$5,500 A Month Saved From One Grafana Query

Checkly’s platform team cut roughly 300 milliseconds from every ephemeral pod startup by tightening how startup time is measured, then attacking the real bottlenecks inside application initialization. The payoff wasn’t just technical—faster readiness meant fewer pods needed to handle load, translating into about $5,500 saved per month (with the team later citing ~$55,000/month savings after the full change set). The core lesson: when checks run at high scale, tiny startup delays compound into major compute cost.

The optimization journey began with a cost goal tied to FinOps—reducing total compute time per user. Initially, checks ran on long-lived containers in AWS ECS that pulled work from SQS. That architecture avoided startup-time concerns because containers were rarely created, but it introduced two major downsides: autoscaling lacked fine granularity, and the same container handled many clients, forcing extra complexity to clean up data between runs. The team moved to Kubernetes ephemeral pods so each check run started cleanly and scaled precisely, but that shift made pod startup time a direct contributor to both platform cost and user experience.

Early attempts focused on standard advice: smaller container images should start faster. The team found the AWS SDK for JavaScript was inflating images by about 70 MB, since it bundled clients for many AWS services. They switched to AWS SDK v3’s modular imports to ship only what they used. That helped container size, but didn’t deliver the startup-time reduction they expected—CPU time was still higher than anticipated.

To pinpoint the bottleneck, the team added a single log line at the moment pods reported ready to receive SQS tasks, then used Grafana Loki with LogQL to treat those unstructured logs as a metric. Startup time clustered around ~3 seconds, far above the team’s estimate for “JavaScript pods” and especially problematic for “scheduling delay,” when users wait from scheduling a check to receiving results.

One fix changed the order of operations: instead of starting the container only after a queue message triggered it, the container started internally and then pulled from the queue. That removed the long startup delay from the user’s critical path, though it still increased AWS bills.

The biggest “weird trick” came from digging into application initialization. By instrumenting which modules took the longest during startup (using a require-time filter), the team discovered the AWS SDK again—but this time the issue wasn’t image size. Different parts of the system referenced different AWS SDK versions across multiple package.json files. Even with modular imports, those version mismatches prevented internal dependency reuse, forcing repeated downloads and initialization of overlapping code. Aligning all AWS SDK client versions to the same release drastically reduced startup time.

After the changes, pods became ready in about half the previous time, cutting startup by ~300 ms per pod. Because faster readiness reduced the number of ephemeral pods required under load, the compute savings accumulated quickly—ultimately reported as about 25% fewer pods and on the order of tens of thousands of dollars per month. The broader message is that observability plus targeted measurement can expose root causes hiding in dependency graphs and initialization paths, not just in infrastructure settings.