Production gets attention because it serves customers. Non-production gets copied from production, scaled for a load test three years ago, and left running. When we tagged our accounts by environment and looked, development, staging, QA and preview environments together were 38 percent of our cloud bill. Most of that capacity was used for about 50 hours a week.

Over two quarters we cut non-production spend by 57 percent. Engineers noticed only one change: preview environments now start in about four minutes instead of being always on. This guide covers what we did, in order of savings per effort.

1. Measure by environment first

You cannot cut what you cannot see. Make environment a first-class allocation dimension:

  • Separate accounts or projects per environment where possible; it makes allocation trivial.
  • An environment tag (or label) on everything else, activated as a cost allocation tag.
  • A weekly report of cost by environment and by team.

Our first report revealed two surprises: a "temporary" load testing environment that had been running for 14 months, and a staging database cluster larger than production's.

2. Turn things off at night and on weekends

A resource needed 50 hours a week but running 168 hours wastes 70 percent of its cost. Schedules are the biggest, simplest lever:

  • Instances and databases: AWS Instance Scheduler, or simple EventBridge schedules that stop and start tagged resources. RDS instances can be stopped for up to seven days at a time, after which they start automatically, so schedules must stop them again.
  • Kubernetes: scale non-production node pools to zero outside working hours, or scale Deployments to zero with a scheduler and let the cluster autoscaler remove nodes.
  • Opt-out, not opt-in: everything in non-production accounts follows the schedule unless tagged schedule=always-on with an owner and a reason. Every always-on tag is reviewed monthly.
# EventBridge Scheduler targets, simplified
stop:  cron(0 20 ? * MON-FRI *)   # 20:00 local
start: cron(0 7 ? * MON-FRI *)    # 07:00 local

Teams across time zones needed wider windows; we defined schedules per team rather than one global window. A Slack command lets anyone start their environment outside hours for a few hours.

3. Replace long-lived shared environments with ephemeral ones

Long-lived feature environments accumulate. Ephemeral preview environments, created per pull request and destroyed on merge or after inactivity, cost money only while someone uses them. Our setup:

  • A namespace per pull request in a shared non-production cluster, created by CI.
  • Shared backing services (one database server with a database per preview, one message broker with a virtual host per preview) instead of a full stack per preview.
  • Automatic deletion when the PR closes, plus a reaper that deletes previews idle for 48 hours.

Watch out for leaks: deleted namespaces can leave volumes and load balancers behind (see our post on orphaned EBS volumes), so the reaper checks cloud resources too.

4. Right-size to non-production reality

Staging does not need production's capacity unless you load test there:

  • Smaller instance sizes and fewer replicas. One replica per service is usually enough in development.
  • Single-AZ databases without read replicas.
  • Burstable instances for idle-most-of-the-time services.
  • Spot capacity for non-production Kubernetes nodes and CI runners.

For load tests, create a production-sized environment for the test and destroy it afterwards.

5. Shrink the data

Full copies of production data in non-production are expensive (storage, snapshots, longer restores) and risky. Alternatives:

  • Seed data generated by scripts for development.
  • A subset of production, anonymised, for staging: for example, 5 percent of customers with all their related rows, with personal data masked.
  • Short retention for logs and metrics in non-production: 7 days instead of 30.

6. Guardrails

  • Service control policies that deny the largest instance families and GPU instances in development accounts, with an exception process.
  • Budgets per non-production account with daily alerts.
  • Terraform policy checks for non-production: no multi-AZ databases, no provisioned IOPS, NAT gateways only where required.

Results

Over two quarters, non-production spend fell by 57 percent. Schedules delivered about half of that, ephemeral previews and the removal of long-lived feature environments about a quarter, and right-sizing and data reduction the rest. Developer satisfaction survey scores for environments went up slightly, mostly because previews became consistent and fast to create. The one complaint, previews taking a few minutes to start in the morning, was solved by pre-warming the non-production node pool fifteen minutes before the schedule start.

Keeping it from growing back

Savings erode quietly. Every quarter we re-run the cost-by-environment report, review all always-on exceptions, and check that new accounts inherited the schedule and policy defaults. New services are created from templates that already include non-production sizing, so the cheap option is also the default one.

Common mistakes

Mistake Why it hurts Do this instead
Opt-in schedules Nobody opts in Opt-out with reviewed exceptions
Copying production sizing to staging Paying for capacity never used Right-size, create big environments only for load tests
Long-lived feature environments Accumulate and are forgotten Ephemeral per-PR environments with reapers
Full production data copies Cost and privacy risk Generated or anonymised subsets
Forgetting RDS auto-restart Stopped databases come back after 7 days Schedule repeated stops

Key takeaways

  • Split costs by environment before optimising.
  • Schedules are the largest, simplest saving for non-production.
  • Prefer ephemeral environments with automatic cleanup.
  • Size non-production for its real use, and shrink its data.
  • Add guardrails so savings do not erode.
  • Review exceptions and defaults every quarter, because waste comes back one reasonable request at a time.

References