During an incident last year, an engineer added an inbound rule to a security group in the AWS console so a vendor could reach a database for debugging. The incident ended, the rule stayed. Eight months later, someone applied an unrelated Terraform change to the same security group, Terraform removed the rule because it was not in code, and a vendor integration that had quietly started depending on it broke at 3 a.m.

Drift, meaning differences between what your infrastructure code says and what actually exists, is unavoidable: incidents, provider-side defaults, other tools and humans all change things. This guide is about making drift visible quickly, deciding what to do with it, and reducing how often it happens.

Kinds of drift

Not all drift is the same, and the response differs:

Kind Example Usual response
Manual change to managed resource Console edit to a security group Revert or codify
Unmanaged resource Bucket created by hand Import or delete
Provider or service defaults New attribute default after an API change Update code to match
Changes by other automation Autoscaling, tags added by a policy engine Ignore specific attributes
Deleted resource Someone deleted a managed instance Re-create or remove from code

Detecting drift on a schedule

The simplest reliable detector is terraform plan run on a schedule against every root module, without applying:

terraform init -input=false -lockfile=readonly
terraform plan -input=false -lock=false -detailed-exitcode -out=drift.tfplan
code=$?
# 0 = no changes, 1 = error, 2 = changes present (drift or unapplied code)

-detailed-exitcode turns the plan into a signal. -lock=false avoids blocking real applies; drift checks are read-only. Run it with a read-only role so a misconfigured job cannot change anything.

A plan shows differences between code and reality, which includes unapplied code changes on the main branch. To isolate drift specifically, use a refresh-only plan, which reports how real resources changed compared to the state file:

terraform plan -refresh-only -detailed-exitcode

We run both nightly for each of our 60 root modules in a CI matrix and post a summary:

on:
  schedule: [{cron: "0 5 * * 1-5"}]
jobs:
  drift:
    strategy:
      matrix:
        stack: ${{ fromJson(needs.list-stacks.outputs.stacks) }}
    steps:
      - run: ./scripts/drift-check.sh ${{ matrix.stack }}

The script converts the plan to JSON with terraform show -json drift.tfplan and extracts resource addresses and changed attributes, so the report says "aws_security_group_rule.db_ingress: cidr_blocks changed", not 400 lines of plan output.

Triage: revert, codify or ignore

Each drift item needs an owner and a decision within a few days:

  • Revert when the change was a mistake or temporary. Apply the code as is.
  • Codify when the change was right. Update the code to match, review it, apply. The plan becomes empty.
  • Ignore when another system legitimately owns the attribute. Use ignore_changes narrowly:
resource "aws_autoscaling_group" "workers" {
  # desired_capacity is managed by the cluster autoscaler
  lifecycle {
    ignore_changes = [desired_capacity]
  }
}

Never ignore whole blocks like tags or ingress to make a report quiet; that hides exactly the changes you want to see.

Bringing unmanaged resources under control

Plans do not show resources Terraform does not know about. For those, use import blocks, which are reviewable in a pull request and work with plan:

import {
  to = aws_s3_bucket.reports
  id = "example-finance-reports"
}

resource "aws_s3_bucket" "reports" {
  bucket = "example-finance-reports"
}

terraform plan -generate-config-out=generated.tf can draft configuration for imported resources, which you then clean up. To find unmanaged resources in the first place, compare a cloud inventory (AWS Config, Resource Explorer, or a tag query for resources without a managed-by tag) against the resources in state.

Preventing drift

Detection is the safety net. Prevention reduces the volume:

  • Read-only console by default. Engineers get read-only roles for daily work; write access goes through Terraform pipelines.
  • Break-glass access with a trail. In incidents, a time-limited elevated role is available, every session is logged, and closing the incident requires listing manual changes for follow-up.
  • Tag everything managed (managed-by = terraform, stack = <name>), so anyone in the console can see a resource is managed and where.
  • Policy as code. Service control policies or permission boundaries can deny changes to resources tagged managed-by = terraform except from the deploy roles.
  • Small, frequent applies. Drift is much easier to reason about when the main branch is always applied. Stacks with weeks of unapplied changes make every plan noisy.

Measuring progress

Track two numbers per month: open drift items and median days to resolve one. When we started, we had 140 open items across 60 stacks and a median age of several months, because nobody looked. After three months of nightly reports with owners, open items dropped to around 10 at any time, most resolved within two days. The count never reaches zero, and that is fine: the goal is that every difference is known and has a decision.

Common mistakes

Mistake Why it hurts Do this instead
Only finding drift at the next apply Surprise reverts during unrelated changes Scheduled plans with detailed exit codes
Broad ignore_changes Real drift hidden Ignore single attributes owned by other systems
Drift reports nobody owns Alerts become noise Owner per stack and a triage deadline
Write access in the console for everyone Drift by default Read-only roles, break-glass for incidents
Long-unapplied main branch Plans mix drift and pending work Apply often; use refresh-only plans to separate

Key takeaways

  • Expect drift; make it visible daily.
  • Use -detailed-exitcode for detection and -refresh-only to isolate real-world changes.
  • Every drift item gets a decision: revert, codify or narrowly ignore.
  • Import unmanaged resources with reviewable import blocks.
  • Reduce drift with read-only defaults, break-glass access and tagging.

References