gp2 to gp3 migration saved money and made our database slower

A cost optimisation sprint included the classic recommendation: migrate EBS volumes from gp2 to gp3, which costs less per GB and is a live modification with no downtime. We modified 140 volumes in one afternoon. Two days later the data team asked why the nightly reporting jobs, which run heavy queries against our self-managed PostgreSQL primary, had gone from 52 minutes to 1 hour 49 minutes.

The database host metrics for the data volume told the story:

                       before (gp2)    after (gp3 default)
VolumeReadOps/s peak       5,940             3,000 (flat top)
VolumeThroughput peak      238 MiB/s         125 MiB/s (flat top)
VolumeQueueLength avg      2.1               17.8

Flat tops at exactly 3,000 IOPS and 125 MiB/s. We were hitting a limit.

Reproducing it

fio on a test volume of the same size, gp2 versus default gp3:

fio --name=randread --filename=/dev/nvme2n1 --rw=randread --bs=16k --iodepth=32 \
    --numjobs=4 --runtime=60 --time_based --direct=1 --group_reporting

gp2 2000 GiB:            read: IOPS=6000,  BW=93.8MiB/s
gp3 2000 GiB (default):  read: IOPS=3000,  BW=46.9MiB/s

A sequential read test showed 250 MiB/s on gp2 versus 125 MiB/s on gp3 defaults.

What we ruled out

• Query plan changes. Plans were identical; the same queries simply waited longer on IO.
• Instance EBS bandwidth limits. The instance supported far more than 250 MiB/s.
• The volume optimisation phase. After modification, a volume goes through an optimizing state; performance remained low after it completed.

Root cause

gp2 performance scales with size: 3 IOPS per GiB, so a 2,000 GiB volume has a baseline of 6,000 IOPS, and throughput up to 250 MiB/s for large volumes. gp3 decouples performance from size: every volume starts at a baseline of 3,000 IOPS and 125 MiB/s, and you provision more if you need it. Our migration script used defaults for every volume, so large volumes lost half their performance.

What is happening under the hood

For small volumes (under about 1 TiB), gp3 defaults match or beat gp2 baseline and remove burst credits entirely, so the migration is a pure win. For large volumes, the default gp3 configuration is slower than gp2. Provisioning extra IOPS and throughput on gp3 usually still costs less than the equivalent gp2 volume, but you have to do it.

Environment

Minimal reproduction

aws ec2 modify-volume --volume-id vol-0123456789abcdef0 --volume-type gp3
# results in Iops=3000, Throughput=125 regardless of size
aws ec2 describe-volumes --volume-ids vol-0123456789abcdef0 --query 'Volumes[0].[Size,Iops,Throughput]'

The fix

We provisioned gp3 performance to at least match the old gp2 baseline, then sized from metrics:

aws ec2 modify-volume --volume-id vol-0123456789abcdef0 --iops 7000 --throughput 300

We chose 7,000 IOPS and 300 MiB/s after looking at the p99 of the old gp2 metrics plus some headroom for growth.

The migration script was rewritten to calculate gp3 settings from the gp2 volume instead of using defaults:

def gp3_settings(size_gib: int) -> tuple[int, int]:
    gp2_iops = max(100, min(16_000, 3 * size_gib))
    gp2_tput = 250 if size_gib > 334 else 128     # approximate gp2 throughput tiers
    iops = max(3000, gp2_iops)
    tput = max(125, gp2_tput)
    return iops, tput

and then compared with each volume's actual CloudWatch peak usage over 30 days, choosing the higher of "matches gp2" and "p99 usage plus 30 percent". Volumes that never came close to gp2 limits got lower settings.

Cost check for the database volume:

gp2 2000 GiB                              : 2000 x 0.11              = 220.00 USD/month
gp3 2000 GiB, 7000 IOPS, 300 MiB/s        : 2000 x 0.088 + 4000 x 0.0055 + 175 x 0.044
                                          = 176.00 + 22.00 + 7.70    = 205.70 USD/month

Still cheaper than gp2, with more performance. (Prices are for eu-west-1 at the time and are only an illustration; check current pricing for your region.)

Alternatives and trade-offs

• Stay on gp2 for large volumes. Simple, slightly more expensive, keeps size-coupled performance.
• io2 for the database. Higher durability and consistent latency for demanding databases, at a higher price.
• Instance store or local NVMe for reporting replicas. Very fast, data is lost when the instance stops; fine for rebuildable replicas.

How to test the fix

fio on a test volume with the new settings matched the expected IOPS and throughput. The real test was the nightly reports: we ran them manually the same evening after modification finished.

Confirming in production

• Nightly reports returned to 50 minutes the first night after the change.
• VolumeQueueLength averaged under 2 again during reporting.
• Across all 140 volumes, the migration still saved about 18 percent on EBS, compared with 20 percent with defaults. 23 large volumes needed provisioned performance.

Regression guard

• CloudWatch alarms on VolumeQueueLength and on IOPS or throughput reaching 90 percent of provisioned values for more than 15 minutes.
• The volume migration runbook now includes "compare current peak IOPS and throughput with the target configuration".
• Cost changes that touch performance-relevant resources need a performance owner sign-off.

Security implications

None. Volume type modification does not change encryption or access. We confirmed all modified volumes remained encrypted with the same KMS keys.

Prevention checklist

• Before changing storage types, compare provisioned limits, not just prices.
• Size gp3 IOPS and throughput from observed peaks.
• Watch queue length and throttling metrics after storage changes.
• Test heavy batch workloads, not just daytime traffic.
• Treat "cost optimisation" changes as changes, with verification.

References

• Amazon EBS volume types — https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html
• General Purpose SSD volumes (gp2 and gp3) — https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html
• Modifying EBS volumes — https://docs.aws.amazon.com/ebs/latest/userguide/ebs-modify-volume.html
• Amazon EBS pricing — https://aws.amazon.com/ebs/pricing/