We switched our semantic search to a cheaper 1024-dimension embedding model and every insert and query failed against the 1536-dimension column. Worse, the half-finished fix would have mixed incompatible vectors. How we migrated with versioned columns, a backfill and dual reads.
A one-off data backfill filled our Postgres RDS instance, storage autoscaling grew it from 400 GB to 2 TB, and after we deleted the temporary tables the bill stayed. RDS allocated storage cannot be reduced in place. We shrank it with a blue/green switchover and capped autoscaling.
A trading system that cannot explain its balances to the cent will eventually make decisions on wrong numbers. This guide shows how we keep an internal double-entry ledger for trades, fees, funding and transfers, reconcile it with exchange balances, and handle the differences we find.
Page 1 of our activity feed loaded in 9 ms, page 4,000 in 3.2 seconds, and a partner's sync job was walking every page. OFFSET makes Postgres read and discard every earlier row. We switched to keyset pagination with a tie-breaker and a matching index.
Every value in our orders API was a bound parameter, except the sort column, which was pasted into ORDER BY from the query string. A pentest proved it was injectable. We replaced it with an allowlist map and added tests that fuzz the sort input.
We moved all EBS volumes from gp2 to gp3 to save 20 percent, as every cost guide recommends. Two days later, nightly reports on our self-managed Postgres took twice as long. The 2 TB gp2 volume had 6,000 IOPS and 250 MiB/s; gp3 defaults are 3,000 and 125. Provisioning both fixed it.
Every deploy and every traffic burst, the first requests to our booking API timed out and Postgres started refusing clients. The handler did heavy setup and opened a fresh connection per invocation. Moving init out of the handler and adding RDS Proxy fixed it.
A routine database failover lasted 20 seconds, but our API stayed down for 41 minutes. Three layers each made up to three attempts with no delay, multiplying load on the database by 27. Here is the math, the fix and how we proved it.
EXPLAIN ANALYZE prints a wall of nodes, costs and loops that most of us skim for the word Seq Scan. This guide walks a real slow query node by node: estimates vs actuals, loops, buffers, and the fixes each pattern points to.
After importing 410,000 tickets with their original ids, most new tickets failed with duplicate key value violates unique constraint tickets_pkey, while a few succeeded. The id sequence was still at 1. Here is why, and how we resynced every sequence safely.
Every ten minutes our catalog API spiked to 4-second responses and the database hit 100% CPU. A hot Redis key expired and 600 requests rebuilt it at once. Single-flight, a short Redis lock, stale-while-revalidate and TTL jitter fixed it.
Small tenants in our RAG product got 2 or 3 chunks back instead of 10, and sometimes none. HNSW finds ef_search candidates first and applies the tenant filter afterwards. Iterative index scans in pgvector 0.8 plus partial indexes for big tenants fixed recall.
A practical guide to time handling that survives DST switches: when to store UTC instants, when to keep local time plus an IANA zone, how gaps and overlaps behave, which database types to use, and how to test with fixed clocks.
Semantic search in our docs assistant took 1.9 seconds and returned odd rankings. The query sorted by L2 distance while the HNSW index was built for cosine, so Postgres scanned every row. Matching the operator to the opclass brought it to 14 ms.
A one-line ALTER TABLE in a deploy hung for eleven minutes and took the whole site down with it. An idle-in-transaction worker held a lock, the ALTER queued behind it, and every query queued behind the ALTER. Here is the diagnosis and the timeouts we now set.
Our API container crashed with ECONNREFUSED on about one in five CI runs and on every fresh laptop setup. depends_on only waited for the Postgres container to start, not to accept connections. A real healthcheck, service_healthy and a small retry loop fixed it.
On November 3, 2024, our 01:30 billing run fired at 01:30 CDT and again at 01:30 CST, charging 1,284 subscriptions twice. The in-house scheduler matched local wall-clock time. We moved schedules to UTC and added idempotency keys, a run ledger and a unique constraint.