Our meeting transcripts contained sentences nobody said, usually during silence or hold music, plus the same line repeated a dozen times. Whisper conditions on its own previous output and has no idea what silence is. A VAD pre-filter and stricter decoding thresholds removed most of it.
Half our Argo CD applications showed OutOfSync seconds after every successful sync, and auto-sync kept fighting the HPA over replica counts. The live objects had fields set by controllers and webhooks that Git never mentioned. Targeted ignoreDifferences and server-side diff fixed it.
Every deploy of one service failed with another operation is in progress after someone cancelled a pipeline mid-upgrade. The release was stuck in pending-upgrade. We recovered with helm rollback, then added CI concurrency, timeouts and atomic upgrades so it cannot recur.
Our self-managed cluster pulled images from ECR fine after setup and failed every morning with no basic auth credentials. The imagePullSecret held an ECR token that expires after 12 hours. We switched to the kubelet credential provider so nodes fetch fresh tokens themselves.
In March 2025 the popular tj-actions/changed-files action was compromised and its version tags were repointed to code that printed CI secrets into workflow logs. Three of our repositories used it by tag. This is how we checked exposure, rotated secrets and moved to SHA pinning.
After Docker Hub tightened limits for unauthenticated pulls in April 2025, our self-hosted CI runners started failing a third of builds with toomanyrequests. All runners shared one NAT IP. We authenticated pulls, added a pull-through cache and moved base images into our own registry.
A hotfix made on a checked-out tag vanished when the developer switched back to main, and a reset --hard later took a day of work with it. Nothing was really gone: the reflog still pointed at every commit. Here is how we got them back and what gc changes.