Our versioning workflow created a tag and pushed it, and the release workflow that listens for tags never started. Events created with the default GITHUB_TOKEN do not trigger new workflow runs. We switched to a GitHub App token for the push and made the release explicit with workflow_call.
Our Ray preprocessing job for image embeddings failed after an hour with OutOfMemoryError. Ray schedules tasks by CPU count, so it packed 16 tasks that each needed 3 GB onto nodes with 30 GB. Declaring memory per task, smaller blocks and a right-sized object store fixed it.
We moved our EKS batch node pool to Spot and cut its cost by 60 percent. Within a week, nightly jobs failed most nights: a single instance type, no interruption handling and jobs with no checkpoints. Diversified instances, graceful draining and checkpointing made Spot reliable.
Prompt and response logs are the fastest way to build an accidental PII database. This guide shows where personal data enters an LLM feature, how to redact it before logging and before the provider, and how to keep retention and access under control.
Our daily signals disagreed with the exchange chart, and intraday signals appeared and disappeared within the same bar. We resampled UTC millisecond timestamps in local time and included the still-forming last candle. UTC everywhere and using only closed candles fixed both problems.
A new subdomain's certificate sat in pending for hours with a self check error: wrong status code 404, expected 200. The challenge solver created its Ingress with an ingress class our controller ignored. Setting the class on the solver and checking the redirect path fixed issuance.
A new model that wins offline can still lose in production, through skew, latency, or effects on users that offline data cannot show. This guide covers a staged rollout for models: offline gates, shadow traffic, canaries on business metrics, A/B tests when needed, and fast rollback.
A new operations dashboard refreshed every 15 minutes and our BigQuery bill jumped by 9,000 USD in a month. Each tile ran SELECT star over a 40 TB events table without a partition filter. Column pruning, required partition filters and a bytes billed cap fixed it.
We added a semantic cache to our help assistant to cut LLM cost. Within days a user asking how to cancel a subscription got instructions for cancelling an order. The two questions were 0.92 similar. We added exact-match first, scoped keys, intent gating and a measured threshold.
Our perpetual futures bot reported a monthly PnL 6 percent higher than the exchange's account history. It ignored trading fees on some fills, never recorded funding payments, and valued open positions at last trade price instead of mark price. Building PnL from the income ledger fixed it.
GitOps says everything lives in Git, and secrets are the exception that breaks the rule. This guide compares encrypting secrets in Git with SOPS or Sealed Secrets against referencing them from a secret manager with External Secrets Operator or the CSI driver, including rotation and bootstrap.
After we redeployed our MLflow tracking server on Kubernetes, every run older than the deploy showed metrics but no artifacts, and model loading failed. The server wrote artifacts to a local directory in the pod. We moved to proxied artifact storage on S3 and recovered what we could.
After we turned on streaming for our assistant, tool calls failed with JSON parse errors and sometimes ran with another call's arguments. We were parsing argument fragments per chunk and keying them by arrival order. Accumulating by index fixed it.
A misconfigured batch job launched 60 GPU instances on a Friday evening. Our monthly budget alert fired on Sunday afternoon, after 14,000 USD was spent. Monthly thresholds and billing data lag make budgets slow. We added daily budgets, anomaly detection and real-time usage guardrails.
A hedging bot meant to short perpetual futures placed spot sell orders instead, selling inventory it was supposed to hedge. In ccxt's unified symbols, BTC/USDT is spot and BTC/USDT:USDT is the linear perpetual. Explicit market types and assertions on market metadata fixed it.
During a traffic spike our checkout service stayed at 3 pods while CPU was pegged. kubectl get hpa showed <unknown>/70% for weeks. A sidecar container added by a platform change had no CPU request, so utilisation could not be computed. Requests on every container fixed it.
A research agent called the same search tool 412 times in one run and spent more overnight than the whole feature spent in a normal week. The tool returned an empty string on errors, the loop had no step limit and nothing tracked spend per run.
Our batch classification job with a small open model produced good output for the longest prompt in each batch and nonsense for the rest. Transformers warned that right-padding was detected for a decoder-only model. Left padding, a proper pad token and the attention mask fixed it.
We spent a quarter tagging every resource with team and service, then grouped Cost Explorer by tag and saw 92 percent under No tag key. User-defined tags must be activated as cost allocation tags, they apply from activation onward, and some spend cannot be tagged at all.
During a volatile minute, our place-order request timed out and the bot retried. Both requests reached the exchange, and we held twice the intended position. A timeout does not mean the order failed. Client order ids, query-before-retry and an unknown state fixed it.
We lowered a rate limit in a ConfigMap, Argo CD reported Synced, and the service kept using the old limit for two days. Environment variables from ConfigMaps are read once at container start, and our file mount used subPath, which never updates. A checksum annotation fixed rollouts.
Our CI compared LLM summaries against stored snapshots at temperature 0 and failed about one run in five. Temperature 0 is not a determinism guarantee on hosted models. We replaced string snapshots with property checks and a scored eval.
A churn model reached 0.97 AUC in validation and 0.68 in production. The training set joined customer features from the current snapshot, so it included support tickets and plan downgrades that happened after the prediction date. Point-in-time joins brought validation to an honest 0.81.
We moved all EBS volumes from gp2 to gp3 to save 20 percent, as every cost guide recommends. Two days later, nightly reports on our self-managed Postgres took twice as long. The 2 TB gp2 volume had 6,000 IOPS and 250 MiB/s; gp3 defaults are 3,000 and 125. Provisioning both fixed it.
Our bot believed two orders were still open when they had filled hours earlier. The exchange closes websocket connections after 24 hours; our reconnect logic resubscribed but never checked what happened while disconnected. Reconnect with backoff plus REST reconciliation fixed the state drift.