Argo CD application always OutOfSync after sync: defaulted fields and HPA replicas

Our Argo CD dashboard had become noise: 31 of 70 applications showed yellow OutOfSync most of the time, even right after a successful sync. Engineers had stopped trusting the status, which defeats the point of GitOps. Worse, for a few apps with auto-sync and self-heal enabled, pods were being scaled down every few minutes during traffic peaks.

The app diff for one service showed two kinds of difference:

 spec:
-  replicas: 2
+  replicas: 7
 template:
   metadata:
     annotations:
+      sidecar.istio.io/status: '{"initContainers":["istio-init"],...}'

Reproducing it

On a test app: sync, wait ten seconds, refresh. OutOfSync again. With self-heal on, Argo CD re-applied replicas: 2, the HorizontalPodAutoscaler raised it back to 7 a minute later, and the cycle repeated. The event log for the Deployment showed scaling events alternating between horizontal-pod-autoscaler and the Argo CD sync.

What we ruled out

• Someone editing resources by hand. kubectl get deploy -o yaml --show-managed-fields showed the fields owned by the HPA controller and the Istio injector, not by humans.
• Helm template drift. helm template output matched what Argo CD considered desired.
• Argo CD cache issues. A hard refresh showed the same diff.

Root cause

Two separate things wrote to the live objects after Argo CD applied them. The HPA controller owns spec.replicas for Deployments it scales, so whatever Git says about replicas is overwritten. And mutating admission webhooks (Istio sidecar injection, a policy engine adding labels) added fields at creation time. Argo CD compares desired manifests with live state, saw extra or different fields, and reported OutOfSync. With self-heal on, it then "fixed" the replicas back to Git's value.

What is happening under the hood

GitOps assumes Git is the single owner of a resource's configuration. In a real cluster, several actors share ownership: autoscalers, admission webhooks, operators that set defaults, and the API server itself, which fills in default values. Kubernetes tracks this in managedFields, which records which manager set which field.

Argo CD's default diff compares normalised manifests on the client side. It knows how to ignore many server defaults, but not fields added by your webhooks or controllers. Argo CD can also do the diff on the server side using a dry-run server-side apply, which lets the API server compute the result including defaults and webhook mutations. Fields owned by other managers can then be ignored by manager name.

Environment

Minimal reproduction

# Deployment in Git sets replicas, an HPA targets it, auto-sync + selfHeal enabled
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 2
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: api}
  minReplicas: 2
  maxReplicas: 20

The fix

1. Stop declaring replicas when an HPA owns them. The cleanest fix for the scaling fight is to remove spec.replicas from the Deployment template in Git when an HPA is enabled. Our chart now does:

spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}

Without the field in Git, Argo CD has nothing to enforce, and the HPA is the only owner.

2. Ignore fields owned by other managers, and make sync respect that.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      managedFieldsManagers:
        - kube-controller-manager      # HPA scaling writes through this manager
    - group: apps
      kind: Deployment
      jqPathExpressions:
        - .spec.template.metadata.annotations."sidecar.istio.io/status"
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - RespectIgnoreDifferences=true

ignoreDifferences affects the diff and status. RespectIgnoreDifferences=true makes the sync itself leave those fields alone, which is what stops self-heal from resetting replicas on existing resources.

3. Server-side diff for webhook defaults. With ServerSideDiff=true, Argo CD asks the API server what the result of applying the manifest would be, including mutating webhooks, so most webhook-added fields no longer show as differences. We enabled it per app first, then globally in argocd-cmd-params-cm once we were confident.

Alternatives and trade-offs

• Global ignore rules in argocd-cm for common cases like /spec/replicas on all Deployments. Less repetition, but they hide differences in apps where Git should own replicas.
• Disable self-heal. Removes the fight, keeps the noise, and gives up drift correction.
• Move mutations into Git. For policy-added labels, rendering them in the chart makes Git the owner. That works for static values, not for sidecar status annotations.

How to test the fix

On a staging app: sync, then generate load to make the HPA scale to 6 replicas, then trigger a sync with an unrelated change:

argocd app sync api --prune
kubectl -n api get deploy api -o jsonpath='{.spec.replicas}'      # stays 6
argocd app get api -o json | jq -r .status.sync.status             # Synced

Confirming in production

• OutOfSync applications dropped from 31 to 0 at rest. The yellow status now means a real difference.
• No more scale-downs caused by Argo CD: Deployment events show scaling only from the HPA.
• We checked that real drift is still detected by editing an env var by hand on a test app; Argo CD flagged it and self-heal reverted it.

Regression guard

• A chart linter rule fails if autoscaling.enabled is true and replicas is rendered.
• A weekly report lists apps that have been OutOfSync for more than an hour.
• ignoreDifferences entries require a comment explaining which controller owns the field.

Security implications

Ignoring differences is also a way to hide changes. Broad ignore rules (whole spec, or all annotations) would let someone change a container image or security context by hand without Argo CD noticing. We keep ignore rules as narrow as possible and review them like code. Self-heal remains on for everything else, which is an important control against manual drift.

Prevention checklist

• Decide which actor owns each field: Git, a controller, or a webhook.
• Do not render replicas when an HPA manages the workload.
• Use managedFieldsManagers or narrow JQ paths, never broad ignores.
• Turn on RespectIgnoreDifferences when using self-heal with ignores.
• Prefer server-side diff to handle webhook mutations.

References

• Argo CD diffing customization — https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/
• Argo CD sync options, RespectIgnoreDifferences — https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/
• Argo CD server-side diff — https://argo-cd.readthedocs.io/en/stable/user-guide/diff-strategies/
• Kubernetes server-side apply and field management — https://kubernetes.io/docs/reference/using-api/server-side-apply/