MLflow runs listed but artifacts return 404 after redeploy: artifacts stored inside the container

The platform team upgraded our MLflow tracking server Deployment to a newer image. Afterwards, the MLflow UI still listed every experiment and run with parameters and metrics, but artifact panels were empty for every run created before the upgrade, and loading registered models failed:

mlflow.exceptions.MlflowException: The following failures occurred while downloading one or more
artifacts from mlflow-artifacts:/3/9f1c0e2a6b1f4b3d8a7e5c2d1b0a9f88/artifacts/model:
##### File model/MLmodel #####
API request to http://mlflow.mlops.svc:5000/api/2.0/mlflow-artifacts/artifacts/3/9f1c0e2a.../artifacts/model/MLmodel failed with exception HTTPError('404 Client Error: Not Found ...')

The serving job for the production fraud model, which loads the model at startup, failed to restart.

Reproducing it

The server's start command from the Deployment:

command: ["mlflow", "server",
          "--host", "0.0.0.0",
          "--backend-store-uri", "postgresql://mlflow@mlflow-db:5432/mlflow"]

No artifact options at all. Inside the new pod:

$ kubectl -n mlops exec deploy/mlflow -- ls /mlartifacts
ls: cannot access '/mlartifacts': No such file or directory
$ kubectl -n mlops exec deploy/mlflow -- ls ./mlartifacts
3

A fresh, nearly empty ./mlartifacts directory with only runs created after the upgrade.

What we ruled out

• Database loss. The PostgreSQL backend store was intact, which is why runs and metrics were visible.
• Permissions on S3. S3 was never configured; that was the problem.

Root cause

When mlflow server runs with artifact serving enabled (the default in MLflow 2.x) and no --artifacts-destination, it stores artifacts in a local ./mlartifacts directory relative to its working directory. In a container without a persistent volume at that path, that directory lives in the pod's writable layer and disappears when the pod is replaced. The server had been running for seven months without a restart, so artifacts accumulated in a place that looked permanent until the upgrade replaced the pod.

What is happening under the hood

MLflow separates two stores:

• Backend store: runs, parameters, metrics, tags, registry metadata. Ours was PostgreSQL.
• Artifact store: files such as models, plots and datasets. Clients either write directly to the artifact store (for example S3) or, in proxied mode, upload through the tracking server, which writes to --artifacts-destination.

Run records store an artifact URI like mlflow-artifacts:/3/<run_id>/artifacts. With proxied artifacts, the server resolves that against its configured destination. If the destination changes or disappears, the URI still exists in the database and points to nothing.

Environment

Minimal reproduction

docker run --name m -p 5000:5000 ghcr.io/mlflow/mlflow mlflow server --host 0.0.0.0
# log a run with an artifact, then:
docker rm -f m && docker run --name m -p 5000:5000 ghcr.io/mlflow/mlflow mlflow server --host 0.0.0.0
# run metadata from a persistent DB would remain; artifacts are gone

The fix

Recovery. The old pod was gone, but two sources helped: the fraud model's artifacts were still in the serving job's model cache volume from its last successful start, and several training jobs had local copies in their own pipeline workspaces. We re-uploaded the production models and their key evaluation reports into the new artifact store under the original run paths. Older experiment artifacts were lost; their metrics and parameters survived in PostgreSQL.

Durable artifact storage with proxying.

command: ["mlflow", "server",
          "--host", "0.0.0.0",
          "--backend-store-uri", "postgresql://mlflow@mlflow-db:5432/mlflow",
          "--artifacts-destination", "s3://example-mlflow-artifacts/",
          "--serve-artifacts"]
env:
  - name: AWS_REGION
    value: eu-west-1

The server's service account uses IAM Roles for Service Accounts with access to that one bucket. Clients keep using mlflow-artifacts:/ URIs and never need S3 credentials themselves, which suits notebooks well.

Bucket protections. Versioning on, a lifecycle rule that expires noncurrent versions after 90 days, and a deny on s3:DeleteBucket. Backups of the PostgreSQL backend store were already in place; they now run with the same retention as the bucket so metadata and artifacts can be restored together.

Alternatives and trade-offs

• Persistent volume for ./mlartifacts. Fixes the restart problem but ties artifacts to one volume in one zone, and blocks scaling the server to multiple replicas.
• Direct client access to S3 (--default-artifact-root s3://... without proxying). Less load on the server, but every client needs S3 credentials.

How to test the fix

We logged a run with an artifact, deleted the server pod, waited for the replacement, and downloaded the artifact successfully. A model was registered, the pod restarted again, and mlflow.pyfunc.load_model("models:/fraud@champion") worked.

Confirming in production

• Artifacts for all runs since the change are in S3 and survive restarts and upgrades.
• The production fraud model loads from the registry again; serving restarted successfully.

Regression guard

• A synthetic check every hour logs a tiny artifact to a canary experiment and reads it back.
• The MLflow Helm values in our platform repository require artifactsDestination; the chart fails to render without it.
• Upgrades of stateful platform tools need a written "where does the data live" check in the change request.

Security implications

Proxied artifacts mean the tracking server is now the gate to model files, so it must have authentication. Our MLflow server sits behind an SSO proxy, and only the server's role can read and write the bucket. Model artifacts can contain pickled objects that execute code when loaded, so write access to the artifact store is effectively code execution in serving; the bucket policy denies writes from anything except the server role.

Prevention checklist

• Configure an explicit artifact store for MLflow; never rely on the default local path in containers.
• Back up the backend store and artifact store together.
• Test platform tools by restarting them before trusting them with data.
• Use proxied artifacts to keep storage credentials out of clients.
• Restrict write access to model artifacts as tightly as code deployment.

References

• MLflow tracking server and artifact stores — https://mlflow.org/docs/latest/tracking/server.html
• MLflow artifact stores — https://mlflow.org/docs/latest/tracking/artifacts-stores.html
• MLflow server CLI options — https://mlflow.org/docs/latest/cli.html#mlflow-server
• IAM roles for service accounts (EKS) — https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html