Scoring 80 million rows with a 600 MB model took over 9 hours on a 20-node Spark cluster. The pandas UDF loaded the model from storage on every Arrow batch, thousands of times per executor. An iterator UDF that loads once per task cut the job to 28 minutes.
After we redeployed our MLflow tracking server on Kubernetes, every run older than the deploy showed metrics but no artifacts, and model loading failed. The server wrote artifacts to a local directory in the pod. We moved to proxied artifact storage on S3 and recovered what we could.
Data scientists promoted a new churn model by setting the champion alias, as the newer MLflow docs recommend. Our deploy script still loaded models:/churn/Production, which pointed at the old version. Stages are deprecated; we moved the whole pipeline to aliases and added a promotion check.