Guide to Serve Machine Learning Models in Production

Here’s a simple checklist for people who deploy machine learning models to production. Based on my personal experience.

Before deployment

• Identify Bias: Will the bias affect users? Discuss with the product team.
• Performance Profile: List your hardware requirements: CPU/GPU/RAM/DISK, sustaining usage or burst?
• Model file size: Avoid network congestion. Decide on push-based or pull-based deployment.
• Continuous testing: Test the model on real-world data. Check for over-fitting.

Deploying

• Prepare for rollback.
• Use canary deployment if possible.

After deployment

• Log Collection: collect log for debugging, remove personal information, sample if needed.
• Error handling: No model is perfect. Respond to user feedback, fast.
• Emergency response: Be prepare to fix the application with heuristic rules.
• Observability: Detect concept drift in the real world asap.