Build E2E CICD Pipeline with GitHub Actions, Docker, and Cloud

PoA — GitHub Actions🚀 | Docker📦 | Docker Hub🛢️ | ️Cloud☁️

Image by Author

Hi folks, Are you ready to revolutionize your ML project workflow? Let’s implement an end-to-end CI/CD pipeline for a machine learning project using GitHub Actions, Docker, Docker Hub, and Render. From version control to deployment, let’s unlock the full potential of automation in ML development!

Table of Content
— Introduction
— References
— Pull Production Model from AWS RDS
— Client-side App
— Containerization using Docker
— CI/CD Workflow
— Deploy Container locally
— Deploy Container to Cloud
— GitHub Repository
— Conclusion

Introduction

Automating Machine Learning deployment is truly the icing on the cake, it makes the entire process smoother and more efficient. In this blog, we’ll build a complete end-to-end CI/CD workflow that will retrieve the current production model from the Mlflow backend store — AWS MySQL RDS, create a Docker container, upload it to the Docker hub, and finally host it to cloud. So let’s roll up your sleeves and deep dive into it.

References

In this blog, my major focus would be on the CI/CD pipeline. I won’t deep dive into data tracking, model training, experiments/artifacts tracking, and all. I’ve already covered them in my previous blogs, so please check it out.

Streamline ML Workflow with MLflow️ — I

Streamline ML Workflow with MLflow — II

Deploy MLflow Server on Amazon EC2 Instance

Configure DVC with Amazon S3 Bucket

Pull Production Model from AWS RDS

Now this is one of the crucial stage when we work with MLflow. Let’s implement code to fetch a current production model from AWS RDS, our MLflow backend storage.

MLflow REST API provides direct methods to fetch the current production model. Don’t be confused with the fancy word “current production model" Actually, Mlflow provides a way to register the outperforming models and we can tag them an alias such as @champion, @production, @staging, or @archive. This alias came up with some properties, I won’t deep dive into it here. Check out my previous blogs for in-depth details. Once we assign the alias, we can fetch that particular model via the API and use it in further scripts. For reference, I’ve included the code snippet below.

Client-side App

Now let’s build a Client-side Web App, it’ll run inside our docker container. I’ve used the Streamlit web framework for creating a simple UI. I am not adding any advanced functionality, just put the model inputs and get the output, That’s it!

For reference, I’ve included a code snippet and a UI snapshot below.

Web App Snapshot

Image by Author

Note: My primary focus is implementing a comprehensive end-to-end CI/CD pipeline, rather than emphasizing model performance.

Containerization using Docker

If you’re not sure about the docker containers, don’t worry. I’ve covered it in my blog. Check it out.

Develop & Dockerize ML Microservices using FastAPI + Docker

Let’s containerize both of these files and remember minimizing the size of the Docker container is crucial for efficient deployment and resource management. To achieve this, ensure that you only include the necessary files and dependencies in your container image. Add all the unnecessary files in .dockerignore to avoid them from being copied into the container image. For reference, I’ve included a code snippet of Dockerfile below.

I’ve externally passed the port 8000 for the Streamlit app to avoid any kind of conflict. I added unnecessary files to .dockerignore based on my project structure. Must follow this step.

CI/CD Workflow

So far, everything is perfect. Now let’s build CI/CD workflow. Here, I spent hours to ensure the workflow is bug-free. I’ll split it into two sections. We’ll first deploy it locally before moving it to the cloud.

— Deploy container locally

Let’s implement the workflow. The steps are quite easy, once you go through it, you’ll understand what’s going on. I’ve also added the comments for better understanding.

Line — 6 of the workflow, provides paths linked to our repository. Any changes detected in these directories will trigger the CI/CD workflow.

At line — 16, in jobs section I split the job into two sub-tasks download_model & build_push_docker_image .

At line — 44, I’ve used upload-artifact@v2 action, it’ll save our artifacts generated while executing the sub-task download_model i.e. model & model_details.json file so that we can utilize them further.

At line — 66 I’m downloading the same artifacts to utilize them in the build_push_docker_image stage.

You’ll notice, I used some ls commands in the workflow. That's just for debugging purposes, you can remove them.

From line — 75, we’re creating the Docker container, logging in to the Docker Hub, & pushing the image to the Docker Hub. Once the image is pushed to the Docker Hub, we can pull it via the Docker Desktop and run it in our local system.

For reference, I’ve included the code snippet & snapshot below.

CI/CD Pipeline Execution

Image by Author

Docker Hub Deployment

Image by Author

Image by Author

Running Container on Docker Desktop

Once the image is deployed successfully, just go to Docker Desktop, pull the image, and run it. That’s it!

Image by Author

Image by Author

— Deploy container to Cloud

As of now, at every trigger, we were just uploading the docker image to Docker Hub, but we got to go one step further than even. Now let’s pull the docker image from Docker Hub and host it on the cloud. We’ve plenty of options to host it on cloud such as ECS (Elastic Container Service — by AWS), EKS (Elastic Kubernetes Service — by AWS), GKS (Google Kubernetes Engine), AKS (Azure Kubernetes Service), and many more. In this blog, we’ll host it on Render.

In the next blog, we’ll deploy it using AWS services. So stay tuned!

Step 1. Go to https://render.com/, Sign-up. Once you’ve successfully signed up, click on New & select Web Services.

Image by Author

Step 2. Click on Deploy an existing image from a registry.

Image by Author

Step 3. Enter the Image URL. You’ll get it from the Docker Hub.

Image by Author

Step 4. Select the Free tier, and click on Create Web Service.

Image by Author

Step 5. Once the container is deployed, you can access the hosted app via the link.

Image by Author

Step 6. Congrats🎉, The container hosted successfully.

Image by Author

Step 7. To automate the deployment process, let us incorporate it into the CI/CD workflow. At line — 92, I used the DEPLOY_HOOK you’ll get it from settings. For reference, I’ve included the code snippet & snapshots below.

CI/CD Workflow

Deploy Hook Snapshot

Image by Author

CI/CD Execution Snapshot

Image by Author

Deployment Snapshot — Client Side

Image by Author

Deployment Snapshot—Cloud (Backend)

Image by Author

GitHub Repo

The codebase is available here, just Fork it and start experimenting with it.

GitHub — Built End-to-End CI/CD Pipeline with GitHub Actions, Docker, and Cloud

Conclusion

If this blog has sparked your curiosity or ignited new ideas, follow me on Medium, GitHub & connect on LinkedIn, and let’s keep the curiosity alive.

Your questions, feedback, and perspectives are not just welcomed but celebrated. Feel free to reach out with any queries or share your thoughts.

Thank you🙌 &,
Keep pushing boundaries🚀