End-to-End Deployment of Video to Audio Converting Microservice Application on AWS EKS

GITHUB REPO — https://github.com/Ahmedgit7/video-audio-microservices-python-app

In this project we will deploy a python microservices application which is going to convert video into audio and deploy it on AWS Cloud using tools like Kubernetes, Helm, Docker, rabbit MQ and also databases including postgresql and Mongo DB.

Architecture:

In this application, we have four different microservices: the Auth Service, Converter Service, Gateway Service, and Notification Service. Additionally, we use Helm charts to deploy RabbitMQ, MongoDB, and PostgreSQL.

The application works as follows:

· The user interacts with the API Gateway, which has three endpoints: login, upload, and download.

· First, the user logs in. The Auth Service checks the credentials against the PostgreSQL database. If the credentials are correct, a JWT token is generated, which allows the user to upload a video.

· When a user uploads a video via the upload endpoint, the video is stored in RabbitMQ. The video is then transferred to the Converter Service, which converts the video into audio and stores the converted data in MongoDB (deployed via Helm charts).

· Once the video is converted to audio, the Notification Service sends a notification to the user’s email address.

· Finally, the user can download the converted audio via the download endpoint, accessing the file through the Gateway Service.

INFRASTRUCTURE HAS BEEN BUILT USING TERRAFORM, THE CODES ARE ADDED IN THE GITHUB REPO.

Prerequisites:

1.Install Helm:

Commands:

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3

$ chmod 700 get_helm.sh

$ ./get_helm.sh

2.Python:

Already installed

3. AWS CLI:

Commands:

“curl “https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o “awscliv2.zip”

unzip awscliv2.zip

sudo ./aws/install”

After installing awscli do aws configure

4. Install kubectl:

Commands:

“curl -LO “https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl

chmod +x kubectl

sudo mv kubectl /usr/local/bin”

5. Databases:

Mongodb: if we don’t have mongodb installed in our machine then we need to install mongodb in it.

Although we will be installing mongodb in kubernetes cluster using helm charts, the mongosh command isn’t automatically available on your local machine because it is a client tool that needs to be installed separately from the MongoDB server. The Helm chart installation sets up MongoDB on your Kubernetes cluster, but you still need mongosh locally to interact with it.

So for that we need to install mongodb in our machine using this commands: AS I AM USING UBUNTU:

“curl -fsSL https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add –

apt-key list

echo “deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.4 multiverse” | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list

sudo apt update

sudo apt install mongodb-org”

Postgres Installation:

“sudo apt install postgresql”

STEP 1: Clone the Repository to our machine:

“git clone https://github.com/Ahmedgit7/video-audio-microservices-python-app.git”

STEP 2: Create a Kubernetes Cluster.

Create a Role for kubernetes cluster which include these policies attached.

Create Nodegroup:

Create an IAM Role for this Nodegroup which should contain these below policies:

Nodegroup has been Created:

Enable EBS CSI Addon

Enable addon ebs csi this is for enabling pvcs once cluster is created

Connected Kubernetes Cluster with cli:

“aws eks update-kubeconfig — name <cluster_name> — region <aws_region>”

Installation of MongoDB, Postgres, RabbitMQ:

1.MongoDB

To install MongoDB, set the database username and password in values.yaml,

Then navigate to the MongoDB Helm chart folder and run:

“cd Helm_charts/MongoDB

helm install mongo .”

Persistent volume is also created for the mongodb

2. Postgres

PostgreSQL

Set the database username and password in values.yaml. Install PostgreSQL from the PostgreSQL Helm chart folder and initialize it with the queries in init.sql.

Go to init.sql file and give the email id to which you need to get the notification:

Postgres running:

Update the security group of nodes:

Once the Installation of Postgres and Mongodb has been completed, we will go and connect to Mongodb and postgresql.

“mongosh mongodb://<username>:<pwd>@<nodeip>:30005/mp3s?authSource=admin”

Username and password of — values.yml

Connected to Mongodb

Basically there is nothing in our mongodb db, So its purpose is to store the videos and audio which gets converted.

(Note If you have changed the username and password of mongodb in values.yml then, make sure to change them in configmap of all the manifests file)

Now we will exit from mongodb and get connect to postgresql.

“psql ‘postgres://<username>:<pwd>@<nodeip>:30003/authdb’”

Username and password from values.yml

Successfully connected

Now here we need to create a table called auth_user and insert user credentials into it.

CREATE TABLE auth_user (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    email VARCHAR (255) NOT NULL,
    password VARCHAR (255) NOT NULL
);

--Add Username and Password for Admin User
-- INSERT INTO auth_user (email, password) VALUES ('thomasfookins007helby@gmail.com', '123456');
INSERT INTO auth_user (email, password) VALUES ('ahmedhshaikh@gmail.com', '123456');

Install RabbitMQ:

Go to the Rabbitmq folder and run the following command to install rabbitmq using helm:

“helm install rabbitmq .”

You can see the rabbit mq has been installed successfully and pod is running.

Now we will go access the rabbitmq dashboard on browser.

Access it using — <nodeip>:30004

Username — guest

Password — guest

Once we log in into the rabbit mq dashboard we will go and create a queues.

In RabbitMQ, a queue is a buffer that stores messages. Queues are the main components where messages are sent to, stored, and consumed from. They are integral to RabbitMQ’s messaging system, which allows communication between different parts of a system or between different applications.

Go to Queues One for MP3 and One for Video.

1. MP3 Queue

2. Video Queue

Queues Created

Once we have installed Mongodb, Postgresql and RabbitMQ, the next step is to install our microservices individualy to create a application and validate if we are able to convert video to audio or not.

1.auth-service

Go to Project Folder>src>auth_service

And built the image for this service.

After creating an image we need to push the image to dockerhub, for that first we need to login into our dockerhub. And create one repository.

So for this I have created auth repo.

After creating repo in docker hub go and push the docker image which has been built before to docker hub repo.

2. Converter

Build the image for this using above steps:

3. gateway

4. notification

Apply the manifest file for each microservice:

1. auth manifest file applied.

Auth Pods are running

2. gateway manifest file applied

Gateway pods are running

3. converter-service file is applied

Converter pods are running

4. notification manifest file is applied

Notification pods are running.

Once our Pods are running Not let us see How we can convert the video into audio part.

For that go to cd projectfolder/assets folder where your video is stored.

And then run this below login endpoint:

“curl -X POST http://nodeIP:30002/login -u <email>:<password>”

As you can see we have successfully logged in with the gmail and password which we have stored in the postgresql. So when user logged in using those creds postgresql checked it and provided us the jwt token which we will be using it to upload the video to convert it to audio.

To Upload Run this below Upload Endpoint:

“curl -X POST -F ‘file=@./video.mp4’ -H ‘Authorization: Bearer <JWT Token>’ http://nodeIP:30002/upload”

You can see that our video has been successfully uploaded.

Now to download that converted mp3 file we need to download it using the endpoint given below:

“curl — output video.mp3 -X GET -H ‘Authorization: Bearer <JWT Token>’ http://nodeIP:30002/download?fid=<Generated fid>”

But before running this command we will need the file id which we will get to our gmail, when we upload the video.

As you can have successfully downloaded the mp3 file:

This completes our End to End deployment of Video to Audio converter Microservice Application in EKS Cluster.