Jenkins Master-Slave Architecture and Agents | Day 28 of 90 Days of DevOps

8 min readSep 2, 2023

Hello everyone and welcome to Day 28 of 90 Days of DevOps. In this blog, I will explain what is Jenkins master-slave architecture and how to set up Jenkins agents on AWS EC2 machines. I will also demonstrate how to run a pipeline job for a Node.js todo app on a Jenkins agent.

What is Jenkins Master?

Jenkins master is the central server that manages the entire Jenkins system. It is responsible for:

Scheduling and executing build jobs
Monitoring and managing Jenkins agents
Storing configuration data and build artifacts
Providing a web interface and REST API for users and administrators
Installing and updating plugins and tools

What is Jenkins Agent?

Jenkins agent is a machine or a container that connects to the Jenkins master and executes build jobs on behalf of the master. It is also known as a slave or a node. An agent can be:

A physical or virtual machine running any operating system
A Docker container running inside a Docker host or a Kubernetes cluster
A cloud instance provisioned by Jenkins plugins such as EC2, Azure, GCP, etc.

The benefits of using Jenkins agents are:

Scalability: You can add more agents to handle more concurrent build jobs and reduce the load on the master
Isolation: You can run different types of build jobs on different agents with different environments and tools
Security: You can restrict the access and permissions of agents to protect sensitive data and resources

How to Set Up Jenkins Master and Agent on AWS EC2 Machines?

In this section, I will show you how to create two EC2 instances for Jenkins master and agent, and how to connect them using SSH keys.

Step 1: Create Two EC2 Instances

First, you need to create two EC2 instances on AWS. One will be the master and the other will be the agent. You can use any Linux distribution that supports Java, such as Ubuntu, CentOS, Amazon Linux, etc. For this demo, I will use Ubuntu 20.04 LTS.

To create an EC2 instance, follow these steps:

Log in to your AWS console and go to the EC2 service
Click on the Launch Instance button
Choose Ubuntu Server 22.04 LTS (HVM), SSD Volume Type as the AMI
Choose t2.micro as the instance type (or any other type that suits your needs)
Configure the security group to allow inbound traffic on port 22 (SSH), port 80 (HTTP), and port 8080 (Jenkins)
Create or select an existing key pair and download the private key file (.pem)
Review and launch the instance

Repeat these steps to create another EC2 instance for the agent. Make sure you use the same key pair for both instances.

Step 2: Connect Master with Agent using SSH Keys

Next, you need to establish a secure connection between the master and the agent using SSH keys. This will allow the master to authenticate and communicate with the agent without requiring a password.

To do this, follow these steps:

Log in to your master instance using SSH. For example: ssh -i my-key.pem ubuntu@master-ip
Generate a public key for the master using ssh-keygen. You can accept the default options or customize them as you wish.

Copy the public key of the master to the authorized_keys file of the agent using ssh-copy-id. For example: ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu@agent-ip

Verify that you can log in to the agent from the master without a password using ssh. For example: ssh ubuntu@agent-ip

Step 3: Install Jenkins on Master and Agent

Now that you have two EC2 instances connected by SSH keys, you need to install Jenkins on both of them. Jenkins requires Java to run, so you also need to install Java on both instances.

To install Java and Jenkins on Ubuntu, follow these steps:

Update the package list using sudo apt update
Install Java using sudo apt install openjdk-11-jdk
Add the Jenkins repository key using wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
Add the Jenkins repository URL using sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
Update the package list again using sudo apt update
Install Jenkins using sudo apt install jenkins
Start and enable Jenkins service using sudo systemctl start jenkins and sudo systemctl enable jenkins

Repeat these steps on both master and agent instances.

Step 4: Configure Jenkins on Master and Agent

After installing Jenkins on both instances, you need to configure them to work together. You need to do the following:

Set up the initial password and plugins on the master
Create an agent on the master and configure its launch method and credentials
Connect the agent to the master and verify its status

Set Up Initial Password and Plugins on Master

To set up the initial password and plugins on the master, follow these steps:

Access the Jenkins web interface on your browser using http://master-ip:8080
Find the initial password using sudo cat /var/lib/jenkins/secrets/initialAdminPassword on the master instance
Enter the initial password and click Continue
Choose Install suggested plugins or Select plugins to install as you prefer
Create an admin user and click Save and Continue
Confirm the Jenkins URL and click Save and Finish
Click Start using Jenkins

Create an Agent on Master and Configure its Launch Method and Credentials

To create an agent on the master and configure its launch method and credentials, follow these steps:

Go to Manage Jenkins > Manage Nodes and Clouds > New Node

Enter a name for the agent (e.g. agent-1) and choose Permanent Agent
Click OK
Enter the following configuration for the agent:
Remote root directory: /home/ubuntu (or any other directory that you want to use for Jenkins workspace)

Labels: agent (or any other label that you want to use to identify the agent)
Usage: Only build jobs with label expressions matching this node (or any other option that suits your needs)
Launch method: Launch agents via SSH
Host: agent-ip (the public IP address of your agent instance)
Credentials: Add > Jenkins > SSH Username with private key > Enter username (ubuntu) and private key (paste the content of your key pair file .pem) > Add > Choose the credential that you just added

Host Key Verification Strategy: Non-verifying Verification Strategy (or any other option that suits your security needs)

Click Save

Connect the Agent to the Master and Verify its Status

To connect the agent to the master and verify its status, follow these steps:

Go to Manage Jenkins > Manage Nodes and Clouds > agent-1 > Launch Agent
Wait for a few seconds until you see the message “Agent successfully connected and online”
Go back to Manage Jenkins > Manage Nodes and Clouds and check that the agent is online and idle

Here is an example of how it looks like:

How to Run a Pipeline Job for a Node.js Todo App on a Jenkins Agent?

In this section, I will show you how to run a pipeline job for a Node.js todo app on a Jenkins agent. The pipeline job will perform the following steps:

Clone the GitHub repository of the Node.js todo app
Install the dependencies using npm
Run the tests using npm
Build a Docker image using Dockerfile
Deploy the Docker image using docker-compose

The GitHub repository of the Node.js todo app is https://github.com/ajitfawade/node-todo-cicd.

Step 1: Create a Pipeline Job on Master

To create a pipeline job on master, follow these steps:

Go to New Item > Enter a name for the job (e.g. node-todo-pipeline) > Choose Pipeline > Click OK
Enter the following configuration for the job:
General: Check GitHub project and enter https://github.com/ajitfawade/node-todo-cicd as Project url
Build Triggers: Check GitHub hook trigger for GITScm polling (or any other option that suits your needs)
Pipeline: Choose Pipeline script from SCM as Definition
SCM: Choose Git as SCM
Repository URL: https://github.com/ajitfawade/node-todo-cicd.git
Credentials: Add > Jenkins > Username with password > Enter your GitHub username and password or token > Add > Choose the credential that you just added
Branch Specifier: */main (or any other branch that you want to build)
Script Path: Jenkinsfile (or any other path that contains your pipeline script)
Click Save

Step 2: Write a Pipeline Script in Jenkinsfile

To write a pipeline script in Jenkinsfile, follow these steps:

Open your GitHub repository of the Node.js todo app in your code editor or IDE
Create a file named Jenkinsfile in the root directory of your repository
Write the following code in Jenkinsfile:

pipeline {
    agent none // This means that the pipeline will not run on any agent by default
    stages {
        stage('Clone') {
            agent { label 'agent' } // This means that this stage will run on an agent with the label 'agent'
            steps {
                echo "cloning the repository"
                git 'https://github.com/ajitfawade/node-todo-cicd.git' // This will clone the GitHub repository to the agent's workspace
            }
        }
        stage('Build') {
            echo "Building the image"
            steps {
                sh 'docker build -t todo-app .'
            }
        }
        stage('Deploy') {
            steps {
                echo "Deploying todo app"
                sh 'docker-compose down'
                sh 'docker-compose up -d'
            }
        }
    }
}

Step 3: Run the Pipeline Job on Master and Verify the Results

To run the pipeline job on master and verify the results, follow these steps:

Go to your pipeline job on Jenkins web interface and click Build Now
Wait for the pipeline to complete and check the console output for each stage
Go to your Docker Hub account and check that the Docker image is pushed successfully
Go to your agent instance and check that the Node.js todo app is running using docker ps and curl localhost:3000