Automate infrastructure provisioning using Terraform, GitHub & Jenkins

Before we deve deep into the project, lets define some important terms

Definition of important terms

1.Jenkins
Jenkins is an open-source automation server used to automate tasks associated with building, testing, and delivering/deploying software.

3.Terraform
Terraform is a tool for building, changing, and versioning infrastructure that has an open-source and enterprise version.

Prerequisites:

• Awa account: If you don’t have one, you can register Here
• GitHub Account
• Install Terraform
• Aws CLI
• IDE (we are using Visual studio Code)
• AWS user with Admin permissions

This tutorial we will walks you through the process of deploying a VPC and it associate subnet in an aws environment. You will launch an EC2 instance, install Jenkins on that instance, and configure Jenkins to automatically spin up Jenkins agents if build abilities need

Explanation

Jenkins Pipeline:- supports Continuous Integration and Continuous Deployment by automating all the steps like build, testing, and deploying. This can be achieved with a Jenkins file which will be committed to GitHub or any project’s code repository.

Jenkins file:-The Jenkins file contains the definition of the pipeline with multiple stages where each stage represents a task or set of tasks to be performed. I will explain each stage individually later in this article.

GitHub:-We used GitHub as our code repository for this project. we committed all our code to the main branch.

Terraform:-Terraform is a free open-source infrastructure as a code tool that can create, change and improve the infrastructure. we used it to automate the process of creating a vpc and it associate subnets in our aws cloud environment.

Before we move on to lunch an Ec2 instance, install and configure jenkins and its plugin , we need to make sure that our terraform code function perfectly.
If you are new to terraform, please follow up with this tutorial to set up, write and Deploy terraform code in your cloud environment manually. Your can as well fork the repository. from our github account . If that is what you intend to do , Please make sure you create and match your s3 bucket name with that in the terraform code.
Once that is done, your are ready to procced with the automation process.

Get started

With above the Prerequisites in place, we will move straight to install and configure Jenkins on the ec2 instance running in our cloud environment. if you need help lunching an ec2 instance, check out this tutorial. Please, make sur that port 22 and 8080 are open and that you can ssh into your ec2 instance.

Installing and configuring Jenkins

To facilitate things, you can insert Jenkins installation script as user data (User data script is the easiest and most popular way to send instructions to an instance at launch.) while lunching an aws EC2 Instance. if you choose to do so, here is the installation script.

                                #!/bin/bash
                                sudo yum update –y
                                sudo wget -O /etc/yum.repos.d/jenkins.repo \
                                https://pkg.jenkins.io/redhat-stable/jenkins.repo
                                sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key
                                sudo yum upgrade
                                sudo dnf install java-17-amazon-corretto -y
                                sudo yum install jenkins -y
                                sudo systemctl enable jenkins
                                sudo systemctl start jenkins
                                

You can check the status of the Jenkins service using the command:

sudo systemctl status jenkins

Downloading and Installing Jenkins step by step after lunching an ec2 Instance

1. Ensure that your software packages are up to date on your instance by using the following command to perform a quick software update:

    sudo yum update –y

2. Add the Jenkins repo using the following command:

   sudo wget -O /etc/yum.repos.d/jenkins.repo \
                                           https://pkg.jenkins.io/redhat-stable/jenkins.repo 

3. Import a key file from Jenkins-CI to enable installation from the package:

   sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key 

    sudo yum upgrade

4. Install Java (Amazon Linux 2023):

   sudo dnf install java-17-amazon-corretto -y 

5. Install Jenkins:

   sudo yum install jenkins -y 

6. Enable the Jenkins service to start at boot:

    sudo systemctl enable jenkins

7. Start Jenkins as a service:

    sudo systemctl start jenkins

You can check the status of the Jenkins service using the command:

sudo systemctl status jenkins

Configuring Jenkins

Jenkins is now installed and running on your EC2 instance. To configure Jenkins:

1. Connect to http://your_server_public_DNS>:8080 from your browser. You will be able to access Jenkins through its management interface:
   
2. As prompted, enter the password found in /var/lib/jenkins/secrets/initialAdminPassword.
   run the following command in your ec2 instance to display this password:

   sudo cat /var/lib/jenkins/secrets/initialAdminPassword

3. The Jenkins installation script directs you to the Customize Jenkins page. Click Install suggested plugins.
4. Once the installation is complete, the Create First Admin User will open. Enter your information, and then select Save and Continue.
   
5. You will get a link to your Jenkins, save it somewhere, and click on save and finish

Configure Terraform on Jenkins

Using the link you save above, sign into Jenkins and follow the various stept to configure terraform on Jenkins

1. Click Manage Jenkins from left hand navigation.
2. Select Plugins from System Configuration section.
3. Click the Available plugins tab and search Terraform.
4. Select Terraform and click Install without restart.
   
5. Click Manage Jenkins from left hand navigation.
6. Click Tool Configuration from System configuration section.
7. Scroll down to the Terraform section and click Add Terraform.
8. Enter a Name of your choice. I’m going to use “Terraform” to make things simple.
9. Ensure Install automatically is NOT selected. It will be selected by default.
10. we will install Terraform on the EC2 instance hosting our Jenkins by SSHing into the server. so, go ahead and SSH into the instance
11. Open any browser of your choice and search for terraform.io. Click on the Download button, then select your operating system on the left and package manger at the top. Right click on AMD64 (version:1.7.0) and click on "copy link"
    
12. In your ec2 terminal run the following command.

13. wget (copied url)

14. If you run ls you should see a Terraform zip file. Run unzip {terraform zip file name}.
15. The unzipped file is an executable and we will need to move it to /usr/bin.
    sudo mv terraform /usr/bin
16. To verify run which terraform
17. Now switch back to Jenkins. For Install directory enter /usr/bin. Click Save.
    

Manage AWS Credentials on Jenkins

1. Click Manage Jenkins.
2. Click Credentials in the Security section.
3. Click system
   
4. Click Global credentials (unrestricted).
5. Click on add credentials
6. For Kind select Secret text. For ID type “AWS_ACCESS_KEY_ID”. For Secret paste your Access Key for your user. Then click OK.
   
7. Repeat step 5 for your “AWS_SECRET_ACCESS_KEY”

Configure Jenkins Pipeline

Before we configure Jenkins Pipeline, we need to have a Jenkinsfile in the root directory of our project residing in github. let us create one now

Creating a Jenkinsfile

A Jenkinsfile is a text file that defines the entire build process for a Jenkins pipeline. It uses a domain-specific language (DSL) that is based on Groovy syntax. Writing a Jenkinsfile involves defining stages, steps, and other configuration elements to automate the build, test, and deployment processes.

Pipeline supports two syntaxes, Declarative (introduced in Pipeline 2.5) and Scripted

Declarative Syntax: This is a more structured and human-readable approach and usually structured as seen below
The agent instructs Jenkins to allocate an executor and workspace for the Pipeline. Without an agent directive, not only is the Declarative Pipeline not valid, it would not be capable of doing any work! By default the agent directive ensures that the source repository is checked out and made available for steps in the subsequent stages.
The stages directive, and steps directives are also required for a valid Declarative Pipeline as they instruct Jenkins what to execute and in which stage it should be executed.

Jenkinsfile (Declarative Pipeline)
pipeline {
    agent any
    //Specify the agent (where the build run)
    //e.g, 'docker', 'node', 'any', etc

    stages {
        stage('Build') {
            steps {
                //Define steps for the 'Build stage'
                //source code is assembled, compiled, or packaged

            
            }
        }
        stage('Test') {
            steps {
                //Define steps for the 'Test stage'
            }
        }

        //Add more stages as needed

        stage('Deploy') {
            steps {
                //Define the deployment step
            }
        }

        Post {
            //Define post-biuld action, such as notification oe clean
        }
    }
}
                                

Scripted Syntax: With the Scripted Pipeline,node is a crucial first step as it allocates an executor and workspace for the Pipeline. In essence, without node, a Pipeline cannot do any work! From within node, the first order of business will be to checkout the source code for this project. sample structured seen below

    node {
    checkout scm
    //The checkout step will checkout code from source control
    //scm is a special variable which instructs the checkout step to clone the specific revision which triggered this Pipeline
    run.
    /* .. snip .. */
    }

you can check out how to set environment variable and Handle credentials within Jenkins Here

Check out the jenkinsfile use for this project in the our GitHub Repo

Configure Jenkins Pipeline

Resources

• Installed and configure Jenkins on Aws
• Create and use Jenkinsfile