Playing CI/CD with Micro Integrator

In Agile development having Continuous integration(CI) and continuous deployment(CD) for Services is a must for delivering changes more frequently and reliably. An organization can have multiple environments, such as development, testing, QA, staging, and production, each with its own instance. Therefore, the Services need to be available in each environment after developers specify the required conditions. Manually promoting Services between environments is a tedious, error-prone, and time-consuming task. This drastically reduces a developer’s productivity.

WSO2 Micro Integrator is an open-source, container friendly Integration runtime of Enterprise Integrator which is used for integrating APIs, services, data, and SaaS, proprietary, and legacy systems. Also, it supports both centralized (ESB style) and decentralized (microservices, cloud native) architectural styles.

In this blog, I will be explaining the recommended way of implementing the CI/CD structure for WSO2 Micro Integrator. For the purpose of explanation I will assume that a typical organization will have the following environments.

• Dev environment for developers to develop and test their code.
• Staging environment which used as a Test environment for functional testing.
• Production environment which is exposed to outside and serves the actual traffic

Before setting up the CI/CD pipeline we need some introduction to the following tools.

Jenkins : Jenkins offers a simple way to set up a continuous integration or continuous delivery (CI/CD) environment for almost any combination of languages and source code repositories using pipelines, as well as automating other routine development tasks. Hence I’ll be using Jenkins to setup the CI/CD pipeline for Micro Integrator.

Git (GitHub) : Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. GitHub is a cloud-based Git repository hosting service. Essentially, it makes it a lot easier for individuals and teams to use Git for version control and collaboration. We will be using Github to manage the source artifacts of our Integration project.

Integration Studio : WSO2 Integration Studio is a low-code development platform in which you can develop Integration solutions that can run on the WSO@ Micro Integrator. It provides the development environment to design, develop, debug, and test the artifacts that you create.

Nexus : Nexus is a repository manager. It allows you to proxy, collect, and manage your dependencies so that you are not constantly juggling a collection of JARs. It makes it easy to distribute your software. Internally, you configure your build to publish artifacts to Nexus and they then become available to other developers.

Let’s look at how we can implement a CI/CD pipeline for Micro Integrator which deployed on a VM using the above tools.

Integration Project Build Job

• We need to maintain one Jenkins job per Integration Project repository.
• The Integration Project has to be a Maven Multi Module project and it has to contain one Composite Exporter module. By default, the sample scripts provided, supports one Exporter module. Users can customize it to support multiple Exporter modules per Integration Project.
• The build phase of the job will build the Integration project and run the unit tests if a Unit test server, if configured.
• The release phase of the job will publish the CApps to the Nexus repository and create a release tag in GitHub.

Deployment Descriptor Build Job

• We need to maintain one Jenkins job per Environment.
• There will be descriptor files for each project inside a separate folder.
• To use a new version / rollback to a previous version, the user should define the change inside the respective descriptor and commit to the respective branch.
• This job contains only the build phase. This will delete the CApps from the VM instance and fetch the defined CApps from the Nexus and add it to the VM instance.

Setting up the environment

PREREQUISITES

1. Integration Studio 8.0.0 or higher
2. Two GitHub repositories
3. Source repository — To maintain the source of the project
4. Deployment repository — To maintain the descriptor files of the environment
5. Jenkins server for Continuous Integration
6. Nexus server — To store the deployable composite artifacts (CApps).
7. Micro Integrator instances in 3 environment (Standalone or Clustered)
8. Separate Micro Integrator instance as Unit Testing Server (Optional)

Sample User guide

1. Clone the samples-apim repository
2. Create Integration Project for your solution via the Integration Studio. By default, Integration studio adds parent maven details in the submodule POMs. You can customize maven details while creating the solution.
3. Update Project pom scm details by filling the git location.

<scm>     <connection>scm:git:https://github.com/username/repository.git</connection>     <developerConnection>scm:git:https://github.com/username/repository.git</developerConnection>     <url>https://github.com/username/repository.git</url>
</scm>

4. Commit your changes to the GitHub source repository

5. Set up Jenkins server

6. Login to the Jenkins server using the credentials given in the Dockerfile of Jenkins instance.

7. Navigate to the project build job and trigger a build.

8. Create WebHooks in relevant GitHub repositories pointing to the Jenkins server. (Source & Deployment)

9. Set up Nexus server

10. Login to the Nexus server using the credentials given in the Dockerfile of nexus instance and confirm that the ‘Integration’ repository has been created.

11. Go to the Jenkins server and perform a maven release by giving release and development versions.

12. Once the release build is passed, confirm that the CApp is available in the Integration repository under the given release version in Nexus.

13. Create a descriptor.yaml inside a folder specific to each environment by including solution project details.

--- - <group-id> <artifact-id> <capp-version> - ...

14. Commit the above created folder to the dev branch (environment) of the deployment repository.

15. Once you commit the changes, you can observe that the descriptor-dev job starts running and it pulls the CApps from the nexus and copies it to the configured Micro Integrator instance in the Dev environment.

16. Verify that the new changes are available in the Dev environment.

17. You can repeat steps 13, 14 and 15 for the Staging and Prod environment.

SETTING UP JENKINS SERVER

1. Docker scripts for setting up a Jenkins environment are provided here. This will spin up a preconfigured Docker image. By default, 4 Jenkins jobs will be created. One project for the Integration project and three more for the environment descriptor repositories (dev, staging, prod).
2. Both the Integration project, and descriptor repositories will be in GitHub. Note: You can customize the Docker scripts to create Jenkins jobs for multiple Integration projects.
3. Navigate to the docker-vm-artifacts directory. cd sample-apim/mi-cicd/docker-vm-artifacts/jenkins
4. Open up the Dockerfile and fill up the project and environment related details.vi Dockerfile
5. [Optional] If you want to customize the Jenkins configuration, update the Jenkins_casc_vm.yaml file.vi jenkins_casc_vm.yaml
6. [Optional] Set up Synapse Unit testing server
7. Run the following build command to build the Docker image.docker build -t <image-name>:<image-tag> .
8. Run the following command to run the image. You need to configure the .ssh folder to access the dev, staging and production environment and mount the folder to the container.docker run -d -p 8080:8080 -v ~/.ssh:/root/.ssh <image-name>:<image-tag>
9. After installing Jenkins, the jobs can be accessed via http://localhost:8080/

SETTING UP NEXUS SERVER

1. Docker scripts for setting up a Nexus environment are provided here. This will spin up a preconfigured Docker image. This Nexus container will be used to host the Carbon applications for different versions. The default repository is named as ‘Integration’
2. Navigate to the Docker VM artifacts directory.cd sample-apim/mi-cicd/docker-vm-artifacts/nexus
3. Open up the Dockerfile and fill up the project and environment related details.vi Dockerfile
4. [Optional] If you want to customize the Nexus configuration, update the nexus_casc.yaml file.vi nexus_casc.yaml
5. Run the following build command to build the Docker image.docker build -t <image-name>:<image-tag> .
6. Run the following command to run the image.docker run -d -p 8081:8081 <image-name>:<image-tag>
7. After installing nexus, the repository browser can be accessed via http://localhost:8081/ .

SETTING UP SYNAPSE UNIT TESTING SERVER

If you have written Synapse unit tests for your Integration project, you can run them during the Jenkins build.

To set up the Synapse Unit testing server, please follow the below steps.

1. Run a separate Micro Integrator Instance in Unit testing mode. To start the server in Unit testing mode please pass this argument -DsynapseTest.
2. If you want to change the synapse testing port, you can pass the -DsynapseTestPort=<new Port> to the above command. Default port is 9008
3. Update Jenkins Dockerfile as below.

SYNAPSE_TEST_FRAMEWORK_CONFIGS= -DtestServerType=remote -DtestServerHost=<IP of testing server> -DtestServerPort=9008

That’s it. We have now successfully implemented a CI/CD pipeline for Micro Integrator which is deployed in a VM.

Micro Integrator

WSO2 Integration Studio

The Love story of Integration Studio and Micro Integrator

Life of MI : How an ESB went on to serve Microservices?