Continuous integration is a necessary development tool. Thanks to automation and standardization, it helps to effectively prevent errors when deploying applications into operation.

Also, you should not miss the previous part of the article series about web integration – Architecture and operating of web applications in a cloud environment (Part 2). It is a must-read!

What is Continuous Integration and What are its Stages?

Man is not a robot, and that’s the reason why there are a number of repeated steps in IT practice that are worth automating. In application development are constantly repeated certain steps. It is boring to do them over and over, people make mistakes, and machines do not. Continuous integration is a cure for such struggle. So what are the possible uses?

I consider continuous integration a necessary addition to software development, often adrenalin, and sometimes even fun. My favourite tool is Jenkins CI, which I have the most experience with. There is a whole range of similar tools used for continuous integration, build automation and deployment. Each has its advantages and disadvantages. Selection criteria can vary depending on the price or user-friendliness, and the possibility of integrating other tools and systems. But before we look at the specific tools, let’s first mention something about continuous integration as such.

Application development goes through various phases, from design and development to testing. Some may run in parallel, and especially in development, it happens that more specialists create more or less interdependent parts of the application.

The phases need to be combined into a functional unit for the customer, which is not quite that simple. Individual team members perform best work possible, which they have tested and would vouch for. But we do not live in an ideal world, and thus the contributions of various people may conflict sometimes, or someone may make a mistake in the project due to his/her carelessness or negligence.

Continuous integration helps prevent the client and the system users from having to deal with errors. It involves using various tools designed for this purpose as well as establishing error rate reducing processes.

Both large and small development teams need to re-build the project over and over.

This means that particular parts, which are developed by individual teams (or individuals), need to be assembled into a functional project or application, where individual modules smoothly communicate with each other. At the same time it is also necessary that responsible personnel are timely advised if an error occurs (the system correctly notifies authorized personnel).

After all, it would be a problem if, say, Internet banking users could not make payments, because some of the developers made a mistake in implementation and none of the responsible personnel knew about it. Continuous integration tools are set up to ensure this doesn’t happen and everything goes smoothly as clockwork.

For this reason continuous integration process should also include automated testing.

Auth. Note: Naturally, an application build and deployment can theoretically also be carried out manually. Though, we could dispute this would be “continuous” integration. Regardless of the possible error rate (and the time consumed) is not desirable to have processes depend on the presence of trained personnel. So it makes more sense to use an existing tool that is already in place.

Such tools are Jenkins CI, Atlassian Bamboo, Teamcity, Travis CI, and others. How to pick one is a matter of personal preference, all listed can deal with sub-tasks outlined in this article.

Stage One of Continuous Integration – Download the Latest Source Code

The first step in continuous integration is to download the latest code from a source code management (SCM) system.

Auth. Note: SCM system addresses the following:

• Hosting – physical housing of code repositories of code version control system
• Centralization of developed code – for submission and checks
• Security setup (individual repositories access setup for particular groups)
• Code verification
• Other tools can be connected to SCM (request tracking, documentation,..).

That is why using SCM is an absolute necessity. I could elaborate on it in more detail, but I could go on as long as this article. For this reason, I will mention only the most important details.

I personally use Git version control system; you may be using Mercurial SCM or Apache Subversion. For more details check the article on version control systems.

Exception in versioning should be a specific configuration in runtime environment -credentials to log in into the production database, keys to connect to API, etc. For security reasons, these shouldn’t be available to every developer, neither should they float in the “public space”.

In addition to taking care of security issues, we also need to set a specific debugging level in certain instances independently of the common standard. This is done by employing a versioned local configuration.

Either way, a versioned code, in its “raw form” may not suffice for the application operation, because after downloading the source code, we must transform it into an executable form.

Stage Two of Continuous Integration – Building the Project

Theoretically, a small project can be a small application in itself, built with only a few classes.

In practice, we do not want to write a code entirely from scratch, and so it is smarter to use one of the frameworks as the basis (Zend, Nette, Spring MVC,..) where the basic requirements, such as safety and programme logic separation in comparison with the presentation, are already sorted out. And of course, each framework includes bits we had prepared to reuse instead of having to create the project from scratch.

Indeed, we can use plenty of frameworks and libraries. Yet, we need to address systematically their definition and subsequent downloading. So here come in play composer install / update (PHP), maven install (Java), and gradle build. They are here to help downloading related libraries and thus completing the project.

Stage Three – Build Front-End Code

In addition to program code, applications also have their own front-end (interfaces) for users. Modern front-end coders routinely prepare styles for LESS / SASS preprocessors. As in the previous case, when working with a program code we need to compile LESS / SASS documents to CSS (into a publishable and usable form). Tools such as Grunt (The JavaScript Task Runner), and/or (A package manager for the web) are here to help us with it.

Stage Four – Editing Front-End Code

As I mentioned above, we don’t usually get the code in the desired form straight away. Also, our secondary goal is to save data and CPU power (on the server and on the client’s side), and therefore we try to convert the front-end code (CSS, JavaScript, images) into a compressed form.

The same rules apply both to front-end codes and programme codes. That means we should do syntax checking and potential error checking.

Such operations as minification, compilation and testing are done by tools like Grunt (The JavaScript Task Runner) or Bower (A package manager for the web).

The above tools are here to help us create a ready-made application, which is possible to deploy. Yet, we should test it, prior to publishing it for real users. Even the most conscientious developer can make a mistake. The truth is that when multiple developers work together it can happen that separate pieces produced by each of them are functional on their own, but the joint piece may behave weird or show errors. And that’s the reason why we should pay duly care to checking the code and testing it.

Monitoring and Testing Continuous Integration

The tests can be divided into the following groups depending on the moment of initiation:

• Pre-commit (prior to handover)
• Pre-deploy (prior to deployment)
• Post-deploy (post deployment into the target environment).

The tests can be initiated prior to the integrated change depending on the test type and integration strategy – pre-commit (see e.g. npm pre-commit package, SonarQube pre-commit analysis) or post integration – i.e. post-commit.

Early stage tests are not related to continuous integration tool, as code testing is done already on the developer side. SonarQube and pre-commit test are done directly in developers’ IDE using Eclipse and InteliJ plugins. Pre-commit tests relate to the source code. This means we don’t test the running application. Instead, we do syntax checking; verify compliance with coding conventions, check whether there are no duplications in the proposed classes, and check the overall code quality.

Later on, in pre-deploy are done unit tests, we check functionality and correct implementation of particular system units in the form of classes and functions. Here we touch upon the issue of test-driven development, where tests are written prior to the code itself. They reflect all functionality requirements, and we cannot go further unless they are in compliance. If the tests go right we can finish the deployment, and then move on to post-deploy tests.

Post-deploy tests are initiated following successful deployment:

• Automated user tests – are created by Selenium IDE and run by CI tool in Selenium server environment.
• Integration tests verify functionality of the system.

Nature of some projects requires manual testing.

These can be, for instance, electronic banking and applications of similar security, which can be logged into only by human users. Some operations may involve two-factor authentication, which we want to test by real human testers. In addition, there are many components, such as forms and calculators that can be tested automatically. Basic tests checking if an application is working after an update, and behaves as expected, can be machine done.

Automated testing brings in the long run:

• Repetitiveness – tests are carried out following the same scenario, in more or less the same conditions,
• Saved time – compared to demanding user testing.

Despite all the efforts of the development team, building or testing can end in a failure. Also, someone always needs to be notified about the negative result (advising about positive outcomes is not always necessary). This is resolved by a notification system, which we will discuss in the second part of the article.

You can now follow to the next part – article called: Stay Up to Date – Continuous Integration Notification System discussion.