Beginner's Guide: Creating DevOps Pipeline
DevOps has become the standard solution for fixing slow, fragmented, or inoperative software development processes. The problem is that if you are new to DevOps and don’t know where to start, then you may lack understanding of these methods. This article will discuss the definition of the DevOps pipeline, and will also offer instructions for creating it in five steps. Although this tutorial is not exhaustive, it should give you the foundation to begin your journey and expand your knowledge in the future. But let's start with the story.
My journey through DevOps
I used to work in the Citi Group cloud team, developing the Infrastructure-as-a-Service (IaaS) web application for managing Citi cloud infrastructure, but I was always interested in how to make the development process more efficient and bring positive cultural changes to the development team. I found the answer in a book recommended by Greg Lavender, Citi's CTO for cloud architecture and infrastructure. The book was called The Phoenix Project ( The Phoenix Project ), and it explains the principles of DevOps, and it reads like a novel.
The table on the back of the book shows how often various companies deploy their systems in a release environment:
Amazon: 23,000 A Day
Google: 5,500 a day
Netflix: 500 a day
Facebook: Once a Day
Twitter: 3 times a week
Typical company: Once every 9 months
How are Amazon, Google and Netflix frequencies possible? That's because these companies figured out how to make an almost perfect DevOps pipeline.
We were far from that until we implemented DevOps in Citi. Then my team had different environments, but deployment to the development server was completely manual. All developers had access to only one development server based on IBM WebSphere Application Server Community Edition. The problem was that the server shut down whenever several users tried to deploy at the same time, so the developers had to tell each other about their intentions, which was pretty painful. In addition, there were problems with low-level test coverage of the code, cumbersome manual deployment processes, and the inability to track the deployment of code associated with a specific task or user story.
I realized that I needed to do something, and found a like-minded colleague. We decided to collaborate in creating the original DevOps pipeline - it installed the Tomcat virtual machine and application server while I was working on Jenkins, integrated Atlassian Jira and BitBucket, and also worked on test code coverage. This side project was very successful: we almost completely automated many processes, achieved almost 100% operability of our development server, provided tracking and improved test coverage of the code, and also added the ability to associate branches in Git with tasks in Jira or deployment. Most of the tools we used to build our DevOps pipeline were open source.
Now I understand how simple our DevOps pipeline was: we did not use extensions like Jenkins files or Ansible. However, this simple conveyor worked well, perhaps thanks to the Pareto principle (also known as the 80/20 rule).
A Brief Introduction to DevOps and CI/CD Pipeline
If you ask a few people, “What is DevOps?”, You will probably get several different answers.DevOps, like Agile, has evolved to span many different disciplines, but most people will agree on a few things: DevOps is a software development practice or software development life cycle (SDLC) that focuses on changing the culture in which developers and non-developers exist in an environment in which:
Automated operations that were previously performed manually;
Everyone does what he knows best;
The number of deployments for a certain period of time increases; Bandwidth increases;
Increased development flexibility.
Although having the right software tools is not the only thing you need to create a DevOps environment, some tools are needed. The key tool is continuous integration and continuous deployment (CI/CD). This pipeline has various stages (for example, DEV, INT, TST, QA, UAT, STG, PROD), many operations are automated, and developers can write high-quality code, achieve development flexibility and high deployment frequency.
This article describes a five-step approach to creating a DevOps pipeline similar to the one in the following diagram using open source tools.
Step 1: CI/CD Methods
The first thing you need is a CI/CD tool. Jenkins, an open source Java-based tool distributed under the MIT license, is the tool that has popularized DevOps and become the de facto standard.
So what is Jenkins? Consider this a kind of magical universal remote control that can talk with and organize various services and tools. A CI/CD tool itself, such as Jenkins, is useless, but it becomes more powerful as it connects to various tools and services.
Jenkins is just one of many open source CI/CD tools you can use to build your DevOps pipeline.
Jenkins: Creative Commons and MIT
Travis CI: MIT
Apache Gump: Apache 2.0
This is what DevOps processes look like with the CI/CD tool:
You have a CI/CD tool running on your localhost, but you can do little at the moment. Let's move on to the next stage of the journey into the world of DevOps.
Step 2: Manage source control systems
This is where source code control becomes almost a necessity. This tool saves your code in repositories, keeps track of versions and coordinates the work of project participants.
Although there are many tools for controlling source code, Git is the standard, and rightly so. I highly recommend using Git, although there are other open source options if you like.
Git: GPLv2 and LGPL v2.1
Subversion: Apache 2.0
Concurrent Versions System (CVS): GNU
Mercurial: GNU GPL v2 +
This is what the DevOps pipeline looks like with the addition of source code controls.
The CI/CD tool can automate verification, source code, and collaboration between members. Not bad? But how to make a working application out of it so that billions of people can use and evaluate it?
Step 3: Create an assembly automation tool
Use the build automation tool. Regardless of which build automation tool you decide to use, they all have the same goal: to collect the source code in some desired format and automate the task of cleaning, compiling, testing and deploying in a specific environment. The build tools will vary depending on your programming language, but here are some common open source options.
Wow! You can put the build automation tool configuration files into the source code management system and let your CI/CD tool put everything together.
All is well, right? But where to deploy your application?
Step 4: Server for web applications
So far, you have a packaged file that can be either executable or installable. For any application to be truly useful, it must provide some kind of service or interface, but you need a container to host your application.
A server for web applications is just such a container. The server provides an environment in which the logic of the package being deployed can be determined. The server also provides an interface and offers web services, opening sockets for the outside world. You need an HTTP server, as well as some environment (such as a virtual machine) to install it. In the meantime, let's assume that you learn more about this later (although I will talk about containers below).
There are several servers for open source web applications.
|WildFly||GNU Lesser Public||Java|
|CDDL & amp; GNU Less Public||Java|
Ваш DevOps-пайплайн почти готов к использованию. Хорошая работа!
Хотя на этом можно остановиться и заниматься интеграцией самостоятельно, качество кода - важная вещь для разработчика приложений, и об этом нужно беспокоиться.
Шаг 5: Покрытие тестирования кода
Реализация тестов может быть еще одним громоздким требованием, но разработчики должны ловить любые ошибки в приложении на ранней стадии и улучшать качество кода, чтобы гарантировать, что конечные пользователи будут удовлетворены. К счастью, существует множество инструментов с открытым исходным кодом для терианиеинниниени ва Еще лучше то, что большинство инструментов CI/CD могут подключаться к этим инструментам и автоматизирсее.
Тестирование кода состоит из двух частей: фреймворки для тестирования кода, которые помогают писать и запускать тесты, а также инструменты для формирования предложений, которые помогают улучшить качество кода.
Системы тестирования кода
|JUnit||Eclipse Public License||Java|
Системы рекомендаций по улучшению кода
|CodeCover||Eclipse Public (EPL)||Java|
|Emma||Common Public License||Java|
|JaCoCo||Eclipse Public License||Java|
Теперь, когда вы реализовали инструменты покрытия кода тестами, ваш DevO-пайплайн дольена по,ана
Как я уже говорил, вы можете размещать свой сервер на виртуальной машине или сервере, но контейненпа
Что такое контейнеры? Краткое объяснение заключается в том, что виртуальная машина нуждается в огромном объеме памяти операционной системы, превышающем размер приложения, в то время как контейнеру нужно всего лишь несколько библиотек и конфигураций для запуска приложения.Obviously, the virtual machine still has important applications, but the container is an easy solution for hosting the application, including the application server.
Although there are other container options, the most popular are Docker and Kubernetes.
Docker: Apache 2.0
Kubernetes: Apache 2.0
Intermediate Automation Tools
Our DevOps pipeline is primarily focused on co-creating and deploying applications, but there are many other things you can do with DevOps tools. One of them is the use of Infrastructure as Code (IaC) tools, which are also known as intermediate automation tools. These tools help automate installation, management, and other tasks for middleware. For example, an automation tool can retrieve applications like a web application server, database, and monitoring tool, with the correct configurations, and deploy them to the application server.
Here are some open source middleware tools:
Ansible: GNU Public
SaltStack: Apache 2.0
Chef: Apache 2.0
Puppet: Apache or GPL
Learn the details of how to get a sought-after profession from scratch or Level Up in skills and salary, taking SkillFactory paid online courses:
- DevOps course (12 months)
- 450 free courses from the Ivy League
- Trends in the Data Scene 2020
- Data Science has died. Long live Business Science
- Steroid Data Scenes: Introducing Decision Intelligence
- The most successful and most scandalous Data Science project: Cambridge Analytica