This report is 2016, but it is still useful for those who want to understand how TeamCity works.

Hello, I’m very glad to be here today and see such a keen interest in our product. I am truly a TeamCity developer. I have been working on the product for over 7 years. My area of ​​responsibility is support for dotnet technologies in TeamCity. I lead dotnet development. But including I communicate a lot with users. And I make all kinds of reports at conferences, including trainings we do for users and help to use our product correctly, efficiently, etc.

Today I want to share the experience of using TeamCity inside JetBrains and the approaches that we use. I hope you find it helpful.

We will definitely have a question and answer session at the end, so if I suddenly don’t cover something, you will have the opportunity to ask.

I will start a little from afar. I’ll tell you about the present company and product.

ITKarma picture

JetBrains today are offices in 4 countries, including a small office in Moscow, where, by the way, several TeamCity developers are also sitting.

We have 20 products. Of these, 16 personal and 4 products for the team. These are the products that we have already announced. Something to boil from what we have not yet announced, but after some time will also be available.

The company is growing fast enough. The number of developers exceeds 500 people, because our company has mainly developers. We have a minimal number of people who are not involved in development. Even those who are listed as managers, this is also usually a developer. Even the girls at the reception sit and do something at Javascript.

Accordingly, the company is growing very rapidly now, especially in the last 5 years. And the number of employees has increased significantly. A lot of products have been released.

And active users of only our personal products, in my opinion, are more than 3,000,000 people in the world, that is, developers. This is a good enough result, it seems to me. And this number is constantly growing.

ITKarma picture

Accordingly, TeamCity is one of the oldest products in the company. This year we released the 10th version. We release one major update during the year and one minor update. In a major update, more and more new features appear. In the secondary, these features are completed and some less global features appear.

And we also release at least 5 bugfix updates for each of these updates. And every month we release something. This is either a bugfix update, or some kind of feature update.

Another important feature of TeamCity is that support is carried out by the development team, that is, of course, we have a separate support where engineers answer the most frequently asked questions, but mostly, if you contact support, more often In total, you are talking to developers. Accordingly, for you this is a good level of competence for those who answer your questions. And for us, for developers, this is an opportunity to have first-hand feedback about the features that we have released.

ITKarma picture

We have an internal installation in TeamCity that collects all JetBrains products. In general, this is a special installation, because the product grew out of it.

All JetBrains products appear and develop in approximately the same scenario. My colleagues, when developing this idea, they were faced with the fact that at that time not a single tool, not a single build system met the requirements that arise when developing a complex java application. And it was decided to make your build server. That's how TeamCity appeared in JetBrains.

All this time the product has evolved. And in many ways evolved as a response to the new requirements of our growing company. When there were a lot of products, there were a lot of teams, we made a hierarchy of projects. When our colleagues started actively using Git, we made support for Git brunch features. That is, a lot of features are a reaction to the need of our colleagues who make complex products for dotnet, java, etc., so the support in TeamCity is quite good.

Here are the approximate figures for the size of our installation. This I took our statistics:

  • We have over 500 active projects.
  • It has about 4,000 active configurations.
  • We have more than 300 build agents connected to this single installation.
  • Our builds are on average quite massive. The maximum number of texts that are executed in one build is 53,269.
  • On average, the build company lasts 25 minutes. And the stream of builds is big enough because there are about 3,000 commits throughout the company per day. And, accordingly, the build service processes all these commits, launches builds.
  • An average of 6,000 tasks starts every day.

T. E. The load on this internal build service is quite large. And one way or another, everything that I will tell today will concern this installation.

ITKarma picture

What exactly would I like to cover today?

  • First, I will introduce my view of the basic concepts of TeamCity such as configuration, project, template, dependency, build agents, etc. I will tell you some details and features of use inside all these concepts.
  • Next, I will dwell separately on a very important feature. This is a snapshot of addiction. I will talk about what are the options for using this feature. As practice shows, users do not quite understand why this is necessary and how to use this feature correctly. She is the cornerstone of TeamCity. And there are no analogues on the market. And I will devote some time to this feature.
  • I’ll talk about the environment management approaches on build agents that we are currently using. There are many Build agents, there are 300 of them. There are different environments in order to run tests of our products. And you need to somehow manage everything.
  • Then I’ll talk about how TeamCity skeleton inside the company, that is, in particular in the 10th version, we rolled out a new feature that allows us to deliver an additional TeamCity server, which takes off part of the load from the main server.
  • Then I will touch upon TeamCity administration practices.
  • And then I’ll tell you about the TeamCity extension, that is, about all kinds of plugins that are not included in the delivery, but we use them internally and they are quite useful to us.

The first two of these points will be mainly useful to developers, build-engineers, those who are not particularly concerned with the administration of the server itself, but want to use it efficiently and correctly.

The last four points will be useful to project administrators and people who are somehow involved in the administration of TeamCity.

ITKarma picture

TeamCity implements the concept of competition green, that is, there is a main node that distributes the task of computing something to secondary nodes. Minor nodes are build agents. The main node contains all the data about configuring these tasks and responds to environmental events. That is, on version control commits, on the appearance of artifacts in the control repository, etc. And it launches the task.

The nodes that are called build agents, they will report to the server this task and all the results of their execution. Accordingly, the server also supports the task queue, i.e.if there are no agents suitable for specific tasks, she is queued and lives there until this agent appears. The queue is restarting, that is, you can turn off the server and turn it back on, but the status of the queue will be preserved.

And, among other things, an important feature is that if at some point the server is gone, for example, you upgrade, then the agents go into the mode of accumulating data, which they then send to the server when the server rises again. Thus, you can not lose a single byte of data that should have been sent from builds to the server when upgrading the server. And after the agents finish their existing tasks after upgrading the server, they will receive a command to upgrade and they will go to upgrade themselves.

ITKarma picture

Build Agent is a java process that runs as a console application, or as a service on some machine. He knows how to accept tasks for building, testing, deploying and whatever tasks from the server. Able to report back the progress of their execution and some artifacts that were generated in the process, etc.

All the many agents authorized on the server are divided into pools. It is very useful to use pools in order to somehow distribute available resources between development teams. That is, we have separate pools for different teams inside. And this ensures that if we run a lot of builds in TeamCity, it will not eat all the server resources, but only in our pool. And the wait time for the builds in our queue will be longer, but this doesn’t affect the command... or the dotnet command.

ITKarma picture

In addition to the agent being able to report information on the tasks that it performs to the server, the agent also tells the server about the environment in which it is wounded. That is, about what tools are at this moment in the operating system. He will report this through his configuration options. Plugins are supported there, that is, there you can write a plugin that will report from the agent some specific and necessary tools for you, if all this is not available from the box.

The server needs this knowledge about the environment on built agents in order to run tasks only on those nodes that can successfully complete this task.

ITKarma picture

Build configuration is the primary TeamCity configuration unit. In fact, it contains all the many settings that describe a particular assembly task. This, for example, a list of repositories where you need to get the source code, or builds scripts, the sequence of steps necessary to build the criteria for launching the task, the criterion for the success of the task and the requirements, for example, to the agents that provide these tasks.

ITKarma picture

Build configuration merges into projects. Projects, in addition to being containers for build configuration, they are containers for other projects, that is, projects can be nested, they form a hierarchy.

Accordingly, at the project level, you can also define certain properties that will be common to the entire configuration from this project. For example, enable or disable saving version control settings, determine the SSH keys necessary for these configurations to work, so that the configurations do not define these keys inside themselves, but simply use them.

In addition, projects form a hierarchy and a mechanism for inheriting entities within the project is implemented, that is, in each project you have access to the entities that are defined in the projects at a higher level. Thus, you can reuse the settings within the team or within the company.

Another application of projects is the delimitation of access rights to various configurations.For example, some configurations related to the deployment of your service or application in production, you can take all these configurations into a separate project, give rights to use and run builds in this project, and only individual users to change the settings of this project. Accordingly, you can thereby limit the circle of people who can do deployment.

This is the trend of transferring everything in TeamCity to the project level, this is a global trend. He continues many versions. And it will continue further.

Our main task is to make sure that when a new team appears inside the company, we, the people who administer the TeamCity server, create a root project for them, appoint someone from their team as a project admin, and then fully delegate it all further TeamCity configuration tasks. That is, we want to write our features so that when using them at the project level, there is no need to make any global admin settings.

And in particular, until the 10th version it was impossible, for example, to authorize agents into your pool. Even if you had your own pool for your project, the project admin did not have the rights to do this. It was necessary to go to the system administrator and ask him to authorize the agents. In the 10th version, we made it so that if you have admin rights to all projects associated with the pool, then you can authorize the agents yourself.

Some tasks will still require you to interact with the system administrator. For example, everything related to users. That is, if you want to get a new user into the system, or configure some group of users, for this you will have to go to the system administrator for now. But we want to do so in the future so that people within their projects can create their own user groups, that is, add users there, authorize, etc.

T. e. This is such a global trend, we gradually rewrite all our features in such a way. And this, by the way, is one example of using TeamCity inside JetBrains, because, as I already said, this internal instance of TeamCity is administered by our TeamCity team itself. And so we do not want to answer and do all the settings for all the people in the company, of whom there are already more than 500 people. Therefore, we try to make it so that we can delegate all this work. At the same time, making sure that their settings did not affect people from other teams.

ITKarma picture

The next entity is the dependencies between the configurations. Configurations may be related to dependencies.

The simplest form of addiction is an artifact of addiction.

I will give the simplest example. If you have an application, you use the API of some library in runtime, then, accordingly, to resolve this dependency during build, you can use the artifact dependency between the two configurations in TeamCity. In this case, you can use the last assembled build library, i.e. the last successfully assembled, or some kind of quilted build. Or, if you use an artifact, then you can publish library artifacts to a third-party repository. And then from there he will be resolved during the bulid’s application.

The second way has some advantages. And in particular, if you use a third-party artifact repository and do not directly use TeamCity dependencies, you do not lose the local reproducibility of builds, which is a very important property. Because if you have permission for this dependency reflected only in TeamCity as a dependency between configurations, you won’t be able to locally build the correct build of your application for the correct revision. Because you will need to go to TeamCity and understand which version of the library in that build was used and, accordingly, do a lot of manual actions.

Local reproducibility of builds is a very important property that you should try not to lose to the last. Refuse this only if there are sharp reasons.

You probably use the artifact of dependency anyway.

ITKarma picture

In addition, there is such an entity as a configuration template.In addition to the mechanism of inheritance of some properties through projects, TeamCity has implemented another inheritance mechanism. This is the build configuration template.

A standard example is testing an application on several platforms. In the simplest case, you have a configuration that drives tests. You have some kind of artifact dependency on configurations that collect artifacts that you are testing. And there are many build agents with different machines. There is, for example, VCS Trigger, which you have mounted to this configuration. And you run these tests on all these machines. This is a working option, but the idea is not very good.

why? Because you have tests, you target several platforms, you have one configuration and many build agents on which you run.

The general problem is that you will have mixed status on different platforms in the builds history of this configuration. It will blink at you. That is, if your tests fall only on Linux, then from time to time your build will be green or red. And this is bad. This is not worth doing. It is worth dividing tests on different platforms into different configurations. And to reuse the settings between them, you need to select a template.

ITKarma picture

The template, in fact, is also a configuration. But what is the difference between a template and a configuration? Some properties in it are parameterized. That is, as the values ​​of this property, a reference to some parameter is used. Accordingly, when creating a new configuration based on a template, in the future, when you have a new target platform, you only need to specify a couple of parameters and you will have a new working configuration. And all other settings will be reused.

ITKarma picture

And we came to an important essence - a snapshot of addiction. The idea of ​​snapshot addiction is quite simple. When you say that you have one configuration depending on the snapshot of the other configuration, then you say that the builds of these two configurations should be built consistently, i.e. with respect to the same revision, or snapshot in version control, or in several repositories. Because even the case is supported if your sources are in one version control or in one repository, and the other sources in another. That is, even such a snapshot between different repositories works.

This type of dependency gives a lot of goodies. And not all of them are obvious.

  • Firstly, by building such dependencies between configurations, you can get consistent builds in some cases that you couldn’t do before. That is, if you have some application or service that has many components, it is often a problem to make some kind of build of the whole system so that all its components are consistent, i.e. assembled relative to a specific point in time and snapshot.
  • Using this feature allows you to trigger your builds efficiently.
  • In addition, versioning your components is convenient.
  • is used when building a build pipeline, that is, if you have some kind of continuous delivery pipeline, then in TeamCity this is all configured using snapshot dependencies.
  • And also it helps a lot when you make releases, because there is such a feature as historical build, I will tell about it too.

Now let's take it in order.

The simplest case is builds consistency.

ITKarma picture

Consider an example. We have some kind of application. It is cross-platform. It contains some kind of Windows-specific code, Linux-specific code, and there is some kind of library that hurts regardless of the platform.

In runtime Linux and Windows, specific parts use the API of this library. And the application contains all these components: a library and platform-dependent parts.

A feature of such an application may be that you cannot collect such an application within the same machine. That is, you should definitely use two build agents with Windows and with Linux.And some parts of the system will be assembled on Windows, some on Linux. In this case, when you cannot collect everything in one sequence of steps, within one build on the same machine, then you may have problems with the consistency of builds.

ITKarma picture

What will the simplest configuration look like? I have a build configuration that compiles sorts libraries and publishes it as artifacts. These arrows are an artifact of dependence. I have platform-specific parts that use these artifacts assembled. They do not compile the library code themselves; they use it as an artifact. And there is a build application that collects everything together, builds some kind of package, etc.

Suppose you configured such a scheme in TeamCity, that is, you use only the dependency artifact. Your developers make some commits, that is, you have some kind of commit stream, respectively, you have some kind of build agents, some of which are compatible with these tasks, some with these, etc. And it may be that the sequence of all these builds is mixed up. When you use the dependency artifact, you have the last build built, the last successfully built build, and quilted build.

ITKarma picture

And if you had several revisions commited in the repository, then some components at a particular point in time will already be assembled relative to A, and some will be assembled relative to revision B. And, accordingly, what revision will the application be built on it’s not very clear which particular artifacts will get there.

ITKarma picture

To avoid these problems, a snapshot of dependency is used. That is, in addition to the artifact of dependencies, which say that I want to reuse artifacts from those builds, you bring in the same set of equivalent dependency snapshots.

After this, TeamCity guarantees that the builds in all these configurations, since they are connected by snapshot dependencies, will form a chain of builds - build chain. This is such a concept in TeamCity. And as part of a specific build chain, you will be guaranteed to have a consistent state.

The problem is that if you publish your artifacts and resolve them from a third-party repository, then this picture does not work for you. Because TeamCity also knows about snapshots and an artifact of addiction. And when he resolves a specific version of an artifact, he draws attention to the snapshot and takes exactly the right artifact. If you publish all this in artifactory, then you still take either a snapshot or some specific version.

This is the case when it is beneficial to use the dependency artifact inside TeamCity and not publish it to a third-party repository of artifacts, or maybe duplicate this publication.

Actually snapshot addiction, artifact addiction, it does not add any semantics. TeamCity doesn’t matter what you collect there, for example, building something, laying out something, etc. If it is important for you to deploy consistently several components of the system, then everything is the same. Imagine that here you are not assembling artifacts, but laying out some set of micro-services on staging or production. Then you have a root’s configuration called deploy everything. And it will depend on snapshots from all other configurations. And with this root’s node you will be able to do everything consistently, all at once. And if the calculation of one of the components is broken, then you can re-lay it again in a consistent state, and not the last assembled.

ITKarma picture

The next aspect is how a snapshot of dependencies allows you to trigger your builds efficiently.

ITKarma picture

Usually the approach here is this. First, I have to get a library.When it is assembled, for example, this component should be assembled. When it is assembled, the application itself must be assembled. And usually this is set as trigger after. That is, it is triggered when the build of another configuration is completed.

If you have such a rhombus, then you will get two builds of your application and in none of them there will be a normal version of artifacts. To avoid this situation, you add a snapshot of dependency.

ITKarma picture

And TeamCity offers an alternative approach to triggering. You should not think at all about how the dependence will be fulfilled, you should think about the result, that is, what you want to get. You want to get the build of your application. There are some changes in this column. Graf can be huge, TeamCity is going exactly the same. He has a huge number of plugins and huge graphs on the left.

But we want to build just build TeamCity. And all we do is hang up only one trigger on the root’s configuration and put there a magic checkmark “trigger build to change the dependency snapshot”, which is turned off by default. And you don’t think about anything further. TeamCity for you determines at each particular point in time the builds, which components should be built and the builds, which components should be rebuilt, because one of the dependencies, for example, has changed.

ITKarma picture

If at some point my Windows-specific code changed, I only have a trigger here, but TeamCity will understand that I want to run the entire build chain with this change, so this build will be rebuilt. You will have a new artifact of this version, the rest of the system with these changes have not been affected and will not be rebuilt. Since you have changed one of the application components, the build of the application itself will also be rebuilt. At the same time, these artifacts will be reused, you save time for build agents, that is, these builds will not be launched at all.

ITKarma picture

And the presence of this feature allows us to save a huge amount of build time with us internally.

This is an old screenshot. When the build time, which occupied all the agents on the server, was 3,558 hours. And every day at that time we saved 870 + 509 hours due to the presence of a snapshot of dependency and reuse of artifacts inside builds.

ITKarma picture

This leads to the fact that you need to buy fewer build agents, because fewer build agents you need are serviced, because builds are reused.

Why didn’t we make this feature private? Because we don’t do that. Everything we do, we do for ourselves and for users as well.

Manage application components version

The next feature, which is available using snapshot dependencies, is version control of applications, versions of components or microservices inside a large service.

ITKarma picture

An important property of dependency snapshots and the builds chain is that not only the artifacts that are available as part of this build chain make up the general context, but the parameters of all builds make up the overall build chain context.

And within the context of this context, you have the opportunity from the build of one configuration to access the values ​​of the parameters of another configuration.

For example, if you have a configuration A, in which there is some parameter that has the value foo within a specific build. If build from configuration B is in the same chain, that is, consistent with this build, then you can use the value of, for example, this parameter as the value of parameter B.

ITKarma picture

What does this give you? Usually, when you talk about versioning some components or services, then you always have some kind of template according to which the version is assigned. And you have some kind of static part that is usually hardcoded at the build configuration level. And there is some kind of dynamic part that changes from build to build. Here is the syntax - percentage is something percent - this is a link to assign some parameter to TeamCity.

Accordingly, the first thing that comes to mind if I have a lot of component systems and all these components need to be assigned some version is to hardcode such a template at the configuration level. But then I will have duplication. I will have 4 places in the system where I will have this template hardcoded.

I can, of course, inherit everything from some kind of template. Define this template at the project level and also reuse it. But there will be a problem that if I do not have these builds executed consistently, that is, if there is no dependency snapshot, then this dynamic part will not necessarily be the same. And it turns out that the components of your application will probably have components that have different build numbers. Because build number is a local counter within each configuration. And it is different in each configuration. If here 10 builds have passed, and here 11, then there will be 11 build number, and here 10. And there will be no common version for all components.

ITKarma picture

How to do it right? I have already said that within the framework of the build chain we have a common parameter context and we can refer from the parameter of one build to the build parameter from another configuration. We can do this with this syntax.

I decipher. It says dependency with such and such ID and property name that I want to use. I can use such a substitution as the value of my parameter. Thus, I can define the version parameter in the very first executable configuration, which will be, for example, CDMY0CDMY, i.e. CDMY1CDMY within this configuration. In all other configurations, as a version, I will use the link to the parameter from build from this configuration. Thus, we have a uniquely defined template, and we are guaranteed to get the same version value in all dependent builds.

And this all works not only in read only mode, that is, we can not only refer to the value of the parameter and the dependency, we can force them to set some values. That is, the reverse syntax works for all inverse dependencies whose IDs satisfy some parameter. That is, it is either an explicitly specified ID or an asterisk. And we can set the parameter with such a name to some specific value.

This can be useful when you have scripts that collect each of the components, for example, some kind of parameter. For example, isRelease, which is compiled into code at compile time. And so you then turn on and turn off exception, or else do something there. And you do not forward this through config, but directly into the code. And, accordingly, you can have many dependencies and the default values ​​of isRelease in each of the dependencies can be different.

But at some point you want to build a build so that all components are guaranteed to get the correct value for this flag. And you start your build in this configuration, in our case it’s the build application, and say "set all dependencies to isRelease true". And you are guaranteed to get the correct value for this flag in all dependencies. Such builds will not be reused, because past builds were built with different values. You will be guaranteed to have all your dependencies reassembled, but on the other hand you will get what you want.

ITKarma picture

The next aspect is a little about continued delivery.

ITKarma picture

Imagine that you have some kind of configuration that builds the distribution of your application, or it is a package that you will post later on staging, production.

Do you have any acceptance tests. You have a deploy on staging procedure. It is framed as a build configuration. And there is deploy to production.

Usually, when such a pipeline is configured, you want to have a balance between automation and manual control. That is, some parts of this pipeline I want to start automatically, but somewhere I still want to have a manual drive, that is, to have some kind of control.

Accordingly, how does snapshot addiction help here? For example, I want my application to be assembled for every version control commit, accept tests run and laid out on staging, but at the same time, only what I said explicitly with my hands laid out in production. How can this be implemented using a snapshot of dependency?

ITKarma picture

Firstly, as shown in the diagram, I am unifying my configurations with snapshot dependencies, that is, I say that deploy production depends on build dist. Deploy on staging depends on acceptance tests. And the tests depend on build dist, because they will test this particular artifact in the future.

ITKarma picture

Next, I add the VCS Trigger here, that is, I want to post on staging with any commit. I don’t add any trigger to this because I want manual control.

ITKarma picture

How to implement this manual control? There is a feature called build promotion. Promote are user actions that are available for a specific build in this configuration, if there are incoming dependencies snapshots in this configuration that allow me to build a specific build further along my pipeline in the direction I want.

ITKarma picture

Here is my scheme. I am logged in under the user ekoshkin, who is a developer. Imagine that I have a development team that is developing some kind of application, there is a team of admins that deploy the whole thing.

ITKarma picture

In my case, my root’s configuration is the build app. There is some kind of build that has been built. And I want to deploy him. He has some addictions.

ITKarma picture

Here, the very rhombus that we talked about is just implemented. And I want to deploy the result of this build somewhere.

ITKarma picture

I go to action. This I do for a specific build.

ITKarma picture

I run promote. A list of incoming snapshots of dependencies is shown to me, that is, a list of configurations, which depends on me by snapshot. And it allows me to get build into one of them.

Accordingly, here I have two configurations that deploy my application in production and staging. And I can run this build in one of these configurations, and this configuration will use exactly this version of the artifact for which I call promote. This is again done due to the snapshot of dependencies, because they are connected by a snapshot of dependencies.

ITKarma picture

When I run build, I can customize something, for example, that is, specify some parameters necessary for the calculation procedure, etc.And after that build will go.

I call promote. Build started in this configuration in production. Accordingly, he has a status. I can watch status and deployment in this configuration. In my case, these are positive statuses.

ITKarma picture

And most importantly, I can then see all the build chains that led to the deployment. And I can see all the versions that were deployed at that moment, and all the versions of all the components that were deployed, which is quite useful.

What are the benefits of such setup?

I am the developer in this example and am the administrator of the Demoapp project, that is, everything that is related to the development of my application, I can change as you like. I have permission to edit anything in the application. But at the same time, as for the deployment situation, I have no rights. All configurations related to deployment, they are concentrated in a separate project. And at the level of the user group that I am a member of, these rights were taken from me. All I can do is just run builds, promote specific artifacts in them.

Thus responsibility is shared. On the one hand, people who are part of the devops team and are administrators in the deployment project, they can change the build procedure as they like. They will probably have their own agent pool associated with deployment. There will be some secrets needed for deployment. For example, SSH keys for access to production servers, etc. I, as a developer, will not have access to this. And my protocol for communicating with them is promote, that is, promote a specific version of the artifact that needs to be posted. I still have the right to decide which version will be posted, and they still have the opportunity to guarantee that the calculation procedure will be correct, safe, no one will change it.

Promote is a manual action?

Yes, promote is a manual action. If you want to upload everything automatically in production, then you hang up a VCS trigger or any trigger in this configuration.

How do I manage the switch between the release of all and the release of the first million?

You will do this through the parameter. Most likely, you will have two configurations: deployment for the first million, deployment for everyone. They will have a common template, i.e. the assembly procedure will be the same, they will have one parameter. This is the first million, or not the first million.

What if you want to roll out the release for the first million not in TeamCity, but in another place?

This means that this commit/release is special. The commit/release is special - it means that you set some tag. Alternatively, so as not to expose tags, I suggest such a scheme. In a specific build of your service, which was compiled relative to this particular commit, which is not specially tested in any way, but you, as a developer, know what exactly it needs to be laid out.

How to do all this for everything except for some individual commits?

I think that this should be done at the branch level. And then this configuration will have several stories for different branches. And it will be that such fixes are dangerous. And there will be a master. The wizard will constantly roll out. And if there is something so dangerous, then there will be a separate procedure that looks at a separate branch and does not bother with it in the master, until you roll it out and see how it works. Because it seems to me that in a master such a commit is not very good to have. We need a procedure that looks not only at the master, but also at some neighboring branch, for example.

Promote can rollback?

Promote is pushing along some kind of flow, in the direction in the task graph. You will need to implement this yourself, because the inverse rollback task is another question.

Is it possible to do downgrade on deploy?

If the rollout procedure itself supports downgrade, then yes, but it’s not always because your data can settle if it is not in the container and there is some kind of state that is saved. And then there will be a problem. But this is the semantics of a specific application. There are such things. For example, in TeamCity, this is done. I talked about the fact that there are major, minor releases, there are bugfix updates.And we have an agreement that data conversion takes place in major and minor releases, i.e. there are 10 and 10.1. And when we release bagfix updates, we have an agreement that we support downgrade as part of bugfix updates. We just do not write converters. We write when everything has died for everyone.

Accordingly, if we want to do some converters, we will make up for it in the next big release, or we can somehow do without them, at the code level, we somehow solve this. We have done so. But when you do the service, then, in my opinion, there is more space for maneuver than when you do the product. This is probably done at the infrastructure level.

ITKarma picture

I’ll also show the trigger settings, I have enough rights to them. Accordingly, I have a VCS trigger. And there is a tick “trigger a build on changes in snapshot dependencies”, which are not defaulted.

ITKarma picture

And a bunch of rules. Add trigger rule and there is include, exclude rule for VCS user name, for VCS root, for comment regexp and for file wildcard even exist. Accordingly, it can be inclusive, exclude and rules can be several. They can be nested.

ITKarma picture

Now about historical builds. This feature is very useful when you make releases.

ITKarma picture

Again, there is our rhombus, which is united by a snapshot of dependencies. And developers make some commits. A commit came to the library, a couple of Windows code commits and some kind of Linux code commit. And this all happens before the release. We need to be released just some last fixes come. We have a VCS trigger, so build is constantly running.

ITKarma picture

Gathered some kind of build. It turned out that it contains a bug, that is, there either some tests fell that were not checked at the level of a specific component, or testers said that there was a problem.

And at the moment we have the latest build Linux component built for the latest revision that came there. This is commit 3. There were 2 commits on Windows and the last commit hit the library. And it turned out that the last commit 4 brought a bug. And if we gather here for this condition, then everything will be fine.

But the problem is that at the moment, in our history, such an artifact was never correctly collected at all, that is, it was collected for the previous revision and the last. And it just contains an error, and all the others for the correct revision are already collected.

And I want to build so quickly to launch the application itself so that as soon as possible to get a new release build, at the same time rebuild this one, but do not touch these.

ITKarma picture

This is how you can do it using the history build feature. You go run custom build of this configuration, there you will have a list of changes. And since you have all configurations united by snapshot dependencies, the changes list will be a list of snapshots relative to any of which you can consistently build your entire application. You will select the previous revision in it and run the build of this application on this previous revision. Everything will become automatic further, that is, TeamCity will understand that regarding this snapshot, this artifact is already suitable, there hasn’t been any such collection yet. Accordingly, he will collect it. And if, for some reason, this build was built for some other audit, then it will also be reused. He will take the necessary artifact, rebuild the application itself, because one of the dependencies has changed. And using the minimum amount of time for build agents, that is, speeding up feedback systems, you get a new release build.

Such a feature has appeared.Despite the fact that the queue is full of builds, I showed that the average wait for build in the queue is still 1 hour, so I want to build a quick release build somehow. And the presence of such a historical build allows you to do this.

If you publish everything to a third-party artifact repository, you will have to do a lot of manual work, that is, for each of these builds you will have to figure out which coordinate of this artifact was published, which artifact was published in each of the build. And you will need to do some action manually, most likely.

What if a commit with a bug contains third-party dependencies?

You have to write a script. It can be launched here, probably. We have done so. We have an artifact script that the developer runs on his machine, where exactly the third-party libraries all resolve the correct version. And then everything is going in the framework of one revision. With us, most of the project can be assembled on one machine, but the whole TeamCity cannot be assembled on one machine.

Why am I drawing a snapshot addiction with a more visible arrow?

Because it is a stronger addiction than an artifact of addiction. An artifact of dependence is a binary, in fact, dependence. That is, a dependency on some version of the API. A snapshot of dependency is, in fact, all of these components, which may even lie in different repositories, but as if it were just one working copy. And you can check out this working copy for yourself and collect everything. Although, maybe this will require several nodes, several machines: Mac, Linux, Windows. That is, such an idea is behind all this.

We were talking about a feature - historical build. Such a build, because it will be the last in this configuration, but assembled for the latest revision, it will be marked as historical. And it will be said that in fact this revision is older for some reason and therefore is called historical build.

Basic things like jobs, slave in Bamboo - everything seems to be the same. I talked a lot about the snapshot of addiction, because it really is such an exotic thing. She is only in TeamCity. And all these features, which are usually solved with some crutches, are useful. As I showed you about promotion, but in reality you can have one component, and the manual drive for continuous delivery is also implemented using snapshot dependencies, because again you need to upload something consistent.

ITKarma picture

On slide 1 is a snapshot of addiction or 4?

4 snapshots of addiction. And most importantly, with these commits there may not be a consistent history of changes in some branch. It can be commits in different repositories. Because TeamCity is such a meta version control. And he does cross-repository snapshots. Therefore, this revision can come in one repository, these two in another, this one in the third. And still it will be some snapshots.

My root is not the one that broke, my average dependency broke. But the application itself was affected. TeamCity synchronizes them in time. If these are revisions from the same repository, there is a chain partners relationship, because there is a directed graph in Git, but if it is between repositories, then TeamCity itself stores it. That is, he simply knows the timestamp of each commit and builds his snapshots himself, and somehow manages.