ITKarma picture

Hello! My name is Mikhail Bulgakov (no, not a relative), I work as a release engineer at Badoo. Five years ago, I started automating iOS application releases, which I described in detail in this article . And then I took up Android applications.

Today I will summarize some of the results: I’ll tell you what we have come to during this time. Long story short: any employee involved in the process can unload at least all our applications on both platforms in a few clicks - without a headache, a lot of time, registration and SMS. So, our release engineer department for 2019 saved about 830 hours.

For details - welcome to kat!

What is behind the mobile release

There are three steps to releasing an application in Badoo:

  1. Development.
  2. Window dressing in the application store: texts, pictures - everything that the user sees in the App Store or Google Play.
  3. The release of the release engineering team.

When the application is completely ready and the first stage is completed, it is important not to fix it at the release stage and bring the product to the “counter”. This last step seems the easiest, but in fact it takes a lot of time and its success depends on a few people.

Most of the time is spent on preparing the application window on the App Store or Google Play: you need to fill in beautiful screenshots, make an enticing description optimized for better indexing, select keywords for search. The popularity of the application directly depends on the quality of this work, that is, in fact, the result of the activities of developers, testers, designers, product managers, marketers - all involved in the creation of the product.

If the application must exist in several languages, at least a separate person is required to prepare the showcase, or even several employees: a product manager who will write texts for the description, organize translation into all languages ​​and prepare a ToR for creating screenshots, a designer who will draw screenshots with text overlaid, device contours, etc., and, of course, translators who translate all screenshots and texts into different languages.

The final part of the work is the release process itself. It takes a considerable amount of time for a small release engineering team. At this crucial, but rather routine stage, we sought to minimize the number of errors and the influence of the human factor. To do this, first of all, it was necessary to automate the loading of metadata (text and graphic design of the application window): this can significantly reduce time costs and quickly carry out business releases (for example, styling the application for Valentine's Day).

Since the decision on whether the application is ready for release in Badoo is made by a team of QA engineers, we decided to give them the right to click on the “red button” for launching the release. At the same time, we wanted it to be accessible even from mobile devices (with a visual display of script progress).

ITKarma picture

First steps towards automation: loading metadata

How it worked at the very beginning: for each release, a table was created in Google Sheets, in which the product manager uploaded the verified master text in English, after which the translators adapted it for a specific country, dialect and audience, and then the release engineer transferred all the information from this table on the App Store or Google Play.

The first step towards automation that we took was to integrate the translation of texts into our overall translation process. I will not dwell on this - this is a separate large system, which can be found in our recent article . The main point is that translators do not waste time on tablets and work with the interface for convenient loading by hand (read: ctrl + c ctrl + v) of the translated options in the page. In addition, there are the makings of versioning and the foundation for Infrastructure-as-Code.

At the same time, we added the unloading of ready-made translations from the database and their implementation in the assembled IPA file (file extension of the iOS application). The application is built with TeamCity. Thus, each version of the application always had a fresh translation without manual intervention in the assembly process.

For a while we lived like that, and on the whole, everything suited us. But the number of applications has increased, and with it the time to prepare each release.

ITKarma picture
Our reality as of 2015

On average, the release of one application with the current version of the screenshots took about one and a half to two hours of work of the release engineer in the case of iOS and about half an hour in the case of Android. The difference is due to the fact that iOS applications must go through the so-called Processing, which takes some time (it is impossible to send the application to Review until Processing is completed successfully). In addition, the App Store itself, by most operations at that time, was much slower than Google Play.

It became obvious that we needed an additional tool for delivering applications to the stores. And just at that moment, a product called Fastlane began to gain popularity in the open-source market. Despite the fact that he was still damp then, he could already solve a huge layer of our problems...

I’ll say a few words about him to make it clearer what will be discussed further.

Briefly about Fastlane

Today Fastlane is a product that is able to almost completely automate all actions from the moment of development to the release of the application on the App Store and Google Play. And it's not just about downloading texts, screenshots and the application itself - here are certificate management, beta testing, code signing, and much more.

We met Fastlane during his “youth” and instability. But now it is a confidently working and integral component of many mobile application development teams that face the problem of huge time costs for delivering their products to users. The most interesting thing about it is the ability to write custom plugins for the main component and use the community-written plugins . For such a specific product, this is a good solution, which (which is important) helps not to produce "extra" technologies in DevTools.

Confidence is also inspired by the fact that the founder and chief developer of Fastlane was hired by Google: now the component supports not only the community, but also Sam.

Over time, we have implemented most of the features provided by Fastlane in the build, sign, fill, etc. systems of our applications. And incredibly happy about it. Why reinvent the wheel, and even maintain its correct shape, when you can write a unified script once, which will spin in a CI/CD system itself?

Automate iOS releases

Due to the fact that Google Play is more friendly to developers, it took very little time to release an Android application: a couple of minutes without updating texts, videos and screenshots. Hence the lack of need for automation. But with the App Store the problem was very tangible: it took too much time to send applications to Review. Therefore, it was decided to start the automation with iOS.

We pondered the resemblance of our system of automation of interaction with the App Store (and even made prototypes), but we did not have the resources to finish and update. There was also no more or less adequate API from Apple. Well, the last nail in the coffin of our custom solution was driven by regular updates to the App Store and its mechanisms. In general, we decided to try Fastlane - then another version of 2015.

The first step was to write a mechanism for uploading translated texts for applications to the desired structure as a component of our common AIDA (Automated Interactive Deploy Assistant) internal system. This system is a kind of hub, a link between all the systems, technologies and components used in Badoo. It works on a self-written queuing system implemented on Golang and MySQL. Maintains and improves it mainly Department Release Engineering.We talked about it in more detail in the article back in 2013, since then a lot has changed. We promise to talk about her again - AIDA is cool!

At the next stage, the downloaded texts were fed by Fastlane, which uploaded them to the App Store. After that, I had to go into the App Store interface, manually select the desired downloaded version and send the application for verification if Processing had already been completed by that time.

This reduced the preparation time of the release from a couple of hours to about 30 minutes, of which only one and a half minutes had to do something with your hands! The rest of the time is to wait. Wait for the end of Processing. The mechanism was a breakthrough at that time just because it almost completely saved us from manual work in preparing the AppStore for release. We created a repository for the script, which was given access to people who are directly related to releases (project managers, release engineers).

In this mode, we lived for some time. But at some point this scheme led to the accumulation of a lot of “sacred knowledge”, the owner of which and, as a result, the overall picture of events, became a single person, and this is not good. Especially for this person himself: you can’t even go on vacation without a laptop.

In addition, around this mechanism there were a lot of disparate infrastructure components that were practically not connected with each other.

  1. We had to go to TeamCity for a fresh build, download the IPA file from there, upload it to the App Store through the Application Manager.
  2. Then go to the interface with translations in AIDA, see if all the translations are ready, run the script, make sure that it worked correctly (after all, Fastlane was still damp at that time).
  3. After that, crawl into the App Store and update the page with the version until Processing is completed.
  4. And only after that send the application to Review.

And so with each application. Let me remind you that at that time we had eight of them.

The next step was to transfer the script to our AIDA, at the same time combining and automating all the steps until the application is sent: checking for the readiness of translations, collecting data from TeamCity, notification, logging and all other benefits of the 21st century. In parallel with this, we started loading all assembled versions into TestFlight at the build stage.

TestFlight is a third-party application, once purchased by Apple to test a finished application by external testers in a practically production environment, that is, with push notifications and that's all.

ITKarma picture
AIDA well done, be like AIDA!

All this led to a reduction in time from half an hour to one and a half minutes for everything about everything: the IPA file managed to go through Processing before the moment when the team of QA engineers gave the go-ahead to launch the release. Nevertheless, we still had to go to the App Store, select the version we need and submit it to Review.

Plus, a simple interface was drawn: we all love klats-klats.

ITKarma picture
Like this, tab by tab, Ctrl + C Ctrl + V.

Automate Android releases

Next, the question arose about the automation of releases of Android applications. Although this process was much faster, I had to do quite a lot with my hands:

  1. Go to the Google Play console to make sure that the previous version is 100% rolled out or frozen.
  2. Create a new release version with updated texts and screenshots (if available).
  3. Download the APK file (Android Package), download the Mapping file.
  4. Go to HockeyApp (used to log crashes at the time), upload the APK file and the Mapping file there.
  5. Chat and unsubscribe about release status.

And so with every application.

Yes, Google Play has its own API. But why make a wrapper, keep track of changes in the protocol, maintain it and produce entities without the need, if we already use Fastlane for iOS releases? In addition, it comfortably exists on our server, boils in its own juice and is generally updated.And by that time, he had also learned to adequately release Android applications. The Stars Come Together!

The first thing we saw from everywhere was everything that was old: separate scripts, automation sketches, old wrappers for the API - this was created once as an experiment and did not represent much value. Immediately after that, we added a team to AIDA, which already knew how to take what we needed from TeamCity, load what we needed into HockeyApp, send alerts, log activity, and in general it was a member of the team.

Fastlane handled the filling in of APK and Mapping files on Google Play. I must say that it is much easier to go along the beaten track: it was realized quickly enough with a minimum amount of effort.

At a certain stage in the implementation of automation, there was a transition from APK-archives to AAB (Android App Bundle). Again, we were lucky that in hot pursuit it turned out pretty quickly to fix everything, but “entertainment” was added in connection with this transition. For example, he spoiled HockeyApp, which did not know how to use AAB archives in connection with preparing for self-sawing. So in order to comfortably continue to use it, it was necessary to disassemble the assembled archive after the AAB assembly, get the Mapping file from it, which will fly to HockeyApp, and from the AAB it was necessary to separately collect the APK file and only then upload it to the same HockeyApp. Sounds fun. At the same time, Google Play itself perfectly decomposes AAB, takes out the Mapping file and inserts it where necessary. So we got rid of one step and added a few, but there was nowhere to go.

An interface was written (again, similar to iOS), which was able to download a new version, check release along and across, manage the current active release (for example, increase rollout percentage). In this form, we gave it to the release team members of the Android QA team, began to collect feedback, fix bugs, finish the logic (and what else happens after release 1.0?).

By the way, in the future, automation gave us the opportunity to upload beta versions of applications to Google Play automatically on a schedule, which, in turn, pretty much accelerated the process of automatic and regression testing.

Unification of flow of mobile releases

By the time of the automation of Android releases, Fastlane had finally learned how to send versions of iOS applications for review. And we have a little improved version control system in AIDA.

It's time to give iOS releases at the mercy of a team of QA engineers. To do this, we decided to draw a beautiful form that would completely cover the needs that arise during the release of iOS applications: it would make it possible to select the desired build in TeamCity according to predefined parameters, choose the option of downloaded texts, whether or not to update optional fields (for example, Promotional Text).

No sooner said than done. The cookie cutter turned out to be very pretty and fully satisfies all requests. Moreover, with its implementation, it became possible to immediately select all the necessary applications with all the required parameters, so that interaction with the interface was minimized. AIDA on command sends a link to the build log, by which you can track errors that occur, make sure everything went well, get some kind of debug information like the version of the downloaded IPA file, release version, etc. That's how beautiful iOS releases are. and were transferred to the iOS QA team.

ITKarma picture
Well, pretty?

We liked the idea with the mold so much that we decided to do the same for Android releases. Whereas, we have an application written entirely in React Native, and the QA team of the application is responsible for both iOS and Android releases.

The interface already used by the Android QA team was integrated with changes in a similar form, the process was adapted to new realities - everything was as close as possible to the processes of the iOS team. This gave an incentive to finally outline a more or less final version of the documentation for both platforms (in the process of constant changes, I categorically did not want to do this), and also to unbind the release process from all the artificial limitations that have developed historically and led to unnecessary movements in unusual situations, requiring action team of release engineers.


In such a fun way for about five years (from the moment we started developing the mobile business to this day), we completely automated the assembly, testing and release processes, made them as efficient as possible and transferred the responsibility for releases to the QA team members who accept decision on application availability.

In addition to the obvious advantages, we completely got rid of scattered scripts, various “secret knowledge” tied to a single person, integrated a new component into our ecosystem, which is supported by a small team of release engineers.

It's great that now there is an opportunity to automate most routine actions, that engineers can write code whenever they want, and third-party developers can support any code without wasting precious time digging into complex interfaces. It’s especially difficult to figure out things like “Where should I tick?”, When the clock is midnight, no one is in the office, and hotfix needs to be uploaded here and now.

Five years. Why so long? Firstly, mobile releases are far from the only area of ​​responsibility for our small team. Secondly, of course, it took time to develop the new open-source project Fastlane; our release system evolved with it.

We have come a long way in this area. Perhaps it is not the most effective, perhaps some rake could be foreseen and circumvented. But it was as it was. When we started it, there were no similar articles - we ourselves worked our way up. And if you are faced with a similar task now and this article will help you with something, I will be extremely happy. But even if you didn’t get any radically new information, I hope that at least it was interesting to read at your leisure. And, perhaps, to compare with their experience. And if you have anything to say on this topic, welcome to comment !.