We continue the story of projects and products that are inextricably linked with the Russian Intel development center. And today we will talk about the Intel peer project in Nizhny Novgorod - this year it is also celebrating its twentieth anniversary! This is a well-known and widely used (as well as widely known and well-used), without exaggeration, the great OpenCV computer vision library . Its creators shared with us their memories - how it all began and what came of it.

ITKarma picture

Open library for the open community


Although we are talking about the twentieth anniversary, in fact, the life of OpenCV began much earlier.

Intel came to Russia in the very beginning of the 90s, at first it was a sales office in Moscow, but other units quickly pulled in to look at the personnel available in the country. A pilot project to create mathematical routines was created by Intel at the Institute of VNIIEF (city of Sarov) in 1992. Legends say that some procedures in the Sarov software implementation with the same accuracy turned out to be faster than the hardware implementation in the i386 FPU. This led, with the assistance of the then-existing Gor-Chernomyrdin commission (on Russian-American cooperation in the field of economics and technology), to a contract between Intel and VNIIEF for the development of optimized mathematical libraries. Here it is, exactly the case when they wanted the best, but got the better.

Under the contract, a library of optimized mathematical cores was developed in Sarov, called Intel MKL (Math Kernel Library) and has not changed it so far. A bit later, Signal Processing Library (SPL), Image Processing Library (IPL), speech recognition primitives, and some other libraries were added that required their creators to have knowledge not only of x86 programming and microarchitecture, but also of mathematics and physics.

ITKarma picture
Very Old OpenCV Banner

The contract developed successfully, it turned out to be very handy for Intel, because the technology of vector instructions MMX had just appeared, and libraries of functions optimized for it, designed for various fields of application, were indispensable for its promotion.

The team participating in the contract was in full swing, full of ideas, but the regime restrictions of the closed city of Sarov and the VNIIEF Institute, which worked for the defense industry, made it difficult to communicate with American colleagues. Having weighed all the pros and cons, part of the team decided to secede and move to Nizhny Novgorod. So, in 1997, the Nizhny Novgorod Software Technology Laboratory (NSTL) appeared, which also collaborated under a contract with Intel.

When Intel already had two laboratories in Russia, the set of libraries under development expanded. There was a library of image compression (Intel JPEG Library, IJL), video compression (Intel MPEG Library, MPL) and, finally, it is a computer vision library (Intel Computer Vision Library, CVL), which later became OpenCV. It was assumed that CVL will replenish the above family of libraries and will develop in parallel with the older "sisters". But a more amazing fate awaited her.

ITKarma picture
System Requirements OpenCV Early 2000s

Why computer vision? Why at Intel? The answer to these questions is in the three editions a book about OpenCV , written by one of the main initiators of creating and opening the world of OpenCV - Gary Bradski.

Namely, visiting the leading American universities, Gary noticed that students have a noticeable interest in research in the field of computer vision, there are studies themselves, but there is no universal and fast-working solution for them - each group was forced to invent its own bicycle.

At the same time, Intel has always been important degree of contribution of software solutions to the company's success in the market. And if MKL and other "solid" libraries of the family went along the well-worn path of commercial products with closed source code, offering the user a guaranteed level of quality and support, then this model was not suitable for CVL.

It was considered that it is more important to interest a larger number of enthusiasts in the relatively young field of computer vision algorithms, to offer them an ecosystem for experimentation and innovation, while taking advantage of the technological advantages of Intel processor architectures. So OpenCV took the path of an open source open source library.

Now, when all the large companies are involved in the development of open source and support projects developed by other teams, this solution seems obvious. But then, 20 years ago, companies were far from inspired by the idea that the programmers to whom they paid their salaries would give the code publicly. In this sense, Intel proved to be a company capable of making very innovative decisions. But it was not easy.

It seems that this was really the first open-source project from Intel! Prior to this, the company most likely already contributed to Linux and other open source projects on this platform, but information about Intel's own projects made before OpenCV could not be found.

ITKarma picture
Gary Bradsky - great and terrible funny

The same Gary Bradsky offered to release the library under an open license, adopting the principle: “It's easier to get forgiveness than permission” (it's easier to get forgiveness than permission). Of course, the library was not released in a pirated way, but it was a real example of using Intel's corporate risk-taking principle.

The final decision to release OpenCV certainly did not go without the approval of Richard Wirt, the founder and then head of a separate software development unit at Intel, who did so much to establish Intel in Russia.

At first, the attitude to this decision was wary, and in the Russian part of the company, but, since after the opening of OpenCV, a group of creators from Intel continued to manage the development, infrastructure and policies in the project, and the library began to gain popularity, everyone was used to this state of affairs, and it has become a new norm.

So, the idea of ​​CVL coming from Gary Bradsky and another U.S. Intel employee, Shinn Horng Lee, was enthusiastically and professionally accepted by the Russian side, led by Valery Fedorovich Kuryakin, who led Intel's research projects.
And work on OpenCV has begun. The first entry in the library logs is dated March 1999 .

For the first year, a certain base of algorithms for classical computer vision was assembled, which included:
  • means of working with images and contour representations of objects. Contour algorithms were important for performance on processors of that time
  • object tracking and motion analysis algorithms, including the famous Lucas-Canada optical flow algorithm, as well as CamShift, a fast color object tracking algorithm developed by the founder of the library, Gary Bradsky.
  • camera calibration ported from MATLAB by Jean-Yves Bouguet, then an Intel engineer, later one of Google Street View’s main developers
  • simple (classic) face recognition algorithms based on their own subspaces. Later, in 2002, one of its most popular algorithms of all time, a face detector based on the Viola-Jones algorithm, improved and expanded by Intel engineer Rainer Lainart, was added to the library.
  • and so on - only about 100 functions

From the first release, a module with the unusual name HighGUI, developed by team member Viktor Erukhimov, became a very important and one of the most popular parts of the library that survived everything, and even the advent of deep learning. HighGUI allowed users with just a few lines of code to read images from files, receive video from the camera, display the results of algorithms on the screen, add sliders to control parameters in real time, select points and areas on the screen with the mouse, etc.

This functionality made the library extremely convenient and popular in research circles, and when later, with the participation of Victor, support for the Python language was added to the library, there was simply no reason to not use OpenCV for experiments in computer vision. Thus, the dream of Gary Bradsky to give researchers a convenient and free tool began to be realized two years after the start of the project.

ITKarma picture
A bit of Russian flavor of OpenCV, illustration from habroposta about building OpenCV 4.0 for Raspbian

How did the name OpenCV come about? There is no consensus on this. One version is from library users, presumably from some US university. In general, the enthusiasm and activity from universities, different research groups were great. And the further - the more. By the way, the library logo is also the work of its user - Adi Shavit. According to another version, the name Gary Bradsky came up with the name OpenCV, but be that as it may, the first version called OpenCV was released in June 2000.

It is noteworthy that until 2006 (!) OpenCV lived in beta status.

OpenCV and its team


To work on the new library, a small, but intellectual, ambitious and creative team of young people was recruited shortly before graduating from UNN . The leader of the team was Vadim Pisarevsky, who subsequently received the title of Intel Principal Engineer - in his existence, in many respects, OpenCV owes it to him.

Moreover, at the time of starting work on CVL, the company already had competence in processing and recognizing images (medical radiographs, electronic boards), maintained creative communication with employees of the University of Nizhny Novgorod and the Scientific Research Institute of Higher Vocational Medicine. Here, first of all, we should recall Boris Modestovich Chudinovich, the author of inventions , an expert on contour algorithms, who generously shared with employees and interns of the company with their not only many years, but many decades of experience. But this, of course, was not enough.

ITKarma picture
Intel OpenCV Team 2019

Not only the OpenCV team was young, but also the computer vision itself - even then, long before the big data era, it was actively developing. Like other technologies in the world - software and hardware. So the team lacked just about all the knowledge. Learned how to write effective C++ code; how to test; how to put into practice articles from the Internet in order to receive and confirm, and not accidentally refute the result described in them; how to work with Linux, how to work with cameras, etc.

The team had several handbooks: Trucco & amp; Verri , Jähne and Faugeras , according to which classical algorithms were implemented. These books then could not be bought in Russia, Gary Bradsky brought them with each business trip to Nizhny Novgorod. Part of the modules (for example, camera calibration) was done using MATLAB articles and codes.

Testing for the correct implementation was a big problem because there was almost no data. Sometimes tests for computer vision algorithms are more complicated than the algorithms themselves.OpenCV had, in particular, to be tested on its creators. And not one of them suffered from this! Rather, even won. In the photo below, one of the creators of OpenCV Valery Cherepennikov, from 2013 to 2016, served as CEO of Intel Research and Development in Russia.

ITKarma picture
2001, Kalman Filter Test for Object Tracking

And here again, Gary Bradsky, showing the operation of the Motion Templates algorithm.

ITKarma picture
Image quality is typical for digital cameras 20 years ago

development time


Meanwhile, the twenty-first century continued to accelerate - it was time for OpenCV to learn new platforms. Just at that time, Intel bought StrongARM, and in addition to the traditional x86 MMX/SSE architecture, OpenCV was also optimized for ARM (XScale) WMMX, the hardware platform of the extinct ancestors of modern smartphones called PocketPC.

ITKarma picture
Pocket PC laptops with Intel XScale. 2002, photo from hpc.ru

They did not know how to call, they did not have a built-in camera and GPS, and as an OS, a memorable Windows CE was used. But it was a real computer, in a revolutionary small form factor. So, he could and should have seen something! Using OpenCV, of course.

At that time, hyper-threading began to appear in processors, the level of technology already allowed placing two cores on the processor chip, and then more. Libraries faced the task of parallelizing computations and algorithms. OpenCV initially used OpenMP technology, but later added parallelization using Intel Threading Building Blocks (TBB) and just system threads.

The architecture and interface of OpenCV has also changed significantly over time. Initially, the library had a C interface, like other libraries of high-performance functions, and was based on the structures and data types of the IPL library, embodying the idea of ​​a hierarchy of levels. Then, as a development of this idea, the C++ interface was added (which later became the main one), the structures and data types of the IPL library that ended its life were replaced with their own.

By the way, do you remember Shinn Lee, together with Gary Bradsky, who brought the idea of ​​OpenCV to Russia? He subsequently became the evangelist of the Intel Performance Primitives (IPP) , the successor to the IPL and SPL, developed in Sarov under the leadership of Boris Sabanin. Perhaps that is why OpenCV and IPP are so closely related. But closely - does not mean "tough." The connection between OpenCV and the Intel IPP library at different times was stronger when many OpenCV functions were called inside functions from IPP, and less strong when functions from IPP were replaced with their own implementation. At the same time, although Intel IPP remained closed and paid, OpenCV could automatically detect the presence of IPP and use it to accelerate.

Moreover, starting with OpenCV 3.0, the IPP functions for working with images used in OpenCV have been allocated to a separate IPP-ICV library, which is distributed free of charge with OpenCV.

In fairness, it should be noted that OpenCV is able to automatically detect and use not only IPP, but also MKL to accelerate, but the use cases of the latter are an order of magnitude smaller due to the specifics of computer vision algorithms, which often require processing of “small” kernels by MKL standards.

ITKarma picture
OpenCV Books - A Little Thing

At the peak of the first phase of development, in 2002-2003, about 15 people worked on the library at Intel, then the resources were sharply reduced. And who knows, what would be the fate of OpenCV, if not for the perseverance and perseverance of Vadim Pisarevsky. When the library had practically no financial support, Vadim changed his work several times, first inside the company, then outside to continue working on the project.

At some point, after the release of OpenCV 2.0 in 2009, the official Intel project called "OpenCV" was discontinued - the library was, it would seem, finally released by Intel in free swimming.

But at the same time, dozens of groups inside Intel continued to use the library in their work. At OpenCV, the performance of new processor architectures was tested - from servers to the Internet of things, with its help impressive demos of new technologies were created. Moreover, Intel's OpenCV was not just used, but also developed! All this time, company employees continued to contribute to the OpenCV code. So, at this time, significant optimization was added to OpenCV using the AVX2 vector instructions.

Without OpenCV, it’s hard to imagine using Intel Real Sense cameras.... Drones? Heterogeneous systems? Unmanned cars? Deep machine learning? In all these cases, one of the cornerstones of successful technology development is OpenCV.

ITKarma picture
Itseez Team 2015

Therefore, Intel's further decision was completely natural and logical. In a new round of development in 2016, Intel bought Itseez, founded by former Intel employees and best known for its key role in the development of the OpenCV library.

So, since 2016, Intel has again led the development of OpenCV.

OpenCV today in numbers and links


What is OpenCV now? For its users, this is, first of all, a repository on GitHub, and not one, but three:


On the page of the main project by stars, you can approximately judge the minimum number of developer users with GitHub - 45.2k . The number of contributors is also substantial - 1151 (about 95% are external contributors to Intel).

Developers are very fond of reading lines of code in their projects - in OpenCV there are now about a million of them. Naturally, this number is constantly increasing - only in 2019 more than 1000 patches were accepted.

According to SourceForge statistics, from September 2001 to January 2019, OpenCV was downloaded a little less than 19 million times, and official OpenCV packages for May 2020 downloaded more than 140 thousand times.

But there are also unofficial ones. Say, an unofficial OpenCV package for Python, existing since September 2016, has more than 2.5 million downloads in the last month.

As already mentioned, OpenCV is written in C++. However, there are a large number of bindings for many popular languages ​​and platforms.

  • Official support: Python, Java, JavaScript.
  • Community support: Node.JS, Matlab, C # and.Net, D, Go, Haskell, R, Rust
  • New Deployments: Swift, Julia
  • Surely you can add to this list.

The company is working on OpenCV with a large (significantly more football) experienced team. Every year in summer, students of Intel Summer Schools join it, implementing interesting additions and improvements to OpenCV.So, in the summer of 2020, students were offered the task of adding functionality to work with the OpenCV computer vision library... with sound. Furthermore, OpenCV continues to participate in the Google Summer of Code .

Taking this opportunity, the OpenCV team thanks the entire community of the project for their invaluable help - without the efforts of many people it would be simply impossible to achieve such successes and such diversity.
OpenCV continues its active development, the fifth version and many new features are ahead!

In conclusion, the news of recent days: OpenCV is moving from a BSD license to a Apache2 license.

ITKarma pictureStarting to work on OpenCV in 1999, and not knowing anything about computer vision at that moment, I did not expect the project to turn out so long-lasting, and that I linger in it for so long. But so many wonderful people passed through it, and I met and worked together with even more wonderful people from the community, so even for the sake of it it was worth it to deal with them. By the way, the community from the very beginning, and especially since 2012, when we moved to github, actively supported, with a kind word, and bug reports and patches. Sometimes there were very funny cases, for example, one guy from Germany sent us some new functionality, very neatly made, we integrated it, wrote about it in the announcement of the next release, and then after a while he wrote that thanks to such a clear demonstration of his skills managed to change his unloved work to work related to computer vision - something that he always wanted to do. Which once again confirms the proverb “lucky for those who are lucky.”
Vadim Pisarevsky, currently an employee of the Shenzhen Institute of Artificial Intelligence and Robotics for Society

ITKarma pictureOf the impressive OpenCV successes, I recalled a project at UNN with 5-7 students who had never been engaged in image recognition and even image processing, and even in programming were not experts. The project was competitive for European universities, we submitted an application, we were accepted, a database of images and videos was sent, on which we had to detect disturbing situations such as a person in the subway, leaving a bag in the passage or on the street, and leaving. Conditions, tasks, requirements, criteria were fixed. All was given a limited time. The group took a place in the middle of the list, even a little higher, at first I was upset, but then I found out that two or three teams that broke away from us noticeably had been working on this topic for years, this was their specialization, and we’ve been like that since the summer, without most universities participating in the competition passed training and experience, and there were about 20 of them. Is it worth mentioning that OpenCV was the main development tool? Without it, in such a limited time, nothing good would have happened. So “my eyes are afraid, my hands are doing” - I said something as a student and they did it.
Valery Fedorovich Kuryakin, at the time of the advent of OpenCV - Intel Research Project Manager

ITKarma pictureIt seems to me that the success of OpenCV in the early 2000s was due to the fact that the library solved standard computer vision problems that everyone needed in this area : From working with a webcam to the camera and stereo pair calibration function. Researchers then, basically, worked on matlab (computer vision toolbox appeared much later) with disparate sets of scripts, it was very inconvenient. Since the code was open, it was possible to adapt it for your tasks, and the BSD license allowed commercial use.
I am very pleased that OpenCV has become a practically necessary element for solutions using computer vision.Like many other popular open source projects, OpenCV has gone from a single-level library created by a small team to a large-scale product that is being developed by the entire community. 20 years for a software project, especially in an area where everything is changing rapidly, is a lot. It’s great that the library is still popular and widely used to solve a variety of problems.

Victor Eruhimov, currently CEO of Itseez3D Inc., OpenCV.org board member

ITKarma pictureStrictly speaking, initially I was not directly involved in the OpenCV project. When it was just the beginning, I worked in IPL, IPP projects, where I grew from an engineer to a project manager. However, OpenCV, of course, played a role in my life. In particular, when it became clear that it was necessary to divide the libraries into levels of detail, i.e. high-level ones like IPL, and subsequently OpenCV and low-level (computing cores) like IPP, and we focused on low-level libraries, after a while, we began to feel a break from the knowledge in what conditions our low-level computing cores should work. It came to understand that you also need to look at the level of algorithms in order to select the correct parts for implementation as computing cores. What do you mean right? It means that they are general enough to be reused in different algorithms, but still quite narrow-minded, containing dense calculations to effectively apply low-level optimization. And in this regard, OpenCV, the algorithms implemented in it, served as a source for discussion and selection of the interface and the low-level functions for IPP themselves. I can say that this kind of analysis helped me broaden my horizons.
Vladimir Dudnik, since 2019, head of the OpenCV team at Intel
.

Source