Go School


As you know, in mid-May, Ozon announced the launch of the Go programming school . They promised the following:

  • free training
  • Ozon’s real Go development knowledge
  • Ozon job opportunity

To get to school, you had to:

  • have experience in industrial programming
  • pass test tasks for programming on the Yandex.Conest platform
  • get skype interviews

Already during the selection it turned out that the online school will only be until the epidemiological situation is improved. However, nonresident residents were still promised the opportunity to take training on the Internet.

At that time, the number of students who were ready to accept the School was announced - about 40 people.

So little by little the conditions of admission grew by new items, among the added ones it also appeared:

  • it is advisable to live in Moscow
  • be a citizen of the Russian Federation
  • age over 18

But all this became clear later, but for now the Ozon offer has attracted many developers. It was time to start the first step: passing the test.

It seemed that everything looked good, the conditions are not so complicated and quite feasible.

ITKarma picture


First, it was necessary to solve five programming problems on the Yandex.Contest platform. The first four tasks were typical algorithmic problems that they love to give at interviews. These tasks could be solved in any language from the proposed list. The last - fifth - was to be decided on Go.

Task E


Description
It is necessary to write the function func Merge2Channels (f func (int) int, in1 & lt; -chan int, in2 & lt; -chan int, out chan & lt; -int, n int) in package main.
Description of her work:
do the following n times:
  • read one number from each of the two channels in1 and in2, call them x1 and x2.
  • calculate f (x1) + f (x2)
  • write the resulting value to out

The Merge2Channels function must be non-blocking, immediately returning control.
Function f can work for a long time, waiting for something or performing calculations.

Input Format
The number of iterations is passed via argument n. Integers are supplied via the channel arguments in1 and in2.
The function for processing numbers before addition is passed through the argument f.

Output Format
The channel for outputting the results is passed through the out argument.

Notes
You must submit the task under the Make compiler. Decisions that give an incorrect answer may, for technical reasons, receive a Runtime Error verdict. Slower solutions get the Idleness Limit verdict, it is worth considering this as an excess of execution time.

Having completed all the tasks except the last one, we proceed to the task E. The description of the task did not seem complicated, it was only necessary to make a non-blocking function code and return the results in the correct order, but all attempts were unsuccessful and the platform constantly returned WA (Wrong Answer). Although 100 tasks were allotted for each task, it was worth spending them carefully, because the most frugal was promised an advantage in the selection. For two days I tried to figure out what was going on, spent 20 attempts, but all the time I got WA.
It was decided to create a telegram group with the same losers and hold a brainstorm. The size of the group increased rapidly and the more options we tried, the more often the guesses sounded that the problem was not a solution, but a platform or test.

Here begins the investigation of how Yandex.Contest works.
Several people have found a way to view the contents of files on the platform through the output in stderr. So we managed to get the task tests (main_test.go), directory structure and makefile.
Locally tests passed, but the platform stubbornly betrayed WA.
The collected information and analysis of the Yandex platform.The contest allowed us to formulate the algorithm of the tests and the cause of the unfortunate error.

How tests work for job E

How tests for job E work


According to the information that we managed to get from the Yandex.Contest platform

Stages (Make):
1) Compilation.
1.1) There is no compilation in our task. Just double-shifts the solution file.
1.2) Archiving.
At this stage, there are already “see input format” and “see output format” files. They are empty.

2) Execution.
2.1) Unzipping.
2.2) go test. This is where the solution is checked. If it is incorrect, then throws panic. If true, then the go run run result is output to STDOUT. This is the MOST important point. Remember this.
Startup File
#!/bin/bash go test 


3) Check by checker.
A checker is simply any scripting language that takes three parameters as an input:
input (test file), answer (expected response of the test file), output (output of a custom solution at the startup stage).
The input and output parameters are specified in the admin panel, and in our case they are not important.

But the most interesting is the answer. There, in our task, the contents of the file “see output format” come. BUT! No one writes anything there and it is always empty in good condition.
As a result, they just needed to redirect the output of the go test result and check the output with a checker.

Since nothing is written to the “see output format” file, the system considers all decisions to be incorrect and we get the result WA.

Knowing how the system works, I managed to create a cheat script. He wrote an arbitrary character to the file “see output format”, which is why the platform checker considered the task complete. In fact, it was the only (albeit dishonest) way to complete Mission E successfully.

Total. If you’re not ready for a little fraud and submit the absolutely correct solution, you will receive only WA .

ITKarma picture

Answers from Ozon


After this study, several people wrote to Ozon and the school representatives with a detailed description of bugs. Answers to this were received in the spirit: “Everything works for us, we see that several people completed the task.” Many letters requesting verification of the correctness of the assignment remained unanswered at all.

Reply by mail:
We have not yet conducted a code review of decisions made by the system through our testing. We will carry out only after the selection. Now there are more successful decisions, these are the decisions that were made with the verdict “ok”. We did not change anything in the task, as I wrote earlier. As you understand, until the end of the selection I can not share information about the decision. We will try to do a task analysis and share information with all the participants in the selection.

During the contest, Ozon twice published comments that all participants could see. The second time there was an extensive piece dedicated to assignment E. It says a lot and at the same time nothing:
After analyzing your premises for the task E, we came to the conclusion that the test for this task is too general and broad. In this case, one test is not enough. Some borderline situations are accepted as correct. An error in the system that is issued in response to a participant’s decision is not always predictable. We have no way to refine this in real time: there is a risk of losing existing solutions. We will work on this after the completion of the selection. Difficulties are also associated with the fact that Go is a new language for the platform. Make, as a compiler, was added at our request in order to process solutions to the problems on Go.

At the same time, Ozon considered it necessary to chide the contestants who decided to cut corners:
Until the time of writing the cheat script, we received 32 correct solutions to this problem. Qualifying competitions did not imply any pentest of the system, therefore, we ask you to treat the selection problems in good faith.

Employees of the company were even seen at a forum where visitors came together to jointly solve task E - they threatened them with a finger and appealed to their honesty.

Numbers, statistics and sadness

Figures, statistics and sadness


On the first of June, messages appeared in the group about the first invitations for interviews. Everyone held their breath. If error messages could still be ignored, then in an interview we will get answers to all questions.
But strange things happened at the interviews - no analysis of tasks, no questions about the contest. Just a few standard questions about location, hobbies and desire to study at school. The interviewer does not know anything about task E, nor about problems with it.

According to the organizers, more than 4,000 people submitted applications, which came as a surprise to them. In response to numerous questions and complaints, Ozon organized an online meeting at Zoom to answer all school questions. But there was no unequivocal answer to the situation with task E. A statement was made that task E would not be taken into account in the selection.

In the telegram group, the number of people reached 180. And not one could pass the task of E.
After analyzing the test system, many in the chat for a couple of days could not believe that checking task E contains an error. It’s hard to put up when a few days of your life are wasted because of someone’s negligence.

After the head of the Ozon school did not admit that task E was broken and impossible to complete, the “bombing” began in the group. Especially we did not understand why it is impossible to fix the test system.
It added bitterness to the fact that they did not take to school those who did the assignments, but those who met the selection criteria. One of the group members said that his friend was invited for an interview, although he did only two of the four tasks that could be completed without tricks.

On June 17, as promised, Ozon sent everyone letters of results (late at night).

Conducted several polls, here are the results:

Age structure of the group (131 votes) :
32% - 25-29 years
27% - 20-24 years
24% - 30-34 years
8% - 35-40 years old
5% - up to and including 19 years
4% - 40+ years

Contrary to expectations, most of the audience gathered mature.

Geography of participants (60 voters) :
35% - Moscow
20% - Volga Region
15% - Central Region
8% - St. Petersburg
7% - Moscow Region
5% - Northwest
3% - South
2% - Ukraine
3% - Belarus
2% - Kazakhstan

Apparently, a lot of people were not from Moscow, but, in general, it is understandable - Ozon promised an online school. Judging by the reviews of those who were invited for an interview, they were people from different regions, moreover, a person from Minsk also interviewed, which made the selection logic completely incomprehensible.

Screening Results (100 Voters) :
80% - refused | 12% - received an invitation
8% - nothing came


Refusal was received by both those who were not invited for an interview, and those who were interviewed.

What should be the selection of a healthy person


1. Rating! Objective and open, which shows a list of those who did the assignment, passed the interview, who scored how many points.

2. Clear criteria for how points were awarded. There must be publicly available criteria by which everyone can calculate their points and check.

3. Why was there a mistake with assignment E and why was it not recognized right away? What kind of tricks and attempts to get away from the recognition of their jamb? Why didn’t they immediately fix the problem, but persistently talked about some 32 participants? Who is this noodle for? What kind of disrespect?

4. Apologize in case of a jamb and tell about further actions, as well as about what will be undertaken to improve the selection procedure.

Unsubscribe level “Everything works, everything is normal, 32 people decided, then it broke, Yandex is to blame, because they have a make-up curve. Start with yourself, look for a mistake in yourself ”, etc. Or they’ve been selected according to “experience in completing test tasks” - that’s absolutely, excuse me, in no way.

How was the selection from Ozon


1. No rating.The decision of who to call and who not was made in a completely unknown way to us. Selected by beauty? According to the age? The length of the arms and other organs? Do you need good specialists or metrosexuals for posing in shop windows? I suppose that the decision was made solely on the basis of the participants' resumes. I hope hr received the award for processing 4000 resumes

Imagine if you sent the Unified State Examination results by 399 points in four subjects to the university and completed the introductory exam by 98, and you would receive a reply “Try again, you have little experience in passing selections” - would you like this option? The situation is exactly the same here.

2. The task does not work, and moreover, the organizers do not want to admit it to the last, considering their competitors complete degenerates - brandishing 32 mythical participants who passed the selection.

What’s the problem of saying: “Yes, we were stuck, we’ll roll the fix, everything will work.”

3. Formal unsubscribes. Like go get the “Experience of passing test tasks” - what, excuse me?

4. Go to school tasks that require Go knowledge... I just killed the statement of the task “The task can be completed only on Go”.

5. Silence when trying to find out, write to the post office or in PM. “We did not expect 4k people.” And how much did you expect? 15 people? It’s strange enough to expect a small number of participants and at the same time publicize the school on all available platforms/chats/channels in RuNet

Conclusion


Ozon launched a school to hire programmers, but the company was not ready for an influx of comers. The situation in the test system exacerbated the situation even more. But even when the companies provided a ready-made analysis, Ozon did not recognize the problem.

The hope that Ozon will change his mind has warmed to the end. In the evening of June 23, company employees published results of selection for Go School . In the post there was a place for statistics, a description of the tasks and even a clumsy recognition of problems with task E. But those who followed this story were waiting for a completely different recognition. Ozon chose a simple path: he lavished generously with accusations, just not to mess himself. It is doubly ironic that the solutions proposed by Ozon for task E also do not pass tests on Yandex.Contest. But for several weeks the company has already had a detailed analysis of the problem. He remained without attention, finding his peace somewhere in bottomless corporate inboxes.

Mistakes happen. It is better to recognize them in a timely manner than pull them to the last, hoping that no one will notice. And when you poke your nose into your failure, then stubbornly and instruct PR managers to hush up the issue is completely pointless. And if this analysis makes at least one company more careful in organizing such contests, then this text is written not in vain.

Links



Thanks for the help in writing this article to the members of the Gozone

UPDATE : OZON fixed the checker, now everything works as it should (1 month has passed).

Source