Dialogue Evaluation Winners - About Tasks, Language Models, ML, and About Me
Vladislav Korzun, winner of the track Dialogue Evaluation RuREBus-2020
What are you doing?
I am a developer at the NLP Advanced Research Group at ABBYY. At the moment, we are solving the one shot learning task for extracting entities. That is, having a small training sample (5-10 documents), you need to learn how to extract specific entities from similar documents. For this, we are going to use the outputs of the NER model trained on standard types of entities (Persons, Locations, Organizations) as attributes for solving this problem. We also plan to use a special language model, which was trained on documents similar in theme to our task.
What tasks did you solve at Dialogue Evaluation?
At the Dialogue, I participated in the competition RuREBus , dedicated to extracting entities and relationships from specific documents of the corps of the Ministry of Economic Development. This building was very different from the buildings used, for example, in the competition Conll . Firstly, the types of entities themselves were not standard (Persons, Locations, Organizations), among them there were even unnamed and substantive actions. Secondly, the texts themselves were not sets of verified sentences, but real documents, because of which they came across various lists, headings and even tables. As a result, the main difficulties arose precisely with data processing, and not with solving a problem, because these are essentially the classic Named Entity Recognition and Relation Extraction tasks.
In the competition there were 3 tracks: NER, RE with the given entities and end-to-end RE. I tried to solve the first two. In the first task, I used the classical approaches. At first, I tried to use a recurrent network as a model, and fasttext word embeddings, capitalization patterns, symbol embeddings and POS tags  as signs. Then I already used various pre-trained BERTs , which pretty much exceeded my previous approach. However, this was not enough to take first place in this track.
But in the second track I succeeded. To solve the problem of extracting relationships, I reduced it to the problem of classifying relationships, similar to SemEval 2010 Task 8 . In this problem, for each sentence, one pair of entities is given for which the relation needs to be classified. And in the track in each sentence there can be as many entities as you like, however, it simply reduces to the previous one by sampling the sentence for each pair of entities. Also, during training, I took negative examples randomly for each sentence in an amount not more than twice the number of positive ones, in order to reduce the training sample.
As approaches to solving the problem of classifying relationships, I used two models based on BERT-e. In the first, I simply concatenated the BERT outputs with NER embeds and then averaged the characteristics for each token using Self-attention . As the second model was taken one of the best for solving SemEval 2010 Task 8 - R-BERT . The essence of this approach is as follows: insert special tokens before and after each entity, average the BERT outputs for tokens of each entity, combine the resulting vectors with the output corresponding to the CLS token, and classify the resulting feature vector. As a result, this model won first place in the track. Competition results are available here .
 Wu, S., He, Y. (2019, November). Enriching pre-trained language model with entity information for relation classification. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management ( pp. 2361-2364 ).
What did you find most difficult in these tasks?
The most problematic was the processing of the case. The tasks themselves are as classic as possible; there are already ready-made frameworks for solving them, for example, AllenNLP. But the answer needs to be issued while preserving tokens spans, so I could not just use the finished pipeline without writing a lot of additional code. Therefore, I decided to write the entire pipeline on pure PyTorch, so as not to miss anything. Although I still used some modules from AllenNLP.
There were also quite a few long sentences in the building, which were inconvenient when training large transformers, for example, BERT, because they become demanding on video memory with increasing length of sentences. However, most of these sentences are enumerations separated by semicolons, and they could be separated by this symbol. I simply divided the remaining offers by the maximum number of tokens.
Have you previously participated in the Dialogue and the tracks?
Last year, he spoke with his master's diploma at a student session.
And why did you decide to participate in competitions this year?
At this time, I was just solving the problem of extracting relationships, but for a different corps. I tried to use a different approach based on parsing trees. The path in the tree from one entity to another was used as input. But this approach, unfortunately, did not show strong results, although it was on a par with the approach based on recurrent networks, using token embeddings and other signs as signs, such as the length of the path from the token to the root or one of the entities in the syntax tree parsing, as well as the relative position of the entities.
In this competition, I decided to participate, because I already had some best practices for solving such problems. And why not apply them in a competition and get a publication? It didn’t turn out as easy as I thought, but this is more likely due to problems with interaction with the cases. As a result, for me it was more likely an engineering task than a research one.
Have you participated in other competitions?
At the same time, our team participated in SemEval . Ilya Dimov was mainly engaged in the task, I only offered a couple of ideas. There was the task of classifying propaganda: a span of text has been highlighted and it needs to be classified. I suggested using the R-BERT approach, that is, highlight this entity in tokens, insert a special token in front of and after it, and average the outputs. As a result, this gave a small increase. Here is the scientific value: to solve the problem, we used a model designed for a completely different one.
He also participated in the ABBYY-hackathon, in the ACM icpc - sports programming competitions in the first courses. We didn’t go very far then, but it was fun. Such competitions are very different from those presented at the Dialogue, where there is enough time to calmly implement and test several approaches. In hackathons, you need to do everything quickly, there is no time to relax, there is no tea.But this is the whole charm of such events - they have a specific atmosphere.
What are the most interesting problems you solved at competitions or at work?
There will soon be a GENEA gesture generation competition, and I'm going to go there. I think it will be interesting. This is a workshop on ACM - International Conference on Intelligent Virtual Agents . In this competition, it is proposed to generate gestures for a 3D model of a person based on voice. This year I spoke at the Dialogue with a similar topic, did a short review of approaches for the task of automatically generating facial expressions and gestures by voice. I need to gain experience, because I still defend a dissertation on a similar topic. I want to try to create a reading virtual agent, with facial expressions, gestures, and of course, voice. Current approaches to speech synthesis allow you to generate quite realistic speech in the text, and gesture generation approaches - gestures by voice. So why not combine these approaches.
By the way, where are you studying now?
I am in graduate school at the Department of Computer Linguistics ABBYY at MIPT. In two years I will defend my dissertation.
What knowledge and skills gained at the university help you now?
Oddly enough, math. Although I do not integrate every day and do not multiply matrices in my mind, mathematics teaches analytical thinking and the ability to understand anything. After all, any exam involves proving theorems, and trying to learn them is useless, and understanding and proving yourself, remembering only the idea, is possible. We also had good programming courses, where we learned from a low level how to understand how everything works, analyzed various algorithms and data structures. And now, dealing with a new framework or even a programming language will not be a problem. Yes, of course, we had courses in machine learning and NLP in particular, but still, as it seems to me, basic skills are more important.
Daniil Anastasiev, winner of the track Dialogue Evaluation GramEval-2020
What are you doing?
I am developing the voice assistant "Alice", I work in the sense search group. We analyze the requests that come to Alice. A standard request example is “What is the weather like tomorrow in Moscow?” You need to understand that this is a weather request, that the request asks about the location (Moscow) and there is an indication of the time (tomorrow).
Tell us about the problem that you solved this year on one of the tracks of Dialogue Evaluation.
I was doing a task very close to what ABBYY is doing. It was necessary to build a model that analyzes the sentence, makes morphological and syntactic analysis, defines lemmas. This is very similar to what they do at school. Building a model took about 5 of my days off.
The model studied in normal Russian, but, as you see, it works in the language that was in the problem.
And does it look like what you do at work?
Probably not. Here it must be understood that this task in itself does not carry much meaning - it is solved as a subtask in the framework of solving some important business task. So, for example, at ABBYY, where I used to work, morpho-parsing is the initial stage in the framework of solving the problem of extracting information. As part of my current tasks, I do not need such an analysis. However, the additional experience of working with pre-trained language models such as BERT in itself seems to be certainly useful for my work. In general, this was the main motivation for participating - I did not want to win, but to practice and get some useful skills. In addition, my diploma was partially related to the topic of the task.
Have you participated in Dialogue Evaluation before this?
I participated in the MorphoRuEval-2017 track at the 5th course and also then took 1st place.Then it was necessary to define only morphology and lemmas, without syntactic relations.
Is it realistic to use your model for other tasks now?
Yes, my model can be used for other tasks - I posted all the source code. I plan to lay out the code using a more lightweight and faster, but less accurate model. Theoretically, if someone wants to, you can use the current model. The problem is that it will be too big and slow for most. Nobody cares about speed in a competition, it’s interesting to achieve the highest possible quality, but in practical applications, it’s usually the opposite. Therefore, the main benefit of such large models here is knowing what quality is the most achievable in order to understand what you are sacrificing.
And why are you participating in Dialogue Evaluation and other similar competitions?
Hackathons and such competitions are not directly related to my activity, but this is still a useful experience. For example, when I participated in the AI Journey hackathon last year, I learned some things that I later used in my work. The task was to learn how to pass the exam in the Russian language, that is, to solve tests and write an essay. It is clear that this is all loosely connected with work. But the ability to quickly come up with and train a model that solves a problem is very useful. My team and I, by the way, took first place.
Tell me, what kind of education did you receive and what did you do after university?
He graduated from the undergraduate and graduate programs of the Department of Computer Linguistics ABBYY at MIPT, graduated in 2018. He also studied at the School of Data Analysis (SHAD). When it came time to choose a basic department for the 2nd year, our majority of the group went to the ABBYY departments - computer linguistics or image recognition and text processing. In the undergraduate we were well taught to program - there were very useful courses. Since 4 years I have been working in ABBYY for 2.5 years. First, in the morphology group, then he worked on tasks related to language models to improve text recognition in ABBYY FineReader. I wrote code, trained models, now I am doing the same thing, but for a completely different product.
How do you spend your free time?
I like to read books. Depending on the season, I try to run or ski. I am fond of photography while traveling.
Do you have plans or goals for the next, say, 5 years?
5 years is too far a planning horizon. I don’t even have 5 years of work experience. Over the past 5 years, much has changed, now there is clearly a different feeling from life. I can hardly imagine what else can change, but there are thoughts of getting a PhD abroad.
What can you advise to young developers who are engaged in computer linguistics and are at the beginning of the way?
It is best to practice, try and compete. Beginners can take one of many courses: for example, from SHAD , DeepPavlov or even my own that I once- then spent at ABBYY.
By the way, we continue to enroll in magistracy at the ABBYY department at MIPT: image recognition and text processing (RIOT) and computer linguistics (CL). Until July 15, inclusive, send a motivation letter to email@example.com indicating the department you would like to enroll and a resume indicating the GPA average score on a 5- or 10-point scale.
Details of the master's program can be found in the video , and about ABBYY departments, read here . .