Especially for Habr, I begin a series of articles tutorials on using the RPA platform OpenRPA . I will be glad to receive comments and comments from you if you have any questions. I hope that this story will not leave you indifferent.


pyOpenRPA Tutorial. Managing window GUI applications


Earlier, I wrote that OpenRPA is the first open source RPA platform that allows you to completely save yourself from Paid RPA analogues. And, as it turned out in the process, this topic allows not only to remove the company from the "licensed needle", but also to increase the resulting business effects from the developed robots. After all, the architecture of the new RPA turned out to be much “lighter” and, as a result, faster.


I thank all the readers who expressed interest in my previous article - I really appreciate the opinions of others, because that’s what allows me to offer the public the most relevant solutions. Thank you again for your interest!


This article will provide detailed instructions for developing a robot that will manipulate windowed GUI applications.


Window applications are understood as all kinds of GUI applications that are not rendered in WEB browsers.


Remark. OpenRPA is now becoming pyOpenRPA


Since the publication of previous article , there have been small changes in the name of the RPA platform, namely: OpenRPA is renamed to pyOpenRPA.


What is the reason?


The fact is that the name OpenRPA itself is “talking,” and “lies on the surface.” For this reason, I chose him, and after a while others. Since the concept of pyOpenRPA is to make absolutely no money and open to everyone, there are no budgets in this RPA platform. In this regard, it is not possible to defend the monopoly on the use of the name (and this is not necessary). In this regard, it was decided to slightly adjust the name in order to save users from possible confusion.


Regarding OpenRPA from another team : I really hope that they will be able to realize their idea, surpass in all respects paid RPA platforms, and keep your open. In the open source world, we are not competitors, but colleagues who work in the same direction - towards creating a useful open source product. If we talk about their RPA platform, then the goal is to create an analogue of a commercial RPA platform with visual programming based on. The idea is very interesting and attractive, but extremely time-consuming to implement (it's not just that the best RPA platforms are constantly being developed by huge development teams, which leads to the commercialization of the project). I wish them good luck in achieving their goals.


Article Navigation


Since pyOpenRPA is a fairly large RPA platform: the tutorial will be composed as a series of articles in which key technologies will be covered. Having already mastered these technologies, you will have the opportunity to delve into what you need.


Below is the planned list of articles on this topic (for navigation):



A bit of theory and terms


Let's try to understand how GUI applications are structured and why we can manage them.


Let's start with a simple one. Let's look at the example of a classic notebook, what we see and what the robot sees.


What do we see?


Notepad_human


What does the robot see?


Notepad_pyOpenRPA


Interpretation


Thanks to the architecture of modern operating systems, third-party programs have the software ability to access UI elements - they are UIO of third-party GUI applications. This feature was originally developed in order to allow programmers to spend regression testing of its software , but later it turned out that this the opportunity can also be used in the company's business processes.


As we see in the image above, for a robot, a notepad is a collection of different UIOs. And not just UIO, but UIO with a set of different attributes, and a set of different actions . And most importantly, our robot has full access to all these attributes and actions.


Attribute Examples

  • hidden - element is hidden in the GUI interface from the eyes of the user
  • disabled - the element is not available for performing actions (clicking, hovering, etc.)


Case Studies
  • left click - left mouse click
  • right click - right click
  • type text - enter text into the active area
  • scroll up - scroll up the active area
  • scroll down - scroll down the active area
  • scroll left - scrolling the active area to the left
  • scroll right - scrolling the active area to the right

Here you can object to me that this is all nonsense, because any operating system implements algorithms that ensure the separation of information flows, and one application cannot "crawl" into another application. And I’ll tell you that it’s, indeed, but only for the software (technical) level . This whole security story doesn't work when it comes to accessing the GUI interfaces of other applications.


What is a UIO?


UIO is a User Interface Object (pyOpenRPA terminology). In order to ensure maximum compatibility, this instance is inherited from the object model developed in the pywinauto library (click to get a list of available class functions) .


This approach allows you to implement useful functionality that has already been successfully developed in other libraries, and complement it with missing functionality. In our case, the lack of functionality is the ability to dynamically access UIO objects by UIO selectors.


Rules for the formation of a UIO selector (UIOSelector)


UIO selector is a list of characteristic dictionaries (UIO specifications). These UIO specifications contain the conditions by which the pyOpenRPA library determines which UIO satisfies the conditions specified in the UIO specification. The UIO specification index in the UIO selector list indicates the nesting level of the target UIO.


In another language, a UIO selector is a list of conditions under which 0, 1, or n UIOs may fall.


The following is a list of attributes — conditions that can be used in UIO specifications:


[ { "depth_start" :: [int, start from 1] :: глубина, с которой начинается поиск (по умолчанию 1), "depth_end" :: [int, start from 1] :: глубина, до которой ведется поиск (по умолчанию 1), "ctrl_index" || "index" :: [int, starts from 0] :: индекс UIO в списке у родительского UIO, "title" :: [str] :: идентичное наименование атрибута *title* искомого объекта UIO, "title_re" :: [str] :: регулярное выражение (python диалект) для отбора UIO, у которого атрибут *title* должен удовлетворять условию данного регулярного выражения, "rich_text" :: [str] :: идентичное наименование атрибута *rich_text* искомого объекта UIO, "rich_text_re" :: [str] :: регулярное выражение (python диалект) для отбора UIO, у которого атрибут *rich_text* должен удовлетворять условию данного регулярного выражения, "class_name" :: [str] :: идентичное наименование атрибута *class_name* искомого объекта UIO, "class_name_re" :: [str] :: регулярное выражение (python диалект) для отбора UIO, у которого атрибут *class_name* должен удовлетворять условию данного регулярного выражения, "friendly_class_name" :: [str] :: идентичное наименование атрибута *friendly_class_name* искомого объекта UIO, "friendly_class_name_re" :: [str] :: регулярное выражение (python диалект) для отбора UIO, у которого атрибут *friendly_class_name* должен удовлетворять условию данного регулярного выражения, "control_type" :: [str] :: идентичное наименование атрибута *control_type* искомого объекта UIO, "control_type_re" :: [str] :: регулярное выражение (python диалект) для отбора UIO, у которого атрибут *control_type* должен удовлетворять условию данного регулярного выражения, "is_enabled" :: [bool] :: признак, что UIO доступен для выполнения действий, "is_visible" :: [bool] :: признак, что UIO отображается на экране, "backend" :: [str, "win32" || "uia"] :: вид способа адресации к UIO (по умолчанию "win32"). Внимание! Данный атрибут может быть указан только для первого элемента списка UIO селектора. Для остальных элементов списка данный атрибут будет проигнорирован. }, {... спецификация UIO следующего уровня иерархии } ] 

Example UIO selector


[ {"class_name":"CalcFrame", "backend":"win32"}, # Спецификация UIO 1-го уровня вложенности {"title":"Hex", "depth_start":3, "depth_end": 3} # Спецификация UIO 1+3-го уровня вложенности (так как установлены атрибуты depth_start|depth_stop, определяющие глубину поиска UIO) ] 

PS.The list of functions for working with the UIO selector is presented in the UIDesktop module (pyOpenRPA/Robot/UIDesktop.py). These functions will be used in the further development of the robot.
See the full list of functions of the UIDesktop module here


(Step by Step) do-it-yourself robot


So we got to the most interesting and important section of this tutorial - this is a step-by-step example of creating your first robot using pyOpenRPA.


As an experimental robot, we set ourselves the following goal: Develop a robot that will control the appearance of the interface in the Calculator application . If the view of the interface differs from the view "Programmer", then the robot will have to set this view in automatic mode.


Step 0. Prepare the Python 3 interpreter for the new robot (expand pyOpenRPA)


Unlike the vast majority of RPA platforms, pyOpenRPA implements a different approach to connecting to the project. If in other RPA platforms you need to write robots in the language of this platform (text, or graphic, or scripting language), then in the case of pyOpenRPA it is you who determine where, how and when to use this library in the project.


Several download options are available pyOpenRPA :


  • Option 1, simple. Download the pre-configured portable version from the GitLab project page
  • Option 2, more complicated. Install pyOpenRPA in your version of the Python 3 interpreter (pip install pyOpenRPA)

Step 1. Create a robot project


In order to start a robot project, you need to create a project folder. In the future, I will touch on the topic of organizing project folders for industrial software robots. But at the moment I will not focus on this in order to concentrate directly on the main thing - on the logic of working with GUI windows.


Create the following project structure:


  • RobotCalc Folder:
    • The file "RobotCalc_1.py" is the script of robot 1 that we are writing now
    • File "RobotCalc_1_Run_x64.cmd" - script to launch the robot 1
    • File "RobotCalc_2.py" - robot script 2, addition
    • File "RobotCalc_2_Run_x64.cmd" - script to launch robot 2

.cmd files play an important role in this project - it is thanks to these files that we will be able to launch the robot of interest to us with a simple click on the file.


Below is an example of the "RobotCalc_1_Run_x64.cmd" file (the file "RobotCalc_2_Run_x64.cmd" is similar):


cd %~dp0 ..\Resources\WPy64-3720\python-3.7.2.amd64\python.exe "RobotCalc_1.py" pause >nul 

Step 2. Launch the pyOpenRPA studio and create the necessary UIO selectors


  • Open calculator (win + r > calc > enter)

If you downloaded the pre-configured version of pyOpenRPA from GitLab (option 1, simple):


  • Run cmd file of web studio pyOpenRPA from the repository "pyOpenRPA \ Studio \ pyOpenRPA.Studio_x64.cmd"

If you downloaded the pyOpenRPA package using pip install pyOpenRPA (option 2, more complicated):


  • Run the python interpreter with the following arguments: python -m pyOpenRPA.Studio ". \ Studio \ SettingsStudioExample.py", where SettingsStudioExample.py is the pyOpenRPA studio startup configuration file. The pre-configured template for this file can be download from the pyOpenRPA repository in GitLab

With any of the options after 5 - 15 seconds. the web studio pyOpenRPA should automatically display (seebelow)


pyOpenRPA_studio
Appearance of the web studio pyOpenRPA


  • In the list of open windowed GUI applications, find the calculator and activate the search mode for the UI element by hovering over the mouse ("Mouse search" button)
  • Switch to the calculator (alt + tab)
  • Hover over the element that we need in order to determine the state of the calculator interface. Select the radio button Hex. A green border appears on top of the calculator thanks to the pyOpenRPA studio - this is how the studio tells us what UI element it sees in the calculator at the point where the mouse pointer is.

calc_radiobutton


Studios pyOpenRPA highlighted with a green border the detected UI element in the place of the mouse pointer on the calculator


  • In order to stop the search for a UI element, hold down the ctrl key for 2-4 seconds, after which a hierarchy will appear in the studio’s WEB interface to the UI of the element, which the studio highlighted with a green border.

pyOpenRPA_studio_calc_ui_hexp>

Studio pyOpenRPA displayed the hierarchy of finding the UI element in the calculator after sending a signal to complete the search for UI elements (long press ctrl)


  • In order to make sure that the element was detected correctly, just click the "Highlight" button on the UI of the element that interests you. The program will re-draw the green border over the UI of the element that was detected.


  • Next, click on the UI element in the hierarchy window in the studio, and then go to the edit window for the UIO selector (the UIO selector will be used later in the robot code in Python 3)



pyOpenRPA_studio_calc_ui_hex_uio

  • Let's transform our UIO selector:


    [{"title":"Калькулятор","class_name":"CalcFrame","backend":"win32"},{"ctrl_index":0},{"ctrl_index":6},{"ctrl_index":1}] 

    in the following form:



  • [{"class_name":"CalcFrame","backend":"win32"},{ "title":"Hex", "depth_start":3, "depth_end": 3}] 

    • As a result of the transformations, the unnecessary "title" condition was removed: "Calculator" and intermediate levels, which were characterized only by the index of finding UI elements. Instead, we added the search condition "title": "Hex" and set the search area "depth_start": 3, "depth_end": 3 (in our case, this is necessary because we removed explicit levels of nesting). The attributes of "class_name" are imposed by the condition that you need to look for an application with class_name=CalcFrame, and the backend tells pyOpenRPA which system to search for UI elements to use (win32 or uia, each of them has + and -)
    • Using the "Hightlight element" button, make sure that the UI element is still detected by pyOpenRPA (when you click on the button, a green border should be drawn over the UI element - the new UIO selector works correctly)

    • We will use this UIO selector in the robot to check the status of the calculator interface: if the UI element is successfully detected, the calculator mode is set to the correct one. If the UI element is not detected, then the calculator mode is set incorrectly, and it will need to be changed. In order to check the presence of a UI element by a UIO selector, we use the function pyOpenRPA.Robot.UIDesktop.UIOSelector_Exist_Bool


      lCalcHex_IsExistBool=UIDesktop.UIOSelector_Exist_Bool(inUIOSelector=[{"class_name":"CalcFrame","backend":"win32"},{ "title":"Hex", "depth_start":3, "depth_end": 3}]) # Проверить наличие UI элемента по UIO селектору 

    • In order to set the programmer mode, we will use another win32 feature - activation of the event located in the application menu (see below).



    Calc Vid Programmist


    View "Programmer" in the calculator


    Activation of the menu item is performed using the special function menu_select at the root UIO of the application GUI object.


    • Using pyOpenRPA studio, we will create a UIO calculator root object selector


      lUIOSelectorCalculator=[{"title":"Калькулятор","class_name":"CalcFrame","backend":"win32"}] # Сформировали UIO селектор из студии pyOpenRPA 

    • Next, we request the UIO object by the UIO selector, after which we call the menu_select function, into which we will pass the string address of the called menu item


      lUIOCalculator=UIDesktop.UIOSelector_Get_UIO(inSpecificationList=lUIOSelectorCalculator) # Получить UIO экземпляр lUIOCalculator.menu_select("&Вид -> &Программист") # Выполнить смену режима калькулятора 


    Step 3. Consolidate the code in the robot project


    Having all the necessary UIO selectors and functions, let's move on to compiling a complete robot script. Below I will give the code of the RobotCalc_1.py file ready to run (python.exe "RobotCalc_1.py") with a detailed description of each line.


    from pyOpenRPA.Robot import UIDesktop # Импорт модуля, который умеет управлять UI элеметами GUI приложений import time # Библиотека сна алгоритма import os # Библиотека системных функций, позволит открять калькулятор, если это потребуется lUIOSelectorCalculator=[{"title":"Калькулятор","class_name":"CalcFrame","backend":"win32"}] # Сформировали UIO селектор из студии pyOpenRPA while True: # Вечный цикл lUIOCalculator=UIDesktop.UIOSelector_Get_UIO(inSpecificationList=lUIOSelectorCalculator) # Получить UIO экземпляр lCalcHex_IsExistBool=UIDesktop.UIOSelector_Exist_Bool(inUIOSelector=[{"class_name":"CalcFrame","backend":"win32"},{ "title":"Hex", "depth_start":3, "depth_end": 3}]) # Проверить наличие UI элемента по UIO селектору if not lCalcHex_IsExistBool: # Проверить, что UI элемент отсутствует lUIOCalculator.menu_select("&Вид -> &Программист") # Выполнить смену режима калькулятора time.sleep(1) # Выполнить сон на 1 сек., после чего перейти на следующую итерацию 

    Attention! When starting the robot, make sure that the calculator is in an active state on the screen of your computer. The robot will begin to track the status of the calculator. If the programmer mode is not set in the calculator, the robot will return it to this mode within 1 second.


    Supplement. We are finalizing the robot so that it also includes a calculator (if it is turned off), opens it (if it is minimized)


    • To solve this problem, we already have all the necessary UIO selectors. It is only necessary to determine the functions that we will additionally use.


    • To start the calculator we will use the function os.system



    os.system("calc") # Открыть калькулятор 

    • To check the status of the window (minimized in the tray or maximized) is_minimized

    lUIOCalculator.is_minimized() 

    • To restore a minimized window, we will use the restore function

    lUIOCalculator.restore() # Восстановить окно калькулятора из свернутого вида 

    • In total, we get the following robot source code (file RobotCalc_2.py).

    from pyOpenRPA.Robot import UIDesktop # Импорт модуля, который умеет управлять UI элеметами GUI приложений import time # Библиотека сна алгоритма import os # Билбиотека системных функций, позволит открять калькулятор, если это потребуется lUIOSelectorCalculator=[{"title":"Калькулятор","class_name":"CalcFrame","backend":"win32"}] # Сформировали UIO селектор из студии pyOpenRPA while True: # Вечный цикл lExistBool=UIDesktop.UIOSelector_Exist_Bool(inUIOSelector=lUIOSelectorCalculator) # Проверить наличие окна по UIO селектору if not lExistBool: # Проверить наличие окна калькулятора os.system("calc") # Открыть калькулятор else: # Проверить, что окно калькулятора не свернуто lUIOCalculator=UIDesktop.UIOSelector_Get_UIO(inSpecificationList=lUIOSelectorCalculator) # Получить UIO экземпляр if lUIOCalculator.is_minimized(): # Проверить, что калькулятор находится в свернутом виде lUIOCalculator.restore() # Восстановить окно калькулятора из свернутого вида else: lCalcHex_IsExistBool=UIDesktop.UIOSelector_Exist_Bool(inUIOSelector=[{"class_name":"CalcFrame","backend":"win32"},{ "title":"Hex", "depth_start":3, "depth_end": 3}]) # Проверить наличие UI элемента по UIO селектору if not lCalcHex_IsExistBool: # Проверить, что UI элемент отсутствует lUIOCalculator.menu_select("&Вид -> &Программист") # Выполнить смену режима калькулятора time.sleep(1) # Выполнить сон на 1 сек., после чего перейти на следующую итерацию 

    PS 1. For comparison: Implementing a similar algorithm in another RPA platform using visual programming tools will require 3-4 times more screen space (due to the specifics of visual programming).


    PS 2. List of all functions in the UIDesktop module (pyOpenRPA/Robot/UIDesktop.py)
    See the full list of functions of the UIDesktop module here


    To summarize


    So, we have successfully overcome the first steps to create free software robots. Of course, this article does not cover all areas of software robotics. In the following tutorial articles, we will focus on the remaining “pillars” of robotic control (mouse, keyboard, screen recognition and web manipulation).


    I hope that the technologies discussed above will first of all help you or your company to achieve your goals. And secondly, let other RPA market participants benefit (yes, I'm talking about vendors of paid RPA platforms, many of which are based in the USA).


    I am always open to your comments, and I will be happy to help you with your questions.


    See you soon!

    .

    Source