ITKarma picture
For the possibility of combining individual simulators into a distributed simulation system, the following standards and technologies are currently used:

  • IEEE1516 (also replaces HLA and DIS);
  • OPC;
  • CAPE-OPEN and other industry standards.

Of greatest interest is the IEEE 1516 standard, because this standard is directly related to simulators, aimed at building distributed simulation systems (protocols, recommended control and feedback methods, system architecture, etc.).

The OPC (OLE for Process Control) family of software technologies that provide a single interface for managing automation objects and technological processes is also of considerable interest, but only if integration with automation objects and technological processes is required. The CAPE-OPEN standard is used for interfacing simulators designed specifically for the chemical industry.

The Institute of Electrical and Electronics Engineers (IEEE) made a significant contribution to the standardization of modeling and simulation. Distributed modeling (imitation) is a technology for exchanging data between simulators over local or global computer networks. This allows for the joint work of individual simulators as a single controlled system of modeling or imitation. The concept of distributed modeling is based on the use of high-level architecture (HLA). In practice, the IEEE 1516 standard defines an architecture by using a single API (application programming interface). The starting points of the standard are:

  1. simple "monolithic" simulation models cannot satisfy the needs of professional users;
  2. all possible areas of application of simulation are unknown in advance;
  3. the possibility of arbitrary combination of individual simulators into complex simulation systems should be provided;
  4. The distributed modeling architecture should be as open as possible to future modeling and simulation technologies.

This IEEE 1516 is the absolute standard for the interaction of simulators and simulators in military applications, due to the stringent requirements of compatibility with simulators developed and used by the US Department of Defense and NATO. Currently, IEEE 1516 is increasingly used in the civilian sphere, in the development of simulators for training personnel of complex technical systems, in aviation, astronautics, transport, etc.

The OPC family of software technologies was developed to reduce the cost of creating and maintaining industrial automation applications. In the early 90s, developers of industrial software arose a need for a universal tool for exchanging data with devices from different manufacturers or with different interaction protocols. OPC provides developers of industrial programs with a universal fixed interface for exchanging data with any device. At the same time, device developers are providing a program that implements this interface.

To create complex simulation systems, you can combine the use of IEEE 1516 and OPC, getting the opportunity to use real equipment and SCADA systems (figure), which can be quite useful in many tasks.

The communication of IEEE 1516 standards (which is the base for simulators) and OPC (used in SCADA systems) can be implemented both directly in the simulator and through an intermediary. The role of such an intermediary, for example, for me, is performed by the National Instruments LabView package. LabView can support mathematical models of any complexity, has built-in OPC support, can act as an OPC server, has effective support for interacting with various input/output cards, which allows you to use the necessary equipment directly, but unfortunately does not have the means of interaction with IEEE 1516 that requires writing the appropriate software components.

The use of IEEE 1516 and OPC allows the creation of relatively complex distributed simulation systems, including many simulators, real equipment, SCADA systems, etc.

A separate consideration is the issue of certification of simulators or simulators regarding support for the IEEE 1516 standard. Both simulators (federates in IEEE 1516 terminology) and software libraries that implement interaction are certified. But the purpose of this certification is not to identify functional defects of the program (only certification of support for the IEEE 1516 standard).

Qualified organizations:

  • USA. The Department of Defense (DoD) Modeling and Simulation Coordination Office (M & amp; S CO). Website:
  • France. ONERA. (Office National d’Etudes et Recherches Aérospatiales) is the French national aerospace research center. Website:
  • Sweden. Pitch Technologies AB. Website:

ITKarma picture

ITKarma picture

ITKarma picture

Consider the issues of building distributed simulation systems based on the IEEE 1516 standard. The basic terms used in information support correspond to the terminology of the standard for IEEE 1516 distributed interactive simulation systems - this is a federation, federate, object, attribute and interaction. The concept of an object is defined as a model of a separate phenomenon of the real world. Objects do not have methods, but only states (only data structure without functions for processing them). The state of objects is characterized by a fixed set of attributes - exact values ​​that can change over time. Each object at any time is characterized by its state, which is determined by the set of current values ​​of its attributes. The feds are mathematical descriptions of the behavior of objects - simulation models defined in software (implemented in a directive language) or represented by the values ​​of hardware sensors. In fact, federators can be both imitators and real equipment or special software. The only requirement is to provide a single interface for interaction. Feds can manage objects by changing (updating) or receiving (displaying) the values ​​of their attributes. In particular, simulator users are also federates. The totality of all the federates participating in simulation modeling forms a federation.

The term “interaction” is defined as an instant message (event) that is not tied to a specific instance of the object or federation, occurring at the federation level (ie it is impossible to determine the sender). Interactions, unlike the states of objects, are not supported in the system constantly, but have an instantaneous nature. An example is the one-way broadcast transmission of a text message to all interested members of the federation. Hierarchical Federation Pattern (HLA/IEEE 1516) shown in Figure

The interaction of the federals is carried out using the general mechanism of interaction (RTI), implemented as a subscription. A federate interested in obtaining certain attributes and interactions of another federation must subscribe to them through RTI. The mechanism for requesting, providing, and changing attribute values ​​is shown in the figure. The mechanism for organizing distributed simulation and collaboration is shown in the figure.

ITKarma picture
Figure. Federation Hierarchical Scheme

The objects in the simulator are, as a rule, 3D models, sound sources, respectively, the attributes of such objects are position and orientation in space, size, volume, etc. In relation to simulators, as an interaction, you can consider the actions of the user (federation), for example - the inclusion of a key.

ITKarma picture
Figure.Common Interworking Mechanism (RTI)

ITKarma picture
Figure. Common Interworking Mechanism (RTI)

ITKarma picture
Figure. Organization of distributed simulation and collaboration

When creating distributed simulation systems that interact through RTI, the following important features must be considered. All elements of federations and federations must be documented in certain files (FOM (federation object model) files are used to describe the federation), federates are described in SOM files (Simulation Object Model). All data is stored only in the federates, RTI does not store any data, but only transfers it. HLA allows at any given time only one federation to change the value of an attribute (there is a special rights management mechanism for transferring rights). The feds can manage local time, the HLA uses various internal mechanisms for managing time (synchronization).

In general, the IEEE 1516 standard addresses a huge number of issues related to the creation of distributed simulation systems, such as maintaining the state of the federation, resuming the state, various mechanisms for synchronizing time, areas of interaction between the federates, etc. Due to the significant volume of the standard itself and, especially, the amount of program code to demonstrate all aspects described in the standard, only the fundamental implementation of the “basic” capabilities will be demonstrated below (figure).

ITKarma picture
Figure. IEEE 1516 Basic Features Flowchart

A more detailed description of the implementation involves the need to provide a sufficiently large listing of the program, for this reason, the reader can independently use the examples of programs that come with the software to support RTI. Quite simple examples, containing many comments, are part of The Portico Project library and are freely available at . Almost all commercial implementations of the standard also contain many examples.

As an example, consider the following federation, which consists of two federates: a radio-controlled machine and a control panel. Suppose that control is carried out by setting the speed of each of the 4 engines of the machine and turning the front wheels. A sensor is installed on the machine, which determines the distance to the obstacle and sends a signal to the control panel. To do this, you need to define two classes of objects, cYpravlenie for the control panel and cDatchik for the distance sensor. The attributes of the cYpravlenie class are wheel1, wheel2, wheel3, wheel4, wheel_angle. The attribute of the cDatchik class is distance. The following is a description file of the federation, in HLA 1.3 format (interactions are given as an example).

;; комментарий — файл федерации (FED файл) для HLA 1.3 (Fed (Federation Test) (FedVersion v1.3) (Federate "fed" "Public") (Spaces (Space "Geo" (Dimension X) (Dimension Y) ) ) (Objects (Class cYpravlenie (Attribute wheel1 reliable timestamp) (Attribute wheel2 reliable timestamp) (Attribute wheel3 reliable timestamp) (Attribute wheel4 reliable timestamp) (Attribute wheel_angle reliable timestamp) ) (Class cDatchik (Attribute distance reliable timestamp) ) ) (Interactions (Class reaction BEST_EFFORT RECEIVE (Sec_Level "Public") (Parameter dx) (Parameter dy) (Parameter dz) ) ) ) 

Next, the control simulator creates a federate and an object based on the cYpravlenie class. The simulator representing the machine also creates a federate and an object based on the cDatchik class. The feds also subscribe to the changes they are interested in, i.e. the federal machine subscribes to receive object data from the cYpravlenie class (i.e., to the cYpravlenie class), and the federal management subscribes to the cDatchik class. Thus, the machine receives changes from the control panel, and the panel receives data from the sensor in the car.

The construction of more complex simulation systems involves a fairly serious design. First, it is necessary to determine the principal composition of the federation as a first approximation, i.e., federates, federation objects, and object attributes. When drawing up the federation scheme, it is also necessary to take into account the hardware components of a distributed simulation system, i.e., sensors and control hardware devices should also be presented in the form of federates, objects, and attributes. On the picture. Shows the structure of the Federation simulator installation of a sucker rod pump.

ITKarma picture
Figure. Federation Structure

After the composition of the federation, it is necessary to determine the relations, i.e., reflect which federates publish (i.e. change) the attributes of objects and which subscribe to changes in these attributes. As a rule, at the stage of determining relations, a large number of “amendments” are established for the structure of the federation. After the necessary number of iterations of "refinement" of the structure and relations, designers must establish the fact of the "correctness of the model" of the federation. An example of the definition of links is shown in the figure (objects that do not have links are hidden).

ITKarma picture
Figure. Example of the first step in identifying relationships

The next step is the determination of the required number of computers and the corresponding distribution of feds. For example, federate “A” will function on computer “1”, federals “B, C, D” will operate on computer “2”.

ITKarma picture
Figure. Computer Distribution of Feds

As a rule, the distribution of federals is based on the cost-effectiveness of their mathematical model, if the mathematical models of the federals do not require significant computing resources, then you can use one computer, if the mathematical models of the federals require significant computing resources, you need to determine the number of computers and the corresponding distribution of federals.

At the next stage, a federation description file is compiled (see the example above), which reflects the approved “correct model” of the federation. Then, the software implementation of the federates is created by writing the appropriate code for interacting with RTI and code that implements the mathematical model of the federate. At the final stage, it is necessary to test a distributed simulation system, identify errors, "overloads" of certain components in the system (based on statistics), correct synchronization, etc.

Statistics for each federation separately, and for the federation as a whole, shows the number and types of completed requests and allows you to identify possible problems during the operation of the system.

Federal statistics example:

RTIA: Statistics (processed messages) Joined federation as car_federate Synchronization point announced: ReadyToRun Achieved sync point: ReadyToRun, waiting for federation... Federation Synchronized: ReadyToRun Time Policy Enabled Published and Subscribed Add instance object: obj_datchik Removing temporary file _RTIA_3033_ExampleFederation.fed on resign federation. Resigned from Federation Didn't destroy federation, federates still joined List of federate initiated services -------------------------------------------------- 1 Message::CLOSE_CONNEXION (MSG#1) 1 Message::DESTROY_FEDERATION_EXECUTION (MSG#3) 1 Message::JOIN_FEDERATION_EXECUTION (MSG#4) 1 Message::RESIGN_FEDERATION_EXECUTION (MSG#5) 1 Message::SYNCHRONIZATION_POINT_ACHIEVED (MSG#10) 1 Message::PUBLISH_OBJECT_CLASS (MSG#28) 1 Message::SUBSCRIBE_OBJECT_CLASS_ATTRIBUTES (MSG#32) 1 Message::SUBSCRIBE_INTERACTION_CLASS (MSG#34) 1 Message::REGISTER_OBJECT_INSTANCE (MSG#40) 1708 Message::UPDATE_ATTRIBUTE_VALUES (MSG#41) 855 Message::TIME_ADVANCE_REQUEST (MSG#91) 3 Message::GET_OBJECT_CLASS_HANDLE (MSG#112) 6 Message::GET_ATTRIBUTE_HANDLE (MSG#114) 1 Message::GET_INTERACTION_CLASS_HANDLE (MSG#116) 120516 Message::TICK_REQUEST (MSG#141) 2564 Message::TICK_REQUEST_NEXT (MSG#142) List of RTI initiated services -------------------------------------------------- 1 NetworkMessage::ANNOUNCE_SYNCHRONIZATION_POINT (MSG#13) 1 NetworkMessage::FEDERATION_SYNCHRONIZED (MSG#15) 1 NetworkMessage::DISCOVER_OBJECT (MSG#43) 1711 NetworkMessage::REFLECT_ATTRIBUTE_VALUES (MSG#45) 49 NetworkMessage::GET_FED_FILE (MSG#84) Number of Federate messages : 125662 Number of RTIG messages : 1763 RTIA: Federate destroyed TCP Socket 3 : total=122015 Bytes sent TCP Socket 3 : total=340249 Bytes received UDP Socket 4 : total=0 Bytes sent UDP Socket 4 : total=0 Bytes received Пример статистики по федерации в целом: CERTI RTIG up and running... New federation: ExampleFederation Looking for FOM file... Trying... open_test.fed --> cannot access. Now trying.../usr/local/share/federations/open_test.fed... opened. TCP Socket 7 : total=340400 Bytes sent TCP Socket 7 : total=122015 Bytes received UDP Socket 4 : total=0 Bytes sent UDP Socket 4 : total=0 Bytes received TCP Socket 6 : total=258616 Bytes sent TCP Socket 6 : total=283044 Bytes received UDP Socket 4 : total=0 Bytes sent UDP Socket 4 : total=0 Bytes received 

Time synchronization

As the practice of designing and implementing distributed simulation systems has shown, the greatest difficulty is caused by issues related to the management of the flow of time (time synchronization).

As a rule, when setting the federation time synchronization method, two parameters are set - TimeRegulating and TimeConstrained. In practice, these modes affect the process of receiving messages from other federates and are directly related to the message ordering mechanism:
  • in order of receipt (messages are transmitted in the order they were received without time control);
  • priority (incoming messages are queued with priorities, its timestamp is used to determine the priority of the message);
  • causal (provides sending messages to the federates in the manner agreed with the previous and subsequent events represented by these messages);
  • by timestamps (when using this service, messages will be sent to the federates in the order of their timestamps).

It is also worth noting the possibility of using different synchronization methods by different federates.

Software libraries for implementing RTI

A list of available implementations of HLA \ IEE1516 is available at . To date, a fairly large number of implementations are available, both commercial and non-commercial. Most of the implementations are made in the JAVA and C++ languages ​​(these languages ​​are used in the standard), but there are also implementations for MatLab, Python (CERTI project), etc.

When choosing a library, special attention should be paid to "certification" for IEEE 1516 support.As a rule, all commercial implementations have a “certificate”, free ones do not (many of the free implementations are preparing for such certification).

Table Commercial Implementations:

Non-Commercial Implementation Table:

I personally use the ONERA CERTI project ( to support the distributed application infrastructure.

Measurement of the speed of interaction of the federates through RTI

Such tests are very important in the design of distributed simulation systems, especially if different federates are located in different computer networks, and even more important when federates interact via the Internet.

To achieve the minimum time delays, it is necessary to select the server with the smallest time delays in the passage of packets (can be checked using the ping command). As an example, let us consider the work of one of the distributed systems created at the Research Institute of Electronic Materials, Tsogu. A 100 megabyte network is used (ping'a delays & lt; 0.231 ms), there is no time synchronization (to reduce delays inside the RTI), 2 computers, a server (rtig) is running on one of the machines. Federation parameters - 2 objects contain 5 attributes each (one object per federal/computer), the interaction between the federals is two-way. As a result of processing the measurements, the dependence of the number of interactions per second on the size of the transmitted data is obtained.

The results of such measurements allow us to draw many conclusions, for example, if the simulator should work in a given "real time", i.e. being updated, for example, at least 60 times per second, then no more than 300 kilobytes (for example 5 attributes, then 60 kilobytes each) should be transferred in one interaction (for Fast Ethernet). At the same time, 300 kilobytes of data transmitted 60 times per second also indicates the possibility of using RTI to transfer voice and video data between simulators, as well as for interacting with VR devices.

When federates interact over the Internet, the minimum delays are more determined by packet transmission delays. For example, if the packet transit delays between the server and the simulator are 300 ms, then the maximum number of interactions per second will not exceed 3.

The use of faster solutions, such as IpoIB (IP over InfiniBand, RFC 4392), 10G Ethernet, Myrinet-10G, etc.will increase throughput and significantly reduce latency (existing solutions allow 30 million interactions per second or more).

Interaction with real systems

As federates, not only mathematical models of objects, that is, artificial systems, but also real systems and devices can act. Examples include:

  • microphone providing audio data;
  • video data camera;
  • various input/output devices, such as joysticks (figure), printers, etc.
  • various sensors and control mechanisms connected to the computer via ADC/DAC boards;
  • real equipment and SCADA systems (Figure 2.10.1., chapter 1.4.1.) through the OPC industrial interface;
  • Interfaces to the “desktop” of a real operating system operating on a separate computer or virtual machine (figure);
  • VR devices (chapter 4.5.9.);
  • Database interface, etc.

Such capabilities are of significant interest to simulators and simulation systems in general.