ITKarma picture

Hi, I'm Nikita Brigak, the server developer at Pixonic. Today I would like to talk about compensation for lags in mobile multiplayer.

A lot of articles have been written about server lag compensation, including in Russian. This is not surprising, because this technology has been actively used to create multi-user FPS since the late 90s. For example, you can recall the QuakeWorld mod, one of the first to resort to it.

We use it in our mobile multiplayer shooter Dino Squad.

In this article, my goal is not to repeat what has been written a thousand times, but to tell how we implemented lag compensation in our game, taking into account our technological stack and the features of core gameplay.

A few words about our core and technology.

Dino Squad - network mobile PvP-shooter. Players control dinosaurs laden with a variety of weapons and fight 6 to 6 with each other.

Both the client and the server are on Unity. The architecture is quite classic for shooters: the server is authoritarian, and client prediction works on clients. The game simulation is written using in-house ECS and is used both on the server and on the client.

If you first heard about lag compensation, here is a brief digression into the problems.

In multiplayer FPS games, the match is usually simulated on a remote server. Players send their input to the server (information about the keys pressed), and in response, the server sends them an updated game state, taking into account the data received. With this interaction scheme, the delay between pressing the “forward” key and the moment the player’s character moves on the screen will always have more ping.

If on local networks this delay (popularly referred to as input lag) can be invisible, then when playing via the Internet it creates a feeling of “sliding on ice” when controlling a character. This problem is doubly relevant for mobile networks, where the case when a player has a ping of 200 ms is considered an excellent connection. Ping is often 350, and 500, and 1000 ms. Then it becomes almost impossible to play fast track shooter with the input lag.

The solution to this problem is client-side simulation prediction. Here, the client itself applies the input to the player’s character, without waiting for a response from the server. And when the answer is received, it simply compares the results and updates the positions of the opponents. The delay between pressing a key and displaying the result on the screen is minimal in this case.

Here it is important to understand the nuance: the client always draws himself according to his last input, and enemies - with a network delay, as before, from the data from the server. That is, when shooting at an opponent, the player sees him in the past relative to himself. Read more about client prediction we wrote earlier .

Thus, client prediction solves one problem, but creates another: if a player shoots at the point where the enemy was in the past, the server may not be in the same place when shooting at the same point of the enemy. Server lag compensation is trying to solve this problem. When firing a weapon, the server restores the state of the game that the player saw locally at the time of the shot and checks if he could really hit the enemy. If the answer is yes, the hit counts, even if the enemy is no longer at this point on the server.

Armed with this knowledge, we started implementing server-based lag compensation in Dino Squad. First of all, I had to understand how to restore on the server what the client saw? And what exactly needs to be restored? In our game, the hits of weapons and abilities are calculated through rakecasts and overlaps - that is, through interactions with the physical colliders of the enemy. Accordingly, the position of these colliders that the player "saw" locally, we needed to play on the server. At that time, we used Unity version 2018.x. The physics API is static there, the physical world exists in a single copy. There is no opportunity to save his condition so that he can later be restored from the box.So what to do?

The solution was on the surface, all of its elements were already used by us to solve other problems:

  1. About each client, we need to know at what time he saw opponents when he pressed the keys. We already wrote this information in an input package and used it to adjust the performance of client prediction.
  2. We need to be able to keep a history of game states. It is in it that we will hold the positions of opponents (and hence their colliders). We already had a state history on the server, we used it to build deltas . Knowing the right time, we could easily find the right state in history.
  3. Now that we have the state of the game from history on hand, we need to be able to synchronize the data about the players with the state of the physical world. Existing colliders - move, missing - create, superfluous - destroy. This logic has already been written for us and consisted of several ECS systems. We used it to keep several game rooms in one Unity process. And since the physical world is one per process, it had to be reused between rooms. Before each tick of the simulation, we “reset” the state of the physical world and reinitialized it with data for the current room, trying to reuse the Unity game objects to the maximum through a tricky pool system. It remained to call the same logic for the game state from the past.

Putting all these elements together, we got a “time machine” that could roll back the state of the physical world to the right moment. The code is straightforward:

public class TimeMachine : ITimeMachine {//История игровых состояний private readonly IGameStateHistory _history;//Текущее игровое состояние на сервере private readonly ExecutableSystem[] _systems;//Набор систем, расставляющих коллайдеры в физическом мире//по данным из игрового состояния private readonly GameState _presentState; public TimeMachine(IGameStateHistory history, GameState presentState, ExecutableSystem[] timeInitSystems) { _history=history; _presentState=presentState; _systems=timeInitSystems; } public GameState TravelToTime(int tick) { var pastState=tick == _presentState.Time ? _presentState : _history.Get(tick); foreach (var system in _systems) { system.Execute(pastState); } return pastState; } } 

It remained to understand how to use this machine for the lag compensation of shots and abilities.

In the simplest case, when the mechanics are built on a single hitscan, everything seems to be clear: before the player shoots, you need to roll back the physical world to the desired state, make a rakecast, count the hit or miss and return the world to its original state.

But in Dino Squad there are very few such mechanics! Most of the weapons in the game are created by projectiles - long-lived bullets that fly several ticks of the simulation (in some cases, dozens of ticks). What to do with them, at what time should they fly?

In the ancient article about the Half-Life network stack, the guys from Valve asked the same question and their answer was that : Project-based lag compensation is problematic and better avoided.

We didn't have this option: weapon based on projectiles was a key feature of the game design. Therefore, we had to invent something. After a little braking, we formulated two options that seemed to us workers:

1. We tie the project to the time of the player who created it. Each tick of server simulation for each bullet of each player, we roll back the physical world to a client state and perform the necessary calculations. This approach made it possible to have a distributed load on the server and a predictable flight time for project files. Predictability was especially important for us, as we have all projectiles, including opponents' projectiles, predicted on the client.

ITKarma picture
In the picture, the player in the 30th tick fires a missile in advance: he sees in which direction the enemy is running, and knows the approximate speed of the rocket. Locally, he sees that he hit the target in the 33rd tick. Thanks to lag compensation, it will also get to the server

2. We do everything the same as in the first version, but, having counted one tick of the bullet simulation, we do not stop, but continue to simulate its flight within the same server tick, each time bringing its time to the server tick one tick and updating collider positions. We do this until one of two things happens:

  • The bullet has expired. This means that the calculations are over, we can count the miss or hit. And this is the same tick in which the shot was fired! For us it was both a plus and a minus.Plus - since for the shooting player this significantly reduced the delay between hitting and reducing the health of the enemy. Minus - since the same effect was observed when shooting opponents at the player: the enemy, it would seem, only fired a slow missile, and the damage has already been counted.
  • The bullet reached server time. In this case, its simulation will continue in the next server tick without lag compensation. For slow projectiles, this could theoretically reduce the number of “kickbacks” of physics compared to the first option. At the same time, the uneven load on the simulation increased: the server was idle, then for one server tick it counted a dozen simulation ticks for several bullets.

ITKarma picture
The same scenario as in the previous picture, but calculated according to the second scheme. The rocket “caught up” server time in the same tick that the shot occurred, and the hit can be counted for the next tick. In the 31st tick, lag compensation is no longer applied in this case

In our implementation, these two approaches differed literally in a couple of lines of code, so we both sawed it off, and for a long time we had them in parallel. Depending on the mechanics of the weapon and the speed of the bullet, we chose one or another option for each dinosaur. A turning point here was the appearance in the game of a mechanic like "if you hit the enemy so many times in such a time, get such a bonus." Any mechanics where the time at which the player hit the enemy had an important role refused to be friends with the second approach. Therefore, in the end, we settled on the first option, and now it is used for all weapons and all active abilities in the game.

We should also raise the issue of performance. If it seemed to you that all this would slow down, I answer: the way it is. Moving colliders, turning them on and off Unity is pretty slow. In Dino Squad, in the “worst case” scenario, several hundred project files can exist simultaneously in a battle. Moving colliders to count each project individually is an impermissible luxury. Therefore, it was absolutely necessary for us to minimize the number of “kickbacks” of physics. To do this, we created a separate component in ECS, in which we record the player’s time. We hung it on all entities requiring lag compensation (project-related, ability, etc.). Before we start processing such entities, we cluster them by this time and process them together, rolling the physical world once for each cluster.

At this stage, we got the whole working system. Its code is somewhat simplified:

public sealed class LagCompensationSystemGroup : ExecutableSystem {//Машина времени private readonly ITimeMachine _timeMachine;//Набор систем лагкомпенсации private readonly LagCompensationSystem[] _systems;//Наша реализация кластеризатора private readonly TimeTravelMap _travelMap=new TimeTravelMap(); public LagCompensationSystemGroup(ITimeMachine timeMachine, LagCompensationSystem[] lagCompensationSystems) { _timeMachine=timeMachine; _systems=lagCompensationSystems; } public override void Execute(GameState gs) {//На вход кластеризатор принимает текущее игровое состояние,//а на выход выдает набор «корзин». В каждой корзине лежат энтити,//которым для лагкомпенсации нужно одно и то же время из истории. var buckets=_travelMap.RefillBuckets(gs); for (int bucketIndex=0; bucketIndex < buckets.Count; bucketIndex++) { ProcessBucket(gs, buckets[bucketIndex]); }//В конце лагкомпенсации мы восстанавливаем физический мир//в исходное состояние _timeMachine.TravelToTime(gs.Time); } private void ProcessBucket(GameState presentState, TimeTravelMap.Bucket bucket) {//Откатываем время один раз для каждой корзины var pastState=_timeMachine.TravelToTime(bucket.Time); foreach (var system in _systems) { system.PastState=pastState; system.PresentState=presentState; foreach (var entity in bucket) { system.Execute(entity); } } } } 

All that was left was to adjust the details:

1. Understand how much to limit the maximum range of movement in time.

It was important for us to make the game as accessible as possible in the conditions of poor mobile networks, therefore we limited the story with a margin of 30 ticks (with a tick rate of 20 Hz). This allows players to hit opponents even at very high pings.

2. Determine which objects can be moved in time and which cannot.

Obviously, we are moving the opponents. But installed energy shields, for example, no. We decided that it is better to give priority to defensive ability, as is often done in network shooters. If a player has set up a shield in the present, lag-compensated bullets from the past should not fly through it.

3. Decide whether it is necessary to compensate for the abilities of dinosaurs: bite, tail, etc. We decided what was needed and process them according to the same rules as bullets.

4. Determine what to do with the colliders of the player for whom the lag compensation is performed. In a good way, their position should not be shifted to the past: the player must see himself in the same time in which he is now on the server. Nevertheless, we also roll back the colliders of the shooting player, and there are several reasons for this.

Firstly, it improves clustering: we can use the same physical state for all players with close ping.

Secondly, in all rakecasts and overlaps, we always exclude the colliders of the player who owns the abilities or projectiles.In Dino Squad, players control dinosaurs that have rather unusual geometry by the standards of shooters. Even if the player shoots at an unusual angle, and the bullet path passes through the player’s dinosaur collider, the bullet will ignore it.

Thirdly, we calculate the positions of the dinosaur’s weapons or the point of application of the ability using data from the ECS even before the start of lag compensation.

As a result, the actual position of the colliders of the lag-compensated player is insignificant for us, so we went along a more productive and at the same time simpler way.

Network latency cannot simply be removed; it can only be masked. Like any other disguise method, server lag compensation has its own tradeoffs. It improves the gaming experience of the shooting player at the expense of the player who is being shot. For Dino Squad, however, the choice here was obvious.

Of course, I had to pay for all this with the increased complexity of the server code as a whole - both for programmers and game designers. If earlier the simulation was a simple sequential call of systems, then with lag compensation, nested loops and branches appeared in it. To make it convenient to work with, we also spent a lot of effort.

In the 2019 version (or maybe a little earlier), Unity has a plus or minus full support for independent physical scenes. Almost immediately after the update, we implemented them on the server, because we wanted to quickly get rid of the physical world common to all rooms.

We gave each game room its own physical scene and thus got rid of the need to “clear” the scene from the data of the next room before calculating the simulation. Firstly, it gave a significant increase in productivity. Secondly, it allowed to get rid of a whole class of bugs that arose if the programmer made a mistake in the code for clearing the scene when adding new game elements. Such errors were difficult to debug, and they often led to the fact that the state of physical objects from the scene of one room "flowed" into another room.

In addition, we did a little research on whether physical scenes can be used to store the history of the physical world. That is, conditionally, to allocate not one scene to each room, but 30 scenes, and to make a circular buffer from them, in which the history is stored. In general, the option turned out to be working, but we did not introduce it: it did not show some kind of crazy increase in productivity, but it required rather risky changes. It was difficult to predict how the server will behave during prolonged work with so many scenes. Therefore, we followed the rule: " If it ain't broke, don't fix it ".