How time synchronization has become safe
How to make sure that time per se does not lie if you have a million large and small devices communicating over TCP/IP? After all, each of them has a watch, and the time should be true for everyone. This problem cannot be circumvented without ntp.
Imagine for a moment that in one segment of the industrial IT infrastructure, there were difficulties with synchronizing services over time. The Enterprise Software cluster stack starts to crash immediately, domains break up, masters and Standby nodes unsuccessfully try to restore the status quo.
It is also possible that an attacker intentionally tries to knock down time through a MiTM, or DDOS attack. In this situation, anything can happen:
- user account passwords will expire;
- X.509 certificates will expire;
- two-factor TOTP authentication will stop working;
- backups are "out of date" and the system will delete them;
- DNSSec will break.
It is clear that every first IT department is interested in the reliable operation of time synchronization services, and it would be nice if they were reliable and safe in industrial operation.
Break NTP in 25 minutes
Network protocols - millennials have one feature, they have been outdated for a long time and are no longer good, but replacing them is not so easy even when a critical mass of enthusiasts and funding is gaining.
The main claim to the classic NTP in the absence of reliable defense mechanisms against attackers. Various attempts have been made to solve this problem. To do this, they first introduced a pre-installed key mechanism (PSK) for exchanging symmetric keys.
Unfortunately, this method has not justified itself in the seed of a simple reason - it does not scale well. Need manual configuration on the client side depending on the server. This means that just like that you just can't add another client. If something changes on the NTP server, you must reconfigure all clients.
Then they came up with AutoKey, but immediately they discovered a number of serious vulnerabilities in the algorithm design itself and had to be abandoned. The thing is that the initial number (seed) contains only 32 bits, it is too small and does not contain enough computational complexity for a frontal attack.
- Key ID - symmetric 32-bit key;
- MAC (message authentication code) - checksum of the NTP packet;
Autokey is calculated as follows.
Where H () is the cryptographic hash function.
The same function is used to calculate the checksum.
It turns out that the whole integrity of packet checks is based on the authenticity of cookies. Having captured them, you can restore autokey and then fake the MAC. However, the NTP server uses a seed to generate them. This is where the catch lies.
Cookie=MSB_32(H(Client IP||Server IP||0||Server Seed))
The MSB_32 function cuts off the 32 high-order bits from the result of calculating the md5 hash. Client cookies do not change until the server settings are unchanged. All that remains for the attacker is to restore the initial number and get the opportunity to generate cookies on their own.
First you need to connect to the NTP server as a client and receive cookies. After that, the attacker recovers the initial number by brute force using a simple algorithm.
An algorithm for attacking the calculation of a seed by brute force.
for i=0:2^32 − 1 do Ci=H(Server-IP||Client-IP||0||i) if Ci=Cookie then return i end if end for
IP addresses are known, so it remains only to create a 2 ^ 32 hash until the created cookie does not match what it received from the NTP server. On a regular home station with Intel Core i5, it will take 25 minutes.
NTS - new Autokey
It was impossible to put up with such security holes in Autokey, and in 2012 there was new version protocol.For the sake of a compromised name, they decided to rebrand, so Autokey v.2 was dubbed Network Time Security.
NTS is an extension of NTP security and currently only supports unicast mode. It provides reliable cryptographic protection against packet manipulation, prevents snooping, scales well, is resistant to loss of network packets and leads to the least loss of accuracy that occurs during connection protection.
An NTS connection consists of two steps in which lower layer protocols are used. At the first stage, the client and server agree on various connection parameters and exchange cookies containing keys with all the associated data set. At the second stage, the actual NTS session between the client and the NTP server occurs.
NTS consists of two lower-level protocols: Network Time Security Key Exchange (NTS-KE), initializing a secure connection over TLS, and NTPv4, the last incarnation of the NTP protocol. A little more about this below.
Stage One - NTS KE
At this point, the NTP client initiates a TLS 1.2/1.3 session over a separate TCP connection to the NTS KE server. The following occurs during this session.
- Parties define the AEAD algorithm parameters for the second stage.
- Parties define a second lower layer protocol, but only NTPv4 is currently supported.
- The parties determine the IP address and port of the NTP server.
- NTS KE server issues cookies under NTPv4.
- The parties extract a pair of symmetric keys (C2S and S2C) from the cookie material.
This approach has the great advantage that the entire burden of transmitting the secret information of the connection parameters lies with the proven and reliable TLS protocol. This eliminates the need to invent your own bike for a safe NTP handshake.
Second stage - NTP protected by NTS
At the second stage, the client safely synchronizes time with the NTP server. For this purpose, it passes four special extensions (extension field) in the structure of the NTPv4 package.
- Unique Identifier Extension contains a random nonce to prevent replay attacks.
- NTS Cookie Extension contains one of the NTP cookies available to the client. Since only the client has the symmetric AAED keys C2S and S2C, the NTP server must extract them from the cookie material.
- NTS Cookie Placeholder Extension is a way for a client to request additional cookies from the server. This extension is necessary so that the NTP server response is not much longer than the request. This helps prevent gain attacks. The
- NTS Authenticator and Encrypted Extension Fields Extension contains an AAED cipher with a C2S key, an NTP header, timestamps, and the EFs mentioned above as related data. Without this extension, it's possible to fake timestamps.
Upon receiving a request from the client, the server authenticates the NTP packet. To do this, he must decrypt the cookies, extract the AAED algorithm and the keys. After successfully checking the NTP packet for validity, the server responds to the client in the following format.
- Unique Identifier Extension Mirror copy of client request, measure against attacks by retry.
- NTS Cookie Extension more cookies to continue the session.
- NTS Authenticator and Encrypted Extension Fields Extension contains AEAD cipher with S2C key.
The second handshake can be repeated many times, bypassing the first stage, since each request and response gives the client additional cookies. This has the advantage that relatively resource-intensive TLS operations for computing and transmitting PKI data are divided by the number of repeated requests. This is especially convenient for specialized FPGA chronometers, when all the main functionality can be packaged into several functions from the field of symmetric cryptography, transferring the entire TLS stack to another device.
What is the feature of NTP? Despite the fact that the author of the Dave Mills project tried to document his code as best as possible, a rare programmer will be able to figure out the intricacies of time synchronization algorithms 35 years ago. Part of the code was written before the POSIX era, and the Unix API was then very different from what is used today. In addition, you need knowledge of statistics to clear the signal from interference on noisy lines.
NTS was not the first attempt to repair NTP. After the attackers learned to use NTP vulnerabilities to enhance DDoS attacks, it became clear that radical changes were needed. And while the NTS drafts were being prepared and brought to mind, the National Science Foundation of the USA at the end of 2014 urgently allocated a grant for the modernization of NTP.
The working group was headed not just by anyone, but Eric Stephen Raymond - one of the founders and pillars of the Open Source community and author of the book Cathedral and Bazaar . First of all, Eric and his comrades tried to transfer the NTP code from the BitKeeper platform to git, but it wasn’t there. Project leader Harlan Stenn was opposed to this decision and negotiations were at an impasse. Then it was decided to fork the project code, so NTPSec arose.
Solid experience, including working on GPSD, a mathematical background and the magical skill of reading ancient code - Eric Raymond was the hacker who could pull out such a project. The team found a code migration specialist and in just 10 weeks, NTP settled in on GitLab. Work has begun to boil.
Eric Raymond's team got down to business the same way as Auguste Rodin when working with a block of stone. By removing 175 KLOC of the old code, they were able to significantly reduce the attack area, closing many security holes.
Here is an incomplete list of those under distribution:
- Undocumented, outdated, outdated, or broken refclock.
- Unused ICS library.
- Old code for Windows.
- C ntpq code rewritten in Python.
- C sntp/ntpdig code rewritten in Python.
In addition to cleaning the code, the project also had other tasks. Here is a partial list of achievements:
- Significantly increased protection of the code from buffer overflow. To prevent buffer overflows, all unsafe string functions (strcpy/strcat/strtok/sprintf/vsprintf/gets) have been replaced with safe versions that implement a buffer size limit.
- Added support for NTS.
- Tenfold increased the accuracy of the time step by linking physical equipment. This is due to the fact that modern computer clocks have become much more accurate than those that were at the time of the birth of NTP. Most of all, GPSDO and dedicated radio stations of the time benefited from this.
- The number of programming languages has been reduced to two. Instead of Perl, awk, and even S scripts, Python is now solid. Due to this, more possibilities for reusing code.
- Instead of autotools script noodles, the project began to use the waf software build system.
- Updated and reorganized the project documentation. From a contradictory, and sometimes archaic collection of documents, they created quite tolerable documentation. Each command line key and each configuration entity now has a single version of the truth. In addition, manual pages and web documentation are now created from the same main files.
NTPSec is available for a number of Linux distributions. Currently the latest stable version is 1.1.8, the penultimate one for Gentoo Linux.
(1:696)$ sudo emerge -av ntpsec These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] net-misc/ntpsec-1.1.7-r1::gentoo USE="samba seccomp -debug -doc -early -gdb -heat -libbsd -nist -ntpviz -rclock_arbiter -rclock_generic -rclock_gpsd -rclock_hpgps -rclock_jjy -rclock_local -rclock_modem -rclock_neoclock -rclock_nmea -rclock_oncore -rclock_pps -rclock_shm -rclock_spectracom -rclock_trimble -rclock_truetime -rclock_zyfer -smear -tests" PYTHON_TARGETS="python3_6" 0 KiB Total: 1 package (1 reinstall), Size of downloads: 0 KiB Would you like to merge these packages? [Yes/No]
There was another attempt to replace the old NTP with a more secure counterpart. Chrony, unlike NTPSec, is written from scratch and is designed to work reliably in a wide range of conditions, including unstable network connections, partial availability or network congestion, and temperature changes.In addition, chrony has other benefits:
- chrony can synchronize the system clock faster with greater accuracy;
- chrony is less, consumes less memory and accesses the processor only when necessary. To save resources and energy, this is a big plus;
- chrony supports hardware-based time stamps on Linux, which provides extremely accurate synchronization on local networks.
However, chrony lacks some of the features of the old NTP, such as broadcast and multicast client/server. In addition, classic NTP supports more operating systems and platforms.
To disable server functionality and NTP requests to the chronyd process, it is enough to write port 0 to the chrony.conf file. This is done in cases where there is no need to maintain time for NTP clients or peers. Starting with version 2.0, the NTP server port is open only when access is allowed with the allow directive or the corresponding command, or the NTP peer is configured, or the broadcast directive is used.
The program consists of two modules.
- chronyd is a service running in the background. It receives information about the difference of the system clock with an external time server and corrects the local time. It also implements the NTP protocol and can act as a client or server.
- chronyc is a command line utility for monitoring and controlling the program. Used to fine-tune various service parameters, for example, allows you to add or remove NTP servers while chronyd continues to work.
Starting with version 7 of RedHat Linux, uses chrony as a time synchronization service. The package is also available for other Linux distributions. Latest stable version 3.5, getting ready for v4.0 release.
(1:712)$ sudo emerge -av chrony These are the packages that would be merged, in order: Calculating dependencies... done! [binary N ] net-misc/chrony-3.5-r2::gentoo USE="adns caps cmdmon ipv6 ntp phc readline refclock rtc seccomp (-html) -libedit -pps (-selinux)" 246 KiB Total: 1 package (1 new, 1 binary), Size of downloads: 246 KiB Would you like to merge these packages? [Yes/No]
How to set up your own remote chrony server on the Internet to synchronize time in an office network. The following is an example setup on VPS.
Example of setting Chrony on RHEL/CentOS on VPS
Now let's get some practice and get our own NTP server on VPS. It is very simple, just select the appropriate tariff on the RuVDS website, get a ready server and type in a dozen simple commands. For our purposes, this option is quite suitable.
We pass to service setup and first of all we put the chrony package.
[root@server ~]$ yum install chrony
RHEL 8/CentOS 8 use a different package manager.
[root@server ~]$ dnf install chrony
After installing chrony, you need to start and activate the service.
[root@server ~]$ systemctl enable chrony --now
If you wish, you can make changes to/etc/chrony.conf, replacing the NPT server with the closest local ones to reduce response time.
# Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). server 0.ru.pool.ntp.org iburst server 1.ru.pool.ntp.org iburst server 2.ru.pool.ntp.org iburst server 3.ru.pool.ntp.org iburst
Next, we configure the synchronization of the NTP server with nodes from the specified pool.
[root@server ~]$ timedatectl set-ntp true [root@server ~]$ systemctl restart chronyd.service
You must also open the NTP port outside, otherwise the firewall will block incoming connections from client nodes.
[root@server ~]$ firewall-cmd --add-service=ntp --permanent [root@server ~]$ firewall-cmd --reload
It’s enough to set the time zone correctly on the client side.
[root@client ~]$ timedatectl set-timezone Europe/Moscow
The file/etc/chrony.conf indicates the IP or host name of our VPS server running the NTP server chrony.
Finally, start time synchronization on the client.
[root@client ~]$ systemctl enable --now chronyd [root@client ~]$ timedatectl set-ntp true
Next time I’ll tell you what options there are time synchronization without the Internet.