Much ado about time — UTC and NTP: Part 2
Hello, everyone, school has been keeping much of my research time occupied, so I have not yet had a chance to dive into kubernetes related software and configurations. Instead I’ve been tinkering. I’ve looked at building out VR gloves for metaverse testing [https://hackaday.io/project/178243-lucidgloves-vr-haptic-gloves-on-a-budget], put custom OS tools on Texas Instruments calculators [http://ndless.me/] and confirmed that the Linux distribution OpenTom is working on the TomTom One 310 device through USB as opposed to SD cards [https://github.com/micky-it/Working-2021-OpenTom]. Links for these projects are attached if you’d like to try them out. We’ll be picking up where we left off on NTP by setting up our own server on a Network Attached Storage device.
To set up an NTP service in Truenas, you have a couple of options. One option is to use the tools available through the system’s onboard NAS server. The advantage of this is the ease of use and setup; you can customize the NTP pool the device chooses to connect within the GUI and OS of Truenas itself with a simple toggle. The other option is to run a dedicated virtual machine through Truenas’ hypervisor system. This lets you have full control over the OS and protocol you’ll be using. For this aspect I looked at free and open source implementation of the protocols with chrony and openntpd, selecting the latter on Alpine Linux. Configuration requires setting up files in 1 of 2 locations, although the 2nd location is a bit unorthodox. The first location is a ubiquitous ‘etc/ntp.conf’ file, making it cross compatible with other clients. The 2nd is actually within the daemon’s startup config file which is documented over on the Alpine Linux wiki [https://wiki.alpinelinux.org/wiki/Alpine_Linux:FAQ#Using_a_cron_job_to_keep_the_time_in_sync].
Ideally changes to this file would be for troubleshooting purposes only, but it can accept variables to feed into boot just like any other file. Once this setup file is created, it needs to be configured with the time server to use. For the setup, we used ‘time.nist.gov’ in order to convey a stratum-1 service through the network. Remember, each hop between NTP servers (not network devices) adds another stratum to the count, making the base host stratum 2 and adjacent stratum 3 respectively.
Once configured with time information, the device has a simple way to attach to a network interface and provide NTP for the network at large. Attaching with ‘ntpd -I <adapter>’ will allow devices on the same network to reach the service. This is surprisingly easy compared to later methods due to hypervisors having access to the kernel.
Testing this varies between clients depends on your OS. Windows uses ‘w32tm’ as its application for interacting with NTP changes, while various linux distros have ‘ntpq’ and ‘ntpview’ as valid options. Successful communications are reported back in east to read formats, meaning that the server is ready to provide time-accurate synchronization should it be used as another server.
One question here might be “why you wouldn’t connect the devices to the NIST time server by default?” Some devices may not be connected to the internet and instead rely on their neighbors to obtain time information. For security reasons, having your own stratum 0 or atomic clock server would be the ideal, however this ends up to be an expensive ordeal in most cases. Stratum 1 servers can be built from common raspberry pi server parts and some wiring, but again this is a process which is labor intensive and certainly not up to par with available hardware and internet resources. For highly secure servers, grabbing the time and then forwarding it from a centralized location is the best way to check compliance needs and keep machines working.
Not all plans work out when setting up NTP however. The idea of utilizing containers for NTP became a very appealing aspect. I had hoped to take the containerization concept and translate it over to kubernetes orchestration, giving me some quality time to learn about how k8s can manage the complex workload need of an NTP pool. This is how the NTP pooling project itself handles their infrastructure, through the use of rancher and some internal server hardware[https://news.ntppool.org/2019/09/ntp-pool-servers-on-kubernetes-on-packet/].
I was able to spin up the open source cuturra project, an alpine container running chrony with some easy environmental variable toggles[https://github.com/cturra/docker-ntp]. These worked great in containers together, and the readouts reported accurate hops across the docker internal 172.17 bridged network.
Ideally this would have been the setup needed for the machine. However, some major obstacles got in the way here. Containers on their own proved to be inadequate to manage a testing ground on my machine. While hypervisors and bare metal hardware are easy to interact with and provide appropriate low level hardware, network and configuration scripting to connect the recommend 4–7 NTP servers together, containers struggle in the networking and hosting aspect. Containers can talk to each other and relay time, but just not get the message out to anything on their own hardware or outside their own network sandbox.
Interacting and making changes to the system’s clock is a critical part of NTP time management. Containers simply aren’t designed to make changes at the kernel level which clock protocols require. This presented obstacles to my work. I tried the basic fixes of exposing ports, removing firewalls, basic network configuration, but the deeper I looked the more complexity and deterrence I found.
One such deterrent was running docker containers in privileged mode. This is a very big no-no for any developer, basically handing over root access not only to the container, but exposing critical secrets and possibly the docker client itself [https://srivatsavkondragunta.medium.com/think-twice-before-running-a-docker-container-in-privileged-mode-ca9d96d7f0]. Exposed ports are one thing, compromising your entire container space and potentially the base OS is another. I decided to abandon the idea for the time being and focus on how NTP is utilized on a computer-to-computer basis.
If you’re going to run a NTP clock server as an engineering concept, you’re better off testing on the physical hardware you’ll be working with than putting it in a container or virtual space. This is a case of the device you have working better than the device you need to protect. It contradicts traditional security best practices, but is necessary for such a low level core functionality to operate.
One other thing noted from research is the following: Do not run NTP completely offline. Unless you’re using a stratum 0 device, every connected device is bound to fall out of synchronization at some point. If you think syncing to the stratum 0 device once and then selling it off is a good idea, well that may be a creative budget hack, but in the long run it’s a risk to the timekeeping sync and accuracy process. Not to mention the adjustments that may happen outside of the closed network environment.
On security, NTP is at its basis very similar to DNS. DNS itself is a very insecure protocol and has been for years due to the nature of needing to be worked around a network firewall. Traffic transmitted through DNS to its destination is often unencrypted, as it’s the first step of connecting somewhere. Nowhere in the handshake is there a chance for TLS (or SSL) to stop you from checking that the domain request should be secured. There are technologies positioned to thwart this, such as the currently in progress DNSSEC, although adoption of this protocol has shown to be less than successful [https://blog.apnic.net/2020/07/10/why-has-dnssec-increased-in-some-economies-and-not-others/]. NTP has this same problem, opening the door to poisoning, allowing times to be modified and manipulated from an outside attacker’s server. From researching NTP pools, I’ve taken note than many also have other open ports and front facing web servers. While these may be honeypots, they also pose a common security issue of lacking necessary hardening.
The most interesting issue I’ve come across comes from an old protocol method of manipulating stratum hops over a network. Individual servers using a configuration file may find themselves susceptible to something I believe has not been explored outside the realm of security theory. I’m penning this as a “Fudge attack” for memorability and for the command name utilized to change a server’s stratum. By simulating a server in an NTP pool, either through a device name, URL or ipv4/6 address, an NTP server can impersonate and manipulate the clock of neighboring devices should synchronization not be configured with enough redundant clients[https://support.ntp.org/bin/view/Support/UndisciplinedLocalClock]. Say you have a peer-to-peer network connection using NTP, if a device’s configuration file is modified, and a time drift is detected as a result of the modification by the device now on the same stratum, then the neighboring client will follow instructions within the config file to react. In one instance, an override filter called “tinker” is used to mitigate large time jump discrepancies, making it a possible mitigation [https://stackoverflow.com/questions/14629704/ntp-client-undisciplined-local-clock/14634941].
The Chron, a well known UTS related task scheduler, is also very susceptible to attacks by malware and malicious actors. Many lolbas attack killchains are leveraged against windows OS’ are handled through the use of Chron systems [https://attack.mitre.org/tactics/TA0003/]. Unlike the above attack, these methods are well explored, and include but are not limited to creating fictitious dates to obfuscate malware on a device, elevate privileges, or establish persistence on a device.
One feature that NTP implements is a key exchange. While the web has SSL, NTP uses a Symmetric key method. In this model, if a key matches 1:1 between two devices, the check passes and communication is allowed. [https://blog.apnic.net/2019/11/08/network-time-security-new-ntp-authentication-mechanism/] According to the article, time data is still plaintext, but it uses a form of TLS security to verify the hosts and servers keys are accurate for timekeeping. The biggest proponent of this tech is the industry is fedora/red hat OS. Secure Linux is their specialty and the amount of guidelines and framework for NTP is very useful for enterprise grade security. All major time critical machines can run as NTP servers such as Windows Server, Red Hat Enterprise and so on.
This brings us back around to the start. NTP and UTC are timekeeping tools used in modern computing. As our time on earth is important and more scarce, so to does measuring the milliseconds and microseconds become more important. There is a potential for a large shift in the landscape, perhaps not as big of a shift as the Camera or Smartphone industry from titans such as Apple and Kodak of years gone by, but a significant look at how we keep track of time alongside out digital future will prevent errors, bolster security and continue to design the hardware-associated fundamentals stronger, faster and more secure from the ground up.
Let this blog be a teaser for the next upcoming piece possibly covering disposable & burner phones. While taking your own phone to conventions, not setting up a burner correctly could expose more information than your home device would. Until then, thank you for reading & stay safe.