Time Stamp Counter
The Time Stamp Counter (TSC) is a 64-bit register present on all x86 processors since the Pentium. It counts the number of CPU cycles since its reset. The instruction RDTSC returns the TSC in EDX:EAX. In x86-64 mode, RDTSC also clears the upper 32 bits of RAX and RDX. Its opcode is 0F 31. Pentium competitors such as the Cyrix 6x86 did not always have a TSC and may consider RDTSC an illegal instruction. Cyrix included a Time Stamp Counter in their MII.
How Linux Keeps Time
A Linux system actually has two clocks: One is the battery powered “Real Time Clock” (also known as the “RTC”, “CMOS clock”, or “Hardware clock”) which keeps track of time when the system is turned off but is not used when the system is running. The other is the “system clock” (sometimes called the “kernel clock” or “software clock”) which is a software counter based on the timer interrupt. It does not exist when the system is not running, so it has to be initialized from the RTC (or some other time source) at boot time. References to “the clock” in the ntpd documentation refer to the system clock, not the RTC.
The two clocks will drift at different rates, so they will gradually drift apart from each other, and also away from the “real” time. The simplest way to keep them on time is to measure their drift rates and apply correction factors in software. Since the RTC is only used when the system is not running, the correction factor is applied when the clock is read at boot time, using clock(8) or hwclock(8). The system clock is corrected by adjusting the rate at which the system time is advanced with each timer interrupt, using adjtimex(8).
A crude alternative to adjtimex(8) is to have chron run clock(8) or hwclock(8) periodically to sync the system time to the (corrected) RTC. This was recommended in the clock(8) man page, and it works if you do it often enough that you don’t cause large “jumps” in the system time, but adjtimex(8) is a more elegant solution. Some applications may complain if the time jumps backwards.
The next step up in accuracy is to use a program like ntpd to read the time periodically from a network time server or radio clock, and continuously adjust the rate of the system clock so that the times always match, without causing sudden “jumps” in the system time. If you always have a network connection at boot time, you can ignore the RTC completely and use ntpdate (which comes with the ntpd package) to initialize the system clock from a time server– either a local server on a LAN, or a remote server on the internet. But if you sometimes don’t have a network connection, or if you need the time to be accurate during the boot sequence before the network is active, then you need to maintain the time in the RTC as well.
Real Time Clock (RTC) / Hareware Clock
With that out of the way, let’s take a look at how Linux keeps time. It starts when the system boots up, when Linux gets the current time from the RTC (Real Time Clock). This is a hardware clock that is powered by a battery so it continues to run even when the machine is powered off. In most cases it is not particularly accurate, since it is driven from a cheap crystal oscillator whose frequency can vary depending on temperature and other factors.
The boot time retrieved from the RTC is stored in memory in the kernel, and is used as an offset later by code that derives wall-clock time from the combination of boot time and the tick count kept by the TSC.
TSC (Time Stamp Counter)
The other thing that happens when the system boots is that the TSC (Time Stamp Counter) starts running. The TSC is a register counter that is also driven from a crystal oscillator – the same oscillator that is used to generate the clock pulses that drive the CPU(s). As such it runs at the frequency of the CPU, so for instance a 2GHz clock will tick twice per nanosecond.
It does not exist when the system is not running, so it has to be initialized from the RTC (or some other time source) at boot time.
There are a number of other clock sources which we’ll discuss later, but in most cases the TSC is the preferred clock source for two reasons: it is very accurate, and it is very cheap to query its value (since it is simply a register). But, there are a number of caveats to keep in mind when using the TSC as a timing source.
- In older CPU’s, each core had its own TSC, so in order to be sure that two measurements were accurate relative to each other, it was necessary to pin the measuring code to a single core.
- Also in older CPU’s, the TSC would run at the frequency of the CPU itself, and if that changed (for instance, if the frequency was dynamically reduced, or the CPU stopped completely for power management), the TSC on that CPU would also slow down or stop. (It is sometimes possible to work around this problem by disabling power management in the BIOS, so all CPU’s always run at 100% no more, no less).
Both of these problems are solved in more recent CPUs: a constant TSC keeps all TSC’s synchronized across all cores in a system, and an invariant (or nonstop) TSC keeps the TSC running at a fixed rate regardless of changes in CPU frequency. To check whether your CPU supports one or both, execute the following and examine the values output in flags:
$ cat /proc/cpuinfo | grep -i tsc
flags : ... tsc rdtscp constant_tsc nonstop_tsc ...
The flags have the following meanings:
| Flag | Meaning |
|---|---|
| tsc | The system has a TSC clock. |
| rdtscp | The RDTSCP instruction is available. |
| constant_tsc | The TSC is synchronized across all sockets/cores. |
| nonstop_tsc | The TSC is not affected by power management code. |
Overhead of Clock Queries
The Heisenberg Uncertainty Principle says, in a nutshell, that the act of observing a phenomenom changes it. A similar issue exists with getting timestamps for latency measurement, since it takes a finite (and sometimes variable) amount of time to read any clock source. In other words, just because the TSC on a 2GHz machine ticks twice per nanosecond doesn’t mean we can measure intervals of a nanosecond – we also need to account for the time it takes to read the TSC from software.
So, how expensive is it to perform these different clock queries? Included is some sample code that you can use to measure the time it takes to query various clock sources, from both C++ and Java (using JNI to call C code).
Both the C++ and Java versions take the same approach: call the particular clock function in a tight loop, and store the result. We do this a large number of times, and hang on to the results from the final iteration. This has the effect of allowing Java to do any jitting it needs to, and for both the C++ and Java versions to help ensure that code and data is in the processor’s cache memory.
Reference
- https://en.wikipedia.org/wiki/Time_Stamp_Counter
- http://btorpey.github.io/blog/2014/02/18/clock-sources-in-linux/
- https://tldp.org/HOWTO/Clock-2.html