Leap Second Trips Up LinkedIn, Gawker And More

Max Smolaks,

Servers froze when they were asked to warp time

LinkedIn, FourSquare, Reddit, Gawker and other popular websites experienced technical problems on Saturday night, caused by an extra second being added to the clocks worldwide.

The so-called “leap second” was added on midnight, in order to compensate for variation in the rotation of the Earth. It led to issues in Java-based applications and servers running Linux, which were using the Network Time Protocol (NTP).

Look first, leap second

NTP is a tool that relays Coordinated Universal Time (UTC) to the various online services. UTC averages 400 atomic clocks, and is mathematically accurate. However, Earth’s rotation speed varies in response to climatic and geological events, which means that from time to time, UTC needs to be adjusted to keep it true to the length of the solar day.

There were 25 “leap seconds” added between their adoption in 1972 and June 2012, all positive. Their use is decided by the International Earth Rotation and Reference Systems Service (IERS).

NTP takes into account the time it takes the time server to reply, and even the speed between the origin of request and the server. And yet, on midnight on Saturday, some web systems returned an error when they were instructed to add a 61st second to the last minute of July by the NTP.

The glitch took out web services including Mozilla and Yelp, as well as the Amadeus airline reservation system. According to the Guardian, more than 400 Qantas flights were delayed by the glitch for as long as two hours, with staff being forced to switch to manual check-ins.

Last year, Google had proposed a method of dealing with the leap second it called the “leap smear”. By modifying its internal NTP servers, the Internet search company gradually added milliseconds to the clock throughout the day. By the time the “leap second” is introduced, Google’s servers are already using the right time, avoiding unnecessary confusion.

The incidents followed severe storms in the US on Friday, which took out Amazon web services, along with Instagram, Netflix and other websites hosted on its cloud platform. It had also left more than two million Americans without electricity.

In March, Microsoft’s Azure cloud service was brought down by a leap year problem. System developers had not prepared for the possibility that February might sometimes have 29 days, with a result that services running on Azure including some parts ofthe UK government’s G-cloud, were not available for at least a day.

Are you fluent in the language of the Internet? Take our quiz!