skip navigation
search

In the fall of 2015, we wrote about a traffic spike that had occurred during one of the Republican primary debates. Traffic on the night of Sept. 16, 2015 peaked between 9 and 10pm, with a total of 47,025 page views during that interval. For that day, traffic totaled 204,905 sessions and 469,680 page views. At the time, the traffic level seemed like a big deal – our server had run out of resources to handle the traffic, and some of the people who had come to the site had to wait to find the content they were looking for – at that time, the 14th Amendment to the Constitution.

A year later, we found traffic topping those levels on most weekdays. But by that time, we barely noticed. Nic Ceynowa, who runs our systems, had, over the course of the prior year, systematically identified and addressed unnecessary performance-drains across the website. He replaced legacy redirection software with new, more efficient server redirects. He cached dynamic pages that we knew to be serving static data (because we know, for instance, that retired Supreme Court justices issue no new opinions). He throttled access to the most resource-intensive pages (web crawlers had to slow down a bit so that real people doing research could proceed as usual). As a result, he could allow more worker processes to field page requests and we could continue to focus on feature development rather than server load.

Then came the inauguration on January 20th. Presidential memoranda and executive orders inspired many, many members of the general public to read the law for themselves. Traffic hovered around 220,000 sessions per day for the first week. And then the President issued the executive order on immigration. By Sunday January 29th, we had 259,945 sessions – more than we expect on a busy weekday. On January 30th, traffic jumped to 347,393. And then on January 31st traffic peaked at 435,549 sessions – and over 900,000 page views.

The servers were still quiet. Throughout, we were able to continue running some fairly resource-hungry updating processes to keep the CFR current. We’ll admit to having devoted a certain amount of attention to checking in on the real-time analytics to see what people were looking at, but for the most part it was business as usual.

Now, the level of traffic we were talking about was still small compared to the traffic we once fielded when Bush v. Gore  was handed down in 2000 (that day we had steady traffic of about 4000 requests per minute for 24 hours). And Nic is still planning to add clustering to our bag of tricks. But the painstaking work of the last year has given us a lot of breathing room – even when one of our fans gives us a really big internet hug. In the meantime, we’ve settled into the new normal and continue the slow, steady work of making the website go faster when people need it the most.

You were expecting only a couple of people.  And you have a couple bags of chips, some beer and a few soft drinks; that should be plenty.  Somewhere in the back there’s probably some pretzels and seltzer if things get low.  But then the door opens and people keep pouring in.  The living room fills, the kitchen, even the bedroom.  You squeeze through and look out the window, and the entire street is mobbed with people coming over.  No doubt about it: your supplies have crashed.

We know how you feel.

As our regular readers know, we’ve been working on building up the eCFR, so our systems mastermind, Nic, has been keeping a close eye on our load numbers.  But because we’re building things — hammers everywhere, and piles of plywood, and mind the cord please — when our numbers started to spike one evening a few weeks ago, Nic figured that it was some extra overhead from the feature deployment that was running.  Whatever it was, it was heavy: Nic said it was the only time he’d seen Nginx, the web server, run out of available worker processes.

But it wasn’t our the deployment.  Nic looked more closely at the server logs, and it was just an enormous spike in traffic.  Not enough to actually crash the site, but enough that not everyone was able to get the information they were looking for.  Nic kept digging.  It was puzzling: the huge wave of hits came at a very specific time — between 9:20 and 9:40 at night.  They were all from different locations, so it wasn’t a crawler run amok.  What was everyone looking for?  Did someone think we had early Star Wars tickets or something?

Well, the clue was in the specific information people were looking for.  Everyone, it seemed, was suddenly interested in the 14th amendment. Cross checking that against a news twitter stream, and there was the answer.  As it turns out the Republican Presidential debate was that night.  At some point shortly after 9:20, someone mentioned the 14th amendment.  And the next thing you know, we’re out of chips and pretzels.

But of course we at the LII, being hospitable folk, don’t like to run out of munchies.  So we’ve got a plan to make sure it doesn’t happen again.  Nic is gearing up to implement demand-driven autoscaling.  This will set up an automated monitor which keeps track of our load numbers, and then, if it detects a spike, adds a new machine and balances the workload between them.  Once the traffic slows down, the extra machine can quietly switch back off.

So next time the house fills up, we hope to have an extra tent ready to pitch itself automatically, and a whole tub of munchies in the garage.  Because rest assured, we’re here to welcome you, and no one likes a party without something to snack on.