A day in the life: Traffic spike

You were expecting only a couple of people.  And you have a couple bags of chips, some beer and a few soft drinks; that should be plenty.  Somewhere in the back there’s probably some pretzels and seltzer if things get low.  But then the door opens and people keep pouring in.  The living room fills, the kitchen, even the bedroom.  You squeeze through and look out the window, and the entire street is mobbed with people coming over.  No doubt about it: your supplies have crashed.

We know how you feel.

As our regular readers know, we’ve been working on building up the eCFR, so our systems mastermind, Nic, has been keeping a close eye on our load numbers.  But because we’re building things — hammers everywhere, and piles of plywood, and mind the cord please — when our numbers started to spike one evening a few weeks ago, Nic figured that it was some extra overhead from the feature deployment that was running.  Whatever it was, it was heavy: Nic said it was the only time he’d seen Nginx, the web server, run out of available worker processes.

But it wasn’t our deployment.  Nic looked more closely at the server logs, and it was just an enormous spike in traffic.  Not enough to actually crash the site, but enough that not everyone was able to get the information they were looking for.  Nic kept digging.  It was puzzling: the huge wave of hits came at a very specific time — between 9:20 and 9:40 at night.  They were all from different locations, so it wasn’t a crawler run amok.  What was everyone looking for?  Did someone think we had early Star Wars tickets or something?

Well, the clue was in the specific information people were looking for.  Everyone, it seemed, was suddenly interested in the 14th amendment. Cross checking that against a news twitter stream, and there was the answer.  As it turns out the Republican Presidential debate was that night.  At some point shortly after 9:20, someone mentioned the 14th amendment.  And the next thing you know, we’re out of chips and pretzels.

But of course we at the LII, being hospitable folk, don’t like to run out of munchies.  So we’ve got a plan to make sure it doesn’t happen again.  Nic is gearing up to implement demand-driven autoscaling. This will set up an automated monitor which keeps track of our load numbers, and then, if it detects a spike, adds a new machine and balances the workload between them.  Once the traffic slows down, the extra machine can quietly switch back off.

So next time the house fills up, we hope to have an extra tent ready to pitch itself automatically, and a whole tub of munchies in the garage.  Because rest assured, we’re here to welcome you, and no one likes a party without something to snack on.