Blue Screen

So you may have noticed the little incident we had this afternoon.

At midday we put the Buffalo Trace Antique Collection up on the site and to much excitement you all came rushing to buy the limited number of bottles available. A few moments later our site went down, preventing a lot of lovely people from purchasing their whisky, for which we are truly sorry. We were down 13 minutes in total, and the lucky few who got on the site right as it recovered snapped up the last few bottles (and we got that wrong earlier too – we thought they’d all been sold before the site went down, but they hadn’t). It’s not the customer experience we wanted to provide; we are going to learn from today and improve.

For the techies out there, here is a brief explanation of what happened. Our database server loads one of its temporary databases in memory to make queries faster. When the high volume of traffic hit the site all at once looking for that elusive Buffalo Trace the temporary database became full. This is supposed to be fine, as it is designed to fall back to a secondary temporary database on disc, however due to a bug in the database software, this failed us. The problem was compounded by Redis (the caching server we use for list pages) trying to update its data from the main database that was struggling to respond. These two issues combined took us down.

The next part is pretty amazing (and not in a good way). We would have been able to recover pretty quickly, however one of our monitoring systems conspired against us to cause an error that could only be resolved by shutting it off. This wasn’t initially apparent, and took time to investigate, which is why we took so long to come back.

A lot of people are also rather disgruntled that they had added a bottle to their basket and weren’t allowed to proceed at checkout. To explain this, the site currently doesn’t reserve your bottle until you click the “Place Order” button at the end of the checkout. We do this in an effort to be fair and to stop people squatting on bottles (and preventing anyone else from buying them) simply by adding them to their basket. When there is a limited release like this however, this is far from ideal.

We understand your frustration with this system, and we know it’s not good enough. To make this a better experience for everyone we plan to introduce a new system: For limited releases which we expect to be in really high demand we’re to allocate half the available stock using a lottery so everyone gets an equal chance of getting a bottle, and the other half, well, we’re going to try something totally new…

We’re going to auction it, and if they go for over the RRP we’ll donate all the extra money to charity.

Anyway, we’re really sorry about what happened today, that’s why it happened, here’s what we’re going to do about it.

#MustDoBetter

Timeline

 

The Chaps at Master of Malt