Mistakes, Apologies, and What We Did For Thanksgiving
An open letter to our customers from CEO Adam Schoenfeld.
Nothing makes me feel worse than seeing you (our customers) unable to get the reports you need, when you need them. As you know from our emails and alerts over the past week, we’ve been dealing with data availability issues that have put unacceptable limitations on your reporting. I want to share a bit more about how we got in this situation, how we screwed up, and what’s been happening behind the scenes to make sure that we use this experience to become better and stronger.
I wish that we could blame our hosting provider, The Election, or even Tony Romo. The truth is that we can only blame ourselves for this one. On the evening of 11/18, we wiped out a large chunk of Twitter data. We’re constantly adding new data sources and reports, and we pride ourselves on fast iteration to keep up with your needs. We’ve been moving at a million miles per hours since the company was founded and this fast pace can lead to the occasional mistake. This isn’t the first mistake we’ve made, but it was the first of this magnitude. We’ve skipped over some policies and safeguards to prevent this kind of issue. That was mistake #1.
Our second mistake was getting overly optimistic about our timelines. We sent out an email to all our customers as soon as we discovered this issue. However, our initial timeline estimates were far from accurate and led to broken promises that we had to scramble to address. Transparency is a priority and a core value for our company, so the feeling that we mislead you was, to be blunt, really shitty. We have learned a ton about dealing with emergencies, scaling in real-time, and measuring timelines.
Where We Stand:
As I write this post, things are nearly back to normal, but we still have a lot of work ahead. We have progressively restored normal reporting in reverse chronological order. We’re now back to normal with all data through October. In the next 2-3 days we’ll have September reporting back to normal. We have a great plan in place to finish the process and we expect to have all date ranges back for you by the week of 12/10. Early in the recovery, we also put together a secondary manual process to deliver your requested reports on unavailable date ranges. We can still promise these within 48 hours for any specific requests while we work toward full recovery in the coming days. That’s not ideal, but it’s allowed us to keep things moving for you. If you have these kind of needs, just ping our support team and we’ll be standing by to help.
What We Did For Thanksgiving:
In 25 years, if you ask our CTO Damon what he did for Thanksgiving in 2012, I’m pretty sure he’ll be able to give you an immediate answer. I’ve probably written or spoken the words “I’m sorry” a hundred times over the last week. Apologies are clearly no substitute for solutions, but I want you to know how deeply we share in your pain. Nothing makes us feel worse than seeing our mistakes cause you frustration. We’ve literally been working around the clock. Most of our team spent their entire Thanksgiving holiday glued to our laptops, working on data recovery or working with you to minimize the impact on your business. Engineers slept in shifts and cancelled trips. Our support team answered emails over turkey. Our analysts sacrificed family time to manually build the reports you needed. If it moved the needle for you at all, we would happily give up 100 more Thanksgiving holidays. I’m very proud of how our team was able to work side-by-side to tackle this big challenge. It doesn’t make up for the mistakes that got us here, but I hope it shows our dedication to learning from those mistakes and making the product even stronger going forward.
This Thanksgiving was a big learning experience for our team. We learned how to come together in a crisis and we identified many new opportunities to make our platform stronger, faster, and more resistant to mistakes. I know that we let you down at times in the past week, but I also hope that you saw us fighting hard to get things on track. If you have any feedback to share about how we handled this situation, we would love to hear from you! In the meantime, keep in touch with issues and we’ll keep updating you as this problem is fully resolved.