What We’re Doing to Prevent This: We’ve implemented additional checks in our maintenance process to ensure that this type of issue doesn’t happen again. We’ve also enhanced our alerting mechanisms to catch such issues sooner. In addition, we’ve rolled out a new feature that allows our CSMs to directly alert and send a page to our on-call engineers for faster response times in the event of any future incidents. Impact: The service instability affected our systems intermittently during Saturday and part of Sunday. We deeply regret the inconvenience caused and are taking steps to improve our processes to avoid similar issues going forward. Conclusion: We’re committed to providing reliable service, and we appreciate your understanding as we work to strengthen our systems and improve incident response times.
Posted Sep 17, 2024 - 21:22 CEST
Resolved
Over the weekend, our service experienced instability starting Saturday and continuing into part of Sunday. This was due to an issue with one of our load balancers, which plays a key role in managing traffic to our systems. What Happened: While performing maintenance on our load balancers, one of them was experiencing issues. This meant it wasn’t working as expected, which caused instability in our service. The issue was identified and brought to our attention by our Customer Success Managers (CSMs) on Sunday. Resolution: Once we were alerted, our team promptly investigated and found the root cause. By 1:20 PM on Sunday, we had fixed the issue and restored the service to normal.