Actimo - We are seeing issues with sendouts

Incident Report for Kahoot

Postmortem

Incident summary: One of the transactions in the DB put a lock on one of the main tables.

Description: Inside the process of publishing a message one of the transactions hanged inside our DB server. Transaction showed resistance towards being killed manually or with any other attempts

Resolution: DB’s writer instance was put into reboot to force clean running DB threads. Fix has been implemented for message dispatches to prevent this happening in the future

Posted Dec 22, 2022 - 15:55 CET

Resolved

The service is working as normal now and we will monitor the service for a post mortem.

Posted Dec 22, 2022 - 14:39 CET

Update

We need to restart hosting services in order to apply the changes. Apologies for the inconvenience and the app will be unavailable for a few minutes as a consequence

Posted Dec 22, 2022 - 14:26 CET

Update

Rollback has been initiated. Processing the queues.

Posted Dec 22, 2022 - 14:06 CET

Update

The database issue has been identified and we have escalated this to our hosting provider.

Posted Dec 22, 2022 - 13:45 CET

Identified

The issue has been identified and a fix is being implemented.

Posted Dec 22, 2022 - 12:50 CET

Investigating

Actimo - We are seeing issues with sendouts - at the moment, clients will not be able to send any messages via any delivery channel, we are on it with the Devs to see what is happening here and trying to get any queued messages sent once we have resolved the issue.

Posted Dec 22, 2022 - 12:41 CET

This incident affected: Actimo (Message sending).