Incident summary: One of the transactions in the DB put a lock on one of the main tables.
Description: Inside the process of publishing a message one of the transactions hanged inside our DB server. Transaction showed resistance towards being killed manually or with any other attempts
Resolution: DB’s writer instance was put into reboot to force clean running DB threads. Fix has been implemented for message dispatches to prevent this happening in the future
Posted Dec 22, 2022 - 15:55 CET
Resolved
The service is working as normal now and we will monitor the service for a post mortem.
Posted Dec 22, 2022 - 14:39 CET
Update
We need to restart hosting services in order to apply the changes. Apologies for the inconvenience and the app will be unavailable for a few minutes as a consequence
Posted Dec 22, 2022 - 14:26 CET
Update
Rollback has been initiated. Processing the queues.
Posted Dec 22, 2022 - 14:06 CET
Update
The database issue has been identified and we have escalated this to our hosting provider.
Posted Dec 22, 2022 - 13:45 CET
Identified
The issue has been identified and a fix is being implemented.
Posted Dec 22, 2022 - 12:50 CET
Investigating
Actimo - We are seeing issues with sendouts - at the moment, clients will not be able to send any messages via any delivery channel, we are on it with the Devs to see what is happening here and trying to get any queued messages sent once we have resolved the issue.