Service slow or unavailable
Incident Report for 360Learning
Postmortem

Incident report

Service slow or unavailable on Sept 19, 2023, for 1 h 29 minutes

Between the hour of 16:02 to 17:31 CET on September 19st, 2023, our users encountered a slow or unresponsive web application and mobile application, as well as issues on 360Learning Public API.

The event was triggered by a routine upgrade operation on the main database, that unfortunately placed the database into an unstable state.

Our teams detected the incident immediately due to our monitoring system. We decided to remove the faulty database nodes from operation, and to restart them. Around 17:02 CET the service was back for most users, but slownesses remained until 17:31 CET.

Following this incident, we deployed further fixes on the next days, causing short episodes of a slower than usual service.

At the date of writing, the incident is now completely resolved.

We have taken preventive measures to minimize the impact of this type of incident in the future.

We apologize for the inconvenience caused by the incident.

Posted Sep 21, 2023 - 19:14 CEST

Resolved
The issue has been resolved. We apologize for any inconvenience caused by this incident
Posted Sep 19, 2023 - 17:58 CEST
Monitoring
A fix has been deployed to production. We will keep monitoring and take further action if needed.
Posted Sep 19, 2023 - 17:32 CEST
Update
Service unavailable.

An issue has been detected on our servers. The service is currently down. We're investigating the root cause and working on a solution.

We will send an additional update around 5:30 p.m. CET.
Posted Sep 19, 2023 - 17:07 CEST
Investigating
We're experiencing an elevated level of API errors and are currently looking into the issue.
Posted Sep 19, 2023 - 16:28 CEST
This incident affected: Web Application and API.