Incident Report Performance Issues
Service Disruption 2019-05-23
Customer Business Impact:
From approximately 2019-05-23 16:15:00 UTC until 2019-05-23 16:37:00 UTC all MindTouch sites experienced a significant service disruption.
An unidentified error when entering parameters into our release script caused servers to be terminated unexpectedly.
MindTouch weekly release process begins with the initiation of a pre-release script. This script prompts us for data that will be deployed to our production environments. While running the script, an incorrect input was accepted by the script as a valid release date/branch name.
As a result, release_20190516 application servers were not available to handle service requests and service was interrupted.
The root cause in this instance was an original error type that had not been encountered previously. We have scheduled an update to the build and deployment scripts that will scan and identify errors of this nature going forward prior to build initiation.
May 24, 16:35 UTC
MindTouch sites are coming back online. The MindTouch Engineering team will continue to monitor the status of the sites.
May 23, 16:42 UTC
The issue has been identified and a fix is being implemented.
May 23, 16:36 UTC
The MindTouch Engineering team is investigating reports of site unavailability.
May 23, 16:23 UTC