MindTouch Service Degradation: Sites unavailable
Incident Report for MindTouch
Resolved
Incident Report Performance Issues
Service Disruption 2019-05-23

Customer Business Impact:
From approximately 2019-05-23 16:15:00 UTC until 2019-05-23 16:37:00 UTC all MindTouch sites experienced a significant service disruption.

Problem Summary:
An unidentified error when entering parameters into our release script caused servers to be terminated unexpectedly.

Root Cause:
MindTouch weekly release process begins with the initiation of a pre-release script. This script prompts us for data that will be deployed to our production environments. While running the script, an incorrect input was accepted by the script as a valid release date/branch name.

As a result, release_20190516 application servers were not available to handle service requests and service was interrupted.

Corrective Actions:
The root cause in this instance was an original error type that had not been encountered previously. We have scheduled an update to the build and deployment scripts that will scan and identify errors of this nature going forward prior to build initiation.
Posted May 24, 2019 - 16:35 UTC
Monitoring
MindTouch sites are coming back online. The MindTouch Engineering team will continue to monitor the status of the sites.
Posted May 23, 2019 - 16:42 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted May 23, 2019 - 16:36 UTC
Investigating
The MindTouch Engineering team is investigating reports of site unavailability.
Posted May 23, 2019 - 16:23 UTC
This incident affected: Application (General Service), Search, In-Product Contextual Help, and MindTouch Success Center.