Search
Incident Report for MindTouch
Postmortem

Customer Business Impact:

From approximately 2019-10-04 14:50:00 UTC until 2019-10-04 18:10:00 UTC, the search functionality for some MindTouch sites was inaccessible.

Problem Summary:

The release introduced an issue affecting some MindTouch sites. As a result, searches on those sites did not return any results.

Recovery:

The MindTouch sites that were affected by this issue were rolled back to the prior, stable release until a resolution was found for the new release.

Root Cause Summary:

The custom classification definition for a few of our customers contained entries with null values. These null values in the tagging system were benign up until this event, as these null values were not being read. The release that went out on October 4th, introduced a new dependency on these values being present (not null), and as a result, searches failed on sites with those unexpected data values.

Corrective actions:

We have added tests for this specific issue to our test suite. In addition, we are researching ways to better simulate unexpected data conditions that might be found in the production environment.

Posted 12 days ago. Oct 09, 2019 - 18:13 UTC

Resolved
This incident has been resolved. An RCA will be provided shortly.
Posted 17 days ago. Oct 04, 2019 - 21:09 UTC
Monitoring
Monitoring
The MindTouch Engineering team is still determining a root cause at this time however there have been no further reports of continued search issues. The MindTouch Engineering team will continue to monitor the situation as they validate a root cause.
Posted 17 days ago. Oct 04, 2019 - 18:22 UTC
Investigating
Investigating
The MindTouch Engineering team is investigating reports of slow search and failed search requests for some MindTouch sites.
Posted 17 days ago. Oct 04, 2019 - 14:55 UTC
This incident affected: Search.