Customer Business Impact:

From approximately 2019-10-04 14:50:00 UTC until 2019-10-04 18:10:00 UTC, the search functionality for some MindTouch sites was inaccessible.

Problem Summary:

The release introduced an issue affecting some MindTouch sites. As a result, searches on those sites did not return any results.

Recovery:

The MindTouch sites that were affected by this issue were rolled back to the prior, stable release until a resolution was found for the new release.

Root Cause Summary:

The custom classification definition for a few of our customers contained entries with null values. These null values in the tagging system were benign up until this event, as these null values were not being read. The release that went out on October 4th, introduced a new dependency on these values being present (not null), and as a result, searches failed on sites with those unexpected data values.

Corrective actions:

We have added tests for this specific issue to our test suite. In addition, we are researching ways to better simulate unexpected data conditions that might be found in the production environment.