Customer Business Impact: Event metadata written from 8:40 AM PST to 10:00 AM PST were not recorded for the following services: Page History Draft History Site History Site Activity Help Requests Search Insights Import/Export Page views Indexing
This incident only affects metadata; content updates were not impacted. For example, if a user edited content on a page during the incident, the changed they made were saved properly. The record of this change, however, would not show up in the revision history of that page / content. Any content updates made before or after the incident were not affected and were recorded properly.
Problem Summary: Event processing was disrupted for approximately 80 minutes while the event pipeline was not writing metadata to the MindTouch servers for cataloging.
Recovery: The MindTouch event pipeline stream and associated APIs were reset as a debugging step in order to normalize service.
Chronology of Events (all times PDT):
09/05/18 [8:51 AM] MindTouch Engineering is alerted about a change in the event pipeline’s status
09/05/18 [8:59 AM] MindTouch Engineering is made aware that that the event pipeline was not writing metadata for events.
09/05/18 [9:31 AM] MindTouch Engineering restarts an afflicted API service as a means of triaging the issue with positive results.
09/05/18 [9:40 AM] MindTouch Engineering confirms that they have restarted all afflicted API services and event metadata is confirmed as being sent over.
Posted Sep 06, 2018 - 01:47 UTC
Investigating
MindTouch Engineering is investigating an issue with reporting data not being available in real time. We will update this page as information becomes available.
Posted Sep 05, 2018 - 18:46 UTC
This incident affected: Application (General Service), Search, In-Product Contextual Help, Email Services, and MindTouch Success Center.