MindTouch degraded performance: Slow load time
Incident Report for MindTouch
Postmortem

Root Cause Analysis:  As part of our release process, when a site is migrated to a new deployment, the site’s custom CSS needs to be processed (compiled and minified).  The service/application responsible for this is called mindtouch-node-services. Changes to our "built-in" CSS caused the mindtouch-node-services application to take longer than 30 seconds to minify the CSS.

Recovery:  At approximately 20230217 4:45 UTC, the `clean-css` upgrade was deployed and everything was back to normal. 

Corrective Actions 

  1. CXone Expert has upgraded to `css-clean` 
  2. CXone Expert is currently working on code changes that can give us a better understanding of the bottlenecks that affect this service. 
  3. CXone Expert is upgrading our version of nodejs to provide better performance. 
  4. CXone Expert is testing a nodejs process manager that can spawn multiple nodejs processes to prevent slowness when the nodejs event loop is blocked and cannot take on any more work.  This will require some performance tests to be written and executed.
Posted Mar 09, 2023 - 00:56 UTC

Resolved
This incident has been resolved.
Posted Feb 16, 2023 - 22:37 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 16, 2023 - 21:32 UTC
Update
We are continuing to investigate this issue.
Posted Feb 16, 2023 - 20:51 UTC
Update
We are continuing to investigate this issue.
Posted Feb 16, 2023 - 20:41 UTC
Investigating
MindTouch Engineering is looking into issues related to slow load on MindTouch sites.
Posted Feb 16, 2023 - 20:40 UTC
This incident affected: Application (General Service), Search, In-Product Contextual Help, Email Services, MindTouch Success Center, Analytics, and Geoblocking for Russia.