On August 8th, 2019, at approximately 7:30am (PT), customer sites with *.mindtouch.us domain were showing a 504 error, and pages on those sites were not loading.
Any customer sites with a *.mindtouch.us domain were unusable until this issue was resolved, approximately around 8:45am (PT).
On August 8th, the automated release process began creating the infrastructure for the new release. New load balancers were launched to replace the previous ones. A code defect caused the tool which generates our load balancer configuration files to fail.
The defect in the current version of the code was identified and a prior, stable version was deployed to replace the faulty one.
During the weekly Product Release an issue with the system kept new load balancers from being able to route site traffic. The issue only affected routing for sites ending with “.mindtouch.us” domains. This load balancer configuration update was not a part of the automated deployment process causing the issue to surface on the day of the release, instead of being found during prior testing.
The tool used to update our load balancer configuration has been moved into the automated deployment process, to ensure it is updated with each deployment and tested by QA. Tests for the affected components will be improved.