Impact Start Time (UTC) 12/31/2025 01:45 AM UTC
Impact End Time (UTC) 12/31/2025 09:01 AM UTC
Incident Summary
Updated on 01/07/2026 - On 12/30/2025, some NiCE CXone Mpower customers reported experiencing intermittent service disruptions when utilizing the CXone Mpower Expert knowledge portal. The issue stemmed from a misconfigured network proxy, which prevented a critical cluster component from starting correctly on newly provisioned or restarted nodes. The impact was resolved after correcting the network proxy configuration and restarting the affected component pods, which fully restored service functionality.
Root Cause
The issue stemmed from a misconfigured network proxy, which prevented a critical cluster component from starting correctly on newly provisioned or restarted nodes. This led to node-specific failures in service routing and Domain Name System (DNS) resolution. Because the network proxy is responsible for programming node-level routing, the affected nodes were unable to reliably resolve or route traffic. The impact was limited to specific nodes and only affected workloads running on them, which accounts for the intermittent nature of the failures. Engineers later determined that the misconfiguration was introduced after enabling a component for enhanced logging due to an oversight.
Corrective Actions
Detection:
Internal support teams received reports from customers indicating intermittent service disruptions when utilizing the CXone Mpower
Expert knowledge portal.
Remediation:
The impact was resolved after correcting the network proxy configuration and restarting the affected component pods, which fully
restored service functionality. Completed on 12/31/2025.
Prevention:
The Engineering team will implement enhanced monitoring and alerting for network proxy crash loops to proactively detect similar issues and enable immediate remediation, preventing customer impact. An update will be provided by End of Day (EOD) MT on 01/23/2026. The Engineering team is implementing an automated validation process for network proxy configuration changes to detect issues early and prevent future service disruptions. An update will be provided by EOD MT on 01/30/2026.
Incident Timeline (UTC)
12/31/2025 01:45 AM (UTC) - The first customer case was opened, and Tech
Support (TS) engineers began their initial validations troubleshooting
investigation.
12/31/2025 02:31 AM (UTC) - TS engineers notified the Network Operations
Center (NOC) engineers about the reported customer impact; a major incident
was proposed and confirmed.
12/31/2025 03:00 AM (UTC) - Engineers observed that sites were restored
but were still experiencing latency.
12/31/2025 04:40 AM (UTC) - Engineers restarted the pods to allow the
system to create new pods.
12/31/2025 04:54 AM (UTC) - The affected sites were working fine but
engineers were still seeing proxy errors on the backend.
12/31/2025 05:22 AM (UTC) - The proxy errors were still present, and
engineers found DNS failures. Investigation and troubleshooting continued.
12/31/2025 06:09 AM (UTC) - Engineers restarted the deploy and
infrastructure pipelines.
12/31/2025 08:00 AM (UTC) - Engineers reported that the automated
deployment of pipelines failed and will need to perform manual deployment.
12/31/2025 09:01 AM (UTC) - The impact was resolved after engineers
corrected the proxy configuration and restarted the pods. Following successful
test validations, the major incident was marked as resolved.