Cradlepoint’s engineering team has completed a Root Cause Analysis (RCA) of the service disruption that occurred on November 19 2021. Please follow the link below to view the RCA in our Customer Connect Community. Any questions or concerns please contact Cradlepoint Support.
NetCloud Perimeter remains operational and stable so we are resolving this incident.
Posted Nov 23, 2021 - 15:22 MST
Update
NetCloud Perimeter remains operational and stable. Cradlepoint engineering will continue monitoring through the day.
If any devices have not reconnected, the user may power cycle/reboot the Cradlepoint device. For MS Windows or other clients that may be experiencing issues reconnecting, we recommend re-authenticating and applying the most recent NCP client update if possible.
Posted Nov 23, 2021 - 10:49 MST
Update
Cradlepoint engineering has completed clearing IP table rules and will continue monitoring.
If any devices have not reconnected, the user may power cycle/reboot the Cradlepoint device. For MS Windows or other clients that may be experiencing issues reconnecting, we recommend re-authenticating and applying the most recent NCP client update if possible.
Posted Nov 22, 2021 - 21:26 MST
Monitoring
Cradlepoint engineering is continuing to clear IP table rules on NCP nodes with positive improvement in system load capacity. Over the next several hours Engineering will slowly apply more updates to nodes and monitor until the system is fully stable.
Potential mitigation tactics, if the user power cycles/reboots the Cradlepoint device it could resolve issues involved with NCP failing to reconnect. For MS Windows or other clients that may be experiencing connection issues, we recommend re-authenticating and applying the most recent NCP client update if possible.
Posted Nov 22, 2021 - 19:00 MST
Update
Cradlepoint engineering has been actively clearing IP table rules on NCP nodes in order to improve system load capacity. The team has been successful in reducing the number of database deadlocks as the root cause of the service degradation affecting client status updates. Engineering will continue to slowly apply updates and monitor until all cases of deadlock conditions cease and the system is fully stable.
Potential mitigation tactics, if the user power cycles/reboots the Cradlepoint device it could resolve issues involved with NCP failing to reconnect.
Posted Nov 22, 2021 - 16:34 MST
Identified
We have identified an issue that the team is addressing. The team is making some updates on the backend and you may see a disruption in your network connectivity over the next couple of hours. The engineering team is aware of this issue and working as quickly as possible to address. We apologize for any inconvenience this has caused you.
Posted Nov 22, 2021 - 13:41 MST
Update
NetCloud Perimeter service is operational but we are continuing to monitor.
Posted Nov 22, 2021 - 12:52 MST
Update
NetCloud Perimeter service is fully functional but we are continuing to monitor the system.
Posted Nov 22, 2021 - 10:55 MST
Monitoring
We are aware of the performance degradation causing a potential delay with clients reconnecting. We are monitoring and during this period all active clients will continue to remain operational.
Posted Nov 21, 2021 - 17:38 MST
Identified
NetCloud Perimeter is experiencing a performance degradation causing clients to reconnect slowly. During this period, all active clients will continue to remain operational.