NCE Client Connectivity
Incident Report for Cradlepoint
Postmortem

After the September 14th standard deployment of Cradlepoint NetCloudEngine Pre-Release we observed an infrastructure component with higher demand than previously tested and experienced.

We instantly responded and increased the scale returning to normal performance within 30 minutes. Some customers may have experienced peak performance reduction.

On Tuesday September 19 we noticed an unusual drop in client connections. After an expedited investigation we discovered a portion of our infrastructure was experiencing a performance degradation so we addressed the issue and redeployed. Services were restored within a couple of hours. Some customers may have experienced disconnections during this time period.

As after every release we are review our process and optimize our controls and monitoring parameters. This specific analysis resulted in enhancing infrastructure redundancy and resilience for client connectivity and optimization of monitoring parameters.

Posted Oct 02, 2017 - 14:27 MDT

Resolved
This incident has been resolved.
Posted Sep 19, 2017 - 13:25 MDT
Monitoring
Most clients have returned to online. We are addressing individual clients and prepping a release that will restore multi-networks and the administration interface.
Posted Sep 19, 2017 - 11:16 MDT
Identified
We are experiencing a loss of some clients. We are temporarily rolling back the ability to administer multi-networks until a root cause is identified.
Posted Sep 19, 2017 - 07:32 MDT