Rival5 - Transit/ISP Investigation – Incident details

All systems operational

Transit/ISP Investigation

Resolved
Partial outage
Started 15 days agoLasted about 2 hours

Affected

RCom Unified Communications Hosted PBX Instances

Partial outage from 12:51 PM to 2:26 PM

RVoice Telecom Service

Partial outage from 12:51 PM to 2:26 PM

Emergency / 911 Call Routing

Partial outage from 12:51 PM to 2:26 PM

International Calling

Partial outage from 12:51 PM to 2:26 PM

Origination (Inbound) Calling

Partial outage from 12:51 PM to 2:26 PM

Termination (Outbound) Calling

Partial outage from 12:51 PM to 2:26 PM

Updates
  • Resolved
    Resolved

    Rival5 engineers have conferred with the affected ISP and identified issues with a peering partner of the affected ISP stemming from recent maintenance performed. Routes and peering relationships between the ISP and providers to/from our infrastructure and data centers are suspected to have failed as a result of this maintenance and have been reset.

    The affected ISP is working closely with their peering partner to ensure stability and is keeping Rival5 apprised of any pertinent information.

    This only affected some customers within Rival5's network, but then also only affected some customers within this affected ISP's network as well. Rival5 is considering this incident resolved at this time but will still continue to monitor traffic to and from previously affected customers as routes are restored.

  • Monitoring
    Monitoring

    Rival5 network engineers have been working with customers' primary ISP involved in the to/from transit between connected sites with endpoints deployed and our data centers. After communicating troubles and providing examples traffic and traces, Rival5 has observed stabilized device registration behavior.

    We will continue to monitor this situation closely throughout today and will confer with the affected ISP.

  • Investigating
    Investigating

    Ok Tuesday 09/18/24 at 06:50 AM Central irregular and repetitious device registrations were beginning to cause anomaly warnings within our monitoring system.

    Rival5 support and network engineers soon began troubleshooting issues with customers that submitted a support request using our Rival5 Support Center.

    Our team has reached out to a shared ISP across customers for troubleshooting assistance and route identification into and out of their network, and we have also been in contact with network architects within our transit connectivity provider to troubleshoot paths and routes currently in use and identify any issues/make changes as necessary.

    We will update this thread shortly with more information.