Rival5 - Notice history

Hello, we are currently updating this incident for session timeouts, unreliable call setup, call operations, and phone registration issues over the past week.

Our engineering and development teams have identified a storage latency issue within our virtualization platform in our primary data center. This issue has been at the root of the recent challenges experienced by some, but not all, customers, including session timeouts, difficulties with call setup and operations, and unreliable phone registrations.

We are coordinating with our private cloud and connectivity provider to implement a fast-track change to be completed in the next few hours aimed at optimizing our storage platform (not service-impacting). This may include reverting to a prior state of the platform if deemed necessary. Our objective with these actions is to resolve the identified issues promptly and restore the reliability of our services.

Post-resolution, Rival5 management & engineering teams will comprehensively review this degradation of services and this overall incident with our private cloud and connectivity provider. The focus will be on understanding the absence of effective storage latency monitoring and the lack of generated alerts, as well as assigning dedicated resources more familiar with our environment and configurations. We will also assess the incident handling and overall management of our environment to prevent similar issues from occurring in the future.

We recognize the inconvenience this may have caused our users and appreciate your patience and understanding as we take the final steps towards a resolution. Our commitment to providing reliable and efficient services remains our top priority.

This update will also be sent to associated tickets. Should you have any further questions or concerns and are not already in contact with our support team, please visit our support center and submit a new ticket at https://support.rival5.com.

Thank you for your continued support and understanding.

-The Rival5 Team

Hello, Rival5 continues monitoring customers' intermittent endpoint connection issues to our private cloud infrastructure. Traffic captures and packet filters on different egress points and edges have alluded to floods of TCP session disconnects originating from customers' networks, which removes device registrations and requires re-registration, which is usually fairly quick from observations.

Our team continues working with virtualization, network, and development teams with our data center provider's footprint to troubleshoot and identify the root cause of these network subsystems by approaching the situation from different layers to restore full, stable connectivity.

We will continue posting updates here on this status page incident and update customers' submitted tickets with this associated issue.

Hello, Rival5 is continuing to observe intermittent connectivity issues with customers' endpoints maintaining registrations to our private cloud infrastructure.

Changes to routing elements and paths for both ingress and egress within our data centers will take place this evening. Tier 1 connectivity provider route shifting has already taken place to troubleshoot routes and take them out of routing.

We will update already submitted customer-facing tickets with this information and continue to monitor the situation as a whole. We thank customers that we have network access to that are affected and those who have provided our network and engineering team members traceroutes and other network diagnostics, we certainly appreciate! Thank you for your patience during this intermittent incident as we work with multiple partners & third-parties to troubleshoot, trace, and identify the root problem causing these "hiccups".