RCom Unified Communications Hosted PBX Instances

99.103% uptime

RVoice Telecom Service

99.103% uptime

Emergency / 911 Call Routing

99.103% uptime

International Calling

99.103% uptime

Origination (Inbound) Calling

99.103% uptime

Termination (Outbound) Calling

99.103% uptime

Toll-Free Origination (Inbound) Calling

99.103% uptime

RFax Electronic Fax Services

100.00% uptime

Recent notices

May 18, 2024

May 17, 2024

May 16, 2024

Performance Issue Outage
  • Resolved
    Resolved

    Hello, we are providing an update after we've continued to collaborate on this escalation bridge with our data center / private cloud provider and have continued to see service and platform stability.

    Once production routes were redirected back to edge routers and active infrastructure, a brief firewall session reset was required due to identified traffic due to observed connection issues. Rival5 continues to monitor our firewall, network, and virtualization platforms closely in our data centers ourselves and with the help of our data center / private cloud provider.

    Please notify our support team should you continue to experience connection or stability issues and we will be more than happy to dive in and investigate as needed.

    Thank you for your continued patience and support; we hope to bring this to a final resolution quickly. There is some underlying fabric switching work that will be completed that will not be service-impacting. Once a RCA (root cause analysis) and overall event/incident investigation are completed by our provider and reviewed both internally and with them, our management and engineering team will construct an "RFO" (Reason for Outage) / service incident report that can be made available to customers upon request.

    Please let us know if you have any questions and notify us if there are any concerns.

    Thank you,

    Rival5 Support Center

  • Monitoring
    Update

    Hello, Rival5 is and has been undergoing emergency storage fabric and logical storage maintenance with the assistance of our data center / private cloud provider. We are working quickly to restore services and will resume monitoring customers' instances once they are restored and verify registration recovery as well as instance health and availability.

    Numerous attempts to reach out and engage with Broadcom (f/k/a VMware) support by our provider's escalation bridge and management teams, with Broadcom's failure to reply delayed this troubleshooting process to pinpoint the exact root cause with hardware and signaling errors observed on data center fabric from primary hypervisor computing hosts. Emergency change maintenance has been scheduled on network-based storage fabric switches which is not expected to impact operations.

    Rival5 engineering and management teams have been working on an escalation bridge with pertinent teams within our data center / private cloud provider since 09:30 AM on 05/16, working through their shift changes.


    We will provide updates as they are available here and on already submitted customer-facing tickets. Thank you for your patience.

  • Monitoring
    Monitoring

    Our engineering and management teams are currently engaged in an incident conference with our private cloud and connectivity provider, troubleshooting the issue, isolating certain hosts, and rolling back yesterday's change to our previous production storage platform. Storage availability and read/write latency is identified as the incident's primary symptoms and are affecting our services' reliable operation.

    An escalation has been initiated to the virtualization manufacturer/vendor (Broadcom/VMware) to verify operation/configurations. Networking assets/infrastructure between our virtualization cluster and the storage platform are also being investigated. We are continuing to monitor customers' connections and will communicate updates on this status page, as well as customer-initiated tickets to/from our support team.

  • Investigating
    Investigating

    We are currently investigating this incident. We are seeing degraded performance after status changes last night at our data center. We are currently in the process of switching over to our recovery site to restore service to our customers.

May 15, 2024

May 14, 2024

May 13, 2024

May 12, 2024

May 11, 2024

May 10, 2024

May 09, 2024

May 08, 2024

Transit Connectivity Issues to Rival5 Services
  • Resolved
    Resolved

    Hello, we are currently updating this incident for session timeouts, unreliable call setup, call operations, and phone registration issues over the past week.

    Our engineering and development teams have identified a storage latency issue within our virtualization platform in our primary data center. This issue has been at the root of the recent challenges experienced by some, but not all, customers, including session timeouts, difficulties with call setup and operations, and unreliable phone registrations.

    We are coordinating with our private cloud and connectivity provider to implement a fast-track change to be completed in the next few hours aimed at optimizing our storage platform (not service-impacting). This may include reverting to a prior state of the platform if deemed necessary. Our objective with these actions is to resolve the identified issues promptly and restore the reliability of our services.

    Post-resolution, Rival5 management & engineering teams will comprehensively review this degradation of services and this overall incident with our private cloud and connectivity provider. The focus will be on understanding the absence of effective storage latency monitoring and the lack of generated alerts, as well as assigning dedicated resources more familiar with our environment and configurations. We will also assess the incident handling and overall management of our environment to prevent similar issues from occurring in the future.

    We recognize the inconvenience this may have caused our users and appreciate your patience and understanding as we take the final steps towards a resolution. Our commitment to providing reliable and efficient services remains our top priority.

    This update will also be sent to associated tickets. Should you have any further questions or concerns and are not already in contact with our support team, please visit our support center and submit a new ticket at https://support.rival5.com.

    Thank you for your continued support and understanding.

    -The Rival5 Team

  • Monitoring
    Update

    Hello, Rival5 continues monitoring customers' intermittent endpoint connection issues to our private cloud infrastructure. Traffic captures and packet filters on different egress points and edges have alluded to floods of TCP session disconnects originating from customers' networks, which removes device registrations and requires re-registration, which is usually fairly quick from observations.

    Our team continues working with virtualization, network, and development teams with our data center provider's footprint to troubleshoot and identify the root cause of these network subsystems by approaching the situation from different layers to restore full, stable connectivity.

    We will continue posting updates here on this status page incident and update customers' submitted tickets with this associated issue.

  • Monitoring
    Monitoring

    Hello, Rival5 is continuing to observe intermittent connectivity issues with customers' endpoints maintaining registrations to our private cloud infrastructure.

    Changes to routing elements and paths for both ingress and egress within our data centers will take place this evening. Tier 1 connectivity provider route shifting has already taken place to troubleshoot routes and take them out of routing.

    We will update already submitted customer-facing tickets with this information and continue to monitor the situation as a whole. We thank customers that we have network access to that are affected and those who have provided our network and engineering team members traceroutes and other network diagnostics, we certainly appreciate! Thank you for your patience during this intermittent incident as we work with multiple partners & third-parties to troubleshoot, trace, and identify the root problem causing these "hiccups".

  • Identified
    Identified

    Hello, since 05/06/24, Rival5 engineering and network teams have been investigating and working with some customers with reliably accessing our services and underlying infrastructure. Rival5 has been in contact with some underlying internet service providers to check routing and providing traceroutes to additional carriers involved in the paths to/from our private clouds that comprise our

    We are continuing to investigate these routing impairments which are causing degraded registration stability with devices connecting to our infrastructure. It is also being observed that some call setup, call signaling, and overall call operations are hindered due to these routing issues.

    We are escalating with our connectivity providers to attempt the removal of routes and network advertisements in place to attempt to test and move traffic around from different originating Tier 1 internet carriers.

    Some customers are entirely unaffected due to their internet service provider taking alternative routing and different Tier 1 internet carriers to reach our infrastructure.



    If you are already in contact with our support team, we are updating associated tickets as updates are available. Please continue to work with our support team on these requests. If you are experiencing noticeable connectivity issues with devices offline or rebooting, please reach out to our Rival5 Support Center using https://support.rival5.com.

    We will send updates on customer communications via tickets as well as this status page incident hosted on https://rival5status.com.

May 07, 2024

May 06, 2024

May 05, 2024

Show notice history