A Ribbon Story – Going The Extra Mile
We at Ribbon had a customer issue in December (2022) that I thought would make an interesting blog post. Sometimes, problems arise. Things break. It’s how you respond that can make all the difference, taking a potentially negative experience and creating something positive.
A Critical Failure
I’m not going to name the customer, but it’s a fairly large hospital with multiple locations based in the upper Midwest. A failure occurred in one of the Ribbon Session Border Controllers (SBC) that is central to this customer’s voice network. A complete failure would render their voice network completely disabled – no calls in or out. With a critical function like healthcare, this could be very damaging.
Fortunately, the customer had a redundant, high availability pair of SBCs designed to fail over when if one of the units were to be disabled. The SBCs behaved as they should and all the voice traffic moved to the redundant SBC, but with one unit out, there would be a single point of failure in their voice network. They needed a quick response from Ribbon to get the failed SBC operational.
Ribbon Steps In
Repairing the unit required the immediate shipment of a new hard disk drive (HDD). Given the customer’s location, we determined that flying it into Chicago then have a courier drive it 4+ hours was the best way to transport it. Given the customer wanted to make this repair immediately, it needed to happen same day and get it there asap.
From here, I’ll share excerpts from the customer’s email:
“I wanted to provide everyone with an overview of events and a quick update on our current status:
12/16 @ 2:10AM CST:
I picked up the drives from the courier, drove the package to the hospital, and handed it to one of my voice engineers.
12/16 @ 10AM CST:
we were redundant when the completely failed unit was able to rejoin the cluster following the HDD replacement and subsequent OS build and software install.
12/16 @ 3PM CST:
the synchronization between the failing node and the rebuilt node completed. With the assistance of a Ribbon engineer, we gracefully failed over call processing to the newly rebuilt node.
12/17 @ 8AM CST:
we took the failing node offline. We replaced the failing HDD and began the OS build and software install.
12/17 @ 9AM CST:
the node rejoined the cluster and we were redundant again.
12/17 @ 9:20AM CST:
the synchronization finished and we now fully operational and running under normal operating conditions.”
The customer goes on:
“The phone environment is used in so many lifesaving processes and services to improve the experience for the patient and their families. Here is a representative list of services that would have been unavailable if we experienced a complete failure of our SBC cores:
- Stroke Protocol – Patients or family members call to connect with a stroke neurologist for consultation and transport
- Admissions and Transfer Center – Transfer of patients from other hospitals to us when their condition requires the specialized care that can only be provided here. We have over 1400 transfers each month.
- Emergency Code Calls – Cardiac arrest and/or resuscitation; violence involving a patient; fire; violence involving family and other non-patients; rapid response team; armed violence; infant or pediatric abduction
- Ambulance – Paramedics will often talk with our ED before arrival while enroute
- 911 – beyond the obvious, remote clinics will use 911 when a patient’s condition warrants a trip to the ED. They use 911 to transfer the patient.
I can honestly say, without hyperbole or exaggeration, that every single person who worked to get us the parts we needed and helped guide us with the replacement process saved lives. Some heroic acts go unnoticed and many are generally unaware that their actions make a large difference in this world, but I witnessed firsthand your acts of persistence, tenacity, determination. I am fully aware of the impact you have made and to each and every person who was involved, no matter their role, thank you.”
It's extremely gratifying when the company you work for meets the needs of their customer, or better yet, goes above and beyond.
