Skip to main content
22 October 2024

We were in dire straits…’ Then Clinisys turned up…

Andy Callow

About

Andy Callow is Chief Digital and Information Officer at Nottingham University Hospitals NHS Trust (NUH).

In early September, the pathology system at Nottingham University Hospitals NHS Trust (NUH) failed, leading it to declare a critical incident. The root cause was a software issue, which meant the failure lay in a server platform that was not supported by Clinisys and was not related to its software.

However, Clinisys chief technical support architect Liam Griffin arrived on site and the system was still restored within 24-hours. Trust Chief Digital and Transformation Officer Andy Callow explains.

On Monday, 9 September, Andy Callow was offsite with executive colleagues when he was made aware of issues with the system.

As the Chief Digital and Transformation Officer at NUH left his meeting with a supplier, it became clear the Trust’s pathology system was down – and his team were working to find a way to restore it.

“The team had tried to reboot the system, but as the afternoon went on it became clear that it wasn’t going to be a quick fix.,” he says. “Typically, the first challenge when something like this happens is to work out where the problem is.

“Has something happened at our end or the supplier end? Is it an infrastructure problem, or software, or integration? By Monday evening, the team was still working hard to identify the cause, but we also logged a call with Clinisys to say: ‘We have a problem, and we might need your help’.”

A critical incident  

By Tuesday morning, Andy and his team had set up calls with a number of suppliers, including senior leaders at Clinisys, who asked chief technical support architect Liam Griffin to head for Nottingham.

Meanwhile, the Trust declared a critical incident and set up a ‘gold command’ incident room to handle the impact of losing pathology at one of the biggest and busiest trusts in the country.

“What failed was not the whole pathology system, but the reporting side of things,” Andy explained. “Doctors and GPs could make requests, and the lab could process the tests, but no results were being sent back. We had runners going into the lab to take pictures and upload them to our electronic patient record.

“It was part of our business continuity planning for that to happen but, of course, there were no notifications to tell clinicians their results were ready, so flow through the hospital was still severely impacted. At the same time, GPs were not getting results at all, so we had failure demand, as people rang their GP to say they were expecting results, and the GP rang us to see what was happening.”

Resolved within 24-hours

By Tuesday lunchtime, the Trust was considering whether it would have to rebuild its entire laboratory set-up, which would have impacted services for weeks. However, once Liam arrived on site, he was able to get results coming through within a few hours.

“Liam turned up and got to work, Andy says “It soon became clear that he not only knew the Clinisys system, but the virtualisation platform that it was sitting on. By working together, we started to see results coming through by 5pm, and by 11pm things were almost back to normal.

“By Wednesday morning, we were confident the results were flowing through – and the critical incident was stood down.” No data was lost in the incident, which was resolved within 24 hours.

In a better position, and ready to roll-out the latest LIMS

So, what happened back on Monday morning? The failure was not down to Clinisys or its WinPath laboratory information management system. Instead, a number of management-servers on a virtualisation platform failed, along with a number of third-party servers supporting digital pathology and pregnancy screening.

Liam, with the assistance of the Trust IT team, was able to configure newly built virtual servers. Astonishingly, the Trust were also due to start its migration to the latest version of Clinisys WinPath just a few days after the incident, with a successful go-live in microbiology on Monday, 16 September.

“We did think about putting the go-live back,” Andy admits, “but all the hard work and preparation had been done, so we weren’t incurring any new risk – although the pathology and the IT teams were pretty tired after working incredibly hard during the outage and recovery stage.” Nottingham and its partner Trust, Sherwood Forest Hospitals NHS Foundation Trust, will complete the roll-out in two more phases over the next few months.

The value of working in partnership

Andy is full of praise for Clinisys’ response to the incident that highlighted the huge importance of pathology to almost every aspect of a health system’s operations.

He wrote to Liam personally to thank him and his colleagues, Kelly Lockwood and Keith Clement-Drew, for their support, saying: “I know you all worked long hours and into the evening, and I really appreciate the sacrifice you made of your personal time.”

Andy says this level of support from a supplier is unusual. “My experience is that our teams work in parallel and solve problems together, but in this case, we reached the limit of our expertise on the way the Pathology LIMS system was configured, and Clinisys was willing to step up quickly and provide expertise we didn’t have,” he says.

“Within hours, we were able to see progress, and now we are in a better position with our pathology systems. Not just the same position that we were in before my phone started pinging that Monday, but a better position than we were in before.”