Virtual Assistant for Patients

Baggrund

From the moment a patient receives a diagnosis they have a significant need for healthcare related information to properly prepare for surgeries and other related steps within the course of their treatment. While hospitals do offer the information needed, it has become evident that many patients experience challenges related to understanding and applying this information, which is essential for their treatment process. Key instructions, such as when to fast or how to adjust medications, are critical for successful outcomes. However, accessing and comprehending this information at the necessary times can be difficult for patients, which affects their surgery preparations and the overall effectiveness of treatment.

Patients can reach out to the hospital staff with their questions, however, hospitals do not offer round-the-clock response services, which leaves a narrow time-frame for patients to receive beneficial guidance. Many patients find it hard to grasp the provided information due to factors like hearing impairment (affecting an estimated 7-8% of the population), stress, anxiety, dyslexia, or personal crises. In the worst-case-scenario, if patients fail to get timely responses, they might not be adequately prepared, leading to the cancellation of their procedures. It is estimated that 6-10% of scheduled surgeries are postponed, impacting patient experiences, exacerbating health conditions, and further straining hospital resources.

This was the background and context for the “Virtual Assistant for Patients” (Virtual Assistant) solution which was developed by Rigshospitalet’s Department of Otorhinolaryngology, Head and Neck Surgery & Audiology (ENT), Rigshospitalet’s Innovations Center, and 2021.AI.

To enhance patient support, we adopted an AI-driven approach using a large language model (LLM) to field questions, initially targeting thyroid cancer patients scheduled for surgery in the ENT department. Annually, about 400 cases of thyroid cancer are suspected, with approximately 150 of these leading to surgery. Our strategy focuses on using safe, non-private data to protect patient confidentiality. WhileLLMs are powerful, we are aware of the need for caution due to their potential for errors. These models, generally trained on broad data, may not always provide specific answers to detailed patient questions about their surgery. Our Virtual Assistant’s goal is to bridge this gap, offering clear, helpful responses to treatment-related queries, enhancing patient understanding and preparation beyond what is possible in traditional pre-surgery meetings.

Løsning

The Virtual Assistant solution consists of several components. It is hosted on 2021.AI’s GRACE platform, which is an AI governance platform. GRACE provides a safe and secure environment and services such as identity and access management, resource management, and service orchestration.

The core LLM service is encapsulated in an application which adds governance, risk management, and compliance services. The application and platform are designed to be loosely coupled and scalable. This ensures that LLM’s are not constrained to only ChatGPT, and that the platform can be deployed on-premises or on any major cloud. Being scalable ensures that the intended use case, the number of users, or the underlying resources can scale up easily and efficiently. The LLM used retrieval augmented generation (RAG), which transforms input documentation into a more optimized representation for retrieval.

The solution is developed to meet four main LLM requirements for the “Virtual Assistant for Patients”:
- It must only base its answers on approved input data, which consists of documents provided by the hospital for this specific purpose.
- It must answer questions correctly based on the input data.
- It must not answer questions related to certain topics, for example for diagnoses or questions about death.
- And it must indicate when answers cannot be synthesized from approved input data, for example by adding: “I couldn’t find any relevant information” to its response.

The project adopted a structured approach, focusing on rigorous testing in partnership with healthcare experts from Rigshospitalet. The testing began with usability trials to gauge the system’s accessibility and user interface, which was expanded to include applicability tests for assessing the clinical accuracy of the responses. Collaboration with hospital staff was essential, involving detailed preparation of test plans and compiling questions from discussions with staff and patient representatives. Evaluation metrics and benchmarks were established upfront ensuring consistent and objective assessment throughout the testing phases. Continuous integration of feedback aimed to refine the process, ensuring the system’s responses were not only accurate but appropriately filtered and redirected to clinical staff when necessary. This approach guaranteed that the project’s outputs were clinically valid and user-friendly, meeting the high standards of healthcare delivery.

The LLM performance optimization process was tightly coupled with the test process, and issues identified during a test-run were addressed immediately or shortly after testing. Optimization entailed the use of prompt engineering, which is the process of tuning a prompt to achieve a desirable solution.