I was a handsome 12-year-old boy.

I was in a drama competition for national schools in a play written by my best friend.

In that play I was a detective trying to solve a mystery of who robbed a fictional hotel.

The hotel was called Hotel El Cheapo Nance Depot.

I was also a boy with a stammer or speech impediment and I was desperately trying to both remember and say my lines.

During that play I spent 70% of my time collecting information to solve this mystery of who robbed the hotel.

So fast forward 26 years and not much has changed.

I'm now working as a medical consultant in the hospital for half of my time and as a senior lecturer in applied clinical data analytics in the university for the other half.

The context has changed from a detective solving a mystery of who robbed a hotel to a doctor solving a mystery what is the cause of illness in this patient in front of me.

I still spend 70% of my time collecting information about the patient and only 30% of my time making decisions on that information and communicating with the patient.

So the information or data that I collect takes many forms.

Patients blood pressure, their medical history, blood test results and this imbalance in how healthcare is delivered this 70-30 is well known all over the world and in many specialties it's been made worse by technology with the introduction of the electronic health record for example that was designed for collecting information about billing and not designed to make healthcare more efficient.

So this extra administrative workload that doctors have to do reduces the face time that they have with patients.

The face time that we were fundamentally trained to do and the face time that the patients want.

So the idea worth spreading that I'm going to share tonight is a potential solution to this.

We've all heard a lot about the risks of artificial intelligence but I want to introduce a new perspective.

One where the responsible use of medical AI could help to solve some of these problems and I'm going to introduce a new type of AI called multimodal AI.

Multimodal AI is AI that takes data in many different forms, text, images, numbers.

When I work as a doctor in the hospital I'm talking to the patient, I'm listening to the patient, I'm listening to their chest with a stethoscope, I'm palpating their abdomen, I'm looking at their blood test results.

This is multimodal human intelligence.

Multimodal meaning lots of different types of data.

So in November last year the media coverage of AI really exploded with the release of chat GPT by open AI.

So chat GPT is a type of AI called a large language model or generative AI.

But it's not the only type of AI.

There's other types of AI that are less familiar to the general public like machine learning, computer vision, natural language processing.

And those AI's mostly take in a single type of data and we call this single modal AI.

So I'm going to give three cutting-edge examples of single modal AI in healthcare.

The image behind me is an image of a chest x-ray with the heart in the center, surrounded by the lungs, the ribs going across the shoulders at the top.

AI has become really good at distinguishing normal from abnormal.

We call this triaging.

And most x-rays done in the world are actually normal.

So this software called ChestLink by a company called OxyPit, it's a medical AI triage system.

And it's the first system that's received regulatory or CE approval to be used in a fully autonomous way to report on chest x-rays.

So what OxyPit does is it looks for 75 abnormalities on the chest x-ray.

And if it doesn't find any of those abnormalities, it reports the x-ray as normal without any human involvement.

If it does find an abnormality, it passes the x-ray back to the human radiologist to report.

This is an example of task sharing between the AI and the human radiologist.

The picture behind me is a retina.

This is the tissue at the back of your eye.

If you've ever been for an eye test, this is what the optician sees.

The optician is looking for reversible causes of blindness like macular degeneration.

A group of researchers in University College London developed an AI model trained on 1.6 million pictures of the retina.

This model is able to diagnose eye disease and predict outcomes from eye conditions like macular degeneration.

And that's very impressive that it can do what most non-specialist doctors find hard, but it doesn't stop there.

If we think about a condition like Parkinson's disease, we don't think about the back of the eye.

Parkinson's disease affects your movement, causes a tremor, affects your walking.

The same AI model can look at the back of your eye and predict Parkinson's disease years before patients develop symptoms.

So now not only can it see what the human can see, but it can see things that the human can't see.

However, this model will never diagnose Parkinson's disease, and it definitely will never give compassionate care for Parkinson's disease.

AI like this must be used in conjunction with highly trained healthcare professionals.

I'm going to move away from computer vision towards large language models.

In December of last year, Google released a medical large language model called MedPAM.

They trained their generic large language model called PAM to perform medical question answering.

And this is the first time ever that a computer or an AI model has passed a US medical licensing exam with a passing score of 67%.

And then only three months later, MedPAM2, the next version, got a score of 86%.

This is expert level on that exam.

If you have a smartphone in your pocket, multimodal AI is available to you right now.

Four weeks ago, OpenAI released a multimodal version of ChatGPT.

And this is an example I gave us last week, where I pass in a picture of an ECG.

This is the electrical activity of the heart, a very little scenario. 60-year-old male presented with palpitations.

That's a sensation of your heart beating in your chest.

He could feel his heart skipping beats.

No past medical history, not currently on medications.

The attached picture is ECG.

What is the next step for this patient?

Now, very unhelpful for this presentation, ChatGPT told me that it's a machine learning model and not a device.

So I asked it to help a friend out and told it I'm doing a TEDx talk about multimodal AI and please play along.

And it did exactly that.

Now, although the ECG analysis wasn't perfect, it was very, very close.

And the follow-up advice that it gave was perfect.

But this gets even better when multimodal large language models are trained on medical tasks.

And the best example of this is MedPamM, M for multimodal, released by Google in July of this year.

MedPamM takes multiple different types of input, pictures of the skin, pictures of chest x-rays, pictures of pathology, texts from radiology images, and performs multiple medical tasks.

So it's not perfect, but the radiology report that generated from MedPamM was compared to a human radiologist report.

And the blinded assessors preferred the MedPamM report in 40% of cases.

So the things we need to implement multimodal AI safely are trust, explainability, and randomized clinical trials.

In relation to trust, there was a survey done in the United States, and over half of the respondents would feel anxious if they knew their healthcare worker was relying on AI for their assistance, feared that doctors were going to integrate AI too quickly before understanding the risks to patients.

So we have a lot of work to do to bridge this gap.

The second thing is explainability.

Explainable AI opens up the black box to tell us why it met the output.

So in our research, what we're interested in is why did the AI model pick a particular blood pressure medication for high blood pressure?

Should we just go along with what the model says, or do we want to know why it got to that conclusion?

If the output from the model agrees with our worldview, then we might just go along with it and not question it.

And that's a very risky area in medicine called confirmation bias.

The third thing we need is randomized clinical trials.

AI models must be the same way that we test medicines.

For medicines, we use randomized control trials.

This is the peak of evidence in medicine.

One group receiving the AI model, another group not receiving the AI model, and follow them up to see who does better.

So what's the missing piece?

Where does the art of first look at the patient?

You never interpret a result, an X-ray, an ECG, without knowing the context of the patient.

We often call this the eyeball test.

And this has been tested where patients coming to emergency departments, nurses would triage them as red, yellow, or green from just looking at the patient.

And this was shown to be more accurate than sophisticated models.

So in the future, I see a world where a picture or a video of the patient is also fed into the multimodal model.

So now, looking back at myself, my 12-year-old self, standing nervously on that stage, I can see the parallels between then and now.

Back then, I was looking for data to solve the mystery of who robbed the fictional hotel.

But now we're looking for data for a more profound reason, and that's to make healthcare more efficient, personalized, and accessible.

Imagine a world where remote corners of low- and middle-income countries that have no access to specialized care can gain insights from these models.

That's a world the medical AI, and especially multimodal medical AI, can help us create.

So as we look to the future, we have to prioritize compassion and understanding.

We have to build this relationship between AI and the humans to allow the doctors more time to spend with the patients, to understand them, and give them a better chance at health and happiness.

Thank you.