I Let Frontier AI Navigate 4,000 Medical Images. It Did Not Become My Radiologist.

I had 4,000 scans I couldn't read

After some hospital scans, I asked for a copy of my own images. What I got back was a folder with about 4,000 files in it, spread across three types of scan: CT, PET/CT and MRI. They were unmistakably mine. They were also almost useless to me.

The folder was just a wall of codes. Names like S0001 and SER0007, with thousands of images buried underneath. A hospital image viewer will open all of it happily enough — but it's built for someone who already speaks radiology. Someone who knows that a single scan gets sliced and rebuilt several different ways, and which version to actually look at.

The file names meant nothing to me on their own. They only made sense once you already knew which machine took the scan, in what order, of which body part, and processed which way — exactly the knowledge the folder assumed I had and I didn't.

I did have one foothold. The written report from the hospital mentioned a tiny spot on my lung — the kind of thing they note down and then, in the same breath, tell you not to worry about. That gave me a simple test to aim at: somewhere in those 4,000 files were the actual pictures behind that one sentence, and I wanted to see them with my own eyes.

So the images were mine on paper. In practice they were locked behind software I couldn't drive and words I didn't know.

So I let an AI drive the software for me

I handed the problem to an AI. I used GPT-5.6 Sol in Codex and let it take control of the hospital's image viewer in my browser, starting from that one sentence in the report and working towards the images behind it.

This wasn't the usual party trick of pasting a screenshot into a chatbot and asking what it sees. Each part of the setup had one job — and, just as important, a clear point where its job stopped.

A note on privacy: I've removed or blurred personal medical details that don't affect the point. Exported scans can carry your name and details baked into both the images and the file data — the DICOM confidentiality profile shows just how much has to be stripped out to properly anonymise them.

What I was testing: could the AI drive the software — open the right scan, find its way around, and show me the evidence.

What I was not testing: whether it could diagnose anything, read the raw 3D scan directly, or stand in for a specialist tool or a doctor.

Fig. 02Interface reconstruction · synthetic imaging

W/LORIENTMEASUREFRAME 089

SPECULATIVE VOLUME · FRAME 089 / 277

123

1. 1Select the reconstruction
2. 2Verify the frame and orientation
3. 3Preserve the candidate coordinate

It found the right spot

Sol started by making sense of the mess I couldn't. It worked out which files were which — sorting the CT from the later PET/CT and MRI, then splitting the CT into the versions tuned for lung, for soft tissue and for bone.

That sorting matters more than it sounds. The same scan gets rebuilt several ways, and they're not interchangeable: the version tuned to show lungs is useless for looking at bone. Land on the right scan but the wrong version, and even a perfectly worded question gets you nowhere.

The report pointed to a tiny spot in a specific place. Sol turned that one sentence into a series of steps:

1. open the right scan;
2. pick the version built for that body part;
3. set the brightness and contrast to see it;
4. double-check which way round the image is;
5. scroll through the area instead of stopping on the first convenient slice;
6. look at the slices either side of it;
7. note down exactly which images it used;
8. take screenshots so I could check its work.

It landed on a believable match right where I expected, and kept the neighbouring images so I could check. That was the genuinely impressive bit: one plain-English request replaced a lot of fiddly software-learning, and left a trail someone else could follow step by step.

The win was real. So was the wall it hit next.

Then it tried to measure, and blew it

The spot was only a few millimetres across. Having found a likely match, Sol tried to measure it with the viewer's measuring tool — and this is where it fell over.

The image shifted while it was working, so the line it drew didn't actually sit on the spot it was measuring. That doesn't make the number slightly off; it makes it meaningless. And this isn't me being fussy: measuring these things properly has published medical guidelines, and a measurement taken on the wrong slice hasn't measured anything at all.

Fig. 03Reconstructed failure · synthetic imaging

INTENTFRAME 089

SPECULATIVE VOLUME · FRAME 089 / 277

Measure the highlighted structure

RETURNED RESULTFRAME 090

14 mmSPECULATIVE VOLUME · FRAME 090 / 277

Caliper placed after the frame changed

Honestly, that failure taught me more than all the smooth navigation did.

Sol had the measuring tool. It knew what the tool was for. It could draw a line on the screen. And none of that added up to a number I could trust.

Being able to use a tool isn't the same as being able to use it well.

Once you spot that gap, you see it everywhere. Getting to the right spot isn't the same as knowing what's wrong. Drawing a line isn't the same as taking a real measurement. And repeating what a report says isn't the same as taking responsibility for what it means.

The AI worked the software beautifully. That never turned into the expertise the software was built to serve.

"Reading" a scan is really two jobs

When I asked whether AI could "read" my scans, that one word was hiding two very different jobs.

The first is finding your way around: open the right scan, pick the right version, check which way up it is, scroll to the right place, and remember where you were.

The second is knowing what you're looking at: telling a real problem from a harmless quirk of the image, weighing it against the whole scan and the patient's history, using tested methods, judging how sure you are, and putting your name to the answer.

The first side was mostly a software problem. The second was still an expertise problem.

Driving an expert tool and understanding what it shows you are two separate skills — and the AI had exactly one of them.

I don't think that's just a medical-imaging thing. It's a pattern worth naming, because it turns up anywhere software is built for specialists: AI can make that software usable for the rest of us without ever becoming the specialist it was built for.

It got worse when I let the question go vague

The first task was easy on the AI in one specific way: I already knew the answer. The report told me what to look for and where, and I could check whether it got there. There was a clear right and wrong.

Then I opened it up. Instead of "find this known thing," I asked it to just go looking — anything worth noticing about my posture, muscles, airways, discs, shoulders.

The quality dropped straight away.

Lots of plausible-sounding observations came back. A few held up when I looked again. But one that looked like a left-right imbalance turned out to be just how I was lying in the scanner; another faded once I compared it with the images either side. Vague, reassuring observations tended to survive. Specific "here's something to fix" claims mostly didn't.

The pattern was consistent: give it a clear target and a way to be proved wrong, and it was strong. Leave "interesting" up to it, and it got noisy.

That matters, because slick software makes a weak guess sound more solid than it is — a known trap in medicine called automation bias, where people trust the confident-looking output too readily. The answer isn't to make the AI hedge more. It's to build in the habits a good doctor already has: actively look for evidence you're wrong, check the surrounding images, and treat a first impression as a hunch until it earns being called a finding.

A picture of the data isn't the data

Here's a distinction that's easy to miss. Everything the AI saw was a picture on a screen — whatever the viewer happened to be showing. It could look at the current image, work the controls and scroll through — genuinely useful, but it was looking at pictures of my scan, not the scan itself.

The real scan is much richer than any screenshot of it. It's a full 3D block of measurements, with exact spacing, real depth, and the true values behind every pixel — all the stuff a flat screenshot throws away. Research on AI models built for full 3D CT scans shows what becomes possible when a model is fed the whole 3D scan instead of a stack of flattened pictures.

Think of it like the difference between letting an AI use your code editor and pasting your entire project into a single message. Either way it can look around — but what it can actually do depends on what you've handed it.

Fig. 04Representation model

Rendered frame

SPECULATIVE VOLUME · FRAME 089 / 277

Current pixelsCurrent windowCurrent frame

≠

Native volume

Slice orderingVoxel spacingOriginal intensityAcquisition geometryInstance links

Driving the viewer got the AI in front of the evidence. It never gave it the raw 3D scan to work from.

Specialist tools aren't just weaker AIs

It's tempting to line up specialist medical AI and general AI like GPT on one scale, as if one is just further ahead. They're not on the same ladder at all. They're fed different things and do different jobs. A few real examples:

General AI is the broad thinker and organiser. Specialist tools get the real 3D scan, but only for one narrow job each.

For one narrow, measurement-type job, the better tool is often a specialist one built — and, just as importantly, properly tested — for exactly that. The general AI's role is to understand the bigger picture and decide which specialist tool to reach for. Neither one gives you a radiologist's judgment, and the tools themselves say so: TotalSegmentator states plainly that it's not a medical device and isn't for clinical use.

To be clear, I hadn't actually run any of these specialist tools on my own scans. I'm describing the next layer you'd build, not quietly claiming results I never got.

How the pieces fit together

None of this kills off normal software. It just points to a cleaner split, where each part does the thing it's actually good at.

Plain language was the right way to say what I wanted: show me the evidence behind this sentence. The image viewer was still the right place to actually look at the pictures. For one narrow measuring job, a tested specialist tool would be the right choice. And a doctor was still the one answerable for what it all meant.

Fig. 05Product reconstruction · synthetic imaging

REPORT · VERIFIED SOURCE

Findings

The study contains a small documented feature in the specified region. No further investigation was recommended.

Open supporting evidence →

STUDY 02SERIES 07FRAME 089

SPECULATIVE VOLUME · FRAME 089 / 277

Put simply: you say what you want in plain words, the medical software shows the evidence, specialist tools do the precise measuring, and a human expert stays responsible for the verdict and its consequences.

I don't think we stop building specialist software. I think we stop having to learn each one's private language before it'll give us anything.

Why this matters now

This all feels newly urgent because scans are getting cheaper and more routine. Midjourney Medical, for example, has announced an ultrasound-based whole-body scanner you'd use almost like a spa treatment. For now their own update is careful about the claims — it's pitched as a body-composition scanner, not a diagnostic device, and today a scan takes over 20 minutes, with the faster, do-more version still to come. But scale that up and you scale up exactly the problem I ran into: making the images is the easy part. Making them safe to explore and safe to understand is the part that doesn't come for free.

What I'd actually build

The thing to build here isn't a robot radiologist. It's a personal scan workspace that lets you follow your own results — without ever pretending the AI is now in charge.

Click a sentence, see the proof

Click a line in your report and land on the exact images it's talking about, with the neighbouring slices and everything set up the way you found it. You walk away with real evidence you can revisit — not a chat message that vanishes when you close the window.

Always show where it came from

Every item on screen should say plainly where it came from:

• written up by the radiologist;
• suggested by AI;
• measured automatically;
• added by me, the patient;
• checked by a clinician.

And anything that argues against a finding — or any doubt about it — should be just as easy to see as the finding itself, not buried a layer down.

Let the AI pick the right tool

The AI should choose and drive the tools, not pretend it's the sharpest eye in the room for every job.

When a question really needs the raw 3D scan, it should hand that off to a proper tested tool, bring the result back into the viewer as something you can see and check, note exactly which tool and settings it used, and keep the original images on screen the whole time.

The AI is good at planning, translating and keeping track. The specialist tools are good at precise measurement. The human expert is good at judgment and carries the responsibility. Everyone still has a job.

What the experiment actually showed

I started with one question: could an AI read my scans? By the end I could see that "read" had been two jobs all along, and the AI was good at one of them.

It could turn what I wanted into the right clicks, bring order to a pile of files I couldn't make sense of, go straight to the spot the report described, and leave a trail I could actually follow. It made hard software easy to use.

What it couldn't do was turn that into a verdict on my health I'd bet anything on.

So the next step isn't one all-knowing AI. It's a team: plain language to say what you want, medical software to show the evidence, specialist tools for the precise work, a clear label on where everything came from, and a human being who's answerable for the call.

AI knocked down the software barrier. It left the expertise barrier standing. And far from being a letdown, that gap is exactly where the opportunity is.

Selected sources

• DICOM PS3.15: Attribute Confidentiality Profiles
• Hamamci et al., generalist foundation models for 3D CT, Nature Biomedical Engineering
• Dratsch et al., automation bias in mammography, Radiology
• Bankier et al., pulmonary-nodule measurement recommendations, Radiology
• TotalSegmentator documentation
• MONAI lung-nodule CT detection bundle, version history
• TotalSpineSeg