> There's something incredibly peaceful about being in the hands of an expert you trust. [...] AI can absolutely shatter that feeling in an uncomfortable way [...] but I don't know if I can fully trust AI either.
This really is key. We know we can't trust the AI, but at the same time we're also more comfortable asking the AI for clarifications or confronting it. Not having a time-bound appointment or paying by the hour helps a lot. But even then, more information doesn't necessarily help!
I once brought my 11-year-old car, a Civic with 150k miles, to multiple garages. I figured I'd play the "second opinion" game to correlate what the garages recommended to decide on what needed to be done...
I got 3 completely unrelated recommendations, including one that I knew was invalid! I felt worse off than when I started!
The solution to uncertain information isn't more information, which the AI can certainly provide, it's better information, and AI cannot currently provide that.
I have multiple LLM subscriptions at any given time, plus an array of local models.
When I ask a question outside of my domain of expertise I like to ask all of the LLMs I have access to. I also create separate sessions and ask the same question multiple ways.
It’s revealing to see how many different and contradictory answers I get, most of which are presented confidently.
The last time I ran a medical question through Claude I couldn’t even get consistent answers between sessions.
It’s also scary how easily you can lead each LLM to the answer you have in mind. When I would start asking questions about different options that other LLMs had presented, each session would drift toward that explanation.
In my day job we tried creating a credit assessor tool using LLM as the credit assessor.
It did great, generated a report on the assessed business that was incredibly detailed and plausible.
Then I started running tests and getting into the details, and found that if you ran the same report on the same data, it generated completely different, still very plausible, results. I could run the same source data through the assessment process 10 times and get 10 very different results. We had to can the project and go a different route.
LLMs are designed to produce plausible results, not factual results. We can fix this when using them for software dev by using linters and tests (though we've all had the experience where the LLM invents an API endpoint). I would not trust raw LLM output in any situation where that kind of testing and verification capability isn't present.
What's crazy is that there are ton of businesses building processes around LLMs that haven't done this exercise and fully believe the LLM is giving them accurate data.
> LLMs are designed to produce plausible results, not factual results.
They are true to their name: Language models. It is precisely the same problem in a language: a grammatically correct sentence is not necessarily true.
Yup I use llm to write scripts for me to process data I don't ask the llm to process the data themselves. Even when I wrote something for my day trading I used llm write scripts that do all the processing and predict price movement from that the more data is pre processed the more all the llm come up with similar trades.
It's funny that if the LLMs had all given the same result each time (it sounds like) you would have considered it more valid, even though it might just be giving a single wrong answer more consistently.
if two llms independently cite the same law given a certain factual legal question then it generally makes it more likely that the law cited does, in fact, apply to the fact pattern in question ; i think you’re using “right” in a moral sense irrelevant to the comment i was responding to.
your follow-up makes no difference to the main point under discussion : if you ask two different llms about a legal fact pattern and they independently reference the same law, then it is, in fact, more likely that the law is more relevant to the fact pattern than if only one llm mentioned it.
My point was that a human can say "sure laws say this but how about it's a screwed up thing and we change it for the better" and LLM will just do whatever is the highest in probability space
yeah, it would have been harder to see that the LLM was making shit up if it had consistently made the same shit up. But this is also true of humans, so...
you can set the "temperature" which is a lever on how stochastic the prediction is. If you are doing your own inference this is clear and easy. If you are consuming tokens this is outsourced.
What happened to VERIFYING an answer? Does nobody do that anymore?
When I ask an LLM, I trace the sources, and see if they make sense.
More often than not the sources don't actually say anything about the topic in particular...
> It’s also scary how easily you can lead each LLM to the answer you have in mind.
Exactly. Which is why "treat an LLM like a human expert who can answer your question" doesn't work. It's more like a human bullshitter who makes up convincing looking answers, and tries to please you. If the answers have actually some grounding in the training material, that's useful as some kind of holistic google, but often it's not.
> What happened to VERIFYING an answer? Does nobody do that anymore?
The problem with medical advice is that you may not be competent to verify the answer, right?
I agree that asking 5 LLMs to vote and trusting the answer is totally the wrong approach, of course. But LLMs (and traditional material) can help getting more informed. For instance, instead of going to your doctor with the LLM diagnosis and trying to convince the doctor that the LLM is right, you can try to build your own understanding of the problem and go ask the doctor to explain to you what you understood correctly and what you misunderstood.
If you have some understanding, it's harder for a specialist to bullshit you. But you need your own critical thinking and you need to put effort into actually learning something, blindly trusting and repeating what LLMs say doesn't help.
Or more specifically in this case: the patient was obviously insisting on a diagnosis and treatment based on ... a slightly hurting shoulder, with zero visible or detectable phenomena.
So the doctors gave him what he wanted: a treatment ... and Claude told him the treatment was a placebo. Correctly, I might add.
Yeah, it is absolutely not what the patient wanted to hear. BAD doctors! Except ... no, not really.
Hmm that's not what I read from the article. The author says the opposite, actually:
> [the orthopedist] suggested I get an MRI, which the clinic conveniently had available. [...] This, of course, means little to me, but their suggested course of treatment was extensive; [...] Coming out of the clinic, I had the feeling they had jumped the gun.
The author also says:
> They injected me with Traumeel, which is registered in Germany as a homeopathic medicine "without a therapeutic indication".
I personally wouldn't want to be injected homeopathic medicine "without a therapeutic indication" without even knowing it is homeopathy.
And my recent experience with multiple doctors at multiple hospitals is that just like LLMs, you shouldn't blindly trust them. Sometimes they make mistakes, and in my experience they never, ever admit it (maybe even to themselves).
So trying to get informed "on the internet" (including with LLMs) feels sane to me, but that's worth what it is worth.
What concerns me in the article is that they have GPT make a diagnosis, Claude review it, and somehow seem to assume that if both LLMs agree, they must not be completely wrong. Just like for code, it takes an expert to leverage an LLM to take an expert decision. A beginner can leverage the LLM to understand the problem better, but they never reach the level of expert just from that.
Unfortunately it's part of lowering the confidence towards doctors, and the solution to that is to try and get informed and ask another doctor. And of course they don't like it if you say "I already asked someone else but I don't trust doctors in general, so I am now asking you to test the both of you".
Quite the opposite. The doctors are disagreeing with the LLM, and when the patient doesn't accept, they're putting in a minimal lie in the deontologically accepted way.
This seems to be to be near the opposite of what a sycophant (ie. an LLM) would do.
I've also noticed the opposite problem: Sometimes the LLM, when asked a detailed question (probably with some lead-in), pushes back in a way that betrays that they fell back to general tropes without really considering the nuances of your specific context.
This happens many times, and I usually have to lead the LLM through a chain of reasoning to prove to it that its objection, through generally sound, do not apply to my specific situation.
Someone not as well versed in the subject matter would think the LLM found a smoking gun (which they love to do), and be led on a wild goose chase.
As you say, often you check up on the LLM's "reasoning" and it doesn't follow at all, or you can easily get it to contradict itself with just as much certainty as it had about its previous convictions.
It is very scary to me that people are entrusting potentially life-altering decisions to these things.
My step mom was having debilitating pain. A year of going to doctors and no one was able to find a cause. I scanned her discharge paper work which had her prescriptions on it and gave it to Claude. It identified a prescription that had that exact side effect. They later confronted her primary care that concurred and took her off it.
A friend of mine's wife recently passed. They were chasing a suspected heart defect for over a year. She had been intermittently fainting. At about the year mark they decided to scope her digestive track. They found bleeding ulcers from cancer that was all over her body. I input her fainting symptoms into Claude and gastro impact was number two suspected after heart issues.
I have a few of other cases it's helped with. I'm not sure it could do worse than my own experience with the medical system. This is doubly true in places that lack any sort of medical care.
My mom had cancer and she was on regular, suppressive chemotherapy. I put her info into an AI and it correctly noted that her chemotherapy had stopped being effective 2 months prior based on factual lab reports. She was unaware of this. I was able to be her health advocate much more effectively by respectfully asking her oncologist targeted questions. He was already on top of it and was addressing the issue. Our conversation was respectful and, due to my educating myself, went up another level. Ultimately, it was a positive interaction. I was satisfied that he was indeed expert at his craft, and he was satisfied that we were aware of the uncertainty of the new treatment with a risk-based understanding of the viability of success. This was a positive engagement with an expert. In parallel situations around non-health issues, I've found the ego of the expert seems to be the determinative factor in whether or not the interaction goes well.
> It’s also scary how easily you can lead each LLM to the answer you have in mind.
Scary in this context of course, but I find that it is an interesting thought for coding: it suggests that maybe, a developer who knows what they are doing will end up leading the LLM to coding something that make more sense than a developer who doesn't know and just vibe-codes blindly.
And all it takes is not blindingly accepting the first thing it spews if you suspect there's a better answer (and are in a position to evaluate that better answer).
As someone who uses Claude Code to summarize published research, you have to ground it in peer-reviewed results or it gets lost. But also, I am grounded with two degrees in the source material. So I am feeding it my views and asking if the published work agrees or disagrees with my opinions and I get fantastic results that way to the point of knowing current clinical trials and treatment regimens than most of the oncologists and which led to a great conversation with the clinical trials team. This doesn't replace people, but it augments existing expertise amazingly well.
But also, I hear so many tales of running out of tokens. I ask Claude Code to build a tool to perform a task. I review the tool and then I let it rip if I'm happy with it. As I understand things, most just ask Claude Code to do the task. That seems a bit fraught.
Anyway, you have to impose constraints IMO and ask the right questions to get the answers you need or yes Claude Code (or any other LLM) will eventually just agree with you.
Yeah a lot of focus lately on making context windows enormous and putting everything in them. (It should know every detail of your life!) But in my experience LLMs are extremely "prime-able" and also tend to hyperfixate on details.
So when asking difficult questions I tend to remove as much context as possible, rather than adding it. I don't want it to reflect my own ideas or biases back to me, I want an actually fresh perspective.
The problem is how do you know whether the answer is just the most persuasive or actually the most accurate one? It's hard to figure this out without domain knowledge.
Worse is that LLMs are trained to be persuasive by default. The "you're absolutely right..." stereotype is because these things are A/B tested on response quality and we know from studies people reliably rate vibes better then anything else - e.g. while the quality of hospital accomodations likely has some impact on patient outcomes, the view and decor of the room certainly did not fundamentally change the quality of the care provided but it is the largest determinant in how well people rate that care.
Do people here not realise that "second opinions" are a thing because humans disagree with each other when presented with the same case all the time? It's not just an LLM thing!
Why should a radiologist have to debunk AI slop? They have enough to do already. That's the same mentality that is frustrating open-source repositories with sloppy pull requests, and saying "here, sort this out for me".
Depending on the disease, even in cancer there's myeloma which may cause bone metastasis in many parts of the body with very focal lesions. Radiologists can't assess each and an every one of them, or even to find them all. So AI can definitely help in these scenarios.
I do something similar with reviewing code: I have one agent write the code and another reviews it, then they go back and forth for a bit improving the code. Seems to yield better results than one agent alone.
The difference is that in the code situation, you can run unit tests on the code, compile it, etc. Unless your LLMs are ordering diagnostics and reviewing the results, there is no further information that the LLMs have on the situation. Having a second LLM review the first is counterproductive, if the 2nd LLM is better, why not use it directly? If not, then what prevents it from sending the first on some incorrect tangent?
Also, there are multiple "correct" ways to code something, so imperfect code that solves the problem is still useful. A medical diagnosis is either correct or incorrect.
Different prompt approaches and training doctors to use LLMs can improve accuracy of LLM-assisted diagnosis. It’s pretty reasonable to hypothesize that LLM “peer review” could improve that as well.
With direct discussion, the same tendency to harmonize towards groupthink applies.
Aside from the statelessness GP mentioned, one can insert anti-conciliatory intermediation. "I saw a random claim go by, but something about it seems not quite right. What am I missing? They said: [...]." Weaponizing the bias, and orchestrating the discourse from the harness.
Run it with temperature 0 if you want to minimize randomness. Sampling from a probability distribution is not a problem by itself. The problem is when the probability distribution prioritizes wrong answers.
LLMs are well suited to my (some would say annoyingly) curious nature.
when i get an answer, and my first instinct is to ask a ton of follow-ups and "what about"s. i've learned to tamp this down with fellow humans, but with LLMs its great because most of the time the response is "you're right, something doesn't add up... let me try again". i think we eventually converge on to something reasonably true
The best mechanic I ever had kept my ‘98 Subaru going past 200k miles. Once during a repair I asked him to do an inspection and tell me if there was anything else I should replace. He told me not to do that, and that any mechanic would always find something, but not necessarily the next thing to break.
He said it better using an expression I hadn’t heard before or since, something like “don’t go looking for goats when your herd is already with you.”
Exactly. Old parts of the system will be working if you leave them undisturbed. Mechanics have very good intuitions of this sort of thing.
I read about before there's proper engineering / physics theory about this too, it's like a car as a machine is a linear/smooth physics system with multiple weaknesses. Overtime longtime period of running many places might weaken but it still evolves into a slightly different smooth system, until you introduce a replacement which cause a mis-match of impedance or something like that.
Maintenance-induced failures are what it’s called with small aircraft.
You’ll do something to prevent a failure (like, replace an old but functional alternator) but cause an oil leak or engine vibrations because you had to remove the propeller to complete the job.
Scammers who do the lowest effort diagnostic and "fix" to get you to pay a smaller amount of money to fix the problem in the short term even though it'll re-present itself a week/month/year later.
Upsellers who will find other things "wrong" with your car and pressure you into paying to fix them because they sound a lot worse than they are.
Good mechanics that will explain what they did to diagnose the issue and recommend different options depending on what the issue is.
Funnily enough, I've found that doctors tend to also fit into these 3 archetypes.
Yes, and that's a problem. Doctors (or experts in general) hate it when people don't trust them, but the thing with experts is that people have to trust them. And in my life (and a few times just in the last few years), enough doctors have been wrong enough that I cannot just trust them anymore [1]. If it is important, I will ask them to explain to me, and sometimes I will just ask for a second opinion.
I have read about doctors complaining that "with AI, patients now come with their own diagnosis and don't trust us when we say it's bullshit, and it is a problem". I can feel for them, but if they give the feeling that they don't listen to the patients and the patients don't trust them, it's not only the patients' fault, I would say.
[1]: I have more than one examples of my relatives like this: A doctor says "wow that's bad go to the ER", the ER says "nope it's all good, go home", first doctor learns about that and says "WTF you GO TO THE ER, call me and I will insult them on the phone", and finally resulting in a surgery where the doctors say "they were lucky we could operate right now, because in a matter of hours they could have died from this". How in the world can I trust them after one event like this? Happened to me (in some variation) 3 times. Not based on an LLM diagnosis in the first place: based on a doctor's diagnosis.
Heh, I hear stories like that everyday from my partner who is an ICU nurse. Not as dire, but there are constant inter-department arguments about moving patients because of resource constraints and the ICU could end up completely understaffed/resource constrained if the wrong NP or charge nurse is working. I'm amazed our healthcare system works at all to be honest.
Maybe a difference here is asking AI for conclusions. When I have it do a buyer's report for me, I ask it for "what questions should I be asking? What are typical things that go wrong with this type of vehicle?" I don't delegate conclusions to the AI but use it to educate myself. Then, I can gather further information to make MY decision .. to buy it or not.
I don't think so. LLMs tend to over-index on providing results in general whether it's a conclusion or not. When you ask it "What are typical things that go wrong with this type of vehicle?" you're forcing it to make a conclusion about which results to include and it will almost certainly provide results even if those issues aren't as much of a concern compared to typical issues with other vehicles.
For example, I just prompted Kimi-K2.6 with:
> I'm considering buying a used base model 2010 Honda Civic with 80k miles that's been garage kept. What are typical things that go wrong with this type of vehicle?
It listed 10 issues including the engine block cracking (which wasn't even an issue with 2010 Civics). Started a new chat and asked about a 2010 Toyota Camry, another unbelievably reliable car, and it listed 9 similar issues. Started a new chat and asked about a 2011 Jeep Grand Cherokee, a notoriously unreliable vehicle, and it listed the same number of issues.
Sure it's data to make decisions on either way, but it really all comes down to how good your prompts are and whether or not you can think critically about the output, whether or not that output is a conclusion or just data collection.
There's a big difference between a _puzzle_ and a _mystery_. In a puzzle, the goal state is known, and as more pieces - data - appears, the goal gets closer. You know how far you are from the goal.
A mystery is worse. With each additional piece of data, the goal gets farther away. Everything is more and more confusing.
Maybe I am missing something but I just find this wrong.
Everything is a puzzle: there is one "Truth" or one diagnosis. You (a smart human) should be able to converge on it by cross-examining your LLMs. By themselves, they have no interest in revealing this, no stakes, which makes them tools only useful at the hands of a capable investigator.
> You (a smart human) should be able to converge on it by cross-examining your LLMs.
What makes you think this is fundamentally different from cross-examining ELIZA? There is no guarantee that the LLM will help you converge on anything. Indeed actually calling out an LLM on BS tends to eventually produce an "I don't know and can't help you further" answer (as it should).
> There is no guarantee that the LLM will help you converge on anything.
Absolutely. The guarantee does not come from the LLM. The LLM is a simply an improved version of Google Search.
The guarantee can only come from a systemic application of epistemic discipline and reasoning, which is very much (smart) human territory.
Put it another way, I could make good decisions with/without LLMs, with some uncertain diagnostics as input. I would have to trawl through 50 papers myself, and it is possible that my decision arrives 5 years too late as a result. LLMs enable trawling and do some of the legwork in connecting the dots, but are ultimately only as capable as the orchestrating human.
The problem is that the diagnosis might not be known for a while. There's a few conditions and diseases that require an autopsy for a guaranteed diagnosis and therefore are diagnosis based on symptoms in clinical settings.
> The solution to uncertain information isn't more information, which the AI can certainly provide, it's better information, and AI cannot currently provide that.
I'd argue that AI _can_ currently provide that, but that it can't do it _reliably_, and that to non-experts it's impossible to differentiate, which makes it all the more dangerous.
Isn't that the case with human "experts"? If you had encounters with doctors, mechanics, etc. you'll know you can get a completely different diagnosis for the same problem which obviously means (in most cases) that the person you thought an expert is wrong.
What is needed are studies that will take a cold look at the actual results because AI seems to be required to be perfect or it is useless. It just needs to be as good as a human for most stuff, but in the long run it will be much better. At least that what extrapolating current reality shows us.
We have systems around humans that exist to manage expertise gaps, credibility signals, and accountability. This is part of what makes humans as good as they are, along with specialized training and some measure of meritocratic selection. We license and regulate and account and litigate to make a system that responds and improves.
Some of this might be applicable to LLMs, but some isn’t and much of it would be resisted. This is one reason we’re not likely to get “as good as a human” because at some level we’re not optimizing for the outcomes; we’re optimizing for speed, convenience, some participant’s economics, and underlying beliefs.
I've been going through PT for a hypermobility disorder related injury and I've use an AI to help me figure out "interview questions" to see if a PT knows anything about hypermobility or is willing to learn. I found it helpful to select a new PT after my first PT I trusted made things worse by prescribing stretches and no load progression from rest and recovery back to deadlifts
People put a lot of faith in human “guardrails”, standards, etc. But the same argument could be made that trusting human experts without discernment is as dangerous as trusting AI or Google or whatever other non-human source. It’s always been the case.
On the plus side, it's not like it was blind faith. Human judgement lead me to seek out another expert when I didn't like with the current PT was suggesting I do (no practicing hip hinge movements before moving to deadlift, advising against valsava for heavy lifts, ignoring feedback that a movement was causing pain, prescribing stretches to increase flexibility in a hypermobile person).
The LLM also gave me a bunch of questions to ask a new PT but I didn't have the understanding to judge the responses so I did more research. One of the things the LLM wanted me to ask about was questions about form and force closure and ideally would get a response about the oblique sling across the back. My PT didn't give me that exact response, but explained it in much more lay person terms, but because I had done my research I was able to validate their response was directionally correct. And so far, my experience with this PT has been much better. We're doing block pulls at 70% of my prior deadlift weight, next week we're going to go way back on weight and lower it some to get closer to proper deadlifting and work in some asymmetric loading exercises.
FYI - if there's any reason to believe your particular hypermobility issues might be rooted in any kind of named or unnamed connective tissue disorder, be sure to familiarize yourself with the potential vascular hazards of heavy lifts in general and Valsalva breath-holding specifically. Hypermobility can be a sign that you've drawn a genetic lot which means that you don't even have to lift all that heavy to create aneurysms.
And in keeping with our theme of the limits of both LLMs and humans, unfortunately many exercise and medical professionals may not focus on how some specific genetic lots work until a catastrophic problem presents itself.
> ignoring feedback that a movement was causing pain, prescribing stretches to increase flexibility in a hypermobile person
So glad you found better advice than this!
LLMs are fantastic tools for exploring a topic, coming up with good questions and lines of understanding to pursue with professionals, and in picking professionals. I think they're also poor outright replacements for people especially when it comes to deep and important domains, but still quite useful.
My doc didn't do the full set of test/exam for an hEDS diagnosis, so technically it's just generalized hypermobility spectrum disorder based on past medical history and a near "perfect" beighton score. It could be hEDS though, as a lay person reading the diagnostic criteria I either fit it, or are borderline.
Injury was to my SI joint, I've historically always irritated it lifting, but I set a new PR for deadlifts and it was debilitating for 3 weeks before my other half made me stop being stubborn and see a doctor about it.
I've also had my left shoulder joint surgically repaired after multiple dislocations.
YouTube has saved me at least that much in appliance repairs... and it doesn't even have an AI. It's amazing how valuable access to information can be.
Tbh LLMs pulling data out of medical documents in it's training set and searchable online is likely a much easier task than fixing some weird CSS alignment issue.
I tried that AI diagnosis for my 15 old Ford C MAx too, however with a diagnostic problem the issue is unless you've got the ground truth, there's simply no way to verify any tool / human with a metric that you can compare and decide on future tasks.
The AI might be very good at diagnosing all minor issues, but might not lead to a successful repair, whereas human mechanics are extremely good on 80% of major issues that's not the ground truth, but will lead to successful repairs (that might not address the root but simply patch it). So it comes down to manage expectation / outcomes.
> I got 3 completely unrelated recommendations, including one that I knew was invalid! I felt worse off than when I started!
I almost had a very similar experience with my beater Lexus. It took 2 independent shops and 3 dealers to finally figure out what was causing the ABS to go off randomly at low speeds. Turns out there's some obscure Toyota-specific tool from the late '90s that picked up a proprietary diagnostic code, and the third dealer was the only one that still had that particular piece of equipment.
...and of course, the thing that's broken has been out of production for 20 years and remanufactured ones cost more than the car is worth. I ended up just unplugging the ABS control module.
Point being: once I knew what was wrong, all the seemingly contradictory information from the other 4 shops suddenly fit together. It's just such a weird thing to go wrong that no reasonable tech would ever have considered it.
You nerd sniped me with the story about your used car. What happened in the end? I really want to know! There are some fun YouTube channels that basically do the same. Someone who is an expert auto mechanic takes a used car to various repair garages and asks them to recommend a course of action.
> The solution to uncertain information isn't more information, which the AI can certainly provide, it's better information, and AI cannot currently provide that.
Aside from the LLM-ism (it isn't foo, it's bar) - this is a thought terminating cliche. You definitionally don't know if some information is better or not given that you were uncertain about the information in the first case.
"I went to three mechanics and got three different answers" - your takeaway is just "Ah - I clearly need better informed mechanics."
Which is on it's face absurd because if you could clearly judge the ability of the mechanics you wouldn't need their evaluation. You'd just do the evaluation yourself.
This really is key. We know we can't trust the AI, but at the same time we're also more comfortable asking the AI for clarifications or confronting it. Not having a time-bound appointment or paying by the hour helps a lot. But even then, more information doesn't necessarily help!
I once brought my 11-year-old car, a Civic with 150k miles, to multiple garages. I figured I'd play the "second opinion" game to correlate what the garages recommended to decide on what needed to be done...
I got 3 completely unrelated recommendations, including one that I knew was invalid! I felt worse off than when I started!
The solution to uncertain information isn't more information, which the AI can certainly provide, it's better information, and AI cannot currently provide that.