Good issue, and the FDA section is the most useful summary of that discussion paper I've seen.
Your point about the disclaimer is the sharpest thing here. That "talk to your doctor" underneath specific advice probably doesn't buy an app the informational classification, because what counts is what the output tells the person to do. What's interesting is that three separate items in this issue are making the same argument from different directions. The FDA says you can't pre-specify every correct output, so you evaluate against a clinician panel and keep monitoring. The Cell piece says current benchmarks test retrospective statistical tasks rather than decisions. The Nature Reviews Neurology perspective says deployment is another stage of evaluation rather than the end of one.
All three are moving evaluation from output-referenced to decision-referenced. Which is exactly why the disclaimer fails: if the unit of assessment is the decision the output produces, a sentence appended after it can't reclassify anything.
Yeah, sharp observation. Essentially, AI is a sophisticated machinery, so one can’t just hide behind simple formalistic disclaimers anymore. Pretty logical. However, I am curious how it will be possible to actually manage risk with systems so powerful and unpredictable in the same time. Wild west.
The FDA paper's answer is that you stop trying to make the output predictable and bound what the unpredictability is allowed to touch instead.
That's what the two-axis framework does. Not "is this system safe" but "what does the output tell someone to do, and how bad is it if wrong." An unpredictable system that only informs sits in a different box from one that acts. Which is also why the doctor analogy is doing real work rather than being a nice metaphor. Clinicians are powerful and unpredictable in the same sense, and nobody certifies their outputs. You test competence, constrain scope by stakes, then monitor while they work. Not the wild west. Closer to the oldest pattern in medicine, applied to a new kind of practitioner.
Good issue, and the FDA section is the most useful summary of that discussion paper I've seen.
Your point about the disclaimer is the sharpest thing here. That "talk to your doctor" underneath specific advice probably doesn't buy an app the informational classification, because what counts is what the output tells the person to do. What's interesting is that three separate items in this issue are making the same argument from different directions. The FDA says you can't pre-specify every correct output, so you evaluate against a clinician panel and keep monitoring. The Cell piece says current benchmarks test retrospective statistical tasks rather than decisions. The Nature Reviews Neurology perspective says deployment is another stage of evaluation rather than the end of one.
All three are moving evaluation from output-referenced to decision-referenced. Which is exactly why the disclaimer fails: if the unit of assessment is the decision the output produces, a sentence appended after it can't reclassify anything.
Yeah, sharp observation. Essentially, AI is a sophisticated machinery, so one can’t just hide behind simple formalistic disclaimers anymore. Pretty logical. However, I am curious how it will be possible to actually manage risk with systems so powerful and unpredictable in the same time. Wild west.
The FDA paper's answer is that you stop trying to make the output predictable and bound what the unpredictability is allowed to touch instead.
That's what the two-axis framework does. Not "is this system safe" but "what does the output tell someone to do, and how bad is it if wrong." An unpredictable system that only informs sits in a different box from one that acts. Which is also why the doctor analogy is doing real work rather than being a nice metaphor. Clinicians are powerful and unpredictable in the same sense, and nobody certifies their outputs. You test competence, constrain scope by stakes, then monitor while they work. Not the wild west. Closer to the oldest pattern in medicine, applied to a new kind of practitioner.
This new approach makes sense, actually.