Coded Diagnosis: The Invisible Algorithms Making Medical Decisions You Never Agreed To

Somewhere between your symptoms and your doctor's recommendation, a piece of software you have never heard of may have already weighed in. It has assessed your risk profile, flagged or filtered your data, and quietly nudged a clinical decision — all without your knowledge, your consent, or any meaningful opportunity for scrutiny. This is not a hypothetical scenario drawn from science fiction. It is the operational reality of an increasing number of American hospitals, and it is happening right now.

The proliferation of proprietary AI diagnostic tools in US healthcare settings has accelerated dramatically over the past several years. Venture-backed health technology companies have flooded the market with software promising to reduce diagnostic error, streamline triage, and predict patient deterioration before clinicians can detect it manually. Many of these promises carry genuine merit. The problem is not the ambition. The problem is the architecture — specifically, what patients and providers are not permitted to see inside it.

The Black Box at the Bedside

Most commercial AI diagnostic platforms deployed in hospital systems today are closed-source. Their underlying code, training data, weighting methodologies, and error rates are proprietary information, shielded from external review by trade-secret law and contractual nondisclosure agreements. When a hospital purchases one of these systems, it typically acquires access to outputs — predictions, risk scores, flagged anomalies — without receiving any meaningful visibility into how those outputs are generated.

This opacity extends to the clinicians themselves. A 2023 study published in npj Digital Medicine found that a significant proportion of physicians using AI-assisted diagnostic tools reported limited understanding of how the system reached its conclusions. They were, in effect, being asked to trust a recommendation whose reasoning they could not interrogate. In medicine, that is an extraordinary ask. Clinical decision-making has always depended on the ability to trace logic, challenge assumptions, and account for individual patient context. Algorithmic recommendations offer none of those affordances when the algorithm is sealed.

The consequences of this opacity are not theoretical. Documented cases of algorithmic bias in healthcare AI have surfaced with troubling regularity. A landmark 2019 study in Science revealed that a widely used commercial algorithm — one influencing care decisions for roughly 200 million Americans annually — systematically underestimated the health needs of Black patients relative to white patients with equivalent conditions. The bias was embedded in the training data, which used healthcare costs as a proxy for health needs, a substitution that encoded decades of systemic inequality directly into the model's logic. Because the system was proprietary, neither patients nor independent researchers had the access necessary to detect the flaw before it caused harm.

Legal Gray Areas and the Consent Vacuum

American law has not kept pace with the deployment of AI in clinical settings. The Food and Drug Administration has developed frameworks for regulating certain categories of AI-driven medical devices, but coverage remains incomplete and enforcement inconsistent. Crucially, no federal statute currently requires hospitals to disclose to patients that an AI system participated in their diagnostic or treatment process. HIPAA governs the handling of patient data but says nothing about the algorithmic systems that process it. The result is a consent vacuum: patients routinely interact with AI-mediated healthcare without any legal entitlement to know it is happening.

This gap has not gone unnoticed by legal scholars. Several prominent health law researchers have argued that existing informed-consent doctrine, which requires physicians to disclose material information relevant to treatment decisions, should logically extend to the use of AI tools that substantively influence those decisions. Courts have not yet settled this question, and until they do — or until Congress acts — hospitals face little formal pressure to be transparent about the algorithmic infrastructure shaping patient care.

Some state legislatures have begun to move. Illinois and Colorado have passed broader algorithmic accountability measures, though neither is specifically tailored to the healthcare context. California has explored legislation that would require disclosure of AI use in high-stakes decisions, and patient advocacy organizations are actively lobbying for federal standards that would mandate transparency, independent auditing, and meaningful consent processes.

The Advocates Demanding a Different Architecture

Inside this legal and institutional vacuum, grassroots organizations have stepped forward to demand what regulators have not yet required. Groups like the Algorithmic Justice League, founded by MIT researcher Joy Buolamwini, have built public campaigns connecting AI bias directly to health equity, making the technical stakes legible to non-specialist audiences. Patient advocacy coalitions working in oncology, cardiology, and maternal health have begun pushing hospital systems to disclose which AI tools they use, how those tools were validated, and what demographic data informed their training.

Some researchers are pursuing a structural alternative: open-source diagnostic AI. Academic medical centers including those affiliated with the University of California system and MIT's Clinical Machine Learning Group have published open models that any clinician, researcher, or regulator can inspect, challenge, and improve. These efforts are deliberately positioned as a counterweight to the closed commercial ecosystem. The argument is straightforward — if an algorithm is influencing life-and-death decisions, its code should be legible to the public in the same way that a drug's clinical trial data is expected to be.

The open-source model is not without its own complications. Transparency can expose vulnerabilities to adversarial manipulation, and open models still require robust governance frameworks to ensure responsible deployment. But these are solvable engineering and policy problems. The harms of opacity, by contrast, compound silently, accumulating in populations who already bear disproportionate burdens from a healthcare system that has historically underserved them.

What Patients Deserve to Know

At its core, this is a question about the basic terms of the patient-provider relationship and who has the authority to rewrite them. When a hospital deploys a proprietary algorithm to assist in your care, it is introducing a third party into your clinical encounter — one that operates without your knowledge, without your consent, and without any obligation to explain itself to you or your physician.

The democratic principle underlying open science is precisely applicable here. Research funded by public dollars, conducted in institutions built on public trust, and deployed in the service of public health should not be permitted to disappear behind proprietary walls. The same logic extends to the algorithmic tools that increasingly mediate that research's clinical application. Patients are not passive recipients of care. They are rights-bearing individuals whose informed participation in medical decisions is not a courtesy — it is a cornerstone of ethical medicine.

The technology exists to build transparent, auditable, open-source diagnostic AI. The expertise exists to validate and deploy it responsibly. What is currently lacking is the political will to demand it and the regulatory architecture to require it. Building both is not a luxury. Given what is already operating invisibly inside America's hospitals, it is an urgent necessity.