Mount Sinai's Big Launch and the Path to Oversight

ECRI Lists 'AI' as a Top Health Tech Hazard for 2025

The ECRI Institute (formerly known as the Emergency Care Research Institute) has identified risks with AI-enabled healthcare technologies as the top hazard in its 2025 annual ranking of health technology concerns. While acknowledging AI's benefits, the report emphasizes the dangers of over-reliance on unvetted AI outputs, including biases, hallucinations, and model drift, which can lead to poor patient care. It also highlights the challenges for small and rural healthcare facilities in managing AI governance, as well as broader concerns like cybersecurity threats from third-party vendors and inadequate support for complex medical devices used in homecare settings.

Key Takeaways

• AI Risks Take Center Stage: AI-enabled technologies were identified as the top health tech hazard for 2025 due to risks like biases, hallucinations, and model drift, which can compromise patient care when outputs are not properly scrutinized.

• The Necessity of AI Governance: Effective oversight, including performance monitoring and risk mitigation, is essential to safely integrate AI into healthcare, but only well-resourced organizations like large academic medical centers often have the capacity to implement this.

• Widening Equity Gaps: Smaller and rural healthcare providers may struggle to adopt AI responsibly due to resource constraints, potentially exacerbating disparities in access to innovative and safe technologies.

• Cybersecurity Threats Persist: Ranked third on the hazard list, vulnerabilities in third-party vendors pose significant risks, as demonstrated by the Change Healthcare cyberattack, which highlights the need for thorough vendor vetting to prevent data breaches and service disruptions.

• Challenges in Homecare Technology: The second-ranked hazard focuses on the risks posed by improperly supported medical devices in homecare settings, where inadequate training for patients and caregivers can lead to misuse and safety incidents.

Why This Matters

The ECRI raises a crucial point: while the healthcare industry is enthusiastically riding the AI hype wave, it is doing so amidst significant, and often overlooked, concerns about AI errors, hallucinations, and potential biases. Despite the immense potential of these technologies to transform patient care, there’s a pressing need for a deeper conversation about how they are being implemented safely and responsibly.

As someone who has been openly bullish on these innovations, I also recognize the blind spots in the industry's approach. Discussions around transparency, standardization, and safety protocols often take a backseat to the rush of adoption, leaving critical questions about governance unanswered.

We’re beginning to see calls for action, with increasing demands for standardized evaluation frameworks and clearer guidelines on how AI tools should be utilized and monitored, especially when they directly influence patient care.

The cynic in me worries that real change might only come after a high-profile scandal or legal fallout forces a reckoning. Full Article

Mount Sinai Unveils $100M Center to Lead AI Innovation

Mount Sinai Health System has launched the Hamilton and Amabel James Center for Artificial Intelligence and Human Health, a $100 million research facility designed to advance AI innovation in healthcare. Hosting eight departments from the Icahn School of Medicine, the center will focus on leveraging AI for patient care across Mount Sinai’s network of hospitals and clinics, with applications spanning personalized medicine, genomics, and clinical operations. With this investment, Mount Sinai hopes to position itself as a leader in healthcare AI development.

Key Takeaways

• Strategic Investment in AI: The $100 million center cements Mount Sinai’s role as a leader in healthcare AI, supporting eight departments and employing 40 principal investigators and 250 staff.

• Homegrown AI Innovation: Mount Sinai is developing proprietary AI tools, such as NutriScan AI, to improve operations and patient care while avoiding external regulation.

• Competitive Edge Against Big Tech: By leveraging its vast medical data, Mount Sinai aims to outpace technology giants in developing AI algorithms tailored for healthcare.

• Focus on Standards and Collaboration: As a founding member of the Coalition for Health AI, Mount Sinai plays a pivotal role in shaping industry standards and validating AI tools.

Why This Matters

In an era where many health systems are focusing on incremental AI implementations like ambient scribes and customer support bots, Mount Sinai is taking a bold and strategic leap into the broader health AI arena. This $100 million investment signals a long-term commitment to shaping the future of AI in medicine, positioning the organization as both a leader and a credible incubator for cutting-edge innovation.

For those who believe in AI’s transformative potential in healthcare, this kind of proactive investment is critical. The systems willing to put significant resources into research and infrastructure today are most likely to drive—and benefit from—the tools and breakthroughs that will reshape the industry over the next decade. Mount Sinai, already a key player through its founding role in CHAI, is doubling down on its leadership by creating a dedicated space for developing proprietary tools, fostering collaboration, and advancing industry standards.

By staking its claim as a hub for innovation, Mount Sinai isn’t just competing with other health systems—it’s setting itself apart by focusing on applications uniquely tailored to healthcare use cases. This initiative not only reflects a deep confidence in AI’s potential but also positions the health system as a pioneer, ensuring it remains at the forefront as AI adoption in healthcare matures. Full Article

Congress Prepares for AI Policy Shift Amid D.C. Leadership Change

As leadership shifts in Washington, Congressional staffers are forecasting bipartisan efforts to advance AI policy, reflecting growing recognition of the technology's transformative potential. At a Coalition for Health AI event, staffers highlighted key legislative priorities, including improving transparency in AI algorithms, addressing bias, supporting early disease detection, and overhauling outdated healthcare reimbursement and regulatory frameworks. President-elect Donald Trump’s intention to appoint an "AI Czar" and work with figures like Elon Musk underscores a proactive approach to integrating AI into government operations. Amid these developments, ensuring equitable access to AI-driven healthcare remains a priority, with policymakers focusing on rural inclusion and smaller providers.

Key Takeaways

• Bipartisan Momentum for AI Legislation: Congressional leaders are collaborating on AI policy, with a focus on promoting transparency, addressing algorithmic bias, and supporting research for disease detection and drug development.

• Trump Administration's AI Agenda: The incoming administration plans to spearhead AI progress through leadership roles like an "AI Czar" and partnerships with tech leaders such as Elon Musk, aiming to prioritize government efficiency and innovation.

• Challenges in Healthcare Regulation: Staffers emphasized the need to overhaul outdated FDA medical device regulations and adjust CMS reimbursement models to align with advancements in AI-driven healthcare tools.

• Equity and Access Concerns: Policymakers are prioritizing the inclusion of rural populations and smaller providers in AI development to ensure equitable benefits across the healthcare landscape.

• Proactive Policy Approach: Legislation is expected to advance through regular order, including committee discussions, draft bills, and hearings, signaling a methodical yet forward-looking approach to AI governance.

Why This Matters

Skepticism about meaningful bipartisan regulation in the U.S. political system is warranted, as progress often stalls in a polarized environment. However, the increasing bipartisan focus on AI’s potential downsides—such as bias, misuse, and lack of oversight—signals a shift in priorities that may lead to actionable reforms. Recent efforts to hold major tech companies accountable have shown that both parties recognize the risks of allowing powerful technologies like AI to proliferate unchecked, even if their motivations differ.

National discussions about AI governance, like those outlined in this article, are a step in the right direction, if for no other reason than shining a spotlight on the urgent need for oversight. Still, meaningful change will require more than conversations and promises; it will require tangible legislative action. For now, we'll believe in any real bipartisan consensus when the ink is dry on the legislation. Full Article

That’s it for now. We’ll catch up again next week.

-Patrick

Enjoy this? Please take the time to forward to a friend or subscribe.