Founding Lead Researcher - Medical AI Evaluations

About Sophont

Sophont builds open, universal medical AI that understands pathology, neuroimaging, clinical text and more, empowering clinicians and researchers worldwide.

We are a public-benefit corporation driven by a mission to build AI models and applications to advance healthcare and benefit humanity. Together with MedARC, our open-source Discord community, we are dedicated to open science and developing frontier AI models for medicine transparently. We already have a strong track record, having published open-source well-cited research in NeurIPS, ICML, CVPR, Nature Biomedical Engineering, and more. As part of our radically transparent approach, you can already join our public meetings to see how we conduct research at Sophont and MedARC.

Sophont is a rare early-stage company with both world-class technical leadership and a mission that matters. Joining Sophont means you are paid to lead and contribute to high-impact medical AI research that will transform healthcare and life sciences.

Our team is composed of leaders in their respective domains, backed by investors from Google (Jeff Dean, Logan Kilpatrick), W&B (Lukas Biewald), Hugging Face (Clem Delangue), Kindred Ventures, Upfront Ventures, and more.

How We Build

Sophont functions differently than other startups. There is no clean separation between thinking and doing. The same people deciding what should exist are the ones building it.

A large part of that work happens in the open. Through MedARC, an open science Discord community, Sophont staff lead groups of highly motivated contributors running experiments, testing ideas, and pushing new directions forward continuously. It is closer to a live research lab than a traditional team.

About the Role

We are seeking an exceptional lead researcher to own the evaluation layer for Sophont's medical AI work across language models, pathology foundation models, neuroimaging models, agentic systems, retrieval, and clinical-facing products. This role sits at the intersection of benchmark design, statistics, infrastructure, data quality, and scientific judgment.

Evaluation is one of the highest-leverage parts of medical AI. The standards this role builds should influence what researchers optimize, what partners trust, and which systems are mature enough for clinical or life-sciences deployment. We want Sophont's evaluation work to help set the standard for how different types of medical foundation models are tested.

This work builds on public Sophont projects such as Medmarks, which is already becoming a reference point for medical LLM evaluation; Brainmarks for fMRI foundation models; Nanopath for pathology model evaluation; and OpenMidnight for open pathology foundation modeling. You will build the systems and standards that let Sophont and the broader field make reliable decisions quickly.

What You'll Do

• Define reusable evaluation methodology across medical language models, pathology, neuroimaging, multimodal models, retrieval and search tools, agents, and clinical AI workflows
• Build benchmark infrastructure that supports public and internal leaderboards, reproducible result packaging, repeatable evaluation jobs, and clear model-to-model comparisons
• Design high-quality evaluation datasets with researchers, clinicians, and MedARC contributors, including annotation rubrics, quality control, leakage checks, and inter-rater reliability
• Set standards for source-grounded systems, including claim decomposition, retrieval quality, evidence recall, evidence precision, uncertainty calibration, guideline grounding, and medical fact verification
• Develop clinically meaningful evaluation protocols across language, vision, pathology, neuroimaging, multimodal reasoning, and product-facing workflows
• Run statistically defensible model comparisons with confidence intervals, subgroup analysis, calibration, robustness checks, ablations, and negative-result reporting
• Translate evaluation results into papers, blog posts, technical reports, partner diligence materials, and internal release decisions
• Lead MedARC contributors through small, reviewable pull requests, clear specs, and reproducible experimental workflows

What We're Looking For

• Experience building rigorous ML evaluations, benchmarks, model validation systems, or scientific measurement infrastructure
• Strong command of statistics, experimental design, data quality analysis, and scientific interpretation
• Strong Python and software engineering skills, including experience with data pipelines, reproducible experiments, and model evaluation infrastructure
• Ability to design annotation protocols and reason carefully about ground truth in noisy biomedical settings
• Experience with LLMs, agents, computer vision models, biomedical foundation models, or other systems that require careful evaluation
• Comfort balancing cost, latency, model quality, and statistical power when deciding what to evaluate and how often to evaluate it
• Clear writing ability for scientific, engineering, and partner-facing audiences
• Comfort operating in an early-stage environment where you will define the process while building it

Preferred

• Experience with medical, biological, clinical, pathology, or neuroimaging datasets
• Nice to have: experience with retrieval-augmented generation, hybrid search, BM25, dense retrieval, ColBERT-style retrieval, LLM-as-judge systems, or human evaluation pipelines, since many medical evaluations will involve building or integrating search tools
• Experience building public benchmarks, leaderboards, crowdsourcing tools, or annotation systems
• Experience with GPU clusters, experiment tracking, distributed evaluation, or model-serving infrastructure
• Strong publication record or open-source record in ML, biomedical AI, or evaluation methodology
• Experience leading open-source research communities or distributed contributor groups

Location

This role is fully remote with the ability to mostly dictate your own working hours, provided you can attend core collaboration meetings.

Compensation

The salary range for this role is $100,000-$300,000 USD per year. Benefits include meaningful equity, OpenAI and Anthropic subscriptions, a 401(k) with 4% match, medical, dental, vision, and basic life insurance.

Employees are encouraged to publish and open-source their work and to participate in workshops, hackathons, and conferences.

We are an equal opportunity employer and welcome applicants from all backgrounds.