Picture a routine moment a few months from now. Your team is updating the scientific platform for a product two years past launch. The new positioning rests on a systematic literature review from a trusted SLR vendor, a real-world dataset licensed from a curated data provider, and slides drafted by your medical writing agency. Then a journal reviewer, a health technology assessment body or your own medical governance committee asks a simple question: was AI used anywhere in producing this evidence, and how do you know it worked?
For many Medical Affairs teams today, the honest answer would be: “We’re not sure.”
That isn’t a failing of any individual. It reflects how quickly AI has moved into the evidence supply chain, and how much of it has moved out of sight. For many teams, most of the AI in their evidence portfolio isn’t in the tools they use themselves. It sits inside the work of the vendors they pay to screen literature, extract data, abstract real-world records and draft documents. It’s often on by default, and as the published record suggests, it is rarely visible to anyone reading the final work.
This article looks at where that AI is likely to be, why its invisibility is the real risk, and what it means for evidence leads who remain accountable for every number that reaches a clinician, a payer or a regulator.
This is not an argument against vendors using AI
Let’s be clear from the start: vendors adopting AI is, on balance, good news. Used well, it can make evidence work faster and cheaper.
The workload case is real. A 2025 review of how much AI can save in evidence synthesis, by authors from Roche and Cytel, found that AI-assisted screening cut the number of abstracts people needed to review by 55–64%. The studies it drew on were published between 2012 and 2023, so they largely predate today’s LLM-based tools, but even then, on a large review, that was a substantial share of the most labour-intensive step.
If your vendor is using AI responsibly, you may be getting better work for less money. The problem isn’t the AI. The problem is that you often can’t tell whether it’s being used responsibly, because you can’t tell whether it’s being used at all.
Where AI is entering your evidence portfolio
Most Medical Affairs teams work with four broad categories of evidence vendor, and AI is now established in all of them.
SLR and evidence-synthesis vendors are the most mature. Review platforms increasingly build machine-learning prioritisation and LLM-based screening and extraction into the workflow itself. A vendor may use AI to rank records, act as a second screener, extract data from full texts or draft the narrative synthesis. Some of this is a feature someone chooses to switch on. Increasingly, it’s simply how the platform works.
Real-world data providers increasingly use AI to abstract variables from unstructured clinical notes: a diagnosis date, a line of therapy, a disease stage, a response assessment. The variable in the file you receive looks the same whether a trained human abstractor, a rule-based algorithm or a large language model produced it.
CROs and medical writing agencies increasingly use AI to help draft protocols, study reports, manuscripts, abstracts, plain language summaries and slide decks.
Medical information and literature-monitoring services increasingly use AI to triage records and draft responses.
Across all four, the pattern is the same. The deliverable looks exactly as it always did. The methods note may or may not mention AI. And unless someone on your side asks, nobody may ever record which model was used, which version, on which tasks, or how well it performed.
What the public record shows
In our work evaluating AI evidence platforms, Eularis has looked closely at how widely agencies use them. For just one widely used platform, one of several in this space, around 25 medical affairs and market access agencies publicly state that they work with it. That is a floor, not a ceiling. Some agencies offer the same capabilities under their own brand, so public partnerships understate how far these tools reach. Multiply that across the other platforms in the market, and AI is clearly a routine part of how agency evidence work gets done.
Now set that against what appears in the published record.
When the BMJ Group added mandatory AI-disclosure questions to its submission system, researchers analysed 25,114 research manuscripts, including systematic reviews and meta-analyses, submitted to 49 BMJ journals in 2024. Only 5.7% disclosed any AI use, even though surveys suggest that somewhere between 28% and 76% of researchers use AI. And of the 1,431 manuscripts that did disclose, just 28 named an evidence synthesis tool: roughly one in every thousand submissions. A separate study of more than 100,000 manuscripts submitted to
JAMA Network journals found a disclosure rate of 3.3%.
Evidence synthesis shows a subtler version of the same pattern. A 2025 analysis of 2,271 evidence syntheses published by Cochrane, the Campbell Collaboration and Environmental Evidence found that around 12% used tools with machine-learning capability, but roughly 90% of those didn’t say whether the machine-learning features had actually been used. The tool was named; whether AI did any of the work was not.
It would be easy to read this as people hiding their use of AI, but that isn’t the useful reading. The BMJ authors offer two possible explanations: authors may be uncertain about what AI use needs to be disclosed, or they may be hesitant to declare it. Both point to a system where expectations are still settling.
There are structural reasons too. Journal requirements still vary in what they ask for and where, and authors are often unsure what counts as reportable AI use. A vendor may disclose its methods fully to the client, but that detail doesn’t automatically travel into a manuscript written months later by different people. When AI is built into a platform by default, it often isn’t thought of as a separate method at all. And much agency evidence work, from HTA dossiers to internal reviews and platform documents, never reaches a journal.
Whatever the mix of reasons, the practical conclusion is the same. The published record can’t tell you where AI was used in the evidence you rely on. If you want to know, you have to ask, and you have to build the answer into how the work is contracted and recorded.
Why invisibility is the real risk
When AI is used visibly, you can calibrate your oversight to it. When it’s invisible, three things become very difficult.
You can’t verify at the right depth. AI errors have a particular character: they look exactly like correct values. A human abstractor who can’t find a value usually leaves the field blank or flags it. A model that hasn’t been carefully constrained may supply a plausible value instead, one that is typical for the drug class but appears nowhere in the paper.
The evidence also shows that errors aren’t spread evenly. A 2026 systematic review of 27 studies of LLM-based data extraction found that accuracy ranged widely across studies, from below 50% to near-perfect. More tellingly, it differed by type of field: categorical and text fields came out at roughly 74–96%, but numerical fields at only about 47–88%. Numbers, the things that end up on platform slides and in payer dossiers, are where AI extraction is weakest. If you don’t know a vendor used AI for extraction, you won’t know to check the numbers hardest.
You can’t disclose. Disclosure expectations for AI in evidence work have firmed up. In 2025, Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence published a joint position statement on AI in evidence synthesis. It permits AI, but requires that any use of AI or automation that makes or suggests judgements be fully and transparently reported, including the name of the AI system, tool or platform, its version, the dates it was used, what it was used for and which parts of the process it affected.
Read that against your vendor portfolio. If your SLR vendor used AI for screening or extraction and you publish the review, the disclosure lands on you. And it can be written only with details that had to be captured while the work was happening. Nobody can reconstruct the model version a vendor’s platform was running eighteen months ago.
You can’t defend it. When a payer, regulator, journal or auditor asks how a number was produced, “the vendor handled it” isn’t an answer. Your accountability does not transfer with the work. A disclosure you can’t make, or a provenance record you can’t produce, is your problem, not the vendor’s.
The real-world data blind spot
Of the four categories, curated real-world data deserves particular attention, because the AI is most deeply buried there.
When a data provider quotes a single accuracy figure for its AI abstraction, it’s worth asking: accuracy of what? A headline figure may be an average across every variable in the dataset. Your study may depend on two or three variables, perhaps a progression date, a biomarker status or a line-of-therapy definition, and performance on those may be quite different from the average. It may also differ by subgroup, by care setting, or by the country whose records the model was validated on.
Recent work has started to set out what good validation of AI-curated real-world data looks like. One example is the VALID framework for AI-extracted oncology data, published in a peer-reviewed journal in 2026. It was developed by Flatiron Health, itself a real-world data provider, which is worth bearing in mind, as with any evidence produced by the people whose products it assesses. The practical lesson for evidence leads is that the right questions to ask a data provider are now quite specific, and quite different from the questions most teams were trained to ask about a traditional dataset. Most data-licensing checklists haven’t caught up yet.
The clause people forget, and regret
One of those questions catches people out more than any other: what happens when the model changes?
Suppose an SLR vendor shows you strong performance evidence for its AI screening tool at the RFP stage. Six weeks into your project, its platform upgrades to a newer model. The evidence you evaluated now describes a system that no longer exists, and your project effectively has two methods. Unless your contract required the vendor to tell you before any change of model or configuration on your work, you may never know.
The same applies to real-world data. A dataset validated under one model version and refreshed under another looks identical in the file you receive.
It isn’t.
Model-change notification is one of seven provisions that belong in the RFP and contract for any AI-assisted evidence work. All seven have the same awkward property: they’re almost impossible to add once work has started, because by then the information you’d need has already gone unrecorded. Teams that get this right agree a standard schedule with procurement and legal once, and attach it to every evidence RFP.
You can’t simply re-do the work
Once the risk is clear, a natural instinct is to check everything a vendor delivers. On a review with dozens of studies and dozens of fields per study, that quickly becomes a second, slower version of the original work. Teams that try it usually stop within weeks.
The answer is a deliberate audit: one that concentrates effort where errors are most likely and most consequential, gives a meaningful view of the whole deliverable’s quality, and sets in advance the point at which the work goes back to the vendor.
This is where the course lab for this topic tends to surprise people. Participants audit a fictional vendor’s extraction deliverable using a small sample of studies, and the sample is enough to surface errors that are completely invisible if you only read the extraction table. The vendor’s methods note has gaps of its own, and they’re easy to miss on a first read. We’ll come back to what that lab involves at the end.
Why this is getting harder, not easier
It would be comforting to think this is a transitional problem that will settle once vendors and sponsors get used to AI. In practice, three trends are making it harder.
First, AI is moving from optional features to platform defaults. When a capability is something a vendor deliberately switches on, it tends to get mentioned in a proposal. When it is simply part of how the platform works, it often doesn’t, because the vendor no longer thinks of it as a separate method.
Second, the models underneath these platforms change frequently. A vendor may be entirely transparent about the tool it uses and still not think to mention that the model inside it was updated last month. The more often models change, the more often the performance evidence you were shown stops describing the system doing your work.
Third, evidence supply chains are getting longer, through subcontracted searches, licensed abstraction and third-party platforms. Each extra link is another place AI can enter without anyone on your side being told.
None of this is anyone behaving badly. It’s simply what happens when a technology spreads faster than the contracts and oversight around it.
The evidence lead’s role is changing
The deeper implication is that the evidence lead’s job is shifting. The traditional model, which was to commission the work, receive it and review it, assumed the main risks would be visible in the deliverable. With AI in the supply chain, more of the risk sits in what the deliverable doesn’t show: which tasks were automated, how the tool was validated, whether it changed mid-project, and what a person actually checked. So the role moves towards two activities either side of the deliverable: specifying the work so the right information is captured as it happens, and auditing what comes back in a way that looks beneath the surface. Neither is a technical skill. Both are the kind of judgement Medical Affairs already applies to clinical evidence, now applied to how the evidence was produced.
Where to start: one question
If you take one action from this article, make it this. On every evidence engagement, new or existing, ask your vendors:
Where in this deliverable was AI used, and what can you show me about how well it performed?
The answers are revealing. Some vendors will reply immediately and precisely, with the tool, the model version, the tasks and performance evidence on comparable work. Others will need time to find out, which tells you their records may not support the disclosure you’ll eventually need to make. A few will be surprised to be asked at all.
A two-question self-check
Take your three most recent vendor deliverables in evidence generation and ask two things.
For each one, could you state today, with confidence, which tasks used AI and which model version was running?
And does any of those contracts say what happens if the vendor changes its model mid-project?
If the answer to either is no, your contracts are missing at least one of the seven provisions, and probably more. Asking the question above will tell you where AI is being used today. It won’t fix the contracts, the data-provider questions or the audit method. Those take a structured approach, and they’re worth getting right before the next renewal rather than after the next disclosure request.
Learning to oversee what you can’t see
The shift Medical Affairs and Market Access teams need to make is easy to state and harder to practise. Evidence leads have been trained to judge a deliverable by reading it. With AI in the supply chain, reading isn’t enough, because AI errors are built to read well. Oversight has to move upstream, into how work is specified and contracted, and it has to be backed by a proportionate way of auditing what comes back.
Your vendors are already using AI on your evidence. The question is whether you find out where, and how well, before someone else asks you.
Learn how
This is the focus of Module 5, Overseeing AI-Assisted Vendor Work, in AI for Evidence Generation, a hands-on, recorded course from our Pharma AI Enablement Institute.
You’ll work through the seven contract provisions, the questions to put to any provider of AI-curated data, and a sampling method for auditing a deliverable without redoing it. Then, in the lab, you audit a fictional vendor’s file end to end and decide whether to accept or return the work. You leave with an RFP and contract checklist for AI-assisted evidence work and a sampling audit checklist for your next engagement.
Module 5 is one of six core modules that give you a single method for judging AI-assisted evidence work at volume. You then apply it to your role in optional blocks for synthesis and RWE, publications and medical information, or leadership.
Every module ends in a lab built around one fictional product, and a learner pack holds every hand-out, prompt and answer key. Each lab includes pre-run AI output, so you can do the checking even without access to your company’s AI tool.
New to AI? Start with our Judgement Layer courses.