Evidence Quality: Not All Studies Are Equal

Both investigate the same treatment.

Study A

Has enough participants. Uses appropriate randomisation. Has a suitable comparison group. Measures relevant outcomes. Has relatively little missing data. Reports its methods clearly. Follows participants for a meaningful period.

Now:

Study B

Has 16 participants. Doesn’t clearly explain randomisation. Loses many participants during follow-up. Measures only an immediate outcome. Reports results selectively. And gives us very little information about what actually happened. Technically: Both may be called RCTs. But should we trust them equally?

No.

The label tells you the study design. It doesn’t guarantee the study quality.

Small studies aren’t automatically bad. Sometimes small studies are appropriate, especially in early research. But small samples usually create more uncertainty around estimates and may be more vulnerable to chance findings.

Imagine testing a treatment on eight people. Six improve. That’s interesting.

But:
Would the same thing happen in 80 people?
800?
Different ages?
Different health conditions?
Different therapists?

We shouldn’t ask a small study to answer a bigger question than it was designed to answer.

Bias doesn’t simply mean: “The researcher has an opinion.” In research, bias refers to systematic problems that can distort the estimated result.

For example:
Were participants allocated to treatment groups appropriately?
Did researchers know which treatment participants received when measuring outcomes?
Were large numbers of participants lost during the study?
Were all planned outcomes reported?
Were groups treated differently apart from the intervention being studied?

These issues can affect how much confidence we have in the findings. The important question isn’t: “Is this study biased: yes or no?” Almost every study has limitations. The better question is: “Could these limitations meaningfully change the result?”

In drug research, it may sometimes be possible to give participants tablets that look identical while keeping them unaware of which treatment they received. Manual therapy is different.

If you’re receiving a massage…you’ve probably noticed. The therapist has definitely noticed. So blinding participants and therapists can be difficult or impossible in many physical interventions. That doesn’t automatically make the study poor quality.

But it does affect what kinds of bias researchers need to consider. Evidence quality should be judged in the context of what was realistically possible for that type of intervention.

Imagine a treatment performs better than doing nothing. Useful. Now compare the same treatment with a well-designed exercise program. Maybe the difference becomes much smaller. Neither result is necessarily wrong. They’re answering different questions.

So when someone says: “Research shows this treatment works.” ask: “Compared with what?” A study’s conclusion can look very different depending on the control or comparison group.

Suppose a study reports that treatment significantly improved range of motion.

Great.
But what if the average improvement was tiny?
Did pain change?
Did function improve?
Could participants return to normal activity?
Did the improvement last?

A study can measure something accurately that doesn’t matter very much to the person receiving treatment. That’s why evidence quality isn’t only about methodology. We also need to consider whether the outcomes are relevant and meaningful.

So don’t stop at: p < 0.05 Look for:

Effect size
How large was the difference?

Confidence interval
How precise was the estimate?

Clinical relevance
Would the change matter in real life?

Numbers need context.

Imagine two studies estimate the same treatment effect. Study A suggests the true effect is probably somewhere within a relatively narrow range. Study B gives an enormous range from potentially useful to potentially negligible.

The average estimate might look similar. But our confidence isn’t. This is where confidence intervals become useful. They remind us that research doesn’t usually give us one perfectly certain number.

It gives us an estimate with uncertainty around it. Good evidence interpretation doesn’t hide that uncertainty. It shows it.

Suppose ten reasonably good studies investigate the same question. Nine produce broadly similar results. One differs. That’s one situation. Now imagine ten studies producing results pointing in completely different directions.

That’s another. When research findings are inconsistent, our confidence in a simple overall conclusion may decrease.

Then we ask:
Why?
Different populations?
Different treatment methods?
Different doses?
Different comparison groups?
Different study quality?
Or simply chance?

Variation doesn’t automatically invalidate the evidence. But it needs explaining.

Because not all studies are equal. Published Doesn’t Mean Proven Getting research published doesn’t transform a finding into a fact. A study can be published and still have important limitations.
Maybe the sample was very small.
Maybe the comparison wasn’t appropriate.
Maybe many participants dropped out.
Maybe the outcome measurement was unreliable.
Maybe the groups were different before treatment.
Maybe the follow-up was extremely short.
Maybe the results are too uncertain to support a strong conclusion.
This doesn’t necessarily make the study useless.

It means we need to interpret it appropriately.

Imagine excellent research demonstrates that a treatment helps healthy athletes aged 18–25. Can we automatically apply that result to older adults recovering from surgery?

No. This is sometimes a problem of indirectness. The evidence may be good. But it may not directly answer the clinical question we actually have.

Ask:
Are these the right people?
The right intervention?
The right comparison?
The right outcome?

Evidence can be high quality for one question and not particularly useful for another.

Research may be funded by companies, organisations, governments, universities or other groups. Funding sources and conflicts of interest should be reported and considered. But: Industry funded = automatically false isn’t good critical thinking either. A conflict of interest is information that helps us evaluate the research.

It should make us pay careful attention to study design, analysis and reporting. Not automatically throw the paper away. Critical thinking works in both directions.

A systematic review can be excellent. It can also be poorly conducted. A meta-analysis can provide a useful pooled estimate. It can also combine studies with serious limitations.

Remember: A sophisticated analysis cannot completely rescue weak underlying evidence. If five unreliable studies are combined, we don’t automatically receive one reliable answer. This is why good systematic reviews don’t simply combine results.

They also assess the quality or certainty of the evidence. What Does “Low-Certainty Evidence” Mean? This phrase is often misunderstood. Suppose a review concludes: “There is low-certainty evidence that the treatment may reduce pain.”

Some people translate that as: “The treatment doesn’t work.” That’s not necessarily what it means. Low certainty means our confidence in the estimated effect is limited. Future research may meaningfully change our understanding of the size or direction of that effect.

Similarly, high-certainty evidence doesn’t mean: “This works for every person.” It means we have greater confidence in the evidence regarding the question being studied. Certainty describes confidence in the evidence not certainty about every individual.

When researchers assess the overall certainty of a body of evidence, confidence may be reduced for reasons such as: Risk of bias Important problems in how studies were designed or conducted. Inconsistency Studies produce substantially different findings. Indirectness

The research doesn’t perfectly match the population, intervention or outcome we’re interested in. Imprecision The estimates are too uncertain. Publication bias

The available published evidence may not represent all the research that was conducted. You don’t need to memorise these terms. Just remember the question behind them: “What could make us less confident in this conclusion?”

Twenty weak studies don’t necessarily beat three excellent ones. A thousand participants don’t automatically rescue poor methodology. A meta-analysis isn’t automatically superior because it has a beautiful forest plot.

And a paper doesn’t become trustworthy because its title contains fourteen complicated scientific words. Evidence quality is not about being impressed. It’s about evaluating.

When you encounter a research claim, ask:

1. What type of study is it?
What question can that design reasonably answer?

2. Was the study conducted well?
Look for important risks of bias and methodological limitations.

3. How many people were studied?
And how precise are the results?

4. What was the treatment compared with?
Context changes interpretation.

5. What outcome changed?
Was it meaningful?

6. Are the results consistent with other research?
One study doesn’t exist in isolation.

7. Does the evidence apply to the person or question we’re interested in?

And finally:

How confident should we actually be?

That’s evidence appraisal. Don’t Ask Only “Is There Evidence?” Ask: “How good is the evidence?” Then: “What does it actually support?” Then: “How confidently can we apply it here?”

That’s not weakness. That’s exactly what evidence-based practice is supposed to do. So the next time someone says: “But there’s research supporting it…” the conversation doesn’t have to end.

There’s one more very useful question: “How good is the research?”