How to Read a Systematic Review?

Before reading the conclusion, find out what question the review was actually trying to answer. For example: Does massage reduce chronic low back pain in adults? That’s much more useful than: “Does massage work?”

Look for details such as:

• the population
• the intervention
• the comparison
• the outcomes
• sometimes the timeframe

You may encounter the framework: PICO

P — Population
Who are we interested in?

I — Intervention
What treatment or approach is being studied?

C — Comparison
What is it being compared with?

O — Outcome
What are researchers measuring?

If you understand those four things, you’ve already understood a large part of the review.

Imagine a systematic review concludes that massage helps pain. Before accepting that statement, ask: What kind of pain?
Neck pain?
Low back pain?
Post-exercise soreness?
Several completely different conditions combined together?

Then look at the treatments. Were all studies investigating similar massage approaches? Or did one use 10 minutes of massage, another use 60 minutes, another combine massage with exercise and another use an entirely different manual therapy?

If very different studies are grouped under one broad label, interpretation becomes more difficult. A review can only synthesise the studies that went into it.

This is one of the features that separates a systematic review from an ordinary narrative overview. Researchers should describe how they searched for studies. That may include:
Which databases were searched?
Which search terms were used?
What dates were covered?
Were there language restrictions?
What types of studies were eligible?

A good systematic review tries to identify the relevant evidence using a transparent, reproducible process. Why does that matter? Because if researchers only look in one convenient place, they may miss important studies.

And if we only include evidence that’s easy to find, we may get a distorted picture.

Before researchers begin selecting studies, they should define what qualifies. Perhaps they include: Adults with chronic non-specific low back pain. Randomised controlled trials. Massage compared with usual care or another intervention.

Pain and disability outcomes. And exclude: Acute injuries. Children. Case reports.

Studies combining massage with several other treatments where the effect of massage can’t be separated. These decisions shape the final review. So don’t only ask: “How many studies were included?” Ask: “Why were these studies included?”

A systematic review containing 40 studies may sound more convincing than one containing six. But numbers alone don’t determine confidence. Suppose 40 poorly designed studies all have serious limitations.

Combining them doesn’t magically create excellent evidence. This is sometimes described with the less elegant principle: Garbage in, garbage out. Perhaps not the most sophisticated academic terminology.

Still memorable. A systematic review is only as informative as the evidence it contains and the quality of the review process itself.

This is a major issue. Imagine five studies investigating massage for shoulder pain.
Study one treats athletes.
Study two treats older adults.
Study three uses one treatment.
Study four uses eight weeks of treatment.
Study five combines massage with exercise.
Can we simply average everything together?

Maybe. Maybe not. Differences between studies are often described as heterogeneity. Studies can differ in:

• participants
• conditions
• interventions
• treatment dose
• comparison groups
• outcome measures
• follow-up periods
• methodology

Some variation is normal. But when studies are very different, a single combined conclusion may become harder to interpret.

These terms are often confused. A systematic review is the structured process of identifying, evaluating and synthesising relevant research. A meta-analysis is a statistical method that may be used to combine numerical results from multiple studies.

A systematic review may contain a meta-analysis. But it doesn’t have to. If studies are too different to combine meaningfully, researchers may synthesise the evidence without calculating one pooled effect.

So:
Systematic Review = the structured review process
Meta-Analysis = statistical combination of compatible study results

We’ll explore meta-analysis separately.

Suppose a systematic review finds a statistically significant reduction in pain. Useful.
But how large was the difference?
Was it large enough for patients to notice?
Did function improve?
Did quality of life change?
How long did the benefit last?

A small statistical difference can still have limited practical importance. This is why we keep returning to: Did it matter? Not simply: Was p < 0.05?

Good reviews don’t only tell us what the estimated effect was. They also help us understand how confident we should be in that estimate. You may see discussion of: confidence intervals or an assessment of the certainty of evidence. A narrow confidence interval generally suggests a more precise estimate.

A wide interval suggests greater uncertainty. Evidence certainty may also be reduced because of issues such as:

• risk of bias
• inconsistent findings
• indirect evidence
• imprecision
• publication bias

So two reviews can both report a positive result while supporting very different levels of confidence. “There appears to be an effect” and “we are highly confident there is an effect” are not the same statement.

Read it. But don’t treat it as the only part of the paper that matters. Researchers interpret their own results. Your job as a reader is to see whether the conclusion matches the evidence presented.

A cautious conclusion might say: “The intervention may provide short-term pain reduction, but certainty is low due to small studies and risk of bias.” Social media may translate that into: “SCIENTISTS PROVE THIS TREATMENT ELIMINATES PAIN.”

Those are not the same conclusion. Always distinguish: What the data showed from What someone says the data means.

Suppose a systematic review suggests massage may reduce pain in a particular musculoskeletal condition. That’s useful evidence. But we still need context.
How large was the benefit?
Compared with what?
Was it short-term?
Did function improve?
How certain was the evidence?
What type of massage was studied?
Did participants also receive exercise or education?
And does the evidence apply to the person we’re treating?

A systematic review doesn’t eliminate clinical reasoning. It gives clinical reasoning better information to work with.

When you open a systematic review, start here:

1. What question was the review asking?
Who, what treatment, compared with what, and which outcome?

2. How did the authors search for studies?
Was the process broad and transparent?

3. Which studies were included?
Do they actually match the question?

4. What was the quality or risk of bias of those studies?
More studies aren’t automatically better evidence.

5. Were the studies similar enough to compare?
Look at differences in participants, treatments and outcomes.

6. How large and meaningful was the effect?
Don’t stop at statistical significance.

7. How certain are we?
Does the evidence support confidence — or caution?

You don’t need to understand every statistical formula to ask these questions.

Systematic reviews are valuable because they help us see the wider evidence rather than relying on one study. But they don’t remove uncertainty. Different reviews may reach different conclusions. New research may appear. Methods may differ. Evidence quality may be limited. And the research still needs to be applied to an individual person.

So don’t ask only: “Is there a systematic review?” Ask: “Is it a good systematic review — and what does it actually tell us?”

A simple framework is:
Question → Search → Select → Assess → Synthesise → Interpret

That’s the logic behind a systematic review. And when you’re reading one: Don’t count studies. Evaluate evidence. Because ten weak studies don’t automatically become strong evidence simply because someone put them in the same PDF.