Challenges in using RoB 2 - A worked example

Risk-of-bias assessment under RoB 2 proved both time-consuming and technically demanding for a systematic review of remote therapies for alcohol and drug misuse—especially when trials reported multiple outcomes, used complex social-intervention designs, and left key methodological details unclear. Across 53 RoB 2 assessments covering 45 randomized controlled trials (with only one study judged low risk of bias), most evidence landed in “high risk” or “some concerns,” underscoring how difficult it is to apply standardized bias criteria to heterogeneous remote-treatment research.

The work focused on adults in OECD countries receiving remotely delivered interventions—online or by telephone or via smartphone apps—such as remote recovery support, remote talking therapy, and self-guided therapy. Comparators included in-person care, no intervention, or alternative remote interventions. Outcomes were quantitative measures of substance use (for example, days of use or relapse), and the review team had to handle wide variation in what trials measured and how they measured it.

A first major challenge was pre-specifying which outcome to assess. The included studies were heterogeneous and there was no consensus on core outcomes for substance-use treatments. To manage this, the team used an iterative four-step decision process: prioritize outcomes relevant to stakeholders (excluding areas like mental health or criminal justice involvement), select commonly reported substance-use outcomes across the dataset, favor the most robust measurement approach (toxicology or self-report verified by toxicology over self-report alone), and ensure comparability by considering measure type (dichotomous vs continuous), statistical approach, and measurement time points.

Second, assessing and reporting multiple outcomes created duplication and workload. RoB 2 requires outcome-specific judgments, but many domains remain unchanged across outcomes within a trial. The team streamlined the process by identifying RoB 2 items likely to vary across outcomes (e.g., missing outcome data, self-report measurement issues, and outcome-ascertainment differences) while treating stable domains more consistently. They also applied pragmatic thresholds for missing data (e.g., ≥95% participant data for continuous outcomes) and treated self-report bias differently depending on whether outcomes were monitored or verified (urine toxicology was treated as more objective than unverified self-report).

Third, interpreting statistics and trial methodology was difficult because RoB 2 expects statistical and methodological sophistication, yet many trials provided limited information. The team often had to triangulate across multiple records—protocols, registration documents, and linked reports—to infer whether participants or intervention deliverers were aware of allocation, how missing data were handled, and what models were used. When protocols were unavailable, assumptions were required; for example, fully automated interventions were treated as less prone to bias from delivery awareness.

Finally, navigating RoB 2 guidance itself was a barrier. The “cryptic” nature of the checklist prompted frequent back-and-forth to full guidance, training videos, and webinars. To reduce ambiguity, the team piloted and built a review-specific guidance document that translated RoB 2 items into concrete decision rules for their remote-therapy context—such as treating imputing missing outcomes as positive drug use as an acceptable correction when missingness was presumed nonrandom, and requiring biochemical verification (e.g., urine toxicology) before judging whether outcome assessment was influenced by knowledge of allocation.

Overall, the experience highlighted that applying RoB 2 to broad, complex remote-therapy reviews demands more examples, clearer scenarios for missing information and advanced methods, and consolidated, multi-layered guidance resources—so assessments remain consistent even when trial reporting is incomplete.