Breaking down Faheem's systematic literature review: From Idea to Execution

TL;DR

A systematic literature review must start with a clear purpose and research questions that are valuable, crisp, and answerable from existing literature.

Briefing Cornell Notes

Briefing

Systematic literature reviews can be built from scratch using a disciplined pipeline—then accelerated with AI tools—without surrendering the researcher’s judgment. The core takeaway is that publishable-quality reviews depend less on which review style is chosen and more on defining the right purpose and research questions, running a controlled search with inclusion/exclusion criteria, and extracting/analyzing data in a structured way. In the manual workflow described, a high-quality review took roughly 4–6 months, reflecting the heavy time cost of screening thousands of papers down to a final set.

The session starts by clarifying what a literature review is: synthesizing existing work to identify patterns, trends, concepts, and directions for future research. It then distinguishes multiple review types—systematic, scoping, integrative/critical, narrative, historical, and meta-synthesis—while emphasizing that systematic reviews follow a protocol tightly enough to satisfy reviewers. The practical “how-to” begins with the steps of a systematic literature review: define the need for the review and its research questions; run a pilot study to test feasibility; design search strings for databases (e.g., Google Scholar, ACM, Scopus); apply inclusion/exclusion criteria; select relevant studies; extract data using a predefined extraction form; analyze the extracted data (the example uses thematic analysis); and report findings with results, discussion, and future research implications.

A detailed case study illustrates the workflow. For a 2019 literature review on architectural tactics for big data systems (published in Journal of Systems and Software), the process began with research questions about quality attributes for BDCA systems and architectural tactics to address quality concerns. Paper identification relied on a manual, trial-and-error search-string approach across major computing databases, then progressively narrowed the pool: 4,634 initial hits, reduced via title screening to 748, deduplicated to 516, abstract screening to 168, full-text review to 69, and finally “snowballing” from references to add 5 more—ending with 74 included papers. Data extraction was guided by a structured form capturing authorship and publication details plus the technique, quality attributes, rationale, and tactics needed to answer the two research questions.

The second half shifts to automation using SciSpace (spelled “skypace” in the transcript), positioning it as a way to reduce the most time-consuming steps. SciSpace is presented as supporting paper identification with transparent filtering and relevance counts, automated data extraction into a spreadsheet-like structure (including extracting specific fields like quality attributes), paraphrasing via a Chrome extension, question-based extraction from papers, and citation generation in formats such as APA. Beyond these, it’s described as offering summaries, outlines, proofreading, formatting, exporting, and other writing-assistance features.

Looking ahead, the session argues that research will become faster and more tool-driven, but “AI supports, not replaces.” The future advantage goes to researchers who adopt tools effectively while still critically evaluating outputs, managing bias and limitations, avoiding AI-generated plagiarism risks (especially when writing is fully AI-produced), and protecting sensitive information by checking terms and data-safety policies. The skills emphasized for future researchers include prompt engineering, tools literacy, ethical AI usage, workflow automation, and critical evaluation of AI outputs—paired with enduring fundamentals like critical thinking, connecting research ideas, and strong reading and writing.

Cornell Notes

A systematic literature review succeeds when it is built on clear purpose and well-crafted research questions, then executed through a protocol: pilot testing, search-string design, inclusion/exclusion screening, structured data extraction, and thematic analysis. In a concrete example, a big-data architecture review narrowed 4,634 initial papers down to 74 through staged screening (title → abstract → full text) plus snowballing from references, and then extracted tactics and quality attributes into a predefined form. The workflow is time-intensive when done manually (about 4–6 months for publishable quality), but AI tools like SciSpace can automate paper identification, data extraction, paraphrasing, and citation generation. The future of research favors researchers who use AI to accelerate repetitive work while maintaining critical judgment, ethical safeguards, and awareness of bias, plagiarism, and data privacy risks.

What makes a systematic literature review different from other review types, and why does that matter for publication?

Systematic reviews rely on a clear, protocol-driven methodology for conducting and analyzing papers. That structure matters because it gives reviewers confidence that each claim is grounded in a defined process—especially when the review follows established guidelines (the example references Barbara Kitchenham’s guidelines). Other review styles (scoping, narrative, historical, integrative/critical, meta-synthesis) can be valuable, but the systematic approach is the one most closely tied to transparent, reproducible selection and analysis.

How do purpose and research questions shape the entire literature review pipeline?

Purpose determines what the review is for—such as understanding a field at the start of a PhD, producing a publishable paper, or identifying industry trends like evaluation metrics. Research questions then drive everything: they must be crisp, valuable, and answerable from the available literature. In the case study, the questions focused on quality attributes for BDCA systems and the architectural tactics used to address quality concerns; those questions then dictated what data fields were extracted from each paper.

Why is pilot testing used before running the full search, and what does it verify?

A pilot study runs the review logic on a small scale (about 8–10 manually selected papers) to check whether the defined research questions can actually be answered using the kinds of papers that appear in the literature. It also helps validate search-string effectiveness: if the search returns the pilot papers, that’s a strong sign the search strategy is aligned with the review’s goals.

How does the case study reduce thousands of papers to a final set without losing relevance?

The example uses staged screening plus deduplication and snowballing. Starting from 4,634 papers, title screening reduced the set to 748, deduplication brought it to 516, abstract screening narrowed it to 168, and full-text screening left 69. Finally, snowballing added 5 more relevant papers found in references that the initial database search missed, resulting in 74 included studies.

What does “data extraction” mean in practice, and how is it kept from becoming unmanageable?

Data extraction uses a predefined extraction form so the reviewer pulls only the information needed to answer the research questions. In the example, the form captured bibliographic details plus the technique, quality attributes, rationale for addressing those attributes, and architectural tactics—fields directly tied to the two research questions. Extracted data then feeds analysis (the example uses thematic analysis) and becomes the basis for results and discussion.

Which parts of a systematic review are most amenable to automation with SciSpace, and what safeguards remain necessary?

Automation is presented as strongest for paper identification (with relevance counts and filters), data extraction into structured outputs (including custom fields like quality attributes), paraphrasing, question-based extraction from papers, and citation generation (e.g., APA). Safeguards still matter: researchers must critically evaluate AI outputs, manage bias/limitations, avoid AI-generated plagiarism when writing is fully AI-produced, and protect sensitive information by reviewing platform terms and data-safety policies.

Review Questions

If you had to start a systematic literature review today, what order would you follow for purpose → research questions → pilot study → search-string design → inclusion/exclusion → extraction → analysis?
In the case study, which screening stage contributed the largest reduction, and how did snowballing change the final included-paper count?
What skills beyond traditional reading/writing are needed to use AI tools effectively in research, and why is critical evaluation still required?

Key Points

1
A systematic literature review must start with a clear purpose and research questions that are valuable, crisp, and answerable from existing literature.
2
A pilot study (8–10 papers) tests whether the questions can be answered and helps validate that the search strategy is aligned with the review’s goals.
3
Search-string design is a trial-and-error process that must balance recall (enough papers) with precision (not hundreds of thousands).
4
Inclusion/exclusion criteria and staged screening (title, abstract, full text) are essential to reduce thousands of hits to a manageable, relevant set.
5
Structured data extraction forms prevent reviewers from getting lost and ensure extracted fields directly map to the research questions.
6
AI tools like SciSpace can automate paper identification, data extraction, paraphrasing, and citation generation, but researchers must still critically evaluate outputs and manage ethics, bias, and privacy.
7
Future research skills include prompt engineering, tools literacy, ethical AI usage, workflow automation, and the ability to judge whether AI-generated content is reliable.

Highlights

A publishable systematic review can take about 4–6 months when done largely manually, largely due to screening and extraction workload.

One example narrowed 4,634 initial papers down to 74 using title/abstract/full-text screening plus snowballing from references.

SciSpace is positioned as automating not just paper discovery but also structured data extraction into custom fields and citation generation (including APA).

The future advantage goes to researchers who use AI for repetitive acceleration while retaining critical judgment, ethical safeguards, and awareness of bias and plagiarism risks.

Topics

Mentioned

SciSpace
Barbara Kitchenham