Webinar - Scale Development and Validation: A thorough guide on how to develop and validate a scale

Scale development is the process of turning an abstract, unobservable concept (a latent construct) into a measurable questionnaire score—then proving that the items work together reliably and validly. Because most quantitative research depends on measurement, researchers often start with existing scales, but when a scale doesn’t fit a specific context (like higher education), a new scale must be built from scratch. A scale is typically a standardized set of self-report items whose responses are summed (or otherwise combined) to represent the underlying construct—such as “university social responsibility” or “servant leadership”—through multiple dimensions.

The foundation comes first: an operational definition. Before writing or selecting items, researchers must define what the construct means in their specific study context and how it will be operationalized. That definition then guides every later decision—what items are included, how they are grouped into dimensions, and even how the dimensions are named. Without a clear operational definition, the entire scale development effort risks becoming misaligned with what the construct is supposed to measure.

Next comes item generation and refinement. A common approach begins with a literature search to collect existing items from prior scales, then expands the item pool using stakeholder input such as interviews, focus groups, or expert discussions. Items are then reviewed by an expert panel—often combining academics and practitioners—to judge whether each statement is relevant, clear, and truly measures the intended construct. Researchers may also use “validation items” (e.g., reverse-phrased or negating items) to detect careless responding. Through iterative revisions and pilot testing, the item pool is reduced to a workable set that still represents the construct’s dimensions.

Psychometric testing follows, starting with exploratory factor analysis (EFA). EFA acts as a data-reduction and structure-checking tool: it tests whether the items actually cluster together in the way the researcher hypothesized. Key diagnostics include KMO (sampling adequacy), Bartlett’s test (whether correlations are sufficient for factor analysis), eigenvalues (often retaining factors with values greater than 1), and factor loadings (how strongly each item relates to a factor). Items with weak loadings or cross-loadings—where an item loads on multiple factors—are removed or revised. The goal is a coherent factor structure that matches the conceptual model.

Confirmatory factor analysis (CFA) then tests whether the proposed measurement model fits the data. CFA evaluates factor loadings and model fit indices such as RMSEA and SRMR (with thresholds discussed), and it may use modification indices to improve fit—though only within constraints like correlating residuals for conceptually similar indicators. After the structure is accepted, reliability and validity are established. Reliability is assessed using composite reliability (and related measures), while convergent validity is checked via Average Variance Extracted (AVE). Discriminant validity is tested using the Fornell–Larcker criterion, comparing the square root of AVE for each construct against its correlations with other constructs.

A practical example in the transcript illustrates the workflow: starting from a large literature-derived item pool, adding interview-based items, using expert marking to cut down the number of items, piloting, then running EFA to remove problematic items until a stable seven-factor structure emerges. The process ends with CFA, followed by composite reliability, AVE, and discriminant validity checks—producing a scale that is both statistically defensible and conceptually grounded for its target context.