Robustness Checks #SmartPLS4 - Unobserved Heterogeneity using SmartPLS4

TL;DR

Unobserved heterogeneity can make full-sample SmartPLS estimates misleading when subgroup effects differ and cancel out.

Briefing Cornell Notes

Briefing

Unobserved heterogeneity can quietly undermine SmartPLS results when different latent subgroups produce substantially different parameter estimates—so analyzing the full sample as one group can make positive and negative effects cancel out. The session focuses on a practical robustness-check workflow for detecting that problem in SmartPLS4 using the FINITE MIXTURE segmentation procedure (often referred to as “FIMIX/FIX” in SmartPLS contexts). The goal is to decide whether the data should be split into multiple segments before interpreting model estimates.

The procedure starts with a key design choice: how many segments to test. The workflow uses a recommended segmentation sample size based on a medium effect size of 0.15 and statistical power of 0.8, yielding a minimum of about 85 observations per segment. With a total sample size of roughly 341, dividing 341 by 85 implies testing about 4 segments (i.e., running solutions for 1 through 4 segments). In SmartPLS4, this is done by running the finite mixture segmentation for segment counts from 1 up to the chosen maximum, then exporting the model selection criteria for each solution.

For each segmentation run, the analysis relies on multiple fit/selection indices—specifically AIC and BIC variants (including AIC3/BIC and AIC4/BIC), plus CAIC and MDL (minimum description length) and entropy. Rather than treating any single index as decisive, the workflow compares where each criterion reaches its minimum (or otherwise indicates the best segment count). In the example, the indices do not agree cleanly: AIC3 and CAIC point to different segment numbers (AIC3 favoring a 4-segment solution while CAIC favors 2 segments). Meanwhile, AIC4 and BIC also converge on a 2-segment solution, and MDL5 points in a direction that conflicts with the AIC4/BIC agreement (in the example, MDL5 favors a 1-segment solution).

That disagreement creates an “ambiguous picture.” Since the segmentation criteria collectively fail to unambiguously identify a single best number of segments, the workflow concludes that there is no clear evidence of unobserved heterogeneity severe enough to justify splitting the data. The practical takeaway is conservative: when the model selection indices disagree across segment counts, the robustness-check outcome is to proceed with the original analysis on the full dataset rather than creating segments for further interpretation.

Cornell Notes

Unobserved heterogeneity arises when distinct subgroups in the data have meaningfully different model estimates, making a single full-sample analysis potentially misleading (effects can cancel). SmartPLS4’s finite mixture segmentation (FIMIX/FIX) checks for this by running solutions with different segment counts and comparing model selection criteria. The example uses a recommended per-segment sample size of about 85 (medium effect size 0.15, power 0.8), so a sample of ~341 implies testing 1–4 segments. Multiple indices are compared—AIC3/CAIC, AIC4/BIC, MDL5, and entropy—but they point to different segment numbers (e.g., AIC3 favors 4 segments, CAIC favors 2, AIC4/BIC favor 2, and MDL5 favors 1). Because the criteria are ambiguous rather than consistent, the workflow opts to analyze the whole dataset without segmentation, concluding no clear unobserved heterogeneity issue.

Why does unobserved heterogeneity threaten SmartPLS results, and what failure mode does it create?

Unobserved heterogeneity occurs when the data contain subgroups that produce substantially different parameter estimates. If the model is estimated on the entire dataset as one group, subgroup-specific positive and negative effects can offset each other, yielding misleading overall estimates that don’t reflect any subgroup accurately.

How does the workflow decide how many segments to test in FINITE MIXTURE segmentation?

It uses a recommended minimum sample size per segment derived from a medium effect size of 0.15 and power of 0.8, giving about 85 observations per segment. With a total sample around 341, dividing 341 by 85 suggests testing roughly 4 segments, so the procedure runs segment solutions for 1 through 4 segments.

Which model selection criteria are used to judge the best segmentation solution?

The workflow compares common FIX/finite-mixture model selection criteria exported from SmartPLS4: AIC variants (AIC3 and AIC4), CAIC, BIC (paired with the AIC variants), plus MDL (minimum description length, noted as MDL5 in the example) and entropy. The decision logic centers on where these criteria indicate minima or preferred segment counts.

What does it mean when different indices recommend different numbers of segments?

It signals an ambiguous segmentation decision. In the example, AIC3 points toward 4 segments while CAIC points toward 2 segments. AIC4 and BIC also point toward 2 segments, but MDL5 points toward 1 segment. Because the indices don’t converge on a single segment number, the evidence for unobserved heterogeneity is not decisive.

What final action does the workflow take when segmentation criteria are ambiguous?

When the indices disagree on the number of segments, the workflow avoids creating segments for further analysis and instead analyzes the whole dataset together. In the example, that leads to the conclusion that there’s no clear unobserved heterogeneity problem.

Review Questions

In unobserved heterogeneity, why can estimating a single model on the full sample produce misleading effects?
Given a total sample size of about 341 and a recommended per-segment minimum of 85, how many segment solutions should be tested in this workflow?
What is the practical decision rule when AIC/CAIC, AIC/BIC, and MDL suggest different segment counts?

Key Points

1
Unobserved heterogeneity can make full-sample SmartPLS estimates misleading when subgroup effects differ and cancel out.
2
FINITE MIXTURE segmentation in SmartPLS4 checks for unobserved heterogeneity by testing multiple segment counts and comparing selection criteria.
3
Use the recommended per-segment minimum sample size of about 85 based on medium effect size 0.15 and power 0.8 to choose how many segments to test.
4
Compare AIC3/CAIC, AIC4/BIC, MDL5, and entropy across segment solutions to see whether criteria converge on one segment number.
5
If selection indices point to different segment counts (an ambiguous picture), avoid segmentation and proceed with the full dataset analysis.
6
A consistent, unambiguous recommendation across indices is the condition that would justify segment-based interpretation.

Highlights

Unobserved heterogeneity can cause subgroup-specific positive and negative effects to cancel when the model is estimated on the full sample.

The segmentation workflow uses a medium effect size (0.15) and power (0.8) to justify ~85 observations per segment, leading to testing 1–4 segments for a ~341 sample.

AIC3/CAIC, AIC4/BIC, and MDL5 can disagree on the optimal number of segments, producing an ambiguous decision.

When indices don’t converge, the robustness-check outcome is to analyze the whole dataset rather than create segments.

Topics

Unobserved Heterogeneity
SmartPLS4
Finite Mixture Segmentation
Model Selection Criteria
Robustness Checks

Mentioned

SmartPLS4
AIC
BIC
CAIC
MDL
FIX