What is the difference between Q²predict and the Q² from blindfolding?

The blindfolding Q² is an in-sample measure of predictive relevance computed from the estimation data. Q²predict comes from out-of-sample k-fold cross-validation and reflects how well the model predicts cases it was not trained on. Only the latter is about genuine prediction.

Should I use RMSE or MAE in PLSpredict?

Use RMSE by default. Switch to MAE when the distribution of prediction errors is strongly non-symmetric, since RMSE weights large errors more heavily.

What counts as good predictive power?

Q²predict above 0 for the target indicators, and lower PLS errors than the linear-model benchmark. If PLS beats the benchmark on all indicators the model has high predictive power, on most it is medium, on a few it is low, and on none it has no predictive power.

PLS-SEM METHOD

PLSpredict: assessing out-of-sample predictive power

PLSpredict evaluates how well your PLS-SEM model predicts new cases it was not trained on, rather than how well it fits the data you already have. This page explains what it measures, how to read Q²predict and the linear-model benchmark, the metric variants, and the common pitfalls, above all confusing in-sample fit with real predictive power.

What PLSpredict measures

Standard PLS-SEM reporting focuses on in-sample measures such as R² and the Q² from blindfolding, which describe how well the model explains the data used to estimate it. They say little about whether the model can predict new observations. PLSpredict (Shmueli et al. 2016) fills that gap with out-of-sample prediction.

It uses k-fold cross-validation: the sample is split into k folds, the model is trained on all but one fold, and the held-out fold is predicted. Repeating this across folds yields a predicted value for every case without ever using that case to train the model. Those predictions are then compared with the actual values to quantify predictive power.

A worked example

For the three indicators of the target construct, Q²predict stays above 0 throughout, and the PLS prediction error (RMSE) is lower than the linear-model benchmark on all three. Because PLS beats the benchmark on every indicator, the model has high predictive power.

Had PLS beaten the benchmark on only one of the three, you would report medium to low predictive power instead.

PLS RMSE is below the linear-model (LM) benchmark on all three indicators, and Q²predict stays above 0: high predictive power.

How to read the result

PLSpredict reports, for each indicator of your key target construct, a Q²predict and a prediction error (RMSE or MAE) for both the PLS model and a naive linear-model (LM) benchmark.

Q²predict > 0: The indicator is predicted better than the most naive benchmark (the training-sample mean). A value at or below 0 means no predictive relevance for that indicator.
PLS < LM (all): High predictive power: the PLS prediction error is lower than the linear-model benchmark on every indicator.
PLS < LM (most / few): Medium predictive power if PLS beats the benchmark on most indicators, low if only on a few.
PLS ≥ LM (none): No predictive power: the model does not beat the benchmark. Do not make predictive claims from it.

Compare errors on the indicators of your main target construct, not every indicator in the model. Use RMSE by default, and switch to MAE when the distribution of prediction errors is highly non-symmetric.

Metrics and variants

Q²predict

Out-of-sample predictive relevance per indicator. Above 0 means the model predicts better than the indicator mean. It is not the same as the Q² from blindfolding, which is in-sample.

RMSE vs MAE

RMSE (root mean squared error) is the default. MAE (mean absolute error) is preferred when prediction errors are strongly non-symmetric, because RMSE is more sensitive to large outlying errors.

LM benchmark

A linear regression of each indicator on the indicators of its direct-antecedent constructs. PLSpredict asks whether the theory-based PLS model beats this simple benchmark.

CVPAT

The Cross-Validated Predictive Ability Test (Liengaard et al. 2021) formally tests whether the model’s average prediction error is significantly lower than a benchmark (indicator average or LM).

Running PLSpredict step by step

Identify the key target construct whose indicators you care about predicting.
Choose the number of folds k (10 is common) and the number of repetitions; more folds and repeats give more stable estimates.
Run PLSpredict to obtain Q²predict and the PLS and LM errors for each indicator.
Check that Q²predict is above 0 for the target indicators.
Compare the PLS error against the LM benchmark across those indicators and classify predictive power as high, medium, low, or none.

Common pitfalls

Treating in-sample R² or the blindfolding Q² as evidence of predictive power. They describe fit, not out-of-sample prediction.
Skipping the Q²predict > 0 check before comparing errors.
Using RMSE when the prediction-error distribution is strongly non-symmetric; MAE is more appropriate there.
Comparing errors on every indicator rather than those of the key target construct.
Claiming predictive power when the PLS model does not beat the simple linear-model benchmark.

PLSpredict in OpenPLS

OpenPLS includes a PLSpredict panel that runs k-fold cross-validation and reports, per indicator, Q²predict and the RMSE, MAE and MAPE for both the PLS model and the linear-model benchmark, with a verdict for each. A separate CVPAT turns the comparison into a formal significance test. Free and runs in the browser.

Test the predictive power of your own model

Build your model in the visual editor and run k-fold PLSpredict in one click. Free, no install.

Open OpenPLS What is PLS-SEM?

Frequently asked questions

What is the difference between Q²predict and the Q² from blindfolding?: The blindfolding Q² is an in-sample measure of predictive relevance computed from the estimation data. Q²predict comes from out-of-sample k-fold cross-validation and reflects how well the model predicts cases it was not trained on. Only the latter is about genuine prediction.
Should I use RMSE or MAE in PLSpredict?: Use RMSE by default. Switch to MAE when the distribution of prediction errors is strongly non-symmetric, since RMSE weights large errors more heavily.
What counts as good predictive power?: Q²predict above 0 for the target indicators, and lower PLS errors than the linear-model benchmark. If PLS beats the benchmark on all indicators the model has high predictive power, on most it is medium, on a few it is low, and on none it has no predictive power.