Copied from TuringLang/Turing.jl#2785 (comment):
In Bayesian modelling workflows (e.g. Gelman's classic workflow or Hess et al. from my own field - I have a more complete list of these kinds of tutorial articles if interested) a set of standard methods exist for validating models. In my field, these are considered best practices, and are sometimes required for publication; they must mostly be handcoded by researchers, however, and are so not always done. It would be very useful to have functionalities for them in Turing.jl, especially given that the methods are not usually specific to any given model. In addition, they seem to be relatively easily implementable, since the required functionality alread exists in Turing.
Often, visualisations and choices of how to summarise the data will be specific to a given model, so it should be possible to make custom methods for these.
In general, the methods below require 1) simulating data with specified parameter values and experimental design features, and 2) fitting models to the simulated data from the first step. It is also sometimes necessary to be able to make predictions for a new dataset.
For some of these methods, including custom functions might not be necessary, or would at least be a very thin wrapper, for others it would be more involved. For some, it might also not be functionalities that belong in Turing. I'll list all the methods that I am familiar with - there might be more.
I'd be very happy to help implement some of these!
The methods include:
Parameter recovery
Simulate data for a range of different known parameter settings. Estimate parameters from simulated data. See if estimated and known parameters match. Tests whether the model+inference method can properly reconstruct parameter values of the gneerative process.
Can be implemented using fix() to set parameter values and predict() to generate data. Then condition the model on the generated data. Can be parallelised across parameter settings. Will be computationally heavy, so storing temporary results and allowing for resuming later would be useful.
Arguments: Turing model; specifications of which variables are parameters and which are data; full set of parameter settings to be used; settings for parallelisation and temporary storage; settings for inference method (e.g., arguments to sample() or vi()), number of repetitions.
Output: Full set of samples across parameters alongside the known values used for generating behaviour, across all parameter settings, across repetitions.
Model recovery
Simulate data with a range of different models. Use model comparison to select models from the generated data. See if estimated and known models match. Tests whether the model comparison method can properly reconstruct which model is used in the data generating process. Requires all models to generate the same kind of data.
Would optimally be combined with parameter recovery within each model (since some parameter settings will make models more or less redcoverable) although this quickly becomes computationally heavy. Maybe a shared interface could be set up.
Arguments: Set of Turing mdoels to compare; specifications of which variables are parameters (for each model separately) and which are data (same across models); full set of parameter settings to be used (for each mdoels separately); settings for parallelisation and temporary storage; settings for inference method (e.g., arguments to sample() or vi()); settings for model comparison method (e.g., loo or other metrics); number of repetitions.
Output: Full set of posteriors over models for each known model and each combination of parameter settings across models (and also parameter estimates, as above), across repetitions.
Prior predictive checks
Simulate data using parameter values sampled from the prior. Visualise or summarise the data in order to check that the prior+model produces a priori plausible data (such as reaction times always above 0 and below some upper bound empirically found in the literature). Features relevant for plausibility will be model-specific.
Arguments: Turing model; specifications of which variables are parameters and which data, number of repetitions.
Output: Container with generated data across repetitions (which can be used for custom visualisations and summarisations).
Posterior predictive checks
Simulate data using parameter values sampled from the posterior. Visualise or summarise the simulated data alongside the real empirical data in order to check whether the model can capture the data well (especially capture important qualitative patterns in the data that a single accuracy measure would miss). Features of data to compare will be model-specific. Can also be used to make predictions for a new dataset, to see if the model generalises.
Arguments: Turing model; specifications of which variables are parameters and which data, number of repetitions; optionally a new dataset or set of specifications to make predictions for.
Output: Container with generated data across repetitions alongside real data (which can be used for custom visualisations and summarisations).
Simulation-based calibration
(NB. I haven't done this myself, so worth confirming I have understood correctly).
Simulate data with parameter values sampled from the prior. Fit the model to the simulated data. Check that the averaged posteriors are approximately equal to the prior (there are varius ways to visualise / quantify this). If the model inversion method is working corerctly, this should be the case. Note that this is equivalent to parameter recovery when only using parameter settings drawn from the prior, although what is done with the results is different.
Arguments: Turing model; settings for parallelisation and temporary storage; settings for inference method (e.g., arguments to sample() or vi()); number of samples to draw from the prior.
Output: Full set of samples across sampled prior values. Standard evaluation methods should be model-agnostic here, so including these would be useful.
Sensitivity analysis
Fit different variants of a model to real data, varying modelling chocies where there are multiple equally valid options. Tests whether arbitrary choices shape results. Most commonly used to test whether the choice of prior determines the result, but there are also often features of the likelihood that are more or less arbitrary. Test-retest checks fall in this category too, where the model is fit to multiple datasets that should give similar results.
Arguments: Turing model; set of different specifications to use (e.g., different priors) (which I suppose the model must be able to take as arguments); settings for inference method (e.g., arguments to sample() or vi()).
Output: Posterior parameter estimates for each set of specifications.
Update checks
Compare (visually or quantitatively) whether the posterior is different from the prior, to check whether the data has been informative for the estimates or not.
Design analysis
A generalisation of frequentist power analysis or Bayesian precision analysis. Simulate data from the model given different specifications (such as number of participants, or other controllable aspects of the experiment, such as the experimental intervention). Fit the model to the simulated data (essentially do a parameter recovery) under each specification. Test whether the posteriors become sufficiently precise under different designs (in order to find the minimum number of participants needed to be able to expect precise estimates, or which of a set of possible designs is optimal). E.g., Bayesian optimal experimental design. Can also include model recovery for designing experiments to best adjudicate between possible models.
Arguments: (set of) Turing model(s); set of design settings (which I suppose the model would have to take as arguments or similar); set of parameter settings to simulate data under; settings for parallelisation and temporary storage; settings for inference method (e.g., arguments to sample() or vi()); settings for model comparison method (e.g., loo or other metrics); number of repetitions.
Out-of-sample predictions
Generate predictions for a new dataset from the same or a sufficiently similar experimental design (to do out-of-sample posterior predictive checks); or generate for a set of settings where the new data is not known (such as a new participant in an experiment where predictors are known but not the outcome measure).
Copied from TuringLang/Turing.jl#2785 (comment):
In Bayesian modelling workflows (e.g. Gelman's classic workflow or Hess et al. from my own field - I have a more complete list of these kinds of tutorial articles if interested) a set of standard methods exist for validating models. In my field, these are considered best practices, and are sometimes required for publication; they must mostly be handcoded by researchers, however, and are so not always done. It would be very useful to have functionalities for them in Turing.jl, especially given that the methods are not usually specific to any given model. In addition, they seem to be relatively easily implementable, since the required functionality alread exists in Turing.
Often, visualisations and choices of how to summarise the data will be specific to a given model, so it should be possible to make custom methods for these.
In general, the methods below require 1) simulating data with specified parameter values and experimental design features, and 2) fitting models to the simulated data from the first step. It is also sometimes necessary to be able to make predictions for a new dataset.
For some of these methods, including custom functions might not be necessary, or would at least be a very thin wrapper, for others it would be more involved. For some, it might also not be functionalities that belong in Turing. I'll list all the methods that I am familiar with - there might be more.
I'd be very happy to help implement some of these!
The methods include:
Parameter recovery
Simulate data for a range of different known parameter settings. Estimate parameters from simulated data. See if estimated and known parameters match. Tests whether the model+inference method can properly reconstruct parameter values of the gneerative process.
Can be implemented using fix() to set parameter values and predict() to generate data. Then condition the model on the generated data. Can be parallelised across parameter settings. Will be computationally heavy, so storing temporary results and allowing for resuming later would be useful.
Arguments: Turing model; specifications of which variables are parameters and which are data; full set of parameter settings to be used; settings for parallelisation and temporary storage; settings for inference method (e.g., arguments to sample() or vi()), number of repetitions.
Output: Full set of samples across parameters alongside the known values used for generating behaviour, across all parameter settings, across repetitions.
Model recovery
Simulate data with a range of different models. Use model comparison to select models from the generated data. See if estimated and known models match. Tests whether the model comparison method can properly reconstruct which model is used in the data generating process. Requires all models to generate the same kind of data.
Would optimally be combined with parameter recovery within each model (since some parameter settings will make models more or less redcoverable) although this quickly becomes computationally heavy. Maybe a shared interface could be set up.
Arguments: Set of Turing mdoels to compare; specifications of which variables are parameters (for each model separately) and which are data (same across models); full set of parameter settings to be used (for each mdoels separately); settings for parallelisation and temporary storage; settings for inference method (e.g., arguments to sample() or vi()); settings for model comparison method (e.g., loo or other metrics); number of repetitions.
Output: Full set of posteriors over models for each known model and each combination of parameter settings across models (and also parameter estimates, as above), across repetitions.
Prior predictive checks
Simulate data using parameter values sampled from the prior. Visualise or summarise the data in order to check that the prior+model produces a priori plausible data (such as reaction times always above 0 and below some upper bound empirically found in the literature). Features relevant for plausibility will be model-specific.
Arguments: Turing model; specifications of which variables are parameters and which data, number of repetitions.
Output: Container with generated data across repetitions (which can be used for custom visualisations and summarisations).
Posterior predictive checks
Simulate data using parameter values sampled from the posterior. Visualise or summarise the simulated data alongside the real empirical data in order to check whether the model can capture the data well (especially capture important qualitative patterns in the data that a single accuracy measure would miss). Features of data to compare will be model-specific. Can also be used to make predictions for a new dataset, to see if the model generalises.
Arguments: Turing model; specifications of which variables are parameters and which data, number of repetitions; optionally a new dataset or set of specifications to make predictions for.
Output: Container with generated data across repetitions alongside real data (which can be used for custom visualisations and summarisations).
Simulation-based calibration
(NB. I haven't done this myself, so worth confirming I have understood correctly).
Simulate data with parameter values sampled from the prior. Fit the model to the simulated data. Check that the averaged posteriors are approximately equal to the prior (there are varius ways to visualise / quantify this). If the model inversion method is working corerctly, this should be the case. Note that this is equivalent to parameter recovery when only using parameter settings drawn from the prior, although what is done with the results is different.
Arguments: Turing model; settings for parallelisation and temporary storage; settings for inference method (e.g., arguments to sample() or vi()); number of samples to draw from the prior.
Output: Full set of samples across sampled prior values. Standard evaluation methods should be model-agnostic here, so including these would be useful.
Sensitivity analysis
Fit different variants of a model to real data, varying modelling chocies where there are multiple equally valid options. Tests whether arbitrary choices shape results. Most commonly used to test whether the choice of prior determines the result, but there are also often features of the likelihood that are more or less arbitrary. Test-retest checks fall in this category too, where the model is fit to multiple datasets that should give similar results.
Arguments: Turing model; set of different specifications to use (e.g., different priors) (which I suppose the model must be able to take as arguments); settings for inference method (e.g., arguments to sample() or vi()).
Output: Posterior parameter estimates for each set of specifications.
Update checks
Compare (visually or quantitatively) whether the posterior is different from the prior, to check whether the data has been informative for the estimates or not.
Design analysis
A generalisation of frequentist power analysis or Bayesian precision analysis. Simulate data from the model given different specifications (such as number of participants, or other controllable aspects of the experiment, such as the experimental intervention). Fit the model to the simulated data (essentially do a parameter recovery) under each specification. Test whether the posteriors become sufficiently precise under different designs (in order to find the minimum number of participants needed to be able to expect precise estimates, or which of a set of possible designs is optimal). E.g., Bayesian optimal experimental design. Can also include model recovery for designing experiments to best adjudicate between possible models.
Arguments: (set of) Turing model(s); set of design settings (which I suppose the model would have to take as arguments or similar); set of parameter settings to simulate data under; settings for parallelisation and temporary storage; settings for inference method (e.g., arguments to sample() or vi()); settings for model comparison method (e.g., loo or other metrics); number of repetitions.
Out-of-sample predictions
Generate predictions for a new dataset from the same or a sufficiently similar experimental design (to do out-of-sample posterior predictive checks); or generate for a set of settings where the new data is not known (such as a new participant in an experiment where predictors are known but not the outcome measure).