run2$validate(limits = c(0, 3))13 Model Validation
Tip
It is essential that you understand simulations first.
Also, make sure you have run the tutorial setup code in your R session before copying, pasting and running example code here.
13.1 Introduction
The fit of a model to the data generates various metrics of goodness-of-fit, such as the objective function value (OFV), Akaike information criterion (AIC), Bayesian information criterion (BIC), and log-likelihood. However, these metrics do not provide a complete picture of how well the model describes the data.
Model validation techniques are used to assess the predictive performance of a model and its ability to generalize to new data. These techniques help identify potential issues with the model, such as overfitting or underfitting, and provide insights into areas where the model may need improvement.
Broadly, two major types of model validation techniques exist: internal and external. External validation involves testing the model on an independent dataset that was not used during model development. Internal validation, on the other hand, uses the original dataset to assess the model’s performance through various resampling methods or simulation-based approaches.
First we consider simulation-based, internal methods of model validation in Pmetrics.
13.2 Simulation-based Internal Validation
Internal methods of validating include visual predictive check (VPC) (Holford 2005), prediction-corrected visual predictive check (pcVPC) (Bergstrand et al. 2011), numerical predictive check, and normalized prediction distribution errors (NPDE) (Comets, Brendel, and Mentré 2008). These are all implemented in the validate() method of a PM_result object.
The common idea is that we simulate many datasets from each subject the model building population and compare the distributions of the observed data and the simulated data. If the model is a good fit, the observed data should fall within the range of simulated data, similar in central tendency and dispersion.
When executing the following code, choose wt as the covariate to bin. Accept all default bin sizes.
Plot the default visual predictive check; ?plot.PM_valid for help.
run2$valid$plot()Generate a prediction-corrected visual predictive check.
run2$valid$plot(type = "pcvpc")Create an npde plot.
run2$valid$plot(type = "npde")Here is another way to generate a visual predicive check…
npc_2 <- run2$valid$simdata$plot(obs = run2$op, log = FALSE, binSize = 0.5)The plot can appear more jagged with smaller bin sizes because different subjects have different doses, covariates, and observation times, which are all combined in one simulation. Collapsing simulation times within 1 hour bins (binSize = 1) smooths the plot, but can change the P-values in the numerical predictive check below.
npc_2…and here is a numerical predictive check. P-values are binomial test of proportion of observations less than the respective quantile.
13.3 Citations
Bergstrand, Martin, Andrew C Hooker, Johan E Wallin, and Mats O Karlsson. 2011. “Prediction-Corrected Visual Predictive Checks for Diagnosing Nonlinear Mixed-Effects Models.” The AAPS Journal 13 (2): 143–51. https://doi.org/10.1208/s12248-011-9255-z.
Comets, Emmanuelle, Karl Brendel, and France Mentré. 2008. “Computing Normalised Prediction Distribution Errors to Evaluate Nonlinear Mixed-Effect Models: The Npde Add-on Package for R.” Computer Methods and Programs in Biomedicine 90 (2): 154–66. https://doi.org/10.1016/j.cmpb.2007.12.002.
Holford, N. 2005. “VPC, the Visual Predictive Check - Superiority to Standard Diagnostic (Rorschach) Plots.” In 14th Population Approach Group Europe (PAGE), Abstract 738. Pamplona, Spain.