NONMEM Users Network Archive

Hosted by Cognigen


From: makamal
Date: Fri, 25 Jul 2008 17:26:36 -0400

Dear Dr. Holford,
Please correct me if I am wrong, however my understanding is that
asymptotic distribution implied by NONMEM's covariance step approaches
normality as the sample size gets larger or we have more data.
However, a non parametric bootstrap distribution may have poor
coverage with a small sample size as well, since it relies on sampling
subjects with repalcement in the data set. So both distributions have
problems when sample size is small (e.g. N<30). Therefore I would
think when N is large the wald based Confidence Intervals from NONMEM
are appropriate enough. It would be helpful to know the criteria when
generating a non parametric bootstrap distribution is really
Thanks, Mohamed
Quoting Nick Holford <n.holford

> Mahesh,
> Thanks for your further info on VPC and PPC. I agree that the
> bootstrap distribution of the parameters is probably better than the
> asymptotic normal distribution implied by NONMEM's covariance step
> results.
> I dont have your experience of comparing VPC and PPC so I hope you
> can find a way to publish these results which are similar to the
> limited exploration reported by Yano et al.
> VPC is not the perfect answer for model evaluation but it has some
> useful properties compared with the traditional methods (standard
> horizontal residual plots and diagonal residual plots (DV vs PRED
> and IPRED). I certainly havent seen any reason to use a PPC for
> model evaluation. It does however have a value (in theory) for
> predicting the uncertainty in outcome of a future trial.
> Nick
> Samtani, Mahesh [PRDUS] wrote:
>> Dear Nick,
>> Thank-you for teaching these important concepts. Could you and
>> others kindly comment on the following 2 aspects:
>> a) The variance-covariance matrix based on the estimated standard
>> errors and their correlation will generate a multi-variate normal
>> distribution for the parameters. However, the posterior
>> distribution of parameters may not be normally dispersed. Wouldn't
>> it be better to use the bootstrap results as a source for getting
>> the uncertainty distribution. I have to admit that the bootstrap
>> method can be quite time-consuming. See one such example at:
>> b) More importantly, after going through the PPC and VPC comparison
>> for several cases I always find that if the parameter estimates
>> have reasonable precision from the original NONMEM run then the PPC
>> and VPC results are essentially identical. This echoes an earlier
>> comment that most of the variation is explained by BSV and RV. Has
>> any one else experienced this behavior also and if so shouldn't VPC
>> be enough for model verification?
>> Kindly advise...Mahesh
>> -----Original Message-----
>> From: owner-nmusers
>> [mailto:owner-nmusers
>> Sent: Wednesday, July 23, 2008 8:38 AM
>> To: Nick Holford; nmusers
>> Subject: RE: FW: [NMusers] PPC
>> Hi Nick,
>> I have been following this discussion and I think it is very helpful to
>> many of us. Can you please elaborate on that last part about binning?
>> What is that for? I must have missed something there.
>> Thanks,
>> Susan Susan Willavize, Ph.D. Global Pharmacometrics Group
>> 860-732-6428
>> This e-mail is classified as Pfizer Confidential; it is confidential and
>> privileged. -----Original Message-----
>> From: owner-nmusers
>> On Behalf Of Nick Holford
>> Sent: Wednesday, July 23, 2008 6:32 AM
>> To: nmusers
>> Subject: Re: FW: [NMusers] PPC
>> Paul,
>> The procedure you describe is a way of producing a posterior
>> predictive check but I don't know of any good examples of its use.
>> A simpler way of
>> doing a PPC samples the population parameter estimates from a
>> distribution centered on the final estimates with a
>> variance-covariance
>> based on the estimated standard errors and their correlation. VPCs
>> are not posterior predictive checks because they do not take
>> account of the posterior distribution of the parameter estimates
>> (i.e. the final estimates with their uncertainty). A VPC typically
>> ignores the parameter
>> uncertainty and uses what has been called the degenerate posterior
>> distribution (See Yano Y, Beal SL, Sheiner LB. Evaluating
>> pharmacokinetic/pharmacodynamic models using the posterior
>> predictive check. J Pharmacokinet Pharmacodyn. 2001;28(2):171-92
>> for terminology, methods and examples).
>> When I spoke of uncertainty I did not mean random variability (OMEGA and
>> SIGMA). A VPC will simulate observations using the final THETA,
>> OMEGA and SIGMA estimates.
>> You can calculate distribution statistics for your observations (such as
>> median and 90% intervals) by combining the observations (one per
>> individual) at each time point to create an empirical distribution.
>> The statistics are then determined from this empirical
>> distribution. In order to get sufficient numbers of points (at
>> least 10 is desirable) you
>> may need to bin observations into time intervals e.g. 0-30 mins,
>> 30-60 mins etc.
>> Nick
>> Paul Matthew Westwood wrote:
>>> ________________________________________
>>> From: Paul Matthew Westwood
>>> Sent: 22 July 2008 13:20
>>> To: Nick Holford
>>> Subject: RE: [NMusers] PPC
>>> Nick,
>>> Thanks for your reply and apologies once again for another confusing
>> email. I think I am using VPC, which as I understand it is simulating n
>> datasets using the final parameter estimates gained from the final
>> model, and then taking the median and 90% confidence interval (for
>> example) for each simulated concentration and comparing these to the
>> real concentrations. Whereas, PPC is where you then run the final model
>> through the simulated datasets and compare selected statistics of these
>> new runs with the original. Is this correct? You mentioned including
>> uncertainty on the parameter estimates in the simulated datasets. Would
>> one usually not include uncertainty (fixing the error terms to zero) in
>> the simulated datasets? Doing this with mine obviously produced much
>> better concentrations with no negative values and no 'significant'
>> outliers. Another thing you mentioned is comparing the median of the
>> simulated concentrations with the median of the original dataset
>> concentrations, but as there is only one sample for any particular time
>> point would this indicate the unsuitability of VPC (and furthermore PPC)
>> for this model?
>>> Thanks again,
>>> Paul.
>>> ________________________________________
>>> From: owner-nmusers
>> Behalf Of Nick Holford [n.holford
>>> Sent: 22 July 2008 10:30
>>> To: nmusers
>>> Subject: Re: [NMusers] PPC
>>> Paul,
>>> Its not clear to me if you did a VPC (visual predictive check) using
>>> just the final estimates of the parameters) or tried to do a posterior
>>> predictive check (PPC) including uncertainty on the parameter
>> estimates
>>> in the simulation.
>>> I dont have any experience with PPC but I dont think its helpful for
>>> model evaluation. Its more of a tool for understanding uncertainties
>> of
>>> predictions for future studies.
>>> I assume you dont have complications like informative dropout
>> processes
>>> to complicate the simulation so if you did a VPC and the median of the
>>> predictions doesnt match the median of the observations then your
>> model
>>> needs more work.
>>> Some negative concs are OK but 'impossibly high values' point to
>>> problems with your model.
>>> So I think you can safely say the VPC has worked very well -- it has
>>> told you that you need to think more about your model. You might find
>>> some ideas in these references:
>>> 1. Tod M, Jullien V, Pons G. Facilitation of drug evaluation in
>>> children by population methods and modelling. Clin Pharmacokinet.
>>> 2008;47(4):231-43.
>>> 2. Anderson BJ, Holford NH. Mechanism-Based Concepts of Size and
>>> Maturity in Pharmacokinetics. Annu Rev Pharmacol Toxicol.
>> 2008;48:303-32.
>>> Nick
>>> Paul Matthew Westwood wrote:
>>>> Hello all,
>>>> I wonder if someone can give me some tips on PPC.
>>>> I am working on a midazolam dataset with a pediatric population, and
>> have decided to use PPC as a model validation technique. The dataset I
>> am modelling has up to 43 patients, at different ages, different
>> weights, different times of dosing and sampling, and different doses. I
>> simulated 100 datasets using NONMEM VI, fixing all parameters to the
>> final estimates from the model. The simulated datasets produced had a
>> large proportion of negative concentrations, and also a few impossibly
>> large concentration values. Also the median, 5th and 95th percentiles
>> were not very promising, and the resulting graphs not very clean.
>>>> Firstly, can I use PPC with any degree of confidence with a dataset
>> such as this, and if so, do I omit the negative concentration values
>> from the analysis?
>>>> Thanks in advance for any help given.
>>>> Paul Westwood,
>>>> PhD Student,
>>>> QUB,
>>>> Belfast.
>>> --
>>> Nick Holford, Dept Pharmacology & Clinical Pharmacology
>>> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New
>> Zealand
>>> n.holford
> --
> Nick Holford, Dept Pharmacology & Clinical Pharmacology
> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zeala=
> n.holford

Mohamed A. Kamal, Pharm.D.
Ph.D. Candidate
Department of Pharmaceutical Sciences
University of Michigan
Received on Fri Jul 25 2008 - 17:26:36 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to:

Once subscribed, you may contribute to the discussion by emailing: