NONMEM Users Network Archive

Hosted by Cognigen

Re: FW: PPC

From: makamal
Date: Fri, 25 Jul 2008 17:26:36 -0400

Dear Dr. Holford,
Please correct me if I am wrong, however my understanding is that
asymptotic distribution implied by NONMEM's covariance step approaches
normality as the sample size gets larger or we have more data.
However, a non parametric bootstrap distribution may have poor
coverage with a small sample size as well, since it relies on sampling
subjects with repalcement in the data set. So both distributions have
problems when sample size is small (e.g. N<30). Therefore I would
think when N is large the wald based Confidence Intervals from NONMEM
are appropriate enough. It would be helpful to know the criteria when
generating a non parametric bootstrap distribution is really
advantageous.
Thanks, Mohamed
Quoting Nick Holford <n.holford

> Mahesh,
>
> Thanks for your further info on VPC and PPC. I agree that the
> bootstrap distribution of the parameters is probably better than the
> asymptotic normal distribution implied by NONMEM's covariance step
> results.
>
> I dont have your experience of comparing VPC and PPC so I hope you
> can find a way to publish these results which are similar to the
> limited exploration reported by Yano et al.
>
> VPC is not the perfect answer for model evaluation but it has some
> useful properties compared with the traditional methods (standard
> horizontal residual plots and diagonal residual plots (DV vs PRED
> and IPRED). I certainly havent seen any reason to use a PPC for
> model evaluation. It does however have a value (in theory) for
> predicting the uncertainty in outcome of a future trial.
>
> Nick
>
> Samtani, Mahesh [PRDUS] wrote:
>> Dear Nick,
>> Thank-you for teaching these important concepts. Could you and
>> others kindly comment on the following 2 aspects:
>>
>> a) The variance-covariance matrix based on the estimated standard
>> errors and their correlation will generate a multi-variate normal
>> distribution for the parameters. However, the posterior
>> distribution of parameters may not be normally dispersed. Wouldn't
>> it be better to use the bootstrap results as a source for getting
>> the uncertainty distribution. I have to admit that the bootstrap
>> method can be quite time-consuming. See one such example at:
>> http://www.page-meeting.org/pdf_assets/2373-MSamtani%20PAGE%20Poster%2020=
07.pdf
>> b) More importantly, after going through the PPC and VPC comparison
>> for several cases I always find that if the parameter estimates
>> have reasonable precision from the original NONMEM run then the PPC
>> and VPC results are essentially identical. This echoes an earlier
>> comment that most of the variation is explained by BSV and RV. Has
>> any one else experienced this behavior also and if so shouldn't VPC
>> be enough for model verification?
>>
>> Kindly advise...Mahesh
>>
>> -----Original Message-----
>> From: owner-nmusers
>> [mailto:owner-nmusers
>> Sent: Wednesday, July 23, 2008 8:38 AM
>> To: Nick Holford; nmusers
>> Subject: RE: FW: [NMusers] PPC
>>
>>
>> Hi Nick,
>>
>> I have been following this discussion and I think it is very helpful to
>> many of us. Can you please elaborate on that last part about binning?
>> What is that for? I must have missed something there.
>>
>> Thanks,
>> Susan Susan Willavize, Ph.D. Global Pharmacometrics Group
>> 860-732-6428
>>
>> This e-mail is classified as Pfizer Confidential; it is confidential and
>> privileged. -----Original Message-----
>> From: owner-nmusers
>> On Behalf Of Nick Holford
>> Sent: Wednesday, July 23, 2008 6:32 AM
>> To: nmusers
>> Subject: Re: FW: [NMusers] PPC
>>
>> Paul,
>>
>> The procedure you describe is a way of producing a posterior
>> predictive check but I don't know of any good examples of its use.
>> A simpler way of
>>
>> doing a PPC samples the population parameter estimates from a
>> distribution centered on the final estimates with a
>> variance-covariance
>>
>> based on the estimated standard errors and their correlation. VPCs
>> are not posterior predictive checks because they do not take
>> account of the posterior distribution of the parameter estimates
>> (i.e. the final estimates with their uncertainty). A VPC typically
>> ignores the parameter
>>
>> uncertainty and uses what has been called the degenerate posterior
>> distribution (See Yano Y, Beal SL, Sheiner LB. Evaluating
>> pharmacokinetic/pharmacodynamic models using the posterior
>> predictive check. J Pharmacokinet Pharmacodyn. 2001;28(2):171-92
>> for terminology, methods and examples).
>>
>> When I spoke of uncertainty I did not mean random variability (OMEGA and
>>
>> SIGMA). A VPC will simulate observations using the final THETA,
>> OMEGA and SIGMA estimates.
>>
>> You can calculate distribution statistics for your observations (such as
>>
>> median and 90% intervals) by combining the observations (one per
>> individual) at each time point to create an empirical distribution.
>> The statistics are then determined from this empirical
>> distribution. In order to get sufficient numbers of points (at
>> least 10 is desirable) you
>>
>> may need to bin observations into time intervals e.g. 0-30 mins,
>> 30-60 mins etc.
>>
>> Nick
>>
>> Paul Matthew Westwood wrote:
>>
>>> ________________________________________
>>> From: Paul Matthew Westwood
>>> Sent: 22 July 2008 13:20
>>> To: Nick Holford
>>> Subject: RE: [NMusers] PPC
>>>
>>> Nick,
>>>
>>> Thanks for your reply and apologies once again for another confusing
>>>
>> email. I think I am using VPC, which as I understand it is simulating n
>> datasets using the final parameter estimates gained from the final
>> model, and then taking the median and 90% confidence interval (for
>> example) for each simulated concentration and comparing these to the
>> real concentrations. Whereas, PPC is where you then run the final model
>> through the simulated datasets and compare selected statistics of these
>> new runs with the original. Is this correct? You mentioned including
>> uncertainty on the parameter estimates in the simulated datasets. Would
>> one usually not include uncertainty (fixing the error terms to zero) in
>> the simulated datasets? Doing this with mine obviously produced much
>> better concentrations with no negative values and no 'significant'
>> outliers. Another thing you mentioned is comparing the median of the
>> simulated concentrations with the median of the original dataset
>> concentrations, but as there is only one sample for any particular time
>> point would this indicate the unsuitability of VPC (and furthermore PPC)
>> for this model?
>>
>>> Thanks again,
>>> Paul.
>>> ________________________________________
>>> From: owner-nmusers
>>>
>> Behalf Of Nick Holford [n.holford
>>
>>> Sent: 22 July 2008 10:30
>>> To: nmusers
>>> Subject: Re: [NMusers] PPC
>>>
>>> Paul,
>>>
>>> Its not clear to me if you did a VPC (visual predictive check) using
>>> just the final estimates of the parameters) or tried to do a posterior
>>> predictive check (PPC) including uncertainty on the parameter
>>>
>> estimates
>>
>>> in the simulation.
>>>
>>> I dont have any experience with PPC but I dont think its helpful for
>>> model evaluation. Its more of a tool for understanding uncertainties
>>>
>> of
>>
>>> predictions for future studies.
>>>
>>> I assume you dont have complications like informative dropout
>>>
>> processes
>>
>>> to complicate the simulation so if you did a VPC and the median of the
>>> predictions doesnt match the median of the observations then your
>>>
>> model
>>
>>> needs more work.
>>>
>>> Some negative concs are OK but 'impossibly high values' point to
>>> problems with your model.
>>>
>>> So I think you can safely say the VPC has worked very well -- it has
>>> told you that you need to think more about your model. You might find
>>> some ideas in these references:
>>>
>>> 1. Tod M, Jullien V, Pons G. Facilitation of drug evaluation in
>>> children by population methods and modelling. Clin Pharmacokinet.
>>> 2008;47(4):231-43.
>>> 2. Anderson BJ, Holford NH. Mechanism-Based Concepts of Size and
>>> Maturity in Pharmacokinetics. Annu Rev Pharmacol Toxicol.
>>>
>> 2008;48:303-32.
>>
>>> Nick
>>>
>>> Paul Matthew Westwood wrote:
>>>
>>>> Hello all,
>>>>
>>>> I wonder if someone can give me some tips on PPC.
>>>> I am working on a midazolam dataset with a pediatric population, and
>>>>
>> have decided to use PPC as a model validation technique. The dataset I
>> am modelling has up to 43 patients, at different ages, different
>> weights, different times of dosing and sampling, and different doses. I
>> simulated 100 datasets using NONMEM VI, fixing all parameters to the
>> final estimates from the model. The simulated datasets produced had a
>> large proportion of negative concentrations, and also a few impossibly
>> large concentration values. Also the median, 5th and 95th percentiles
>> were not very promising, and the resulting graphs not very clean.
>>
>>>> Firstly, can I use PPC with any degree of confidence with a dataset
>>>>
>> such as this, and if so, do I omit the negative concentration values
>> from the analysis?
>>
>>>> Thanks in advance for any help given.
>>>>
>>>> Paul Westwood,
>>>> PhD Student,
>>>> QUB,
>>>> Belfast.
>>>>
>>>>
>>>>
>>>>
>>> --
>>> Nick Holford, Dept Pharmacology & Clinical Pharmacology
>>> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New
>>>
>> Zealand
>>
>>> n.holford
>>> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
>>>
>>>
>>>
>>
>>
>
> --
> Nick Holford, Dept Pharmacology & Clinical Pharmacology
> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zeala=
nd
> n.holford
> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
>
>
>
>
>



Mohamed A. Kamal, Pharm.D.
Ph.D. Candidate
Department of Pharmaceutical Sciences
University of Michigan
Received on Fri Jul 25 2008 - 17:26:36 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.