From: Samtani, Mahesh [PRDUS] <*MSamtani*>

Date: Wed, 23 Jul 2008 11:08:45 -0400

Dear Nick,

Thank-you for teaching these important concepts. Could you and others =

kindly comment on the following 2 aspects:

a) The variance-covariance matrix based on the estimated standard errors =

and their correlation will generate a multi-variate normal distribution =

for the parameters. However, the posterior distribution of parameters =

may not be normally dispersed. Wouldn't it be better to use the =

bootstrap results as a source for getting the uncertainty distribution. =

I have to admit that the bootstrap method can be quite time-consuming. =

See one such example at:

http://www.page-meeting.org/pdf_assets/2373-MSamtani%20PAGE%20Poster%2020=

07.pdf

b) More importantly, after going through the PPC and VPC comparison for =

several cases I always find that if the parameter estimates have =

reasonable precision from the original NONMEM run then the PPC and VPC =

results are essentially identical. This echoes an earlier comment that =

most of the variation is explained by BSV and RV. Has any one else =

experienced this behavior also and if so shouldn't VPC be enough for =

model verification?

Kindly advise...Mahesh

-----Original Message-----

From: owner-nmusers

[mailto:owner-nmusers

Sent: Wednesday, July 23, 2008 8:38 AM

To: Nick Holford; nmusers

Subject: RE: FW: [NMusers] PPC

Hi Nick,

I have been following this discussion and I think it is very helpful to

many of us. Can you please elaborate on that last part about binning?

What is that for? I must have missed something there.

Thanks,

Susan

Susan Willavize, Ph.D.

Global Pharmacometrics Group

860-732-6428

This e-mail is classified as Pfizer Confidential; it is confidential and

privileged.

-----Original Message-----

From: owner-nmusers

On Behalf Of Nick Holford

Sent: Wednesday, July 23, 2008 6:32 AM

To: nmusers

Subject: Re: FW: [NMusers] PPC

Paul,

The procedure you describe is a way of producing a posterior predictive

check but I don't know of any good examples of its use. A simpler way of

doing a PPC samples the population parameter estimates from a

distribution centered on the final estimates with a variance-covariance

based on the estimated standard errors and their correlation. VPCs are

not posterior predictive checks because they do not take account of the

posterior distribution of the parameter estimates (i.e. the final

estimates with their uncertainty). A VPC typically ignores the parameter

uncertainty and uses what has been called the degenerate posterior

distribution (See Yano Y, Beal SL, Sheiner LB. Evaluating

pharmacokinetic/pharmacodynamic models using the posterior predictive

check. J Pharmacokinet Pharmacodyn. 2001;28(2):171-92 for terminology,

methods and examples).

When I spoke of uncertainty I did not mean random variability (OMEGA and

SIGMA). A VPC will simulate observations using the final THETA, OMEGA

and SIGMA estimates.

You can calculate distribution statistics for your observations (such as

median and 90% intervals) by combining the observations (one per

individual) at each time point to create an empirical distribution. The

statistics are then determined from this empirical distribution. In

order to get sufficient numbers of points (at least 10 is desirable) you

may need to bin observations into time intervals e.g. 0-30 mins, 30-60

mins etc.

Nick

Paul Matthew Westwood wrote:

*> ________________________________________
*

*> From: Paul Matthew Westwood
*

*> Sent: 22 July 2008 13:20
*

*> To: Nick Holford
*

*> Subject: RE: [NMusers] PPC
*

*>
*

*> Nick,
*

*>
*

*> Thanks for your reply and apologies once again for another confusing
*

email. I think I am using VPC, which as I understand it is simulating n

datasets using the final parameter estimates gained from the final

model, and then taking the median and 90% confidence interval (for

example) for each simulated concentration and comparing these to the

real concentrations. Whereas, PPC is where you then run the final model

through the simulated datasets and compare selected statistics of these

new runs with the original. Is this correct? You mentioned including

uncertainty on the parameter estimates in the simulated datasets. Would

one usually not include uncertainty (fixing the error terms to zero) in

the simulated datasets? Doing this with mine obviously produced much

better concentrations with no negative values and no 'significant'

outliers. Another thing you mentioned is comparing the median of the

simulated concentrations with the median of the original dataset

concentrations, but as there is only one sample for any particular time

point would this indicate the unsuitability of VPC (and furthermore PPC)

for this model?

*>
*

*> Thanks again,
*

*> Paul.
*

*> ________________________________________
*

*> From: owner-nmusers *

Behalf Of Nick Holford [n.holford

*> Sent: 22 July 2008 10:30
*

*> To: nmusers *

*> Subject: Re: [NMusers] PPC
*

*>
*

*> Paul,
*

*>
*

*> Its not clear to me if you did a VPC (visual predictive check) using
*

*> just the final estimates of the parameters) or tried to do a posterior
*

*> predictive check (PPC) including uncertainty on the parameter
*

estimates

*> in the simulation.
*

*>
*

*> I dont have any experience with PPC but I dont think its helpful for
*

*> model evaluation. Its more of a tool for understanding uncertainties
*

of

*> predictions for future studies.
*

*>
*

*> I assume you dont have complications like informative dropout
*

processes

*> to complicate the simulation so if you did a VPC and the median of the
*

*> predictions doesnt match the median of the observations then your
*

model

*> needs more work.
*

*>
*

*> Some negative concs are OK but 'impossibly high values' point to
*

*> problems with your model.
*

*>
*

*> So I think you can safely say the VPC has worked very well -- it has
*

*> told you that you need to think more about your model. You might find
*

*> some ideas in these references:
*

*>
*

*> 1. Tod M, Jullien V, Pons G. Facilitation of drug evaluation in
*

*> children by population methods and modelling. Clin Pharmacokinet.
*

*> 2008;47(4):231-43.
*

*> 2. Anderson BJ, Holford NH. Mechanism-Based Concepts of Size and
*

*> Maturity in Pharmacokinetics. Annu Rev Pharmacol Toxicol.
*

2008;48:303-32.

*>
*

*> Nick
*

*>
*

*> Paul Matthew Westwood wrote:
*

*>
*

*>> Hello all,
*

*>>
*

*>> I wonder if someone can give me some tips on PPC.
*

*>> I am working on a midazolam dataset with a pediatric population, and
*

have decided to use PPC as a model validation technique. The dataset I

am modelling has up to 43 patients, at different ages, different

weights, different times of dosing and sampling, and different doses. I

simulated 100 datasets using NONMEM VI, fixing all parameters to the

final estimates from the model. The simulated datasets produced had a

large proportion of negative concentrations, and also a few impossibly

large concentration values. Also the median, 5th and 95th percentiles

were not very promising, and the resulting graphs not very clean.

*>> Firstly, can I use PPC with any degree of confidence with a dataset
*

such as this, and if so, do I omit the negative concentration values

from the analysis?

*>>
*

*>> Thanks in advance for any help given.
*

*>>
*

*>> Paul Westwood,
*

*>> PhD Student,
*

*>> QUB,
*

*>> Belfast.
*

*>>
*

*>>
*

*>>
*

*>>
*

*>
*

*> --
*

*> Nick Holford, Dept Pharmacology & Clinical Pharmacology
*

*> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New
*

Zealand

*> n.holford *

*> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
*

*>
*

*>
*

*>
*

--

Nick Holford, Dept Pharmacology & Clinical Pharmacology

University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New

Zealand

n.holford

http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Received on Wed Jul 23 2008 - 11:08:45 EDT

Date: Wed, 23 Jul 2008 11:08:45 -0400

Dear Nick,

Thank-you for teaching these important concepts. Could you and others =

kindly comment on the following 2 aspects:

a) The variance-covariance matrix based on the estimated standard errors =

and their correlation will generate a multi-variate normal distribution =

for the parameters. However, the posterior distribution of parameters =

may not be normally dispersed. Wouldn't it be better to use the =

bootstrap results as a source for getting the uncertainty distribution. =

I have to admit that the bootstrap method can be quite time-consuming. =

See one such example at:

http://www.page-meeting.org/pdf_assets/2373-MSamtani%20PAGE%20Poster%2020=

07.pdf

b) More importantly, after going through the PPC and VPC comparison for =

several cases I always find that if the parameter estimates have =

reasonable precision from the original NONMEM run then the PPC and VPC =

results are essentially identical. This echoes an earlier comment that =

most of the variation is explained by BSV and RV. Has any one else =

experienced this behavior also and if so shouldn't VPC be enough for =

model verification?

Kindly advise...Mahesh

-----Original Message-----

From: owner-nmusers

[mailto:owner-nmusers

Sent: Wednesday, July 23, 2008 8:38 AM

To: Nick Holford; nmusers

Subject: RE: FW: [NMusers] PPC

Hi Nick,

I have been following this discussion and I think it is very helpful to

many of us. Can you please elaborate on that last part about binning?

What is that for? I must have missed something there.

Thanks,

Susan

Susan Willavize, Ph.D.

Global Pharmacometrics Group

860-732-6428

This e-mail is classified as Pfizer Confidential; it is confidential and

privileged.

-----Original Message-----

From: owner-nmusers

On Behalf Of Nick Holford

Sent: Wednesday, July 23, 2008 6:32 AM

To: nmusers

Subject: Re: FW: [NMusers] PPC

Paul,

The procedure you describe is a way of producing a posterior predictive

check but I don't know of any good examples of its use. A simpler way of

doing a PPC samples the population parameter estimates from a

distribution centered on the final estimates with a variance-covariance

based on the estimated standard errors and their correlation. VPCs are

not posterior predictive checks because they do not take account of the

posterior distribution of the parameter estimates (i.e. the final

estimates with their uncertainty). A VPC typically ignores the parameter

uncertainty and uses what has been called the degenerate posterior

distribution (See Yano Y, Beal SL, Sheiner LB. Evaluating

pharmacokinetic/pharmacodynamic models using the posterior predictive

check. J Pharmacokinet Pharmacodyn. 2001;28(2):171-92 for terminology,

methods and examples).

When I spoke of uncertainty I did not mean random variability (OMEGA and

SIGMA). A VPC will simulate observations using the final THETA, OMEGA

and SIGMA estimates.

You can calculate distribution statistics for your observations (such as

median and 90% intervals) by combining the observations (one per

individual) at each time point to create an empirical distribution. The

statistics are then determined from this empirical distribution. In

order to get sufficient numbers of points (at least 10 is desirable) you

may need to bin observations into time intervals e.g. 0-30 mins, 30-60

mins etc.

Nick

Paul Matthew Westwood wrote:

email. I think I am using VPC, which as I understand it is simulating n

datasets using the final parameter estimates gained from the final

model, and then taking the median and 90% confidence interval (for

example) for each simulated concentration and comparing these to the

real concentrations. Whereas, PPC is where you then run the final model

through the simulated datasets and compare selected statistics of these

new runs with the original. Is this correct? You mentioned including

uncertainty on the parameter estimates in the simulated datasets. Would

one usually not include uncertainty (fixing the error terms to zero) in

the simulated datasets? Doing this with mine obviously produced much

better concentrations with no negative values and no 'significant'

outliers. Another thing you mentioned is comparing the median of the

simulated concentrations with the median of the original dataset

concentrations, but as there is only one sample for any particular time

point would this indicate the unsuitability of VPC (and furthermore PPC)

for this model?

Behalf Of Nick Holford [n.holford

estimates

of

processes

model

2008;48:303-32.

have decided to use PPC as a model validation technique. The dataset I

am modelling has up to 43 patients, at different ages, different

weights, different times of dosing and sampling, and different doses. I

simulated 100 datasets using NONMEM VI, fixing all parameters to the

final estimates from the model. The simulated datasets produced had a

large proportion of negative concentrations, and also a few impossibly

large concentration values. Also the median, 5th and 95th percentiles

were not very promising, and the resulting graphs not very clean.

such as this, and if so, do I omit the negative concentration values

from the analysis?

Zealand

--

Nick Holford, Dept Pharmacology & Clinical Pharmacology

University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New

Zealand

n.holford

http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Received on Wed Jul 23 2008 - 11:08:45 EDT