From: Nick Holford <*n.holford*>

Date: Fri, 25 Jul 2008 09:22:03 +1200

Mahesh,

Thanks for this practical advice on how to do binning with S-Plus.

Here are some more comments on VPCs and binning:

Simulating at the same set of times for every subject is useful because

of the usual scatter of observed times around protocol times. VPCs based

only on observed times are possible but can be very hard to intepret

visually when there is a lot of between subject variability in

observation times. It can also be computationally difficult with large

data sets which are themselves simulated 1000 times. Note that the

simulated values themselves are not binned. There is no need to do

binning because you can always simulate enough times to get reliable

statistics at each simulation time.

Simulation times would normally be based on the nominal protocol time.

It can be helpful to simulate more frequently if the protocol was rather

sparse. Mats has pointed out that any simulations done at non-observed

times cannot give you any diagnostic information about whether the model

is predicting well at these non-observed times. The shape of the model

predictions can be helpful in understanding where your design was

deficient and what models might be identified from the data.

If you simulate at non-observed times and you have more than one

independent variable (e.g. time and weight) you will almost always want

to use the covariates from the original data set for each subject. I

choose the observed covariate set which is closest in time to the

simulation time. This is not realy binning but it uses the same

algorithm of associating observations at times close to the simulation

time with the simulation time. The alternative is to try and build a

parametric multivariate distribution for covariates to use for

simulation -- a procedure full of assumptions and high likelihood of

model misspecification.

The binning of the observations is frequently necessary in order to get

sufficient observations in the sample to compute reasonable statistics

(e.g. median, 5%ile, 95%ile). I bin the observations around the times

chosen for the simulations. The observed statistics are then plotted as

observation median and percentile bands ('the percentile VPC'). A VPC

which does not do this but only shows the scatter of observations

without showing these observation statistics is of only limited value

('the scatterplot VPC'). The combination of a percentile VPC and a

scatterplot VPC is much more useful.

Mats and I need to do some additional work on our PAGE tutorial

presentation before we post it on the PAGE website. Its not enough just

to put the slides on the web. We also want to add some explanatory notes.

Best wishes,

Nick

Samtani, Mahesh [PRDUS] wrote:

*> Dear Susan,
*

*> The cut function in S-plus is quite useful for binning. The cut function creates a category object by dividing continuous data into intervals. One can use the nominal (protocol) times as breakpoints and labels in the cut function. To read more about binning please see the abstract by Drs. Karlsson and Holford on VPC from this year's PAGE meeting.
*

*>
*

*> http://www.page-meeting.org/?abstract=1434
*

*>
*

*> Dr. Holford / Dr. Karlsson could you kindly post your presentation from this year's PAGE VPC tutorial on their webpage?
*

*>
*

*> Thanks...Mahesh
*

*>
*

*> -----Original Message-----
*

*> From: owner-nmusers *

*> [mailto:owner-nmusers *

*> Mohamad-Samer
*

*> Sent: Thursday, July 24, 2008 11:54 AM
*

*> To: Willavize, Susan A; Nick Holford; nmusers *

*> Subject: RE: FW: [NMusers] PPC
*

*>
*

*>
*

*>
*

*> Dear Susan,
*

*>
*

*> Binning is to have sufficient number of points to compute quantiles of interest.
*

*>
*

*> PSN. 2.2.5 has a predictive check utilities and very extensive options regarding binning and stratifying. The description document may be useful to understand more about binning.
*

*>
*

*> For uncertainties you may use the bootstrap distribution or the asymptotic distribution from a covariance step.
*

*>
*

*> Kind Regards,
*

*>
*

*> Samer
*

*>
*

*>
*

*>
*

*> -----Original Message-----
*

*> From: owner-nmusers *

*> Sent: Wed 7/23/2008 08:38
*

*> To: Nick Holford; nmusers *

*> Subject: RE: FW: [NMusers] PPC
*

*>
*

*> Hi Nick,
*

*>
*

*> I have been following this discussion and I think it is very helpful to
*

*> many of us. Can you please elaborate on that last part about binning?
*

*> What is that for? I must have missed something there.
*

*>
*

*> Thanks,
*

*> Susan
*

*>
*

*> Susan Willavize, Ph.D.
*

*> Global Pharmacometrics Group
*

*> 860-732-6428
*

*>
*

*> This e-mail is classified as Pfizer Confidential; it is confidential and
*

*> privileged.
*

*>
*

*>
*

*> -----Original Message-----
*

*> From: owner-nmusers *

*> On Behalf Of Nick Holford
*

*> Sent: Wednesday, July 23, 2008 6:32 AM
*

*> To: nmusers *

*> Subject: Re: FW: [NMusers] PPC
*

*>
*

*> Paul,
*

*>
*

*> The procedure you describe is a way of producing a posterior predictive
*

*> check but I don't know of any good examples of its use. A simpler way of
*

*>
*

*> doing a PPC samples the population parameter estimates from a
*

*> distribution centered on the final estimates with a variance-covariance
*

*>
*

*> based on the estimated standard errors and their correlation. VPCs are
*

*> not posterior predictive checks because they do not take account of the
*

*> posterior distribution of the parameter estimates (i.e. the final
*

*> estimates with their uncertainty). A VPC typically ignores the parameter
*

*>
*

*> uncertainty and uses what has been called the degenerate posterior
*

*> distribution (See Yano Y, Beal SL, Sheiner LB. Evaluating
*

*> pharmacokinetic/pharmacodynamic models using the posterior predictive
*

*> check. J Pharmacokinet Pharmacodyn. 2001;28(2):171-92 for terminology,
*

*> methods and examples).
*

*>
*

*> When I spoke of uncertainty I did not mean random variability (OMEGA and
*

*>
*

*> SIGMA). A VPC will simulate observations using the final THETA, OMEGA
*

*> and SIGMA estimates.
*

*>
*

*> You can calculate distribution statistics for your observations (such as
*

*>
*

*> median and 90% intervals) by combining the observations (one per
*

*> individual) at each time point to create an empirical distribution. The
*

*> statistics are then determined from this empirical distribution. In
*

*> order to get sufficient numbers of points (at least 10 is desirable) you
*

*>
*

*> may need to bin observations into time intervals e.g. 0-30 mins, 30-60
*

*> mins etc.
*

*>
*

*> Nick
*

*>
*

*> Paul Matthew Westwood wrote:
*

*>
*

*>> ________________________________________
*

*>> From: Paul Matthew Westwood
*

*>> Sent: 22 July 2008 13:20
*

*>> To: Nick Holford
*

*>> Subject: RE: [NMusers] PPC
*

*>>
*

*>> Nick,
*

*>>
*

*>> Thanks for your reply and apologies once again for another confusing
*

*>>
*

*> email. I think I am using VPC, which as I understand it is simulating n
*

*> datasets using the final parameter estimates gained from the final
*

*> model, and then taking the median and 90% confidence interval (for
*

*> example) for each simulated concentration and comparing these to the
*

*> real concentrations. Whereas, PPC is where you then run the final model
*

*> through the simulated datasets and compare selected statistics of these
*

*> new runs with the original. Is this correct? You mentioned including
*

*> uncertainty on the parameter estimates in the simulated datasets. Would
*

*> one usually not include uncertainty (fixing the error terms to zero) in
*

*> the simulated datasets? Doing this with mine obviously produced much
*

*> better concentrations with no negative values and no 'significant'
*

*> outliers. Another thing you mentioned is comparing the median of the
*

*> simulated concentrations with the median of the original dataset
*

*> concentrations, but as there is only one sample for any particular time
*

*> point would this indicate the unsuitability of VPC (and furthermore PPC)
*

*> for this model?
*

*>
*

*>> Thanks again,
*

*>> Paul.
*

*>> ________________________________________
*

*>> From: owner-nmusers *

*>>
*

*> Behalf Of Nick Holford [n.holford *

*>
*

*>> Sent: 22 July 2008 10:30
*

*>> To: nmusers *

*>> Subject: Re: [NMusers] PPC
*

*>>
*

*>> Paul,
*

*>>
*

*>> Its not clear to me if you did a VPC (visual predictive check) using
*

*>> just the final estimates of the parameters) or tried to do a posterior
*

*>> predictive check (PPC) including uncertainty on the parameter
*

*>>
*

*> estimates
*

*>
*

*>> in the simulation.
*

*>>
*

*>> I dont have any experience with PPC but I dont think its helpful for
*

*>> model evaluation. Its more of a tool for understanding uncertainties
*

*>>
*

*> of
*

*>
*

*>> predictions for future studies.
*

*>>
*

*>> I assume you dont have complications like informative dropout
*

*>>
*

*> processes
*

*>
*

*>> to complicate the simulation so if you did a VPC and the median of the
*

*>> predictions doesnt match the median of the observations then your
*

*>>
*

*> model
*

*>
*

*>> needs more work.
*

*>>
*

*>> Some negative concs are OK but 'impossibly high values' point to
*

*>> problems with your model.
*

*>>
*

*>> So I think you can safely say the VPC has worked very well -- it has
*

*>> told you that you need to think more about your model. You might find
*

*>> some ideas in these references:
*

*>>
*

*>> 1. Tod M, Jullien V, Pons G. Facilitation of drug evaluation in
*

*>> children by population methods and modelling. Clin Pharmacokinet.
*

*>> 2008;47(4):231-43.
*

*>> 2. Anderson BJ, Holford NH. Mechanism-Based Concepts of Size and
*

*>> Maturity in Pharmacokinetics. Annu Rev Pharmacol Toxicol.
*

*>>
*

*> 2008;48:303-32.
*

*>
*

*>> Nick
*

*>>
*

*>> Paul Matthew Westwood wrote:
*

*>>
*

*>>
*

*>>> Hello all,
*

*>>>
*

*>>> I wonder if someone can give me some tips on PPC.
*

*>>> I am working on a midazolam dataset with a pediatric population, and
*

*>>>
*

*> have decided to use PPC as a model validation technique. The dataset I
*

*> am modelling has up to 43 patients, at different ages, different
*

*> weights, different times of dosing and sampling, and different doses. I
*

*> simulated 100 datasets using NONMEM VI, fixing all parameters to the
*

*> final estimates from the model. The simulated datasets produced had a
*

*> large proportion of negative concentrations, and also a few impossibly
*

*> large concentration values. Also the median, 5th and 95th percentiles
*

*> were not very promising, and the resulting graphs not very clean.
*

*>
*

*>>> Firstly, can I use PPC with any degree of confidence with a dataset
*

*>>>
*

*> such as this, and if so, do I omit the negative concentration values
*

*> from the analysis?
*

*>
*

*>>> Thanks in advance for any help given.
*

*>>>
*

*>>> Paul Westwood,
*

*>>> PhD Student,
*

*>>> QUB,
*

*>>> Belfast.
*

*>>>
*

*>>>
*

*>>>
*

*>>>
*

*>>>
*

*>> --
*

*>> Nick Holford, Dept Pharmacology & Clinical Pharmacology
*

*>> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New
*

*>>
*

*> Zealand
*

*>
*

*>> n.holford *

*>> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
*

*>>
*

*>>
*

*>>
*

*>>
*

*>
*

*>
*

--

Nick Holford, Dept Pharmacology & Clinical Pharmacology

University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand

n.holford

http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Received on Thu Jul 24 2008 - 17:22:03 EDT

Date: Fri, 25 Jul 2008 09:22:03 +1200

Mahesh,

Thanks for this practical advice on how to do binning with S-Plus.

Here are some more comments on VPCs and binning:

Simulating at the same set of times for every subject is useful because

of the usual scatter of observed times around protocol times. VPCs based

only on observed times are possible but can be very hard to intepret

visually when there is a lot of between subject variability in

observation times. It can also be computationally difficult with large

data sets which are themselves simulated 1000 times. Note that the

simulated values themselves are not binned. There is no need to do

binning because you can always simulate enough times to get reliable

statistics at each simulation time.

Simulation times would normally be based on the nominal protocol time.

It can be helpful to simulate more frequently if the protocol was rather

sparse. Mats has pointed out that any simulations done at non-observed

times cannot give you any diagnostic information about whether the model

is predicting well at these non-observed times. The shape of the model

predictions can be helpful in understanding where your design was

deficient and what models might be identified from the data.

If you simulate at non-observed times and you have more than one

independent variable (e.g. time and weight) you will almost always want

to use the covariates from the original data set for each subject. I

choose the observed covariate set which is closest in time to the

simulation time. This is not realy binning but it uses the same

algorithm of associating observations at times close to the simulation

time with the simulation time. The alternative is to try and build a

parametric multivariate distribution for covariates to use for

simulation -- a procedure full of assumptions and high likelihood of

model misspecification.

The binning of the observations is frequently necessary in order to get

sufficient observations in the sample to compute reasonable statistics

(e.g. median, 5%ile, 95%ile). I bin the observations around the times

chosen for the simulations. The observed statistics are then plotted as

observation median and percentile bands ('the percentile VPC'). A VPC

which does not do this but only shows the scatter of observations

without showing these observation statistics is of only limited value

('the scatterplot VPC'). The combination of a percentile VPC and a

scatterplot VPC is much more useful.

Mats and I need to do some additional work on our PAGE tutorial

presentation before we post it on the PAGE website. Its not enough just

to put the slides on the web. We also want to add some explanatory notes.

Best wishes,

Nick

Samtani, Mahesh [PRDUS] wrote:

--

Nick Holford, Dept Pharmacology & Clinical Pharmacology

University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand

n.holford

http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Received on Thu Jul 24 2008 - 17:22:03 EDT