From: Nick Holford <*n.holford*>

Date: Sat, 26 Jul 2008 17:23:04 +1200

Mohamed,

When the number of subjects is small then any confidence interval is

going to be wide and probably no-one is really interested in it. With

studies more suitable for population analysis (at least 25 subjects and

preferably over 100 if you want to look for covariate effects) then the

CIs may be more interesting.

With linear models or parameters which are nearly linear in non-linear

models then I would expect quite good agreement between CIs obtained by

bootstrapping and by using NONMEM SEs. But the models get interesting

when one tries to estimate non-linear parameters e.g. EC50 in an Emax

model. In that case the CIs will often be asymmetrical and the normal

distribution assumption used to compute SEs from CIs will be wrong.

Leonid does not discuss the issue of assymetry of CIs in his poster --

but when I look at Figure 3 I see evidence for disagreement between

bootstrap and NONMEM SE based CIs. The scatter of bootstrap points

relative to the solid line shows an excess of bootstrap upper CI values

above the SE prediction. For the lower CI prediction there also seem to

be more bootstrap values above the SE predictions. Its hard to be sure

that these upper and lower bootstrap predictions belong to the same

parameters but if so this would be evidence for asymmetry of the

bootstrap CI. This is exactly what one would expect because the SE

method has to assume symmetrical CIs yet the bootstrap estimate is not

restricted in this way.

I think this poster is a nice example of why correlation coefficients

are a very poor way to compare predictions (as pointed out by Sheiner

and Beal in their classic paper Sheiner LB, Beal SL. Some Suggestions

for Measuring Predictive Performance. J Pharmacokinet Biopharm.

1981;9(4):503-12.). A better way would be to compute the prediction

error for the absolute larger CI arm and smaller CI arm obtained by

bootstrapping to the symmetrial CI from the SE. If bootstraps CIs are

indeed asymmetrical then there would be a difference shown by the mean

prediction error ('bias'). Note that I use absolute value larger and

smaller CI arm to refer to the larger or smaller part of the CI that is

constructed around zero. I dont mean the upper and lower parts of the CI

interval.

Best wishes,

Nick

Leonid Gibiansky wrote:

*> I cannot make any general statements but here is the summary of the 13
*

*> different models that I tested for comparison of bootstrap and nonmem CI.
*

*>
*

*> http://www.quantpharm.com/pdf_files/2572-GibianskyPage2007Poster2007final.pdf
*

*>
*

*>
*

*> Note that all bootstrap samples were appropriately stratified by major
*

*> covariates (such as study, dose, weight as necessary, etc.).
*

*>
*

*> Leonid
*

*>
*

*> --------------------------------------
*

*> Leonid Gibiansky, Ph.D.
*

*> President, QuantPharm LLC
*

*> web: www.quantpharm.com
*

*> e-mail: LGibiansky at quantpharm.com
*

*> tel: (301) 767 5566
*

*>
*

*>
*

*>
*

*>
*

*> makamal *

*>> Dear Dr. Holford,
*

*>> Please correct me if I am wrong, however my understanding is that
*

*>> asymptotic distribution implied by NONMEM's covariance step
*

*>> approaches normality as the sample size gets larger or we have more
*

*>> data. However, a non parametric bootstrap distribution may have poor
*

*>> coverage with a small sample size as well, since it relies on
*

*>> sampling subjects with repalcement in the data set. So both
*

*>> distributions have problems when sample size is small (e.g. N<30).
*

*>> Therefore I would think when N is large the wald based Confidence
*

*>> Intervals from NONMEM are appropriate enough. It would be helpful to
*

*>> know the criteria when generating a non parametric bootstrap
*

*>> distribution is really advantageous.
*

*>> Thanks, Mohamed
*

*>> Quoting Nick Holford <n.holford *

*>
*

--

Nick Holford, Dept Pharmacology & Clinical Pharmacology

University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand

n.holford

http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Received on Sat Jul 26 2008 - 01:23:04 EDT

Date: Sat, 26 Jul 2008 17:23:04 +1200

Mohamed,

When the number of subjects is small then any confidence interval is

going to be wide and probably no-one is really interested in it. With

studies more suitable for population analysis (at least 25 subjects and

preferably over 100 if you want to look for covariate effects) then the

CIs may be more interesting.

With linear models or parameters which are nearly linear in non-linear

models then I would expect quite good agreement between CIs obtained by

bootstrapping and by using NONMEM SEs. But the models get interesting

when one tries to estimate non-linear parameters e.g. EC50 in an Emax

model. In that case the CIs will often be asymmetrical and the normal

distribution assumption used to compute SEs from CIs will be wrong.

Leonid does not discuss the issue of assymetry of CIs in his poster --

but when I look at Figure 3 I see evidence for disagreement between

bootstrap and NONMEM SE based CIs. The scatter of bootstrap points

relative to the solid line shows an excess of bootstrap upper CI values

above the SE prediction. For the lower CI prediction there also seem to

be more bootstrap values above the SE predictions. Its hard to be sure

that these upper and lower bootstrap predictions belong to the same

parameters but if so this would be evidence for asymmetry of the

bootstrap CI. This is exactly what one would expect because the SE

method has to assume symmetrical CIs yet the bootstrap estimate is not

restricted in this way.

I think this poster is a nice example of why correlation coefficients

are a very poor way to compare predictions (as pointed out by Sheiner

and Beal in their classic paper Sheiner LB, Beal SL. Some Suggestions

for Measuring Predictive Performance. J Pharmacokinet Biopharm.

1981;9(4):503-12.). A better way would be to compute the prediction

error for the absolute larger CI arm and smaller CI arm obtained by

bootstrapping to the symmetrial CI from the SE. If bootstraps CIs are

indeed asymmetrical then there would be a difference shown by the mean

prediction error ('bias'). Note that I use absolute value larger and

smaller CI arm to refer to the larger or smaller part of the CI that is

constructed around zero. I dont mean the upper and lower parts of the CI

interval.

Best wishes,

Nick

Leonid Gibiansky wrote:

--

Nick Holford, Dept Pharmacology & Clinical Pharmacology

University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand

n.holford

http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Received on Sat Jul 26 2008 - 01:23:04 EDT