NONMEM Users Network Archive

Hosted by Cognigen


From: Nick Holford <n.holford>
Date: Sat, 26 Jul 2008 17:23:04 +1200


When the number of subjects is small then any confidence interval is
going to be wide and probably no-one is really interested in it. With
studies more suitable for population analysis (at least 25 subjects and
preferably over 100 if you want to look for covariate effects) then the
CIs may be more interesting.

With linear models or parameters which are nearly linear in non-linear
models then I would expect quite good agreement between CIs obtained by
bootstrapping and by using NONMEM SEs. But the models get interesting
when one tries to estimate non-linear parameters e.g. EC50 in an Emax
model. In that case the CIs will often be asymmetrical and the normal
distribution assumption used to compute SEs from CIs will be wrong.

Leonid does not discuss the issue of assymetry of CIs in his poster --
but when I look at Figure 3 I see evidence for disagreement between
bootstrap and NONMEM SE based CIs. The scatter of bootstrap points
relative to the solid line shows an excess of bootstrap upper CI values
above the SE prediction. For the lower CI prediction there also seem to
be more bootstrap values above the SE predictions. Its hard to be sure
that these upper and lower bootstrap predictions belong to the same
parameters but if so this would be evidence for asymmetry of the
bootstrap CI. This is exactly what one would expect because the SE
method has to assume symmetrical CIs yet the bootstrap estimate is not
restricted in this way.

I think this poster is a nice example of why correlation coefficients
are a very poor way to compare predictions (as pointed out by Sheiner
and Beal in their classic paper Sheiner LB, Beal SL. Some Suggestions
for Measuring Predictive Performance. J Pharmacokinet Biopharm.
1981;9(4):503-12.). A better way would be to compute the prediction
error for the absolute larger CI arm and smaller CI arm obtained by
bootstrapping to the symmetrial CI from the SE. If bootstraps CIs are
indeed asymmetrical then there would be a difference shown by the mean
prediction error ('bias'). Note that I use absolute value larger and
smaller CI arm to refer to the larger or smaller part of the CI that is
constructed around zero. I dont mean the upper and lower parts of the CI

Best wishes,


Leonid Gibiansky wrote:
> I cannot make any general statements but here is the summary of the 13
> different models that I tested for comparison of bootstrap and nonmem CI.
> Note that all bootstrap samples were appropriately stratified by major
> covariates (such as study, dose, weight as necessary, etc.).
> Leonid
> --------------------------------------
> Leonid Gibiansky, Ph.D.
> President, QuantPharm LLC
> web:
> e-mail: LGibiansky at
> tel: (301) 767 5566
> makamal
>> Dear Dr. Holford,
>> Please correct me if I am wrong, however my understanding is that
>> asymptotic distribution implied by NONMEM's covariance step
>> approaches normality as the sample size gets larger or we have more
>> data. However, a non parametric bootstrap distribution may have poor
>> coverage with a small sample size as well, since it relies on
>> sampling subjects with repalcement in the data set. So both
>> distributions have problems when sample size is small (e.g. N<30).
>> Therefore I would think when N is large the wald based Confidence
>> Intervals from NONMEM are appropriate enough. It would be helpful to
>> know the criteria when generating a non parametric bootstrap
>> distribution is really advantageous.
>> Thanks, Mohamed
>> Quoting Nick Holford <n.holford

Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand

Received on Sat Jul 26 2008 - 01:23:04 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to:

Once subscribed, you may contribute to the discussion by emailing: