Ken, In defense of the mechanistic modeler:
I suspect that generally what we want to do with models is extrapolate.&nb=
sp; That is, predict how people who are older, younger, larger, smaller, on=
drug longer, on higher doses, have interacting meds, 2D6 deficiency, other=
disease etc will behave. Predicting data within the range of w=
hat you've studied isn't really all that interesting, and can, for the most=
part be left to traditional statistics  and falls into the "stamp collect=
ing" category from Rutherford (another good Kiwi I believe). That, I =
think is an important difference between hypothesis testing (which is very =
important) and modeling/estimation (which is a lot more interesting, and in=
herently, more risky) So, if you model a linear relationship because tha=
t is all the range of your data will support (even though you know linear r=
elationships are very rare in biology) you've essentially precluded any opp=
ortunity to extrapolate beyond your data. If you do so, you will cert=
ainly be wrong. Your model is well supported, not risky, but not very inter=
esting. Imposing an Emax (or other biologically plausible) model will=
result in you being wrong sometimes (as opposed to always wrong with the l=
inear model). But, we must always make the "customer" aware of the limit=
ations of the analysis  some guess at the chances of it being very wrong.<=
br> Bottom line  if we want to say something interesting, more interest=
ing that traditional statistics, we will need to take risks with less than =
optimally supported mechanistic models.
Ma=
rk Sale MD
Next Level Solutions, LLC
www.NextLevelSolns.com
9198469185
 Original Message 
Subject: RE: [NMusers] What does convergence/covariance show?
From: Ken Kowalski <ken.kowalski Date: Tue, August 25, 2009 12:03 pm
To: "'nmusers'" <nmusers
Nick,
It sounds like you do recognize that models are often overparameterized by=
your statements:
" It is quite common to find that the
estimates EC50 and Emax are highly correlated (I assume SLOP=EMAX/EC50). =
It would also be common to find that the random effects of EMAX and EC50
are also correlated. That is expected given the limitations of most
pharmacodynamic designs."
When EC50 and Emax are highly correlated I think you will find that a
simplified linear model will fit the data just as well with no real impact<=
br>
on goodnessoffit (e.g., OFV). If we only observe concentrations in the
linear range of an Emax curve because of a poor design then it is no
surprise that a linear model may perform as well as an Emax model within th=
e
range of our data. If the design is so poor in information content
regarding the Emax relationship because of too narrow a range of
concentrations this will indeed lead to convergence and COV step failures i=
n
fitting the Emax model.
Your statement that you would be unwilling to accept the linear model in
this setting really speaks to the plight of the mechanistic modeler. It is=
important to note that an overparameterized model does not mean that the
model is missspecified. A model can be correctly specified but still be
overparameterized because the data/design simply will not support
estimation of all the parameters in the correctly specified model. The
mechanistic modeler who has a strong biological prior favoring the more
complex model is reluctant to accept a simplified model that he/she knows
has to be wrong (e.g., we would not expect that the linear model would hold=
up at considerably higher concentrations than those observed in the existin=
g
data). The problem with accepting the more complex model in this setting i=
s
that we can't really trust the estimates we get (when the model has
convergence difficulties and COV step failures as a result of
overparameterization) because there may be an infinite set of solutions to=
the parameters that give the same goodnessoffit (i.e., a very flat
likelihood surface). You can do all the bootstrapping you want but it is
not a panacea for the deficiencies of a poor design.
While I like to fit mechanistic models just as much as the next guy, I also=
like my models to be stable (not overparameterized). In this setting, the=
pragmatist in me would accept the simpler model, acknowledge the limitation=
s
of the design and model, and I would be very cautious not to extrapolate my=
model too far from the range of my existing data. More importantly, I woul=
d
advocate improving the situation by designing a better study so that we can=
get the information we need to support a more appropriate model that will
put us in a better position to extrapolate to new experimental conditions.<=
br>
We review the COV step output (looking for high correlations such as betwee=
n
the estimates of EC50 and Emax) and fit simpler models not because we prefe=
r
simpler models per se, but because we want to fully understand the
limitations of our design. Of course this simple example of a poor design<=
br>
with too narrow a concentration and/or dose range to estimate the Emax
relationship can be easily uncovered in a simple plot of the data, however,=
for more complex models the nature of the overparameterization and the
limitations of the design can be harder to detect which is why we need a
variety of strategies and diagnostics including plots, COV step output,
fitting alternative simpler models, etc. to fully understand these
limitations.
Just my 2 cents. :)
Ken
Original Message
From: ownernmusers ation==top.location){Popup.composeWindow('pcompose.php?sendto=ownern=
musers%40globomaxnm.com');}else{top.Popup.composeWindow('pcompose.php?sendt=
o=ownernmusers%40globomaxnm.com');} return false;" href="http://email0=
1.secureserver.net/pcompose.php#Compose" mce_href="http://email01.secures=
erver.net/pcompose.php#Compose">mailto:ownernmusers /a>] On
Behalf Of Nick Holford
Sent: Tuesday, August 25, 2009 1:09 AM
To: nmusers
Subject: Re: [NMusers] What does convergence/covariance show?
Leonid,
I did not say NONMEM stops at random. Whether or not the stopping point
is associated with convergence or a successful covariance step appears
to be at random. The parameter values at the stopping point will
typically be negligibly different. Thus the stopping point is not at
random. You can easily observe this in your bootstrap runs. Compare the
parameter distribution for runs that converge with those that dont and
you will find there are negligible differences in the distributions.
I did not say that I ignore small changes in OFV but my decisions are
guided by the size of the change.
I do not waste much time modelling absorption. It rarely is of any
relevance to try to fit all the small details.
I dont see anything in the plot of SLOP vs EC50 that is not revealed by
R=0.93. If the covariance step ran you would see a similar number in the =
correlation matrix of the estimate. It is quite common to find that the
estimates EC50 and Emax are highly correlated (I assume SLOP=EMAX/EC50). =
It would also be common to find that the random effects of EMAX and EC50
are also correlated. That is expected given the limitations of most
pharmacodynamic designs. However, I would not simplify the model to a
linear model just because of these correlations. I would pay much more
attention to the change in OFV comparing an Emax with a linear model
plus whatever was known about the studied concentration range and the EC50.=
I do agree that bootstraps can be helpful for calculating CIs on
secondary parameters.
Nick
Leonid Gibiansky wrote:
> Nick,
> Concerning "random stops at arbitrary point with arbitrary error" I
> was referring to your statement: "NONMEM VI will fail to converge or <=
br>
> not complete the covariance step more or less at random"
>
> For OFV, you did not tell the entire story. If you would look only on =
> OF, you would go for the absolute minimum of OF. If you ignore small <=
br>
> changes, it means that you use some other diagnostic to (possibly)
> select a model with higher OFV (if the difference is not too high,
> within 51020 units), preferring that model based on other signs
> (convergence? plots? number of parameters?). This is exactly what I
> was referring to when I mentioned that OF is just one of the criteria.=
>
> One common example where OF is not the best guide is the modeling of <=
br>
> absorption. You can spend weeks building progressively more and more <=
br>
> complicated models of absorptions profiles (with parallel, sequential,=
> timedependent, Mtimemodeled absorption etc.) with large drop in OF =
> (that corresponds to minor improvement for a few patients), with no
> gain in predictive power of your primary parameters of interest, for <=
br>
> example, steadystate exposure.
>
> To provide example of the bootstrap plot, I put it here:
>
> http://quantph=
arm.com/pdf_files/example.pdf
>
> For 1000 bootstrap problems, parameter estimates were plotted versus <=
br>
> parameter estimates. You can immediately see that SLOP and EC50 are
> strongly correlated while all other parameters are not correlated. CI =
> and even correlation coefficient value do not tell the whole story
> about the model. You can get similar results from the covariancestep =
> correlation matrix of parameter estimates but it requires simulations =
> to visualize it as clearly as from bootstrap results. Advantage of
> bootstrap plots is that one can easily study correlations and
> variability of not only primary parameters (such as theta, omega,
> etc), but also relations between derived parameters.
>
> Leonid
>
> 
> Leonid Gibiansky, Ph.D.
> President, QuantPharm LLC
> web: www.quantpharm.com
> email: LGibiansky at quantpharm.com
> tel: (301) 767 5566
>
>
>
>
> Nick Holford wrote:
>> Leonid,
>>
>> I do not experience "random stops at arbitrary point with arbitrar=
y
>> error" so I don't understand what your problem is.
>>
>> The objective function is the primary metric of goodness of fit. I=
>> agree it is possible to get drops in objective function that are <=
br>
>> associated with unreasonable parameter estimates (typically an OME=
GA
>> estimate). But I look at the parameter estimates after each run so=
>> that I can detect this kind of problem. Part of the display of the=
>> parameter estimates is the correlation of random effects if I am <=
br>
>> using OMEGA BLOCK. This is also a weaker secondary tool. By explor=
ing
>> different models I can get a feel for which parts of the model are=
>> informative and which are not by looking at the change in OBJ. Sma=
ll
>> (510) changes in OBJ are not of much interest. A change of OBJ of=
at
>> least 50 is usually needed to detect anything of practical importa=
nce.
>>
>> I don't understand what you find of interest in the correlation of=
>> bootstrap parameter estimates. This is really nothing more than yo=
u
>> would get from looking at the correlation matrix of the estimate f=
rom
>> the covariance step. High estimation correlations point to poor
>> estimability of the parameters but I think they are not very helpf=
ul
>> for pointing to ways to improve the model.
>>
>> Nevertheless I can agree to disagree on our modelling art :)
>>
>> Nick
>>
>> Leonid Gibiansky wrote:
>>> Nick,
>>>
>>> I think it is dangerous to rely heavily on the objective funct=
ion
>>> (let alone on ONLY objective function) in the model developmen=
t
>>> process. I am very surprised that you use it as the main diagn=
ostic.
>>> If you think that nonmem randomly stops at arbitrary point wit=
h
>>> arbitrary error, how can you rely on the result of this random=
>>> process as the main guide in the model development? I pay atte=
ntion
>>> to the OF but only as one of the large toolbox of other diagno=
stics
>>> (most of them graphics). I routinely see examples when
>>> overparametrized unstable models provide better objective fun=
ction
>>> values, but this is not a sufficient reason to select those. I=
f you
>>> reject them in favor of simpler and more stable models, you wo=
uld
>>> see less random stops and more models with convergence and
>>> successful covariance steps.
>>>
>>> Even with bootstrap, I see the main real output of this proced=
ure in
>>> revealing the correlation of the parameter estimates rather th=
en in
>>> computation of CI. CI are less informative, while visualizatio=
n of
>>> correlations may suggest ways to improve the model.
>>>
>>> Any way, it looks like there are at least the same number of <=
br>
>>> modeling methods as modelers: fortunately for all of us, this =
is
>>> still art, not science; therefore, the time when everything wi=
ll be
>>> done by the computers is not too close.
>>>
>>> Leonid
>>>
>>> 
>>> Leonid Gibiansky, Ph.D.
>>> President, QuantPharm LLC
>>> web: www.quantpharm.com
>>> email: LGibiansky at quantpharm.com
>>> tel: (301) 767 5566
>>>
>>>
>>>
>>>
>>> Nick Holford wrote:
>>>> Mats, Leonid,
>>>>
>>>> Thanks for your definitions. I think I prefer that provide=
d by Mats
>>>> but he doesn't say what his test for goodnessoffit might=
be.
>>>>
>>>> Leonid already assumes that convergence/covariance are dia=
gnostic
>>>> so it doesnt help at all with an independent definition of=
>>>> overparameterization. Correlation of random effects is oft=
en a very
>>>> important part of a model  especially for future predict=
ions 
>>>> so I dont see that as a useful test  unless you restrict=
it to
>>>> pathological values eg. correlation>0.9?. Even with v=
ery high
>>>> correlations I sometimes leave them in the model because s=
etting
>>>> the covariance to zero often makes quite a big worsening o=
f the OBJ.
>>>>
>>>> My own view is that "overparameterization" is not a black =
and white
>>>> entity. Parameters can be estimated with decreasing degree=
s of
>>>> confidence depending on many things such as the design and=
the
>>>> adequacy of the model. Parameter confidence intervals (pre=
ferably
>>>> by bootstrap) are the way i would evaluate how well parame=
ters are
>>>> estimated. I usually rely on OBJ changes alone during mode=
l
>>>> development with a VPC and boostrap confidence interval wh=
en I seem
>>>> to have extracted all I can from the data. The VPC and CIs=
may well
>>>> prompt further model development and the cycle continues.<=
br>
>>>>
>>>> Nick
>>>>
>>>>
>>>> Leonid Gibiansky wrote:
>>>>> Hi Nick,
>>>>>
>>>>> I am not sure how you build the models but I am using =
convergence,
>>>>> relative standard errors, correlation matrix of parame=
ter
>>>>> estimates (reported by the covariance step), and corre=
lation of
>>>>> random effects quite extensively when I decide whether=
I need
>>>>> extra compartments, extra random effects, nonlinearity=
in the
>>>>> model, etc. For me they are very useful as diagnostic =
of
>>>>> overparameterization. This is the direct evidence (pr=
oof?) that
>>>>> they are useful :)
>>>>>
>>>>> For new modelers who are just starting to learn how to=
do it, or
>>>>> have limited experience, or have problems on the way, =
I would
>>>>> advise to pay careful attention to these issues since =
they often
>>>>> help me to detect problems. You seem to disagree with =
me; that is
>>>>> fine, I am not trying to impose on you or anybody else=
my way of
>>>>> doing the analysis. This is just an advise: you (and o=
thers) are
>>>>> free to use it or ignore it :)
>>>>>
>>>>> Thanks
>>>>> Leonid
>>>>
>>>>
>>>> Mats Karlsson wrote:
>>>>> <<I would say that if you can remove parameters/=
model components
>>>>> without
>>>>> detriment to goodnessoffit then the model is overpar=
ameterized. >>
>>>>>
>>>>
>>

Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealan=
d
n.holford mobile: +64 21 46 23 53
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
