NONMEM Users Network Archive

Hosted by Cognigen

Re: What does convergence/covariance show?

From: Leonid Gibiansky <LGibiansky>
Date: Mon, 24 Aug 2009 19:50:07 -0400

Nick,

I think it is dangerous to rely heavily on the objective function (let
alone on ONLY objective function) in the model development process. I am
very surprised that you use it as the main diagnostic. If you think that
nonmem randomly stops at arbitrary point with arbitrary error, how can
you rely on the result of this random process as the main guide in the
model development? I pay attention to the OF but only as one of the
large toolbox of other diagnostics (most of them graphics). I routinely
see examples when over-parametrized unstable models provide better
objective function values, but this is not a sufficient reason to select
those. If you reject them in favor of simpler and more stable models,
you would see less random stops and more models with convergence and
successful covariance steps.

Even with bootstrap, I see the main real output of this procedure in
revealing the correlation of the parameter estimates rather then in
computation of CI. CI are less informative, while visualization of
correlations may suggest ways to improve the model.

Any way, it looks like there are at least the same number of modeling
methods as modelers: fortunately for all of us, this is still art, not
science; therefore, the time when everything will be done by the
computers is not too close.

Leonid

--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566




Nick Holford wrote:
> Mats, Leonid,
>
> Thanks for your definitions. I think I prefer that provided by Mats but
> he doesn't say what his test for goodness-of-fit might be.
>
> Leonid already assumes that convergence/covariance are diagnostic so it
> doesnt help at all with an independent definition of
> overparameterization. Correlation of random effects is often a very
> important part of a model -- especially for future predictions -- so I
> dont see that as a useful test -- unless you restrict it to pathological
> values eg. |correlation|>0.9?. Even with very high correlations I
> sometimes leave them in the model because setting the covariance to zero
> often makes quite a big worsening of the OBJ.
>
> My own view is that "overparameterization" is not a black and white
> entity. Parameters can be estimated with decreasing degrees of
> confidence depending on many things such as the design and the adequacy
> of the model. Parameter confidence intervals (preferably by bootstrap)
> are the way i would evaluate how well parameters are estimated. I
> usually rely on OBJ changes alone during model development with a VPC
> and boostrap confidence interval when I seem to have extracted all I can
> from the data. The VPC and CIs may well prompt further model development
> and the cycle continues.
>
> Nick
>
>
> Leonid Gibiansky wrote:
>> Hi Nick,
>>
>> I am not sure how you build the models but I am using convergence,
>> relative standard errors, correlation matrix of parameter estimates
>> (reported by the covariance step), and correlation of random effects
>> quite extensively when I decide whether I need extra compartments,
>> extra random effects, nonlinearity in the model, etc. For me they are
>> very useful as diagnostic of over-parameterization. This is the direct
>> evidence (proof?) that they are useful :)
>>
>> For new modelers who are just starting to learn how to do it, or have
>> limited experience, or have problems on the way, I would advise to pay
>> careful attention to these issues since they often help me to detect
>> problems. You seem to disagree with me; that is fine, I am not trying
>> to impose on you or anybody else my way of doing the analysis. This is
>> just an advise: you (and others) are free to use it or ignore it :)
>>
>> Thanks
>> Leonid
>
>
> Mats Karlsson wrote:
>> <<I would say that if you can remove parameters/model components without
>> detriment to goodness-of-fit then the model is overparameterized. >>
>>
>
Received on Mon Aug 24 2009 - 19:50:07 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.