NONMEM Users Network Archive

Hosted by Cognigen

Re: What does convergence/covariance show?

From: Nick Holford <n.holford>
Date: Wed, 26 Aug 2009 08:28:30 +1200

Mats,

Thanks for stating more clearly what I tried to say before. Once again
-- I agree that OFV is not a measure of clinical importance. But it is
correlated with discernible differences in model predictions that may be
of clinical importance.

A change of OFV of 5 in a survival model may well be useful to reject a
null hypothesis and point to some explanatory variable. There are
numerous 'statistically significant' findings in the clinical literature
like this that have no practical impact. You do not indicate what else
in the survival analysis convinced you that the OFV was associated with
something of practical consequence. I trust your decision was not based
only on the OFV!

Nick

Mats Karlsson wrote:
> Nick,
>
> I agree that small changes (5-10) in OFV often are not practically important
> and and big changes more often are. However, my point is that OFV is not the
> right scale to judge importance. You should judge it on the consequence of
> you additional complexity to the model (the magnitude of the found drug
> effect/covariate/etc). Just the other day did I analyze survival data where
> a small (5) change in OFV is of practical consequence.
>
> A true treatment effect of a certain size will improve the OFV in relation
> to the size of the dataset. The larger the data set, the larger the change
> in OFV. However, the estimate of the treatment effect does not change
> systematically with the size of the data set. The size of the treatment
> effect is what is more appropriate diagnostic for practical consequences.
> OFV we would use only to make sure that we have found the effect by chance.
>
> Best regards,
> Mats
>
> Mats Karlsson, PhD
> Professor of Pharmacometrics
> Dept of Pharmaceutical Biosciences
> Uppsala University
> Box 591
> 751 24 Uppsala Sweden
> phone: +46 18 4714105
> fax: +46 18 471 4003
>
>
> -----Original Message-----
> From: owner-nmusers
> Behalf Of Nick Holford
> Sent: Tuesday, August 25, 2009 7:25 AM
> To: nmusers
> Subject: Re: [NMusers] What does convergence/covariance show?
>
> Mats,
>
> When I referred to a change of 50 being needed to detect something of
> practical importance I was not saying that was of clinical relevance.
> That cannot be judged from the OFV alone. But small OFV changes are
> rarely if ever indicators of something that is clinically relevant.
>
> I expect you will agree on this point :-)
>
> Nick
>
> Mats Karlsson wrote:
>
>> Nick,
>>
>> I too would use OFV as the most important goodness-of-fit diagnostic when
>> comparing models, especially when deeming something to be redundant. If
>> adding a component doesn't reduce OFV, I see no reason to include it (I
>> think we're agreeing on something!). However, you write
>>
>> " Small (5-10) changes in OBJ are not of much interest. A change of OBJ of
>> at least 50 is usually needed to detect anything of practical importance."
>>
>> Today we use population methods for everything from very rich pop pk
>> meta-analyses to very sparsely informative data sets on survival. To use
>>
> OFV
>
>> as a measure of goodness-of-fit is central and look at the risk something
>> improved the fit by chance, but I would not use it as measure of clinical
>> importance.
>>
>> Best regards,
>> Mats
>>
>> Mats Karlsson, PhD
>> Professor of Pharmacometrics
>> Dept of Pharmaceutical Biosciences
>> Uppsala University
>> Box 591
>> 751 24 Uppsala Sweden
>> phone: +46 18 4714105
>> fax: +46 18 471 4003
>>
>>
>> -----Original Message-----
>> From: owner-nmusers
>>
> On
>
>> Behalf Of Nick Holford
>> Sent: Tuesday, August 25, 2009 12:14 AM
>> To: nmusers
>> Subject: Re: [NMusers] What does convergence/covariance show?
>>
>> Mats, Leonid,
>>
>> Thanks for your definitions. I think I prefer that provided by Mats but
>> he doesn't say what his test for goodness-of-fit might be.
>>
>> Leonid already assumes that convergence/covariance are diagnostic so it
>> doesnt help at all with an independent definition of
>> overparameterization. Correlation of random effects is often a very
>> important part of a model -- especially for future predictions -- so I
>> dont see that as a useful test -- unless you restrict it to pathological
>> values eg. |correlation|>0.9?. Even with very high correlations I
>> sometimes leave them in the model because setting the covariance to zero
>> often makes quite a big worsening of the OBJ.
>>
>> My own view is that "overparameterization" is not a black and white
>> entity. Parameters can be estimated with decreasing degrees of
>> confidence depending on many things such as the design and the adequacy
>> of the model. Parameter confidence intervals (preferably by bootstrap)
>> are the way i would evaluate how well parameters are estimated. I
>> usually rely on OBJ changes alone during model development with a VPC
>> and boostrap confidence interval when I seem to have extracted all I can
>> from the data. The VPC and CIs may well prompt further model development
>> and the cycle continues.
>>
>> Nick
>>
>>
>> Leonid Gibiansky wrote:
>>
>>
>>> Hi Nick,
>>>
>>> I am not sure how you build the models but I am using convergence,
>>> relative standard errors, correlation matrix of parameter estimates
>>> (reported by the covariance step), and correlation of random effects
>>> quite extensively when I decide whether I need extra compartments,
>>> extra random effects, nonlinearity in the model, etc. For me they are
>>> very useful as diagnostic of over-parameterization. This is the direct
>>> evidence (proof?) that they are useful :)
>>>
>>> For new modelers who are just starting to learn how to do it, or have
>>> limited experience, or have problems on the way, I would advise to pay
>>> careful attention to these issues since they often help me to detect
>>> problems. You seem to disagree with me; that is fine, I am not trying
>>> to impose on you or anybody else my way of doing the analysis. This is
>>> just an advise: you (and others) are free to use it or ignore it :)
>>>
>>> Thanks
>>> Leonid
>>>
>>>
>> Mats Karlsson wrote:
>>
>>
>>> <<I would say that if you can remove parameters/model components without
>>> detriment to goodness-of-fit then the model is overparameterized. >>
>>>
>>>
>>>
>>
>>
>
>

--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
n.holford
mobile: +64 21 46 23 53
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Received on Tue Aug 25 2009 - 16:28:30 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.