NONMEM Users Network Archive

Hosted by Cognigen

Re: What does convergence/covariance show?

From: Nick Holford <n.holford>
Date: Wed, 26 Aug 2009 09:53:32 +1200

Mats,

You are right - I replied before the coffee had started working so I was
indeed in a strange world!

Nevertheless the isolated finding of P<0.05 should not be uncritically
interpreted as being of clinical relevance without other considerations
such as adequate a priori power and if possible some plausible mechanism
even if the P value suggests an increased hazard of death.

Nick

Mats Karlsson wrote:
>
> Nick,
>
> You’re living in a strange world if killing patients is benefit :)
>
> Mats
>
> Mats Karlsson, PhD
>
> Professor of Pharmacometrics
>
> Dept of Pharmaceutical Biosciences
>
> Uppsala University
>
> Box 591
>
> 751 24 Uppsala Sweden
>
> phone: +46 18 4714105
>
> fax: +46 18 471 4003
>
> *From:* Nick Holford [mailto:n.holford
> *Sent:* Tuesday, August 25, 2009 11:15 PM
> *To:* Mats Karlsson
> *Subject:* Re: [NMusers] What does convergence/covariance show?
>
> Mats,
>
> If the trial was powered to test the effect of the treatment on
> survival then I would think that it would be reasonable to consider
> some practical consequences. However, FDA would not accept one trial
> alone as evidence of benefit without other strong supporting evidence
> from a different trial i.e. the OFV alone is not enough to accept
> clinical importance.
>
> Nick
>
>
> Mats Karlsson wrote:
>
> Nick,
>
> If the hazard of patients are dying is significantly (p<0.05) higher on the
> new treatment compared to reference, I don't think you need other evidence
> before it has practical consequences. Without mechanistic understanding,
> would you ignore it and move on to the next analysis?
>
> Mats
>
> Mats Karlsson, PhD
> Professor of Pharmacometrics
> Dept of Pharmaceutical Biosciences
> Uppsala University
> Box 591
> 751 24 Uppsala Sweden
> phone: +46 18 4714105
> fax: +46 18 471 4003
>
>
> -----Original Message-----
> From: owner-nmusers
> Behalf Of Nick Holford
> Sent: Tuesday, August 25, 2009 10:29 PM
> To: nmusers
> Subject: Re: [NMusers] What does convergence/covariance show?
>
> Mats,
>
> Thanks for stating more clearly what I tried to say before. Once again
> -- I agree that OFV is not a measure of clinical importance. But it is
> correlated with discernible differences in model predictions that may be
> of clinical importance.
>
> A change of OFV of 5 in a survival model may well be useful to reject a
> null hypothesis and point to some explanatory variable. There are
> numerous 'statistically significant' findings in the clinical literature
> like this that have no practical impact. You do not indicate what else
> in the survival analysis convinced you that the OFV was associated with
> something of practical consequence. I trust your decision was not based
> only on the OFV!
>
> Nick
>
> Mats Karlsson wrote:
>
>
> Nick,
>
>
>
> I agree that small changes (5-10) in OFV often are not practically
>
>
>
> important
>
>
> and and big changes more often are. However, my point is that OFV is not
>
>
>
> the
>
>
> right scale to judge importance. You should judge it on the consequence of
>
> you additional complexity to the model (the magnitude of the found drug
>
> effect/covariate/etc). Just the other day did I analyze survival data
>
>
>
> where
>
>
> a small (5) change in OFV is of practical consequence.
>
>
>
> A true treatment effect of a certain size will improve the OFV in relation
>
> to the size of the dataset. The larger the data set, the larger the change
>
> in OFV. However, the estimate of the treatment effect does not change
>
> systematically with the size of the data set. The size of the treatment
>
> effect is what is more appropriate diagnostic for practical consequences.
>
> OFV we would use only to make sure that we have found the effect by
>
>
>
> chance.
>
>
> Best regards,
>
> Mats
>
>
>
> Mats Karlsson, PhD
>
> Professor of Pharmacometrics
>
> Dept of Pharmaceutical Biosciences
>
> Uppsala University
>
> Box 591
>
> 751 24 Uppsala Sweden
>
> phone: +46 18 4714105
>
> fax: +46 18 471 4003
>
>
>
>
>
> -----Original Message-----
>
> From: owner-nmusers
>
>
>
> On
>
>
> Behalf Of Nick Holford
>
> Sent: Tuesday, August 25, 2009 7:25 AM
>
> To: nmusers
>
> Subject: Re: [NMusers] What does convergence/covariance show?
>
>
>
> Mats,
>
>
>
> When I referred to a change of 50 being needed to detect something of
>
> practical importance I was not saying that was of clinical relevance.
>
> That cannot be judged from the OFV alone. But small OFV changes are
>
> rarely if ever indicators of something that is clinically relevant.
>
>
>
> I expect you will agree on this point :-)
>
>
>
> Nick
>
>
>
> Mats Karlsson wrote:
>
>
>
>
>
> Nick,
>
>
>
> I too would use OFV as the most important goodness-of-fit diagnostic when
>
> comparing models, especially when deeming something to be redundant. If
>
> adding a component doesn't reduce OFV, I see no reason to include it (I
>
> think we're agreeing on something!). However, you write
>
>
>
> " Small (5-10) changes in OBJ are not of much interest. A change of OBJ
>
>
>
> of
>
>
> at least 50 is usually needed to detect anything of practical
>
>
>
> importance."
>
>
> Today we use population methods for everything from very rich pop pk
>
> meta-analyses to very sparsely informative data sets on survival. To use
>
>
>
>
>
> OFV
>
>
>
>
>
> as a measure of goodness-of-fit is central and look at the risk something
>
> improved the fit by chance, but I would not use it as measure of clinical
>
> importance.
>
>
>
> Best regards,
>
> Mats
>
>
>
> Mats Karlsson, PhD
>
> Professor of Pharmacometrics
>
> Dept of Pharmaceutical Biosciences
>
> Uppsala University
>
> Box 591
>
> 751 24 Uppsala Sweden
>
> phone: +46 18 4714105
>
> fax: +46 18 471 4003
>
>
>
>
>
> -----Original Message-----
>
> From: owner-nmusers
>
>
>
>
>
> On
>
>
>
>
>
> Behalf Of Nick Holford
>
> Sent: Tuesday, August 25, 2009 12:14 AM
>
> To: nmusers
>
> Subject: Re: [NMusers] What does convergence/covariance show?
>
>
>
> Mats, Leonid,
>
>
>
> Thanks for your definitions. I think I prefer that provided by Mats but
>
> he doesn't say what his test for goodness-of-fit might be.
>
>
>
> Leonid already assumes that convergence/covariance are diagnostic so it
>
> doesnt help at all with an independent definition of
>
> overparameterization. Correlation of random effects is often a very
>
> important part of a model -- especially for future predictions -- so I
>
> dont see that as a useful test -- unless you restrict it to pathological
>
> values eg. |correlation|>0.9?. Even with very high correlations I
>
> sometimes leave them in the model because setting the covariance to zero
>
> often makes quite a big worsening of the OBJ.
>
>
>
> My own view is that "overparameterization" is not a black and white
>
> entity. Parameters can be estimated with decreasing degrees of
>
> confidence depending on many things such as the design and the adequacy
>
> of the model. Parameter confidence intervals (preferably by bootstrap)
>
> are the way i would evaluate how well parameters are estimated. I
>
> usually rely on OBJ changes alone during model development with a VPC
>
> and boostrap confidence interval when I seem to have extracted all I can
>
> from the data. The VPC and CIs may well prompt further model development
>
> and the cycle continues.
>
>
>
> Nick
>
>
>
>
>
> Leonid Gibiansky wrote:
>
>
>
>
>
>
>
> Hi Nick,
>
>
>
> I am not sure how you build the models but I am using convergence,
>
> relative standard errors, correlation matrix of parameter estimates
>
> (reported by the covariance step), and correlation of random effects
>
> quite extensively when I decide whether I need extra compartments,
>
> extra random effects, nonlinearity in the model, etc. For me they are
>
> very useful as diagnostic of over-parameterization. This is the direct
>
> evidence (proof?) that they are useful :)
>
>
>
> For new modelers who are just starting to learn how to do it, or have
>
> limited experience, or have problems on the way, I would advise to pay
>
> careful attention to these issues since they often help me to detect
>
> problems. You seem to disagree with me; that is fine, I am not trying
>
> to impose on you or anybody else my way of doing the analysis. This is
>
> just an advise: you (and others) are free to use it or ignore it :)
>
>
>
> Thanks
>
> Leonid
>
>
>
>
>
>
>
> Mats Karlsson wrote:
>
>
>
>
>
>
>
> <<I would say that if you can remove parameters/model components without
>
> detriment to goodness-of-fit then the model is overparameterized. >>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Nick Holford, Professor Clinical Pharmacology
> Dept Pharmacology & Clinical Pharmacology
> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
> n.holford
> mobile: +64 21 46 23 53
> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
n.holford
mobile: +64 21 46 23 53
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Received on Tue Aug 25 2009 - 17:53:32 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.