NONMEM Users Network Archive

Hosted by Cognigen

RE: What does convergence/covariance show?

From: Mats Karlsson <mats.karlsson>
Date: Wed, 26 Aug 2009 05:06:08 +0200

Nick,

Could you elaborate on how you reason around the necessity of showing a
priori power when you find a significant effects from the study data? How
would you show it?

Best regards,
Mats

Mats Karlsson, PhD
Professor of Pharmacometrics
Dept of Pharmaceutical Biosciences
Uppsala University
Box 591
751 24 Uppsala Sweden
phone: +46 18 4714105
fax: +46 18 471 4003


-----Original Message-----
From: owner-nmusers
Behalf Of Nick Holford
Sent: Tuesday, August 25, 2009 11:54 PM
To: nmusers
Subject: Re: [NMusers] What does convergence/covariance show?

Mats,

You are right - I replied before the coffee had started working so I was
indeed in a strange world!

Nevertheless the isolated finding of P<0.05 should not be uncritically
interpreted as being of clinical relevance without other considerations
such as adequate a priori power and if possible some plausible mechanism
even if the P value suggests an increased hazard of death.

Nick

Mats Karlsson wrote:
>
> Nick,
>
> You're living in a strange world if killing patients is benefit :)
>
> Mats
>
> Mats Karlsson, PhD
>
> Professor of Pharmacometrics
>
> Dept of Pharmaceutical Biosciences
>
> Uppsala University
>
> Box 591
>
> 751 24 Uppsala Sweden
>
> phone: +46 18 4714105
>
> fax: +46 18 471 4003
>
> *From:* Nick Holford [mailto:n.holford
> *Sent:* Tuesday, August 25, 2009 11:15 PM
> *To:* Mats Karlsson
> *Subject:* Re: [NMusers] What does convergence/covariance show?
>
> Mats,
>
> If the trial was powered to test the effect of the treatment on
> survival then I would think that it would be reasonable to consider
> some practical consequences. However, FDA would not accept one trial
> alone as evidence of benefit without other strong supporting evidence
> from a different trial i.e. the OFV alone is not enough to accept
> clinical importance.
>
> Nick
>
>
> Mats Karlsson wrote:
>
> Nick,
>
> If the hazard of patients are dying is significantly (p<0.05) higher on
the
> new treatment compared to reference, I don't think you need other evidence
> before it has practical consequences. Without mechanistic understanding,
> would you ignore it and move on to the next analysis?
>
> Mats
>
> Mats Karlsson, PhD
> Professor of Pharmacometrics
> Dept of Pharmaceutical Biosciences
> Uppsala University
> Box 591
> 751 24 Uppsala Sweden
> phone: +46 18 4714105
> fax: +46 18 471 4003
>
>
> -----Original Message-----
> From: owner-nmusers
[mailto:owner-nmusers
> Behalf Of Nick Holford
> Sent: Tuesday, August 25, 2009 10:29 PM
> To: nmusers
> Subject: Re: [NMusers] What does convergence/covariance show?
>
> Mats,
>
> Thanks for stating more clearly what I tried to say before. Once again
> -- I agree that OFV is not a measure of clinical importance. But it is
> correlated with discernible differences in model predictions that may be
> of clinical importance.
>
> A change of OFV of 5 in a survival model may well be useful to reject a
> null hypothesis and point to some explanatory variable. There are
> numerous 'statistically significant' findings in the clinical literature
> like this that have no practical impact. You do not indicate what else
> in the survival analysis convinced you that the OFV was associated with
> something of practical consequence. I trust your decision was not based
> only on the OFV!
>
> Nick
>
> Mats Karlsson wrote:
>
>
> Nick,
>
>
>
> I agree that small changes (5-10) in OFV often are not practically
>
>
>
> important
>
>
> and and big changes more often are. However, my point is that OFV is
not
>
>
>
> the
>
>
> right scale to judge importance. You should judge it on the
consequence of
>
> you additional complexity to the model (the magnitude of the found
drug
>
> effect/covariate/etc). Just the other day did I analyze survival data
>
>
>
> where
>
>
> a small (5) change in OFV is of practical consequence.
>
>
>
> A true treatment effect of a certain size will improve the OFV in
relation
>
> to the size of the dataset. The larger the data set, the larger the
change
>
> in OFV. However, the estimate of the treatment effect does not change
>
> systematically with the size of the data set. The size of the
treatment
>
> effect is what is more appropriate diagnostic for practical
consequences.
>
> OFV we would use only to make sure that we have found the effect by
>
>
>
> chance.
>
>
> Best regards,
>
> Mats
>
>
>
> Mats Karlsson, PhD
>
> Professor of Pharmacometrics
>
> Dept of Pharmaceutical Biosciences
>
> Uppsala University
>
> Box 591
>
> 751 24 Uppsala Sweden
>
> phone: +46 18 4714105
>
> fax: +46 18 471 4003
>
>
>
>
>
> -----Original Message-----
>
> From: owner-nmusers
<mailto:owner-nmusers
>
>
>
> On
>
>
> Behalf Of Nick Holford
>
> Sent: Tuesday, August 25, 2009 7:25 AM
>
> To: nmusers
>
> Subject: Re: [NMusers] What does convergence/covariance show?
>
>
>
> Mats,
>
>
>
> When I referred to a change of 50 being needed to detect something of
>
> practical importance I was not saying that was of clinical relevance.
>
> That cannot be judged from the OFV alone. But small OFV changes are
>
> rarely if ever indicators of something that is clinically relevant.
>
>
>
> I expect you will agree on this point :-)
>
>
>
> Nick
>
>
>
> Mats Karlsson wrote:
>
>
>
>
>
> Nick,
>
>
>
> I too would use OFV as the most important goodness-of-fit
diagnostic when
>
> comparing models, especially when deeming something to be
redundant. If
>
> adding a component doesn't reduce OFV, I see no reason to include
it (I
>
> think we're agreeing on something!). However, you write
>
>
>
> " Small (5-10) changes in OBJ are not of much interest. A change
of OBJ
>
>
>
> of
>
>
> at least 50 is usually needed to detect anything of practical
>
>
>
> importance."
>
>
> Today we use population methods for everything from very rich pop
pk
>
> meta-analyses to very sparsely informative data sets on survival.
To use
>
>
>
>
>
> OFV
>
>
>
>
>
> as a measure of goodness-of-fit is central and look at the risk
something
>
> improved the fit by chance, but I would not use it as measure of
clinical
>
> importance.
>
>
>
> Best regards,
>
> Mats
>
>
>
> Mats Karlsson, PhD
>
> Professor of Pharmacometrics
>
> Dept of Pharmaceutical Biosciences
>
> Uppsala University
>
> Box 591
>
> 751 24 Uppsala Sweden
>
> phone: +46 18 4714105
>
> fax: +46 18 471 4003
>
>
>
>
>
> -----Original Message-----
>
> From: owner-nmusers
<mailto:owner-nmusers
>
>
>
>
>
> On
>
>
>
>
>
> Behalf Of Nick Holford
>
> Sent: Tuesday, August 25, 2009 12:14 AM
>
> To: nmusers
>
> Subject: Re: [NMusers] What does convergence/covariance show?
>
>
>
> Mats, Leonid,
>
>
>
> Thanks for your definitions. I think I prefer that provided by
Mats but
>
> he doesn't say what his test for goodness-of-fit might be.
>
>
>
> Leonid already assumes that convergence/covariance are diagnostic
so it
>
> doesnt help at all with an independent definition of
>
> overparameterization. Correlation of random effects is often a
very
>
> important part of a model -- especially for future predictions --
so I
>
> dont see that as a useful test -- unless you restrict it to
pathological
>
> values eg. |correlation|>0.9?. Even with very high correlations I
>
> sometimes leave them in the model because setting the covariance
to zero
>
> often makes quite a big worsening of the OBJ.
>
>
>
> My own view is that "overparameterization" is not a black and
white
>
> entity. Parameters can be estimated with decreasing degrees of
>
> confidence depending on many things such as the design and the
adequacy
>
> of the model. Parameter confidence intervals (preferably by
bootstrap)
>
> are the way i would evaluate how well parameters are estimated. I
>
> usually rely on OBJ changes alone during model development with a
VPC
>
> and boostrap confidence interval when I seem to have extracted all
I can
>
> from the data. The VPC and CIs may well prompt further model
development
>
> and the cycle continues.
>
>
>
> Nick
>
>
>
>
>
> Leonid Gibiansky wrote:
>
>
>
>
>
>
>
> Hi Nick,
>
>
>
> I am not sure how you build the models but I am using
convergence,
>
> relative standard errors, correlation matrix of parameter
estimates
>
> (reported by the covariance step), and correlation of random
effects
>
> quite extensively when I decide whether I need extra
compartments,
>
> extra random effects, nonlinearity in the model, etc. For me
they are
>
> very useful as diagnostic of over-parameterization. This is
the direct
>
> evidence (proof?) that they are useful :)
>
>
>
> For new modelers who are just starting to learn how to do it,
or have
>
> limited experience, or have problems on the way, I would
advise to pay
>
> careful attention to these issues since they often help me to
detect
>
> problems. You seem to disagree with me; that is fine, I am not
trying
>
> to impose on you or anybody else my way of doing the analysis.
This is
>
> just an advise: you (and others) are free to use it or ignore
it :)
>
>
>
> Thanks
>
> Leonid
>
>
>
>
>
>
>
> Mats Karlsson wrote:
>
>
>
>
>
>
>
> <<I would say that if you can remove parameters/model
components without
>
> detriment to goodness-of-fit then the model is
overparameterized. >>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Nick Holford, Professor Clinical Pharmacology
> Dept Pharmacology & Clinical Pharmacology
> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New
Zealand
> n.holford
tel:+64(9)923-6730 fax:+64(9)373-7090
> mobile: +64 21 46 23 53
> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
n.holford
mobile: +64 21 46 23 53
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Received on Tue Aug 25 2009 - 23:06:08 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.