NONMEM Users Network Archive

Hosted by Cognigen

Re: What does convergence/covariance show?

From: Nick Holford <n.holford>
Date: Wed, 26 Aug 2009 21:58:15 +1200

Mats,

The issue of selection bias with underpowered studies has been discussed
at length by Ribbing and Jonsson 2004.

Steve Duffull gave a very nice talk a couple of years ago at PAGANZ on
this problem and the difficulties of interpreting the controversial
phase IV studies of Vioxx. Perhaps Steve can explain this issue better
than I can.

Nick

Ribbing J, Jonsson EN. Power, Selection Bias and Predictive Performance
of the Population Pharmacokinetic Covariate Model. Journal of
Pharmacokinetics and Pharmacodynamics. 2004;31(2):109-34.


Mats Karlsson wrote:
> Nick,
>
> Could you elaborate on how you reason around the necessity of showing a
> priori power when you find a significant effects from the study data? How
> would you show it?
>
> Best regards,
> Mats
>
> Mats Karlsson, PhD
> Professor of Pharmacometrics
> Dept of Pharmaceutical Biosciences
> Uppsala University
> Box 591
> 751 24 Uppsala Sweden
> phone: +46 18 4714105
> fax: +46 18 471 4003
>
>
> -----Original Message-----
> From: owner-nmusers
> Behalf Of Nick Holford
> Sent: Tuesday, August 25, 2009 11:54 PM
> To: nmusers
> Subject: Re: [NMusers] What does convergence/covariance show?
>
> Mats,
>
> You are right - I replied before the coffee had started working so I was
> indeed in a strange world!
>
> Nevertheless the isolated finding of P<0.05 should not be uncritically
> interpreted as being of clinical relevance without other considerations
> such as adequate a priori power and if possible some plausible mechanism
> even if the P value suggests an increased hazard of death.
>
> Nick
>
> Mats Karlsson wrote:
>
>> Nick,
>>
>> You're living in a strange world if killing patients is benefit :)
>>
>> Mats
>>
>> Mats Karlsson, PhD
>>
>> Professor of Pharmacometrics
>>
>> Dept of Pharmaceutical Biosciences
>>
>> Uppsala University
>>
>> Box 591
>>
>> 751 24 Uppsala Sweden
>>
>> phone: +46 18 4714105
>>
>> fax: +46 18 471 4003
>>
>> *From:* Nick Holford [mailto:n.holford
>> *Sent:* Tuesday, August 25, 2009 11:15 PM
>> *To:* Mats Karlsson
>> *Subject:* Re: [NMusers] What does convergence/covariance show?
>>
>> Mats,
>>
>> If the trial was powered to test the effect of the treatment on
>> survival then I would think that it would be reasonable to consider
>> some practical consequences. However, FDA would not accept one trial
>> alone as evidence of benefit without other strong supporting evidence
>> from a different trial i.e. the OFV alone is not enough to accept
>> clinical importance.
>>
>> Nick
>>
>>
>> Mats Karlsson wrote:
>>
>> Nick,
>>
>> If the hazard of patients are dying is significantly (p<0.05) higher on
>>
> the
>
>> new treatment compared to reference, I don't think you need other evidence
>> before it has practical consequences. Without mechanistic understanding,
>> would you ignore it and move on to the next analysis?
>>
>> Mats
>>
>> Mats Karlsson, PhD
>> Professor of Pharmacometrics
>> Dept of Pharmaceutical Biosciences
>> Uppsala University
>> Box 591
>> 751 24 Uppsala Sweden
>> phone: +46 18 4714105
>> fax: +46 18 471 4003
>>
>>
>> -----Original Message-----
>> From: owner-nmusers
>>
> [mailto:owner-nmusers
>
>> Behalf Of Nick Holford
>> Sent: Tuesday, August 25, 2009 10:29 PM
>> To: nmusers
>> Subject: Re: [NMusers] What does convergence/covariance show?
>>
>> Mats,
>>
>> Thanks for stating more clearly what I tried to say before. Once again
>> -- I agree that OFV is not a measure of clinical importance. But it is
>> correlated with discernible differences in model predictions that may be
>> of clinical importance.
>>
>> A change of OFV of 5 in a survival model may well be useful to reject a
>> null hypothesis and point to some explanatory variable. There are
>> numerous 'statistically significant' findings in the clinical literature
>> like this that have no practical impact. You do not indicate what else
>> in the survival analysis convinced you that the OFV was associated with
>> something of practical consequence. I trust your decision was not based
>> only on the OFV!
>>
>> Nick
>>
>> Mats Karlsson wrote:
>>
>>
>> Nick,
>>
>>
>>
>> I agree that small changes (5-10) in OFV often are not practically
>>
>>
>>
>> important
>>
>>
>> and and big changes more often are. However, my point is that OFV is
>>
> not
>
>>
>>
>> the
>>
>>
>> right scale to judge importance. You should judge it on the
>>
> consequence of
>
>> you additional complexity to the model (the magnitude of the found
>>
> drug
>
>> effect/covariate/etc). Just the other day did I analyze survival data
>>
>>
>>
>> where
>>
>>
>> a small (5) change in OFV is of practical consequence.
>>
>>
>>
>> A true treatment effect of a certain size will improve the OFV in
>>
> relation
>
>> to the size of the dataset. The larger the data set, the larger the
>>
> change
>
>> in OFV. However, the estimate of the treatment effect does not change
>>
>> systematically with the size of the data set. The size of the
>>
> treatment
>
>> effect is what is more appropriate diagnostic for practical
>>
> consequences.
>
>> OFV we would use only to make sure that we have found the effect by
>>
>>
>>
>> chance.
>>
>>
>> Best regards,
>>
>> Mats
>>
>>
>>
>> Mats Karlsson, PhD
>>
>> Professor of Pharmacometrics
>>
>> Dept of Pharmaceutical Biosciences
>>
>> Uppsala University
>>
>> Box 591
>>
>> 751 24 Uppsala Sweden
>>
>> phone: +46 18 4714105
>>
>> fax: +46 18 471 4003
>>
>>
>>
>>
>>
>> -----Original Message-----
>>
>> From: owner-nmusers
>>
> <mailto:owner-nmusers
>
>>
>>
>> On
>>
>>
>> Behalf Of Nick Holford
>>
>> Sent: Tuesday, August 25, 2009 7:25 AM
>>
>> To: nmusers
>>
>> Subject: Re: [NMusers] What does convergence/covariance show?
>>
>>
>>
>> Mats,
>>
>>
>>
>> When I referred to a change of 50 being needed to detect something of
>>
>> practical importance I was not saying that was of clinical relevance.
>>
>> That cannot be judged from the OFV alone. But small OFV changes are
>>
>> rarely if ever indicators of something that is clinically relevant.
>>
>>
>>
>> I expect you will agree on this point :-)
>>
>>
>>
>> Nick
>>
>>
>>
>> Mats Karlsson wrote:
>>
>>
>>
>>
>>
>> Nick,
>>
>>
>>
>> I too would use OFV as the most important goodness-of-fit
>>
> diagnostic when
>
>> comparing models, especially when deeming something to be
>>
> redundant. If
>
>> adding a component doesn't reduce OFV, I see no reason to include
>>
> it (I
>
>> think we're agreeing on something!). However, you write
>>
>>
>>
>> " Small (5-10) changes in OBJ are not of much interest. A change
>>
> of OBJ
>
>>
>>
>> of
>>
>>
>> at least 50 is usually needed to detect anything of practical
>>
>>
>>
>> importance."
>>
>>
>> Today we use population methods for everything from very rich pop
>>
> pk
>
>> meta-analyses to very sparsely informative data sets on survival.
>>
> To use
>
>>
>>
>>
>>
>> OFV
>>
>>
>>
>>
>>
>> as a measure of goodness-of-fit is central and look at the risk
>>
> something
>
>> improved the fit by chance, but I would not use it as measure of
>>
> clinical
>
>> importance.
>>
>>
>>
>> Best regards,
>>
>> Mats
>>
>>
>>
>> Mats Karlsson, PhD
>>
>> Professor of Pharmacometrics
>>
>> Dept of Pharmaceutical Biosciences
>>
>> Uppsala University
>>
>> Box 591
>>
>> 751 24 Uppsala Sweden
>>
>> phone: +46 18 4714105
>>
>> fax: +46 18 471 4003
>>
>>
>>
>>
>>
>> -----Original Message-----
>>
>> From: owner-nmusers
>>
> <mailto:owner-nmusers
>
>>
>>
>>
>>
>> On
>>
>>
>>
>>
>>
>> Behalf Of Nick Holford
>>
>> Sent: Tuesday, August 25, 2009 12:14 AM
>>
>> To: nmusers
>>
>> Subject: Re: [NMusers] What does convergence/covariance show?
>>
>>
>>
>> Mats, Leonid,
>>
>>
>>
>> Thanks for your definitions. I think I prefer that provided by
>>
> Mats but
>
>> he doesn't say what his test for goodness-of-fit might be.
>>
>>
>>
>> Leonid already assumes that convergence/covariance are diagnostic
>>
> so it
>
>> doesnt help at all with an independent definition of
>>
>> overparameterization. Correlation of random effects is often a
>>
> very
>
>> important part of a model -- especially for future predictions --
>>
> so I
>
>> dont see that as a useful test -- unless you restrict it to
>>
> pathological
>
>> values eg. |correlation|>0.9?. Even with very high correlations I
>>
>> sometimes leave them in the model because setting the covariance
>>
> to zero
>
>> often makes quite a big worsening of the OBJ.
>>
>>
>>
>> My own view is that "overparameterization" is not a black and
>>
> white
>
>> entity. Parameters can be estimated with decreasing degrees of
>>
>> confidence depending on many things such as the design and the
>>
> adequacy
>
>> of the model. Parameter confidence intervals (preferably by
>>
> bootstrap)
>
>> are the way i would evaluate how well parameters are estimated. I
>>
>> usually rely on OBJ changes alone during model development with a
>>
> VPC
>
>> and boostrap confidence interval when I seem to have extracted all
>>
> I can
>
>> from the data. The VPC and CIs may well prompt further model
>>
> development
>
>> and the cycle continues.
>>
>>
>>
>> Nick
>>
>>
>>
>>
>>
>> Leonid Gibiansky wrote:
>>
>>
>>
>>
>>
>>
>>
>> Hi Nick,
>>
>>
>>
>> I am not sure how you build the models but I am using
>>
> convergence,
>
>> relative standard errors, correlation matrix of parameter
>>
> estimates
>
>> (reported by the covariance step), and correlation of random
>>
> effects
>
>> quite extensively when I decide whether I need extra
>>
> compartments,
>
>> extra random effects, nonlinearity in the model, etc. For me
>>
> they are
>
>> very useful as diagnostic of over-parameterization. This is
>>
> the direct
>
>> evidence (proof?) that they are useful :)
>>
>>
>>
>> For new modelers who are just starting to learn how to do it,
>>
> or have
>
>> limited experience, or have problems on the way, I would
>>
> advise to pay
>
>> careful attention to these issues since they often help me to
>>
> detect
>
>> problems. You seem to disagree with me; that is fine, I am not
>>
> trying
>
>> to impose on you or anybody else my way of doing the analysis.
>>
> This is
>
>> just an advise: you (and others) are free to use it or ignore
>>
> it :)
>
>>
>>
>> Thanks
>>
>> Leonid
>>
>>
>>
>>
>>
>>
>>
>> Mats Karlsson wrote:
>>
>>
>>
>>
>>
>>
>>
>> <<I would say that if you can remove parameters/model
>>
> components without
>
>> detriment to goodness-of-fit then the model is
>>
> overparameterized. >>
>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Nick Holford, Professor Clinical Pharmacology
>> Dept Pharmacology & Clinical Pharmacology
>> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New
>>
> Zealand
>
>> n.holford
>>
> tel:+64(9)923-6730 fax:+64(9)373-7090
>
>> mobile: +64 21 46 23 53
>> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
>>
>
>

--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
n.holford
mobile: +64 21 46 23 53
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Received on Wed Aug 26 2009 - 05:58:15 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.