NONMEM Users Network Archive

Hosted by Cognigen

RE: Models that abort before convergence Addendum

From: Ken Kowalski <ken.kowalski>
Date: Fri, 21 Nov 2008 16:39:53 -0500

Leonid,

I have never reported out as a final model a run that failed to converge =
or failed the COV step. My guess is that individuals who frequently do =
probably tend to be more mechanistic in their model building than I am =
and often push the complexity of their models beyond what can be =
supported by the data in hand. For those that do report out models that =
don't converge, I wonder if they have tried re-running their models with =
different starting values (15-20% different) and see if NONMEM fails to =
converge at the same set of parameter estimates. My guess is in many =
cases it won't although both sets of estimates may appear "reasonable" =
and give similar fits and VPC.

For individuals who have strong prior beliefs about their mechanistic =
models, my thinking is that rather than using approximate maximum =
likelihood methods and ignoring the diagnostics that might suggest their =
model is unstable or not fully supported by the data, I think they would =
be better served by using a Bayesian approach. That way they can be =
explicit about the strength of their priors and they don't have to worry =
about convergence and COV step failures. JMHO.

Ken

Kenneth G. Kowalski
President & CEO
A2PG - Ann Arbor Pharmacometrics Group, Inc.
110 E. Miller Ave., Garden Suite
Ann Arbor, MI 48104
Work: 734-274-8255
Cell: 248-207-5082
Fax: 734-913-0230
ken.kowalski



-----Original Message-----
From: owner-nmusers
On Behalf Of Leonid Gibiansky
Sent: Friday, November 21, 2008 3:53 PM
To: Mark Sale - Next Level Solutions
Cc: nmusers
Subject: Re: [NMusers] Models that abort before convergence Addendum

Mark,
"Useful" is the relative and subjective term. Error messages and
convergence information are useful to me (i.e., they make my search of
the final model more efficient), and I'd like to understand whether they =

are useful to other people. I do not try to prove that the model
completed without error messages is correct, or that the model completed =

with rounding error is wrong, or whether the error messages provide
information not readily available in NPC, NPDE and PPC. I am interested
to see how many people find it useful: full stop here, do not try to
interpret the poll beyond this simple statement. In addition, questions
4-7 will help us to understand how widespred is the use of models with
failed convergence step and/or with failed minimization step.
Thanks
Leonid

--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566




Mark Sale - Next Level Solutions wrote:
>
> Leonid,
> Let me understand:
> You now have a theory that the way to determine whether the NONMEM =
error
> messages are useful (i.e., they tell you something about the model
> "goodness") is a poll. This, I think is a theory (and one well
> established in epistomolgy) of how to find an optimal solution - =
appeal
> to a large number of presumably well informed people. As data that =
may
> be relavant to this theory, I would point out that a poll gave us GW
> Bush as our 43rd president.
> Nick, in contrast has suggested that the error messages could be used =

> as a source of random numbers. This also, I think, is a theory =
without
> data to support or contradict it.
> So ....
> Let me propose a solution - let's generate some data. Suppose we
> randomly generate 1000 models. We could tests the hypotheses:
>
> Are the error messages random (I suspect they are not, that there is
> some information in them). To test this, see if the error messages =
are
> predictive of other (presumably non-random) measure of goodness - NPC
> and NPDE, and perhaps PPC come to mind.
>
> Do the error messages provide information not readily available in =
NPC,
> NPDE and PPC.
> Not really sure how to test this, without some "gold standard" of
> goodness, except perhaps to compare the different measures to the =
model
> that was used to simulate the data (seems like measures based on that
> would be "correct" in some way??). I need some ideas on this.
>
>
> I can generate, run and extract results from random models (using the =
GA
> software) - I already have NPDE and PPC in it, was thinking of adding =
NPC.
>
> Any interest/collaborators??
>
>
>
>
> Mark Sale MD
> Next Level Solutions, LLC
> www.NextLevelSolns.com <http://www.NextLevelSolns.com>
> 919-846-9185
>
> -------- Original Message --------
> Subject: Re: [NMusers] Models that abort before convergence =
Addendum
> From: Leonid Gibiansky <LGibiansky
> Date: Thu, November 20, 2008 9:57 pm
> To:
> Cc: nmusers <nmusers
>
> Nick, Mark, and All,
> We can argue indefinitely, but let me propose a poll. If you like =
to
> participate, reply directly to me (use "reply", not "reply to =
all"). I
> will summarize all the replies received up to the end of November. =
Skip
> the questions that you do not like to answer, write NA if the =
question
> is not applicable. Summaries will be blinded.
>
> 1. Would you like Nonmem to stop producing all run-time (not =
syntax)
> error/warning messages (134, 137, number of significant digits, =
etc.)
> and "MINIMIZATION SUCCESSFUL" messages (YES/NO):
>
> 2. Do you remember at least one example when the run-time error =
message
> helped you to find an error in your code (YES/NO):
>
> 3. In your experience, run-time error messages allow you to detect
> model
> errors or problems quicker than it would be done without error
> messages:
> (agree/disagree)
>
> 4. Have you ever used in your report/publication ANY model that =
did not
> have $COV step completed (YES/NO):
>
> 5. Have you ever used in your report/publication ANY model that =
did not
> converge (YES/NO):
>
> 6. Have you ever used in your report/publication FINAL model that =
did
> not have $COV step completed (YES/NO):
>
> 7. Have you ever used in your report/publication FINAL model that =
did
> not converge (YES/NO):
>
> 8. Define yourself as novice/intermediate/experienced Nonmem user:
>
> Thanks
> Leonid
>
> --------------------------------------
> Leonid Gibiansky, Ph.D.
> President, QuantPharm LLC
> web: www.quantpharm.com <http://www.quantpharm.com>
> e-mail: LGibiansky at quantpharm.com
> tel: (301) 767 5566
>
>
Received on Fri Nov 21 2008 - 16:39:53 EST

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.