NONMEM Users Network Archive

Hosted by Cognigen

unscientific poll

From: Leonid Gibiansky <LGibiansky>
Date: Wed, 03 Dec 2008 13:00:15 -0500

Dear All,
Here is the summary of the the replies for the questions that I sent out
recently. I received a total of 35 replies. This e-mail consists of 4
parts, as follows:

Part 1: Each original questions is followed by the summary of replies

Part 2: All comments that I received are copy-pasted after the summary
of replies.

Part 3: CSV file with the original data is copy-pasted after the comments

Part 4: R code that I used to summarize the results is provided

Thanks to all who participated.
Leonid

--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566

############ RESULTS ###########################

> 1. Would you like Nonmem to stop producing all run-time (not syntax)
> error/warning messages (134, 137, number of significant digits, etc.)
> and "MINIMIZATION SUCCESSFUL" messages (YES/NO):
      question YES No Missing
            Q1 2 (5.7%) 33 (94.3%) 0 (0%)

> 2. Do you remember at least one example when the run-time error
message helped you to find an error in your code (YES/NO):
     question YES No Missing
           Q2 31 (88.6%) 3 (8.6%) 1 (2.9%)

> 3. In your experience, run-time error messages allow you to detect
model errors or problems quicker than it would be done without error
messages: (agree/disagree)
     question AGREE No Missing
         Q3 27 (77.1%) 3 (8.6%) 5 (14.3%)

> 4. Have you ever used in your report/publication ANY model that did
not have $COV step completed (YES/NO):
     question YES No Missing
           Q4 25 (71.4%) 9 (25.7%) 1 (2.9%)

> 5. Have you ever used in your report/publication ANY model that did
not converge (YES/NO):
     question YES No Missing
            Q5 13 (37.1%) 21 (60%) 1 (2.9%)

> 6. Have you ever used in your report/publication FINAL model that did
> not have $COV step completed (YES/NO):
     question YES No Missing
           Q6 16 (45.7%) 18 (51.4%) 1 (2.9%)

> 7. Have you ever used in your report/publication FINAL model that did
> not converge (YES/NO):
     question YES No Missing
           Q7 3 (8.6%) 31 (88.6%) 1 (2.9%)

> 8. Define yourself as novice/intermediate/experienced Nonmem user:

Missing Novice Intermediate Experienced
  1 3 15 16


############ Comments ###########################

Honestly, nobody is proposing to remove the minimization successful
statement or any non syntax error message. I agree with the comments
that stress it's important to take them with a pinch of salt as these
"errors" not always point you in the right directions, and not always
being picky about $COV step or number of significant digits help in
selecting the best model

-------------------------
Question 1 has at least 3 parts and cannot be answered YES or NO in any
meaningful way. Please note the messages we have discussed are not ERROR
or WARNING messages. They are a message about the minimization status.

I would be happy if NONMEM stuck to the facts. It can tell me if it
achieved the requested sigdigs (CONVERGED) or ran out of function evals
(PREMATURE TERMINATION). But it should keep its subjective judgements to
itself.

If you made a list of run-time error messages and another of run-time
warning messages then perhaps your survey could be more helpful in
deciding which are meaningful?

----------------------

My 2 cents on the discussion, for what itís worthÖ The error message is
usually related to an error in the dataset / dataset programming issue /
initial estimates / model parameterization. Generally, all need to be
addressed. Just as important, I wouldnít trust a MINIMIZATION
SUCCESSFUL message either as you will often see flip-flop or 3-CMT
identifiability issues that NONMEM doesnít flag. NONMEM has its flaws,
but Iíve seen many of NONMEMís error messages disappear after correcting
a programming error in the dataset, refining the initial estimates, or
re-parameterizing the model. I think the practical pharmacometrician
isnít going to accept NONMEM outputs strictly at face value, but would
challenge the findings to confirm the results. Thatís just good science.
----------------------

I have never reported out as a final model a run that failed to converge
or failed the COV step. My guess is that individuals who frequently do
probably tend to be more mechanistic in their model building than I am
and often push the complexity of their models beyond what can be
supported by the data in hand. For those that do report out models that
don't converge, I wonder if they have tried re-running their models with
different starting values (15-20% different) and see if NONMEM fails to
converge at the same set of parameter estimates. My guess is in many
cases it won't although both sets of estimates may appear "reasonable"
and give similar fits and VPC.

For individuals who have strong prior beliefs about their mechanistic
models, my thinking is that rather than using approximate maximum
likelihood methods and ignoring the diagnostics that might suggest their
model is unstable or not fully supported by the data, I think they would
be better served by using a Bayesian approach. That way they can be
explicit about the strength of their priors and they don't have to worry
about convergence and COV step failures. JMHO.

------------------

############## Original Data ##################

ID,Q1,Q2,Q3,Q4,Q5,Q6,Q7,Q8
1,0,0,-1,1,1,1,0,1
2,0,1,1,1,0,1,0,3
3,0,1,-1,1,0,1,0,2
4,0,1,-1,1,1,1,1,3
5,0,0,-1,1,0,1,0,2
6,0,1,-1,1,0,1,0,3
7,0,1,1,0,0,0,0,3
8,1,1,0,0,0,0,0,3
9,0,1,1,0,0,0,0,2
10,0,1,1,1,0,0,0,-1
11,0,1,1,1,0,1,0,3
12,0,0,1,0,0,0,0,2
13,0,1,1,1,1,1,0,2
14,0,1,1,0,0,0,0,3
15,0,1,1,1,1,1,0,3
16,1,-1,0,1,1,1,0,2
17,0,1,1,-1,-1,-1,-1,1
18,0,1,1,0,0,0,0,3
19,0,1,1,1,0,0,0,3
20,0,1,1,1,0,0,0,3
21,0,1,1,1,1,0,0,1
22,0,1,1,0,0,0,0,2
23,0,1,0,1,1,1,0,3
24,0,1,1,1,1,0,0,2
25,0,1,1,1,1,0,0,3
26,0,1,1,0,0,0,0,2
27,0,1,1,1,0,0,0,2
28,0,1,1,1,1,1,0,2
29,0,1,1,1,0,1,0,3
30,0,1,1,1,0,0,0,3
31,0,1,1,1,1,1,1,2
32,0,1,1,1,1,1,1,3
33,0,1,1,1,1,1,0,2
34,0,1,1,1,0,0,0,2
35,0,1,1,0,0,0,0,2


############## R code ############################

raw.data <- read.table("C:/poll.csv",sep=",",header=T)
res <- NULL
for(Qname in paste("Q",1:7,sep="") ){
     x <- raw.data[,Qname]
     n.yes <- sum(x == 1)
     n.no <- sum(x == 0)
     n.na <- sum(x == -1)
     n <- length(x)
    temp <- data.frame(n=n,question=Qname,
                   YES=paste(n.yes," (",round(100*n.yes/n,1),"%)",sep=""),
                   No=paste(n.no," (",round(100*n.no/n,1),"%)",sep=""),
                   Missing=paste(n.na,"
(",round(100*n.na/n,1),"%)",sep=""))
    res <- rbind(res,temp)
}
res
table(raw.data$Q8)

##########################################################

Received on Wed Dec 03 2008 - 13:00:15 EST

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.