NONMEM Users Network Archive

Hosted by Cognigen

RE: Funny behaviour with MCETA>1 and parallel computation

From: Bauer, Robert <Robert.Bauer>
Date: Wed, 30 Jul 2014 15:51:01 +0000

Paolo:
This may not be a matter of MCETA, it might have something to do with that particular individual’s data plus your model, and is there some unusual evaluation that can accidentally occur, causing the optimization for that subject to fail. Would you mind sharing with me your control stream file and data set, that I might give it a try.

Robert J. Bauer, Ph.D.
Vice President, Pharmacometrics, R&D
ICON Development Solutions
7740 Milestone Parkway
Suite 150
Hanover, MD 21076
Tel: (215) 616-6428
Mob: (925) 286-0769
Email: Robert.Bauer ilto:Robert.Bauer
Web: www.iconplc.com<http://www.iconplc.com/>

From: Paolo Denti [mailto:paolo.denti
Sent: Wednesday, July 30, 2014 3:54 AM
To: Bauer, Robert; nmusers
Subject: Re: [NMusers] Funny behaviour with MCETA>1 and parallel computation

Hi Bob,
thanks for the prompt response and suggestions.

I have tried to implement with RANMETHOD=P and increasing MCETA to 1000, but unfortunately without success.
Even using MCETA=1000, the result is the same: when I use the parallel computation feature, it performs worse than using MCETA=0. With one processor, things work as expected.

Maybe I did not explain the situation well. The issue is not that the optimisation takes a slightly different path and it reaches a different minimum, that would not surprise me, as I know different rounding and other random factor can influence that.
The problem here is that when using parallel computation even on the first iteration (using MAXEVAL=0) MCETA>1 gives a worse OFV that MCETA=0. This still does not make any sense to me, irrespectively of random number generators, numerical approximation,etc. My understanding is that, in each individual, NONMEM will try 0 and other initial estimates to find the optimal ETAs, and then it will choose the solution giving the lowest individual OFV. Even if this is done with different seeds and on different CPUs, they will all try 0, so whatever MCETA=0 gives out, it should be the upper bound for the OFV for that individual. Then all these individual OFVs are summed together to find the total OFV. Since NONMEM is trying 0 in each subject - plus other random values that may vary - it should at least be able to use those results. In each individual it can only do better by trying extra values, and if all the individual OFVs are lower or at worst the same as the ones provided by MCETA=0, then the total can only be better.

Am I missing something? Is NONMEM maybe minimising only some other individual likelihood in the MAP step, and that does not coincide with the individual OFV?

To better understand I have looked at the values of individual OFVs in the two runs (MCETA=1000 and MCETA=0, both with parallel computaion) and all the tables are exactly the same cell by cell (to the 5th digit or so), except for the records of that outlying individual. In spite of trying 1000 initial estimates (including 0) MCETA=1000 gets the individual ETA for that subject wrong, and gives a worse iOFV than MCETA=0. And it's not a matter of numerical noise, the individual OFV is 100 points worse and the individual parameters are very different..

MCETA=0 is a sub-case of MECTA>1, so MCETA>1 should not do any worse. I have tried and re-tried, and it is unlikely that the estimator was unlucky all the times with that subject, even trying 1000 initial estimates..

I don't know how to explain this. But maybe I am misunderstanding how this works?
Any explanation?

Thank you,
Paolo







On 2014/07/29 22:45, Bauer, Robert wrote:
Paolo:
MCETA=5 is a small number of random samples of etas to test, and when you parallelize, your random seed pattern changes, unless, you include $EST RANMETHOD=P. Yes, sometimes a worse result follows if a lower OBJ is chosen, yet those etas that gave a lower OBJ offered a worse path than when eta=0. I suggest the following:

$EST METHOD=1 INTERACTION MAXEVAL=0 MCETA=1000 RANMETHOD=P ... ; spend a lot of random samples doing a thorough search with MCETA=1000, but it takes time, so do it just for the first iteration.
$EST METHOD=1 INTERACTION MAXEVAL=9999 MCETA=5 RANMETHOD=P ... ; remaining iterations use MCETA more lightly.



Robert J. Bauer, Ph.D.
Vice President, Pharmacometrics, R&D
ICON Development Solutions
7740 Milestone Parkway
Suite 150
Hanover, MD 21076
Tel: (215) 616-6428
Mob: (925) 286-0769
Email: Robert.Bauer
Web: www.iconplc.com<http://www.iconplc.com>

-----Original Message-----
From: owner-nmusers er-nmusers
Sent: Tuesday, July 29, 2014 3:54 PM
To: nmusers
Subject: [NMusers] Funny behaviour with MCETA>1 and parallel computation

Dear all,
I am working with FOCE-INTERACTION and a PK model with some issues in optimising the individual ETAs.
A couple of subjects in the dataset are very fast absorbers and sometimes NONMEM fails to find the optimal value for their individual ETAs. This results in the OFV randomly jumping up a lot (~100 points) when I restart the model from the final estimates of a previous run. The model may sometimes find its way back to the same minimum, but not always.

So I decided to use the new feature MCETA, which is supposed to address these kinds of issues by trying several randomly generated sets of initial ETAs, increasing the chance of finding the best value of individual ETA for each subject.

I first tested the new feature on a single CPU and everything worked as
expected: using MCETA=0 (the default) the OFV jumps up after restarting with the final values, while with MCETA=5 the problem is solved and the OFV is stable.

I then ran the same test using the parallel computation feature to speed it up, but I noticed that here the OFV was jumping up anyway. Not only, using MCETA=5 was actually giving a WORSE OFV than the default value of MCETA=0, which is indeed giving the same result irrespectively of whether I use the parallel computation or not.
I have retried several times to make sure I hadn't made any mistakes and when I use MCETA>1 and the parallel computation feature, it sometimes perform worse than the default MCETA=0. I tried other values of MCETA too, but same story.
When I say that the OFV goes up, I am referring to the first iteration, so I have replicated these results using MAXEVAL=0.

Any suggestion about what may be going on? My understanding is that using MCETA>1 CANNOT be any worse than the default behaviour (MECTA=0) of just using 0 as initial value. The guide says NONMEM is supposed to try both 0 AND some other values as initial estimates, and choose in every subject the option that minimises the OFV. So in the worst case scenario NONMEM will just use 0 and I will have wasted some electrons.
Am I misunderstanding something?

Sorry for the lengthy email and thanks for any input on this.
Paolo


--
------------------------------------------------
Paolo Denti, PhD
Pharmacometrics Group
Division of Clinical Pharmacology
Department of Medicine
University of Cape Town

K45 Old Main Building
Groote Schuur Hospital
Observatory, Cape Town
7925 South Africa
phone: +27 21 404 7719
fax: +27 21 448 1989
email: paolo.denti to:paolo.denti
------------------------------------------------

________________________________
UNIVERSITY OF CAPE TOWN

This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.


ICON plc made the following annotations.
------------------------------------------------------------------------------
This e-mail transmission may contain confidential or legally privileged information that is intended only for the individual or entity named in the e-mail address. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or reliance upon the contents of this e-mail is strictly prohibited. If you have received this e-mail transmission in error, please reply to the sender, so that ICON plc can arrange for proper delivery, and then please delete the message.

Thank You,

ICON plc
South County Business Park
Leopardstown
Dublin 18
Ireland
Registered number: 145835



--

------------------------------------------------

Paolo Denti, PhD

Pharmacometrics Group

Division of Clinical Pharmacology

Department of Medicine

University of Cape Town



K45 Old Main Building

Groote Schuur Hospital

Observatory, Cape Town

7925 South Africa

phone: +27 21 404 7719

fax: +27 21 448 1989

email: paolo.denti .za<mailto:paolo.denti

------------------------------------------------
Received on Wed Jul 30 2014 - 11:51:01 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.