NONMEM Users Network Archive

Hosted by Cognigen

RE: [NMusers] Humongous log file with parallel NONMEM

From: <E.Olofsen_at_lumc.nl>
Date: Mon, 22 Jun 2015 19:23:54 +0000

Dear Devin,

The I/O to the null device is quite efficient, because it is not related to=
 hardware. Write protect the log file gives an error indeed.

Best regards,

Erik
________________________________
From: Devin Pastoor [devin.pastoor_at_gmail.com]
Sent: Monday, June 22, 2015 8:58 PM
To: Olofsen, E. (ANST); ron_at_pirana-software.com; bgillespiester_at_gmail.com
Cc: nmusers_at_globomaxnm.com
Subject: Re: [NMusers] Humongous log file with parallel NONMEM

Ron and Erik,

The problem with those solutions is they don't actually address the I/O con=
cern. While you will save some disk space at any given time (which I assume=
 shouldn't be an issue for Bill on AWS) , you are actually increasing the I=
/O burden (eg now you have to deal with creation and deletion). IF disk I/=
O is a bottleneck, a better solution would be to delete the file only when =
the file reaches a critical size, or wait to some other time when there is =
lower I/O burden, perhaps at the end of the run with a post-run hook.

Bill, a little bit of an out there solution (maybe) could be to create a du=
mmy file with the same name as the log file, then write/append protect it, =
so even if nonmem attempts to overwrite/append it will not be able to. I do=
n't know how much nonmem would complain trying to write to that file but be=
ing unable to. It'd be pretty hacky but could be worth a shot depending on =
how much I/O is a bottleneck.


Devin

On Mon, Jun 22, 2015 at 2:42 PM <E.Olofsen_at_lumc.nl<mailto:E.Olofsen_at_lumc.nl=
>> wrote:
Dear Bill and Ron,

I was thinking about:

$ ln -s /dev/null <control_stream>.log

(After deleting the log if one already exists)

Data written to the log file is actually written to the null device, which =
discards the data - two tests seem to indicate that it could work.

Best regards,

Erik
________________________________
From: owner-nmusers_at_globomaxnm.com<mailto:owner-nmusers_at_globomaxnm.com> [ow=
ner-nmusers_at_globomaxnm.com<mailto:owner-nmusers_at_globomaxnm.com>] on behalf =
of Ron Keizer [ron_at_pirana-software.com<mailto:ron_at_pirana-software.com>]
Sent: Monday, June 22, 2015 7:32 PM
To: Bill Gillespie
Cc: nmusers_at_globomaxnm.com<mailto:nmusers_at_globomaxnm.com>
Subject: Re: [NMusers] Humongous log file with parallel NONMEM

hi Bill,

a simple hack is to delete the file continually, by running the following c=
ommand before you start NONMEM:

watch -n 60 'find . -name *.log -delete' &>/dev/null &

Notes:
- will delete all log files every minute
- I'm using 'find ...' instead of just 'rm *.log' here to ensure that log-f=
iles in subfolders will also be deleted, e.g. useful when using via PsN
- run the command only once, the watch process will stay active
- only works on linux

best regards,
Ron


----------------------------------------------
Ron Keizer, PharmD PhD
Pirana Software & Consulting BV
California / the Netherlands
www.pirana-software.com<http://www.pirana-software.com/>
----------------------------------------------

On Mon, Jun 22, 2015 at 6:22 AM, Bill Gillespie <bgillespiester_at_gmail.com<m=
ailto:bgillespiester_at_gmail.com>> wrote:
Hi all,

I'm running NONMEM (METHOD = BAYES) in parallel on 32 cores and it genera=
tes a humongous log file with repeated entries like the following:

 ITERATION -577
 STARTING SUBJECTS 1 TO 4 ON MANAGER: OK
 STARTING SUBJECTS 5 TO 8 ON WORKER1: OK
 STARTING SUBJECTS 9 TO 11 ON WORKER2: OK
 STARTING SUBJECTS 12 TO 15 ON WORKER3: OK
 STARTING SUBJECTS 16 TO 18 ON WORKER4: OK
 STARTING SUBJECTS 19 TO 20 ON WORKER5: OK
 STARTING SUBJECTS 21 TO 24 ON WORKER6: OK
 STARTING SUBJECTS 25 TO 27 ON WORKER7: OK
 STARTING SUBJECTS 28 TO 29 ON WORKER8: OK
 STARTING SUBJECTS 30 TO 32 ON WORKER9: OK
 STARTING SUBJECTS 33 TO 35 ON WORKER10: OK
 STARTING SUBJECTS 36 TO 39 ON WORKER11: OK
 STARTING SUBJECTS 40 TO 42 ON WORKER12: OK
 STARTING SUBJECTS 43 TO 46 ON WORKER13: OK
 STARTING SUBJECTS 47 TO 50 ON WORKER14: OK
 STARTING SUBJECTS 51 TO 53 ON WORKER15: OK
 STARTING SUBJECTS 54 TO 58 ON WORKER16: OK
 STARTING SUBJECTS 59 TO 62 ON WORKER17: OK
 STARTING SUBJECTS 63 TO 66 ON WORKER18: OK
 STARTING SUBJECTS 67 TO 70 ON WORKER19: OK
 STARTING SUBJECTS 71 TO 71 ON WORKER20: OK
 STARTING SUBJECTS 72 TO 74 ON WORKER21: OK
 STARTING SUBJECTS 75 TO 77 ON WORKER22: OK
 STARTING SUBJECTS 78 TO 80 ON WORKER23: OK
 STARTING SUBJECTS 81 TO 84 ON WORKER24: OK
 STARTING SUBJECTS 85 TO 86 ON WORKER25: OK
 STARTING SUBJECTS 87 TO 88 ON WORKER26: OK
 STARTING SUBJECTS 89 TO 90 ON WORKER27: OK
 STARTING SUBJECTS 91 TO 93 ON WORKER28: OK
 STARTING SUBJECTS 94 TO 96 ON WORKER29: OK
 STARTING SUBJECTS 97 TO 99 ON WORKER30: OK
 STARTING SUBJECTS 100 TO 103 ON WORKER31: OK
 COLLECTING SUBJECTS 1 TO 4 ON MANAGER
 COLLECTING SUBJECTS 5 TO 8 ON WORKER1
 COLLECTING SUBJECTS 9 TO 11 ON WORKER2
 COLLECTING SUBJECTS 12 TO 15 ON WORKER3
 COLLECTING SUBJECTS 16 TO 18 ON WORKER4
 COLLECTING SUBJECTS 19 TO 20 ON WORKER5
 COLLECTING SUBJECTS 21 TO 24 ON WORKER6
 COLLECTING SUBJECTS 25 TO 27 ON WORKER7
 COLLECTING SUBJECTS 28 TO 29 ON WORKER8
 COLLECTING SUBJECTS 30 TO 32 ON WORKER9
 COLLECTING SUBJECTS 33 TO 35 ON WORKER10
 COLLECTING SUBJECTS 36 TO 39 ON WORKER11
 COLLECTING SUBJECTS 40 TO 42 ON WORKER12
 COLLECTING SUBJECTS 43 TO 46 ON WORKER13
 COLLECTING SUBJECTS 47 TO 50 ON WORKER14
 COLLECTING SUBJECTS 51 TO 53 ON WORKER15
 COLLECTING SUBJECTS 54 TO 58 ON WORKER16
 COLLECTING SUBJECTS 59 TO 62 ON WORKER17
 COLLECTING SUBJECTS 63 TO 66 ON WORKER18
 COLLECTING SUBJECTS 67 TO 70 ON WORKER19
 COLLECTING SUBJECTS 71 TO 71 ON WORKER20
 COLLECTING SUBJECTS 72 TO 74 ON WORKER21
 COLLECTING SUBJECTS 75 TO 77 ON WORKER22
 COLLECTING SUBJECTS 78 TO 80 ON WORKER23
 COLLECTING SUBJECTS 81 TO 84 ON WORKER24
 COLLECTING SUBJECTS 85 TO 86 ON WORKER25
 COLLECTING SUBJECTS 87 TO 88 ON WORKER26
 COLLECTING SUBJECTS 89 TO 90 ON WORKER27
 COLLECTING SUBJECTS 91 TO 93 ON WORKER28
 COLLECTING SUBJECTS 94 TO 96 ON WORKER29
 COLLECTING SUBJECTS 97 TO 99 ON WORKER30
 COLLECTING SUBJECTS 100 TO 103 ON WORKER31

The result is a lot of disk I/O and a file in the GB+ range. It dwarfs the =
file containing the MCMC samples. Is there some way to suppress that file o=
r reduce what gets written to it?

Thanks,
Bill

William R Gillespie, VP Strategic Modeling & Simulation
Metrum Research Group LLC
2 Tunxis Road, Tariffville, CT 06081
Direct & FAX: 919-371-2786<tel:919-371-2786>, Main: 860-735-7043<tel:860-73=
5-7043>
billg_at_metrumrg.com<mailto:billg_at_metrumrg.com>
www.metrumrg.com<http://www.metrumrg.com/>


Received on Mon Jun 22 2015 - 15:23:54 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request_at_iconplc.com. Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.