NONMEM Users Network Archive

Hosted by Cognigen

RE: Validation Strategy for NONMEM

From: Joachim.Grevel
Date: Fri, 24 Oct 2008 08:09:11 +0200

Dear Mark,

In our risk analysis we had to rank the associated risk. Just to give you
a feel for what we came up with:

Low risk: Reproducibility of results as long as control stream and
data file don't change. Even with the most unstable models, and the
largest data files this risk remains low.

High risk: Model misinterpretation (also misspecification) fall into =

that category. For example, you have a series of models which satisfy you
numerically (successful termination, small standard errors of estimation,
acceptable shrinkage, nice goodness of fit, etc.), but the supposed lesson =

you learned from the modeling, and the message you project to the world
(outside your modeling group) is either misleading or misunderstood.

Now, how do you deal with this 'high' risk? Measures to minimize that risk =

include: training of users, training of your customers, review of modeling =

results and conclusions, continued education (which involves also this
user network). Indeed, the NONMEM user network is listed in our risk
analysis as a means to mitigate.

Thank you for your continued interest,

Joachim
_________________________=
_________________
Joachim GREVEL, Ph.D.
MERCK SERONO International S.A.
Exploratory Medicine
1202 Geneva
Tel: +41.22.414.4751
Fax: +41.22.414.3059
Email: joachim.grevel





"Vilicich, Mark" <Mark.Vilicich
Sent by: owner-nmusers
10/23/2008 08:35 PM

To
<nmusers
cc

Subject
RE: [NMusers] Validation Strategy for NONMEM






Thank you for everyone's insight into this.
 
The insight into performance considerations is excellent Joachim and Jun,
thank you.
 
It would be great to hear from you or anyone else with experience running
NONMEM on a VMware environment as we are considering it. Since we are
starting with a very small set of users, it offers the ability to dedicate =

greater CPU power and memory to NONMEM as our analytic needs grow over
time.
 
Regarding validation; requiring proof of reproducible results seems
appropriate and Jeff's mention of developing a level of "confidence in the =

performance of a NONMEM installation" alludes to the risk based approach
in my mind and helps guide validation efforts to where the value is.
 
How could one generically describe a spectrum of testing levels for NONMEM =

based on a spectrum of risk such as-high, medium, low? What would be the
minimum tests done for a base level of confidence versus what testing is
included to support greater confidence (i.e. the high risk situation) in
reproducibility? Do NONMEM's or NMQUAL's standard provided testing cover
that spectrum or just the base level of confidence?
 
Thanks in advance for your responses,
 
Mark
From: owner-nmusers
On Behalf Of Joachim.Grevel
Sent: Wednesday, October 22, 2008 7:44 AM
To: nmusers
Subject: Re: [NMusers] Validation Strategy for NONMEM


Dear Jun,

the outside reference used: NONMEM (Version VI) with Wings for NONMEM;
installed using NMQUAL version 6.3.2 (Metrum Institute); Intel Visual
Fortran compiler (Version 10.1 ) or GNU Fortran g77 (Version 3.1 release
20020514); compiler options for Intel Fortran were /nologo /nbs /w
/4Yportlib /Gs /Ob1gyti /Qprec_div. ; compiler options for g77 were
-fno-backslash ?O; hardware: Intel Core Duo 7500 2.19GHz processor on an
IBM Thinkpad Model T61 with 1 Gb RAM. The operating system was Windows XP
Professional 2002 Service Pack 3.
We internally used: NONMEM (Version VI,2) PsN (Version 2.2.5) installed
without NMQual; gfortran compiler with option -O ; hardware: HP PROLIANT
DL585 G2 Servers with many cores (speed 2,8 GHz) with 16 Gb RAM and the
LINUX (SUSE Enterprise Edition 10 SP1 64 Bit) operating system with SUN
Grid engine (N1).
We reproduced the reference models well. Speed was equal with ANVAN6 (long =

runtimes in the hours). But with precompiled models ADVAN4 and with naked
PRED the Intel compiler was sometimes twice as fast (runtimes < 1 min).
Hope this is helpful for some of you about to make decisions about hard-
and software.
Joachim

_________________________=
_________________
Joachim GREVEL, Ph.D.
MERCK SERONO International S.A.
Exploratory Medicine
1202 Geneva
Tel: +41.22.414.4751
Fax: +41.22.414.3059
Email: joachim.grevel




"Jun Shen" <jun.shen.ut
10/22/2008 04:08 PM


To
Joachim.Grevel
cc
nmusers
Subject
Re: [NMusers] Validation Strategy for NONMEM








Dear Joachim,

Can you share some of your findings? What system configuration seems to be =

optimal (hardware/OS system, compiler, any aiding software)? Thanks.

Jun

On Wed, Oct 22, 2008 at 1:21 AM, <Joachim.Grevel

Dear all,

one of the more useful things we did was an exchange of data files and
models with a prominent member of our modeling community. This way we
challenged various aspects of our installation, found some soft spot,
corrected it, and wrote a document that is worth more than the ink and the =

paper. Yet again, all we did was assuring that our results match those of
a number of other installations (different compilers, operating systems,
processors). Furthermore, we got an idea of the comparative speed of our
installation. These are useful things to have documented when you are a
system administrator and modeler at the same time.

Joachim

_________________________=
_________________
Joachim GREVEL, Ph.D.
MERCK SERONO International S.A.
Exploratory Medicine
1202 Geneva
Tel: +41.22.414.4751
Fax: +41.22.414.3059
Email: joachim.grevel



"A.J. Rossini" <blindglobe
10/21/2008 10:41 PM


To
Michael.J.Fossler
cc
Joachim.Grevel
Subject
Re: [NMusers] Validation Strategy for NONMEM










There is useless validation, and then there is useful validation. The
latter is about making sure your computational results are
reproducible, the former is about making sure that your documentation
can be photocopied. It's sort of the same thing, if you aren't a
modeler.

Unfortunately, most on this list tend to be modelers.

On Tue, Oct 21, 2008 at 7:37 PM, <Michael.J.Fossler
>
> After reading this, it is no wonder scientific productivity is at an
> all-time low. Imagine if Marie Curie had to qualify her radium-purifying
> equipment, or if Alexander Fleming had to validate his petri dishes
before
> culturing Penecillium. One day scientists are going to push back against
> these IT people, who just make busy work for everyone.
>
>
>
> Joachim.Grevel
> Sent by: owner-nmusers
>
> 21-Oct-2008 03:18
>
>
> To
> nmusers
> cc
> Subject
> Re: [NMusers] Validation Strategy for NONMEM
>
>
>
>
>
> Dear Mark,
>
> I am engaged in this question since the beginning of this year (not
finished
> yet), and I am happy to share some basics of my experiences:
>
> 1. It is important to specify where the data for NONMEM analysis
come
> from. If they come from a GCP source and are already QA-ed when they
arrive
> at the doorstep of NONMEM then your system will only subject to GCP
> regulation. Otherwise, you also have to comply with GLP.
>
> 2. There is no way around what is called here a 'Validation Plan' =

and
> a 'Risk Analysis'. These documents will trigger a slate of other
documents
> (in our case here about 15) which describe Installation, Installation
> Validation, Qualification of Users, Modeling Strategy, Review Processes,
> System Life Cycle Management etc.
>
> 3. We found it useful to differentiate between 'Exploratory Work' =

and
> 'Submission Work'.
>
> 4. Before you worry about passing inspection by the FDA, you need =

to
> worry about passing inspection of your own company QA officers.
>
> 5. Just installing NONMEM with NMQual does not render you new
system
> 'validated' or 'qualified'. Here my apologies to the excellent folks at
> Metrum, but for various reasons, we ended up not using NMQual.
>
> 6. You have to know what you are trying to build before you
concern
> yourself about QA processes. A number of separate installations on PCs
> linked to a file server is a different animal from a server-based
> installation with a grid engine.
>
> 7. It all takes more time than you think: make generous budgets
and
> time lines.
>
> I hope I helped more than I confused,
>
> Joachim
>
> ________________________=
__________________
> Joachim GREVEL, Ph.D.
> MERCK SERONO International S.A.
> Exploratory Medicine
> 1202 Geneva
> Tel: +41.22.414.4751
> Fax: +41.22.414.3059
> Email: joachim.grevel
>
>
>
> "Vilicich, Mark" <Mark.Vilicich
> Sent by: owner-nmusers
>
> 10/17/2008 10:08 PM
>
> To
> <nmusers
> cc
> Subject
> [NMusers] Validation Strategy for NONMEM
>
>
>
>
>
>
> Dear All,
>
> I am interested in perspectives on strategies for "validating" NONMEM.
Also,
> experiences from or with the FDA since the FDA is: a key user, customer
of
> analysis and auditor of NONMEM use in the industry. Without a large
nonmem
> staff here, the challenge I see is in scaling the validation strategy to
> provide the most efficient environment for doing analysis that is
defensible
> to both internal and external audits based on the associated GxP risk
level.
>
> Below are the concepts I've cobbled together, though instead of my
> reinventing the wheel I appreciate anything you could share. Any and all
> gems of insight you can share whether it regard the big picture or some
> detailed specifics, IT centric or business process related. You may send
> them back to the listserver or me directly as you feel appropriate.
>
> Details:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> From searching the archives and other random bits of knowledge on
NONMEM,
> part of the validation strategy is to recognize that NONMEM is not to be
> literally validated. NONMEM may be considered more of a development
> environment, optimized for developing specialized forms of complex
analysis
> and modeling. As a development platform, an approach could be that
NONMEM
> itself is qualified and each specific analysis is validated
individually.
>
> To support establishing a defensible NONMEM environment, I've also read
> discussions on integrating common software development best practices
such
> as version control of the "programming" of nonmem, NMQual and other
> commercial and custom tools for capturing all the metadata related to
> running a specific NONMEM job. These themes support defining the state
of
> the NONMEM environment and ability to reproduce the outcomes.
>
> Also, reading in the archives about the differences in the numeric
outcomes
> of NONMEM analyses based on the hardware platform, etc. are helpful to
know
> up front and to consider in the validation strategy so it is not
destined to
> failure if the target environment is multiplatform or otherwise complex.
>
> Gaps noticed/topics not discussed:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Is there opportunity in looking at the risk based approached sanctioned
by
> the FDA a few years ago that would make the total validation
deliverable,
> including both the application and the model development process, more
lean
> and targeted at the primary risk targets?
>
> Does this scientific software environment lend itself to use of modern
agile
> software development methodologies that go far beyond basic iterative
> approaches. These methodologies are being used in software development
for
> the regulated/GxP industry.
>
> I've seen the excellent presentation from 2004 that Joga Gobburu from
the
> FDA gave, seems like there has been some progression of thought or
actions
> on the proposals included there. Any references to follow-up information =

on
> it would be helpful?
>
> Regards ,
>
> Mark Vilicich
> Early Development
> mark.vilicich
>
> ________________________=
________
>
> This message and any attachment are confidential, may be privileged or
> otherwise protected from disclosure and are intended only for use by the
> addressee(s) named herein. If you are not the intended recipient, you
must
> not copy this message or attachment or disclose the contents to any
other
> person. If you have received this transmission in error, please notify
the
> sender immediately and delete the message and any attachment from your
> system. Merck Serono does not accept liability for any omissions or
errors
> in this message which may arise as a result of E-Mail-transmission or
for
> damages resulting from any unauthorized changes of the content of this
> message and any attachment thereto. If verification is required, please
> request a hard-copy version. Merck Serono does not guarantee that this
> message is free of viruses and does not accept liability for any damages
> caused by any virus transmitted therewith.



--
best,
-tony

blindglobe
Muttenz, Switzerland.
"Commit early,commit often, and commit in a repository from which we
can easily roll-back your mistakes" (AJR, 4Jan05).

Drink Coffee: Do stupid things faster with more energy!



Received on Fri Oct 24 2008 - 02:09:11 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.