RE: OMEGA selection

From: Ken Kowalski <ken.kowalski>
Date: Tue, 21 Apr 2009 09:50:51 -0400

NMusers,

My apologies for entering into this discussion a bit late as I was on =
vacation last week. Rather than rehash previous debates about \$COV, I =
thought I would just list some of the ways I use the \$COV step output to =
assist my model building and clinical trial simulation efforts.

Before I do so, let me preface my comments by saying that for me the =
real diagnostic value of the \$COV step is in the output reported by the =
\$COV step and not simply whether or not \$COV runs successfully. Thus, I =
strive for a successful \$COV step because I find diagnostic value in the =
\$COV output to guide my model-building efforts.

There are 3 basic ways I use the \$COV step output:

1) Inspection of the standard errors, pairwise correlations among the =
parameter estimates, and the eigenvalue analysis of the correlation =
matrix helps me to understand the limitations of the design/data via the =
model.

2) I find building full covariate models much easier to obtain by first =
ensuring that I have a stable base model through an inspection of the =
\$COV step output. I tend to like to use the full model to make =
inference about the covariate parameter estimates (e.g., CIs) as they =
will not suffer from model selection bias which occurs with stepwise =
procedures.

3) Based on asymptotic statistical theory for maximum likelihood =
estimation I will often assume that the parameters estimates follow a =
multivariate normal distribution with mean vector set to the population =
parameter estimates and covariance matrix set to the covariance matrix =
of the parameter estimates for THETA, OMEGA and SIGMA reported in the =
\$COV output. This assumption allows me to easily generate random sets =
of population parameters reflecting parameter uncertainty when =
conducting clinical trial simulations. Of course one could do =
non-parametric bootstrapping to accomplish this as well but it is easier =
and faster to use the multivariate normal distribution when it is =
reasonable to assume that the asymptotics hold.

Below are examples that illustrate some of the ways I use the \$COV =
output:

• Identify largest standard errors relative to the point =
estimates and rationalize the limitations of the data/design that would =
give rise to these large SEs (e.g., a standard error for ka may be large =
if few sample times are observed prior to Tmax).

• Screen for high pairwise correlations. For example, a high =
correlation in the population parameter estimates for CL/F and V/F may =
result when fitting a base model to steady-state PK data. This would =
suggest that the same information in the data is being used to estimate =
both parameters. This can be problematic for building full covariate =
models where one or more covariates may have effects on both parameters. =
In this setting I may use clinical judgment as to whether a particular =
covariate effect is more likely to be on CL/F or V/F if the limitations =
of the design/data preclude estimating it on both.

• The covariance matrix of the estimates from a full model run =
are helpful in determining a subset of potential parsimonious final =
models using the WAM algorithm (see Kowalski & Hutmacher, JPP =
2001;28:253-275).

• I use SAS (or Splus) to generate a random set of population =
parameters from the multivariate normal distribution using the =
population parameter estimates and the covariance matrix of the =
estimates from the \$COV output in clinical trial simulations so that I =
can quantify operating characteristics such as probability of success =
(probability of a Go decision) and probability of a correct decision in =
contrast to power calculations which assume a fixed effect size. Power =
is a conditional probability (conditioning on an assumed effect =
magnitude) whereas POS (prob of success) is an unconditional probability =
that takes into account the uncertainty in achieving a given effect =
magnitude. Power is a performance characteristic of the design whereas =
POS is a performance characteristic of both the design and compound =
(dose of treatment).

Kind regards,

Ken

-----Original Message-----
From: owner-nmusers
On Behalf Of Nick Holford
Sent: Wednesday, April 15, 2009 2:49 PM
To: nmusers
Subject: Re: [NMusers] OMEGA selection

Mark,

I agree with your logic. In the meantime I will ignore the \$COV step (it =

rarely happens for me) and wait for some empirical evidence that the
\$COV step is of demonstrable value for model building. Perhaps your grid =

computing system could take on that challenge by comparing the results
of automated model building with and without \$COV or convergence?

Nick

Mark Sale - Next Level Solutions wrote:
>
> Nick et al.
> At this risk of starting an discussion that probably has little
> mileage left in it. First I agree with Nick on covariance - it
> probably doesn't matter. But, I'd like to point out what may be an
> error in our logic.
> We content that we have demonstrated that covariance doesn't matter.
> Our evidence is that, when bootstrapping, the parameters for the
> sample that have successful covariance are not different from those
> that failed. So, we conclude that the results are the same regardless =

> of covariance outcome across sampled data sets - the independent
> variable in this test is the data set, the model is fixed.
> In model selection/building, we have a fixed data set and the
> independent variable is the model structure. Whether covariance
> success is a useful predictor across different models with a fixed
> data set is a different question than whether covariance is a useful
> predictor across data sets with a fixed model.
> But, in the end, I do agree that biological plausibility, diagnostic
> plots, reasonable parameters and some suggestion of numerical
> stability/identifiably (such as bootstrap CIs) are more important than =

> a successful covariance step.
>
> Mark
>
> Mark Sale MD
> Next Level Solutions, LLC
> www.NextLevelSolns.com <http://www.NextLevelSolns.com>
> 919-846-9185
>
> -------- Original Message --------
> Subject: Re: [NMusers] OMEGA selection
> From: Nick Holford <n.holford
> Date: Wed, April 15, 2009 12:17 pm
> To: nmusers
>
> Ethan,
>
> Do not pay any attention to whether or not the \$COV step runs or
> even if
> the run is 'SUCCESSFUL' to conclude anything about your model. =
Your
> opinion is not supported experimentally e.g. see
> http://www.mail-archive.com/nmusers
for
> discussion and references.
>
> NONMEM has no idea if the parameters make sense or not and will
> happily
> converge with models that are overparameterised. You cannot rely =
on a
> failed \$COV step or a MINIMIZATION TERMINATED message to conclude =
the
> model is not a good one. You need to use your brains (NONMEM does =
not
> have a brain) and your common sense to decide if your model makes
> sense
> or is perhaps overparameterised.
>
> Nick
>
> Ethan Wu wrote:
> >
> > Dear all,
> >
> > I am fitting a PD response, and the equation goes like this:
> >
> > total response = baseline+f(placebo response) +f(drug =
response)
> >
> > first, I tried full omega block, and model was able to converge, =
but
> > \$COV stop failed.
> >
> > To me, this indicates that too many parameters in the model. The
> > structure model is rather simple one, so I think probably too
> many Etas.
> >
> > I wonder is there a good principle of Eta reduction that I could
> > implement here. Any good reference?
> >
> >
>
> --
> Nick Holford, Dept Pharmacology & Clinical Pharmacology
> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland,
> New Zealand
> n.holford
> mobile: +33 64 271-6369 (Apr 6-Jul 17 2009)
> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
>
>

--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New =
Zealand
n.holford
mobile: +33 64 271-6369 (Apr 6-Jul 17 2009)
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Received on Tue Apr 21 2009 - 09:50:51 EDT

The NONMEM Users Network is maintained by ICON plc. Requests to subscribe to the network should be sent to: nmusers-request@iconplc.com.

Once subscribed, you may contribute to the discussion by emailing: nmusers@globomaxnm.com.