From: Ken Kowalski <*ken.kowalski*>

Date: Tue, 21 Apr 2009 09:50:51 -0400

NMusers,

My apologies for entering into this discussion a bit late as I was on =

vacation last week. Rather than rehash previous debates about $COV, I =

thought I would just list some of the ways I use the $COV step output to =

assist my model building and clinical trial simulation efforts.

Before I do so, let me preface my comments by saying that for me the =

real diagnostic value of the $COV step is in the output reported by the =

$COV step and not simply whether or not $COV runs successfully. Thus, I =

strive for a successful $COV step because I find diagnostic value in the =

$COV output to guide my model-building efforts.

There are 3 basic ways I use the $COV step output:

1) Inspection of the standard errors, pairwise correlations among the =

parameter estimates, and the eigenvalue analysis of the correlation =

matrix helps me to understand the limitations of the design/data via the =

model.

2) I find building full covariate models much easier to obtain by first =

ensuring that I have a stable base model through an inspection of the =

$COV step output. I tend to like to use the full model to make =

inference about the covariate parameter estimates (e.g., CIs) as they =

will not suffer from model selection bias which occurs with stepwise =

procedures.

3) Based on asymptotic statistical theory for maximum likelihood =

estimation I will often assume that the parameters estimates follow a =

multivariate normal distribution with mean vector set to the population =

parameter estimates and covariance matrix set to the covariance matrix =

of the parameter estimates for THETA, OMEGA and SIGMA reported in the =

$COV output. This assumption allows me to easily generate random sets =

of population parameters reflecting parameter uncertainty when =

conducting clinical trial simulations. Of course one could do =

non-parametric bootstrapping to accomplish this as well but it is easier =

and faster to use the multivariate normal distribution when it is =

reasonable to assume that the asymptotics hold.

Below are examples that illustrate some of the ways I use the $COV =

output:

• Identify largest standard errors relative to the point =

estimates and rationalize the limitations of the data/design that would =

give rise to these large SEs (e.g., a standard error for ka may be large =

if few sample times are observed prior to Tmax).

• Screen for high pairwise correlations. For example, a high =

correlation in the population parameter estimates for CL/F and V/F may =

result when fitting a base model to steady-state PK data. This would =

suggest that the same information in the data is being used to estimate =

both parameters. This can be problematic for building full covariate =

models where one or more covariates may have effects on both parameters. =

In this setting I may use clinical judgment as to whether a particular =

covariate effect is more likely to be on CL/F or V/F if the limitations =

of the design/data preclude estimating it on both.

• The covariance matrix of the estimates from a full model run =

are helpful in determining a subset of potential parsimonious final =

models using the WAM algorithm (see Kowalski & Hutmacher, JPP =

2001;28:253-275).

• I use SAS (or Splus) to generate a random set of population =

parameters from the multivariate normal distribution using the =

population parameter estimates and the covariance matrix of the =

estimates from the $COV output in clinical trial simulations so that I =

can quantify operating characteristics such as probability of success =

(probability of a Go decision) and probability of a correct decision in =

contrast to power calculations which assume a fixed effect size. Power =

is a conditional probability (conditioning on an assumed effect =

magnitude) whereas POS (prob of success) is an unconditional probability =

that takes into account the uncertainty in achieving a given effect =

magnitude. Power is a performance characteristic of the design whereas =

POS is a performance characteristic of both the design and compound =

(dose of treatment).

Kind regards,

Ken

-----Original Message-----

From: owner-nmusers

On Behalf Of Nick Holford

Sent: Wednesday, April 15, 2009 2:49 PM

To: nmusers

Subject: Re: [NMusers] OMEGA selection

Mark,

I agree with your logic. In the meantime I will ignore the $COV step (it =

rarely happens for me) and wait for some empirical evidence that the

$COV step is of demonstrable value for model building. Perhaps your grid =

computing system could take on that challenge by comparing the results

of automated model building with and without $COV or convergence?

Nick

Mark Sale - Next Level Solutions wrote:

*>
*

*> Nick et al.
*

*> At this risk of starting an discussion that probably has little
*

*> mileage left in it. First I agree with Nick on covariance - it
*

*> probably doesn't matter. But, I'd like to point out what may be an
*

*> error in our logic.
*

*> We content that we have demonstrated that covariance doesn't matter.
*

*> Our evidence is that, when bootstrapping, the parameters for the
*

*> sample that have successful covariance are not different from those
*

*> that failed. So, we conclude that the results are the same regardless =
*

*> of covariance outcome across sampled data sets - the independent
*

*> variable in this test is the data set, the model is fixed.
*

*> In model selection/building, we have a fixed data set and the
*

*> independent variable is the model structure. Whether covariance
*

*> success is a useful predictor across different models with a fixed
*

*> data set is a different question than whether covariance is a useful
*

*> predictor across data sets with a fixed model.
*

*> But, in the end, I do agree that biological plausibility, diagnostic
*

*> plots, reasonable parameters and some suggestion of numerical
*

*> stability/identifiably (such as bootstrap CIs) are more important than =
*

*> a successful covariance step.
*

*>
*

*> Mark
*

*>
*

*> Mark Sale MD
*

*> Next Level Solutions, LLC
*

*> www.NextLevelSolns.com <http://www.NextLevelSolns.com>
*

*> 919-846-9185
*

*>
*

*> -------- Original Message --------
*

*> Subject: Re: [NMusers] OMEGA selection
*

*> From: Nick Holford <n.holford *

*> Date: Wed, April 15, 2009 12:17 pm
*

*> To: nmusers *

*>
*

*> Ethan,
*

*>
*

*> Do not pay any attention to whether or not the $COV step runs or
*

*> even if
*

*> the run is 'SUCCESSFUL' to conclude anything about your model. =
*

Your

*> opinion is not supported experimentally e.g. see
*

*> http://www.mail-archive.com/nmusers *

for

*> discussion and references.
*

*>
*

*> NONMEM has no idea if the parameters make sense or not and will
*

*> happily
*

*> converge with models that are overparameterised. You cannot rely =
*

on a

*> failed $COV step or a MINIMIZATION TERMINATED message to conclude =
*

the

*> model is not a good one. You need to use your brains (NONMEM does =
*

not

*> have a brain) and your common sense to decide if your model makes
*

*> sense
*

*> or is perhaps overparameterised.
*

*>
*

*> Nick
*

*>
*

*> Ethan Wu wrote:
*

*> >
*

*> > Dear all,
*

*> >
*

*> > I am fitting a PD response, and the equation goes like this:
*

*> >
*

*> > total response = baseline+f(placebo response) +f(drug =
*

response)

*> >
*

*> > first, I tried full omega block, and model was able to converge, =
*

but

*> > $COV stop failed.
*

*> >
*

*> > To me, this indicates that too many parameters in the model. The
*

*> > structure model is rather simple one, so I think probably too
*

*> many Etas.
*

*> >
*

*> > I wonder is there a good principle of Eta reduction that I could
*

*> > implement here. Any good reference?
*

*> >
*

*> >
*

*>
*

*> --
*

*> Nick Holford, Dept Pharmacology & Clinical Pharmacology
*

*> University of Auckland, 85 Park Rd, Private Bag 92019, Auckland,
*

*> New Zealand
*

*> n.holford *

*> mobile: +33 64 271-6369 (Apr 6-Jul 17 2009)
*

*> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
*

*>
*

*>
*

--

Nick Holford, Dept Pharmacology & Clinical Pharmacology

University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New =

Zealand

n.holford

mobile: +33 64 271-6369 (Apr 6-Jul 17 2009)

http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Received on Tue Apr 21 2009 - 09:50:51 EDT

Date: Tue, 21 Apr 2009 09:50:51 -0400

NMusers,

My apologies for entering into this discussion a bit late as I was on =

vacation last week. Rather than rehash previous debates about $COV, I =

thought I would just list some of the ways I use the $COV step output to =

assist my model building and clinical trial simulation efforts.

Before I do so, let me preface my comments by saying that for me the =

real diagnostic value of the $COV step is in the output reported by the =

$COV step and not simply whether or not $COV runs successfully. Thus, I =

strive for a successful $COV step because I find diagnostic value in the =

$COV output to guide my model-building efforts.

There are 3 basic ways I use the $COV step output:

1) Inspection of the standard errors, pairwise correlations among the =

parameter estimates, and the eigenvalue analysis of the correlation =

matrix helps me to understand the limitations of the design/data via the =

model.

2) I find building full covariate models much easier to obtain by first =

ensuring that I have a stable base model through an inspection of the =

$COV step output. I tend to like to use the full model to make =

inference about the covariate parameter estimates (e.g., CIs) as they =

will not suffer from model selection bias which occurs with stepwise =

procedures.

3) Based on asymptotic statistical theory for maximum likelihood =

estimation I will often assume that the parameters estimates follow a =

multivariate normal distribution with mean vector set to the population =

parameter estimates and covariance matrix set to the covariance matrix =

of the parameter estimates for THETA, OMEGA and SIGMA reported in the =

$COV output. This assumption allows me to easily generate random sets =

of population parameters reflecting parameter uncertainty when =

conducting clinical trial simulations. Of course one could do =

non-parametric bootstrapping to accomplish this as well but it is easier =

and faster to use the multivariate normal distribution when it is =

reasonable to assume that the asymptotics hold.

Below are examples that illustrate some of the ways I use the $COV =

output:

• Identify largest standard errors relative to the point =

estimates and rationalize the limitations of the data/design that would =

give rise to these large SEs (e.g., a standard error for ka may be large =

if few sample times are observed prior to Tmax).

• Screen for high pairwise correlations. For example, a high =

correlation in the population parameter estimates for CL/F and V/F may =

result when fitting a base model to steady-state PK data. This would =

suggest that the same information in the data is being used to estimate =

both parameters. This can be problematic for building full covariate =

models where one or more covariates may have effects on both parameters. =

In this setting I may use clinical judgment as to whether a particular =

covariate effect is more likely to be on CL/F or V/F if the limitations =

of the design/data preclude estimating it on both.

• The covariance matrix of the estimates from a full model run =

are helpful in determining a subset of potential parsimonious final =

models using the WAM algorithm (see Kowalski & Hutmacher, JPP =

2001;28:253-275).

• I use SAS (or Splus) to generate a random set of population =

parameters from the multivariate normal distribution using the =

population parameter estimates and the covariance matrix of the =

estimates from the $COV output in clinical trial simulations so that I =

can quantify operating characteristics such as probability of success =

(probability of a Go decision) and probability of a correct decision in =

contrast to power calculations which assume a fixed effect size. Power =

is a conditional probability (conditioning on an assumed effect =

magnitude) whereas POS (prob of success) is an unconditional probability =

that takes into account the uncertainty in achieving a given effect =

magnitude. Power is a performance characteristic of the design whereas =

POS is a performance characteristic of both the design and compound =

(dose of treatment).

Kind regards,

Ken

-----Original Message-----

From: owner-nmusers

On Behalf Of Nick Holford

Sent: Wednesday, April 15, 2009 2:49 PM

To: nmusers

Subject: Re: [NMusers] OMEGA selection

Mark,

I agree with your logic. In the meantime I will ignore the $COV step (it =

rarely happens for me) and wait for some empirical evidence that the

$COV step is of demonstrable value for model building. Perhaps your grid =

computing system could take on that challenge by comparing the results

of automated model building with and without $COV or convergence?

Nick

Mark Sale - Next Level Solutions wrote:

Your

for

on a

the

not

response)

but

--

Nick Holford, Dept Pharmacology & Clinical Pharmacology

University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New =

Zealand

n.holford

mobile: +33 64 271-6369 (Apr 6-Jul 17 2009)

http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Received on Tue Apr 21 2009 - 09:50:51 EDT