PsN-Toolkit—A collection of computer intensive statistical methods for non-linear mixed effect modeling using NONMEM

https://doi.org/10.1016/j.cmpb.2005.04.005Get rights and content

Summary

PsN-Toolkit is a collection of statistical tools for pharmacometric data analysis using the non-linear mixed effect modeling software NONMEM. The toolkit is object oriented and written in the programming language Perl using the programming library Perl-speaks-NONMEM (PsN). Five methods: the Bootstrap, the Jackknife, Log-likelihood Profiling, Case-deletion Diagnostics and Stepwise Covariate Model building are included as separate classes and may be used in user-written Perl scripts or through stand-alone command line applications. The tools are designed with the ability to cooperate and with an emphasis on common structures for workflow and result handling. Parallel execution of independent tool sections is supported on shared memory multiprocessor (SMP) computers, Mosix/openMosix clusters and distributed computing environments following the NorduGrid standard. In conclusion, PsN-Toolkit makes it easier to use the Bootstrap, the Jackknife, Log-likelihood Profiling, Case-deletion Diagnostics and Stepwise Covariate Model building in pharmacometric data analysis.

Introduction

Pharmacometrics formulate the dynamic interaction between drugs and individuals in terms of quantitative models based on (patho-) physiological and pharmacological knowledge. The insight that is thereby gained about the properties of a drug in individual patients or in patient populations is used in a descriptive or predictive manner to optimize the therapy of existing drugs and to design future clinical trials. Due to the complexity and heterogeneity of biological data, a central part of pharmacometrics is the knowledge about and development of methods for data analysis.

NONMEM [1] is the most widely used computer program for pharmacometric data analysis (also known as population pharmacokinetic/pharmacodynamic (PK/PD) analysis). It implements non-linear mixed effects (NLME) models, which allow for the characterization of the population parameters, i.e. the parameter values for a typical individual in the study population as well as the inter-individual variability around the typical individual's parameter estimates. In contrast to traditional single-subject analysis, which requires rich sampling in all individuals, NLME based analyses handles both sparse and rich sampling.

Linear models allow for closed form solutions to aspects such as estimates of parameter precision, and influence of single observations. Non-linear models, on the other hand, generally have no such solutions. At best, there may be numerical approximations such as estimates of standard errors. It is usually unknown how good an approximation is in a particular application. Alternatives exist of which some are based on approaches aiming at describing the statistic of interest not in terms of closed form solutions but instead by applying brute force computations. These computer intensive methods have begun to be applied in pharmacometrics but have yet to be evaluated further before they find general acceptance. A major obstacle has been the lack of such computational tools suitable for pharmacometric problems. PsN-Toolkit is an attempt at creating a set of such tools and making them available to the pharmacometric community.

Covariates are patient specific factors, such as age, body weight, concomitant medication and markers for disease progression that may identify sources of between patient variability as well as sub-groups of patients that are at risk of sub-therapeutic or toxic drug concentrations.

The number of parameter–covariate combinations is often large (for example, 4 model parameters and 20 covariates equal 80 potential combinations) and the task of identifying the relevant ones can be a time consuming exercise. An efficient approach to accomplish this is to use Stepwise Covariate Model building (SCM) directly in NONMEM [2], [3]. This procedure is also known as (orthogonal) Forward Selection–Backward Elimination. Stepwise procedures in general have been shown to exhibit a risk of including false parameter–covariate relations, of giving rise to biased estimates of the included relations as well as of yielding too narrow confidence limits [4], [5]. Other studies have reported that these problems may not be large for pharmacokinetic models [6], [7].

SCM assumes that there exists a structural model, i.e. a model that relates the main response variable, for example drug concentrations in pharmacokinetics, to the main independent variable, for example time since drug administration. The task of SCM is to identify the covariates that explain variability in the parameters of the structural model.

In a first step, each relevant parameter–covariate combination is added and estimated one by one in the structural model. The model with the largest improvement over the starting model is retained as the starting model for the next step. In each subsequent step the remaining parameter–covariate combinations are tried. This forward inclusion is continued until no improvement can be gained by adding new model components. The measure of model improvement is usually based on statistical significance. Optionally, the forward inclusion step can be followed by a backward elimination step. This proceeds according to the same general scheme as the forward step, but reversely, using stricter improvement criteria.

This adaptive procedure for Covariate Model building relies heavily on the validity of the statistics used for model discrimination.

The standard way of estimating the confidence intervals of parameter estimates for NLME models is to calculate the interval limits from the standard errors under the assumption that the estimates are normally distributed. Log-likelihood Profiling (LLP) is one alternative method where no assumption regarding symmetry of the interval has to be made [8], [9]. Fixing a parameter to values close to the estimate obtained from a maximum likelihood procedure (as the one implemented in NONMEM) and refitting this reduced model generates a likelihood profile. Often, as is the case in NONMEM, minus two times the natural logarithm of the likelihood is used and the maximum likelihood then corresponds to the minimum of this quantity. If a parameter is fixed, this model can be regarded as an alternative, competing with the full non-fixed model in being the most appropriate for describing the data at hand. The rival models are nested and the difference in the log-likelihoods of the data for the two models is approximately χ2-distributed. At p = 0.05, a statistically significantly improvement is achieved when the log-likelihood difference is 3.84. Thus, the challenging alternative model with a fixed parameter can be rejected when the −2 log-likelihood difference is above this value. The confidence interval limits for a parameter is then where the log-likelihood is 3.84 higher than at the maximum likelihood estimate. It is however well known that the actual significance levels for log-likelihood differences acquired through NONMEM may not agree with the expected using a χ2-distribution. For example, it has been shown that the first-order approximation method (FO) gives higher significance levels than the nominal when studying covariate effects [10]. Also, tests for non-zero variance components cannot be performed using standard likelihood ratios since the null-hypothesis puts the parameter value on the boundary of the parameter space (zero) [11]. This must be taken into consideration when performing Log-likelihood Profiling (see Command Line Example 1.

The Bootstrap [12] is a general method for measuring statistical accuracy and precision. Briefly, it involves creating “new” data sets by sampling with replacement from the original data and applying the same analysis steps to each of the new data sets as was performed on the original data. In population PK/PD, these analysis steps usually correspond to model fits generating parameter estimates but it may be any kind of statistical procedure. The results from the new data sets form distributions, which reflect the uncertainty in the original analysis. These distributions can be used to assess covariate selection stability [13], uncertainty of parameter estimates [14] and to correct for certain types of bias [15]. The resampling is performed with replacement on statistically independent parts of the data, which in population PK/PD usually corresponds to individuals. Depending on what the statistic of interest is, different numbers of resampled data sets are needed. Bias correction of parameter estimates typically needs 50 Bootstrap data sets, whereas estimation of standard errors requires 200. Using the BCa method [16] to calculate second-order correct 95% confidence intervals requires approximately 2000 bootstrap data sets.

Case-deletion Diagnostics (CDD) is a standard method for detecting observations that are the most important for a model fit, e.g. to determine which observation or individual that influence the parameter estimates the most. The Cook-score and the covariance ratio tests:COOKi=(θˆiθˆ)Tcov(θˆ)1(θˆiθˆ)COVratioi=det(cov(θˆi))det(cov(θˆi))where θˆ and θˆi are vectors of all parameter estimates from regressions of the model to the full and i reduced data sets, respectively, may be used to estimate the effect of the removal of one individual at a time on the parameter estimates [17]. Numerical methods for case-deletion exist for linear models but for NLME models it is necessary to fit the model to data sets in which one individual at a time is excluded [18]. This requires a substantial amount of CPU time,1 especially for large data sets.

The Jackknife, first proposed by Quenouille [19], [20] as a means to reduce bias, is very similar to Case-deletion Diagnostics. Parts of the data are excluded one by one from the training data set, followed by the calculation of a statistic of some kind on each reduced data set. The Jackknife estimate of the bias of a parameter estimate θˆ for a data set with n subjects is the difference between the Jackknife estimate of the mean θˆ() and the estimate from the full data set scaled by a factor of n  1, i.e.biasˆ=(n1)(θˆ()θˆ)whereθˆ()=1ni=1nθˆ(i)

As computers have grown faster, the above methods have started to become more widely used in pharmacometric data analysis. The methods may potentially also be used in combination to further increase their usefulness. For example, running a Bootstrap for each potential parameter–covariate combination of each step in an SCM would circumvent some pitfalls of the commonly used approaches to statistical significance [10]. The inverse link between the SCM and the Bootstrap is also of interest. By running a Bootstrap of the complete stepwise covariate selection procedure, the frequencies of the inclusion of parameter–covariate relations in the model may help in determining which relations are of highest importance. The combination of a Case-deletion Diagnostics and either a Log-likelihood Profiling or a Bootstrap may help in revealing which individuals have the greatest impact on the confidence intervals for the parameter estimates.

Section snippets

Prior work by the authors

We have previously published an application programming interface to NONMEM, called Perl-speaks-NONMEM (PsN) [21]. PsN is an object oriented library of Perl modules designed around NONMEM's input and output files that makes it easier for developers to integrate NONMEM's computing power into their own applications. As the name of the present work implies, PsN-Toolkit depends heavily on PsN for communication with NONMEM.

Prior work by others

Routines for some of the computer intensive methods described in this article

Design considerations

PsN-Toolkit is designed around the concept of tools or methods used in pharmacometric analyses. An analysis consists of a number of steps being performed in consecutive order or in parallel, each aiming at verifying or discarding some hypothesis specified for the structure of the NLME model, the data or the design of the experiment. These hypotheses are tested using tools, ranging from very simple likelihood ratio tests to complex (manual or automated) procedures involving many model fits to

Choice of programming language

The programming language Perl [26] was chosen for implementing PsN-Toolkit for two reasons. First, Perl is available on many computer platforms including Microsoft Windows®, Sun Solaris® and Linux, and second, good Perl modules exists for many otherwise programmatically difficult tasks. PsN-Toolkit uses the following Perl modules: Config::Tiny, Math::SigFigs, Math::Random, Parallel::ForkManager, Statistics::Distributions and Text::More. The modules can be downloaded for free from //www.cpan.org/

Status report

PsN-Toolkit can be obtained for free from http://psn.sourceforge.net. Detailed technical documentation and user manuals can be found at the same address.

Future plans

The five methods included in PsN-Toolkit are just a few of a larger set of computer intensive methods that have been suggested for use in pharmacometric data analysis. Methods like the Posterior Predictive Check, Monte-carlo Simulations, Cross Validation, Model Selection using Genetic Algorithms and the LASSO could all be valuable in an expanded PsN-Toolkit.

Acknowledgement

The Swedish Foundation for Strategic Research, Stockholm, Sweden, is gratefully acknowledged for providing the financial means to perform this study.

References (27)

  • L. Lindbom et al.

    Perl-speaks-NONMEM (PsN)—a Perl module for NONMEM related programming

    Comput. Methods Programs Biomed.

    (2004)
  • S.L. Beal, L.B. Sheiner, NONMEM Users Guide, 1998 NONMEM Project Group, University of California, San Francisco,...
  • E.N. Jonsson et al.

    Automated covariate model building within NONMEM

    Pharm. Res.

    (1998)
  • J.W. Mandema et al.

    Building population pharmacokinetic–pharmacodynamic models. I. Models for covariate effects

    J. Pharmacokinet. Biopharm.

    (1992)
  • S. Derksen et al.

    Backward, forward and stepwise automated subset-selection algorithms—frequency of obtaining authentic and noise variables

    Br. J. Math. Stat. Psychol.

    (1992)
  • D.G. Altman et al.

    Bootstrap investigation of the stability of a Cox regression-model

    Stat. Med.

    (1989)
  • U. Wahlby et al.

    Comparison of stepwise covariate model building strategies in population pharmacokinetic–pharmacodynamic analysis

    AAPS PharmSci.

    (2002)
  • J. Ribbing et al.

    Power, selection bias and predictive performance of the Population Pharmacokinetic Covariate Model

    J. Pharmacokinet. Pharmacodyn.

    (2004)
  • L.B. Sheiner

    Analysis of pharmacokinetic data using parametric models. III. Hypothesis tests and confidence intervals

    J. Pharmacokinet. Biopharm.

    (1986)
  • N.H. Holford et al.

    Results and validation of a population pharmacodynamic model for cognitive effects in Alzheimer patients treated with tacrine

    Proc. Natl. Acad. Sci. U.S.A.

    (1992)
  • U. Wahlby et al.

    Assessment of actual significance levels for covariate effects in NONMEM

    J. Pharmacokinet. Pharmacodyn.

    (2001)
  • D.O. Stram et al.

    Variance–components testing in the longitudinal mixed effects model

    Biometrics

    (1994)
  • B. Efron

    Bootstrap methods: another look at the Jackknife

    Ann. Stat.

    (1979)
  • Cited by (926)

    View all citing articles on Scopus
    View full text