--- title: "Getting started with pmsims" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Getting started with pmsims} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r} #| label: setup #| include: false knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 6, fig.height = 4, warning = FALSE, message = FALSE ) ``` ## What pmsims does **pmsims** estimates the **minimum sample size** needed to develop a prediction model to achieve a target level of performance **with assurance**. Rather than relying on simple rules of thumb or closed‑form formulae, pmsims uses **simulation** to: - Generate synthetic datasets that reflect your target setting (outcome type, prevalence or $R^2$, signal vs. noise predictors, and how complex the underlying signal is); - Fit a specified **model** (e.g., logistic regression or linear regression); - Evaluate a chosen **performance metric** (e.g., calibration slope, AUC); and - Trace a **learning curve** of performance as the training size increases.

A diagram showing the pmsims workflow, consisting of the data generator, model function, metrics function, which are passed to the simulation engine.

The recommended design objective is **assurance**: the **smallest** $n$ such that a high proportion of repeated studies (e.g., 80%) meet the target performance. In pmsims, this is implemented via the **20th percentile** of the simulated performance distribution at each $n$. ## Required inputs at a glance There are three wrapper functions for binary, continuous, and survival outcomes, respectively: - `simulate_binary()` - `simulate_continuous()` - `simulate_survival()` All three functions share the same basic structure. The table below lists the key inputs. ```{=html}
Select wrapper
Argument Description
signal_parameters (int) Number of true signal predictors associated with the outcome.
Applies: all Default: — Required
noise_parameters (int) Number of noise predictors unrelated to the outcome.
Applies: all Default: 0
complexity (int 1–4) Signal structure of the data-generating mechanism: 1 purely linear, 2 linear + quadratic, 3 linear + quadratic + interaction, 4 the Friedman function.
Applies: all Default: 1
data_control (list) Optional list fine-tuning the predictors:
  • nonlinear_strength (num [0, 1)) — fraction of the signal variance carried by the nonlinear, linearly-inaccessible component. Complexity 2 and 3 only; ignored with a warning for 1 and 4.
  • correlation (num [-1, 1]) — pairwise correlation among candidate predictors. Default 0.3.
  • predictor_distribution (chr) — one of "normal", "uniform", "binary", "exponential", "lognormal", "t", "laplace". Default "normal".
  • binary_predictor_prevalence (num (0, 1)) — required when predictor_distribution = "binary", ignored otherwise. Binary predictors are incompatible with complexity 2/3.
Applies: all Default: NULL
outcome_prevalence (num 0–1) Target prevalence of the binary outcome.
Applies: binary only Default: — Required
maximum_achievable_cstatistic (num 0–1) Maximum achievable C-statistic with effectively unlimited data. This calibrates the data generator and is not the minimum acceptable threshold.
Applies: binary only Default: — Required
maximum_achievable_rsquared (num 0–1) Maximum achievable R2 with effectively unlimited data. This calibrates the data generator and is not the minimum acceptable threshold.
Applies: continuous only Default: — Required
maximum_achievable_cindex (num 0–1) Maximum achievable concordance index with effectively unlimited data. This calibrates the data generator and is not the minimum acceptable threshold.
Applies: survival only Default: — Required
baseline_hazard (num > 0) Baseline hazard used by the survival data-generating mechanism. Larger values imply shorter event times, all else equal.
Applies: survival only Default: 1
censoring_rate (num 0–1) Proportion of individuals expected to be censored in the simulated survival datasets.
Applies: survival only Default: — Required
model (chr) Model used for fitting: "glm" / "lm" / "coxph" depending on the outcome, or one of the experimental machine-learning options "lasso", "ridge", "rf", "xgboost".
Applies: all Default: "glm" / "lm" / "coxph"
metric (chr) Performance metric used to estimate the minimum required sample size. Metric identifiers take one canonical form throughout the package: "calibration_slope", "calibration_in_the_large", "auc", "r2", "cindex", and "csse" (calibration slope squared error).
Applies: all Default: "calibration_slope"
target_performance (num) Minimum acceptable performance in the units of the chosen metric (e.g. calibration slope ≥ 0.9), used as the threshold for selecting the required sample size.
Applies: all Default: — Required
n_reps_total (int) Total number of simulation replications.
Applies: all Default: 1000 Required
mean_or_assurance (chr) Criterion for summarising results; "assurance" recommended.
Applies: all Default: "assurance"
``` > Notes: > > - `maximum_achievable_*` represents the best plausible performance with > effectively unlimited data and calibrates the data generator. > - `target_performance` is the minimum acceptable performance threshold used > to determine the required sample size. > - `complexity` and `data_control` describe the data-generating mechanism, not > the model you plan to fit. The same configuration is used both to calibrate > the generator against `maximum_achievable_*` and to simulate the data, so a > more complex signal generally implies a larger required sample size. > - For reproducibility, set a random seed (`set.seed()`). ## Installation ```{r} # install.packages("remotes") # remotes::install_github("pmsims-package/pmsims") library(pmsims) ``` ## Binary-outcome example We target the smallest *n* that meets the **assurance** criterion. ```{r} #| echo: true #| eval: false set.seed(123) binary_example <- simulate_binary( signal_parameters = 20, noise_parameters = 0, complexity = 1, data_control = list(correlation = 0.3), outcome_prevalence = 0.30, maximum_achievable_cstatistic = 0.80, model = "glm", metric = "calibration_slope", target_performance = 0.85, n_reps_total = 1000, mean_or_assurance = "assurance" ) binary_example ``` ```{r Run binary} #| echo: false # `binary_example` is the result of the call above, shipped with the package. print(binary_example) ``` The printed summary is a human-readable report. Implementation detail --- the internal metric identifiers, the engine settings used for the search, and any quantities recorded on an internal search scale --- is available through `summary(binary_example)` or, equivalently, `print(binary_example, verbose = TRUE)`. Plot the estimated learning curve and identified sample size: ```{r, fig.alt="Plot showing learning curve for binary outcome"} plot(binary_example) ``` ## Continuous-outcome example ```{r} #| echo: true #| eval: false continuous_example <- simulate_continuous( signal_parameters = 15, noise_parameters = 0, complexity = 1, data_control = list(correlation = 0.3), maximum_achievable_rsquared = 0.50, model = "lm", metric = "calibration_slope", target_performance = 0.95, n_reps_total = 1000, mean_or_assurance = "assurance" ) continuous_example ``` ```{r} #| echo: false # `continuous_example` is the result of the call above, shipped with the package. print(continuous_example) ``` ```{r, fig.alt="Plot showing learning curve for continuous outcome"} plot(continuous_example) ```