Generalizability Path Example: Characterizing Underrepresented Populations

Overview

A common challenge in translating evidence from randomized controlled trials (RCTs) to real-world practice is that trial participants may not reflect the broader target population. By definition in Parikh et al. 2025, subgroups that are “underrepresented” or “insufficiently represented” often occupy regions of the covariate space with heterogeneous treatment effects and insufficient representation in the trial data. If certain subgroups are underrepresented in the trial, estimates of the Target Average Treatment Effect (TATE) can be imprecise or misleading when transported to that population. The Sample Average Treatment Effect (SATE) is a finite sample equivalent version of the TATE.

The resulting estimand from ROOT is the Weighted Target Average Treatment Effect (WTATE): the average treatment effect restricted to the sufficiently represented subpopulation, estimated with lower variance than the unweighted TATE.

This vignette walks through a complete generalizability analysis using the built-in diabetes_data dataset.

The diabetes_data Dataset

diabetes_data is a simulated dataset that mimics a diabetes intervention study. There are 2,000 individuals in a randomized controlled trial (RCT) sample, and there are 8,000 individuals in this simulated population we are making inferences to.

library(ROOT)

data(diabetes_data, package = "ROOT")
str(diabetes_data)
#> 'data.frame':    10000 obs. of  7 variables:
#>  $ Race_Black: int  0 1 1 0 0 0 0 0 0 0 ...
#>  $ Sex_Male  : int  1 0 1 1 1 1 1 0 1 1 ...
#>  $ DietYes   : int  0 0 0 0 0 0 0 0 0 0 ...
#>  $ Age45     : int  0 1 0 0 1 0 0 0 1 0 ...
#>  $ S         : int  1 0 0 0 0 0 0 0 0 0 ...
#>  $ Tr        : int  0 NA NA NA NA NA NA NA NA NA ...
#>  $ Y         : num  0.818 NA NA NA NA ...

The key columns are:

# How many trial vs target population units?
table(S = diabetes_data$S)
#> S
#>    0    1 
#> 8000 2000

# Treatment breakdown within the trial
table(Tr = diabetes_data$Tr[diabetes_data$S == 1])
#> Tr
#>    0    1 
#>  977 1023

Checking Covariate Overlap

Before running ROOT, it is good practice to check whether trial participants differ from the target population on key covariates. Systematic differences signal which subgroups may be underrepresented.

# Mean of each covariate by S
covariate_cols <- c("Age45", "DietYes", "Race_Black", "Sex_Male")

overlap <- sapply(covariate_cols, function(v) {
  tapply(diabetes_data[[v]], diabetes_data$S, mean, na.rm = TRUE)
})

knitr::kable(
  t(overlap),
  digits  = 3,
  caption = "Covariate means by sample membership (S = 1: trial, S = 0: target)"
)

Differences across rows flag potential sources of underrepresentation that ROOT will attempt to characterize.

Fitting ROOT in Generalizability Mode

We use characterizing_underrep(), which is the high-level wrapper around ROOT() for generalizability/transportability analyses. It expects data to contain Y, Tr, and S, and internally:

1. Estimates transportability scores using logistic regression models (default) for \(P(S = 1 \mid X)\) and \(P(\text{Tr} = 1 \mid X, S = 1)\).
2. Constructs Horvitz–Thompson-style influence scores \(v_i\).
3. Grows a forest of weighted trees that minimize the variance of the weighted estimator \(\widehat{\text{WTATE}}\).
4. Selects a Rashomon set of the top-\(k\) trees and aggregates their weight assignments by majority vote (default).
5. Fits a single summary tree characterizing the final \(w_{\text{opt}}\) assignments.

gen_fit <- characterizing_underrep(
  data                  = diabetes_data,
  generalizability_path = TRUE,
  num_trees             = 20,
  top_k_trees           = TRUE,
  k                     = 10,
  seed                  = 123
)

Inspecting the Results

print(gen_fit)
#> characterizing_underrep object
#>   --- ROOT brief summary ---
#> ROOT object
#>   Generalizability mode: TRUE 
#> 
#> Summary classifier (f):
#> n= 2000 
#> 
#> node), split, n, loss, yval, (yprob)
#>       * denotes terminal node
#> 
#>  1) root 2000 735 1 (0.36750000 0.63250000)  
#>    2) Race_Black>=0.5 345   0 0 (1.00000000 0.00000000) *
#>    3) Race_Black< 0.5 1655 390 1 (0.23564955 0.76435045)  
#>      6) Age45>=0.5 260   0 0 (1.00000000 0.00000000) *
#>      7) Age45< 0.5 1395 130 1 (0.09318996 0.90681004)  
#>       14) DietYes>=0.5 130   0 0 (1.00000000 0.00000000) *
#>       15) DietYes< 0.5 1265   0 1 (0.00000000 1.00000000) *
#> 
#> Estimand summary (generalization mode):
#>   Unweighted SATE = 5.48424, SE = 0.4290411
#>   Weighted   WTATE = 3.55263, SE = 0.3609503

summary(gen_fit)
#> characterizing_underrep object
#>   --- ROOT summary ---
#> ROOT object
#>   Generalizability mode: TRUE 
#> 
#> Summary classifier (f):
#> n= 2000 
#> 
#> node), split, n, loss, yval, (yprob)
#>       * denotes terminal node
#> 
#>  1) root 2000 735 1 (0.36750000 0.63250000)  
#>    2) Race_Black>=0.5 345   0 0 (1.00000000 0.00000000) *
#>    3) Race_Black< 0.5 1655 390 1 (0.23564955 0.76435045)  
#>      6) Age45>=0.5 260   0 0 (1.00000000 0.00000000) *
#>      7) Age45< 0.5 1395 130 1 (0.09318996 0.90681004)  
#>       14) DietYes>=0.5 130   0 0 (1.00000000 0.00000000) *
#>       15) DietYes< 0.5 1265   0 1 (0.00000000 1.00000000) *
#> 
#> Global objective function:
#>   User-supplied: No (default objective used)
#> 
#> Estimand summary (generalization mode):
#>   Unweighted SATE = 5.48424, SE = 0.4290411
#>   Weighted   WTATE = 3.55263, SE = 0.3609503
#> 
#> Diagnostics:
#>   Number of trees grown: 20
#>   Rashomon set size: 10
#>   % observations with w_opt == 1: 63.2%
#> 
#>   Leaf summary:    4 terminal nodes
#>  leaf_id                                               rule predicted_w    n
#>        2                             root & Race_Black>=0.5           0  345
#>        6                root & Race_Black< 0.5 & Age45>=0.5           0  260
#>       14 root & Race_Black< 0.5 & Age45< 0.5 & DietYes>=0.5           0  130
#>       15 root & Race_Black< 0.5 & Age45< 0.5 & DietYes< 0.5           1 1265
#>   pct                           label
#>  17.2 Under-represented (drop, w = 0)
#>  13.0 Under-represented (drop, w = 0)
#>   6.5 Under-represented (drop, w = 0)
#>  63.2       Represented (keep, w = 1)

gen_fit$leaf_summary
#>   leaf_id                                               rule predicted_w    n
#> 1       2                             root & Race_Black>=0.5           0  345
#> 2       6                root & Race_Black< 0.5 & Age45>=0.5           0  260
#> 3      14 root & Race_Black< 0.5 & Age45< 0.5 & DietYes>=0.5           0  130
#> 4      15 root & Race_Black< 0.5 & Age45< 0.5 & DietYes< 0.5           1 1265
#>    pct                           label
#> 1 17.2 Under-represented (drop, w = 0)
#> 2 13.0 Under-represented (drop, w = 0)
#> 3  6.5 Under-represented (drop, w = 0)
#> 4 63.2       Represented (keep, w = 1)

Visualizing the Characterization Tree

plot() renders the final characterized tree from the Rashomon set. Blue leaves (\(w = 1\)) denote well-represented subgroups; orange leaves (\(w = 0\)) denote underrepresented subgroups. The percentage shown in each leaf is the share of trial units falling into that node.

plot(gen_fit)

The tree reads top-down as a decision rule: starting from the root (all trial units), the first split separates subgroups that are wholly underrepresented from those that may be included. Follow the branches down to each leaf to read the complete inclusion/exclusion rule for that subgroup.

Interpreting the Output

From the characterized tree and leaf summary, we can describe the underrepresented subgroups in plain language:

• Black participants are flagged as underrepresented (\(w = 0\)) regardless of other characteristics.
• Participants aged 45 or older who are not Black are also underrepresented.
• Participants on a diet programme who are neither Black nor aged 45+ are underrepresented.
• The remaining participants, those who are not Black, under 45, and not on a diet programme, form the well-represented subpopulation (\(w = 1\)) for whom the WTATE is estimated.

The Rashomon set provides multiple near-optimal characterizations of these subgroups. The final summary tree aggregates across all trees in the set, giving a single interpretable rule.

Key Parameters

Reference

Parikh, H., Ross, R. K., Stuart, E., & Rudolph, K. E. (2025). Who Are We Missing?: A Principled Approach to Characterizing the Underrepresented Population. Journal of the American Statistical Association, 120(551), 1414–1423. https://doi.org/10.1080/01621459.2025.2495319