Case study: scoring the CDISC pilot ADaM package

This case study scores a real, public ADaM package end to end. Nothing is hand-tuned: we take two artifacts that were produced independently and ask how ready the result is for submission.

Because the spec and the data were not made to match each other, the score is not a formality. It tells us where the package stands.

The inputs

library(r4subpharma)

# The CDISC pilot ADaM specification (covers ADSL, ADAE, ADLBC, ADADAS, ADTTE)
e <- new.env()
load(metacore::metacore_example("pilot_ADaM.rda"), envir = e)
spec <- e$metacore

# One of the admiral-built datasets
adsl <- pharmaverseadam::adsl
dim(adsl)
#> [1] 306  55

One call to a score

submission_readiness() harvests documentation evidence from the spec and conformance evidence from the dataset, then computes the Submission Confidence Index.

ctx <- r4subcore::r4sub_run_context("CDISCPILOT01", "PROD")
#> ℹ Run context created: "R4S-20260831223707-n4yxpuf6"
spec_adsl <- metacore::select_dataset(spec, "ADSL", verbose = "silent")

res <- submission_readiness(list(ADSL = adsl), spec_adsl, ctx)
#> ℹ metacore_to_evidence: 102 rows from 51 variables
#> ✔ Evidence table created: 102 rows
#> ℹ adam_to_evidence: 125 rows for dataset "ADSL"
#> ✔ Evidence table created: 125 rows
#> ✔ Bound 2 evidence tables: 227 total rows
#> ℹ Submission Confidence Index: 79.5 (minor_gaps)
res
#> <submission_readiness>
#>   evidence rows: 227 
#>   SCI:           79.5 
#>   band:          minor_gaps

Reading the score

The pillar breakdown shows where the number comes from.

as.data.frame(res$sci$pillar_scores)
#>      pillar pillar_score n_indicators weight
#> 1   quality    0.9934641            3   0.35
#> 2     trace    0.3933824            2   0.25
#> 3      risk           NA            0   0.25
#> 4 usability    1.0000000            1   0.15

Quality (documentation and types) and usability (labels) are strong. The weak pillar is traceability: the specification lists variables that the built dataset does not yet contain. That single gap is what holds the score back.

The gap, made actionable

The value of a score is the fix list behind it. The failing traceability checks name exactly which specified variables are missing from the data.

ev <- res$evidence
missing <- ev[ev$indicator_id == "T-ADAM-001" & ev$result == "fail", ]
nrow(missing)
#> [1] 30
sub(".*: ", "", missing$message)
#>  [1] "Variable SITEGR1 is described in metadata but missing from ADSL" 
#>  [2] "Variable TRT01PN is described in metadata but missing from ADSL" 
#>  [3] "Variable TRT01AN is described in metadata but missing from ADSL" 
#>  [4] "Variable AVGDD is described in metadata but missing from ADSL"   
#>  [5] "Variable CUMDOSE is described in metadata but missing from ADSL" 
#>  [6] "Variable AGEGR1N is described in metadata but missing from ADSL" 
#>  [7] "Variable AGEGR2 is described in metadata but missing from ADSL"  
#>  [8] "Variable AGEGR2N is described in metadata but missing from ADSL" 
#>  [9] "Variable RACEN is described in metadata but missing from ADSL"   
#> [10] "Variable ITTFL is described in metadata but missing from ADSL"   
#> [11] "Variable EFFFL is described in metadata but missing from ADSL"   
#> [12] "Variable COMP8FL is described in metadata but missing from ADSL" 
#> [13] "Variable COMP16FL is described in metadata but missing from ADSL"
#> [14] "Variable COMP24FL is described in metadata but missing from ADSL"
#> [15] "Variable DISCONFL is described in metadata but missing from ADSL"
#> [16] "Variable DSRAEFL is described in metadata but missing from ADSL" 
#> [17] "Variable BMIBL is described in metadata but missing from ADSL"   
#> [18] "Variable BMIBLGR1 is described in metadata but missing from ADSL"
#> [19] "Variable HEIGHTBL is described in metadata but missing from ADSL"
#> [20] "Variable WEIGHTBL is described in metadata but missing from ADSL"
#> [21] "Variable EDUCLVL is described in metadata but missing from ADSL" 
#> [22] "Variable DISONSDT is described in metadata but missing from ADSL"
#> [23] "Variable DURDIS is described in metadata but missing from ADSL"  
#> [24] "Variable DURDSGR1 is described in metadata but missing from ADSL"
#> [25] "Variable VISIT1DT is described in metadata but missing from ADSL"
#> [26] "Variable VISNUMEN is described in metadata but missing from ADSL"
#> [27] "Variable RFENDT is described in metadata but missing from ADSL"  
#> [28] "Variable DCDECOD is described in metadata but missing from ADSL" 
#> [29] "Variable DCSREAS is described in metadata but missing from ADSL" 
#> [30] "Variable MMSETOT is described in metadata but missing from ADSL"

These are the CDISC pilot analysis variables: treatment-coded flags, age and site groupings, and completion flags. An analysis-ready ADSL is expected to carry them, so their absence is a real readiness finding, not a formatting nit.

We can also see the smaller quality signals: any variable whose type does not match the specification.

ev[ev$indicator_id == "Q-ADAM-001" & ev$result == "warn",
   c("location", "message")]
#> [1] location message 
#> <0 rows> (or 0-length row.names)

Scoring more of the package

The same call scales to several datasets at once; the score then reflects the whole set of evidence.