The Cram Method for Efficient Simultaneous Learning and Evaluation
The Cram Method for Efficient Simultaneous Learning and Evaluation
Please visit the official website for more details.
The Cram method is an efficient approach to simultaneous learning and evaluation using a generic machine learning (ML) algorithm. In a single pass of batched data, the proposed method repeatedly trains an ML algorithm and tests its empirical performance.
Because it utilizes the entire sample for both learning and evaluation, cramming is significantly more data-efficient than sample-splitting, which reserves a portion of the data purely for evaluation. Also, a key distinction from cross-validation is that Cram evaluates the final learned model directly, rather than using as a proxy the average performance of multiple fold-specific models trained on different data subsets—resulting in sharper inference and better alignment with real-world deployment. More details about the Cram procedure can be found in the article Introduction & Cram Policy part 1.
The Cram method naturally applies to the policy learning setting, which is a popular subfield of ML focused on learning a decision rule (also called treatment rule or policy) that assigns treatments or actions to individuals based on their features, with the goal of maximizing an expected outcome (e.g., health, profit, welfare). Cramming allows users to both learn an individualized decision rule and estimate the average outcome that would result if the learned decision rule were to be deployed to the entire population beyond the data sample.
It is particularly relevant in high-stakes applications where decisions must be both personalized and statistically reliable.
Common examples include:
🧠 Cram Policy (cram_policy): Learn
and evaluate individualized binary treatment rules using Cram. Supports
flexible models, including causal forests and custom learners. Common
examples include whether to treat a patient, send a discount offer, or
provide financial aid based on estimated benefit.
📈 Cram ML (cram_ml): Learn and
evaluate standard machine learning models using Cram. It estimates the
expected loss at the population level, giving you a reliable measure of
how well the final model is likely to generalize to new data. Supports
flexible training via caret or custom learners, and allows evaluation
with user-defined loss metrics. Ideal for classification, regression,
and other predictive tasks.
🎰 Cram Bandit (cram_bandit):
Perform on-policy evaluation of contextual bandit algorithms using Cram.
Supports both real data and simulation environments with built-in
policies.
For users with an ML background, it may be informative to compare with supervised learning to introduce the contextual bandit setting. In supervised learning, each data point comes with a known label. In contrast, the contextual bandit setting involves a context (feature vector), a choice among multiple actions, and a reward observed for the chosen action. Thus, the label (reward) is at first unknown and is only revealed after an action is chosen - note that the labels (rewards) associated with the non-chosen actions will remain unknown (partial feedback), which makes learning and evaluation more challenging.
Contextual bandits appear in applications where an online system selects actions based on context to maximize outcomes—like showing ads or recommendations and observing user clicks or purchases. Contextual bandit algorithms aim to learn a policy that chooses the best action for each context to maximize expected reward, such as engagement (clicks) or conversion. Cram Bandit estimates how well the final learned policy would perform if deployed on the entire population, based on data collected by the same policy.
You can also explore additional tutorials and examples through the “Articles” menu in the top navigation bar of the website.
To install the development version of Cram from GitHub:
# Install devtools if needed
install.packages("devtools")
# Install cramR from GitHub
devtools::install_github("yanisvdc/cramR")If you use Cram in your research, please cite the following papers:
@techreport{jia2024cram,
title = {The Cram Method for Efficient Simultaneous Learning and Evaluation},
author = {Jia, Zeyang and Imai, Kosuke and Li, Michael Lingzhi},
institution = {Harvard Business School},
type = {Working Paper},
year = {2024},
url = {https://www.hbs.edu/ris/Publication%20Files/2403.07031v1_a83462e0-145b-4675-99d5-9754aa65d786.pdf},
note = {Accessed April 2025}
}@misc{jia2025crammingcontextualbanditsonpolicy,
title={Cramming Contextual Bandits for On-policy Statistical Evaluation},
author={Zeyang Jia and Kosuke Imai and Michael Lingzhi Li},
year={2025},
eprint={2403.07031},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2403.07031},
}You can also cite the R package:
@Manual{,
title = {cramR: The Cram Method for Efficient Simultaneous Learning and Evaluation},
author = {Yanis Vandecasteele and Michael Lingzhi Li and Kosuke Imai and Zeyang Jia and Longlin Wang},
year = {2025},
note = {Package developed and maintained by Yanis Vandecasteele.},
url = {https://github.com/yanisvdc/cramR},
}We welcome contributions! To contribute:
# 1. Fork the repository.
# 2. Create a new branch
git checkout -b feature/your-feature
# 3. Commit your changes
git commit -am 'Add some feature'
# 4. Push to the branch
git push origin feature/your-feature
# 5. Create a pull request.
# 6. Open an Issue or PR at:
# https://github.com/yanisvdc/cramR/issuesFor questions or issues, please open an issue.