---
title: "Getting Started with evoFE"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting Started with evoFE}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 5
)
```

## What is evoFE?

**evoFE** (Evolutionary Feature Engineering) uses a genetic algorithm to
automatically discover useful feature transformations for tabular data. Instead
of manually crafting interaction terms, ratios, or binning strategies, you let
evolution explore the space of possible transformations and keep the ones that
improve predictive performance.

The result is an **evo_recipe** -- a reusable transformation pipeline that can be
applied to new data at prediction time.

### How it works

1. **Initialisation** -- A population of individuals is created. Each individual
   is a "recipe" containing a set of feature transformations (genes) and an
   **active feature mask** that controls which raw input columns are visible.
2. **Evaluation** -- Every individual is scored via cross-validated or
   split model performance (LightGBM, XGBoost, or any custom evaluator).
3. **Selection** -- The top 50 % survive to breed.
4. **Breeding** -- Survivors are combined (crossover) and randomly altered
   (mutation) to produce the next generation. Mutations may add/remove/modify
   genes *or* toggle which raw features are active.
5. **Repeat** -- The cycle continues until the fitness plateaus or the
   generation budget is exhausted.

## Installation

```r
# Install the released version from CRAN
install.packages("evoFE")

# Or install the development version directly from GitHub
# devtools::install_github("tanopereira/evoFE")
```

## Quick Start -- Binary Classification

Let's classify whether a car has an automatic or manual transmission using the
`mtcars` dataset.

```{r binary-classification}
library(evoFE)

data(mtcars)
df <- mtcars
df$am <- as.integer(df$am)   # target: 0 = automatic, 1 = manual

set.seed(42)
res <- evolve_features(
  data      = df,
  target_col = "am",
  task       = "classification",
  evaluator  = "xgboost",
  generations = 5,
  pop_size    = 8,
  cv_folds    = 3,
  early_stopping_generations = 3,
  verbose     = TRUE
)
```

The returned `evo_recipe` object contains the best individual (feature recipe),
the fitted model, and the evolution history.

```{r binary-inspect}
# Print high-level overview of the recipe
print(res)

# View a detailed structured summary
summary(res)
```

### Applying the recipe to new data

`predict()` applies the evolved transformations to new data and returns the
engineered feature matrix:

```{r binary-predict-features}
engineered <- predict(res, df[1:5, ])
head(engineered)
```

`predict_model()` goes one step further -- it applies the transformations **and**
runs the trained model to produce predictions:

```{r binary-predict-model}
preds <- predict_model(res, df[1:5, ])
preds
```

## Regression

Predict petal length from the iris dataset:

```{r regression}
data(iris)

set.seed(123)
res_reg <- evolve_features(
  data       = iris[, c("Sepal.Length", "Sepal.Width", "Petal.Width", "Petal.Length")],
  target_col = "Petal.Length",
  task       = "regression",
  evaluator  = "xgboost",
  generations = 5,
  pop_size    = 8,
  cv_folds    = 3,
  early_stopping_generations = 3,
  verbose     = TRUE
)

cat("Best recipe:", individual_to_recipe_string(res_reg$best_individual), "\n")
cat("Fitness (neg RMSE):", res_reg$best_individual$fitness, "\n")
```

```{r regression-predict}
preds_reg <- predict_model(res_reg, iris[1:10, ])
# Compare predictions to actuals
data.frame(
  actual    = iris$Petal.Length[1:10],
  predicted = round(preds_reg, 2)
)
```

## Multiclass Classification

Classify iris species (3 classes). Note `task = "multiclass"`:
  
```{r multiclass}
iris_mc <- iris
iris_mc$Species <- as.character(iris_mc$Species)

set.seed(99)
res_mc <- evolve_features(
  data       = iris,
  target_col = "Species",
  task       = "multiclass",
  evaluator  = "xgboost",
  generations = 5,
  pop_size    = 8,
  cv_folds    = 3,
  early_stopping_generations = 3,
  verbose     = TRUE
)

cat("Best recipe:", individual_to_recipe_string(res_mc$best_individual), "\n")
```

For multiclass, `predict_model()` returns a probability matrix -- one column per
class:

```{r multiclass-predict}
probs <- predict_model(res_mc, iris_mc[c(1, 51, 101), ])
round(probs, 3)
```

## Transformer Reference

evoFE ships with **42 built-in transformers** that the genetic algorithm can
select from during evolution. The table below groups them by category.

### Arithmetic (numeric -> numeric)

| Transformer | Arity | Description |
|:---|:---:|:---|
| `log` | unary | Safe natural logarithm: `log1p(abs(x))` |
| `sqrt` | unary | Safe square root: `sqrt(abs(x))` |
| `reciprocal` | unary | `1/x` (0 where `x == 0`) |
| `power` | unary | Signed exponentiation: `sign(x) * |x|^p`, $p \in \{0.5, 1/3, 2, 3\}$ |
| `displaced_log` | unary | `log1p(|x + d|)` where `d` is sampled from [10, 1000] |
| `add` | multi | Element-wise sum of 2+ columns |
| `subtract` | binary | $x_1 - x_2$ |
| `multiply` | multi | Element-wise product of 2+ columns |
| `divide` | binary | $x_1 / x_2$ (0 where denominator is 0) |
| `normalized_difference` | binary | $(x_1 - x_2) / (|x_1| + |x_2| + 10^{-6})$ |
| `log_ratio` | binary | $\log(1+|x_1|) - \log(1+|x_2|)$ |

### Rank / Distribution (numeric -> numeric)

| Transformer | Description |
|:---|:---|
| `rank_transform` | ECDF-based percentile rank mapped to [0, 1]; fit on training data, robust to outliers |

### Group-by Aggregations (mixed cat x num -> numeric)

These combine a **categorical** grouping column with a **numeric** value column.
All are stateful (fit on training data).

| Transformer | Description |
|:---|:---|
| `groupby_mean` | Per-group mean |
| `groupby_sd` | Per-group standard deviation |
| `groupby_max` | Per-group maximum |
| `groupby_min` | Per-group minimum |
| `groupby_median` | Per-group median (robust to outliers) |
| `groupby_quantile` | Per-group Q1 or Q3 ($q \in \{0.25, 0.75\}$) |
| `groupby_ratio` | `value / group_mean` |
| `groupby_zscore` | $(value - group\_mean) / group\_sd$ |

### Supervised Categorical Encodings (categorical -> numeric)

All are stateful (fit on training data only -- no leakage).

| Transformer | Description |
|:---|:---|
| `target_encode` | Smoothed mean-target encoding for binary classification / regression |
| `pooled_target_encode` | Empirical Bayes pooled target encoding with dynamic shrinkage based on target variance |
| `target_encode_multiclass` | Class-wise smoothed target encoding for multiclass tasks |
| `woe_encode` | Weight of Evidence: `ln(P(event|cat) / P(non-event|cat))` with Laplace smoothing; binary classification only |

### Unsupervised Encoding & Binning

| Transformer | Input -> Output | Description |
|:---|:---:|:---|
| `frequency_encode` | cat -> num | Count of each category level in training data |
| `one_hot_encode` | cat -> num | Binary indicator for up to 5 top categories plus an "other" bucket |
| `concat` | cat x cat -> cat | Concatenates 2 or 3 categorical columns with underscore separator |
| `quantile_binning` | num -> num | Quantile-based bin index (numeric output) |
| `quantile_binning_cat` | num -> cat | Quantile-based bin label (categorical output) |
| `log_binning` | num -> num | Log-scale bin index (numeric output) |
| `log_binning_cat` | num -> cat | Log-scale bin label (categorical output) |
| `datetime_extract` | date -> num | Extracted datetime component: year, month, day, hour, day-of-week, or weekend indicator |
| `date_diff` | date x date -> num | Signed difference in days between two datetime columns |

### Dimensionality Reduction (numeric -> numeric)

All are stateful (fit on training data).

| Transformer | Description |
|:---|:---|
| `pca` | Selected principal component from `prcomp` |
| `truncated_svd` | Selected component from truncated SVD |
| `random_projection` | Random unit-vector linear combination |
| `umap` | UMAP projection component (requires **uwot**) |

### Manifold & Graph Learning (numeric -> categorical or numeric)

All are stateful. Clustering is fit on a (optionally downsampled) training set;
new data is assigned to clusters via 1-NN lookup.

| Transformer | Output | Description |
|:---|:---:|:---|
| `genie` | categorical | Genie robust hierarchical cluster label (requires **genieclust**) |
| `genie_centroid_dist` | numeric | Distance to each Genie cluster centroid |
| `umap_genie` | categorical | Genie cluster labels on UMAP embedding (requires **uwot** + **genieclust**) |
| `lumbermark` | categorical | Lumbermark MST-based hierarchical cluster label (requires **lumbermark**) |
| `lumbermark_centroid_dist` | numeric | Distance to each Lumbermark cluster centroid |
| `umap_lumbermark` | categorical | Lumbermark cluster labels on UMAP embedding (requires **uwot** + **lumbermark**) |
| `mst_score` | numeric | MST-based anomaly score (requires **quitefastmst**) |
| `deadwood` | categorical | Deadwood outlier indicator (requires **deadwood**) |

## Hierarchical Features (Gene Chaining)

One of evoFE's powerful capabilities is **hierarchical feature construction**.
After a gene has been evaluated and proven useful, subsequent generations can
build *on top of* its output.

For example:

```
Gen 1: log_ratio(Sepal.Length, Petal.Width)        -> tested [OK]
Gen 2: divide(Petal.Width, logratio(...))            -> chains from tested gene [OK]
```

**Important safety rule**: a gene can only chain from outputs that have been
evaluated in a **previous** generation. A brand-new untested gene is never used
as input for another gene in the same individual. This prevents fragile
dependency chains built on unproven transformations.

## Hybrid Active Feature Mask

Each individual in evoFE not only carries a set of transformation genes, but
also an **active feature mask** -- a subset of the original raw input columns
that the individual sees. This allows the evolutionary search to simultaneously
optimise *which features to include* and *how to transform them*.

### How the mask works

At initialisation, the mask is seeded using the **baseline model importances**.
Features with higher importance (relative to the average) have a higher
probability of being included via a sigmoid function:

$$P(\text{include } x_i) = \sigma\!\left(\frac{\text{importance}(x_i) - \bar{\text{importance}}}{\tau}\right)$$

where $\tau$ is the temperature controlled by `mask_temp_factor`.

### Mask mutation operators

Three mask-specific mutation operators are randomly triggered during each
mutation event:

| Operator | Trigger probability | Effect |
|:---|:---:|:---|
| **Recalculate mask** | `recalculate_mask_prob` (default 0.05) | Redraws the entire mask from scratch using current importances |
| **Toggle raw feature(s)** | `raw_toggle_prob` (default 0.15) | Activates or deactivates one or more features; count drawn from a geometric distribution |
| **Gene mutation** | `1 - recalculate_mask_prob - raw_toggle_prob` | Standard add/remove/modify gene operation |

### Tuning advice

```r
# More aggressive feature selection (wider exploration)
recipe <- evolve_features(
  data = df, target_col = "y", task = "classification",
  raw_toggle_prob = 0.25,       # toggle features more often
  recalculate_mask_prob = 0.10, # recalculate mask more often
  mask_temp_factor = 1.0        # flatter importance distribution
)

# Conservative -- mostly stick to all features, rely on gene mutations
recipe <- evolve_features(
  data = df, target_col = "y", task = "classification",
  raw_toggle_prob = 0.05,
  recalculate_mask_prob = 0.02,
  mask_temp_factor = 0.3  # sharper -- concentrate on highest-importance features
)
```

## Island Model

The island model partitions the population into independent sub-populations
(**islands**) that evolve in parallel. Periodic **migration** exchanges
successful recipes and genes between islands, helping the overall search escape
local optima while preserving diversity.

### Recipe-level migration (Ring topology)

Every `migration_interval` generations, the top `migration_rate` individuals
from each island are copied into the neighbouring island's population (ring
topology). This preserves co-adapted gene interactions.

### Gene-level migration (Injection)

In addition to whole-recipe migration, individual genes from each island's best
individual are periodically injected into a neighbour island's mutation pool.
When a mutation event fires on the receiving island, there is a
`gene_migration_prob` chance that a migrated gene is injected instead of a
random new gene.

### Basic island example

```r
recipe <- evolve_features(
  data = df,
  target_col = "am",
  task = "classification",
  evaluator = "xgboost",
  generations = 10,
  pop_size = 8,
  islands = 4,                  # 4 independent sub-populations
  migration_interval = 5,       # migrate every 5 generations
  migration_rate = 2,           # top-2 individuals migrate
  gene_migration_prob = 0.2,    # 20% chance to inject a migrated gene
  verbose = TRUE
)
```

### Row-split islands

When `row_split_islands = TRUE`, each island receives a distinct subset of the
training rows. This is useful for very large datasets where evaluating every
individual on the full dataset is prohibitively slow.

```r
recipe <- evolve_features(
  data = big_df,
  target_col = "y",
  task = "regression",
  islands = 4,
  row_split_islands = TRUE,     # each island sees ~25% of rows
  evaluation_strategy = "split",
  split_ratio = c(0.6, 0.2, 0.2)
)
```

When combined with `per_island_validation = TRUE`, each island also uses its
own local validation split during the evolutionary search. Only the final
tournament (between islands) uses the global validation set. This prevents
cross-island data leakage during evolution.

### Heterogeneous transformer pools

Each island can search over a different transformer vocabulary:

```r
recipe <- evolve_features(
  data = df,
  target_col = "am",
  task = "classification",
  islands = 2,
  allowed_transformers = list(
    "basic",       # island 1: arithmetic + encodings only
    "clustering"   # island 2: UMAP, Genie, MST, etc.
  )
)
```

### Multi-Evaluator Island Portfolios

Islands can also run different machine learning backends simultaneously. Passing a vector of evaluator names assigns each model family to a dedicated island:

```r
recipe <- evolve_features(
  data = df,
  target_col = "am",
  task = "classification",
  islands = 2,
  evaluator = c("lightgbm", "xgboost"),
  generations = 5,
  pop_size = 6
)
```

Island 1 discovers features optimized for LightGBM's histogram-based splitting, while Island 2 discovers features optimized for XGBoost's exact/approximate greedy trees.

### Advanced Topologies & Migration Policies

For large-scale or structured exploration, evoFE provides graph-theoretic island topologies and adaptive probabilistic migration policies via `migration_config()`:

```r
# Torus topology with demand-driven Gibbs pull migration
mig_cfg <- migration_config(
  topology = topology_torus(c(2, 2)), # 4 islands arranged on a 2x2 toroidal grid
  policy = policy_gibbs_pull(),        # stagnant islands actively pull innovations from leading islands
  interval = 4,
  rate = 1,
  gene_prob = 0.25
)

recipe <- evolve_features(
  data = df,
  target_col = "am",
  task = "classification",
  migration = mig_cfg,
  generations = 10
)
```

Available topologies include:
- `topology_ring(k)`: Unidirectional ring (default)
- `topology_complete(k)`: Fully connected mesh
- `topology_torus(dims)`: N-dimensional torus with wrap-around boundaries
- `topology_hypercube(dim)`: $2^d$-vertex binary hypercube
- `topology_grid(dims)`: N-dimensional Cartesian grid with fixed boundaries
- `topology_tiered(tiers)`: Hierarchical fitness-stratified tiers (HFC architecture)
- `topology_custom(adjacency_matrix)`: User-defined directed graph

Available migration policies include:
- `policy_push_uniform()`: Classic uniform broadcast to downstream neighbors
- `policy_gibbs_push()`: Softmax-weighted push directing migrants toward lower-fitness islands
- `policy_gibbs_pull()`: Demand-driven pull where stagnated islands pull elite solutions from high-performing neighbors
- `policy_tiered_admission()`: Fitness-gated upward migration through hierarchical tiers

## Caruana Post-Hoc Island Ensembling

Multi-island evolution naturally produces diverse, complementary feature recipes across islands. Rather than picking only the single best recipe, `ensemble_islands()` uses **Caruana greedy forward selection with replacement** to construct an optimal weighted ensemble across all island champions:

```r
# 1. Evolve features across 3 islands
set.seed(42)
multi_recipe <- evolve_features(
  data = mtcars,
  target_col = "am",
  task = "classification",
  evaluator = "xgboost",
  islands = 3,
  generations = 5,
  pop_size = 6,
  verbose = FALSE
)

# 2. Greedily build the Caruana ensemble from island populations
ensemble <- ensemble_islands(
  multi_recipe,
  data = mtcars,
  max_rounds = 10,
  verbose = TRUE
)

# Inspect the ensemble weights and member recipes
summary(ensemble)

# 3. Predict directly with the ensemble
ensemble_preds <- predict_model(ensemble, mtcars[1:5, ])
ensemble_preds
```

`predict_model.evo_ensemble()` automatically routes the new observations through each member recipe's transformation pipeline and computes the weighted blend of model predictions.

## Custom Transformer Registration

evoFE makes it easy to register your own custom transformations. Use
`create_transformer()` to define your transformer, and `register_transformer()`
to make it available during evolution:

```r
library(evoFE)

# 1. Define a transformer that adds a constant to a numeric variable
add_five_trans <- create_transformer(
  name = "add_five",
  type = "unary",
  input_type = "numeric",
  apply_func = function(data, gene, state = NULL) {
    data[[gene$input_cols[1]]] + 5
  },
  name_generator = function(gene) paste0("add5_", gene$input_cols[1])
)

# 2. Register it with the package registry
register_transformer("add_five", add_five_trans)

# Now "add_five" is part of the transformer pool for all future evolution runs.
# You can also restrict to only your custom transformer:
recipe <- evolve_features(
  data = df, target_col = "am", task = "classification",
  allowed_transformers = c("log", "sqrt", "add_five")
)
```

### Stateful (fit-on-train) transformers

If your transformer needs to fit parameters on the training set (e.g., a
custom normalisation), provide a `fit_func`:

```r
z_score_trans <- create_transformer(
  name = "z_score",
  type = "unary",
  input_type = "numeric",
  fit_func = function(data, gene, target_col = NULL) {
    x <- data[[gene$input_cols[1]]]
    list(mean = mean(x, na.rm = TRUE), sd = sd(x, na.rm = TRUE))
  },
  apply_func = function(data, gene, state = NULL) {
    x <- data[[gene$input_cols[1]]]
    if (is.null(state) || state$sd == 0) return(rep(0, length(x)))
    (x - state$mean) / state$sd
  },
  name_generator = function(gene) paste0("zscore_", gene$input_cols[1])
)
register_transformer("z_score", z_score_trans)
```

## Custom Evaluator Registration

evoFE is not limited to LightGBM and XGBoost. You can register any ML backend
using `register_evaluator()`:

```r
# Register a simple linear model evaluator backed by glmnet
register_evaluator(
  "my_lm",
  train_func = function(x_train, y_train, x_val = NULL,
                        task = "regression", ...) {
    model <- lm(y ~ ., data = as.data.frame(cbind(y = y_train, x_train)))
    preds <- if (!is.null(x_val)) {
      predict(model, newdata = as.data.frame(x_val))
    } else {
      NULL
    }
    list(model = model, predictions = preds,
         importances = setNames(rep(1, ncol(x_train)), colnames(x_train)))
  },
  predict_func = function(model, x_new, task, ...) {
    predict(model, newdata = as.data.frame(x_new))
  }
)

# Use it in evolution
recipe <- evolve_features(
  data = df, target_col = "y", task = "regression",
  evaluator = "my_lm"
)
```

## Bayesian Hyperparameter Tuning

`make_tunable()` wraps any registered evaluator in a `mlr3mbo` Bayesian
Optimisation loop that automatically tunes hyperparameters during each
fitness evaluation:

```r
# Define parameter search space for XGBoost
param_ranges <- list(
  eta         = list(type = "numeric", lower = 0.01, upper = 0.3),
  max_depth   = list(type = "integer", lower = 3,    upper = 9),
  subsample   = list(type = "numeric", lower = 0.5,  upper = 1.0)
)

# Register a tunable version of xgboost
make_tunable("xgboost", param_ranges, tuner_name = "xgboost_tuned")

# Use it exactly like a regular evaluator
recipe <- evolve_features(
  data = df,
  target_col = "am",
  task = "classification",
  evaluator = "xgboost_tuned",
  generations = 5,
  pop_size = 6,
  mbo_iters = 5,          # Bayesian optimisation iterations per fitness eval
  mbo_init_design = 8     # initial LHS design points
)
```

evoFE ships two pre-configured tunable evaluators: `lightgbm_mbo` and
`xgboost_mbo` (registered at package load).

## Understanding the Output

`evolve_features()` returns an `evo_recipe` S3 object with:

| Field | Description |
|:---|:---|
| `best_individual` | The winning recipe (list of genes, active column sets, fitness) |
| `best_model` | The final LightGBM/XGBoost model trained on all data |
| `history` | Full final-generation population (for inspection) |
| `fitness_history` | Best fitness per generation (for `plot()`) |
| `task` | The task type used |
| `evaluator` | The evaluator used |
| `metric` | The optimization metric used |
| `classes` | Class labels (multiclass only) |

### Inspecting the recipe

```{r inspect-recipe}
ind <- res$best_individual

# Human-readable recipe string
cat(individual_to_recipe_string(ind), "\n")

# Number of evolved genes
cat("Evolved genes:", length(ind$genes), "\n")

# Original columns retained (active feature mask)
cat("Numeric cols: ", paste(ind$numeric_cols, collapse = ", "), "\n")
cat("Categorical cols:", paste(ind$categorical_cols, collapse = ", "), "\n")

# Individual gene details
for (g in ind$genes) {
  cat(sprintf("  %s(%s) -> %s\n",
    g$transformer_name,
    paste(g$input_cols, collapse = ", "),
    g$output_col))
}
```

## Evaluation Strategies

`evoFE` supports multiple evaluation strategies and cross-validation partition schemes designed to balance search speed, compute cost, and strict prevention of data leakage:

### Cross-Validation vs. Train/Val Splits

1. **Cross-Validation (`cv`)**: The default strategy. Evaluates fitness across $K$ folds (`cv_folds` parameter).
2. **Train/Validation/Holdout Split (`split`)**: Fast evaluation on large datasets using `split_ratio = c(0.6, 0.2, 0.2)` or explicit `split_ids`.

```r
split_ids <- sample(
  c("train", "val", "holdout"),
  nrow(df),
  replace = TRUE,
  prob = c(0.6, 0.2, 0.2)
)

recipe <- evolve_features(
  data = df, target_col = "am", task = "classification",
  split_ids = split_ids   # overrides evaluation_strategy -> "split" automatically
)
```

### Confirmation Holdout & Search-Gap Diagnostic

In iterative automated feature engineering, repeatedly evaluating hundreds of candidate pipelines on the same cross-validation folds can lead to *search-level overfitting* (selecting recipes that fit fold artifacts).

To safeguard against this, `holdout_frac` sets aside an untouched, stratified holdout sample before the search begins:

```r
recipe <- evolve_features(
  data = df,
  target_col = "am",
  task = "classification",
  evaluator = "xgboost",
  holdout_frac = 0.20, # 20% untouched holdout
  generations = 5,
  pop_size = 8
)

# Inspect the holdout fitness and search gap
cat("Selection CV fitness:", recipe$best_individual$fitness, "\n")
cat("Confirmation holdout:", recipe$holdout_fitness, "\n")
cat("Search gap:          ", recipe$search_gap, "\n")
```

The `search_gap` (holdout score minus selection score) provides an unbiased check of generalization. A large negative search gap immediately alerts you to search-level overfitting.

### Leakage-Safe Cross-Validation Strategies

Standard random K-fold CV can cause severe optimistic leakage when working with time-series or grouped observations. evoFE provides three fold construction strategies via `cv_strategy`:

| Strategy | When to use | Behavior |
|:---|:---|:---|
| `"random"` | Independent i.i.d. observations | Standard stratified random K-fold partitioning |
| `"time"` | Chronological / time-series data | Folds are contiguous chronological blocks ordered by `time_col`. The validation fold is strictly in the future of the training folds, preventing lookahead leakage. |
| `"group"` | Grouped / clustered observations (e.g. subjects, patients, stores) | All rows sharing a `group_col` entity are assigned to the same fold using greedy bin-packing, ensuring no subject appears in both train and validation. |

```r
# Temporal data -- chronological forward validation
recipe_time <- evolve_features(
  data = financial_df,
  target_col = "returns",
  task = "regression",
  cv_strategy = "time",
  time_col = "date",
  cv_folds = 5
)

# Grouped data -- entity-isolated folds
recipe_group <- evolve_features(
  data = patient_df,
  target_col = "outcome",
  task = "classification",
  cv_strategy = "group",
  group_col = "patient_id",
  cv_folds = 5
)
```

### Multi-Fidelity Evolutionary Screening

For large datasets, evaluating every candidate generation at full dataset size can be compute-intensive. Enabling `multi_fidelity = TRUE` uses a warm-up screening schedule:

```r
recipe_fast <- evolve_features(
  data = big_df,
  target_col = "target",
  task = "classification",
  multi_fidelity = TRUE,
  mf_warmup_frac = 0.40,  # first 40% of generations screen on subsamples
  mf_sample_frac = 0.50,  # screen on 50% row subsamples
  generations = 10,
  pop_size = 12
)
```

During warm-up, candidates are screened rapidly on subsampled folds. Crucially, the top-performing half of candidates is **re-evaluated at full fidelity** before selection, ensuring that all fitness comparisons remain 100% apples-to-apples.

## Alternative and Custom Metrics

By default, evoFE optimises LogLoss (classification) and RMSE (regression).
Pass the `metric` parameter to change this:

| Task | Supported metrics |
|:---|:---|
| `classification` | `"default"` (LogLoss), `"auc"`, `"f1"`, `"ts-refinement"` |
| `multiclass` | `"default"` (Multiclass LogLoss), `"auc"`, `"ts-refinement"` |
| `regression` | `"default"` (neg RMSE), `"mae"` |
| Any | Custom function `function(y_true, y_pred)` returning numeric (higher = better) |

```r
# Optimize AUC
recipe_auc <- evolve_features(
  data = df, target_col = "am", task = "classification",
  metric = "auc", generations = 5, pop_size = 8
)

# Custom metric (MAPE, negated because evoFE maximises)
mape_metric <- function(y_true, y_pred) {
  -mean(abs((y_true - y_pred) / (y_true + 1e-8)))
}

recipe_mape <- evolve_features(
  data = iris[, 1:5], target_col = "Petal.Length", task = "regression",
  metric = mape_metric, generations = 5, pop_size = 8
)
```

### TS-Refinement metric

TS-Refinement is a calibration-aware classification metric that finds the
temperature $T$ minimizing the log-loss of temperature-scaled prediction
margins, then reports the unsmoothed log-loss at that temperature:

$$\mathcal{L}_\text{TS} = \min_{T > 0} \, \text{LogLoss}\!\left(\sigma\!\left(\frac{z}{T}\right),\, y\right)$$

It is more discriminating than raw log-loss when models differ mainly in
calibration rather than ranking quality. Enable it with:

```r
recipe <- evolve_features(
  data = df, target_col = "am", task = "classification",
  metric = "ts-refinement"
)
```

You can also call `compute_ts_refinement()` directly on your own predictions:

```r
score <- compute_ts_refinement(
  y_true = c(0, 1, 1, 0),
  y_pred = c(0.2, 0.8, 0.7, 0.3),
  task   = "classification"
)
cat("TS-Refinement log-loss:", score, "\n")
```

## Parameter Reference

### `evolve_features()` -- full parameter table

| Parameter | Default | Description |
|:---|:---:|:---|
| `data` | - | Input data.frame or data.table |
| `target_col` | - | Name of the target column |
| `task` | `"classification"` | `"classification"`, `"multiclass"`, or `"regression"` |
| `generations` | 10 | Maximum evolutionary generations |
| `pop_size` | 10 | Individuals per generation |
| `cv_folds` | 3 | CV folds (only with `evaluation_strategy = "cv"`) |
| `evaluation_strategy` | `"cv"` | `"cv"` or `"split"` |
| `split_ratio` | `c(0.6, 0.2, 0.2)` | Train/Val/Holdout proportions for `"split"` |
| `split_ids` | `NULL` | User-supplied split assignments (`"train"`, `"val"`, `"holdout"`) |
| `holdout_frac` | 0 | Stratified holdout fraction frozen during search for unbiased confirmation |
| `cv_strategy` | `"random"` | CV fold construction: `"random"`, `"time"`, or `"group"` |
| `time_col` | `NULL` | Timestamp column used when `cv_strategy = "time"` |
| `group_col` | `NULL` | Group entity column used when `cv_strategy = "group"` |
| `multi_fidelity` | `FALSE` | Enable two-stage row-subsampling evaluation during warm-up |
| `mf_sample_frac` | 0.5 | Row fraction kept per fold during multi-fidelity screening |
| `mf_warmup_frac` | 0.5 | Fraction of total generations run in multi-fidelity screening mode |
| `early_stopping_generations` | 3 | Stop after N generations without improvement |
| `evaluator` | `"lightgbm"` | ML backend: `"lightgbm"`, `"xgboost"`, `"catboost"`, `"lm"`, `"keras3"`, or vector of backends per island |
| `seed` | `NULL` | Integer RNG seed for fully reproducible evolutionary search (CRAN-safe) |
| `dynamic_population` | `TRUE` | Expand population dynamically during stagnation |
| `dynamic_population_growth_rate` | 1.5 | Growth multiplier during stagnation |
| `dynamic_population_decay_rate` | 0.7 | Decay multiplier when improvement resumes |
| `crossover_type` | `"both"` | `"random"`, `"union"`, or `"both"` (50/50 mix) |
| `threads` | 2 | Threads for model training and clustering |
| `max_clustering_size` | 5000 | Max unique rows passed to clustering transformers |
| `verbose` | `TRUE` | Print progress to console |
| `metric` | `"default"` | Optimisation metric (`"default"`, `"auc"`, `"f1"`, `"mae"`, `"ts-refinement"`, `"cal_rmse"`, `"cal_mae"`, or custom function) |
| `model_all_final_genes` | `FALSE` | Train final model on union of all genes in final population |
| `model_all_historical_genes` | `FALSE` | Train final model on union of all genes across all generations |
| `allowed_transformers` | `"all"` | Transformer pool: `"all"`, `"basic"`, `"robust"`, `"clustering"`, or a character vector |
| `complexity_penalty` | 0 | Multiplier for parsimony penalty (e.g. 1.0 for standard BIC/PAC-Bayes) |
| `complexity_mode` | `"bic_dynamic"` | Penalty mode: `"bic_dynamic"`, `"bic"`, `"pac_bayes_dynamic"`, `"pac_bayes"`, or `"none"` |
| `complexity_floor` | 0.20 | Minimum safety floor (e.g. 20%) for dynamic penalty relaxation |
| `complexity_target` | `"all_features"` | Target to penalize: `"all_features"` (rewards raw feature pruning) or `"genes"` |
| `migration` | `NULL` | Optional `evo_migration_config` object from `migration_config()` |
| `islands` | 1 | Number of independent sub-populations |
| `migration_interval` | 5 | Generations between island migrations |
| `migration_rate` | 1 | Top-N individuals migrated from each island |
| `gene_migration_prob` | 0.2 | Probability of injecting a migrated gene during mutation |
| `migration_topology` | `"ring"` | Topology: `"ring"`, `"gibbs_stagnation"`, `"gibbs_fitness"`, `"dual_gibbs_pull"`, `"random"` |
| `row_split_islands` | `FALSE` | Split training rows across islands |
| `per_island_validation` | `FALSE` | Use per-island local validation split (requires `row_split_islands = TRUE`) |
| `raw_toggle_prob` | 0.15 | Probability of toggling raw feature(s) in active mask during mutation |
| `recalculate_mask_prob` | 0.05 | Probability of redrawing entire active mask from importances |
| `mask_temp_factor` | 0.5 | Temperature for importance-guided mask sampling |
| `record` | `FALSE` | Enable live evolution viewer |
| `port` | `NULL` | Port for the live viewer server |
| `...` | - | Extra arguments passed to the evaluator's `train_func` |

### Transformer presets for `allowed_transformers`

| Preset | # Transformers | When to use |
|:---|:---:|:---|
| `"all"` | 42 | Default -- full search space |
| `"basic"` | ~18 | Fast runs; arithmetic + key encodings only |
| `"robust"` | ~24 | Outlier-resistant ops (rank, power, groupby median/quantile, WOE...) without heavy clustering |
| `"clustering"` | ~10 | Manifold & graph features only (UMAP, Genie, Lumbermark, MST...) |

## Advanced Options

### Dynamic BIC & PAC-Bayes Complexity Penalties

Without regularization, complex feature engineering algorithms risk evolving bloated recipes that overfit cross-validation noise. evoFE provides theoretically grounded complexity penalties that scale with dataset size $N$:

#### 1. Bayesian Information Criterion (`complexity_mode = "bic"` / `"bic_dynamic"`)
Uses asymptotic BIC scaling per feature:
$$p = \lambda \cdot \frac{\ln(N)}{2N}$$

#### 2. PAC-Bayes Generalization Bound (`complexity_mode = "pac_bayes"` / `"pac_bayes_dynamic"`)
Uses PAC-Bayes generalization bound scaling per feature:
$$p = \lambda \cdot \frac{1}{2\sqrt{N}}$$

#### Dynamic Relaxation & Safety Floor
Under dynamic modes (`"bic_dynamic"` and `"pac_bayes_dynamic"`), the penalty dynamically relaxes as evolution converges towards ideal fitness, preventing late-generation stagnation while enforcing a safety floor (`complexity_floor = 0.20`, or 20% of base penalty).

#### Feature Count Targets
- `complexity_target = "all_features"` (default): Penalizes the total number of active features (retained raw inputs + derived genes). This actively incentivizes the genetic algorithm to prune uninformative raw columns from the active mask.
- `complexity_target = "genes"`: Penalizes only the number of newly derived transformation genes.

```r
# Enable standard dynamic BIC parsimony with total feature regularization
recipe <- evolve_features(
  data = df,
  target_col = "am",
  task = "classification",
  complexity_penalty = 1.0,           # standard 1.0x BIC multiplier
  complexity_mode = "bic_dynamic",
  complexity_target = "all_features", # reward pruning redundant raw features
  complexity_floor = 0.20             # 20% minimum floor
)
```

### Redundancy Pruning

During evolution, evoFE automatically rejects any new feature whose absolute
Pearson correlation with an existing feature exceeds a threshold (default
`0.95`). This prevents wasting evaluations on near-duplicate columns.

The threshold is tunable:

```r
# Tighten to 0.90 -- reject anything correlated above 90%
options(evoFE.redundancy_cor_threshold = 0.90)

# Disable entirely
options(evoFE.redundancy_cor_threshold = 1.0)
```

### Gene Pooling for the Final Model

By default the final model is trained only on the **best individual's** genes.
Two flags expand this:

```r
# Model trains on the union of all unique genes in the final population
recipe <- evolve_features(
  ..., model_all_final_genes = TRUE
)

# Model trains on the union of all unique genes across all generations
recipe <- evolve_features(
  ..., model_all_historical_genes = TRUE
)
```

This can help recover useful genes eliminated by stochastic noise.

### Convergence Plot

`plot(recipe)` produces a generation-by-generation fitness curve:

```r
plot(recipe, type = "fitness")    # fitness trajectory
plot(recipe, type = "importance") # top feature importances of winning model
```

### Live Evolution Viewer

Enable the real-time browser dashboard:

```r
recipe <- evolve_features(
  data = df, target_col = "am", task = "classification",
  record = TRUE  # opens a browser tab with live streaming updates
)
```

## Reproducibility

Calling `set.seed()` before `evolve_features()` guarantees identical results across runs:

```{r reproducibility}
set.seed(42)
r1 <- evolve_features(iris[,1:5], "Petal.Length", task = "regression",
                       generations = 3, pop_size = 5, evaluator = "xgboost",
                       verbose = FALSE)
set.seed(42)
r2 <- evolve_features(iris[,1:5], "Petal.Length", task = "regression",
                       generations = 3, pop_size = 5, evaluator = "xgboost",
                       verbose = FALSE)

identical(r1$best_individual$fitness, r2$best_individual$fitness)
identical(
  individual_to_recipe_string(r1$best_individual),
  individual_to_recipe_string(r2$best_individual)
)
```

## End-to-End Example: Train/Test Split

A realistic workflow with hold-out evaluation:

```{r end-to-end}
data(iris)
set.seed(1)
idx <- sample(nrow(iris), 0.7 * nrow(iris))
train <- iris[idx, ]
test  <- iris[-idx, ]

# Evolve on training data only
set.seed(7)
recipe <- evolve_features(
  data      = train[, 1:4],               # exclude Species
  target_col = "Petal.Length",
  task       = "regression",
  evaluator  = "xgboost",
  generations = 5,
  pop_size    = 8,
  verbose     = FALSE
)

# Predict on held-out test data
test_preds <- predict_model(recipe, test[, 1:4])

# Evaluate
rmse <- sqrt(mean((test$Petal.Length - test_preds)^2))
cat(sprintf("Test RMSE: %.4f\n", rmse))
cat(sprintf("Recipe:    %s\n", individual_to_recipe_string(recipe$best_individual)))
```

## Session Info

```{r session-info}
sessionInfo()
```