---
title: "Cross-Validation"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Cross-Validation}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

<!-- markdownlint-disable MD024 -->

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

## Overview

![Cross-validation comparison](../man/figures/cv_comparison.svg)

Cross-validation helps select optimal parameters (especially `fraction`)
by evaluating performance on held-out data.

---

## K-Fold Cross-Validation

Split data into K folds, train on K-1, validate on 1.

```{r cross_validation_1}
library(rfastlowess)
set.seed(42)
x <- seq(0, 2 * pi, length.out = 100)
y <- sin(x) + rnorm(100, sd = 0.3)

model <- Lowess(
    cv = cv_opts(method = "kfold", k = 5, fractions = c(0.2, 0.3, 0.5, 0.7))
)
result <- fit(model, x, y)

cat("Selected fraction:", result$fraction_used, "\n")
cat("CV scores:", result$cv_scores, "\n")
```

---

## Leave-One-Out (LOOCV)

Each point is held out once. Most thorough but slowest.

```{r cross_validation_2}
library(rfastlowess)
set.seed(42)
x <- seq(0, 2 * pi, length.out = 100)
y <- sin(x) + rnorm(100, sd = 0.3)

model <- Lowess(
    cv = cv_opts(method = "loocv", fractions = c(0.2, 0.3, 0.5, 0.7))
)
result <- fit(model, x, y)

cat("Selected fraction (CV):", result$fraction_used, "\n")
```

---

## Seeded Randomization

Set `seed` on `Lowess()` for reproducible fold assignments (the same seed
is also used for bootstrap intervals):

```{r cross_validation_3}
library(rfastlowess)
set.seed(42)
x <- seq(0, 2 * pi, length.out = 100)
y <- sin(x) + rnorm(100, sd = 0.3)

model <- Lowess(
    cv = cv_opts(
        method = "kfold",
        k = 5,
        fractions = c(0.3, 0.5, 0.7)
    ),
    seed = 42
)
result <- fit(model, x, y)
cat("Selected fraction (CV):", result$fraction_used, "\n")
```

---

## Comparison

| Method | Folds | Speed | Variance | Bias |
| --- | --- | --- | --- | --- |
| **KFold(5)** | 5 | Fast | Moderate | Low |
| **KFold(10)** | 10 | Medium | Lower | Lower |
| **LOOCV** | N | Slow | Lowest | Lowest |

> **Recommendation:** Use **5-fold** or **10-fold** CV for most
> applications. LOOCV is only worth it for small datasets (N < 100).

---

## CV Metrics

Cross-validation uses MSE (Mean Squared Error) by default:

```text
MSE = mean((y_true - y_pred)^2)
```

Lower MSE indicates better fit on held-out data.

---

## Interpreting Results

`result$cv_scores` contains the CV score for each candidate fraction in
the order they were passed. The fraction with **lowest CV score** is
automatically selected.

```{r cross_validation_4}
library(rfastlowess)
set.seed(42)
x <- seq(0, 2 * pi, length.out = 100)
y <- sin(x) + rnorm(100, sd = 0.3)

fractions <- c(0.1, 0.3, 0.5, 0.7)
model <- Lowess(cv = cv_opts(method = "kfold", k = 5, fractions = fractions))
result <- fit(model, x, y)

plot(fractions, result$cv_scores, type = "b",
    xlab = "Fraction", ylab = "CV Score (MSE)",
    main = "Cross-Validation Score by Fraction")
abline(v = result$fraction_used, col = "red", lty = 2)

cat("Selected fraction (CV):", result$fraction_used, "\n")
```

---

## Availability

> **Batch Mode Only:** Cross-validation is only available in **Batch** mode.

| Feature | Batch | Streaming | Online |
| --- | --- | --- | --- |
| K-Fold CV | ✓ | ✗ | ✗ |
| LOOCV | ✓ | ✗ | ✗ |

---

## Best Practices

1. **Test a range**: Include fractions from 0.1 to 0.9
2. **Use enough folds**: 5-10 folds balance speed and accuracy
3. **Set a seed**: For reproducible results
4. **Check the curve**: CV optimizes MSE, but visual inspection matters

```{r session-info}
sessionInfo()
```