Cross-Validation

Overview

Cross-validation comparison

Cross-validation helps select optimal parameters (especially fraction) by evaluating performance on held-out data.

K-Fold Cross-Validation

Split data into K folds, train on K-1, validate on 1.

library(rfastlowess)
set.seed(42)
x <- seq(0, 2 * pi, length.out = 100)
y <- sin(x) + rnorm(100, sd = 0.3)

model <- Lowess(
    cv = cv_opts(method = "kfold", k = 5, fractions = c(0.2, 0.3, 0.5, 0.7))
)
result <- fit(model, x, y)

cat("Selected fraction:", result$fraction_used, "\n")
#> Selected fraction: 0.3
cat("CV scores:", result$cv_scores, "\n")
#> CV scores: 0.4950802 0.4716297 0.5044107 0.545229

Leave-One-Out (LOOCV)

Each point is held out once. Most thorough but slowest.

library(rfastlowess)
set.seed(42)
x <- seq(0, 2 * pi, length.out = 100)
y <- sin(x) + rnorm(100, sd = 0.3)

model <- Lowess(
    cv = cv_opts(method = "loocv", fractions = c(0.2, 0.3, 0.5, 0.7))
)
result <- fit(model, x, y)

cat("Selected fraction (CV):", result$fraction_used, "\n")
#> Selected fraction (CV): 0.3

Seeded Randomization

Set seed on Lowess() for reproducible fold assignments (the same seed is also used for bootstrap intervals):

library(rfastlowess)
set.seed(42)
x <- seq(0, 2 * pi, length.out = 100)
y <- sin(x) + rnorm(100, sd = 0.3)

model <- Lowess(
    cv = cv_opts(
        method = "kfold",
        k = 5,
        fractions = c(0.3, 0.5, 0.7)
    ),
    seed = 42
)
result <- fit(model, x, y)
cat("Selected fraction (CV):", result$fraction_used, "\n")
#> Selected fraction (CV): 0.3

Comparison

Method	Folds	Speed	Variance	Bias
KFold(5)	5	Fast	Moderate	Low
KFold(10)	10	Medium	Lower	Lower
LOOCV	N	Slow	Lowest	Lowest

Recommendation: Use 5-fold or 10-fold CV for most applications. LOOCV is only worth it for small datasets (N < 100).

CV Metrics

Cross-validation uses MSE (Mean Squared Error) by default:

MSE = mean((y_true - y_pred)^2)

Lower MSE indicates better fit on held-out data.

Interpreting Results

result$cv_scores contains the CV score for each candidate fraction in the order they were passed. The fraction with lowest CV score is automatically selected.

library(rfastlowess)
set.seed(42)
x <- seq(0, 2 * pi, length.out = 100)
y <- sin(x) + rnorm(100, sd = 0.3)

fractions <- c(0.1, 0.3, 0.5, 0.7)
model <- Lowess(cv = cv_opts(method = "kfold", k = 5, fractions = fractions))
result <- fit(model, x, y)

plot(fractions, result$cv_scores, type = "b",
    xlab = "Fraction", ylab = "CV Score (MSE)",
    main = "Cross-Validation Score by Fraction")
abline(v = result$fraction_used, col = "red", lty = 2)


cat("Selected fraction (CV):", result$fraction_used, "\n")
#> Selected fraction (CV): 0.3

Availability

Batch Mode Only: Cross-validation is only available in Batch mode.

Feature	Batch	Streaming	Online
K-Fold CV	✓	✗	✗
LOOCV	✓	✗	✗

Best Practices

Test a range: Include fractions from 0.1 to 0.9
Use enough folds: 5-10 folds balance speed and accuracy
Set a seed: For reproducible results
Check the curve: CV optimizes MSE, but visual inspection matters

sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04.1 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] rfastlowess_5.0.0 rmarkdown_2.32   
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.39    R6_2.6.1         fastmap_1.2.0    xfun_0.61       
#>  [5] maketools_1.3.2  cachem_1.1.0     knitr_1.52       htmltools_0.5.9 
#>  [9] buildtools_1.0.0 lifecycle_1.0.5  cli_3.6.6        sass_0.4.10     
#> [13] jquerylib_0.1.4  compiler_4.6.1   sys_3.4.3        tools_4.6.1     
#> [17] evaluate_1.0.5   bslib_0.12.0     yaml_2.3.12      otel_0.2.0      
#> [21] jsonlite_2.0.0   rlang_1.3.0