Preface

This document is an online appendix to the paper “Statistical Models for the Analysis of Optimization Algorithms with Benchmark Functions” by David Issa Mattos, Jan Bosch and Helena Holmström Olsson. It shows the step-by-step process to analyze the data, including data preparation, modeling and plotting for all the models described on the paper.

Pre-requisites

To follow the code, we assume that the reader has some familiarity with the R environment including packages included of the tidyverse, such as dplyr and ggplot2. The code presented is described and fairly commented to help readers follow the modeling process. Other programming languages such as Python with numpy, pandas, matplotlib etc are capable of performing the same steps, but this is out of the scope of this document. For the Bayesian models, we try to minimize dependency on a specific R package such as brms or rstanarm, and therefore we discuss the model in Stan only, since it has bindings for multiple programming languages. The reader familiar with other languages might be interested in adapting these models and plots to the desired language,

Source code

The full source code is available in the repository https://github.com/davidissamattos/statscomp.

  • The dataset and the final data for each model (after the described data transformation) is also available for download in the ./data folder.
  • The Stan models are available in the ./stanmodels folder.
  • The utils folder contains some helper functions.

Compiling this document

This document was created with the bookdown package. To compile it (and run every command to generate the models, figures and etc. ) type:

bookdown::render_book('index.Rmd', 'all')

or alternatively use the custom function from the utils.R file. This function besides compiling the book generate the tables for the paper. We cannot generate latex tables (with correct labels) while compiling to bookdown_site. So this function takes the saved tables and generate them separately

compile_book()

Software environment

The environment used to compile this document is

sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 10.16

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] gtools_3.8.2         progress_1.2.2       ggthemr_1.1.0        viridis_0.5.1        viridisLite_0.3.0   
 [6] patchwork_1.0.1      coda_0.19-4          rstan_2.21.2         StanHeaders_2.21.0-6 glue_1.4.2          
[11] forcats_0.5.0        stringr_1.4.0        dplyr_1.0.2          purrr_0.3.4          readr_1.3.1         
[16] tidyr_1.1.2          tibble_3.0.4         ggplot2_3.3.2        tidyverse_1.3.0      kableExtra_1.2.1    
[21] rmdformats_0.3.7     knitr_1.30          

loaded via a namespace (and not attached):
 [1] matrixStats_0.57.0 fs_1.5.0           usethis_1.6.3      lubridate_1.7.9    devtools_2.3.2    
 [6] webshot_0.5.2      httr_1.4.2         rprojroot_2.0.2    tools_4.0.3        backports_1.2.1   
[11] R6_2.5.0           DBI_1.1.0          colorspace_2.0-0   withr_2.3.0        tidyselect_1.1.0  
[16] gridExtra_2.3      prettyunits_1.1.1  processx_3.4.5     curl_4.3           compiler_4.0.3    
[21] cli_2.2.0          rvest_0.3.6        HDInterval_0.2.2   xml2_1.3.2         desc_1.2.0        
[26] labeling_0.4.2     bookdown_0.21      scales_1.1.1       callr_3.5.1        digest_0.6.27     
[31] rmarkdown_2.6      pkgconfig_2.0.3    htmltools_0.5.0    sessioninfo_1.1.1  highr_0.8         
[36] dbplyr_1.4.4       rlang_0.4.9        readxl_1.3.1       rstudioapi_0.13    farver_2.0.3      
[41] generics_0.1.0     jsonlite_1.7.2     inline_0.3.17      magrittr_2.0.1     loo_2.4.1         
[46] Rcpp_1.0.5         munsell_0.5.0      fansi_0.4.1        lifecycle_0.2.0    stringi_1.5.3     
[51] yaml_2.2.1         pkgbuild_1.1.0     grid_4.0.3         blob_1.2.1         parallel_4.0.3    
[56] crayon_1.3.4       lattice_0.20-41    haven_2.3.1        hms_0.5.3          ps_1.5.0          
[61] pillar_1.4.7       codetools_0.2-16   stats4_4.0.3       pkgload_1.1.0      reprex_0.3.0      
[66] evaluate_0.14      V8_3.4.0           remotes_2.2.0      RcppParallel_5.0.2 modelr_0.1.8      
[71] vctrs_0.3.5        testthat_3.0.0     cellranger_1.1.0   gtable_0.3.0       assertthat_0.2.1  
[76] xfun_0.19          broom_0.7.0        memoise_1.1.0      ellipsis_0.3.1