Validation

GeoStats.jl was designed to, among other things, facilitate rigorous scientific comparison of different geostatistical solvers in the literature. As a user of geostatistics, you may be interested in applying various solvers on a given data set and pick the ones with best performance. As a researcher in the field, you may be interested in benchmarking your new method against other established methods.

Errors of geostatistical solvers can be estimated on given geostatistical problems:

Base.error — Method

error(solver, problem, method)

Estimate error of solver in a given problem with error estimation method.

Below is the list of currently implemented error estimation methods.

Leave-one-out validation

GeoStatsBase.LeaveOneOut — Type

LeaveOneOut(; loss=Dict())

Leave-one-out validation. Optionally, specify loss function from LossFunctions.jl for some of the variables.

References

Stone. 1974. Cross-Validatory Choice and Assessment of Statistical Predictions

Leave-ball-out validation

GeoStatsBase.LeaveBallOut — Type

LeaveBallOut(ball; loss=Dict())

Leave-ball-out (a.k.a. spatial leave-one-out) validation. Optionally, specify loss function from the LossFunctions.jl package for some of the variables.

LeaveBallOut(radius; loss=Dict())

By default, use Euclidean ball of given radius in space.

References

Le Rest et al. 2014. Spatial leave-one-out cross-validation for variable selection in the presence of spatial autocorrelation

Cross-validation

GeoStatsBase.CrossValidation — Type

CrossValidation(k; shuffle=true, loss=Dict())

k-fold cross-validation. Optionally, shuffle the data, and specify loss function from LossFunctions.jl for some of the variables.

References

Geisser, S. 1975. The predictive sample reuse method with applications
Burman, P. 1989. A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods

Block cross-validation

GeoStatsBase.BlockCrossValidation — Type

BlockCrossValidation(sides; loss=Dict())

Cross-validation with blocks of given sides. Optionally, specify loss function from LossFunctions.jl for some of the variables. If only one side is provided, then blocks become cubes.

References

Roberts et al. 2017. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure
Pohjankukka et al. 2017. Estimating the prediction performance of spatial models via spatial k-fold cross-validation

Weighted cross-validation

GeoStatsBase.WeightedCrossValidation — Type

WeightedCrossValidation(weigthing, folding; lambda=1.0, loss=Dict())

An error estimation method which samples are weighted with weighting method and split into folds with folding method. Weights are raised to lambda power in [0,1]. Optionally, specify loss function from LossFunctions.jl for some of the variables.

References

Sugiyama et al. 2006. Importance-weighted cross-validation for covariate shift
Sugiyama et al. 2007. Covariate shift adaptation by importance weighted cross validation

Density-ratio validation

GeoStatsBase.DensityRatioValidation — Type

DensityRatioValidation(k; [parameters])

Desntity ratio validation where weights are first obtained with density ratio estimation, and then used in k-fold weighted cross-validation.

Parameters

shuffle - Shuffle the data before folding (default to true)
estimator - Density ratio estimator (default to LSIF())
optlib - Optimization library (default to default_optlib(estimator))
lambda - Power of density ratios (default to 1.0)

Please see DensityRatioEstimation.jl for a list of supported estimators.

References

Hoffimann et al. 2020. Geostatistical Learning: Challenges and Opportunities