In previous chapters, we learned a large number of transforms for manipulating and processing geotables. In all those code examples, we used Juliaโs pipe operator |> to apply the transform and send the resulting geotable to the next transform:
In this chapter, we will learn two new powerful operators โ and โ provided by the framework to combine transforms into pipelines that can be optimized and reused with different geotables.
7.1 Motivation
The pipe operator |> in Julia is very convenient for sequential application of functions. Given an input x, we can type x |> f1 |> f2 to apply functions f1 and f2 in sequence, in a way that is equivalent to f2(f1(x)) or, alternatively, to the function composition (f2 โ f1)(x). Its syntax can drastically improve code readability when the number of functions is large. However, the operator has a major limitation in the context of geospatial data science: it evaluates all intermediate results as soon as the data is inserted in the pipe. This is known in computer science as eager evaluation.
Taking the expression above as an example, the operator will first evaluate f1(x) and store the result in a variable y. After f1 is completed, the operator evaluates f2(y) and produces the final (desired) result. If y requires a lot of computer memory as it is usually the case with large geotables, the application of the pipeline will be slow.
Another evaluation strategy, known as lazy evaluation, consists of building the entire pipeline without the data in it. The major advantage of this strategy is that it can analyze the functions, and potentially simplify the code before evaluation. For example, the pipeline cos โ acos can be replaced by the much simpler pipeline identity for some values of the input x.
7.2 Operator โ
In our framework, the operator โ (\to) can be used in place of the pipe operator to build lazy sequential pipelines of transforms. Consider the synthetic data from previous chapters:
N =10000a = [2randn(Nรท2) .+6; randn(Nรท2)]b = [3randn(Nรท2); 2randn(Nรท2)]c =randn(N)d = c .+0.6randn(N)table = (; a, b, c, d)gtb =georef(table, CartesianGrid(100, 100))
10000ร5 GeoTable over 100ร100 CartesianGrid
a
b
c
d
geometry
Continuous
Continuous
Continuous
Continuous
Quadrangle
[NoUnits]
[NoUnits]
[NoUnits]
[NoUnits]
๐ Cartesian{NoDatum}
3.57728
1.49707
-0.604643
-0.0807463
Quadrangle((x: 0.0 m, y: 0.0 m), ..., (x: 0.0 m, y: 1.0 m))
4.93356
-4.33672
-1.18603
-1.69285
Quadrangle((x: 1.0 m, y: 0.0 m), ..., (x: 1.0 m, y: 1.0 m))
6.10929
1.45729
2.40183
2.38922
Quadrangle((x: 2.0 m, y: 0.0 m), ..., (x: 2.0 m, y: 1.0 m))
9.90976
-1.45903
1.59394
1.69104
Quadrangle((x: 3.0 m, y: 0.0 m), ..., (x: 3.0 m, y: 1.0 m))
9.61208
-0.698047
0.0432304
0.607825
Quadrangle((x: 4.0 m, y: 0.0 m), ..., (x: 4.0 m, y: 1.0 m))
3.02813
1.32257
2.46016
2.92883
Quadrangle((x: 5.0 m, y: 0.0 m), ..., (x: 5.0 m, y: 1.0 m))
4.65723
-4.04275
-0.100213
-0.9566
Quadrangle((x: 6.0 m, y: 0.0 m), ..., (x: 6.0 m, y: 1.0 m))
8.74681
0.304121
-0.416786
-0.158384
Quadrangle((x: 7.0 m, y: 0.0 m), ..., (x: 7.0 m, y: 1.0 m))
5.65619
4.29829
-0.315877
-0.806317
Quadrangle((x: 8.0 m, y: 0.0 m), ..., (x: 8.0 m, y: 1.0 m))
7.78555
0.529702
-0.983216
-1.03447
Quadrangle((x: 9.0 m, y: 0.0 m), ..., (x: 9.0 m, y: 1.0 m))
โฎ
โฎ
โฎ
โฎ
โฎ
And suppose that we are interested in converting the columns โaโ, โbโ and โcโ of the geotable with the Quantile transform. Instead of creating the intermediate geotable with the Select transform, and then sending the result to the Quantile transform, we can create the entire pipeline without reference to the data:
The operator โ creates a special SequentialTransform, which can be applied like any other transform in the framework:
gtb |> pipeline
10000ร4 GeoTable over 100ร100 CartesianGrid
a
b
c
geometry
Continuous
Continuous
Continuous
Quadrangle
[NoUnits]
[NoUnits]
[NoUnits]
๐ Cartesian{NoDatum}
0.144127
0.61948
-0.587899
Quadrangle((x: 0.0 m, y: 0.0 m), ..., (x: 0.0 m, y: 1.0 m))
0.373737
-1.68287
-1.16357
Quadrangle((x: 1.0 m, y: 0.0 m), ..., (x: 1.0 m, y: 1.0 m))
0.713397
0.599259
2.39106
Quadrangle((x: 2.0 m, y: 0.0 m), ..., (x: 2.0 m, y: 1.0 m))
2.24763
-0.60949
1.59729
Quadrangle((x: 3.0 m, y: 0.0 m), ..., (x: 3.0 m, y: 1.0 m))
2.11301
-0.291329
0.0468911
Quadrangle((x: 4.0 m, y: 0.0 m), ..., (x: 4.0 m, y: 1.0 m))
0.0818072
0.549591
2.46243
Quadrangle((x: 5.0 m, y: 0.0 m), ..., (x: 5.0 m, y: 1.0 m))
0.314949
-1.58927
-0.0880965
Quadrangle((x: 6.0 m, y: 0.0 m), ..., (x: 6.0 m, y: 1.0 m))
1.75301
0.127683
-0.395329
Quadrangle((x: 7.0 m, y: 0.0 m), ..., (x: 7.0 m, y: 1.0 m))
0.574839
1.67466
-0.302593
Quadrangle((x: 8.0 m, y: 0.0 m), ..., (x: 8.0 m, y: 1.0 m))
1.35694
0.226259
-0.95298
Quadrangle((x: 9.0 m, y: 0.0 m), ..., (x: 9.0 m, y: 1.0 m))
โฎ
โฎ
โฎ
โฎ
It will perform optimizations whenever possible. For instance, we know a priori that adding the Identity transform anywhere in the pipeline doesnโt have any effect:
The operator โ (\sqcup) can be used to create lazy parallel transforms. There is no equivalent in Julia as this operator is very specific to tables. It combines the geotables produced by two or more pipelines into a single geotable with the disjoint union of all columns.
Letโs illustrate this concept with two pipelines:
Quadrangle((x: 0.0 m, y: 0.0 m), ..., (x: 0.0 m, y: 1.0 m))
false
true
true
-1.84033
-1.69827
Quadrangle((x: 1.0 m, y: 0.0 m), ..., (x: 1.0 m, y: 1.0 m))
false
false
true
3.14675
0.575166
Quadrangle((x: 2.0 m, y: 0.0 m), ..., (x: 2.0 m, y: 1.0 m))
false
false
true
2.15839
-0.565761
Quadrangle((x: 3.0 m, y: 0.0 m), ..., (x: 3.0 m, y: 1.0 m))
false
false
true
0.41134
-0.270647
Quadrangle((x: 4.0 m, y: 0.0 m), ..., (x: 4.0 m, y: 1.0 m))
false
true
true
3.51355
0.524095
Quadrangle((x: 5.0 m, y: 0.0 m), ..., (x: 5.0 m, y: 1.0 m))
false
true
true
-0.631099
-1.58194
Quadrangle((x: 6.0 m, y: 0.0 m), ..., (x: 6.0 m, y: 1.0 m))
false
false
true
-0.375954
0.118924
Quadrangle((x: 7.0 m, y: 0.0 m), ..., (x: 7.0 m, y: 1.0 m))
false
false
true
-0.698901
1.67743
Quadrangle((x: 8.0 m, y: 0.0 m), ..., (x: 8.0 m, y: 1.0 m))
false
false
true
-1.30394
0.20483
Quadrangle((x: 9.0 m, y: 0.0 m), ..., (x: 9.0 m, y: 1.0 m))
โฎ
โฎ
โฎ
โฎ
โฎ
โฎ
All 5 columns are present in the final geotable.
7.4 Revertibility
An important concept related to pipelines that is very useful in geospatial data science is revertibility. The concept is useful whenever we need to answer geoscientific questions in terms of variables that have been transformed for geostatistical analysis.
Letโs illustrate the concept with the following geotable and pipeline:
We saw that our pipelines can be evaluated with Juliaโs pipe operator:
gtb |> pipeline
4ร4 GeoTable over 4 PointSet
a
b
c
geometry
Continuous
Continuous
Continuous
Point
[NoUnits]
[NoUnits]
[NoUnits]
๐ Cartesian{NoDatum}
-3.0
-0.4
1.4
(x: 0.0 m, y: 0.0 m)
2.0
1.4
0.0
(x: 1.0 m, y: 0.0 m)
-0.4
-3.0
1.6
(x: 1.0 m, y: 1.0 m)
1.4
2.0
-3.0
(x: 0.0 m, y: 1.0 m)
In order to revert a pipeline, however; we need to save auxiliary constants that were used to transform the data (e.g., mean of selected columns). The apply function serves this purpose:
newgtb, cache =apply(pipeline, gtb)newgtb
4ร4 GeoTable over 4 PointSet
a
b
c
geometry
Continuous
Continuous
Continuous
Point
[NoUnits]
[NoUnits]
[NoUnits]
๐ Cartesian{NoDatum}
-3.0
-0.4
1.4
(x: 0.0 m, y: 0.0 m)
2.0
1.4
0.0
(x: 1.0 m, y: 0.0 m)
-0.4
-3.0
1.6
(x: 1.0 m, y: 1.0 m)
1.4
2.0
-3.0
(x: 0.0 m, y: 1.0 m)
The function produces the new geotable as usual and an additional cache with all the information needed to revert the transforms in the pipeline. We say that a pipeline isrevertible, if there is an efficient way to revert its transforms starting from any geotable that has the same schema of the geotable produced by the apply function:
isrevertible(pipeline)
true
revert(pipeline, newgtb, cache)
4ร4 GeoTable over 4 PointSet
a
b
c
geometry
Continuous
Continuous
Continuous
Point
[NoUnits]
[NoUnits]
[NoUnits]
๐ Cartesian{NoDatum}
-1.0
1.6
3.4
(x: 0.0 m, y: 0.0 m)
4.0
3.4
2.0
(x: 1.0 m, y: 0.0 m)
1.6
-1.0
3.6
(x: 1.0 m, y: 1.0 m)
3.4
4.0
-1.0
(x: 0.0 m, y: 1.0 m)
A very common workflow in geospatial data science consists of:
Transforming the data to an appropriate sample space for geostatistical analysis
Doing additional modeling to predict variables in new geospatial locations
Reverting the modeling results with the saved pipeline and cache
We will see examples of this workflow in Part V of the book.
7.5 Congratulations!
Congratulations on finishing Part II of the book. Letโs quickly review what we learned so far:
Transforms and pipelines are powerful tools to achieve reproducible geospatial data science.
The operators โ and โ can be used to build lazy pipelines. After a pipeline is built, it can be applied to different geotables, which may have different types of geospatial domain.
Lazy pipelines can always be optimized for computational performance, and the Julia language really thrives to dispatch the appropriate optimizations when they are available.
Map projections are specific types of coordinate transforms. They can be combined with many other transforms in the framework to produce advanced geostatistical visualizations.
There is a long journey until the technology reaches its full potential. The good news is that Julia code is easy to read and modify, and you can become an active contributor after just a few weeks working with the language. We invite you to contribute new transforms and optimizations as soon as you feel comfortable with the framework.