7  Building pipelines

In previous chapters, we learned a large number of transforms for manipulating and processing geotables. In all those code examples, we used Juliaโ€™s pipe operator |> to apply the transform and send the resulting geotable to the next transform:

geotable |> transform1 |> transform2 |> ... |> viewer

In this chapter, we will learn two new powerful operators โ†’ and โŠ” provided by the framework to combine transforms into pipelines that can be optimized and reused with different geotables.

7.1 Motivation

The pipe operator |> in Julia is very convenient for sequential application of functions. Given an input x, we can type x |> f1 |> f2 to apply functions f1 and f2 in sequence, in a way that is equivalent to f2(f1(x)) or, alternatively, to the function composition (f2 โˆ˜ f1)(x). Its syntax can drastically improve code readability when the number of functions is large. However, the operator has a major limitation in the context of geospatial data science: it evaluates all intermediate results as soon as the data is inserted in the pipe. This is known in computer science as eager evaluation.

Taking the expression above as an example, the operator will first evaluate f1(x) and store the result in a variable y. After f1 is completed, the operator evaluates f2(y) and produces the final (desired) result. If y requires a lot of computer memory as it is usually the case with large geotables, the application of the pipeline will be slow.

Another evaluation strategy, known as lazy evaluation, consists of building the entire pipeline without the data in it. The major advantage of this strategy is that it can analyze the functions, and potentially simplify the code before evaluation. For example, the pipeline cos โ†’ acos can be replaced by the much simpler pipeline identity for some values of the input x.

7.2 Operator โ†’

In our framework, the operator โ†’ (\to) can be used in place of the pipe operator to build lazy sequential pipelines of transforms. Consider the synthetic data from previous chapters:

N = 10000
a = [2randn(Nรท2) .+ 6; randn(Nรท2)]
b = [3randn(Nรท2); 2randn(Nรท2)]
c = randn(N)
d = c .+ 0.6randn(N)

table = (; a, b, c, d)

gt = georef(table, CartesianGrid(100, 100))
10000ร—5 GeoTable over 100ร—100 CartesianGrid
a b c d geometry
Continuous Continuous Continuous Continuous Quadrangle
[NoUnits] [NoUnits] [NoUnits] [NoUnits] ๐Ÿ–ˆ Cartesian{NoDatum}
6.90934 1.01635 -1.04514 -1.54456 Quadrangle((x: 0.0 m, y: 0.0 m), ..., (x: 0.0 m, y: 1.0 m))
9.03891 1.38566 -0.0333763 0.555749 Quadrangle((x: 1.0 m, y: 0.0 m), ..., (x: 1.0 m, y: 1.0 m))
4.48749 4.00113 -0.156711 -0.331299 Quadrangle((x: 2.0 m, y: 0.0 m), ..., (x: 2.0 m, y: 1.0 m))
6.04195 -0.119898 -0.042759 0.165475 Quadrangle((x: 3.0 m, y: 0.0 m), ..., (x: 3.0 m, y: 1.0 m))
5.28475 -1.1691 -0.371992 -0.712822 Quadrangle((x: 4.0 m, y: 0.0 m), ..., (x: 4.0 m, y: 1.0 m))
6.00031 -3.28344 0.17003 -0.404186 Quadrangle((x: 5.0 m, y: 0.0 m), ..., (x: 5.0 m, y: 1.0 m))
5.45968 1.75558 1.45305 1.5119 Quadrangle((x: 6.0 m, y: 0.0 m), ..., (x: 6.0 m, y: 1.0 m))
8.30869 -3.73731 -0.282818 -0.203502 Quadrangle((x: 7.0 m, y: 0.0 m), ..., (x: 7.0 m, y: 1.0 m))
8.27907 0.810659 -0.0387099 0.793324 Quadrangle((x: 8.0 m, y: 0.0 m), ..., (x: 8.0 m, y: 1.0 m))
6.53663 2.67851 -0.242385 -0.249597 Quadrangle((x: 9.0 m, y: 0.0 m), ..., (x: 9.0 m, y: 1.0 m))
โ‹ฎ โ‹ฎ โ‹ฎ โ‹ฎ โ‹ฎ

And suppose that we are interested in converting the columns โ€œaโ€, โ€œbโ€ and โ€œcโ€ of the geotable with the Quantile transform. Instead of creating the intermediate geotable with the Select transform, and then sending the result to the Quantile transform, we can create the entire pipeline without reference to the data:

pipeline = Select("a", "b", "c") โ†’ Quantile()
SequentialTransform
โ”œโ”€ Select(selector: [:a, :b, :c], newnames: nothing)
โ””โ”€ Quantile(selector: all, dist: Distributions.Normal{Float64}(ฮผ=0.0, ฯƒ=1.0))

The operator โ†’ creates a special SequentialTransform, which can be applied like any other transform in the framework:

gt |> pipeline
10000ร—4 GeoTable over 100ร—100 CartesianGrid
a b c geometry
Continuous Continuous Continuous Quadrangle
[NoUnits] [NoUnits] [NoUnits] ๐Ÿ–ˆ Cartesian{NoDatum}
0.980174 0.432745 -1.04332 Quadrangle((x: 0.0 m, y: 0.0 m), ..., (x: 0.0 m, y: 1.0 m))
1.88227 0.585517 -0.0195529 Quadrangle((x: 1.0 m, y: 0.0 m), ..., (x: 1.0 m, y: 1.0 m))
0.280883 1.61367 -0.142861 Quadrangle((x: 2.0 m, y: 0.0 m), ..., (x: 2.0 m, y: 1.0 m))
0.683011 -0.0408694 -0.0308364 Quadrangle((x: 3.0 m, y: 0.0 m), ..., (x: 3.0 m, y: 1.0 m))
0.472458 -0.493867 -0.363007 Quadrangle((x: 4.0 m, y: 0.0 m), ..., (x: 4.0 m, y: 1.0 m))
0.672603 -1.32552 0.165183 Quadrangle((x: 5.0 m, y: 0.0 m), ..., (x: 5.0 m, y: 1.0 m))
0.518657 0.739176 1.43742 Quadrangle((x: 6.0 m, y: 0.0 m), ..., (x: 6.0 m, y: 1.0 m))
1.5382 -1.48857 -0.271769 Quadrangle((x: 7.0 m, y: 0.0 m), ..., (x: 7.0 m, y: 1.0 m))
1.52926 0.349385 -0.0263226 Quadrangle((x: 8.0 m, y: 0.0 m), ..., (x: 8.0 m, y: 1.0 m))
0.848427 1.10168 -0.229861 Quadrangle((x: 9.0 m, y: 0.0 m), ..., (x: 9.0 m, y: 1.0 m))
โ‹ฎ โ‹ฎ โ‹ฎ โ‹ฎ

It will perform optimizations whenever possible. For instance, we know a priori that adding the Identity transform anywhere in the pipeline doesnโ€™t have any effect:

pipeline โ†’ Identity()
SequentialTransform
โ”œโ”€ Select(selector: [:a, :b, :c], newnames: nothing)
โ””โ”€ Quantile(selector: all, dist: Distributions.Normal{Float64}(ฮผ=0.0, ฯƒ=1.0))

7.3 Operator โŠ”

The operator โŠ” (\sqcup) can be used to create lazy parallel transforms. There is no equivalent in Julia as this operator is very specific to tables. It combines the geotables produced by two or more pipelines into a single geotable with the disjoint union of all columns.

Letโ€™s illustrate this concept with two pipelines:

pipeline1 = Select("a") โ†’ Indicator("a", k=3)
SequentialTransform
โ”œโ”€ Select(selector: [:a], newnames: nothing)
โ””โ”€ Indicator(selector: :a, k: 3, scale: :quantile, categ: false)
pipeline2 = Select("b", "c", "d") โ†’ PCA(maxdim=2)
SequentialTransform
โ”œโ”€ Select(selector: [:b, :c, :d], newnames: nothing)
โ”œโ”€ ZScore(selector: all)
โ””โ”€ EigenAnalysis(proj: :V, maxdim: 2, pratio: 1.0)

The first pipeline creates 3 indicator variables from variable โ€œaโ€:

gt |> pipeline1
10000ร—4 GeoTable over 100ร—100 CartesianGrid
a_1 a_2 a_3 geometry
Categorical Categorical Categorical Quadrangle
[NoUnits] [NoUnits] [NoUnits] ๐Ÿ–ˆ Cartesian{NoDatum}
false false true Quadrangle((x: 0.0 m, y: 0.0 m), ..., (x: 0.0 m, y: 1.0 m))
false false true Quadrangle((x: 1.0 m, y: 0.0 m), ..., (x: 1.0 m, y: 1.0 m))
false true true Quadrangle((x: 2.0 m, y: 0.0 m), ..., (x: 2.0 m, y: 1.0 m))
false false true Quadrangle((x: 3.0 m, y: 0.0 m), ..., (x: 3.0 m, y: 1.0 m))
false false true Quadrangle((x: 4.0 m, y: 0.0 m), ..., (x: 4.0 m, y: 1.0 m))
false false true Quadrangle((x: 5.0 m, y: 0.0 m), ..., (x: 5.0 m, y: 1.0 m))
false false true Quadrangle((x: 6.0 m, y: 0.0 m), ..., (x: 6.0 m, y: 1.0 m))
false false true Quadrangle((x: 7.0 m, y: 0.0 m), ..., (x: 7.0 m, y: 1.0 m))
false false true Quadrangle((x: 8.0 m, y: 0.0 m), ..., (x: 8.0 m, y: 1.0 m))
false false true Quadrangle((x: 9.0 m, y: 0.0 m), ..., (x: 9.0 m, y: 1.0 m))
โ‹ฎ โ‹ฎ โ‹ฎ โ‹ฎ

The second pipeline runs principal component analysis with variables โ€œbโ€, โ€œcโ€ and โ€œdโ€ and produces 2 principal components:

gt |> pipeline2
10000ร—3 GeoTable over 100ร—100 CartesianGrid
PC1 PC2 geometry
Continuous Continuous Quadrangle
[NoUnits] [NoUnits] ๐Ÿ–ˆ Cartesian{NoDatum}
1.68708 0.366616 Quadrangle((x: 0.0 m, y: 0.0 m), ..., (x: 0.0 m, y: 1.0 m))
-0.303829 0.562045 Quadrangle((x: 1.0 m, y: 0.0 m), ..., (x: 1.0 m, y: 1.0 m))
0.35228 1.57605 Quadrangle((x: 2.0 m, y: 0.0 m), ..., (x: 2.0 m, y: 1.0 m))
-0.0740307 -0.0357563 Quadrangle((x: 3.0 m, y: 0.0 m), ..., (x: 3.0 m, y: 1.0 m))
0.683434 -0.467476 Quadrangle((x: 4.0 m, y: 0.0 m), ..., (x: 4.0 m, y: 1.0 m))
0.0925121 -1.28355 Quadrangle((x: 5.0 m, y: 0.0 m), ..., (x: 5.0 m, y: 1.0 m))
-1.92792 0.751889 Quadrangle((x: 6.0 m, y: 0.0 m), ..., (x: 6.0 m, y: 1.0 m))
0.28284 -1.46881 Quadrangle((x: 7.0 m, y: 0.0 m), ..., (x: 7.0 m, y: 1.0 m))
-0.451337 0.339025 Quadrangle((x: 8.0 m, y: 0.0 m), ..., (x: 8.0 m, y: 1.0 m))
0.348789 1.05512 Quadrangle((x: 9.0 m, y: 0.0 m), ..., (x: 9.0 m, y: 1.0 m))
โ‹ฎ โ‹ฎ โ‹ฎ

We can combine the two pipelines into a single pipeline that executes in parallel:

pipeline = pipeline1 โŠ” pipeline2
ParallelTableTransform
โ”œโ”€ SequentialTransform
โ”‚  โ”œโ”€ Select(selector: [:a], newnames: nothing)
โ”‚  โ””โ”€ Indicator(selector: :a, k: 3, scale: :quantile, categ: false)
โ””โ”€ SequentialTransform
   โ”œโ”€ Select(selector: [:b, :c, :d], newnames: nothing)
   โ”œโ”€ ZScore(selector: all)
   โ””โ”€ EigenAnalysis(proj: :V, maxdim: 2, pratio: 1.0)
gt |> pipeline
10000ร—6 GeoTable over 100ร—100 CartesianGrid
a_1 a_2 a_3 PC1 PC2 geometry
Categorical Categorical Categorical Continuous Continuous Quadrangle
[NoUnits] [NoUnits] [NoUnits] [NoUnits] [NoUnits] ๐Ÿ–ˆ Cartesian{NoDatum}
false false true 1.68708 0.366616 Quadrangle((x: 0.0 m, y: 0.0 m), ..., (x: 0.0 m, y: 1.0 m))
false false true -0.303829 0.562045 Quadrangle((x: 1.0 m, y: 0.0 m), ..., (x: 1.0 m, y: 1.0 m))
false true true 0.35228 1.57605 Quadrangle((x: 2.0 m, y: 0.0 m), ..., (x: 2.0 m, y: 1.0 m))
false false true -0.0740307 -0.0357563 Quadrangle((x: 3.0 m, y: 0.0 m), ..., (x: 3.0 m, y: 1.0 m))
false false true 0.683434 -0.467476 Quadrangle((x: 4.0 m, y: 0.0 m), ..., (x: 4.0 m, y: 1.0 m))
false false true 0.0925121 -1.28355 Quadrangle((x: 5.0 m, y: 0.0 m), ..., (x: 5.0 m, y: 1.0 m))
false false true -1.92792 0.751889 Quadrangle((x: 6.0 m, y: 0.0 m), ..., (x: 6.0 m, y: 1.0 m))
false false true 0.28284 -1.46881 Quadrangle((x: 7.0 m, y: 0.0 m), ..., (x: 7.0 m, y: 1.0 m))
false false true -0.451337 0.339025 Quadrangle((x: 8.0 m, y: 0.0 m), ..., (x: 8.0 m, y: 1.0 m))
false false true 0.348789 1.05512 Quadrangle((x: 9.0 m, y: 0.0 m), ..., (x: 9.0 m, y: 1.0 m))
โ‹ฎ โ‹ฎ โ‹ฎ โ‹ฎ โ‹ฎ โ‹ฎ

All 5 columns are present in the final geotable.

7.4 Revertibility

An important concept related to pipelines that is very useful in geospatial data science is revertibility. The concept is useful whenever we need to answer geoscientific questions in terms of variables that have been transformed for geostatistical analysis.

Letโ€™s illustrate the concept with the following geotable and pipeline:

a = [-1.0, 4.0, 1.6, 3.4]
b = [1.6, 3.4, -1.0, 4.0]
c = [3.4, 2.0, 3.6, -1.0]
table = (; a, b, c)

geotable = georef(table, [(0, 0), (1, 0), (1, 1), (0, 1)])
4ร—4 GeoTable over 4 PointSet
a b c geometry
Continuous Continuous Continuous Point
[NoUnits] [NoUnits] [NoUnits] ๐Ÿ–ˆ Cartesian{NoDatum}
-1.0 1.6 3.4 (x: 0.0 m, y: 0.0 m)
4.0 3.4 2.0 (x: 1.0 m, y: 0.0 m)
1.6 -1.0 3.6 (x: 1.0 m, y: 1.0 m)
3.4 4.0 -1.0 (x: 0.0 m, y: 1.0 m) 
pipeline = Center()
Center transform
โ””โ”€ selector: all

We saw that our pipelines can be evaluated with Juliaโ€™s pipe operator:

geotable |> pipeline
4ร—4 GeoTable over 4 PointSet
a b c geometry
Continuous Continuous Continuous Point
[NoUnits] [NoUnits] [NoUnits] ๐Ÿ–ˆ Cartesian{NoDatum}
-3.0 -0.4 1.4 (x: 0.0 m, y: 0.0 m)
2.0 1.4 0.0 (x: 1.0 m, y: 0.0 m)
-0.4 -3.0 1.6 (x: 1.0 m, y: 1.0 m)
1.4 2.0 -3.0 (x: 0.0 m, y: 1.0 m)

In order to revert a pipeline, however; we need to save auxiliary constants that were used to transform the data (e.g., mean of selected columns). The apply function serves this purpose:

newtable, cache = apply(pipeline, geotable)

newtable
4ร—4 GeoTable over 4 PointSet
a b c geometry
Continuous Continuous Continuous Point
[NoUnits] [NoUnits] [NoUnits] ๐Ÿ–ˆ Cartesian{NoDatum}
-3.0 -0.4 1.4 (x: 0.0 m, y: 0.0 m)
2.0 1.4 0.0 (x: 1.0 m, y: 0.0 m)
-0.4 -3.0 1.6 (x: 1.0 m, y: 1.0 m)
1.4 2.0 -3.0 (x: 0.0 m, y: 1.0 m)

The function produces the new geotable as usual and an additional cache with all the information needed to revert the transforms in the pipeline. We say that a pipeline isrevertible, if there is an efficient way to revert its transforms starting from any geotable that has the same schema of the geotable produced by the apply function:

isrevertible(pipeline)
true
revert(pipeline, newtable, cache)
4ร—4 GeoTable over 4 PointSet
a b c geometry
Continuous Continuous Continuous Point
[NoUnits] [NoUnits] [NoUnits] ๐Ÿ–ˆ Cartesian{NoDatum}
-1.0 1.6 3.4 (x: 0.0 m, y: 0.0 m)
4.0 3.4 2.0 (x: 1.0 m, y: 0.0 m)
1.6 -1.0 3.6 (x: 1.0 m, y: 1.0 m)
3.4 4.0 -1.0 (x: 0.0 m, y: 1.0 m)

A very common workflow in geospatial data science consists of:

  1. Transforming the data to an appropriate sample space for geostatistical analysis
  2. Doing additional modeling to predict variables in new geospatial locations
  3. Reverting the modeling results with the saved pipeline and cache

We will see examples of this workflow in Part V of the book.

7.5 Congratulations!

Congratulations on finishing Part II of the book. Letโ€™s quickly review what we learned so far:

  • Transforms and pipelines are powerful tools to achieve reproducible geospatial data science.
  • The operators โ†’ and โŠ” can be used to build lazy pipelines. After a pipeline is built, it can be applied to different geotables, which may have different types of geospatial domain.
  • Lazy pipelines can always be optimized for computational performance, and the Julia language really thrives to dispatch the appropriate optimizations when they are available.
  • Map projections are specific types of coordinate transforms. They can be combined with many other transforms in the framework to produce advanced geostatistical visualizations.

There is a long journey until the technology reaches its full potential. The good news is that Julia code is easy to read and modify, and you can become an active contributor after just a few weeks working with the language. We invite you to contribute new transforms and optimizations as soon as you feel comfortable with the framework.