Oleksandr Matsibora. Large geoJSON files compressed with gzip, brotli, geobuf and rendered with MapLibre GL JS deployed to the Storj decentralized cloud

Author
Oleksandr Matsibora
Published

DOI

In a case of working with a large vector datasets (as the geoJSON files) we always have to solve two major problems:

  1. rendering of a huge amount of vector features at once

  2. transferring of a large files to the client-side

Both of these problems could cause a poor user experience and “freezing” of the web browser (desktop or mobile). To find the ways of solving these questions and to compare the different approaches to the large vector datasets management we used the Local Administrative Units (LAU) spatial layer as an example. It is officially provided by Eurostat and available to public access. For building of the choropleth map population density index was used. LAU layer is represented with a 77MB single geoJSON file which consists of more than 98000 geometry features.

Population density of EU based on data of the Local Administrative Units on 2021

1. Rendering the layer. Such large files could not only cause the slow performance in the web browser but the huge amount of features can also “break” the basic functionality as a popup element in Leaflet.js (InternalError: too much recursion). That is why we prefer to use a mapping library with native support of GPU-accelerated vector tile rendering, in our case it was open source MapLibre GL JS.

2. Transferring of 77MB file. It requires fast and reliable network to deliver a large geoJSON file from the storage to the client-side and even in that case a user should wait for fetching of the vector data. One of the best approaches in order to avoid geometry simplification and removing attributes is to use compression for the spatial vector data. So, we have compared implementation of the original large geoJSON file and a few popular compression formats in the same web-GIS application:

  • basic (lau.geojson, 77.0 MB)

  • geobuf (lau.pbf, 15.6 MB)

  • gzip (lau.gzip, 18.1 MB)

  • brotli (lau.br, 11.2 MB)

Large geoJSON files compressed with gzip, brotli, geobuf

Original geoJSON and geobuf files were stored to the regular shared web hosting. Geobuf file was decompressed on the client side with two JS packages (“geobuf” and “pbf”). But gzip and brotli were stored to the Storj cloud. Because this storage allows to set explicitly the custom response header “Content-Encoding” to values “gzip” and “br” respectively and to get all advantages of the native web browser decompression.

After the series of tests on different web-browsers (Firefox, Chromium) and quite slow network the best performance and user’s experience were noticed by Brotli compression. The main benefit of this algorithm is the smallest data transfer size and high speed of decompression without additional libraries and packages.

Online version of the web-GIS:
https://webgis.geohub.org.ua/eu-lau