Producing GeoPackages with massivegeopackage

Massivegeopackage is a Python package for building a GeoPackage from very large raster or elevation datasets.

The core GDAL tools gdalwarp, gdal_translate and gdaladdo seem to struggle with mosaicing very large raster datasets. When the size of the source dataset is larger than 15-20 GB, processing time seems to increase exponentially.

This Python package will convert each source file to individual GeoPackages within a common tile matrix. This is done in a user configurable number of parallel processes. Completed files will immediately be queued to be merged into a base GeoPackage. The merging is very efficient because it consists of SQL INSERT statements only - no geoprocessing.

Dependencies

• numpy
• Pillow
• GDAL >= 3.1.4

The easiest way to get a Python environment where these dependencies are met, is to install QGIS 3.16 or newer. Included is the OSGeo4W shell, where you can install and use the package.

Installation

The package is distributed as a Python wheel-file. Install it using pip:

python -m pip install c:\path\massivegeopackage-1.3.0-py3-none-any.whl

Usage

The package consists of two separate modules: raster and elevation. After installation, the modules can be run with the commands:

python -m massivegeopackage.raster

python -m massivegeopackage.elevation

If the package cannot be installed for whatever reason, it is also possible to unzip the .whl file, and then run the script files directly:

python c:\massivegeopackage-1.3.0-py3-none-any\massivegeopackage\raster.py

python c:\massivegeopackage-1.3.0-py3-none-any\massivegeopackage\elevation.py

Parameters are then provided through input prompts.

massivegeopackage.raster

Input files should be homogenous (same projection, dimensions, bands, pixel size).

Supported input:

Format	Band configuration
Any raster format supported by GDAL	1 band color index (8 bit)
	1 band grayscale (8 bit)
	3 band RGB (24 bit)
	4 band RGBA (32 bit)

Parameters:

Parameter	Description
srcfolder	Full path to a folder containing source raster files.
areapath	Full path to a folder containing shape files. Each shape file should contain a single polygon feature. One clipped GeoPackge will be created for each shape file.
targetfolder	Empty folder for temporary files, logs and the completed GeoPackage.
input_file_type	File extension on input files. Default is tif
nodata	Pixel value to make transparent in the completed GeoPackage. Example: 0 0 0
co	Creation options for the GDAL GeoPackage driver. Example: tile_format=jpeg,quality=50
num_processes	Number of parallel processes to use. Most common storage devices will become a bottleneck with 20 or more parallel processes.

massivegeopackage.elevation

Input files should be homogenous (same projection, dimensions, bands, pixel size). Supported input:

Format	Band configuration
Any raster format supported by GDAL	1 band (Int16)
	1 band (UInt16)
	1 band RGB (Float32)

Parameter	Description
srcfolder	Full path to a folder containing source raster files.
areapath	Full path to a folder containing shape files. Each shape file should contain a single polygon feature. One clipped GeoPackge will be created for each shape file.
targetfolder	Empty folder for temporary files, logs and the completed GeoPackage.
input_file_type	File extension on input files. Default is tif
targetdatatype	Datatype for output GeoPackage - 32 or 16 bits. If source is Float32 and target is set to 16 bit, the value range in each tile will be streched to utilize the entire space that an 16 bit unsigned integer provides (0-65535). A scale factor and offset will be computed, which will be applied when reading from the file output file. This way, an effective precision of around 0.01 - 0.001 meters is achieved (less varied source data results in higher precision).
num_processes	Number of parallel processes to use. Most common storage devices will become a bottleneck with 20 or more parallel processes.