Importing large raster datasets: Difference between revisions
No edit summary |
(→) |
||
| (4 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
Importing raster datasets in the order of 10 GB or less can usually be done with the default Map Import parameters. The general documentation on importing maps can be found [[Importing data|here]]. | Importing raster datasets in the order of 10 GB or less can usually be done with the default Map Import parameters. The general documentation on importing maps can be found [[Importing data|here]]. | ||
However, if the dataset you are importing is significantly larger - 20 GB or more - the | However, if the dataset you are importing is significantly larger - 20 GB or more - the parameters become more important to get a good result, while balancing processing time and file size. This article will go more in-depth on the different parameters and how they affect the import process and result. | ||
== Image | == Image quality == | ||
Image quality in this context refers to how similar the output result is to the input data. The following Map Import parameters will affect image quality and processing time. | Image quality in this context refers to how similar the output result is to the input data. The following Map Import parameters will affect image quality and processing time. | ||
=== Resampling === | === Resampling === | ||
This is the method to use when creating downsampled versions of the main dataset (also called ''pyramids'' or ''overviews''). The default resampling method is <code>average</code>. The following table shows a benchmark comparing the different methods when used on a typical 700 MB dataset. The column "Difference vs. baseline" shows | This is the method to use when creating downsampled versions of the main dataset (also called ''pyramids'' or ''overviews''). The default resampling method is <code>average</code>. The following table shows a benchmark comparing the different methods when used on a typical 700 MB dataset. The column "Difference vs. baseline" shows the difference in execution time from <code>average</code>. | ||
{| class="wikitable" | {| class="wikitable" | ||
|+Resampling methods benchmark | |+Resampling methods benchmark | ||
| Line 63: | Line 63: | ||
==== PNG ==== | ==== PNG ==== | ||
Uses 32 bit RGBA PNG format on all tiles. This is a lossless format, which means that the output will be identical to the input (as long as no resampling is neccessary). This comes at the cost of a very large file size. Recommended for: | Uses 32 bit RGBA PNG format on all tiles. This is a lossless format, which means that the output will be identical to the input (as long as no resampling is neccessary). This comes at the cost of a '''very''' large file size. Recommended for: | ||
* Datasets where lossless quality is needed, with no regard for file size. | * Datasets where lossless quality is needed, with no regard for file size. | ||
| Line 72: | Line 72: | ||
=== Quality === | === Quality === | ||
The '''Quality''' parameter controls how aggresively JPEG tiles are compressed. It is expressed as a value between 10 and 100 where | The '''Quality''' parameter controls how aggresively JPEG tiles are compressed. It is expressed as a value between <code>10</code> and <code>100</code> where | ||
* Higher values = less compression, larger files, better image quality. | * Higher values = less compression, larger files, better image quality. | ||
* Lower values = more compression, smaller files, more visible artifacts. | * Lower values = more compression, smaller files, more visible artifacts. | ||
The relationship between file size and visual quality is highly nonlinear. Using '''Quality'''=100 is not a lossless conversion, but it will result in a very large dataset. Decreasing the number will retain most of the visible quality while giving substantial size reductions. The default value of <code>75</code> gives a good balance between visual quality and file size. | The relationship between file size and visual quality is highly nonlinear. Using '''Quality'''=<code>100</code> is not a lossless conversion, but it will result in a very large dataset. Decreasing the number will retain most of the visible quality while giving substantial size reductions. The default value of <code>75</code> gives a good balance between visual quality and file size. | ||
Visual quality can be objectively measured as Structural Similarity (SSIM), which is a comparison of the compressed image to the original. We have done a benchmark of several compression algorithms, comparing SSIM to file size at various ''quality'' measures. Notice how the curve starts to flatten out at around 60-75 quality. Increasing '''Quality''' from <code>85</code> to <code>95</code> will almost double file size with minimal increase in visual quality. The full article can be found at [[Comparing compression in different formats]] | Visual quality can be objectively measured as Structural Similarity (SSIM), which is a comparison of the compressed image to the original. We have done a benchmark of several compression algorithms, comparing SSIM to file size at various ''quality'' measures. Notice how the curve starts to flatten out at around 60-75 quality. Increasing '''Quality''' from <code>85</code> to <code>95</code> will almost double file size with minimal increase in visual quality. The full article can be found at [[Comparing compression in different formats]] | ||
Latest revision as of 10:14, 11 September 2026
Importing raster datasets in the order of 10 GB or less can usually be done with the default Map Import parameters. The general documentation on importing maps can be found here.
However, if the dataset you are importing is significantly larger - 20 GB or more - the parameters become more important to get a good result, while balancing processing time and file size. This article will go more in-depth on the different parameters and how they affect the import process and result.
Image quality
Image quality in this context refers to how similar the output result is to the input data. The following Map Import parameters will affect image quality and processing time.
Resampling
This is the method to use when creating downsampled versions of the main dataset (also called pyramids or overviews). The default resampling method is average. The following table shows a benchmark comparing the different methods when used on a typical 700 MB dataset. The column "Difference vs. baseline" shows the difference in execution time from average.
| Method | Average time (seconds) | Difference vs. baseline | Sample tile |
|---|---|---|---|
average
|
42.45 | 0% | 
|
nearest
|
38.50 | -9% | 
|
bilinear
|
57.66 | 36% | 
|
cubic
|
61.80 | 46% | 
|
cubicspline
|
60.64 | 43% | 
|
lanczos
|
66.82 | 57% | 
|
gauss
|
54.28 | 28% | 
|
Full article here: Comparing resampling methods in massivegeopackage
Tile format
Tile format refers to the image format to use when storing tiles in the output GeoPackage. The best choice of tile format depends on the source data.
Auto
Uses JPEG on opaque tiles and PNG on tiles where transparency is needed. Recommended for:
- Imagery from satellites, aeroplanes or drones
- Topographical maps with elevation shading or other smooth gradient effects.
PNG
Uses 32 bit RGBA PNG format on all tiles. This is a lossless format, which means that the output will be identical to the input (as long as no resampling is neccessary). This comes at the cost of a very large file size. Recommended for:
- Datasets where lossless quality is needed, with no regard for file size.
PNG8
Calculates a color table with maximum 256 different colors for each tile. If the input data has discrete gradients and a small number of unique colors, this is the best option. Recommended for:
- Topograhical maps without elevation shading,
- Thematic datasets such as land classification.
Quality
The Quality parameter controls how aggresively JPEG tiles are compressed. It is expressed as a value between 10 and 100 where
- Higher values = less compression, larger files, better image quality.
- Lower values = more compression, smaller files, more visible artifacts.
The relationship between file size and visual quality is highly nonlinear. Using Quality=100 is not a lossless conversion, but it will result in a very large dataset. Decreasing the number will retain most of the visible quality while giving substantial size reductions. The default value of 75 gives a good balance between visual quality and file size.
Visual quality can be objectively measured as Structural Similarity (SSIM), which is a comparison of the compressed image to the original. We have done a benchmark of several compression algorithms, comparing SSIM to file size at various quality measures. Notice how the curve starts to flatten out at around 60-75 quality. Increasing Quality from 85 to 95 will almost double file size with minimal increase in visual quality. The full article can be found at Comparing compression in different formats
CPU use
If the input dataset consists of multiple files, Maria Map Maker will process each file in parallel, before subsequently merging them together into a single file. This allows the program to more efficiently utilize a multi-core CPU. The parameter CPU use controls the number of logical processors to use. The following options are available:
Light(25% of logical processors)Moderate(50% of logical processors)Aggressive(75% of logical processors)Extreme(100% of logical processors)
The default is Moderate, which is usually a good choice. More parallel processes do not automatically result in higher performance. Beyond a certain point, processes compete for shared resources including disk read/write throughput, memory bandwidth and CPU caches.