Processing FIL Norway imagery: Difference between revisions

From Maria GDK Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
The biannual delivery of the FIL Norway ECW imagery dataset can be efficiently processed and converted to GeoPackage using the Python package filnorway_processor. This package will split the input according to a custom grid with 32768 x 32768 meters cells. The output is stored as GeoPackages with WEBP compression, and organized by county.  
The biannual delivery of the FIL Norway ECW imagery dataset can be efficiently processed and converted to GeoPackage using the Python package '''''filnorway_processor'''''. This package will split the input according to a custom grid with 32768 x 32768 meters cells. The cells will be processed in a user configurable number of parallel processes. The output is stored as GeoPackages with WEBP compression, and organized by county. These GeoPackages are then used to update the NorwayImageryCompressed product series.  


== Dependencies ==
== Dependencies ==
Line 11: Line 11:
  python -m pip install c:\path\filnorway_processor-1.0.6-py3-none-any.whl
  python -m pip install c:\path\filnorway_processor-1.0.6-py3-none-any.whl


== Arguments ==  
== Arguments ==
Raster specific arguments:
{| class="wikitable"
{| class="wikitable"
! style="width: 200px" | Argument
! style="width: 200px" | Argument
!Description
!Description
|-
|-
|<code>-f, --format</code>
|<code>--input</code>
|Output format. Should be <code>gpkg</code> or <code>mbtiles</code>. Default is <code>gpkg</code>
|Input directory containing all FIL Norway ECW files
|-
|-
| <code>-nd, --nodata</code>
| <code>--fylker</code>
|Pixel value to make transparent in the completed GeoPackage. Example: <code>0 0 0</code>
|Vector dataset with one layer per Norwegian county. Does not have to contain all counties.
|-
|-
|<code>-tf, --tile_format</code>
|<code>--fotodato</code>
|Image format for saving tiles. Should be one of <code>AUTO, JPEG, PNG, PNG8, WEBP</code>
|"Fotodato" dataset with photography dates and coverage included in the delivery. Should be a single file/layer. This is used to clip datasets to valid data areas.
|-
|-
|<code>-q, --quality</code>
|<code>--outpath</code>
|Quality setting for compressed tiles (JPEG and WEBP). Should be in the range <code>1-100</code>
|Output directory.
|-
|-
| <code>-scl, --scalefactor</code>
| <code>-q, --quality</code>
|Factor to use when it is neccessary to scale down to 8 bit. The range of values to use from each source band is calculated as (mean ± stddev × scalefactor).
|WEBP quality on the output GeoPackages. Default value: 5
|-
|-
|<code>--defrag</code>
|<code>-p, --num_processes</code>
|Increase the page size of the base GeoPackage to avoid fragmentation issues when the file grows very large (several hundred gigabytes). When processing is done, the page size will be changed back to default and VACUUM is run to apply the change and to compact the database.
|Maximum number of parallel processes to use
|-
|<code>--debug</code>
|Enable more detailed log messages
|}
|}
== Examples ==
Display help:
python -m filnorway_processor --help
Process FIL Norway for the entire country using maximum 12 parallel processes:
python -m filnorway_processor --input \\files\filnorway\ecw\ --fotodato \\files\filnorway\fotodato\fotodato_2025_02.shp --fylker c:\data\norway_counties_shapefiles\ --output \\files\filnorway\gpkg -p 12
Process FIL Norway for Oslo county using maximum 12 parallel processes. However, Oslo county only covers 3 cells in the grid, so only 3 parallel processes will actually be used.
python -m filnorway_processor --input \\files\filnorway\ecw\ --fotodato \\files\filnorway\fotodato\fotodato_2025_02.shp --fylker c:\data\norway_counties_shapefiles\oslo.shp --output \\files\filnorway\gpkg -p 12

Latest revision as of 13:20, 7 May 2025

The biannual delivery of the FIL Norway ECW imagery dataset can be efficiently processed and converted to GeoPackage using the Python package filnorway_processor. This package will split the input according to a custom grid with 32768 x 32768 meters cells. The cells will be processed in a user configurable number of parallel processes. The output is stored as GeoPackages with WEBP compression, and organized by county. These GeoPackages are then used to update the NorwayImageryCompressed product series.

Dependencies

  • GDAL >= 3.8.0 (with ECW driver)

The easiest way to get a Python environment where these dependencies are met, is to install the latest version of QGIS . Included is the OSGeo4W shell, where you can install and use the package.

Installation

The package is distributed as a Python wheel-file. Install using pip:

python -m pip install c:\path\filnorway_processor-1.0.6-py3-none-any.whl

Arguments

Argument Description
--input Input directory containing all FIL Norway ECW files
--fylker Vector dataset with one layer per Norwegian county. Does not have to contain all counties.
--fotodato "Fotodato" dataset with photography dates and coverage included in the delivery. Should be a single file/layer. This is used to clip datasets to valid data areas.
--outpath Output directory.
-q, --quality WEBP quality on the output GeoPackages. Default value: 5
-p, --num_processes Maximum number of parallel processes to use
--debug Enable more detailed log messages

Examples

Display help:

python -m filnorway_processor --help

Process FIL Norway for the entire country using maximum 12 parallel processes:

python -m filnorway_processor --input \\files\filnorway\ecw\ --fotodato \\files\filnorway\fotodato\fotodato_2025_02.shp --fylker c:\data\norway_counties_shapefiles\ --output \\files\filnorway\gpkg -p 12

Process FIL Norway for Oslo county using maximum 12 parallel processes. However, Oslo county only covers 3 cells in the grid, so only 3 parallel processes will actually be used.

python -m filnorway_processor --input \\files\filnorway\ecw\ --fotodato \\files\filnorway\fotodato\fotodato_2025_02.shp --fylker c:\data\norway_counties_shapefiles\oslo.shp --output \\files\filnorway\gpkg -p 12