Importing data: Difference between revisions
(→) |
(→) |
||
(58 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
To use georeferenced data in M3, it needs to be optimized and imported into your workspace. There are several ways to | To use georeferenced data in M3, it needs to be optimized and imported into a product in your workspace. There are several ways to start an import: | ||
* Drag and drop [[Supported formats|any supported]] files or folders from Windows File Explorer onto the map area in M3. You can do this at any time regardless of where you are in your workflow. The only prerequisite is that you have an active workspace. | * Drag and drop [[Supported formats|any supported]] files or folders from Windows File Explorer onto the map area in M3. You can do this at any time regardless of where you are in your workflow. The only prerequisite is that you have an active workspace. | ||
Line 28: | Line 28: | ||
== Map Tasks == | == Map Tasks == | ||
When you start the Map Import process, a '''Map Task''' will be created. You can monitor it in the '''Map Task Queue''' window. You can have several map tasks active at the same time. | When you start the Map Import process, a '''Map Task''' will be created. You can monitor it in the '''Map Task Queue''' window. You can have several map tasks active at the same time. | ||
''From version 4.2'': Double-click a finished map task to open the product which it was imported to. | |||
[[File:M3 mapimport maptaskqueue.png|none]] | |||
== Placeholder layer == | |||
[[File:M3_mapimport_temp_layer.PNG|left|700px|thumb|The dataset test-N1000 is still processing and will be visible when it is finished]] | |||
When you choose to add your vector or raster dataset as a layer in an overlay/basemap using the '''Add to''' option, a '''placeholder layer''' will be created there. The animated icon [[File:M3_mapimport_raster_temp_layer_icon.PNG]] indicates that the dataset referenced by the layer is still processing. When the map task is finished, it will turn into a normal layer, and the dataset will be visible in the map. This happens automatically, regardless of what you are doing in the application. | |||
<br clear=all> | <br clear=all> | ||
== | == Raster data (version 4.3 and older) == | ||
Importing raster data is usually quite simple. However, you have to make sure that if you're importing multiple files into a single dataset, they have to be homogenous (same file format, pixel size, projection, band types) | |||
Depending on the source file format you can either choose to save as GeoPackage, or use the source format without translation. With some formats (ECW and MrSID), it's recommended to use the source format. These are formats which already have good performance, and translating them to GeoPackage leads to larger file size and loss of image quality. | |||
[[File:M3 rasterimport nodata checked.PNG|right|frame|Making the RGB value "255 255 252" transparent in the imported dataset]] | |||
=== Parameters === | |||
Different source formats will have different parameters. | |||
* '''Nodata''' - If the source data is an RGB raster, the nodata parameter should be a space-separated list of RGB values to make transparent - for example <code>0 0 0</code>. If the source data uses a color palette, then the parameter is single number - the color index to make transparent. | |||
* '''Projection/Proj4''' - These parameters only appear if projection information cannot be found in the source files. A few common projections are provided in the dropdown menu, or you can type in a projection in Proj4 format. | |||
* '''Resampling''' - Resampling method to use when creating downsampled overviews. | |||
* '''Jpeg quality''' - the level of compression to use when creating jpeg tiles. A larger number results in better quality and larger file size. 75-90 is usually a good balance between size and quality. | |||
* '''Tile format - ''' ''From version 4.2'' - Image format for tiles in the output GeoPackage. "Auto" uses JPEG on opaque tiles and PNG on tiles where transparency is needed. "PNG" uses 32 bit RGBA PNG format on all tiles. "PNG8" calculates a color table with maximum 256 different colors for each tile. The "PNG8" option can reduce file size significantly when the input data has few unique colors. | |||
=== Raster data with more than 8 bit === | |||
If a file has bands with more than 8 bit, it is usually elevation data. But in some cases, it can also be imagery from a satellite. Because Maria GDK and the GeoPackage format only supports 8 bit bands for maps and imagery, these datasets must be scaled down in the import process. In M3, this is handled as follows: | |||
{| class="wikitable" | |||
!Number of bands | |||
!Bits per band | |||
!Comment | |||
|- | |||
| 1 || > 8 || Use '''Save as GeoPackage (Scale to 8 bit)'''. Values are automatically scaled using default parameters as described in the [https://gdal.org/programs/gdal_translate.html#cmdoption-gdal-translate-scale gdal_translate documentation] | |||
|- | |||
| 3 || > 8 || Use '''Save as Raster GeoPackage'''. You can change the scale interval with the parameter '''Scale factor'''. This will affect the contrast of the resulting GeoPackge - a lower number results in a brighter image. | |||
|} | |||
== | == Raster data (version 4.4 and newer) == | ||
=== Parameters === | |||
* '''Projection/EPSG code''' - These parameters only appear if projection information cannot be found in the source files. A few common projections are provided in the dropdown menu, or you can type in a projection in EPSG format. | |||
= | * '''Nodata''' - If the source data is an RGB raster, the nodata parameter should be a space-separated list of RGB values to make transparent - for example <code>0 0 0</code>. If the source data uses a color palette, then the parameter is single number - the color index to make transparent. | ||
* '''Resampling''' - Resampling method to use when creating downsampled overviews. | |||
* '''Tile format '''- Image format for tiles in the output GeoPackage: | |||
** ''Auto'' uses JPEG on opaque tiles and PNG on tiles where transparency is needed. | |||
** ''PNG'' uses 32 bit RGBA PNG format on all tiles. | |||
** ''PNG8'' calculates a color table with maximum 256 different colors for each tile. This option can reduce file size significantly when the input data has few unique colors. | |||
* '''JPEG quality''' - the level of JPEG compression to use when Tile format = "Auto" is used. A larger number results in better quality and larger file size. 75-90 is usually a good balance between size and quality. | |||
* '''Scale factor''' - Factor to use when it is neccessary to scale down to 8 bit. This parameter becomes available if the input dataset has bands with more than 8 bit, and '''Save as''' is set to ''Raster GeoPackage'' The range of values to use from each source band is calculated as (mean ± stddev × scalefactor), and then scaled to 8 bit (values 0-255). | |||
* '''CPU use -''' the number of logical processors to use. When the input dataset consists of multiple files, Maria Map Maker will split the work across the logical processors of the CPU. | |||
* '''Clip dataset''' - a vector dataset to use for clipping the input raster dataset. | |||
* '''Clip layer''' - a named layer from the clip dataset to use for clipping. If there are multiple area features in the layer, only the first feature will be used. If the raster dataset falls completely outside the clip layer, an error will be thrown, and no output GeoPackage will be created. | |||
== Vector data == | |||
For importing and styling MARIA GDK vector data, see [[MARIA GDK vector datasets]] (version 4.2 and older). | |||
For importing and styling Mapbox vector data, see [[Mapbox vector datasets]] (version 4.3 and newer). | |||
== Elevation/Coverage data == | |||
For any regular gridded input data of 16 bit or more, the result will be a [http://docs.opengeospatial.org/is/17-066r1/17-066r1.html tiled gridded coverage GeoPackage]. In Maria Map Maker, these are classified as either an ''Elevation dataset'', or a ''Coverage dataset'' (chosen in the '''Save as''' drop-down) | |||
Importing | === Elevation GeoPackage === | ||
Datasets which represent the heights above a reference. Importing data as an elevation dataset requires that you classify the elevation type as one of the following: | |||
* Terrain Model - Models the ground terrain, where trees and man-made structures have been removed. | * Terrain Model - Models the ground terrain, where trees and man-made structures have been removed. | ||
* Surface Model - Models the surface of the earth, where trees and man-made structures are present. | * Surface Model - Models the surface of the earth, where trees and man-made structures are present. | ||
* Bathymetry - Models the sea bed | * Bathymetry - Models the sea bed | ||
=== Coverage GeoPackage === | |||
Datasets which can represent any geographical phenomenon. It's recommended that you create a '''Tag''' which describes the dataset for instance "landcover", "trafficdensity", "temperature", etc. This allows the dataset to be visualized in a [[Color Map Layers|Color map layer]]. | |||
=== | === Common parameters === | ||
Parameters which are shared for Elevation and Coverage dataset imports. | |||
[[ | * '''Tags''' - Allows the dataset to be used in [[Color map layers#Data source from tags|Color maps]] with matching tags | ||
* '''Tile format''' | |||
** ''PNG'' - Tiles are saved as PNG with data type Int16. If the input is Float32, a scale and offset will be calculated to achieve an effective precision of around 0.01 - 0.001 meters (less varied source data results in higher precision). This option is the most space efficient, and usually sufficient for elevation data with 10 meter resolution or lower. | |||
** ''TIFF'' - Tiles are saved as TIFF with data type Float32. This option will retain the precision of the source data at the cost of larger file size. | |||
*'''Resampling''' - Resampling method to use when creating overviews. | |||
The ''' | == Location data == | ||
M3 can create geolocation databases from text files. The text files must be formatted according to one of several well established schemas/sources: | |||
* GNS - [https://geonames.nga.mil/gns/html/ NGA GEOnet Names Server - Non-US placenames] | |||
* GNS_US - [https://www.usgs.gov/core-science-systems/ngp/board-on-geographic-names/download-gnis-data US placenames] | |||
* Geonames - [https://www.geonames.org/ Global placenames] | |||
[[File:M3_locationimport_geonames.PNG]] | |||
== Routing databases == | |||
''From version 4.1'' | |||
M3 can create routing databases based on Openstreetmap raw data (pbf files). These files can be exported from the map at [https://www.openstreetmap.org openstreetmap.org], or they can be downloaded from services such as [https://download.geofabrik.de/ Geofabrik]. The routing databases can be used by a Maria GDK-based application to calculate the fastest route between two points. | |||
=== | === Profiles === | ||
When importing the data, you can select which profiles to include in the output database. A profile describes a vehicle and its behaviour when travelling the road network. Note that the profiles cannot be changed after the database has been created. | |||
Hover the mouse over each profile for a short description. | |||
[[File: | [[File:M3_mapimport_routingDB.PNG]] | ||
<!-- PAGE BREAK --> | |||
== 3D Tiles == | |||
''From version 4.2'' | |||
[[File:M3 import 3DTiles.PNG|right]] | |||
Importing models is handled by the library assimp. A full list of supported formats can be found at https://github.com/assimp/assimp/blob/master/doc/Fileformats.md, however many of the formats have not yet been tested in Maria Map Maker. | |||
The imported 3D tileset will consist of the models themselves in the format b3dm, as well as texture files, and several text files which define the position and relationships between the models. | |||
[[Category:Maria Map Maker]] | === Parameters === | ||
* '''3D Tileset''' - Choose an existing 3D tileset as a destination for the model, or create a new tileset. | |||
* '''Model''' - Choose an existing model as a parent for the current import, or define it as a top-level model. | |||
* '''Compress textures''' - Compress textures into a DirectDraw Surface container file. This reduces file size, but should not be used if you intend to use the tileset in the Cesium platform, as it's not supported there. | |||
* '''Auto place''' - Attempt to find geographic position for the model in the source files. | |||
* '''Latitude / Longitude / Elevation''' - Manually enter position and elevation for the placement of the model. Only used if "Auto place" is disabled or unable to find geographic position in the source files. | |||
* '''Unit select''' - Unit in the source data - meters or feet. | |||
<br clear=all> | |||
[[Category:Maria Map Maker|400]] |
Latest revision as of 11:04, 14 September 2023
To use georeferenced data in M3, it needs to be optimized and imported into a product in your workspace. There are several ways to start an import:
- Drag and drop any supported files or folders from Windows File Explorer onto the map area in M3. You can do this at any time regardless of where you are in your workflow. The only prerequisite is that you have an active workspace.
- When you have a basemap or overlay open, click Add Layer → Import from file/folder
- For elevation data, in addition to the two methods above, you can also use Add content → Elevation in an open product.
All of these will result in a Map Import window on the right side of the map area. This is where you set your options and start the import process. These options will differ depending on of both the source and destination data formats. There are a few common options, however.
Common options
Name | Used to generate the map signature, which uniquely identifies this map when adding it to basemaps and overlays. It is also used as the product name, if you choose to import it into a new product. |
Workspace | Lists all your active workspaces. Choose which of them to import the map to. |
Product | Lists all the available (Draft) products in the chosen workspace. If you have a product, basemap or overlay open, this product will be the default choice. You can also choose to Create new product specifically for the map you are importing. |
Add to | Automatically add the map as a layer to a new or existing overlay/basemap in the selected product. You can also choose to not add it as layer. |
Save as | Lists the available destination formats for the source data you are importing. |
Advanced | These options vary according to the input data and the chose destination format, and can often be left to their default values. See the pages for raster and vector imports for a description of these options. |
Map Tasks
When you start the Map Import process, a Map Task will be created. You can monitor it in the Map Task Queue window. You can have several map tasks active at the same time.
From version 4.2: Double-click a finished map task to open the product which it was imported to.
Placeholder layer
When you choose to add your vector or raster dataset as a layer in an overlay/basemap using the Add to option, a placeholder layer will be created there. The animated icon indicates that the dataset referenced by the layer is still processing. When the map task is finished, it will turn into a normal layer, and the dataset will be visible in the map. This happens automatically, regardless of what you are doing in the application.
Raster data (version 4.3 and older)
Importing raster data is usually quite simple. However, you have to make sure that if you're importing multiple files into a single dataset, they have to be homogenous (same file format, pixel size, projection, band types)
Depending on the source file format you can either choose to save as GeoPackage, or use the source format without translation. With some formats (ECW and MrSID), it's recommended to use the source format. These are formats which already have good performance, and translating them to GeoPackage leads to larger file size and loss of image quality.
Parameters
Different source formats will have different parameters.
- Nodata - If the source data is an RGB raster, the nodata parameter should be a space-separated list of RGB values to make transparent - for example
0 0 0
. If the source data uses a color palette, then the parameter is single number - the color index to make transparent. - Projection/Proj4 - These parameters only appear if projection information cannot be found in the source files. A few common projections are provided in the dropdown menu, or you can type in a projection in Proj4 format.
- Resampling - Resampling method to use when creating downsampled overviews.
- Jpeg quality - the level of compression to use when creating jpeg tiles. A larger number results in better quality and larger file size. 75-90 is usually a good balance between size and quality.
- Tile format - From version 4.2 - Image format for tiles in the output GeoPackage. "Auto" uses JPEG on opaque tiles and PNG on tiles where transparency is needed. "PNG" uses 32 bit RGBA PNG format on all tiles. "PNG8" calculates a color table with maximum 256 different colors for each tile. The "PNG8" option can reduce file size significantly when the input data has few unique colors.
Raster data with more than 8 bit
If a file has bands with more than 8 bit, it is usually elevation data. But in some cases, it can also be imagery from a satellite. Because Maria GDK and the GeoPackage format only supports 8 bit bands for maps and imagery, these datasets must be scaled down in the import process. In M3, this is handled as follows:
Number of bands | Bits per band | Comment |
---|---|---|
1 | > 8 | Use Save as GeoPackage (Scale to 8 bit). Values are automatically scaled using default parameters as described in the gdal_translate documentation |
3 | > 8 | Use Save as Raster GeoPackage. You can change the scale interval with the parameter Scale factor. This will affect the contrast of the resulting GeoPackge - a lower number results in a brighter image. |
Raster data (version 4.4 and newer)
Parameters
- Projection/EPSG code - These parameters only appear if projection information cannot be found in the source files. A few common projections are provided in the dropdown menu, or you can type in a projection in EPSG format.
- Nodata - If the source data is an RGB raster, the nodata parameter should be a space-separated list of RGB values to make transparent - for example
0 0 0
. If the source data uses a color palette, then the parameter is single number - the color index to make transparent. - Resampling - Resampling method to use when creating downsampled overviews.
- Tile format - Image format for tiles in the output GeoPackage:
- Auto uses JPEG on opaque tiles and PNG on tiles where transparency is needed.
- PNG uses 32 bit RGBA PNG format on all tiles.
- PNG8 calculates a color table with maximum 256 different colors for each tile. This option can reduce file size significantly when the input data has few unique colors.
- JPEG quality - the level of JPEG compression to use when Tile format = "Auto" is used. A larger number results in better quality and larger file size. 75-90 is usually a good balance between size and quality.
- Scale factor - Factor to use when it is neccessary to scale down to 8 bit. This parameter becomes available if the input dataset has bands with more than 8 bit, and Save as is set to Raster GeoPackage The range of values to use from each source band is calculated as (mean ± stddev × scalefactor), and then scaled to 8 bit (values 0-255).
- CPU use - the number of logical processors to use. When the input dataset consists of multiple files, Maria Map Maker will split the work across the logical processors of the CPU.
- Clip dataset - a vector dataset to use for clipping the input raster dataset.
- Clip layer - a named layer from the clip dataset to use for clipping. If there are multiple area features in the layer, only the first feature will be used. If the raster dataset falls completely outside the clip layer, an error will be thrown, and no output GeoPackage will be created.
Vector data
For importing and styling MARIA GDK vector data, see MARIA GDK vector datasets (version 4.2 and older).
For importing and styling Mapbox vector data, see Mapbox vector datasets (version 4.3 and newer).
Elevation/Coverage data
For any regular gridded input data of 16 bit or more, the result will be a tiled gridded coverage GeoPackage. In Maria Map Maker, these are classified as either an Elevation dataset, or a Coverage dataset (chosen in the Save as drop-down)
Elevation GeoPackage
Datasets which represent the heights above a reference. Importing data as an elevation dataset requires that you classify the elevation type as one of the following:
- Terrain Model - Models the ground terrain, where trees and man-made structures have been removed.
- Surface Model - Models the surface of the earth, where trees and man-made structures are present.
- Bathymetry - Models the sea bed
Coverage GeoPackage
Datasets which can represent any geographical phenomenon. It's recommended that you create a Tag which describes the dataset for instance "landcover", "trafficdensity", "temperature", etc. This allows the dataset to be visualized in a Color map layer.
Common parameters
Parameters which are shared for Elevation and Coverage dataset imports.
- Tags - Allows the dataset to be used in Color maps with matching tags
- Tile format
- PNG - Tiles are saved as PNG with data type Int16. If the input is Float32, a scale and offset will be calculated to achieve an effective precision of around 0.01 - 0.001 meters (less varied source data results in higher precision). This option is the most space efficient, and usually sufficient for elevation data with 10 meter resolution or lower.
- TIFF - Tiles are saved as TIFF with data type Float32. This option will retain the precision of the source data at the cost of larger file size.
- Resampling - Resampling method to use when creating overviews.
Location data
M3 can create geolocation databases from text files. The text files must be formatted according to one of several well established schemas/sources:
- GNS - NGA GEOnet Names Server - Non-US placenames
- GNS_US - US placenames
- Geonames - Global placenames
Routing databases
From version 4.1
M3 can create routing databases based on Openstreetmap raw data (pbf files). These files can be exported from the map at openstreetmap.org, or they can be downloaded from services such as Geofabrik. The routing databases can be used by a Maria GDK-based application to calculate the fastest route between two points.
Profiles
When importing the data, you can select which profiles to include in the output database. A profile describes a vehicle and its behaviour when travelling the road network. Note that the profiles cannot be changed after the database has been created. Hover the mouse over each profile for a short description.
3D Tiles
From version 4.2
Importing models is handled by the library assimp. A full list of supported formats can be found at https://github.com/assimp/assimp/blob/master/doc/Fileformats.md, however many of the formats have not yet been tested in Maria Map Maker. The imported 3D tileset will consist of the models themselves in the format b3dm, as well as texture files, and several text files which define the position and relationships between the models.
Parameters
- 3D Tileset - Choose an existing 3D tileset as a destination for the model, or create a new tileset.
- Model - Choose an existing model as a parent for the current import, or define it as a top-level model.
- Compress textures - Compress textures into a DirectDraw Surface container file. This reduces file size, but should not be used if you intend to use the tileset in the Cesium platform, as it's not supported there.
- Auto place - Attempt to find geographic position for the model in the source files.
- Latitude / Longitude / Elevation - Manually enter position and elevation for the placement of the model. Only used if "Auto place" is disabled or unable to find geographic position in the source files.
- Unit select - Unit in the source data - meters or feet.