Importing data

From Maria GDK Wiki
Revision as of 07:58, 6 April 2020 by Mbu (talk | contribs) ()
Jump to navigation Jump to search

To use georeferenced data in M3, it needs to be optimized and imported into your workspace. There are several ways to do this:

  • Drag and drop any supported files or folders from Windows File Explorer onto the map area in M3. You can do this at any time regardless of where you are in your workflow. The only prerequisite is that you have an active workspace.
  • When you have a basemap or overlay open, click Add Layer → Import from file/folder
  • For elevation data, in addition to the two methods above, you can also use Add content → Elevation in an open product.

All of these will result in a Map Import window on the right side of the map area. This is where you set your options and start the import process. These options will differ depending on of both the source and destination data formats. There are a few common options, however.

Common options

Name Used to generate the map signature, which uniquely identifies this map when adding it to basemaps and overlays. It is also used as the product name, if you choose to import it into a new product.
Workspace Lists all your active workspaces. Choose which of them to import the map to.
Product Lists all the available (Draft) products in the chosen workspace. If you have a product, basemap or overlay open, this product will be the default choice. You can also choose to Create new product specifically for the map you are importing.
Add to Automatically add the map as a layer to a new or existing overlay/basemap in the selected product. You can also choose to not add it as layer.
Save as Lists the available destination formats for the source data you are importing.
Advanced These options vary according to the input data and the chose destination format, and can often be left to their default values. See the pages for raster and vector imports for a description of these options.
M3 import full app.png

Map Tasks

When you start the Map Import process, a Map Task will be created. You can monitor it in the Map Task Queue window. You can have several map tasks active at the same time.

M3 mapimport maptaskqueue.png

Placeholder layer

The dataset N50 is still processing and will be visible when it is finished

When you choose to add your vector or raster dataset as a layer in an overlay/basemap using the Add to option, a placeholder layer will be created there. When the map task is finished, it will turn into a normal layer, and the data will be visible in the map. This happens automatically, regardless of what you are doing in the application.

Raster data

Importing raster data is usually quite simple. Depending on the source file format you can either choose to save as GeoPackage, GeoPackage with nodata values, or use the source format without translation. With some formats (ECW and MrSID), it's recommended to use the source format. These are formats which already have good performance, and translating them to GeoPackage leads to larger file size and loss of image quality.

M3 rasterimport nodata checked.PNG

Parameters

Different source formats will have different parameters.

  • Projection/Proj4 - These parameters only appear if projection information cannot be found in the source files. A few common projections are provided in the dropdown menu, or you can type in a projection in Proj4 format.
  • Resampling - Resampling method to use when creating downsampled overviews.
  • Jpeg quality - the level of compression to use when creating jpeg tiles. A larger number results in better quality and larger file size. 75-90 is usually a good balance between size and quality.
  • Nodata - If the source data is an RGB raster, the nodata parameter should be a space-separated list of RGB values to make transparent. If the source data uses a color palette, then the parameter is single number - the color index to make transparent.

Raster data with more than 8 bit

If a file has bands with more than 8 bit, it is usually elevation data. But in some cases, it can also be imagery from a satellite. Because the GDK and the GeoPackage format only supports 8 bit bands for raster data, datasets must be scaled down in the import process. M3 can handle the following source data:

  • 1 band of more than 8 bit: Use Save as GeoPackage (Scale to 8 bit)
  • 3 bands of more than 8 bit: Use Save as Raster GeoPackage. You can change the scale interval with the parameter Scale factor. This will affect the contrast of the resulting GeoPackge - a lower number results in a brighter image.



Vector data

The number of tiles quadruples for each increasing zoom level

When Maria Map Maker imports vector data, it is translated to Vector Tiles. This is a very high-performant format, but the translation process requires some tuning and know-how to get the result you want.

The Vector Tiles format splits the vector data into tiles, and puts each tile into a particular zoom level. Zoom level 0 is a single tile which covers the entire Earth (in Web Mercator projection). With every increasing zoom level, each tile is divided into four. As an example, zoom level 15 roughly equals map scale 1:15 000 at the equator. At this level the Earth is split into slightly more than a billion tiles.

When importing vector data to Vector Tiles, you need to decide which zoom levels (z) to create. This zoom extent is not possible to change later. The vector data simply does not exist on the levels outside the zoom extent you choose. This means that you have to be familiar with your source data and how they are going to be used. Using zoom levels which are not appropriate for the source data leads to two things:

  • Using too low zoom level - too much data is stored into each tile, which leads to poor performance.
  • Using too high zoom level - large, simple features are split across an exaggerated number of small tiles, which leads to unnecessary large files and long processing time.

An example could be some very detailed building data for a city. If you try to import these into z10-z12, all the data will probably be pushed into a single, huge tile at z10. This would cause very poor performance in the finished map.

Overzoom

The overzoom function alleviates the zoom extent choice somewhat. It lets you display vector features at higher zoom levels than what they were created for. If you for example create a dataset at z10-z14, the default use of overzoom in M3 (6 levels) lets the data be visible all the way down to z20.

The downside with using overzoom is that performance will decrease as overzoom increases. The precision of the vertices of the vector data also decreases.

Starting import

M3 import vectordata.png

Choose the zoom extent with the two sliders. The zoom indicator shows the current zoom level in the map. Below, all the zoom levels in the zoom extent are listed. They can each be expanded, and you can uncheck layers that you don't need at that particular zoom level. When you are satisfied with your settings, start the map import.

A map task will start, and you can monitor its progress in the Map Task Queue. When it is finished, click the Review button.

M3 import vectordata finished review.png

This will open the Map Import window again, but now an analysis has been performed on the vector tiles. You get the opportunity to re-run the translation process with different settings according to the analysis.

Mouse over each layer to read the result. Zoom levels with a ! have been determined to be sub-optimal. The messages from these will inform you of one of the following:

  • Max size too large and Fraction/number of large tiles too high - You have put data which is too detailed into a zoom level which is too low. Each tile should ideally not be more than 0.1 MB. Depending on the size of the dataset, however, having a few tiles a little over 0.1 MB is not necessarily a problem. When many of them exceed 0.2 MB however, you should consider re-running the process with a higher minimum zoom level.
  • Tiles contain little data - this is not as much of a problem as the opposite. The size of the data file will be larger than it could be, but performance will not be affected.

If you change any settings according to the analysis, you have to use Rerun Map Import, and the translation process will start over. If you use Deploy the map import will be finalized to your workspace as it is.

This dataset on buildings in NYC is too detailed for the zoom extent chosen. A better extent would be z15-z17

Elevation data

Importing elevation data with M3 results in an elevation GeoPackage. You can get the technical details of the format here. Before starting the import, you have to specify the type of elevation data being imported in the Data Type menu:

  • Terrain Model - Models the ground terrain, where trees and man-made structures have been removed.
  • Surface Model - Models the surface of the earth, where trees and man-made structures are present.
  • Bathymetry - Models the sea bed

There are two choices in the Save as dropdown:

Elevation GeoPackage (16 bit)

M3 import elevation 16bit black.png

The elevation data is stored in PNG tiles with 16 bit integer pixels. This option is the most space efficient, and usually sufficient for elevation data with 10 meter resolution or lower. The Precision parameter lets you choose how many decimal places to use in each sample. The default (1) means that no decimals are used.

The ResamplingMethod parameter controls the resampling method to use when creating overviews.


Elevation GeoPackage (32 bit)

M3 import elevation 32bit black.png

The elevation data is stored in TIFF tiles with 32 bit float pixels. This option will retain the precision of the source data at the cost of larger file size.

The ResamplingMethod parameter controls the resampling method to use when creating overviews.


Location data

M3 can create geolocation databases from text files. The text files must be formatted according to one of several well established schemas/sources:

M3 locationimport geonames.PNG