Patching and converting DNC
This page details the process of patching DNC data to the most current version and converting to Maria GDK Vector Tiles.
DNC (Digital Nautical Chart) is a vector-based product containing maritime significant features between latitudes 84°N and 81°S. The format of the vector data is Vector Product Format (VPF). DNC is split into 29 databases, which combined covers most of the world's oceans and coastlines. Each database is again split into a number of libraries, each covering a smaller area, at one of five scale ranges. The scale ranges are Browse, General, Coastal, Approach and Harbor. You can find more information on the DNC NGA webpage (often has problems with invalid certificates)
Prerequisites
The toolchain to patch and convert DNC databases to Vector Tiles requires a number of Powershell modules, scripts and executables:
| Name | Type | Source | Comment |
|---|---|---|---|
| Geofunctions | Powershell module | Teleplan Globe | |
| Import-Excel | Powershell module | Powershell gallery | Install with command Install-Module ImportExcel -scope CurrentUser
|
| DNC_reports_patch.ps1 | Powershell script | Teleplan Globe | Extracts version information and patches databases. |
| patch.exe | Executable | NGA | Applies patches to databases. Found in \WINDOWS\SOFTWARE\WIN in every patch.
|
| DNC_vpf2m6m.fmw | FME workspace | Teleplan Globe | This workspace is referenced in DNC_workspacerunner.fmw |
| DNC_workspacerunner.fmw | FME workspace | Teleplan Globe | This workspace references DNC_vpf2m6m.fmw and runs it in several parallel processes. |
| tovectortiles.exe | Executable | Teleplan Globe | Its location must be included in the Path Environment variable. |
| DNC_m6m_to_vtiles.ps1 | Powershell script | Teleplan Globe | Converts all databases from m6m to Vector Tiles. |
| DNC_merge_vtiles.ps1 | Powershell script | Teleplan Globe | Merges all indvidual vector tiles files into one file per scale range. |
Patches and versioning
Each database has something similar to a major version and a minor version: the Edition number and Notice to Mariners number. The edition number of a database is updated infrequently with database patches. The NtM number has the form (week number)/(year), for instance 37/18.
Patches for individual databases are released every four weeks. Regular patches will update the NtM number, while database patches will update both edition number and NtM number. Databases recieved from NGA are baselines. This means that they can be updated with a regular patch or a database patch. However, once you apply a regular patch on a baseline, it is no longer a baseline and cannot be patched further. Furthermore, all patches are intended for a specific edition of the database. If the patch is applied to the wrong edition, the database will be corrupted. For this reason, it is very important keep a set of original databases untouched, and only work on copies.
When a database patch is successfully applied to a baseline, the edition number will increase by one. The database will still be a baseline.
Steps
1. Extract version information
When a new set of databases and patches have been recieved, it is a good idea to extract some version information from both. We have several Powershell scripts/functions to do this. Use the script DNC_reports_patch.ps1 to create Excel files with versions of the monthly patches, as well as the databases themselves (you have to fill in the first three variables before you run it).
As an example, we can look at how baseline database 25 (DNCDX025_020_Gulf_of_Alaska) appears in the spreadsheets:
Databases_versions.xlsx
| Date | database | NtM |
|---|---|---|
| ... | ... | ... |
| 20140628 | DNC25 | 26/14 |
| ... | ... | ... |
DNC_Patches.xlsx
| Edition | Patchnumber | database | NtM |
|---|---|---|---|
| 21 | 181 | DNC25 | 01/16 |
| 21 | 182 | DNC25 | 05/16 |
| 21 | 183 | DNC25 | 09/16 |
| 21 | 184 | DNC25 | 13/16 |
| 22 | 185 | DNC25 | 17/16 |
| 22 | 186 | DNC25 | 21/16 |
| 22 | 187 | DNC25 | 25/16 |
| 22 | 188 | DNC25 | 29/16 |
| 22 | 189 | DNC25 | 33/16 |
| 23 | 190 | DNC25 | 37/16 |
| 23 | 191 | DNC25 | 41/16 |
| 23 | 192 | DNC25 | 45/16 |
| 23 | 193 | DNC25 | 49/16 |
| 23 | 194 | DNC25 | 53/16 |
| 23 | 195 | DNC25 | 04/17 |
| 24 | 196 | DNC25 | 08/17 |
| 24 | 197 | DNC25 | 12/17 |
| 24 | 198 | DNC25 | 16/17 |
| 24 | 199 | DNC25 | 20/17 |
| 24 | 200 | DNC25 | 24/17 |
| 24 | 201 | DNC25 | 28/17 |
| 24 | 202 | DNC25 | 32/17 |
| 24 | 203 | DNC25 | 36/17 |
| 25 | 204 | DNC25 | 40/17 |
| 25 | 205 | DNC25 | 44/17 |
| 25 | 206 | DNC25 | 48/17 |
| 25 | 208 | DNC25 | 04/18 |
| 25 | 209 | DNC25 | 08/18 |
| 25 | 210 | DNC25 | 12/18 |
| 25 | 211 | DNC25 | 16/18 |
| 25 | 213 | DNC25 | 24/18 |
| 25 | 214 | DNC25 | 28/18 |
2. Patching
Databases_versions.xlsx tells us that DNC25 is at NtM 26/14, which means week 14 of 2014. Cross-checking this NtM number with DNC_patches, sheet "DNC25" tells us that the latest patch applied was patch number 172 (not included in the above table). We want to get the database up to the latest edition and NtM. To do this, we have to sequentially apply every database patch until we get to the latest edition, and then apply the very latest regular patch.
To find which patch is a database patch, look for changes in the edition number. Also, a database patch will consist of a single RTP file, instead of one file for each library in the database. In the DNC25 example, the following patches should be applied in sequence:
| Edition | Patchnumber | database | NtM |
|---|---|---|---|
| 21 | 172 | DNC25 | 17/15 |
| 22 | 185 | DNC25 | 17/16 |
| 23 | 190 | DNC25 | 37/16 |
| 24 | 196 | DNC25 | 08/17 |
| 25 | 204 | DNC25 | 40/17 |
| 25 | 214 | DNC25 | 28/18 |
Patch number 214 is the latest patch we have. It is not a database patch, and thus it will cause the database to no longer be a baseline, and unable to be patched further. It will however contain the latest updates on maritime features.
Script code to apply a patch to a single database is found in DNC_reports_patch.ps1 and looks like this:
Import-Module Geofunctions
Install-DNCPatch "C:\WORK\DNC2018\tools\win-patch\patch.exe" `
"C:\WORK\DNC2018\full_regions\DNCDX025_020_Gulf_of_Alaska\DNC25" `
"C:\WORK\DNC2018\patcher_oppdateringer\DNCDXVDU_214_VPF_Database_Update\WINDOWS\PATCH\DNC25_25\WINDOWS"
As shown above, the Install-DNCPatch function takes parameters:
- path to patch.exe
- the database to be patched
- patch directory.
The above code should be run for each database until they are all patched to the latest NtM. At this point it's a good idea to create a new Databases_versions.xlsx to make sure that all databases were succsefully patched.
The databases are then ready to be converted to m6m.
3. Converting to M6M
To convert all patched databases to m6m, we use the FME workspace DNC_workspacerunner.fmw. In the workspace, the reader should be set to recursively read the folder containing the 29 databases looking for DHT files. Then, set the value for the public parameter Destination_directory
The Workspacerunner transformer needs to reference the DNC_vpf2m6m.fmw workspace, which will do the actual work in multiple parallel processes. When you have entered the path to the workspace, the other parameters will probably disappear. However, it should look like this:
The workspace is now ready to run.
4. Converting to Vector Tiles
When all the databases have been converted to m6m, it is time to convert them to Vector Tiles. This is done with the script DNC_m6m_to_vtiles.ps1. Enter the folder where the m6m files were output in the previous step in $srcfolder and an empty folder in $targetfolder. As a guideline, $threads should be set to 1-2 less than the number of logical processors on the computer. However, more than ~7 will probably max out I/O capacity on most storage devices.
This step will create one Vector Tile file per library per zoom level. The zoom levels are configured in the function ConvertTo-DNCVectorTiles based on trial and error, and it should not be neccessary to change them.
5. Merging Vector Tiles
The final step is to merge all the individual library Vector Tiles into one file for each scale group. This is done with the script DNC_merge_vectortiles.ps1. In the parameter $srcfolder, enter the target folder from the previous step. Enter an empty folder in the $targetfolder parameter. In this script there are only four processes, so the maximum number of $threads that makes sense is also 4.
You should now have the files * General.vtiles.sqlite * Costal.vtiles.sqlite * Approach.vtiles.sqlite * Harbor.vtiles.sqlite
They should replace the existing files in the DNC map product. Browse is such a high-scale dataset that it does not need to be updated.
Troubleshooting
- The DNC patching and conversion process is prone to errors. It's not uncommon for patches to corrupt databases so that they will not work in FME. In this case, you have to delete the database and get a fresh copy, and try again with the previous patch. In some cases, you just have to use the unpatched database.
- After each step, you should make sure that all 29 databases have been processed and output.
- After completing all steps in this guide, it can be useful to analyze the Vector Tiles files to identify problems.