In the previous post I introduced two Geoscape datasets that have been made available on the Australian Government's data.gov.au website: National Roads and Administrative Boundaries. The datasets are distributed in two different formats, neither of which is optimal for my intended spatial processing model. A first task is then to transform the data to a common format, and one that has the right performance characteristics for the project.

My spatial processing model will cache required meta-data and geometry in RAM, trading significantly higher memory requirements in exchange for significantly faster data access. Loading data into memory requires a high-performance storage engine, and for that I have selected DuckDB.

DuckDB is highly performant in terms of storage and execution, and is further recommended for this application by a trait that might often be seen as a limitation: it's an embedded database. So, while it can't do the client-server dance, DuckDB will deliver data to my application without an intermediate network and the overheads that brings. Better still, DuckDB's spatial extension – and particularly GDAL integration – make it reasonably trivial to ingest both National Roads in GDB format and Administrative Boundaries in SHP format. For example:

create table map_feature_state_polygon as
select from ST_Read('ACT_STATE_POLYGON_shp.dbf');

However, the code I've written does get a just little more complicated. Firstly, the datasets are distributed in a hierarchical directory structure, sometimes with separate files (or actually sets of files) for each state or territory. So I'm fishing through the directory hierarchy for those files, and then joining their contents into single tables.

And secondly, I have elected to “normalise out” coded values and recurring text values (e.g. road names), replacing them with integer foreign keys. My rationale is thus:

• this is how I'll store the data in memory once loaded (to save space), and so it avoids doing any such conversion during the load
• notwithstanding DuckDBs storage smarts, I'm still hoping for space efficiency on disk

I am initially focusing on the following datasets / layers, but may add more down the track:

• National Roads (4,340,757 rows)
• Administrative Boundaries
  • State Polygon (12,844 rows)
  • Local Government Area Polygon (2,210 rows)
  • Locality Polygon (15,782 rows)

The code for this article is in the TagJobSpatial repository here.

On my MacBook Pro M1 Max processor the load takes approximately 1 minute, and the resultant DuckDB database is about 2.5 gigabytes.

Tags: #TagJob #Geospatial #DuckDB

The project uses as reference data Geoscape Datasets for Australian roads and administrative boundaries. Over the years, I have worked with these datasets under commercial licence, but happily they are now available for public use under The Australian Government's Data and Digital Government Strategy. Let's take a look at them.

Geoscape National Roads

Note: National Roads © Geoscape Australia licensed by the Commonwealth of Australia under Creative Commons Attribution 4.0 International license (CC BY 4.0).

The Geoscape National Roads dataset is a digital representation of the road network of Australia, inclusive of bus ways, walking trails, cycleways, and even ferry routes. The dataset has rich metadata, inclusive of road hierarchy (e.g. national highway, arterial, local road) and travel direction. One facet I like is that each row – and its associated geometry – is defined intersection-to-intserection (e.g. an actual road might be broken into multiple rows, with each row being the segment between two intersections).

The open edition of this dataset varies subtly from the commercially licensed edition. Firstly, the metadata is almost (but not quite) identical:

• road_id – primary key
• contributor_id
• jurisdiction_control – for example Aboriginal Land Council, Botanical Gardens, Commonwealth, Council, Department of Environment and Conservation, Department of Transport, Dept of Defence, etc
• operator – for example Port Authority, SAG: Forestry SA, SAG: Water, Southern Rural Water, etc
• date_created, date_modified
• national_route, state_route
• full_street_name (upper case)
• street_name (upper case) / street_name_label (title case)
• street_type (upper case) / street_type_label (title case)
• street_suffix (upper case) / street_suffix_label (title case)
• street_alias_name (upper case) / street_alias_type (upper case) / street_alias_suffix (upper case)
• feature_type – one of DUAL CARRIAGEWAY, FERRY ROUTE, MOTORWAY, PATHWAY, RAMP, ROUNDABOUT, SINGLE CARRIAGEWAY, VEHICULAR_TRACK, or null
• hierarchy – one of ACCESS ROAD, ARTERIAL ROAD, BUSWAY, COLLECTOR ROAD, CYCLEPATH, FERRY, FOOTPATH, LOCAL ROAD, NATIONAL OR STATE HIGHWAY, SUB-ARTERIAL ROAD, VEHICLE TRACK, or null
• subtype – BRIDGE, FERRY, FIRE TRAIL, PATHWAY, RAMP, ROAD, ROUNDABOUT, TUNNEL, or null
• ground_relationship – ABOVE GROUND, BELOW GROUND, ON GROUND, or null
• lane_count – range 1-6 or null
• lane_description – ONE, TWO OR MORE, or null (note all three values apply where lane_count is null)
• one_way – ONE WAY, TWO WAY, null
• status – CLOSED, OPERATIONAL, PROPOSED, UNDER CONSTRUCTION, or null
• surface – SEALED, UNSEALED, or null
• trafficability – 2WD, 4WD, or null
• travel_direction – BOTH, FROM TO, TO FROM, null
• speed – 10 – 110, or null
• state, source – ACT, NSW, NT, QLD, SA, TAS, VIC, WA (note: no OT)
• horizontal_accuracy – range 1-250
• shape_length, shape

Secondly, there are more limited data formats and datums. My intention is to work with GDA2020 in GDB format.

Geoscape Administrative Boundaries

Note: Administrative Boundaries © Geoscape Australia licensed by the Commonwealth of Australia under Creative Commons Attribution 4.0 International license (CC BY 4.0).

The Geoscape Administrative Boundaries is a comprehensive set of national boundaries, including government, statistical and electoral boundaries.

The dataset comprises seven products:

• Localities
• Local Government Areas (LGAs)
• Wards
• Australian Bureau of Statistics (ABS) Boundaries
• Electoral Boundaries
• State Boundaries
• Town Points

The ABS Boundaries product is organised in three themes related to 2011, 2016 and 2021 boundary definitions. Of most interest to me is the 2021 ABS Mesh Blocks and Statistical Areas layer within the ABS Boundaries 2021 theme. This is described in detail on the ABS website product comprises:

• Australia
• States and Territories (S/T)
• Statistical Areas Level 4 (SA4s)
• Statistical Areas Level 3 (SA3s)
• Statistical Areas Level 2 (SA2s)
• Statistical Areas Level 1 (SA1s)
• Mesh Blocks (MBs)
• Greater Capital City Statistical Areas (GCCSAs) – not part of main structure

Like National Roads, this dataset is available only in limited datums and formats. I am going to work with GDA2020 downloaded in SHP format.

Tags: #TagJob #Geospatial