Geospatial Data¶

Ingest¶

When uploaded to Dataverse, the .zip is unpacked (same as all .zip files). Shapefile sets are recognized by the same base name and specific extensions. These individual files constitute a shapefile set. The first four are the minimum required (.shp, .shx, .dbf, .prj)

For example:

• bicycles.shp (required extension)
• bicycles.shx (required extension)
• bicycles.prj (required extension)
• bicycles.dbf (required extension)
• bicycles.sbx (NOT required extension)
• bicycles.sbn (NOT required extension)

Upon recognition of the four required files, Dataverse will group them as well as any other relevant files into a shapefile set. Files with these extensions will be included in the shapefile set:

• Required: .shp, .shx, .dbf, .prj
• Optional: .sbn, .sbx, .fbn, .fbx, .ain, .aih, .ixs, .mxs, .atx, .cpg, shp.xml

Then Dataverse creates a new .zip with mimetype as a shapefile. The shapefile set will persist as this new .zip.

Example¶

1a. Original .zip contents:

A file named bikes_and_subways.zip is uploaded to the Dataverse. This .zip contains the following files.

• bicycles.shp (shapefile set #1)
• bicycles.shx (shapefile set #1)
• bicycles.prj (shapefile set #1)
• bicycles.dbf (shapefile set #1)
• bicycles.sbx (shapefile set #1)
• bicycles.sbn (shapefile set #1)
• bicycles.txt
• the_bikes.md
• readme.txt
• subway_line.shp (shapefile set #2)
• subway_line.shx (shapefile set #2)
• subway_line.prj (shapefile set #2)
• subway_line.dbf (shapefile set #2)

1b. Dataverse unzips and re-zips files:

Upon ingest, Dataverse unpacks the file bikes_and_subways.zip. Upon recognizing the shapefile sets, it groups those files together into new .zip files:

• files making up the “bicycles” shapefile become a new .zip
• files making up the “subway_line” shapefile become a new .zip
• remaining files will stay as they are

To ensure that a shapefile set remains intact, individual files such as bicycles.sbn are kept in the set – even though they are not used for mapping.

1c. Dataverse final file listing:

• bicycles.zip (contains shapefile set #1: bicycles.shp, bicycles.shx, bicycles.prj, bicycles.dbf, bicycles.sbx, bicycles.sbn)
• bicycles.txt (separate, not part of a shapefile set)
• the_bikes.md (separate, not part of a shapefile set)
• readme.txt (separate, not part of a shapefile set)
• subway_line.zip (contains shapefile set #2: subway_line.shp, subway_line.shx, subway_line.prj, subway_line.dbf)

For two “final” shapefile sets, bicycles.zip and subway_line.zip, a new mimetype is used:

• Mimetype: application/zipped-shapefile
• Mimetype Label: “Shapefile as ZIP Archive”

How Geoconnect Uses Join Target Information¶

When a user attempts to map a tabular file, the application looks in the Geoconnect database for JoinTargetInformation. If this information is more than 10 minutes* old, the application will retrieve fresh information and save it to the db.

(* Change the timing via the Django settings variable JOIN_TARGET_UPDATE_TIME.)

This JoinTarget info is used to populate HTML forms used to match a tabular file column to a JoinTarget column. Once a JoinTarget is chosen, the JoinTarget ID is an essential piece of information used to make an API call to the WorldMap and attempt to map the file.

Retrieving Join Target Information from WorldMap API¶

The get_join_targets() function in dataverse_layer_services.py uses the WorldMap API, retrieves a list of available tabular file JointTargets. (See the dataverse_layer_services code in GitHub.)

Saving Join Target Information to Geoconnect Database¶

The get_latest_jointarget_information() in utils.py retrieves recent JoinTarget Information from the database. (See the utils code in GitHub.)

Setting Up WorldMap Test Data¶

For the dataset page, this script gives a query to add test WorldMap map data. After the query is run, the “Explore Map” button should appear for a tabular file or shapefile. In the example SQL queries below, substitute $DATASET_ID and $DATAFILE_ID with the appropriate ID’s.

To add sample map data for a tabular file:

INSERT INTO maplayermetadata (id, isjoinlayer, joindescription, embedmaplink, layerlink, layername, mapimagelink, worldmapusername, dataset_id, datafile_id)
VALUES (DEFAULT, true, 'This file was joined with WorldMap layer x, y, z',
'https://worldmap.harvard.edu/maps/embed/?layer=geonode:zip_codes_2015_zip_s9i','https://worldmap.harvard.edu/data/geonode:zip_codes_2015_zip_s9i',
'geonode:zip_codes_2015_zip_s9i',
'http://worldmap.harvard.edu/download/wms/27289/png?layers=geonode%3Azip_codes_2015_zip_s9i&width=865&bbox=-71.1911091251%2C42.2270382738%2C-70.9228275369%2C42.3976144794&service=WMS&format=image%2Fpng&srs=EPSG%3A4326&request=GetMap&height=550',
'admin',$DATASET_ID,$DATAFILE_ID});

To add sample map data for a tabular shapefile:

INSERT INTO maplayermetadata (id, isjoinlayer, embedmaplink, layerlink, layername, mapimagelink, worldmapusername, dataset_id, datafile_id)
VALUES (DEFAULT, false,
'https://worldmap.harvard.edu/maps/embed/?layer=geonode:zip_codes_2015_zip_s9i','https://worldmap.harvard.edu/data/geonode:zip_codes_2015_zip_s9i',
'geonode:zip_codes_2015_zip_s9i',
'http://worldmap.harvard.edu/download/wms/27289/png?layers=geonode%3Azip_codes_2015_zip_s9i&width=865&bbox=-71.1911091251%2C42.2270382738%2C-70.9228275369%2C42.3976144794&service=WMS&format=image%2Fpng&srs=EPSG%3A4326&request=GetMap&height=550',
'admin',$DATASET_ID,$DATAFILE_ID);

Previous: Universal Numerical Fingerprint (UNF) | Next: Shibboleth and OAuth