Fast, static D3 maps built with Turf.
js and the command-lineCombining Mike Bostock’s command-line cartography tutorial with the flexibility of Node.
jsLuke WhyteBlockedUnblockFollowFollowingJan 28Estimated percent of undocumented residents in U.
S.
metro areas.
Source: Pew Research CenterRecently, I needed to build a handful of U.
S.
state bubble maps to be embedded in a story for San Antonio Express-News.
I wanted to use D3 but was concerned about slow asset loading and map rendering, especially because I’d need to iframe each map in separately due to CMS limitations.
In hopes of finding a way to export D3-generated SVGs, I did some googling.
This guy built an export button, which is pretty cool.
The New York Times has SVG Crowbar, which provides similar functionality.
However, the most exciting result I found was this four part article, Command-Line Cartography, by Mike Bostock.
I highly recommend Bostock’s article, from which much of this tutorial is derived.
My only issue with his approach is that, for me, writing d3 code on the command line can get complicated and confusing.
Here’s a snippet:(topo2geo tracts=- < ca-topo.
json | ndjson-map -r d3 -r d3=d3-scale-chromatic 'z = d3.
scaleThreshold().
domain([1, 10, 50, 200, 500, 1000, 2000, 4000]).
range(d3.
schemeOrRd[9]), d.
features.
forEach(f => f.
properties.
fill = z(f.
properties.
density)), d' | ndjson-split 'd.
features'; opo2geo counties=- < ca-topo.
json | ndjson-map 'd.
properties = {"stroke": "#000", "stroke-opacity": 0.
3}, d') | geo2svg -n –stroke none -p 1 -w 960 -h 960 > ca.
svgI wanted to find a solution that utilized Node.
js, allowing me to still work from the command line with Bostock’s d3-geo-projection, but pass off the handling of much of the cartographical manipulation and map rendering to JavaScript.
That’s where Turf.
js comes in — an amazing, open source geospatial analysis tool for writing and manipulating GeoJSON.
This tutorial walks step by step through generating static maps with node and the command line.
We’ll build the map seen above in SVG format, which shows the 50 metro areas with the highest percent of undocumented residents among the 100 largest U.
S.
metro areas.
Step I: Load your dataThe files we’ll need for the map are:The undocumented resident data from a 2017 Pew Research Center study.
A shapefile defining the location of the U.
S.
metro areas.
A shapefile defining the outline of the U.
S.
states.
Fire up the command line and create a directory to work in:mkdir undocumented-residents-mapcd undocumented-residents-mapCreate a bash file that we’ll use as our main build script:touch buildmap.
shThe original Pew data is available here.
I simplified it into two columns and stored it on data.
world.
The shapefiles are provided by the Census Bureau.
We could download all these files via a browser, but it’s more fun to do it programmatically.
Open buildmap.
sh in your favorite text editor and write something to the effect of:The above conditionally downloads and extracts all three files into the root of our project folder.
Now that we have our assets, we can start building our map.
Step II: Convert Shapefiles to GeoJSONWe’re going to use an NPM package to achieve this, so now is a good time to set up package.
json.
Run npm init and follow the prompts to generate the JSON.
Next, you’ll want to add the following to the generate package.
json:"scripts": { "buildmap": "sh buildmap.
sh" }This will allow us to run buildmap.
sh from within the context of our current NPM package.
Install Mike Bostock’s streaming shapefile parser (npm i shapefile -D) and add the following lines to the bottom of buildmap.
sh that will convert our shapefiles to GeoJSON:These two GeoJSON files serve the following purposes:metroareas.
geojson: The geodata we’ll use to bind our undocumented resident CSV data to the map.
states.
geojson: The U.
S.
state outlines we’ll use to draw our map background.
Ultimately, both these files will be merged with the CSV file into one master GeoJSON file that we’ll convert to SVG.
But first…Step III: Add location data to our undocumented resident data pointsIn this step we’re going to use Node to convert the data in our CSV — percent of undocumented residents in U.
S.
metro areas — to scaled circles centered on their respective metro areas in a GeoJSON file.
If you are new to GeoJSON, I recommend starting with the Wikipedia page.
Our outputted GeoJSON will include a single feature — a point– for each metro area found in the CSV.
The coordinates for these features will be derived from metroarea.
geojson and each point’s “properties” object will include a pointRadius property used to scale the size of the point relative to the corresponding percentage of undocumented residents found in that metro area.
Here’s an overview of the steps we’ll take in our JS file:Convert metroareas.
geojson and metro-area-percent-undocumented.
csv to JS objectsFilter metroareas.
geojson to only include metro areas that also exist in our CSVUse Turf.
js to find the center of each metro area polygon and pin our point to the center’s coordinates.
Add styles (color, etc) to our point in each feature’s “properties” object, including our pointRadius property, which is scaled using d3.
Export the set of features as a new GeoJSON file.
To start, create the file: touch mapMetroPathsToCircles.
jsOpen mapMetroPathsToCircles.
js and write the following, which completes the 5 steps above with details for each step in the comments:Before we add this to our bash file, we’ll need to install the required dependencies:npm i @turf/turf d3 csv-parse -DThen, in buildmap.
sh we can add:node mapMetroPathsToCircles.
js # Create circles.
geojsonStep IV: Add the state outlines to our metro area GeoJSONNow we have our GeoJSON of circles but nothing to put it in context — no background map.
Thus, we’ll merge in the U.
S.
state outlines.
On the command line: touch mapAndMergeStates.
js.
Populate the file with the following (details in comments):Finally add this to buildmap.
sh:mapAndMergeStates.
js # merge in states, output topo.
geojsonStep V: Convert GeoJSON to SVGWe have our master GeoJSON file now (topo.
geojson)and can convert it to SVG.
However, we should first apply a geographic projection to our data so the SVG is generated to scale.
To do this, we’ll use Bostock’s geoproject.
We’ll also want to split our GeoJSON so that each feature is on a single line.
Thus, when we convert GeoJSON to SVG, each feature will be on a new line in the XML.
This will be useful when we use the command line to splice in our legend at the end of this tutorial.
Install: npm i d3-geo-projection ndjson-cli -DAt the top of buildmap.
sh put the projection you need (in our case Albers USA) and the SVG output height and width into bash variables:# USA albersPROJECTION='d3.
geoAlbersUsa()'# Display height and widthWIDTH=960HEIGHT=600Then at the bottom ofbuildmap.
sh add:Note line 7, where we run the sed command to pluck the last line from our SVG: Since we used ndjson-split to convert our GeoJSON to a newline delimited JSON file, our SVG XML is also newline delimited.
Thus, with the sed command, we’re shaving off the closing bracket of the SVG.
When we build our legend SVG, we’ll shave off the opening bracket from it too and concat the two together into our final output.
You can learn more about newline delimited JSON and its uses here.
Step VI: Add a legend to our mapWe’re almost done.
The final step is to use node and D3 to generate a legend in SVG format and then merge said legend into our map.
We’ll handwrite our legend in D3 (and export the SVG) using the D3-Node library by Brad Oyler.
We’ll also pass in our SVG height and width variables using flags and parse said flags with the minimist library.
Install: npm i d3-node minimist -DCreate our JS file on the command line: touch buildLegend.
jsThen write the following (details in the comments again):Finally now, in buildmap.
sh we can add:The above runs buildLegend.
js, passing in our SVG width and height.
It then uses tail to concat the legend and the map into our final output topo.
svg.
The last two lines simply clean up the directory to remove non-essential files.
And we’re done!!!.On the command line just run:npm run buildmapIf all goes well, this should generate topo.
svg that looks a little like this.
For reference, here’s a repo including all these files in final form and a direct link to buildmap.
sh in said repo.
.