I created an interactive map of my hikes using the powerful Datasette tool by Simon Willison. The result was an intuitive, easy-to-use map that links each hike to a full report on my blog. Many interesting technologies are at play.
The underlying database is sqlite3, a self-maintaining SQL compliant standalone database. The mapping functionality is provided by a datasette plugin, and it was deployed to Google Cloud Run as a service, then mapped to a custom domain (ironhiker.cloud). Cloud Run automatically scales your application instances as needed, but is a scale to zero service, meaning when it is not in use, all instances are killed and no charges are incurred.
A map of 800+ hikes is great, but even better, I created view tables that filter them into more useful groups: Favorites, Ultras (mountains with prominence > 1500m/4,900’), and one for each Sierra Club list. When you click on a point, a custom popup is shown with a link to the report.
Creating the database
I started with a CSV file exported from my peakbagger.com account. I did a fair bit of cleaning and polishing in Google sheets, verifying links to reports, ensuring the correct latitude and longitude for each mountain, pruning some columns, etc. I also added Boolean fields to track membership in each list. Data cleaning was the most time consuming part of the project, spread out over a couple of weeks.
I set up my work environment in Fedora Linux. First, I created a python virtual environment, then installed datasette, sqlite-utils, and the mapping plugins (leaflet and cluster-map). Datasette has exquisite documentation. If only more tools were so well documented.
# datasette
pip install datasette
pip install sqlite-utils
# plugins
datasette install datasette-cluster-map
datasette install datasette-block-robots
Importing the data into sqlite3 with datasette is straightforward because datasette automatically inspects the data and assigns data types to each field. If the data isn't imported as expected, you can easily modify the data types afterward to suit your needs.
sqlite-utils insert hikes.db All_Hikes hikes.csv --csv -d
I wrote a script that imports the data, creates the popup, and all the view tables. Here is an example of how to create the popup column (it needs to be named “popup”), and one of the view tables:
---- add the popup to the main table
ALTER TABLE All_Hikes ADD COLUMN popup TEXT;
UPDATE All_Hikes
SET popup = json_object(
'title', PeakPointName,
'description', ElevationFeet || ''' elevation, '
|| VertUpFt || ''' gain, '
|| DistMi || ' miles',
'link', TripReport
);
---- create the view tables
CREATE VIEW Favorites AS
SELECT
popup,
latitude,
longitude
FROM All_Hikes
WHERE Favorite == 'TRUE';
A side note on duplicate points. Several mountains were ascended using different routes. Each route has it’s own row in the table with a different report, but the same latitude and longitude. The cluster map elegantly breaks each one out when you zoom in on a point.
Testing the map locally
With the database ready and the mapping plugins installed, the map is auto-generated as long as the database contains latitude and longitude columns. You can test it locally by running datasette with your database. This starts up a local web server that you can browse to test out functionality.
keithw@ai3:~/datasette$ source activate_env.sh
(datasette) keithw@ai3:~/datasette$ datasette hikes.db
INFO: Started server process [789119]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
Deploying to Google Cloud
Datasette wields deeper magic. There is a publish option that can bundle python, datasette, plguins, and the database into a container and deploy it to a number of cloud service providers. I used it to create a Google Cloud Run service. Before you can publish to the cloud, you need to install the CLI for that service. For Google, I installed the Google Cloud CLI. Commands for the Google Cloud use the gcloud program. Finally, you need a Google account and have to set up billing in the cloud dashboard. This is what the publish command looked like (I left out the —about option):
datasette publish cloudrun hikes.db \
--service iron-hiker --install=datasette-cluster-map --install=datasette-block-robots \
--title='Iron Hiker Trip Report Database' --max-instances 10
The first time I ran it, I had to grant some permissions to my Google Cloud account. Most of the permissions I was able approve at the command line, but one or two needed some approval in the dashboard. I got it to work on the third try. The entire upload process took a couple of minutes, but the Google CLI provided real time feedback at each step. When it finished, it gave me the URL where the site was running. The default URL had the service name in it and ended in .run.app. Everything worked great using the system generated URL.
I decided to go one step farther and map a custom domain to the service. Domain mapping is available in most Google Cloud regions, but not all. I registered a domain at porkbun.com, then returned to the Google Cloud dashboard and went through the steps to verify ownership of the domain. Google provided a DNS TXT record that I added to my domain for verification. Once verified, the dashboard generated DNS A and AAAA records that must be added to the domain. There were 4 IPv4 addresses, and 4 IPv6 addresses. I dutifully added them. It took about thirty minutes for the DNS updates to propagate. After that, the site was running under the custom domain. This was a rewarding project and great fun to see the map come to life.