Simon Willison’s Weblog

Spinning up a new Django app to act as a backend for VaccinateCA

23rd February 2021

Originally posted to my internal blog at VaccinateCA

My goal by the end of this week is to have a working proof of concept for a Django + PostgreSQL app that can replace Airtable as the principle backend for the https://www.vaccinateca.com/ site. This proof of concept will allow us to make a go or no-go decision and figure out what else needs to be implemented before we can start using it to track calls.

I’m calling it a “prototype” and a “proof of concept”, but my career has taught me that prototypes often end up going into production—so I’m building it with that in mind.

Today I started building that app. The repo is currently https://github.com/CAVaccineInventory/django.vaccinate though we are likely to rename it soon—possibly to VIAL (for Vaccine Information Archive and Library)—Jesse is good at actually relevant codenames!

Here’s what I have so far:

• Proof of concept for Auth0 SSO—mostly from following their tutorial. You can try that out on the staging site homepage at https://vaccinateca-preview.herokuapp.com/
• The very beginnings of a test suite—built using pytest-django. The tests run against PostgreSQL and I had to figure out how to do that inside GitHub Actions—here’s my TIL.
• The tests run in GitHub Actions! Continous Integration—should work against pull requests too.
• ... which means we can take the next step and go for Continuous Deployment. Every commit that passes the tests is now deployed instantly to the staging environment.
• The staging environment itself is currently on Heroku, because they make it ridiculously easy to setup Continuous Deployment—it’s literally a checkbox in their admin panel. I wrote about this a few years ago: How to set up world-class continuous deployment using free hosted tools. It’s likely we’ll move this to Google Cloud at some point since other VaccinateCA stuff is running there. I know how to run Continous Deployment using Google Cloud Run so that could be a good option here.
• Errors now get logged to a new project in the VaccinateCA Sentry instance.

And the biggest thing: I’ve implemented a set of Django models against the SQL schema that has been coming together in this pull request. These are exposed in the Django Admin (just with default settings, no customization yet) in the staging environment.

You can try those out by visiting https://vaccinateca-preview.herokuapp.com/admin/ and signing in with username demo and password demo (this account will be deleted the second we have any real data in the prototype).

I wrote data migrations to insert states and insert counties—you can see the results in the admin here (counties) and here (states).

Some engineering principles

I’ve invested a lot of effort today in getting some fundamental things set up: a test suite, continuous integration, continuous deployment, and a detailed and up-to-date README.

The reason I’m investing that effort now is that I know from experience that these things are incredibly valuable, and very easy to implement at the start of a project... but become massively harder the longer you delay them. Adding comprehensive tests, documentation and CI to a six month old project can take weeks. Adding them to a project that is just starting takes just a few hours.

I also plan to lean very heavily on the Django migrations system.

I’ve worked at companies in the past where database migrations—any kind of schema change—are slow, rare and exciting. This has horrible knock-on effects: engineers will go to great lengths to avoid adding a column to a table, which can lead to a rapid acretion of technical debt.

I want schema changes to be quick, common and boring. Django’s migration system—especially against PostgreSQL, which can execute schema changes inside transactions—is ideally suited to this. I want to start using it agressively as early as possible, to ensure we have a culture that says “yes” to schema changes and executes them promptly and frequently.

Next steps

I want to get some real data into the system! I’m going to lock down the security a bit more, then take some exports from Airtable, convert them to the new schema and load them into the prototype. This will allow us to really start kicking the tires on it.

I’m tracking all of the work on the Django app in the issues for that repository.

Part of series VaccinateCA internal blog

1. Getting started - Feb. 22, 2021, 5 p.m.
2. Spinning up a new Django app to act as a backend for VaccinateCA - Feb. 23, 2021, 5 p.m.
3. Importing data from Airtable into Django, plus a search engine for all our code - Feb. 24, 2021, 5 p.m.
4. Django admin customization, JSON in our PostgreSQL - Feb. 25, 2021, 5 p.m.
5. Drawing the rest of the owl - March 1, 2021, 5 p.m.
6. API ready for testing, first video status update - March 2, 2021, 5 p.m.
7. Replaying logs to exercise the new API - March 3, 2021, 5 p.m.
8. … more