Team Pittsburgh Development Update

March 31, 2015

As we’ve focused our work down into discrete projects, there have been three main guiding principles that we’ve used to guide our thinking and our development: use simple and consistent tools, do things on the server, and always be shipping. I want to talk a bit more about our choices and how we’ve tried to apply these principles.

An early version of the template-maker

Use simple and consistent tooling

Across our two main apps, we’re using basically identical stacks: Flask with Postgres. With Flask, we’ve been taking advantage of several great patterns: the app factory pattern and the blueprint pattern. We are managing both a simple app (wexplorer) and a more complicated app with many moving pieces (the template-maker) and several smaller apps tied together by a larger flow. This makes blueprints indispensable — breaking the larger piece into smaller components allows for easier debugging and testing.

We’ve also worked hard to make sure that the parts of the app that touch the database are pulled into their own module. Separating the presentation layer from the data layer frees us up to try many things later, including potentially building an API and perhaps working towards a larger client-facing application should that be something users need. For now, we are making an effort to do as much as possible server side (more on that below).

As for Postgres, it is both familiar and a good choice. It is fast, flexible, and comes with a host of brilliant features, including native JSON support and full-text search. It has sane defaults, and is overall a great solution for our needs. While we may need to expand into something more document-based (like elasticsearch) in the future, Postgres is a great choice for now.

Do things on the server

Choosing Flask and Postgres was easy because they are familiar tools. One thing that was much harder, though, was actively deciding to do as much processing on the server as possible. Originally, we were designing some of our tooling around heavy client-side interactions. But we had to ask the fundamental question: what is truly gained from avoiding page refreshes? What core interactions are essential to the application? Do you truly need a large client-side application to capture many of these experiences?

There are many, many, many, many posts about the downsides of javascript-heavy applications. For us, removing the javascript application made the application simpler to test, made the page load much faster, and cut down on the time it took before content could be read and interactions could take place. It also reduced the size of the codebase and made it much easier to keep track of state. For a prototype-stage app like the template-maker, keeping everything on the server has allowed us to move faster on new features.

These decisions have taken us this far, but we may decide to go for a javascript-heavy client side app in the future. We’ve planned for this as well — separation of the data processing and the presentation will allow us to quickly create a JSON-based REST API to build a rich client app should the need arise.

Always be shipping

Improvements to Wexplorer, originally released during build week

Part of the CfA playbook is that the strategy is delivery, and we’ve delivered software almost every week of the fellowship, including during the residency. Here is what we’ve developed so far:

• Wexplorer - a tool for finding out what’s currently on contract with the City of Pittsburgh (Flask + Postgres/Github)
• Wextractor (or the W-Drive-Extractor) - a general purpose ETL tool designed to move data from one format into another (originally Excel spreadsheets into relational Postgres tables)
• The Pittsburgh Procurement Explorer - a tool for figuring out which process you should use to buy new things (Jekyll/Github)
• Cleaning-Pgh - a site to advertise an upcoming RFP, along with some language about that RFP simplified into a much more digestible Q&A format (Jekyll/Github).
• The template-maker (currently password-protected) - a tool for building document templates and then documents from those templates (Flask + Postgres/Github)

We’re planning to continue to develop new features and improvements for all of these sites! All of the code is available on github and is open source — feel free to open issues and submit pull requests.