Communities Matter

Philadelphia is unique among big cities in how it publishes open data for civic hackers, journalists, entrepreneurs, researchers and other users.

The City of Philadelphia has designated the community-built Open Data Philly website as it’s official data directory for open data – we’re the only big city in the country (maybe the only city period) that does not unilaterally control the data portal where city departments publish their data.

This website is shut down

Pursuing a strategy like this is not without its challenges, but I believe that it is ultimately a better way to engage the community of users around open data releases. The members of our open data community are all stakeholders in the operation and management of the city’s open data portal. They can submit new data sources for inclusion in the portal, and they can suggest changes in how the platform works.

The federal government shutdown this week – which resulted in the federal open data site, data.gov, going offiline – offers an object lesson in the benefits of community managed open data portals like those in Philadelphia, Louisville, Colorado and other places. A government shutdown can’t impact sites that the government does not unilaterally control.

And this raises some interesting questions – can an open data initiative be truly open if the government that starts it can shut it down? What happens to an open data portal when municipal leaders that start open data initiatives leave office, are voted out or are replaced by those that are less enlightened?

Some things to consider as we continue to build our community here in Philly.

Experiments in GitHub Based Procurement

The City of Philadelphia is experimenting with some new ideas that we hope will change the way that city departments procure technology solutions. The “petrie dish” for some of the more interesting of these experiments is the social coding site GitHub.

The Background

GitHub logo

Philadelphia is looking for ways to partner more closely with local technology companies on small to mid-sized technology projects undertaken by city departments. One of the biggest challenges we face in creating these partnerships is our procurement process, which wasn’t designed in a way that is conducive to working with small companies and startups. This is not a new problem, and there are actually initiatives underway in Philadelphia to mitigate this challenge and create valuable partnerships with local startups.

One of the unique characteristics of Philadelphia’s procurement system is the process used for “Miscellaneous Purchase Orders” (MPOs). MPOs are are personal and professional services contracts valued at $30,000 or less for which city departments are required to obtain quotes from no less than three different vendors. Traditionally, the process of outreach to vendors for MPO projects has been decentralized and opaque. Department personnel would (and still do) reach out to vendors that they are personally aware of and obtain a quote for services.

This traditional practice is suboptimal for city departments because the pool of vendors from which quotes are obtained is limited. It is also not optimal for local vendors, many of whom may be qualified to provide the service desired but who may be unaware of the opportunity.

With the support of Philadelphia’s Deputy Mayor of Administration & Coordination and Managing Director, Rich Negrin, and at the direction of the Chief Innovation Officer, Adel Ebeid, we are experimenting with ways to change this practice for MPO projects (starting with technology projects), and we’re using GitHub to do it.

The Overview

$30,000 may not seem like a sizable budget for a big city project, but for some technology projects – specifically things like web site redesigns, mobile app development, etc. – this amount can go a long way, and provide ample funding for a small or mid sized effort. It’s probably not the right budget for larger companies but for small companies, or even freelance developers, this is nothing to sneeze at. It is these small companies that we want to create new partnerships with – today’s small startup is (potentially) tomorrow’s big company.

Philadelphia has a thriving technology community, and a robust network of developers and designers who would be ideal for many of the projects that city departments undertake as MPO projects. But until recently, there hasn’t been a mandate to rework how we conduct outreach for quotes on MPO projects. In addition, there hasn’t been the right concentration of knowledge within city government to think outside the box and try some new ways of doing old things.

The Managing Director and CIO have given the Innovation Management Team in the Office of Innovation and Technology latitude to try new things. A good chunk of this team is made up of former civic hackers, most of whom have collaborated on software projects with other developers in the past. When we got the green light to think of some new strategies, our collective interest settled almost immediately on GitHub.

The Experiment

We decided to try and use GitHub – from start to finish – as the mechanism for soliciting bids on a new technology project for the City of Philadelphia.

There are lots of reasons that we decided to use GitHub for this experiment. First, we wanted to utilize a platform that would resonate with technologists (duh, Github). Second, we wanted to have some insight into the quality of other work done by bidders – we expected that firms interested in bidding would already have GitHub accounts, allowing us to look at public repos to see what kind of solutions they were working on and how active they were in participating in other projects.

And finally, we were hopeful that by doing something different – something that potential vendors probably had not seen from a city government before – that we would encourage some “creative” responses.

To conduct our experiment, we needed the right project. The one we ended up choosing is called myPhillyRising – it involves the development of a mobile app to support the efforts of the PhillyRising Collaborative. PhillyRising is a “boots on the ground” effort that is run under the auspices of the Managing Director’s Office. It places city staff in some of the toughest neighborhoods in Philadelphia to support residents that want to bring together their neighbors and organize community events and address chronic neighborhood problems. The objective of the mobile app is to give both PhillyRising staff and residents in the neighborhoods they serve an easy to use tool for communicating, sharing information and organization events.

The first thing we wanted to do was to post the description of the myPhillyRising project publicly, to enable us to share it widely through various channels with our local technology community. We decided to use a simple GituHub Gist for this.

We asked interested parties that had questions to post them as a comment on this public Gist, allowing other interested parties to see both the question and our answers. For those that wanted to submit a response, we asked them to do one of two things:

  1. If they didn’t have a paid GitHub account, we asked that they create a private Gist containing their response and send us a link for review.
  2. For those with paid GitHub accounts, we asked that they create a new private repo containing their response and invite us to be collaborators on that repo.

While we anticipated that the process of using GitHub to solicit bids would be interesting, the response that we got from local vendors – and the nature of some of these responses – exceeded our expectations.

The Outcome

By the end of the submission process, we had received 9 high quality responses from local vendors – a number that far outstrips the number of responses received for similar MPO projects, and three times the number of responses required. This underscores the importance of publicly posting opportunities of this kind – if the city gives local technologists an opportunity to bid on work, local companies will respond.

In addition, several of the responses were highly creative – responding firms took advantage of the fact that the submission platform was a GitHub repo and made the most of the opportunity. One firm created a compelling webpage using GitHub pages to tell the story of Philadelphia residents who might use the new mobile app – they didn’t do this in addition to their standard response, this was their response. Another firm committed prototype code to their repo that let the evaluation team test drive how the proposed app might look and work on a mobile device.

Beyond the initial submissions from interested firms, we found that GitHub Issues were an effective means of interacting with respondents. We used this mechanism for followup questions, to request additional information and even to help us schedule in person meetings with finalists.

bug

question

Going into the final vendor selection, we expect to use the original GitHub repo that was set up to submit vendor responses as the mechanism for delivering the final working mobile application (and for all of the iterations and code reviews leading up to that point).

And ultimately, if we decide to release the code for the myPhillyRising app as an open source project, we can add the appropriate license and flip the winning vendor’s repo from private to public.

The Road Ahead

This experiment focused on a specific kind of city project. Even within the universe of MPO projects (which represents only part of the larger procurement picture), this effort was very narrowly focused on technology projects. This represents just a drop in the bucket of what the City of Philadelphia spends on technology projects.

That said, this exercise has given us valuable insights into ways that we can make the process of engaging with local technology companies easier and more efficient. The latitude provided by the MDO and CIO gave the project team room to try something that the city had never done before.

We are now expanding our efforts to reach out to local technology companies for small to mid-sized technology projects – aggregating new projects that have the right profile to a central website (a simple WordPress-based site) to make them easy to find and allow users to set up subscriptions for the kind of projects they are interested in. This will not be the last project to use GitHub as an integral part of the vendor selection process, and we are actively looking for ways to integrate GitHub into the processes used to interact with vendors on these projects.

We have much more work to do, but the steps that the City of Philadelphia is taking to engage more efficiently with small, nimble, local technology companies is really exciting.

It’s pretty cool to see GitHub being used as a part of these efforts.

Shutting Down a Civic Project

It’s never easy to shut down a civic project or app, but sometimes its the right thing to do.

And so it is with a project I started at the original Apps for SEPTA event in Philadelphia during the Fall of 2011. Effective last Friday, SEPTAlking stopped accepting phone calls and text messages for the next available SEPTA regional rail trains.

I’m proud of this little application that I first threw together during an insanely crazy weekend while helping to manage a civic hacking event I organized with my good friends at Devnuts. This application went on to be featured in a groundbreaking SEPTA marketing campaign that saw the agency promote apps developed by outside hackers to regional rail riders. Awesome stuff.

The application was used almost 20,000 times since late December 2012 by transit riders looking for the next available train at regional rails stations in and around the City of Philadelphia. This level of usage was made possible by SEPTA’s promotion of the app and also by the generous suport of my friends and former colleagues at Voxeo Labs.

The code for the app is open source, and I hope it inspires others as they work on transit apps and civic projects. Philadelphia is a city where many citizens face challenges to traditional Internet access and I think it’s important that this influences how we build civic apps. The apps we build must be able to serve the widest community of citizens possible.

Most of all, I’m immensely proud to be a part of the civic hacking community in the City of Philadelphia – a community that has grown by leaps and bounds because of events like Apps for SEPTA.

Not the first civic app I’ve shut down, and it certainly won’t be the last. Nevertheless, more hacking to come.

Urban Storytelling with Open Data

One of the most important qualities of open data is that it can provide unparalleled insight into how a city works.

Open data empowers urban storytelling – the process of identifying a trend, or some important characteristic of an urban area and then presenting that information in a compelling way for others.

When I speak about open data, one of the things I emphasize is that the tools for creating stories with data are becoming more powerful and widespread everyday. With the right data, it’s not hard to tell an important or compelling story about a city. To prove my point, I decided to “put my money where my mouth is” at a recent civic hacking meetup in the Northern Liberties section of Philadelphia.

The City of Philadelphia and the Philadelphia Police Department recently released a hugely important dataset – major crime incidents occurring in the city. This data is available as both a static download (updated with the latest information each morning) or through set of APIs. I decided to use this data set, and a tool that’s been on my list to experiment with for a while – Torque, from CartoDB – to tell an urban story.

The code I wrote to tell my story is on GitHub, and you can see the data visualization I built here.

Below is a screencast of the steps I took to build this visualization – as I have said many times, the amount of effort required to build a compelling civic application (provided the right data is available) is trivial. See if you agree after watching this short video.

The technology is elegant. The visualization is cool. But the story itself is rather sad – there are way too many homicides in Philadelphia.

Our Mayor and other city leaders are working hard to change this. Just recently, Mayor Nutter articulated the frustrations of mayors from across the country who are calling for tighter restrictions on the guns that cause so many of the homicides in Philadelphia and elsewhere.

I think part of the change that Mayor Nutter and other mayors are calling for is showing people the magnitude of the problem. This is what a compelling visualization can do – it can reach someone the way that a written story or even a one-on-one conversation often can not.

The problem of violence that plagues urban areas isn’t just a challenge for our city leaders. It’s a challenge for all of us – we all need to be a part of making positive change happen. One way to reach people and motiviate them to be a part of the change is to build a compelling visualization – to tell a compelling urban story. With the right tools, almost anyone can do it.

This is the power of open data.

Mid-Atlantic Osmosis

I had the pleasure of attending the UnWIREd conference in Baltimore this past weekend, and got a chance to watch people that love their city engage in a productive dialog about how to make it better.

Dave Troy and Beth Blauer of Baltimore

I have lots of friends in Baltimore, and attending civicly-focused events there is always fun because of the passion and dedication of people in the community. I estimate that I knew more than half of the attendees at the event from previous hackathons and similar events.

I attended the event mainly to serve my own purposes – I’ll soon be taking a new position with the City of Philadelphia, and I wanted to get a sense from those attending UnWIREd of how they see open data from their city being a key ingredient in building solutions to urban problems.

Read More

From “Zero to Civic” in 5 Minutes

One of the most powerful things about open government data is all of the options it enables.

Open data is the single most important ingredient in civic apps, and it can also power visualizations, mashups and countless other civic uses. The very best open data typically lends itself to several different kinds of uses.

When paired with the increasingly sophisticated tools and APIs available to developers, building powerful and useful civic applications with open government data has never been easier.

This post is meant to provide a quick overview to using open government data to build a civic application – to demonstrate that someone with moderate coding ability, even if they’ve never built a civic application before – can go from “zero to civic” in just minutes.

The screencast below demonstrates how to take an open data set for health inspections from the City of Louisville, KY and build a simple but easy to use text messaging application using open source and free tools.

Read More

Speech Recognition for Open311

Really excited about a new project I started recently to enable phone-based speech recognition for 311 service requests.

Here is a screen cast demonstrating the solution.

I write about it in detail on the Tropo blog. Head on over the get the details, or check out the code for this solution (still a work in progress, but under active development) on GitHub.

Building an Open311 Application with Node.js and CouchDB

Lots of work is being done to finalize the next version of the Open311 API spec (officially referred to as GeoReport V2).

Almost a year ago I launched TweetMy311 – a service that lets people report non-emergency service requests using a smart phone and Twitter. Since then, a lot has changed – not only with the Open311 specification but with the tools available to build powerful Twitter-based applications.
Node.js
In the last several months, I’ve spent a lot of time learning about and working with Node.js. Some of the things I did in the initial version of TweetMy311 (written in PHP) are so much easier to do in Node.js that I’ve decided to completely rewrite the application to use Node. In addition, since I initially launched TweetMy311 CouchDB (the NoSQL database on which the app is built) has also seen a lot of enhancements.

I’ve expecting the overhaul I’m currently working on to make the application code a lot more efficient and easy to understand. Once this overhaul is complete, I intend to release a big chunk of it as open source software, so that anyone that wants to build a powerful Node.js/CouchDB-based civic app can do so.

It’s also exciting to see new cities get on board the Open311 bandwagon. The City of Boston is now supporting Open311 and has started to issue API keys to developers.

As part of my work to overhaul TweetMy311, I’ve developed a neat little Node.js library for interacting with the Open311 API. Since I just started to work with the Boston implementation, I thought it would be helpful to others interested in doing so to walk through a quick example.

If you want to run this example for yourself, you’ll need to have Node.js installed, specifically the latest version – v0.4.2. If you have the Node Package Manager installed, you can simply do:

npm install open311

Once you’ve done this, you should be able to run the following script:

Which will output:

This is just a quick example of how to make the most basic of API calls with the Node.js Open311 module. You can use this module to build fully feature Open311 applications.

I’ll be doing some more blogging in the weeks ahead as the rewrite of TweetMy311 continues, and work on this phase of the GeoReport V2 spec is concluded.

Stay tuned!

Experiments in Open Data: Baltimore Edition

A lot of my open gov energy of late has been focused on replicating a technique pioneered by Max Ogden (creator of PDXAPI) to convert geographic information in shapefile format into an easy to use format for developers.

Specifically, Max has pioneered a technique for converting shapefiles into documents in an instance of GeoCouch (the geographic -enabled version of CouchDB).

I was thrilled recently to come across some data for the City of Baltimore and since I know there are some open government developments in the works there, I decided to put together a quick screencast showing how open data – when provided in an easily used format – can form the basis for some pretty useful civic applications.

The screencast below walks through a quick demonstration of an application I wrote in PHP to run on the Tropo platform – it currently supports SMS, IM and Twitter use.

Just send an address in the City of Baltimore to one of the following user accounts along with a hashtag for the type of location you are looking for:

  • SMS: (410) 205-4503
  • Jabber / Gtalk: bmorelocal@tropo.im
  • Twitter: @baltimoreAPI

This demo application interacts with a GeoCouch instance I have running in Amazon EC2 – you can take a look at the data I populated it with by going to baltapi.com and accessing the standard CouchDB user interface. I haven’t really locked this instance down all that tight, but there really isn’t anything in it that I can’t replace.

Locate places in Baltimore via SMS

Besides, one of the nice things about this technique is how easy it is to convert data from shapefile format and populate a GeoCouch instance. Hopefully others with GIS datasets will look at this approach as a viable one for providing data to developers. (If anyone has some shapefiles for the City of Baltimore and you want to share them, let me know and I’ll load them into baltapi.com.

There are a number of people in Baltimore pushing for an open data program from their city government, and I have heard that there are some really cool things in the pipeline. I can’t wait to see how things develop there, and I want to do anything I can to help.

Hopefully, this simple demo will be useful in illustrating both the ease with which data can be shared with developers and the potential benefit that applications built on top of open data can hold for municipalities.

UPDATE (4/18/2011): I’ve actually replicated all of the Baltimore data from the EC2 instance discussed in this blog post to the new Iris Couch instance. Iris Couch is by far the easiest way to get started using CouchDB, and Couch’s replication feature makes it easy to move data into an Iris Couch instance.