5 Minute REST API

Inspired by one of Waldo Jaquith’s recent projects to create an easy solution for converting data into a REST API, I wanted to try doing the same thing with a bit of a different approach.

Below is a quick screencast demonstrating how to take a CSV file and convert it into a powerful, flexible REST API.

I use the following tools in this demo, some of which I have mentioned before:

I haven’t used CouchDB for a project in a while, but I still wonder at its power and utility – for my money, there is no easier way to create a REST API than with CouchDB.

What I think this example demonstrates is that there are an abundance of tools that can be used to create powerful APIs, making it easier and more practical to build solutions with open data.

Find a tool or approach that best fits your technical skills and get hacking!

Speech Recognition for Open311

Really excited about a new project I started recently to enable phone-based speech recognition for 311 service requests.

Here is a screen cast demonstrating the solution.

I write about it in detail on the Tropo blog. Head on over the get the details, or check out the code for this solution (still a work in progress, but under active development) on GitHub.

“Phind It For Me” Live in Philly

Really excited to launch a new OpenGov project in Philadelphia – Phind It For Me.

The service is built on PHLAPI and the point data sets it houses. As such, one could understand why I’d be interested in enhancing the data sets currently in PHLAPI.

I’m really excited about this project – source code available on GitHub – and would love to see if there is an interest in launching in other cities with CouchDB-based geospatial data repositories, like Baltimore.

It’s built on the awesome new SMSified platform from Voxeo (disclaimer, I work there) and uses a Node.js module I built for working with the SMSified API.

As always, dear readers, any comments or feedback is welcomed.

Do head on over to the project website and check it out!

Building an Open311 Application with Node.js and CouchDB

Lots of work is being done to finalize the next version of the Open311 API spec (officially referred to as GeoReport V2).

Almost a year ago I launched TweetMy311 – a service that lets people report non-emergency service requests using a smart phone and Twitter. Since then, a lot has changed – not only with the Open311 specification but with the tools available to build powerful Twitter-based applications.
Node.js
In the last several months, I’ve spent a lot of time learning about and working with Node.js. Some of the things I did in the initial version of TweetMy311 (written in PHP) are so much easier to do in Node.js that I’ve decided to completely rewrite the application to use Node. In addition, since I initially launched TweetMy311 CouchDB (the NoSQL database on which the app is built) has also seen a lot of enhancements.

I’ve expecting the overhaul I’m currently working on to make the application code a lot more efficient and easy to understand. Once this overhaul is complete, I intend to release a big chunk of it as open source software, so that anyone that wants to build a powerful Node.js/CouchDB-based civic app can do so.

It’s also exciting to see new cities get on board the Open311 bandwagon. The City of Boston is now supporting Open311 and has started to issue API keys to developers.

As part of my work to overhaul TweetMy311, I’ve developed a neat little Node.js library for interacting with the Open311 API. Since I just started to work with the Boston implementation, I thought it would be helpful to others interested in doing so to walk through a quick example.

If you want to run this example for yourself, you’ll need to have Node.js installed, specifically the latest version – v0.4.2. If you have the Node Package Manager installed, you can simply do:

npm install open311

Once you’ve done this, you should be able to run the following script:

Which will output:

This is just a quick example of how to make the most basic of API calls with the Node.js Open311 module. You can use this module to build fully feature Open311 applications.

I’ll be doing some more blogging in the weeks ahead as the rewrite of TweetMy311 continues, and work on this phase of the GeoReport V2 spec is concluded.

Stay tuned!

Experiments in Open Data: Baltimore Edition

A lot of my open gov energy of late has been focused on replicating a technique pioneered by Max Ogden (creator of PDXAPI) to convert geographic information in shapefile format into an easy to use format for developers.

Specifically, Max has pioneered a technique for converting shapefiles into documents in an instance of GeoCouch (the geographic -enabled version of CouchDB).

I was thrilled recently to come across some data for the City of Baltimore and since I know there are some open government developments in the works there, I decided to put together a quick screencast showing how open data – when provided in an easily used format – can form the basis for some pretty useful civic applications.

The screencast below walks through a quick demonstration of an application I wrote in PHP to run on the Tropo platform – it currently supports SMS, IM and Twitter use.

Just send an address in the City of Baltimore to one of the following user accounts along with a hashtag for the type of location you are looking for:

  • SMS: (410) 205-4503
  • Jabber / Gtalk: bmorelocal@tropo.im
  • Twitter: @baltimoreAPI

This demo application interacts with a GeoCouch instance I have running in Amazon EC2 – you can take a look at the data I populated it with by going to baltapi.com and accessing the standard CouchDB user interface. I haven’t really locked this instance down all that tight, but there really isn’t anything in it that I can’t replace.

Locate places in Baltimore via SMS

Besides, one of the nice things about this technique is how easy it is to convert data from shapefile format and populate a GeoCouch instance. Hopefully others with GIS datasets will look at this approach as a viable one for providing data to developers. (If anyone has some shapefiles for the City of Baltimore and you want to share them, let me know and I’ll load them into baltapi.com.

There are a number of people in Baltimore pushing for an open data program from their city government, and I have heard that there are some really cool things in the pipeline. I can’t wait to see how things develop there, and I want to do anything I can to help.

Hopefully, this simple demo will be useful in illustrating both the ease with which data can be shared with developers and the potential benefit that applications built on top of open data can hold for municipalities.

UPDATE (4/18/2011): I’ve actually replicated all of the Baltimore data from the EC2 instance discussed in this blog post to the new Iris Couch instance. Iris Couch is by far the easiest way to get started using CouchDB, and Couch’s replication feature makes it easy to move data into an Iris Couch instance.

Tapping into Open Government

This past weekend, I had the pleasure of hanging out with a bunch of fellow geeks for the better part of a day in Philly at Neomind Labs to work on an open government / open data project as part of the Open Data Hackathon.

Hacking on Open Data

Though most of us had just met, we were able to quickly identify a project that we all thought was worth spending some time on – a more user friendly way of submitting reports of potholes to the City of Philadelphia. (The current interface for doing this is not especially user friendly.)

Once we had decided what we wanted to do, we got down to talking about how to do it – after some coffee and doughnuts (graciously supplied by the guys at Neomind) we began a productive brainstorming session that included lots of talk about the Twitter API, CouchDB, Sinatra and Heroku. Shortly thereafter, we committed some tasks to a large whiteboard in the office and began working on our project.

As the day progressed, I couldn’t help but be struck by how much untapped potential there is in the developer community for open government initiatives. There is so much potential benefit for municipalities if they would just reach out to the developers that live there and engage them to help solve problems and make their communities better.

There are lots of smart, civic-minded people out there that want to help. They’re just waiting to be asked. Sometimes, like this past weekend, an opportunity comes around that provides a glimpse of how significant this untapped potential.
Twitter
The guys at Neomind are a perfect example of this.

The City of Philadelphia is working towards releasing open data sets and engaging developers to use them to build civic apps. But right now opportunities to build open data apps are not always that obvious. (There is a lot of progress being made on this front and I hope to blog more about the good work being done by Philadelphia very soon.)

But if developers are coming together to build open government apps in cities that aren’t doing all that much yet with open data or developer outreach – like they did this past weekend – imagine what will happen when they do.

Cities like Philadelphia and Baltimore that are moving rapidly toward releasing open government data sets will reap significant benefits as developers in those cities start to use them to build useful applications.

These cities stand to reap the benefits of all of the civic-minded geeks that are out there waiting to be engaged to help make their communities better.

Building an Open311 Application

Earlier this year, I had an idea to build a Twitter application that would allow a citizen to start a 311 service request with their city.

At the time, there was no way to build such an application as no municipality had yet adopted a 311 API that would support it (although the District of Columbia did have a 311 API in place, it did not – at the time – support the type of application I envisioned).

That changed recently, when San Francisco announced the deployment of their Open311 API. I quickly requested an API key and began trying to turn my idea into reality.

My idea resulted in an application that I soft launched last week. TweetMy311 is now live and can be used in the City/County of San Francisco to report 311 service requests. The project website has a detailed description of how it works, but its very close to my original idea.
TweetMy311
More good news on the Open311 front came recently when it was announced that San Francisco and the District of Columbia had come to agreement on a shared Open311 standard. This means that apps built to work with the San Francisco 311 API will also work with the 311 API in Washington DC. I’m working on enabling TweetMy311 for Washington DC now, and hope to have this service live there in a few weeks.

Ultimately, I hope people use my application, that they like it, and that it makes it easier to report an issue to their municipality. I did, however, have some other motives in developing this application that I think are equally important.

Are You Experienced?

Since 311 APIs are rare, and (right now) applications that use 311 APIs are also rare, I think there is value in being able to capture the experience of developing an Open311 application from scratch. This information can provide tremendous value to the governments that deploy 311 APIs (what works, what doesn’t, what can be improved, etc.), and for developers thinking about building an Open311 application.

I hope to use TweetMy311 to provide feedback to governments that deploy 311 APIs (and to those thinking about deploying one) so that they can get a sense of how the experience works from a developer that has used one. At the end of the day the ease of use of an API, the quality of documentation, the ability to test applications in a meaningful way and a number of other factors will determine how many developers decide to take the step and become a “civic coder” by building an Open311 application.

Getting to Open

For me, the use of open source technologies in TweetMy311 was important. This project provided a great opportunities to learn more about a technology that I have become fascinated with of late – CouchDB. TweetMy311 is a NoSQL application that uses CouchDB at its core. It runs on Ubuntu Linux with Apache and was built with the PHP scripting language (I guess that makes it the CLAP stack – CouchDB, Linux, Apache, PHP)

Building with open source technologies was important because I hope to be able to share the code I have developed with interested governments that want to learn how an Open311 application is put together. I also believe it’s important because I think the Open311 initiative can be a great mechanism for encouraging the use of open source technologies.

Leading up to this project, I developed a small PHP library for interacting with the San Francisco Open311 API. I make use of this library in TweetMy311 and any other developer that wants to use it in their project is free to do so. I plan on branching this library soon so that it can work with the new version of the Open311 standard.

Give it a twhirl

So if you live in San Francisco and you want to give TweetMy311 a twhirl, check out the description on the project website. I’d appreciate any feedback – positive or negative – because ultimately I think it will make the project better.

I had a great experience developing TweetMy311, and I learned a lot. I’m looking forward to sharing my experience with interested governments and other developers.

NoSQL Telephony with Tropo and CouchDB

In the last two posts, I’ve provided a basic overview of how to create cloud telephony applications using the Tropo platform and CouchDB.

Apache CouchDB Logo

In the first post of this series, I walked through a quick install of CouchDB and provided information on getting a Tropo account set up. In the second post, we created a simple auto attendant Tropo script in PHP that populates a CouchDB database with a call record for each inbound call that is transferred.

I’ll conclude the series with information on how to retrieve information from a CouchDB instance for use in a cloud telephony application, and talk about design documents. This post will also introduce the reader to the concepts of CouchDB Views and Show Functions – powerful tools that can be harnessed to create truly cutting edge cloud phone apps.

First, let’s create a CouchDB database to hold our call settings.

Creating a Call Settings Database

As mentioned in the previous CouchDB posts, you can create a new call settings database using curl from the command line, or using the Futon GUI.

$ curl -X PUT http://your_new_couchdb_ip:5984/call_settings

You should see a response from CouchDB like this:

{“ok”:true}

You can add a record to the call settings database the same way. This time, however, we’ll append the URL for our CouchDB database with a document ID, in this case ‘1000’ – this is the extension that a caller to our cloud telephony app will dial. We’ll use the document ID and and the CouchDB REST API to get all of the settings we’ll need to conduct the transfer – these settings can be seen in the document structure below (feel free to add others to meet your needs or preferences).

$ curl -X PUT http://your_new_couchdb_ip:5984/call_settings/1000 -d ‘{“first_name”:”Joe”,”last_name”:”Blow”,”phone”:”17777777777″,”title”:”Master of Disaster”,”ring_tone”:”audio/ring.wav”}’

You should see a response from CouchDB like this:

{“ok”:true,”id”:”1000″,”rev”:”1-0cf5a7c3a70ac5760f1a5d8dcb8b48d2″}

Let’s add a few more documents to our call settings database (replacing the telephone numbers below with real ones that you want callers to transfer to) and then view all of the documents that we have created.

$ curl -X PUT http://your_new_couchdb_ip:5984/call_settings/2000 -d ‘{“first_name”:”Harry”,”last_name”:”Smith”,”phone”:”18888888888″,”title”:”President of the World”,”ring_tone”:”audio/ring.wav”}’

$ curl -X PUT http://your_new_couchdb_ip:5984/call_settings/3000 -d ‘{“first_name”:”Martin”,”last_name”:”Scorsese”,”phone”:”19999999999″,”title”:”The Departed”,”ring_tone”:”audio/ring.wav”}’

You can view all of the documents in a CouchDB database using the HTTP GET method:

$ curl -X GET http://your_new_couchdb_ip:5984/call_settings/_all_docs

You should see a response from CouchDB like this:

{“total_rows”:3,”offset”:0,”rows”:[
{“id”:”1000″,”key”:”1000″,”value”:{“rev”:”1-0cf5a7c3a70ac5760f1a5d8dcb8b48d2″}},
{“id”:”2000″,”key”:”2000″,”value”:{“rev”:”1-ee2f09516df8b191a89791b01828d788″}},
{“id”:”3000″,”key”:”3000″,”value”:{“rev”:”1-a1399e8218ae75e1efb73ba3f87862ff”}}
]}

Now we need to modify our Tropo PHP script to retrieve the settings we want to use with each transferred call.

Note, for now we’ll keep the logic simple – if a caller enters an extension that does not exist we’ll get a specific HTTP response back from CouchDB – something in the 400 class of responses. If this happens, we’ll just end the call – in the real world you’d want to do something a little more friendly, but you can sort that out when you build your own cloud telephony application. 😉

Modifying the Tropo Script

So, our new Tropo script looks like this:

Note that the getPhoneNumberByExtension() method no longer returns a hard coded phone number – it is using the 4-digit extension entered by the caller to access our CouchDB database using the REST API. The response from CouchDB is a document in JSON format, that we can easily parse using PHP’s handy json_decode() function.

I’ve also modified the value of the $callLog variable to correctly capture some of the variables exposed in the Tropo environment (i.e., the session ID of the call, and the caller ID – see this thread for more information).

So now we have a working cloud telephony application built on Tropo that uses CouchDB to get its call settings, and also to write a call record for billing, reconciliation, etc.

As cool as this is, there is still a lot more we can do with CouchDB in our cloud telephony apps. Note the constants declared at the top of the Tropo script – the last two are blank; one for a design document name, and one for a show function.

define(“COUCH_DB_DESIGN_DOCUMENT_NAME”, “”);
define(“COUCH_DB_SHOW_FUNCTION_NAME”, “”);

Let’s talk about those concepts now, and explore how they could be used in a cloud telephony application.

Getting more out of CouchDB – Design Documents, Map/Reduce and Show Functions

As the title of this post suggests, we’re building cloud-based phone applications without SQL. CouchDB doesn’t use SQL – instead it uses a Map/Recuce framework to index documents in a database.

Map functions can be used to emit a key-value listing of documents in a CouchDB database. Reduce functions are used to aggregate the key-value pairs emitted by a Map function. Map/Reduce functions (or Views) live inside of a special document in a CouchDB database called a “design document“, which has a document ID prefixed with “_design/”.

For example, suppose we have a special design document in our database called “_design/extensions” with a View called “getExtensions” – our View is made up of a Map function and (optionally) a Reduce function. Let’s assume our View has only a Map function to return data on extensions with valid phone numbers to transfer a caller to.

function(doc) {
  if(doc.phone.length == 11 && doc.phone.substr(0,1) == ‘1’) {
    emit(doc._id, doc.phone);
  }
}

Our Map function (which is written in JavaScript, and stored in our design document) has one parameter – doc. This function is called for each document in our database, and the doc parameter represents the document itself. As can be seen, we simply examine each document in the database to see if it has a valid phone number (11 digits, starting with 1).

Views are accessed using a specific URI structure (do note, however, that the REST API for querying Views can change significantly between CouchDB versions), and the response is a set of key-value pairs formatted as JSON.

http://server_ip_address:5984/database_name/_design/design_document_id/_view/view_name

$ curl -X GET http://your_new_couchdb_ip:5984/call_settings/_design/extensions/_view/getExtensions

You should see a response from CouchDB like this:

{“total_rows”:3,”offset”:0,”rows”:[
{“id”:”1000″,”key”:”1000″,”value”:”17777777777″},
{“id”:”2000″,”key”:”2000″,”value”:”18888888888″},
{“id”:”3000″,”key”:”3000″,”value”:”19999999999″}
]}

You can check to see if your Map function is working properly by adding a document with an invalid phone number.

$ curl -X PUT http://your_new_couchdb_ip:5984/call_settings/4000 -d ‘{“first_name”:”Richard”,”last_name”:”Kimble”,”phone”:”4444444″,”title”:”The Fugitive”,”ring_tone”:”audio/ring.wav”}’

Accessing the getExtensions view will return the same results as before, as the phone number for the new document does not pass validation. Using design documents and Views, cloud telephony developers can use CouchDB to build grammars for user input which will significantly enhance the usability of the sample application we’ve used during the last few posts.

But there is even more potential with another piece of functionality in CouchDB – show functions. Show function also live in design documents, alongside Views. Show functions allow a developer to return specifically formatted content from a CouchDB instance, not just data in JSON format.

A basic show function that can be used to return information from our CouchDB database in the format of a SRGS grammar might look like this.

function(doc, req) {
 var grammer = ‘<?xml version=”1.0″?><grammar xmlns=”http://www.w3.org/2001/06/grammar”>&#8217;;
 grammar += ‘<rule id=”R_1″><one-of>’;
 grammar += ‘<item>’ + doc.phone + ‘<item>’;
 grammar += ‘</one-of></rule></grammar>’;
 return grammar;
}

Like Views, Show Functions are accessed using a specific URI structure.

http://{server_ip_address}:5984/{database_name}/_design/{design_document_id}/_show/{show_function_name}/{document_id}

Note that the Show function above is different from the Map function discussed earlier in that it takes two parameters – doc and req. As before, the doc parameter represents the document the function is called against. The req parameter represents a parameter that is sent in with the HTTP request, which can be used inside the function to render output. So a Show function canbe accessed using the above URL with an optional parameter as well, like so.

http://{server_ip_address}:5984/{database_name}/_design/{design_document_id}/_show/{show_function_name}/{document_id}?{parameter}={some_value}

Conclusion

I hope this series of posts has provided a helpful overview of CouchDB, with an emphasis on how it can be used to build cloud telephony applications.

Cloud telephony platforms like Tropo, CloudVox, CallFire and others provide enormous flexibility to developers in building and deploying sophisticated cloud telephony applications.

Pair these tools with CouchDB and you’ve got a powerful combination for building full featured, easy to maintain cloud-based phone apps.