RCOS: Project Floodlight

Scraper.py

2010-07-19T06:37:00.000-07:00

The Short (and sweet)

I finally added in the script I've been using to scrape nysenate.gov for a while now. I should have added this to the repo long ago but alas I didn't. By integrating this information into my library I give developers access to information they otherwise wouldn't have and create connections that would otherwise be difficult to follow.

The Long (and detailed)

I just added a file to scrape the nysenate.gov for senator and committee information. This file depends heavily on the awesome BeautifulSoup library for python. It just makes finding information in pages incredibly easy and almost natural, right out of the box.

The scraping process takes a long time though (remote server response times), so I've been using the cPickle standard module to save data in between sessions and save data for library usage once its fully assembled. In the future I'll be looking into multi-threading the scraping process to make it faster and so that I can do multi-threading. I've always wanted to get into that but never really had to before.

Once the information is stored into the .dat file it can be pulled into the Open Legislation Library. This is primarily done by replacing name references to Senators and Committees with their corresponding objects, filled with the scraped data. By doing this we gain access to two main things:

Committee Relationships: We can now directly find senators by their committee relationships in ways that we could only infer (after much work) before.
Senator Details: We can also place the senators in context by having direct access to the parties they are affiliated with, the committees they sit on, and the leadership positions they hold. They aren't just another name in the cloud.

Looking Forward

Things are starting to fall in place and things are looking up.

There are still a few fairly major issues that need to be dealt with.

When search results are pulled down, they don't contain very good information. To get all the useful information you need to make separate queries to the server for each object. This is slow and painful to do manually. Fortunately, I'm fixing this on the other end and the problem will go away shortly.
You still need to make all your queries through the OpenLegislation client. I'd like to provide hooks into the Objects somehow to automatically create the requests and fill themselves when asked by a user. Still looking into this.

Setup.py

Those issues aside, the library is in perfectly good usable condition. I need to finish up my setup.py file so I can add myself to pypi. I'll probably speak with Rob about this, he's been very helpful throughout this summer.

Firmant

I swear, I'm still looking to move over there (and self host my blog), its just not a simple process (yet)...at least as far as I know. Its been busy but I'll get there.

Upcoming

2010-07-09T06:06:00.000-07:00

This blog will be moving to Firmant in the upcoming days such that I can be of assistance to Rob Escriva in its development and explore a different kind of blogging experience.

I'll keep everyone posted.

Documentation and Accessability

2010-06-07T21:10:00.001-07:00

Documentation Progress

So as I mentioned before, I finished up a version of the OpenLegislation library that works really well for bills and transcripts (but transcripts are boring right now) in the last couple days. Since then I've further developed the documentation to have an examples page, uploaded it to my web server, and created links between my git repo and my documentation.

Accessability

In order to make my script more accessible I am looking into creating a setup.py file to enable my library to be distributed through PyPi and the easy_install interface. Python has a really cool framework for doing this type of thing, I hear Rob will be posting some of his insight into this area soon. I'll put up my thoughts when I've got more experience with.

Web App?

Then I got thinking, why make people install python and your library just to try it out? Could I make the library available in a command line type environment on a web app for anyone to try?

I consulted the #RCOS IRC (always a good idea) and Moorthy thought it was an interesting idea. Rob suggested I use pyparser to create a domain specific language (DSL) that covers the use cases for my library. Good idea.

So today I learned to use pyparser (cool tool) and developed a fairly nice and clean language for recognizing library calls. I even had time to hook that parsing up with eval expressions so I could translate strings matching the DSL into actual commands and store/output the results!

In the upcoming days I'm going to look at having some persistence tied to browser sessions and pushing things back and forth with Ajax. I wasn't initially sure how viable this idea might be, but it looks like its more viable than I thought. If all works out well from here on out I think I'll have taken a big step towards try it before you buy it!

Details on how I did what? I'm still figuring things out, I'll put them up when I get parts done, (hopefully in the next couple days).

NYSS OpenLegislation Library

2010-06-05T10:05:00.000-07:00

Its been a long time since I've posted an update here.

Open Legislation Library

Just today I split the OpenLegislation Library off into a separate project since people in the NY Senate will now be looking at/using it. You can find it here. I've found out at least one other person is writing a library for OpenLegislation as well and hope to get in contact with him.

Documentation

I've also prepared what (I think anyway) is some pretty solid documentation on the libary which can be found in the docs/build/html folder of the repo. I built it using a tool called Sphinx which is very awesome and satisfied (almost) every need easily and out of the box. Not to mention produces documentation that looks clean, consistent, and professional. I'd definitely recommend it to others as a python documentation tool of choice.

Web App?

Since not everyone wants to pull it down (with git) and set up a python environment to play with the library, I am thinking about making a simulated environment in a web app so people can play around with it and see how it works (and how easy it is). I'm not sure how difficult this would be but I'm going to look into it.

Floodlight

I've started work on the Floodlight API now. First step was scrapping some information not available in the OpenLegislation API off of the Senate Drupal site with Beautiful Soup. Let me make special note of the fact that Beautiful Soup is AWESOME. I could not have asked for a cleaner or easier way to scrape data off those web pages. So far I've scraped a senator list, a committee list, and a list of senators on each committee (as well as the committee chair). I'm currently working to scrape all the senator contact information but its got really wild and inconsistent formatting that is making it difficult and full of special cases. Its getting there though.

I'm not sure what other information I should scrape off but if I think if it, Beautiful Soup will be there to make it easy.

Restructuring Data

Now that the OpenLegislation library is in good usable condition I will be using it to do a batch job and pull all of OpenLegislation's information and restructuring for local storage. I think that a lot can be done simply by providing better data organization and linkages. Even more can by done by inserting the information I've scraped into the mix and by providing calculated statistics in some of the views. What exactly I plan to do in this area will largely be the subject of next week's work.

Spreading the Word

In the meantime, I'm going to work on spreading the word in the CIO office that the library has developed and have some of the developers working on/with OpenLegislation take a look at it. I'd encourage anyone else to head over and take a look at it too. Especially the documentation.

Updates and the RCOS Mid-Semester Presentation

2010-04-15T16:39:00.000-07:00

So I just put together our presentation for tomorrow's RCOS meeting. There are a lot of presentations slated for the meeting, so greater elaborations on some of the topics will be posted on the blog in the coming days in lieu of having time to present them to the group.

By the way, to avoid confusion: we're renaming the project very slightly to FloodLight instead of Floodlight because we thought it looked a bit better. The final web application using all of our back end tools will however be called FloodlightProject.org (FloodLight.org was already taken, oh well).

Our presentation is also available for viewing on SlideShare as shown below:

RCOS Mid-Semester Presentation

View more presentations from cihancaglayan.

As a note, we did mean to have our first presentation up when we first posted it, but Graylin had issues uploading it. For your viewing pleasure, I've uploaded it and included it in this post as follows.

Floodlight

View more presentations from cihancaglayan.

Clarifications

2010-04-12T11:50:00.000-07:00

One of the major difficulties we've had with Floodlight to date lie has been making sense of some of the legislative terminology used in NYS Senate and thus OpenLegislation.

One feature of bills in OpenLegislation is their action list. Actions are what they sound like, and they iterate every step that the bill moves through, from introduction, to committee, through the final vote. But the action list are filled with formal legislative vocabulary, and in order for us to put in the business logic to place a bill in a certain stage of the law making process, we need to understand what ever step is.

However, the inner workings of the NYS Senate aren't particularly well documented online. But after speaking with Dean Hill, he directed me to a peer of his in the legislative staff, Mitzi Hart, to answer my questions.

Her clarifications have been immensely helpful, and although some further clarifications on the software end are still needed, we have a lot more of the information we needed to move on in our code.

Later today we will be posting a more formal plan and schedule for the project, as we feel that we haven't been publicly organized enough as of yet. We will also be posting up a layout of all of our planned software, some of which is in the works as of now, and some of which has yet to be started.

Django + WSGI, Ubuntu + OpenSSH

2010-04-05T23:52:00.000-07:00

Tonight we finally fixed up a couple issues that had become major time sinks and obstacles to us getting anything done.

Problem: Django + WSGI
First, we discovered why we couldn't access our CSS files off the server. We had deployed Django on the apache server built into Ubuntu via mod_wsgi on a name based virtual host. The WSGIScriptAlias directive was taking all of our requests and running them through the urls file in our Django app. Upon finding no match, the server would return an empty and/or error page detailing the problem.

Solution
To solve this we created employed the following Alias and AliasMatch directives to catch the static files before they can be dispatch to WSGI for handling.
AliasMatch /media/(.*)
Alias /favicon.ico
Alias /robots.txt

Problem: Inability to Pull/Push
Cihan has spent the last couple weeks being unable to push/pull on the git repository. She set up nearly a dozen different public key identities on github but frustratingly none of them would validate.

Solution
Cihan runs Ubuntu and the Github connection seems to use OpenSSH. There is appearently an open issue here were identities aren't loaded into OpenSSH and recognized by a default scan of the ~/.ssh directory. Instead, a manual ssh-add is required to force the scan and load the identities. After this command everything turns out to work fine.

By adding the code provided by Github for automatic identity confirmation to her .bashrc file we were able to fix this issue (I did this first thing which explains why I never had this issue).

Additional Successes
Unfortunately because this was the last fix we worked out she was unable to take proper credit for setting up the Django Admin tool (I pushed the changes since she could not) so we'll give it to her now. Not that enabling the Admin tool would have been a big deal if not for the problems caused by our deployment solution (my fault there).

Such is life. Work continues in preparation for our presentation on Friday.

Open Legislation Library

2010-03-31T19:50:00.001-07:00

So today I finally got around to uploading a extremely alpha version of an Open Legislation Library. The library (python) allows object oriented access to the bill data available through the Open Legislation API and its web interface.

The Functionality

Currently the library can search by sponsor, bill id, committee, and keyword. After grabbing the data (XML for now) using urllib the library uses the xml.dom.minidom package to parse and build a list of bills. A basic example of usage is as follows:

openleg = OpenLegislation()
bills = openleg.searchByKeyword('healthcare')

for bill in bills:
for member in bill.__dict__:
print str(member) + ': ' + str(getattr(bill,member))
print '\n'

As the output will show you, each Bill has the follow attributes currently defined and pulled from Open Legislation:

id - the Bill idea in the senate (unique identifier)
title
year - year the bill was introduced
sponsor - currently plain text
cosponsors - currently a plain text list
committee - Committee currently/last reviewing the bill
law_section
summary
text - available only on specific bill request
memo - available only on specific bill request

Its functionality is still quite limited but for most basic queries it gets the job done.

Limitations

Current limitations are partly due too poor documentation and/or consistency with the API on the part of the CIO and partly due to our remaining questions about what each field really means and how all the different parts fit together (meaningfully). Until we fully understand the function of an element and its relationships with everything else we're going to leave it out of the library. Better to do one thing well than 10 things poorly.

Future Plans

Current future plans include:

adding actions and sameAs to the bill attributes (need to figure out implementation)
Adding range functionality. The API supports {page}/{results per page} format.
Possibly creating additional queries (as they prove useful)
Possibly finding a new XML package or switching to json. I don't like minidom much.
Packaging some examples/documentation.
Creating a test suite (how do you test an library to a live API effectively?)
Exporting the library to its own Github project

So give it a quick test drive if you want, let us know what you think.

Bill Creation in NYS and Other Legislative Information Websites

2010-02-27T23:40:00.000-08:00

This site, found off the NYS Senate website, details how a bill becomes a law with a focus on the process through the NYS Senate.

Next is a flowchart that I've put together, which displays the important steps, relationships, and players in the bill creation process through the NYS Senate:

This process is part of what we want to visualize in Floodlight. Illustrating where a bill is in the law-making process is imperative in increasing understanding of government and providing a tool that will be useful for citizens to gauge where they will be able to influence the lawmaking process through their elected officials.

Moving from here, Graylin and I are planning on heading to the NYS Library in Albany next week. We're hoping to delve a little bit deeper into the NYS Senate process (as online sources are rather lacking) to make sure that we understand all information necessary to be able to build an accurate and useful tool.

This type of visualization, that of the current state of a bill in the law-making process, has been done by other web services. One example I've found is through the Texas State Legislature.

Although the legislative search section of their site is most definitely lacking in its usability, their display of legislative information in the bill-lookup is useful, although the information layout could stand some improvement.

A more well developed legislative information solution can be found at Open Congress:

Open Congress features better information visualization and organization than the Texas Legislature website, and this is an application that we're using as a source of inspiration.

Client and Project Profile

2010-02-18T17:04:00.000-08:00

Today I gave my Client and Project Profile presentation as the first part of a semester long project for my Information Architecture class. You can find a copy of the presentation here.

In the presentation I discussed our client, their mission, and their vision for fullfilling that mission. We then moved on to the developments that lead to their current situation, took an overview of where they are currently on, and talked a little about where they would like to go. During this segment I referenced a presentation by Andrew Hoppins on their Open Initiative and went over (roughly) the content of this previous post. Finally I gave a little back ground on the goals of Floodlight.

The Questions and Answers

As a final slide, I asked the audience a series of questions probing their thoughts. Summarized Q&A is provided below along with some additional thoughts of my own.

Q: What sites exist for Government Legislation?

A: No suggestions from the audience

I didn't really count on much of a response here, government transparency isn't a common day concern for most people. They aren't likely to know offhand of other source of this information.

Q: Why would you come to a Legislation site?

A: To back up a view point. General concensus that most people are politically opinionated and instead of coming with an open mind, they will come looking for evidence to support their already strong beliefs. Alternatively their belief might not be strong but they would have a basic hypothosis.

A: To see what other people think. This would require a thread like interaction feature set to be enabled.

A: To check up on my senator before elections. Keep him or Kick him.

The feedback here was great. We had previously framed the site's usage as people having a question and looking for an answer. This was strong evidence for an alternative motivation of having an answer and looking so back it up and solidify it. This was the kind of feedback we were looking for here.

Q: What questions do you have?

A: What does this bill mean? Perhaps you could have crowd sourced bill "translations" into common english that everyone can understand. Maybe then allow people to switch between the two views of the bill in some sort of tab structure.

A: Additionally they confirmed the general questions we had outlined in this previous post.

Translations would be very tricky since you want to simplify the bill but can't afford to change the meaning, crowd sourcing this would be difficult since as mentioned above, many/most users may be biased. Additionally it seemed like people were drifting into implementation details which was not productive. How to keep people on the right track?

Q: What information is most important to you?

A: No real additional thoughts were provided here.

After all the previous discussion on the middle two questions I think people were more or less burned out on the topic. This is a big problem for us to figure out. Perhaps next time we'll get a better response.

Information Visualization for Knowledge Discovery: A Computer Science Colloquium by Ben Shneiderman

2010-02-18T17:00:00.000-08:00

Ben Shneiderman is a professor of Computer Science at the University of Maryland and Founding Director of their Human Computer Interaction Lab. Known as the "father of HCI" he has recieved "more awards than we have time to go into" in the words of Debra McGuinness. In his talk at RPI, which Graylin and and I attended today, he spoke about the powerful Knowledge Discovery tools he and his students have created in his lab and some of the effective principles and techniques they have developed and applied.

The Information Visualization Mantra: Overview, zoom and filter, details on demand

Overview: Always give the full set of data first
Zoom and Filter: Then allow users to zoom in on the information they do want, and filter out the information they don't want.
Details on Demand: Now focused, make details readily attainable as the user wants them (by click)

An Interesting Info. Visualization Example:

Social Action seemed to me to be the most relevant visualization tool to our own project, although the others presented were also interesting in their own right. This is a tool for social network analysis. For example, one use demo-ed was common voting records between U.S. Senators. Although this particular use isn't something we would add to Floodlight, related ideas may be. For example, how senators' voting records relate to certain topics of interest is a type of information that would be very useful to a NYS citizen.

Important Take Away Points:

Although this presentation was focused on a lot of numerical and case study data, we still feel that the concepts discussed are important for our own legislative data visualization tool. One of Shneiderman's more important points was that good information visualization is naturally followed by information communication and knowledge discovery, which is certainly one of our main goals. Professor Barb Cutler (RPI CS Department) made another pertinent point; she added during the Q&A portion that any visualization tool should take great care to not display data in such a way that would lead a user towards baseless and otherwise incorrect conclusions, which is another crucial concept we must consider.

Information Architecture Spring 2010

2010-02-18T16:15:00.000-08:00

Information Architecture Spring 2010

I am taking Information Architecture with Roger Grice this semester. For those of you out there wondering what Information Architecture is, take a look at the syllabus exert below:

The field of Information Architecture aims to identify and utilize principles of information organization, collect and interpret empirical data on human information behavior, and develop and apply methods of information design all in the service of creating usable architectures of information.

What does this have to do with Floodlight?

The main task in the class is production of an extensive Architecture proposal for a client and project of our choosing. I have managed to convince the professor to let me double this task up with my RCOS research this semester.

This arrangement will allow me to dedicate more development time to RCOS work on Floodlight and delegate significant design time to my Information Architecture class. Everyone wins.

Additionally this arrangement allows me to solicit the feedback of a fresh and diverse audience of HCI, IT, MBA, EMAC, and CompSci graduate students, many of whom hail from industry and come from a number of countries. Each has a vastly different perspective and it is that perspective I wish gain through the course of this class.

The Deliverables and Feedback

The deliverables for this class include the following elements:

Initial Client and Project Profile
Competitor and Heuristic Analysis
Audience, Personas, and Scope
User Experience Design
Task Flows & Site Map
User Evaluations
Wire frames
Final Presentations
Final Proposal

As each element is completed and turned in for feedback I will post the deliverable and feedback I receive, as well as my thoughts on the feedback, in a separate post and link to it here.

Open Senate Initiative

2010-02-18T08:42:00.000-08:00

The Open Senate Initiative

The Open Senate Initiative is an initiative to Open up the New York senate. It aims to open up the senate in two primary ways: switching operations over to open source solutions and creating open APIs and databases of all Senate Data. Such an initiative is a first in state legislature and they hope to pave the way for other legislatures to follow, leveraging their experience for the benefit of the county at large.

Open Source
By utilizing open source software they will save money, take advantage of community innovation, and create further leverage of tax dollars by sharing their code improvements and enhancements back to the community. They will avoid vendor lock in and gain access to the top developer talent that is frequently attracted to the open source community

Open Data

By opening up their data with APIs they enable developers all over the world to grab, process, and digest their information however they want, whenever they want. By creating feeds of this data they allow constituents to easily follow Senate activities in real time and share them with others. With this flexible platform, they Senate hopes to attract a diverse group of developers to put the data to work and develop solutions to civil problems.

Andrew Hoppin - February, 2009

The first step in this journey was taken when they appointed Andrew Hoppin as the first ever Senate CIO. In the months since, he and his team have moved swiftly to embrace both open software and open information through the release of the new nysenate.gov website and its sub components.

Drupal powered NYSenate.gov - May, 2009

Just 3 months after bringing Andrew on board, the New York senate got a massive overhaul of their web presence. The new site is now powered primarily by Drupal (of which Andrew has been a long time enthusiast) with subcomponents powered by Wordpress (OpenData) and/or/but not limited to inhouse software (OpenLegislation). This new and improved site features blogs for all senators, committees, and the senate as a whole is packed with new features encouraging open communication and participation with constituents.

Open Legislation API Beta - June, 2009

The Open Legislation is a RESTful API that makes available Legislative Data from the State Senate and Assembly. This data includes all bills and the votes, amendments, senators, and committees associated with them (timestamps included). It currently has been implemented through only two interfaces, the Open Legislation Search on the NYSenate website and through a series of mobile apps created by Mark Headd. Project Floodlight will be the first (to our knowledge) 3rd party web application built on the API.

Floodlight Introduction

2010-02-18T00:27:00.000-08:00

What is Floodlight?

Floodlight is a data visualization tool for legislative data. It will feature a dash-board style and provide multiple views of legislative data depending upon the context (explained later). Floodlight leverages data from the NYS Senate's Open Legislation RESTful API.

What is Open Legislation?

Open Legislation's is a RESTful API whose purpose is to make legislative data available and useable online. Currently in beta testing, the API is only known to be used on the main NY Senate website and in a few mobile applications developed by Mark Headd. The NY Senate's current implementation of the API features searching capabilities for legislation, either by keyword, recent action, recent vote, by sponsor, or by committee.

Project Motivation

The main reason for making such legislative information available is to increase government transparency and hopefully to increase citizen participation in state government. This in turn fosters greater trust in government and helps citizens make informed decisions about both their own public officials and the state issues that are important to them.

Project Goals

We feel that for this goal to be realized, the information needs to be visualized in a more digestible and relatable format in order for it to be useful to the average citizen. This is Floodlight's purpose, and our project seeks to help answer the following questions for a NYS citizen:

What has my state senator done/what is he doing now?

What is his/her voting record?
What bills have they sponsored?
What causes, initiatives, and movements (through legislation) does he/she support or not support?

What is happening with _____ issue that is important to me?

What are relevant bills?
Which senators are for/against these relevant bills?
Where does my senator stand on this issue?

What is happening with _____ bill?

What stage of the law-making process is the bill is currently at?
Are there amendments to the bill?
Who is supporting the bill or not supporting it?
Where does my senator stand on this bill?

Where can I influence the state law-making process through the NYS Senate?

At what points of the process can I have influence?
How can I exert my influence and make my opinion known?

Project Links

Wiki
Code Repository