Scout

June 27, 2012 in Craft, Experiments, Resources, Technology

[W]e’re proud to introduce Scout, a tool that allows you to create customized keyword alerts to notify you whenever issues you care about are included in legislative or regulatory actions. This project embodies our goals as an organization: anyone can now freely get the immediate access to information that previously required significant organizational capacity in your state capital or Washington, D.C.

Start by entering a keyword or phrase you would like to get updates about. Scout then saves your subscriptions and sends notifications via email or text message whenever the subscribed issue or bill is talked about in Congress, mentioned in the Federal Register or comes up in state legislation. Through your profile you can create as many alerts as you’d like and group them by tags with the additional option to make them public for others to follow your issues. Users can also complement a Scout subscription by adding optional external RSS feeds, such as press releases from a member of Congress or an issue-based blog.” Source: Sunlight Foundation

[W]e’d be remiss if we didn’t give credit to the Sunlight Foundation’s latest legislative language tool, Scout (http://scout.sunlightfoundation.com/), for alerting us to the existence of the original provision. You can use the tool to set up email alerts for key phrases or follow a particular bill. It covers Congress, regulations across the whole executive branch, and legislation in all 50 states.

We use Scout to get updates anytime Congress is considering expanding what can be withheld under the federal FOIA by setting up an alert that searches bills in Congress for the term “552 (b)” (thanks to a reform written into law last year, all new b3 statutes must cite FOIA – USC 552).” Source: OpenTheGovernment.org, via the Sunlight Foundation

37 responses to Scout

  1. Hello and welcome to the JA’s chat about Scout, a tool developed by the Sunlight Foundation, designed to help make it easier to cover Congress and state legislatures. This is the first in a series of conversations about resources listed on the JA. The aim of these discussions is to dig a little deeper into the possibilities each offers, hear feedback from people who have used or would like to use them, and learn what the people behind the tool wanted to accomplish and what they they’re thinking of next.

    As you may know, we are always adding to the compilation on this site of resources useful to journalists, publishers and anyone interested in exploring and supporting sustainable business models for journalism. On our resources page you’ll find tools, analyses, tips, organizations, reports and more.

    Today Eric Mill is here to talk about Scout. He is the Sunlight developer behind the tool, and he says that of all the work he’s done at Sunlight, he is most proud of building Scout. Eric, welcome.

    • Hi, Emily, thanks!

      Just to describe what Scout is – it’s a search engine and alert system for government action. It covers bills wending their way through Congress and all 50 state legislatures, newly proposed and final regulations across the federal government, and speeches made on the floor of Congress.

      More than that – if you find a bill you care about, either in Congress or one of the states, you can “follow” that specific bill to get smarter notifications, such as votes, activity, and upcoming hearings and debate.

      • I’ve described it as being developed with journalists in mind, but useful to anyone interested in following legislation.

        Just a couple more logistical points before we dig more into Scout: We’ll chat for the next hour, give or take a bit. If anyone would like to join in during this live chat, please sign in to the JA then just jump right in the thread. You can sign in by choosing your preferred social network, where you have an existing profile, and click on that social icon button on the upper right of this page. To post, write in the comment box at the top of the thread reply below an already posted comment. To see what’s new, refresh your screen, or check your email box.

        If you’re reading this after the live chat window, never fear! Comments and questions are still welcome! The conversation thread remains open and active, so you can post a question or comment anytime and anyone who’s participated will get an email notification so they can respond.

        If you are a Twitter fan, we have set up #ScoutJAchat to share there as well.

        Eric, what do people do to use Scout?

      • The easiest way to start off using Scout is to show up to the homepage and type a keyword or phrase into the search box. It’ll search over everything all at once. You can either create an alert right away by hitting the “Create Alert” button, or you can first narrow and filter the search down to what you’re really looking for. Whatever search you see in front of you when you hit the “Create Alert” button is what we’ll notify you about new results for.

      • By default, it treats your search terms as an entire phrase, and looks for the words next to each other. But we do have an Advanced search mode – so if you have fancier searches in mind, you can turn this on by selecting the “Advanced” radio button beneath the search box, and use multiple terms and advanced search operators. This includes a minus sign (-) to exclude terms, and an asterisk as a wildcard

        For example, this search for…

        science nano* -nanomachines

        https://scout.sunlightfoundation.com/search/all/science%20nano*%20-nanomachines/advanced

        …will find any results that have “science”, anything starting with “nano”, but exclude results that contain the word “nanomachines”.

      • When I typed in ‘money in politics’ I got just one hit of a bill in Congress that includes that exact phrase in the bill. I hoped it might get the concept – find similar ideas, etc.

        I got lots more when I typed in ‘campaign finance.’ How “smart” is the search?

      • Great question. Scout doesn’t currently search for synonyms and related words to what you’re typing, so for the most part, it’s as smart as the search query is. And some phrases we commonly use to discuss an issue tend not to appear in legislation.

        For example, “same sex marriage” returns many results, but gay marriage” returns not a single bill.

        same sex marriage – https://scout.sunlightfoundation.com/search/federal_bills/same%20sex%20marriage
        gay marriage – https://scout.sunlightfoundation.com/search/federal_bills/gay%20marriage

        We would love to work on ways to improve this and make the tool smarter, but it’s also the case that this is a power user tool, and precision and predictability are also virtues that power users want out of search engines.

        What we suggest is to create searches for all of the related terms and synonyms around the issue you’re interested in. Where we’d like to go in the future is to make use of the related terms that expert users enter, and offer suggestions and other smart help to users who aren’t as familiar with those synonyms and terms of art.

        As an aside – we do have a particular kind of “smart search” around citations of the law (the US Code),

      • Oops, I got cut off – what I meant to end with is:

        As an aside – we do have a particular kind of “smart search” around citations of the law (the US Code), so if you search for “5 USC 552”, you’ll receive results where that piece of law is cited in various formats, whether or not it matches your exact search:

        https://scout.sunlightfoundation.com/search/federal_bills/5%20usc%20552

      • One great researcher I worked with told me to always start my searches by brainstorming a list of terms! (“Pencil and paper!” she would say. But I figure typing out my list is okay!)

        Maybe you could build that in – a notepad place on the site to brainstorm.

        Anyway, if this is designed for ‘power users’, who do you have in mind? Reporters who follow Congress or state legislatures regularly, or people who cover a million things, including issue that are affected by legislation?

      • I should also add that we do have some specific intelligence around “popular names” for some legislation. For example, the term “Obamacare” never appears in the text or title of the big health care reform bills of 2010. However, a search for “obamacare” will return PPACA and the accompanying Reconciliation Act:

        https://scout.sunlightfoundation.com/search/federal_bills/obamacare

      • Interesting. Did you build that in specifically? However you did it, what was the reason?

      • Yes, I built in that feature with “obamacare” in mind. Other popular terms we have special searches for include “CISPA”, “SOPA”, and “PIPA”. This is less of a power user feature, and more to make the site make more intuitive sense to someone who’s testing it out, or simply has a broader interest in what they know of as Obamacare, even if they don’t know the official names or bill numbers of the original legislation.

        Congress tries to keep up with popular names of legislation, but it’s often slow. It was frustrating to me that, for a while, the word “SOPA” never appeared anywhere in the official data around the Stop Online Piracy Act, so you could go to THOMAS.gov and type in “sopa” and get nothing. Eventually, the Library of Congress added “SOPA” as an official nickname for the bill, but by the time they did the public fight over the legislation had ended.

        We accomplished this through extremely simple means: maintaining a spreadsheet that links popular names to specific bill numbers. We make this spreadsheet public and welcome contributions. You can find more details about this here:

        http://sunlightlabs.com/blog/2012/data-for-better-bill-searching/

      • Eric, this looks like a great service. Will you be offering any sort of API to be able to follow issues from other web or mobile apps?

      • I guess the answer to what kind of power users I mean is: all of the above. Our goals are to make it possible to carve very specific searches that minimize the number of false positives you have to sift through, while also making it quick and easy to just sign up for a large number of alerts about everything. These goals actually tend not to be in tension, but we’re definitely looking for feedback on how to do both of these better, and to make it easier to manage large numbers of alerts.

        Scout is primarily pitched to the audience of people who might love to have a subscription to an expensive pay service on government activity, but don’t have the budget. Political intelligence services, such as Bloomberg Government and CQ Roll Call, are extremely lucrative and powerful – to the point that they very nearly became regulated under the STOCK Act (an amendment that removed these regulatory provisions passed near the end of the legislative process).

        For more on that: http://sunlightfoundation.com/blog/2012/03/22/stock-act-passes-bill-on-political-intelligence-firms-to-be-introduced/

      • The reason these services are so successful is because this information matters, and organizations with that kind of money know it does. Scout is a free service provided by a non-profit using public data, so we can’t hope to offer every kind of feature that a company with a large dedicated staff can – but we feel that we can offer something foundational and useful, to everyone.

      • Hi Eric, I’m a grassroots political activist, and it has been a huge challenge connecting people with policy- the level of distrust with the government and a general lack of engagement is prominent. I see Scout as a great opportunity to get people connected with the decisions being made about them, but not everyone is technologically/politically savvy– they might not even know where to begin. Is there, say, a beginner’s guide or tutorial for someone just starting out in their efforts to use Scout to deepen their knowledge?

      • Jeremy: a lot of this actually already exists. For one, Scout operates entirely on public APIs. When you do a search on Scout and are waiting for search results to come back, Scout is doing an API request to our Open States API, Real Time Congress API, and Capitol Words API.

        http://openstates.org/api/
        http://services.sunlightlabs.com/docs/Real_Time_Congress_API/
        http://capitolwords.org/api/

        For any alerts you create on Scout, we provide an RSS version of your subscription, which can be used to integrate with other websites. We also offer a JSON version — though in many cases, you’re better off directly integrating with one of the source APIs Scout is using, all of which offer JSON. All feeds in Scout require a user account and associated alert, while our public APIs only require that you have registered an API key.

        Finally – if you’ve already registered a Sunlight Labs API key (which can be done here: http://services.sunlightlabs.com) under the same email your Scout account is registered under, we’ll detect that and start showing you direct links to the source data, on the sidebar of your search results. This makes Scout a sort of living documentation for our other public APIs.

      • Eric, that’s so cool! I’ll definitely be looking into this in more depth.

      • Ramiro: we do have a video tutorial that tours the various features of the site and how to make use of them:

        https://www.youtube.com/watch?v=cKbRsZjoIBU&feature=player_embedded

        It’s not complete, though (it doesn’t cover advanced searches), and nor do we have a written guide to all the different ways you can use the site. This is a priority of mine, and something we’ll be expanding in the near future. If you’d like to help out, or offer any suggestions, feel free to email me at eric@sunlightfoundation.com.

      • Thanks Jeremy and Ramiro, for your questions.

        Eric, I appreciate that the spreadsheet for like search terms you point to explains how to contribute in both a tech and non-tech way.

        I want to hear about what you’re doing next with Scout, but first, a couple or people posted their thoughts on Scout early. Luke Rosiak said it made an immediate and tangible difference in his reporting.
        Scout even alerted him to regulations in the federal register before they actually even appeared in the Register.

        How do you do that?

      • Advance copies of regulations, called “public inspection documents” are actually published on the FederalRegister.gov website *before* they are formally published in the Federal Register as a publication.

        Basically, all agencies are required to make their new rules available for physical public inspection somewhere — like a desk or filing cabinet somewhere in Washington, DC. The Office of the Federal Register takes care of making these physically available, but there’s nothing stopping them from simply digitizing these documents and publishing them online too — so that’s what they do.

        FederalRegister.gov is an amazing website, with an amazing story – it was built by a team of citizen developers that first built an unofficial version, GovPulse.us, before they were contacted by the Office of the Federal Register and asked to make it official. They are also a prolific and restriction-free data provider, and we make use of this to tie advance copies of regulations directly into Scout.

        You can read more about this on our blog:
        http://sunlightfoundation.com/blog/2012/07/11/getting-the-jump-on-regulations/

      • Thanks Eric. I’ll be sure to share the information with my networks. Good luck with further developments.

      • Another person who posted early, David Herzog, founder of Open Missouri, mentioned a couple things in his comment – first, he tried it at launch and wasn’t successful at setting up a RSS feed for my alert. Was that a problem you were aware of?

        He also mentions using the OpenStates iPad app from Sunlight and wants to know how the features of Scout might get integrated with that. Anything you’re looking into?

        (I should note here, for people reading after the live chat window, that comments and questions are always welcome! People who participate will get an email notification when new comments go up so they can respond.)

      • I wasn’t aware of any specific issues around setting up RSS feeds for alerts, but depending on the particular time when it happened, it’s certainly possible there was a bug or outage of some kind, they do occasionally happen. I’m glad that it’s been resolved – and David, you can always email scout@sunlightfoundation.com for support with technical issues.

        How and whether we integrate Scout with our Open States app for iOS (http://itunes.apple.com/app/open-states/id500672932?mt=8) is more up to our Open States team than it is to me – I work closely with them, but it is a separate project. That said, it’s something we’ve talked about and there’s no clear answer yet. They may wish to do more kinds of alerts in the Open States iPad app than Scout supports.

        For example, our Congress app for Android (https://play.google.com/store/apps/details?id=com.sunlightlabs.android.congress), which I built and was an inspiration for Scout, contains alerts for tweets, videos, and news mentions, which aren’t currently supported by Scout. There are other technical complications of how this integration might be accomplished, but that’s the main issue to think about.

      • From the many resources listed on the JA, we chose Scout to learn more about because it’s new, easy and useful in many ways. But I also had been interested in Sunlight’s Politiwidgets, developed in 2010, until I learned Sunlight’s not really supporting those or developing those any further, in part because people wanted to work with them in a way they weren’t exactly set up to work. I understand they still draw on current data, but that made me wonder: How do you, as a developer, learn from potential users what they want before you develop a tool?

      • That’s an excellent question. We invested a lot of time into Scout, and so we certainly wanted to have some confidence that it would be useful to other people.

        In Scout’s case, we learned what users wanted in two main ways – firstly, by being those users ourselves. We are an advocacy group that follows all kinds of legislation and regulation in federal and state government, and so we very much wanted something that would scratch our own itches.

        The other way is by observing the success and use of the app I mentioned in a previous comment, Congress for Android. It began as an experiment of mine that I did on a whim, without expectations of much success – but it turned out to be a great success, and has almost half a million downloads to date. More importantly, our users were clearly comprised of both ordinary citizens and professionals who carried it with them on the Hill. It was gratifying that something could be attractive to both ends of the spectrum of expertise.

        Finally, it turned out that push notifications were a very popular feature, and so we learned that people of all sorts want to keep their finger on the pulse of what their government is doing.

      • What do you want to do next with Scout? And what are the challenges you’re facing?

      • Most immediately, we want to expand what information is there: there are tons of reports, memos, filings, and executive orders that we’d like to make as easy to search and be notified of as bills and regulations. This includes reports from the Congressional Budget Office, the Congressional Research Service, the Government Accountability Office, Statements of Administration Policy, memos from the Office of Management and Budget, and whatever other reports that we find or people tell us is useful to their work.

        As I mentioned earlier, I’d like to work on expanding on how smart our searches can be, in a thoughtful way. I’d like to expand what we did with US Code citations to include citations to the Code of Federal Regulations, and direct references to previously passed legislation. We can also be smarter about how we handle searches that users expect to “just work”, like state names and abbreviations, bill numbers, names of Congresspersons, and more.

      • More long term, one vision I’ve found interesting is to grow Scout into a place where users can take their alerts and share them publicly with other people. There’s actually some basic abilities to do just that, now – here’s a set of alerts we’ve created around regulatory reform:

        https://scout.sunlightfoundation.com/user/sunlightfoundation/regulatory-review

        You can subscribe to that set of alerts “all at once” by hitting the “Follow” button, and you’ll get all the same notifications that we do around regulatory reform.

        We’d like to take this sort of thing much, much further and really make Scout a platform for people to help others better track issues, and show off their expertise while they’re at it.

      • Badges for tracking maybe?

        Thank you so much for all your time and insights. Just finally, you’ve said that Scout is the tool you are “most proud of building” in your three years at Sunlight. What are you proud of?

      • What I mean by that is that I think we were able to create something that is useful in an obvious way, right away. The idea behind Scout is in some ways very boring and unoriginal: a search engine, and email alerts. Everyone understands these ideas. But they’re very powerful, and we don’t know of anything that does this well across a broad set of government information, for free.

        I feel confident enough saying this in part because shortly before launch of the product, we actually used the tool to catch a dangerous proposed exemption to the Freedom of Information Act that we would never have caught otherwise:

        http://sunlightfoundation.com/blog/2012/05/25/scout-is-already-delivering-results/

        Because of Scout’s tipoff, we were able to bring this change to the attention of public interest groups working on the legislation and eventually have it amended. We expect that detecting this kind of direct cause and effect between email alerts and actual policy change will generally be much more difficult. But it was certainly a promising beginning.

      • Eric, thank you for building Scout and for talking about it on the JA.

        To everyone, we’ll be continuing our new Resource Chats series in two weeks. We’re in the process of picking the next resource to focus on. You can follow us on Twitter to get an early alert. Also, we always welcome suggestions for new resource listings! Tell us about a tool, a report, a database, a blog, an ad platform, an event planner, whatever you use that helps you do your journalism or publishing work better and ultimately can support your news business being sustainable. Go to the JA Resources Index to check out what’s there already, and look for the Suggest box on the right side of the page.

        Thanks again Eric! Good luck with the next iterations of Scout.

      • And thanks for having me, Emily. It was great to talk all this out!

  2. This tool has already made a tangible difference in several political stories I’ve written for The Washington Times in the last few weeks. It’s not superficial–it searches a breadth of repositories–and it’s not behind the times. A couple weeks ago, it alerted me to some Congressional floor speeches that provided color and timeliness to a story on a topic I was already working on. Last week, I wrote a story about proposed regulations, based on an interview with a Cabinet undersecretary. The next day, the regulations themselves were found by Scout in the federal register’s preview mode–in other words, before they actually even appeared in the Register. And you can just set it and forget it–I mean, who actually reads the federal register? Great work, Eric.

  3. Scout has been called “an alert system for things you care about in state and national government” and hailed as a “promising new tool.” Join me here Thursday, August 16 at 1:30 PM Eastern / 10:30 AM Pacific to talk with Scout developer Eric Mill about what he envisioned, how Scout is being used so far, and where he wants to take it next.

    We welcome input from users or potential users. If you’ve tried Scout, what’s your verdict? What would you change or add? If you cover a state legislature or Congress and haven’t tried it yet, have a look and see what you think. Or tell us your wish list for tracking legislation and Eric will tell you if Scout fulfills it! Drop by during the live chat window or add your comments later and we’ll still get a response. Please sign in to comment. Just click on one of the social network logos on the upper right.

    See you then!

    Emily

    • I’m looking forward to learning more about Scout and Sunlight’s plans for developing it. I tried it at launch and wasn’t successful at setting up a RSS feed for my alert. I gave it a shot tonight and it worked out fine.

      I think the state alerts will be a big hit. I’ve already set one up for “open records” and “sunshine law” in Missouri, so I can stay on top of the action here.

      I’m also using the OpenStates iPad app from Sunlight and am interested in hearing how the features of Scout might get integrated with that.

  4. As a technologist who does politics, I love the concept behind Scout. Good stuff.

Leave a reply

You must be logged in to post a comment.

You can also subscribe without commenting and unsubscribe at any time.