Question:
What tools and metrics are news organizations using to measure social media impact?
New tools like Seattle’s Newsdex offer the ability for local news sites to quickly assess their network size, and how they rank amongst their peers. What are the different ways you are measuring impact, reach and quality of service to the readers you serve?
Who wants to know?
Michael Fancher is a co-convenor of The Seattle Journalism Commons, which connects people and ideas, in person and online, in order to catalyze journalists and the public in creating, disseminating and engaging with news and information. Fancher retired from The Seattle Times in 2008 after 20 years as the executive editor.
Answers: Remember to refresh often to see latest comments!
24 answers so far.
-
Emily Harris says:
January 26, 2012 at 12:24 pm
-
Andre says:
July 31, 2011 at 8:12 pm
-
Scout Analytics says:
June 23, 2011 at 3:01 pm
-
Mike Fancher says:
June 17, 2011 at 12:07 pm
-
Joy Mayer says:
June 17, 2011 at 11:56 am
-
West Seattle Blog says:
June 17, 2011 at 11:41 am
-
Mike Fancher says:
June 17, 2011 at 1:13 pm
-
West Seattle Blog says:
June 17, 2011 at 1:45 pm
-
Rose Kelsch says:
June 16, 2011 at 3:55 pm
-
Mike Fancher says:
June 18, 2011 at 11:52 am
-
Journalism Accelerator says:
June 20, 2011 at 7:38 pm
-
Indian Drama Serials says:
June 15, 2011 at 10:06 am
-
Rusty Coats says:
June 14, 2011 at 2:32 pm
-
Gary Love says:
June 14, 2011 at 4:24 pm
-
Gary Love says:
June 10, 2011 at 9:54 am
-
Winning Mark says:
June 13, 2011 at 6:50 am
-
Chuck Taylor says:
June 10, 2011 at 8:00 am
-
Gary Love says:
June 9, 2011 at 10:44 pm
-
Mike Fancher says:
June 9, 2011 at 10:16 pm
-
Lisa Skube says:
June 9, 2011 at 8:49 pm
-
Gary Love says:
June 9, 2011 at 4:44 pm
-
Chuck Taylor says:
June 9, 2011 at 9:41 am
-
Mike Fancher says:
June 8, 2011 at 5:04 pm
-
Gary Love says:
June 8, 2011 at 4:16 pm
| ParticipantsWhat about influence in terms of change? I’m intrigued by the measure “how many conversations did you provoke today?”- as Gary put it. Sometimes news organizations can trace their reporting directly to changing laws – like novelist and muckraker Upton Sinclair did with The Jungle more than 100 years ago. Anderson Cooper got trackable popularity points during Hurricane Katrina when he called out Senator Landrieu for appearing removed and unsympathetic to the plight of people in New Orleans. What is the right measure of the impact of journalism?
You always have to look at growth. Are the number of your facebook page likes increasing at an encouraging rate? Are your tweets being retweeted by those who matter and can get your name out there? Metrics like these are the ones that matter and that such organization should focus on.
It depends on whether the metrics apply to audience acquisition or monetization.
Social and search are audience development techniques. Metrics on social influence are great in terms of figuring out how to use a loyal audience to attract more audience members. Just as measuring keyword rankings helps to acquire traffic, measuring influence helps to acquire traffic. The key becomes once on the site how the publisher translates fly-by traffic into a loyal audience member. Knowing whether a visitor is a fly-by or a loyal audience member as well as the source should help a publisher optimize acquisition results.
Once acquired, a direct relationship with the reader is the basis for a recurring revenue model. Consequently, the most “profitable” publishers measure engagement of their loyal audience, because engagement is the unit of monetization. A publisher can charge a reader for engagement with content or sell the readers engagement to an advertiser. What share of online time are is the publsiher garnering? Publishers who measure and optimize their readers engagement have the most profit.
ProPublica has announced a new feature to engage readers in sharing investigative stories, called #Muckreads. See http://bit.ly/lnqYlb
Rusty’s right. The most valuable tool will be when we can demonstrate our value to our actual customers — the people who pay us, meaning the advertisers. That doesn’t exist yet.
In the meantime, you have to enter into social media knowing what you’re trying to accomplish. Is it traffic back to your website? More specifically, is it more eyeballs and new traffic? Or more loyalty from the core audience? Is it more conversation, reaction and feedback from the community?
There are ways to strategically approach each of those things. But smart news organizations will know what they’re trying to do, and what success looks like for their specific situation.
In May, a group of smart engagement folks met at the Reynolds Journalism Institute to talk about metrics for engagement. You can find some information about their recommendations, and download their full report, here: http://www.rjionline.org/news/resource-newsrooms-measuring-success-audience-engagement-efforts-0
Hi all. From the micro-news-organization perspective, there ain’t enough hours in the day to use more “tools.” It’s all I can do to get into the first screen of Google Analytics every day, and to check in on Stat Counter a time or two during the day to see if we’re having a traffic spike about something, or not. All I can tell you is, we were early adopters of the social-media basics – Twitter and Facebook both, dating back to 2007, and it took a couple more years before everybody, big and small, was really jumping on the bandwagon. We think it’s another great way to engage with people wherever they are – and we do this WITHOUT technohelpers – we use both in their native environments (no Tweet Deck or whatever here) – and our main Twitter account is conversational rather than link-heavy (which is probably why we don’t get a ton of referrals from Twitter, though we do have a secondary account @westseattlenews hooked up to RSS from our news feed). I’m old-fashioned in that, if you think too hard about it, try to analyze it, try to play too hard to it, you tend to lose whatever secret sauce got you where you are. In TV, this ruination was the “overnight rating service.” I see way too many people playing social media the same way “oh, everybody shared the link about the pregnant skunk, therefore we will focus heavily on expectant-animal stories from hereon out” and it makes me sad. Measure if you must, but don’t leave your gut out of it – that’s the measurement, and what you hear from people, that matters. We have won awards national and regional for our “community engagement” without being particularly scientific about it. Obviously harder to say that for a big multi-employee organization, but figure out who gets the “secret sauce” and have them teach it 🙂 – Tracy
Tracy,
Thanks for the first-hand testimonial. I’m curious to know about your experience with advertisers. How do they assess your engagement with readers? (See Joy Mayer’s comment.) What’s helping to drive ad revenue so far?
If you had a really simple tool to reflect the value of your content to readers, would that help you on the revenue side?
When people stop talking about what they read on “the blog” – businesses will stop buying ads. Word of mouth is everything. I don’t know how you measure that. Businesses buy us because they read us, or because their customers tell them they have to “be on the blog.” The fact we also have one of the largest social media audiences in the city, and the largest in West Seattle, doesn’t matter much to prospective advertisers at this point, though if we feel the need to impress someone, we throw some numbers around (3,600 on Facebook – we are building back from having to abandon the old “profile” page when it maxed out at 5K – and 10,500 on Twitter) and sometimes that helps enhance the numbers we can toss around from our Web readership. But very rarely – because it’s a relatively small number of businesses that really and truly understand the whole so/media thing thus far, overall. Some do it, because someone told them they had to.
Planned Parenthood Advocates of Oregon considers several factors when evaluating the impact of our social media tools. With an email list of 80,000 members, we are careful when crafting the subject line of our messages. We look at the open rate to determine the number of people we reached per message. This helps us evaluate whether the subject line was effective in terms of enticing people to read more. In addition, we also look at the click throughs per link featured in the email message and whether the message was forwarded on to friends. When using an email message to call our members to participate in an online campaign, we also review participation rates.
Facebook is another tool utilized by the organization. Measuring the number of impressions and feedback per post is helpful. The summaries provided by Facebook also allow us to understand our growth and participation during the week. Often times, links to our blog are posted on Facebook. Google analytics allows us to see how many of our facebook fans clicked through to read the blog.
This information from Planned Parenthood causes me to wonder what click through and forwarding rates are considered successful for email and Facebook calls to action. Seems like organizations would benefit from knowing how their results comparable to peer organizations.
This blog post from a major outbound email provider (Campaign Monitor) has an interesting breakdown of average email open rates by industry (including nonprofit, e-commerce, religious, travel, etc) Read the blog post
One option you may want to consider is using Sysomos Heartbeat, a social media monitoring service that also includes an extensive number of traditional media sources.
I think we’re scratching around the edges with these tools – which are terrific and evolving and so much cooler than anything we’ve had before – but lack the final mile of measuring impact: Conversion.
For media companies, a big following in social media usually means only that you have an enhanced distribution system. Size matters only if these people convert from drive-by users (visiting 4 times a month, clicking on 3 stories per visit) to more loyal users – and that THAT conversion (loyalty) leads to higher awareness/click-thru of my advertisers, and that THAT conversion (advertiser awareness) leads to positive sales impact for advertisers. Being able to draw a visible thread between social followers/friends to loyalty, advertiser awareness and point-of-sale impact would be very tasty because it would tie social media to overall sustainability.
I have not seen such a tool.
Peer comparisons are hugely valuable to drive behavior internally; we’re all competitive beasts at some level. It’s very interesting to learn how Newsdex has evolved and how it’s also a white-label product. But if I were to use it in a newsroom and focus on which of my reporters were making the most use of social media, I’d also want to track how many people they were following. The listening aspect of social media is far more important to creating great journalism than the distribution side. And I think appreciation of that – and its possibilities – are just starting to emerge.
I like work being done by matching CRM tools (Highrise) with social media, and love sentiment analysis tools for listening to conversations (Radian6, e-cairn, twitter analysis tools like CrowdyNews). But until we can tie our social media footprint to these levels of conversion, sustainability will continue to plague us – in both the non-profit and for-profit spheres.
I absolutely agree that creating loyal readers should be a driving goal behind a media company’s social media strategy. I’ve tried to think through that for websites (leaving out social media for a moment) and had trouble quantifying it. The difficulty that I found was while print mediums allowed an off or on metric (subscriber or non), the web is more of a slope of user activity. Any line in the sand on that slope felt like an arbitrary measurement point. Some folks have made some strides in this area, for instance Belo’s mult-measurement approach (http://www.rjionline.org/blog/analytics-stoplights-belos-tv-stations)…however, I think we’re all looking for actionable audience metrics.
I’m curious about your thoughts for measuring/comparing the actions of individual social brands. Specifically, I’ve been reluctant to place actions performed by those social brands (# of friends, # of posts a day, # of links shared) simply because I lacked a measurement to show one was better than another. Since the correct # of posts is a mixture of how much you have to say and how much your audience wants to hear. And the correct # of friends is the highest number of folks that you can listen to without it becoming an undifferentiated crowd.
In other words, Newdex does the easy stuff. Measuring things where “more = better”. However, when dealing with softer subjects like engagement, conversion, loyalty, etc. I’m not sure if there is a standard way to measure good vs. bad.
Lastly, I should note that capturing information like # of posts, # of friends, links shared, etc is actually very easy to track. If there is user interest in it, I can absolutely incorporate it into Newsdex.
I’ll take that challenge Chuck.
If I was looking to build the tool to support the sort of social growth that I’m talking about, here’s the framework.
1. Site/section/story/topic/author breakdowns – Since the goal is to give editors and writers tools to that track and support conversations post-publish, each person would need a page that would act as their jumping off point (dashboard). This information could be gathered relatively easily through the XML sitemaps that most sites are generating for Google to help with their SEO.
2. Social mentions – Tracking social mentions or conversations likely starts with the page url. Since the convention is to link back to originating content. Each network has its own methods to get this information. Some of the more popular ones:
a. Facebook – The same API that I use to get fan counts can also reach deeper into the analytics to grab page-level insights. Specifically the links.getStats function https://developers.facebook.com/docs/reference/rest/links.getStats/
b. Twitter – Twitter itself doesn’t provide link-level analytics, and make life harder through the extensive use of link shorteners. Backtweets (http://backtweets.com/) fills in this gap, allowing you to have link-level analytics and expand links.
c. Comments – For sites that use custom-built or built-in comment systems they’d need to figure out their own system. However, for those who use the awesome Disqus tool, they provide page-level analytics (http://disqus.com/api/docs/threads/details/) for each conversation.
d. Blog/site mentions – Since this is the oldest method of sharing content online, more services have been built up to track this. Including a backlinks API from Google, Linkback/Pingback/Refback standards, or create your own like UrbanSpoon’s Spoonback (http://www.urbanspoon.com/e/restaurant_link/2328).
4. Fertile soil (where was the seed planted) – Where the content was seeded matters…a story that was placed at the bottom of a subsection and never placed into any external networks likely was never given a chance to grow. So, tracking where a piece of content is placed (section index, primary twitter account, email newsletter, etc) will add context. Likewise, and article given a placement at the top of the front page that doesn’t elicit any conversations also tells an important story. This data would need to originate from the CMS itself and the tools used to post the content onto social media sites.
5. Harvest – All of this should feed back into the story itself. So tracking the views of the the story by different sources, and different audience categories (loyalist, drive-by, etc) is the last piece of the puzzle. Site tracking software like Google Analytics, Chartbeat, or Omniture could provide that data.
The last piece isn’t a tool, but a mindset. Traditionally, we’ve had a large wall between the folks who produce content and those that consume it. We’ve built up mechanisms to cope with that, relying on the prophets of the newsroom who we rely on to foresee what is important or interesting to the community. Then building up mythologies regarding what the “typical” reader is like (consuming raw agate or avoiding all but easiest to consume Pippa Middleton gossip). We now have the opportunity to see what our audience is like, as well as what they want or need. Technology can help with managing the scale of this new information, but new workflows/mindsets need to be built up to take advantage of this trend.
We have found Backtweets to be very useful for tracking shared content. With all the tools out there, it is difficult to zero in on the best combination, but it is important that you do put together a system that gets you the information you need to track and adjust for success.
So Gary, as the geek here, if the APIs are robust enough, are we talking about being able to track links in tweets — who clicks on them and who re-tweets them, and the relationships between those people? (Using Twitter as an easy example.) Do I then get to see a tree showing how my site’s content spreads from person to person? Maybe identify loyal sharers?
How might I act on that information? (I guess that’s my thing to figure out. I’ll give it some thought.)
I’m trying to imagine a concrete use case.
I’m starting to think that social metrics may be looking in the wrong direction. I think we’re appropriately dissatisfied with measurements of “influence” (Chuck made a good point about influence having very little to do with size) and “engagement” (a conversation about the writer’s yoga class hardly benefits the community).
Instead of looking at those macro properties, what if social measurements started at the story or section level. Perhaps having metrics answer the question, “how many conversations did you provoke today?”
If you were able to get a measurement of how many times your story was mentioned, discussed, added onto, etc via comments, twitter, facebook, blogs, emails sent, and pageviews, you might be able to get a sense of whether you’re activating the community, rather than just just informing them.
Scout Analytics of Seattle (scoutanalytics.com) is doing some groundbreaking work to understand the revenue implications of audience engagement with digital content. Matt Shanahan of Scout is featured in “The Story So Far: What We Know About the Business of Digital Journalism,” a new report from the Columbia University Graduate School of Journalism.
http://www.cjr.org/the_business_of_digital_journalism/executive_summary.php
Gary’s observations about social media would seem to provide yet another level of insight to Scout’s analysis of what people do while they are on a site. I wonder if Scout has explored this.
It seems like tools are a big part of the change. If we can access them, afford them and understand how, and why to use them. Their potential for technology to help us get smarter how to build and convey the value of content products is tremendous. With engagement as the sweet spot. But the rules of engagement have changed, with dozens of ways the term is applied. Joy Mayer, at the Reynolds Journalism Institute, did an entire fellowship year on the subject of engagement. She’s compiled and posted on RJI’s site a ton of information, research, writing and articles for anyone who is curious.
Margaret Francis is another thought-leader on the for-profit side, her business is social measurement, typically for big brands. She works for an outfit called Scout Labs, and offers smart tips like “measure strategy, not stuff.” She’s also a big advocate of key performance indicators (KPIs). The thrust of this is that by better understanding performance goals, what is measured targets what’s most important to an organization’s success.
With the amazing advances around how much we can measure, at the center of this is what we measure. Might good tools get a bad rap due to measuring the wrong things? How do news producers keep on top of what the range of “right” things are? Are there tools out there that are helping us measure smarter?
This kind of business intelligence gathering is often beyond the limited budget and bandwidth of smaller news sites, many of them operating as non-profits. Are there are less expensive network influence mapping tools that others have tried? Gary and Chuck do you know of any new tools that are emerging on the market?
I’d love to bring this conversation in a slightly different direction. I agree 100% with Chuck that measuring “influence” via social reach is a bit silly. It reminds me of the absurdity of media sales teams who show powerpoint slides of their 100K+ audience levels, while selling advertisers a campaign of 10K impressions.
What interests me more is how social media provides a scope to peek into peer-to-peer communications outside of the walls of a site. We have always been aware of our role as thought-leaders, but never have we been able to see those thoughts spread from person to person.
This is where I think Jonah Peretti (the founder of Buzzfeed) has it right. The language he uses is horticultural. Media organizations have been focused on planting a seed of good content in fertile soil, like a well trafficked website with a natural audience. Typically, that has been the end of it. The better organizations work to plant more seeds, or prepare better soil.
However, a farmer knows that is not the end of the line. Once the plant starts to grow, it needs to be maintained. Thinning crops where there is too much competition, cutting back bad growths, providing the right fertilizer throughout its life. The same is true of content, where a media organization now has a responsibility to help the content flourish post-publish. Social media is part of that.
I’m curious what sorts of tools could help. Potentially a system that watches where the conversation is developing (internally with comments, externally with Facebook, Twitter, blogs, etc). Or maybe a tool that helps find the most fertile soil, suggesting the appropriate place to spark a conversation. Maybe it is simply a mater of priorities/mindset at and organization, and tools are just a minor part of that change.
Gary alluded to the challenge of determining who should be included in the measurement metric. Corporate social-media accounts, certainly, but what about individuals who control their own social accounts but also speak officially on behalf of their employer? At Newsdex, we decided, it depends. http://seattle.newsdex.us/about.php
Here is what’s different today (less than you might think):
Used to be a publication either had subscribers or not. Broadcast outlets presented a bigger challenge in measuring influence because only God knew for certain who was listening or watching and for how long and what. For that matter, newspaper advertisers, the saying went, knew half of their ad dollars were effective, but they didn’t know which half. A newspaper delivered is not necessarily a newspaper read.
With the Internet, the metrics are hard facts, like print circulation, but often stupid ones, albeit less so than print circulation. We can use tools to see who clicks on a link, whether it’s on a website or a Twitter tweet. But it’s still very difficult to measure the influence. News can be disseminated and a brand can be promoted without having the end user click or leave a trail. So even though I follow @seattletimes on Twitter, it doesn’t mean @seattletimes is influencing me any more than any of the other 209 accounts I follow, even if I click through once in a while.
In short, online we can get closer to measuring media influence, and we have more ways to do it and more precise data, but we’re still a long way from making those metrics unambiguous.
Gary,
Thanks for these observations. I think the currency of engagement may be the pathway to economic viability in the online media world. I’ve heard this called the Return On Connection. NewsDex is bringing new insight to the calculation.
There is so much that news media people need to learn about the value proposition they offer. This is as true for non-profit and public media as for for-profit companies.
As one of the co-creators of Newsdex, I appreciate the positive reaction to it in the Seattle community.
It has been interesting to watch the usage of the site. I originally saw two main uses of it:
1. To allow local businesses to track their own growth in the social media space, instead of using an intern and spreadsheet (or even worse, not tracking it at all).
2. To look across different media organizations picking out trends. For instance, most newspapers rely most heavily on Twitter, while broadcast has a heavier Facebook presence.
What took me a bit by surprise was the usage of the site as a sort of ComScore or Nielsen measurement service for local social media reach. Although, I was surprised by that usage, it makes sense. Right now, local media have lacked tools in the social space that allows for apples-to-apples comparisons across organizations. Coming up with the rules-of-thumb for what to include or not include (http://seattle.newsdex.us/about.php) was actually far more challenging than the technical work of the site.
While Newsdex is (hopefully) a nice service to check the size of networks across multiple media companies, there are many great services that allow you to optimize a single social media account…HootSuite (http://hootsuite.com/) and Radian 6 (http://www.radian6.com/) come to mind.
I should mention that the technology behind Newsdex is also available on a white-label basis for any company that it looking to track its own network (internal or competitive).
Lastly, I’m most curious what measures of success media outlets are using related to social media. Size of network? Engagement? Referrals? Positive/negative brand mentions? Reach of message? I’m looking towards technology like the nytimes skunkworks project (http://www.niemanlab.org/2011/04/the-new-york-times-rd-lab-has-built-a-tool-that-explores-the-life-stories-take-in-the-social-space/) that use social media to examine how the conversation flows outside of their walls.