Background

Many funders and storytellers work to shift the cultural narratives that condition public understanding and constrain institutional action on issues like racial justice, immigration, or climate. Sometimes this work lacks a shared understanding of what narrative is, how to identify and characterize it reliably, and how it connects to people with the power to make change.

Harmony Labs’ Narrative Observatory aims to solve some of the technical, definitional, and practical challenges bedeviling narrative and cultural strategy. The Narrative Observatory provides narrative and cultural strategists tools, like this website, along with industry-grade data infrastructure to understand audiences relative to their place in culture; to identify, measure, and track narratives within audiences over long time scales; and to surface audience-specific story opportunities and threats.

With funding from Bill & Melinda Gates Foundation, the Narrative Observatory’s first iteration has focused on poverty and economic mobility in the U.S. We are in the process of expanding this work to additional issue and partners and would like to have a conversation with you about supporting your work.

Methods & Data

As with all the tools we create, this website strives to present as simply as possible only what is useful. Keeping our findings short and sweet is really hard for us, because we’re data nerds. Behind everything you see here is some fancy data dancing, which we’re eager to share.

To create our audience classification, we started by reviewing existing surveys (like this one from GOOD) and reading media related to poverty, economic mobility, COVID, Black Lives Matter, and other major events in 2020. This helped inform survey questions exploring attitudes on race, gender, place, and class, along with core values. Using these questions, we conducted a 2,900 respondent, voter-file matched survey and analyzed results to yield four audiences and generate an 8-question audience classifier.

We relied on demographic details from the survey (e.g., age, gender, race, zip code) to create predictive models and project these audiences onto nationally representative, opt-in media consumption panels. These media consumption panels give us visibility into the minute-by-minute media behaviors—of over 300,000 people in the U.S., who opted in and are compensated for their participation—across desktop, mobile, tablet, and TV. This allowed us to enrich our audience profiles with the media artifacts—videos, news articles, TV shows, Tweets, Pinterest pins—that actual audience members actually engaged with.

Next, we worked to extract from all the media audiences engaged with only artifacts relevant to poverty and economic mobility. We did this using keywords and human annotation to build supervised relevance models for each media type. We considered a media artifact poverty-relevant, if it told us what people experiencing poverty or financial wellbeing are like (e.g., hard-working, virtuous, impure); what it is or feels like to experience poverty or financial wellbeing; or how people go from poverty to financial wellbeing or vice versa.

Then, our human analysts—representing each of our audiences—read a randomly generated poverty-relevant sample of each media type, in order to extract key dimensions for story pattern variations. At the same time, we used machines and natural language processing to capture and cluster naturally occurring story patterns. The outputs of both human and machine analyses were used to generate a preliminary narrative structure for news, TV, and Twitter. This structure was used by another team of human annotators to code another sample of poverty-relevant news articles, TV transcripts, and tweets. We built supervised narrative models from these annotations, which predict which narratives each article, tweet, or song is associated with.

We subjected relevant media artifacts to an additional layer of qualitative analysis to surface important story opportunities, threats, and strategically important features, which we share in this site’s Stories & Opportunities section.

All the media data in the Narrative Observatory come from commercial partners, detailed in the Partners section below, who donate their data to this work. As a 501(c)3 organization, bound to serve the public good, Harmony Labs has adopted a set of principles and practices around data to: ensure we only gather, use, and store data that supports our mission; anonymize at ingest any data that contains personally identifiable information; maintain robust security, limited access, and encryption; and actively work with our partners and in-house data science team to adhere to the highest standards for scientific integrity, clearly communicating methods, assumptions, and practices.

In this site’s charts and graphs, percentages may not add up to 100 due to rounding.

About Harmony Labs

Harmony Labs has been supporting narrative and cultural strategy for more than a decade, helping storytellers channel the immense power of story to shape the future. One of the first papers we co-authored looked at fracking narratives in documentary film. Since then, we’ve worked on narratives for climate, gun violence, political corruption, artificial intelligence, reproductive rights, and other issues. With the Narrative Observatory, for the first time ever, we’re harnessing powerful industry relationships and an academic research network to develop data infrastructure purpose-built for narrative identification and tracking over long time scales, across media types.

Harmony Labs builds communities and tools to reform and transform media systems. Our mission is to create a world where media systems support democratic culture and healthy, happy people.

Email us to start a conversation. Follow us @harmonylabs on Twitter and LinkedIn, and Medium.

FAQ

Below are answers to some of the most common questions about the Narrative Observatory. Please reach out to us with your questions.




Isn’t the Narrative Observatory just another social listening tool?expand_more

The Narrative Observatory differs from other social listening tools insofar as it centers audience, the actual lived media experience and behaviors of actual people. At the core of the Narrative Observatory’s data infrastructure are nationally representative audience panels from Nielsen and Comscore that offer visibility into what over 300,000 people in the U.S. are actually doing online and on TV across desktop, mobile, and other devices.

Putting these audience panels at the heart of what we do means we’re not just counting tweets, hashtags, articles produced, or bot traffic. Nor are we relying on self-reporting from surveys to know which media to pay attention to. We’re only counting and analyzing content that actual people are actually engaging with. Which is what sets us apart from many media listening tools on the market, and a lot of narrative and media research too.

To those audience panel data, we join terabytes of content data: TV transcripts, online news articles, song lyrics, YouTube video transcripts, and more. These data allow us to look across media types for patterns in the actual content people are choosing to consume. We use machine learning models to help us with at scale content analysis, of course, but the training data for these models are always generated by actual audience members reading and making judgements about content. Judgements like: is this content about that issue? Or does this content represent that narrative? This is a time consuming, expensive process, but, through trial and error, we’ve found it to be the only way to produce meaningful, actionable, accurate results.

How does your work serve the public good?expand_more

Media systems have evolved to be outrage machines, sorting machines that are reproducing and reifying division and difference, in ways that are starting to pose a real threat to the conduct of civic life, in the U.S. and elsewhere. We think of the Narrative Observatory as a tool that helps people better navigate today’s media minefield and, maybe over time, heal some of these real or perceived divisions, making the work of advocacy much easier. Specifically, we see this work as serving three communities: content creators, advocacy organizations, and philanthropic funders. People making content to support cultural or narrative strategy often don’t have the time or people power to do much more than a little anecdotal desk research on audience, story, or narrative and are often looking for off-the-shelf frameworks, insights, and validated hypotheses that can help them do their work more effectively. Advocacy organizations often have national segmentations that capture how people relate to their issue, but sometimes lack the nuanced cultural understanding that can help them make or distribute strong content. Philanthropic funders working on social issues at scale are bound to be using media as part of their strategy, and they need to know whether their media investments are positively affecting public conversation. We designed the Narrative Observatory to address these needs.

Where does data for the Narrative Observatory come from?expand_more

All the media data in the Narrative Observatory come from commercial partners, detailed in the Partners section above, who donate their data to this work. This community of data philanthropists includes big companies, like Nielsen and Comscore, and also smaller startups, like Peakmetrics, that are scraping different corners of the web, news, television, radio, and other kinds of media. We are grateful to them for supporting the Narrative Observatory. We also make these data available to a network of mostly academic researchers, who use them to better understand how media systems work, and how to improve them.

What are your data practices with respect to privacy and security?expand_more

As a 501(c)3 organization, bound to serve the public good, Harmony Labs has adopted a set of principles and practices around data to: ensure we only gather, use, and store data that supports our mission; anonymize at ingest any data that contains personally identifiable information; maintain robust security, limited access, and encryption; and actively work with our partners and in-house data science team to adhere to the highest standards for scientific integrity, clearly communicating methods, assumptions, and practices. In general, we adhere to the principle of least privilege. Which means that the Harmony Labs’ team, partner, and technical infrastructure are only given access to the resources and permissions necessary to complete pre-specified goals. Even our applications and supporting software are only able to interact with specific services and data, limiting unintentional cross-contamination and spillage vectors. Annual independent security audits—most recently completed in December 2020—validate our commitment to data security and strong access controls.

Is data from the Narrative Observatory open and accessible?expand_more

Data donated to this project and to Harmony Labs are available to participants in our research network. The terms and conditions for data availability derive from agreements with our data partners. We are always happy to share our data and methods with others and have tried our best to develop the Narrative Observatory in a way that is open, accessible, and well documented on our Medium page.