What gets measured and what doesn't get measured
MAY 10TH, 2021
TOPIC: What gets measured and what doesn’t get measured
BACKGROUND: An expert roundtable explores measurement
LISTEN: On the web, Apple, Spotify, RSSWATCH: YouTube
Last month, a Pew Research survey methodologist, a rabbi, a Tlingit tribal leader, an Islamic scholar, an entrepreneur, a sociologist, a law professor, a journalist, an investor, and a BBC researcher walked into a virtual room for a profound conversation about how our lives are measured.
The group explored five questions:
- What kinds of values do we measure? What don’t we?
- Who decides what gets measured?
- What is the impact of being measured?
- How do we measure a value well?
- What are values we could be measuring but are not?
The outcome was an extraordinary and deep discussion representing many points of view. Listen on the web, Apple, or Spotify. Watch on YouTube. An edited transcript featuring one key idea from each speaker is below.
YANCEY: For most of human history people have been guided by values. Moral notions of right and wrong. What’s beautiful and what’s not. What’s just and what’s unjust. In recent years we’ve been ruled by value. Numerical expressions of what’s valuable, what’s not, and to what degree. It’s turned out that Value is a lot more convenient than Values.
Right now a lot of what we think of as values are being translated into value. Dimensions of human existence are being measured in fundamentally new ways. This is what we’ll discuss today.
Morgan X’agatkeen Howard is from the Tlingit nation. Born and raised in Alaska, his early childhood took place in the small village of Yakutat. He currently serves on the Board of Directors for Sealaska Corporation.
MORGAN: The questions you’re posing today are maybe more important than the answers we come up with. We’re going through life constantly measuring and taking those results and making decisions because of them. We can’t help as humans but to do that. We’re constantly evaluating, we’re constantly posing questions and looking for results. But not as many people are asking if we’re measuring the right things.
I try to think in the ways of my ancestors and use the same thinking that they had, which was our concept of Haa Shagoon, which is the idea that we look equally at both the past, present, and future. So we consider what our ancestors thought of as important, what we think of as important today to make our decisions for today’s population, but also for our children, our grandchildren, and use that in our decision making. How does it benefit those three groups of people? It’s long term thinking for sure.
Recently there was a carbon credit sale in the southern United States to not cut some trees down through this carbon credit exchange. The most recent sale came about because a company got better at measuring. So now they’re getting paid for not doing something. So how do we measure not doing something? There’s an important value we don’t put much importance on: not doing something. We’ve always got to be doing something. Maybe progress was that I decided not to do anything.
Salome Viljoen is a NYU post-doc and a soon-to-be Columbia University School of Law Academic Fellow and Lecturer.
SALOME: What we measure is a reflection of our larger economic and political priorities. That often means that we measure what’s profitable to measure. More precisely, when we’re talking about those collective priorities, it’s not really our collective priorities — it’s the priorities of those looking to maximize profit who already hold power.
A lot of my thinking as a legal scholar focuses on developing legal institutions to recognize and enforce more priorities in what we measure and how we govern what’s measured. That changes the kinds of measurements you see. One thing that’s measured are the things that we legally mandate to measure. That can be especially useful when we’re talking about measuring things that it’s not in the self-interest of someone, or that might be inconvenient, cumbersome, or might shift the measurement priorities away from just reflecting their own priorities and actually reflecting a far greater set of priorities. How we govern and mandate measurement can be really useful.
Shannon Arvizu, Ph.D. is a sociologist and Founder at Epic Teams, an organizational performance consultancy with an initiative called Citizen Dashboard to help governments measure and improve citizen outcomes.
SHANNON: We don’t measure things that we don’t want to be on the hook for. With the COVID pandemic we had this plethora of measurement of data. But every state, county, city had their own way of measuring and defining those measures. What actually is a COVID death or what actually is a COVID infection? It took a journalistic entity like The Atlantic to create the default way of measuring COVID stats. We tend to not measure things that are too cumbersome, costly, or require too much coordination.
Ashley Amaya is a Senior survey methodologist at Pew Research Center who advises projects involving address-based sampling, multiple modes of data collection, and surveys of rare populations, with doctorates from the University of Maryland and a master’s degree from the University of Michigan, both in survey methodology.
ASHLEY: We’ve talked about self identity, an abstract concept. So how do you ask somebody: what’s your self identity, and then actually put all of the different people’s perceptions together to tell a story and measure something? One of the things that we try to measure is social integration. How tied to your community are you? We can ask questions and operationalize that: do you belong to the PTA? Do you play a group sport? Do you talk to your neighbors? Do you have family dinners? If we don’t operationalize that correctly, then we’re not measuring the right thing.
The other things that are harder are sensitive things. People are happy to talk to us, but they might not be happy to answer that specific question, or provide that specific data point. Or they might flat out lie. For example, people tell us they vote, even though they did not vote. People love to tell us they have a library card. They want to project an image that makes them seem more likely to be liked. So whether or not we’re getting a true answer or not is really problematic.
That’s migrated us from using surveys. Stop asking people when they can lie or not answer and start using big data or alternative data. That’s great, except that gives us a false sense of security that we’re getting a better product, and that’s not necessarily true. There’s error in those big data too.
Marlon Wayne is cofounder of Impulse, a social polling app that calls itself the Nielsen’s for the TikTok generation.
MARLON: When I look at traditional surveys, at least for Gen Z, I just don’t want to fill that out. I’m not incentivized to give my opinion or my thoughts. Yet I have a lot of thoughts on this thing and I’d love to share them, but I don’t want it to feel like it’s a waste of my time. I want to feel like there’s a high value add. We do a few things with Impulse to extrinsically align the incentives in order to make that the best possible outcome. April 20th just passed, which was Weed Week. We were asking some pretty sensitive questions. But because we’ve built a lot of trust over time, people have no trouble answering them. In fact, joking about them and having fun with it. It’s been enlightening to learn how people share when they feel comfortable. Still, we’ve definitely seen a lot more skips on certain questions than we typically do. That shows there’s something about being measured or being asked in an official way that triggers something for us.
Alex Varley-Winter is a UK-based investigative journalist with a focus on finance and climate.
ALEX: When you were talking about fire it reminded me immediately of an amazing documentary called Position Among the Stars, which follows three generations of an Indonesian family. Towards the end of that, one of the younger generation goes to the woman in the village, her grandmother, bringing her a stove cooker, saying you don’t have to collect wood every day, now you can cook on this. She gets really depressed because it doesn’t cost her money to collect wood every day, it’s much more efficient to do it that way. It’s a divide between the urban thinker and the rural thinker, and it made me think about how women’s labor isn’t counted. We don’t measure women’s needs and women’s labor. Iceland had a women’s strike decades ago. All the women went on strike, and it forced people to measure women’s labor and what we contribute. Not something that gets measured properly.
Pauline Pigott is a senior consultant at Deloitte Sustainability France.
PAULINE: Companies are very used to measuring their profits. They’re very advanced on the financial aspects, but when we start to look at other other impacts that they have, like social impacts or environmental impacts, we find multiple ways of measuring these, but it’s very hard to link them to profitability. What we’re trying to work on right now is the triple bottom line accounting framework, the design methodology where you take the accounting framework and apply it to environmental and social impacts. This provides you with a picture of the present and also helps you better understand the future.
Esther Dyson is a tech journalist, investor, and Executive Founder of Wellville.
ESTHER: As we measure things, we’re going more and more towards real time measurement. In some ways, that’s really great. But what it means is we’re not measuring the long term. Not only are we not measuring it, we’re not thinking about it. We tend to be too reactive, whether it’s in the stock market, or, “Oh my gosh, my child’s score is X.” We’re confusing precision with meaning.
Irwin Kula is a Rabbi, author, and Director of the National Jewish Center for Learning and Leadership.
IRWIN: There’s a dance between the measurable and the unmeasurable. Morgan’s incredible insight of how do you measure ancestry? If someone is a seventh generation rabbi, like I am, how do I measure the value of that? Everybody introduces me that way, but what does that mean? It’s not worth anything, or is it worth something that’s priceless? That’s the dance between the measurable and the unmeasurable.
One of the things my organization is involved in is inviting religious leaders to collaborate on measuring whether the job they actually want to get done is actually getting done. If you imagine that religious practice and being in your congregation is an upgrade of your virtue or the qualities of human flourishing, are you willing to use some of those tools to evaluate whether the ends that you yourself claim are happening? There were a lot of synagogues and a lot of leadership in that synagogue that weren’t interested in actually being measured. People are often scared to actually measure if something works or doesn’t work.
Lianne Kerlin is a Senior Researcher at the BBC leading a multi-year project that seeks to identify and measure core human values.
LIANNE: One of the things you mentioned at the beginning was about self interest, and there’s this idea of self preservation as well. You have this narrative about your identity, say that I’m a healthy person. To preserve that identity you’ll probably discard measurements that challenge that idea. That self-interest in preservation happens on every single scale. The things we end up measuring becoming proxies for things that fit in with our narrative — I’m a good organization, I deliver great content, I’m a great healthy person — but we don’t really have the holistic view.
Übeyd Ruff is a Technology Transfer Officer at Istanbul Sabahattin Zaim University, Editor at the geopolitical thinktank the Center for Islam and Global Affairs, and policy advisor for the Islamic Reporting Initiative, an award-winning NGO that promotes corporate sustainability and social responsibility (CSR) based on Islamic values.
ÜBEYD: Historically in nations with Muslims and other minorities, when Western powers come into places they brought their definitions and their measurements with them, and their motivations were not good. Now when these Western powers who have evolved, repented in some way, are trying to repair the societies that they trashed with the Sustainable Development Goals, you have to expect that the local population is a little suspicious. “Here we go again, the UN is telling us to use these measures.” My task as a policy advisor for the Islamic Reporting Initiative was to find a way for Muslim nations to understand that this is not an attempt to re-colonize, that this is not evil. The methodology we used was to show that Islamic civilization was sustainable. There was a golden age, and within Islamic scriptures in the Quran, and the Hadith, and numerous Translations by many scholars, all of the Sustainable Development Goals are there. We found scholars to explain to people basically at street level that this is not new. The Sustainable Development Goals are in the Quran, and we can actually teach the UN how to do this better. If you look at social media four years ago, no one in the Muslim community was talking about the Sustainable Development Goals. After our efforts now, when you get on Facebook and LinkedIn everyone is writing papers about this Islam and the Sustainable Development Goals. We’ve changed the conversation and it’s been a success because we used the local culture, the local context, to show that this is not new.
Listen to the full conversation on the web, Apple, or Spotify. Watch on YouTube.
Panelists
- Ashley Amaya (Senior survey methodologist at Pew Research Center)
- Shannon Arvizu, Ph.D. (Sociologist and Founder at Epic Teams)
- Esther Dyson (Tech journalist, investor, and Founder of Wellville)
- Morgan X’agatkeen Howard (Tlingit nation, Board of Directors for Sealaska)
- Lianne Kerlin (Senior Researcher at the BBC)
- Irwin Kula (Rabbi, Director of the National Jewish Center for Learning and Leadership)
- Pauline Pigott (Senior consultant, Deloitte Sustainability France)
- Übeyd Ruff (Technology Transfer Officer at Istanbul Sabahattin Zaim University, Editor at the Center for Islam and Global Affairs)
- Salome Viljoen (Soon-to-be Columbia University School of Law Academic Fellow and Lecturer)
- Alex Varley-Winter (UK-based investigative journalist)
- Marlon Wayne (Cofounder of Impulse)
Further Reading
The Ideaspace
Yancey Strickler's private desktop