Early Toothbrush testing machine at Consumer Reports

As our lives become more and more shaped by digital technologies, how can consumer protection organizations work together with the public for safety and fairness, supported by evidence?

Today at Cornell’s Public Interest Technology Community of Practice, we were joined by Ben Moskovitz, Leah Fishman, and Ginny Fahs of the Consumer Reports Digital Lab. Consumer Reports is an independent nonprofit working to make the marketplace more safe and fair for consumers since 1936. CR conducts independent, rigorous research and uses its consumer insights and policy expertise to inform people’s purchase decisions, improve products and services that businesses deliver, and drive regulatory and fair competitive prices. Within Consumer Reports, The Digital Lab investigates existing and emerging consumer harms based on interactions with and use of different technologies, and creates new solutions and services to help consumers be better protected.

In this discussion, we spoke with Consumer Reports’ digital team about these choices. How has a vision of consumer advocacy transmitted to digital and digitized products? How is data used to manipulate behaviors, and exploit cognitive biases? How do consumers understand and react to automated decision-making by opaque algorithms, which disproportionately harm low-income consumers and communities of color?

Consumer Reports Origins & Theory of Change

Ben Moskowitz starts out by telling us about Colson Warne, the co-founder of Consumer Reports (then Consumer Union). In the 1930s, Consumer Reports was a big believer in markets, and they held the view that an uninformed decision is not a free decision. Today, Consumer Reports is an independent non-profit with six million members based in Yonkers New York. These people form the heart of Consumer Reports and are key to its power in DC. Consumer Reports also reaches more than 20 million consumers every month with media about products that are good or unsafe, as well as systemic issues. Consumer Reports has the world’s largest independent product testing operation. They test a wide range of consumer products from cars to stoves to bike helmets and robot vacuums. And this research by Consumer Reports has led to product safety and policy changes over the years. Their research contributed to seatbelt laws, car safety, and regulation of food safety.

In the past, Consumer Reports was focused on people as consumers. In 2022, we’re now all “users” as well, which means we need to ask not just if something works, but also now  how companies and products treat us as part of an interactive system— how do they use our data, do they ask for consent, what influence do they have on people’s safety? Ben tells us about products that stop working if we stop paying, about digital toll-booths for using digital services, for subscriptions that trick people into ongoing fees.

How can we create change when products are unsafe or unfair? Ben tells us that the consumer movement has tended to focus on three directions. By organizing consumers, it’s possible to influence the demand side. This sometimes enables Consumer Reports to influence companies directly. When that doesn’t make sufficient headway, Consumer Reports works on regulation. Work by Consumer Reports to support consumers has a real impact. Ben shows us data from 2017 suggesting that out of 17.5 million cars purchased in the US, more than 3 million of those purchase decisions are linked with information from Consumer Reports. For example, CR ratings and reporting often make the news, and can even affect companies’ stock prices–Tesla dropped from number one in their electric car ratings to seven or eight, it was a major news item; a bellwether that Tesla isn’t the only game in town when it comes to high quality electric cars.

The Challenge of Testing Digital Products

How might this model of consumer power make a difference in our digital world? To extend their work to digital products, Consumer Reports contributed to the Digital Standard of indicators for rating digital products: are they secure, private, fair, etc? Every internet product that Consumer Reports tests now goes through this basic suite and is incorporated into the final ratings. That means that when Consumer Reports evaluates a blender, a car, or digital financial services, they can rate its digital aspects.

Rating companies for their privacy or security is hard to do, especially when companies are not forthcoming about their practices or failures. In Digital Standard based testing, Consumer Reports often has to rely on public statements. And since product software is updated regularly, continuous testing can be difficult and go out of date. Consumer Reports can score a company for what they say they do about bias, for example, but the Digital Standard can’t provide guidance on what they actually do. 

So what should a digital watchdog like CAT Lab or Consumer Reports be doing in 2022 to have maximum impact? What kind of research, reporting, or advocacy is going to have the most impact on consumer behavior and the structure of the market? We need to go beyond ratings. And people’s experiences with complex systems differ— for some people something might work very well, and for other people it could be the worst thing ever, depending on characteristics like race, gender, disability, etc. That’s not true just for digital products, and crash test dummies (anthropomorphic test devices) tend to be white and male, leading to biases in product safety more widely.

What kind of research has the Consumer Reports Digital Lab been doing recently?? Ben tells us about an audit study they conducted to demonstrate that drivers who were less educated could pay up to 16% more for car insurance. This demonstration of a known harm helped CR and partner advocacy groups to get the New Jersey State Senate to pass a bill which would bar insurance companies from using certain non-driving factors in setting insurance premiums. 

In 2020, Consumer Reports did a project to post bad ads and found that Facebook was approving ads with coronavirus information. In another project, the Consumer Reports team analyzed the policies of insurers to find out how the company might use driving data to determine the pricing of auto insurance. In another example, the team looked at the impact of new Amazon Warehouses, finding a huge increase in warehouses in 2020. They found that these warehouses are predominantly being built in communities of color. Working with the Brown Institute at Columbia, they found that these warehouses are going to have disparate impact on communities of color, including pollution, traffic, and income inequality.

With six million members, Consumer Reports also organizes crowd-sourced research. In 2021, Consumer Reports organized people to submit water samples and worked with the Guardian to analyze those water samples for lead and other poisonous chemicals. The study prompted many hearings and policy actions. Consumer Reports also asked people to submit their internet bills for a pricing study — people submitted nearly 30,000 bills from nearly 60,000 participants, along with internet speed tests and a demographic questionnaire. To support this, they also built up a technology backend to do other document reading in the future.

Leah tells us a bit more about the broadband speed test project, which involved a collaboration with mLab. We found that people enjoyed the fact that they were getting information back that had them asking more questions about the Internet service. The process wasn’t laborious. Consumer Reports also contacted people afterward, rather than leaving them hanging. But given that the bills are often confusing, analyzing the data has taken longer than they hoped. They also worked with 50 regional and national partners to develop the project. They realized that the Consumer Reports audience tends to skew toward certain demographics. So in order to meet people where they are at, they needed to link arms with trusted organizations that have communities we weren’t already speaking with. We started with people our advocates in the Hill are working with, along with community organizations at a local level.

In the long term, Consumer Reports is imagining how it might evolve to become a “consumer data union.” If their members can trust them to receive data, or to exercise data rights on their behalf, then Consumer Reports can provide individual insights to people, collective insights to take to policymakers, and in the future carry out collective bargaining on behalf of its members. For example, people could connect a device to their network, Consumer Reports could provide people with information about the security of the devices on the network, and also spot problems with networks across the country. Through consumer data rights, members of the data union might authorize CR to retrieve and analyze information on their behalf. This might include data about purchase histories, product preferences, bills or financial transactions, areas where a trusted non-profit might provide better advice or ways to save money.. 

Ben concludes by telling us that The Consumer Reports Digital Lab is often hiring and offers fellowships and other opportunities for researchers: https://digital-lab.consumerreports.org/