Looking up from the bottom of a spiral staircase, steps twist up. In the center of the spiral is a sky blue octagon painted on the ceiling.
photo by Valerian Guillot: https://www.flickr.com/photos/valerian1985/48561016036

How can communities create lasting change on the most pressing challenges in online behavior, from navigating racism to reducing burnout among those who moderate social conversations?

Are we doomed to react to what can feel like a growing avalanche of social problems, or might it be possible to create positive cascades that have lasting benefits for communities over time?

This summer, CAT Lab convened a group of community leaders from the Reddit platform to discuss ideas for research that could make a difference in communities and also advance basic science. In all of our Community Research Summits, we discuss the goals that leaders have for their communities, the ways they might measure those goals, and ideas for interventions that could help their communities achieve those goals.

As researchers, we listen, take notes, and make connections between community goals and scientific questions. We summarize these possibilities into “Citizens Agendas” that help CAT Lab and our scientific collaborators identify possible research partnerships with communities. Those seeds of ideas eventually grow into new studies like our recent collaboration on harassment prevention across multiple communities.

How might we think differently if we focused on creating lasting positive change rather than just responding to problems?

At our summer meeting, we gathered with leaders who facilitate broad online conversations about knowledge and culture. Many moderators reflected on the challenges of building community on software platforms that are largely focused on reacting to problems rather than cultivating the behaviors and virtues communities aspire to. 

How might we think differently about online community leadership online if we focused on creating lasting change rather than just responding to problems? Our conversations this summer led the CAT Lab team to think about an idea from social psychology called “Recursive change.” In this approach, organizations focus on early changes that have cumulative effects over time. A famous example includes changing the details of college orientation, which can increase graduation rates by changing how students start their educational career. After the Research Summer, we’re thinking about similar ideas that could cause cascading benefits for individuals and communities over time.

Challenges for Communities focused on Knowledge, Culture, and Politics

The communities that gathered at our workshop were primarily organized around sharing knowledge, culture, or politics. Some groups organize spaces for the public to ask questions and receive in-depth answers that are so well crafted that people link back to them for years. Some community leaders host discussions for people in marginalized groups to meet each other, find their voices, and share in a common culture. A third group seeks to facilitate civil discussions about contentious political topics.

All three groups struggle with a two-sided challenge. On one hand, a small number of highly committed people participate regularly to uphold the conversation. On the other hand, a large number of not-always self-aware people join the conversation to learn (and sometimes argue) about the community’s topic of focus. Conflicts and misunderstandings can drive away participants that leaders want to include—at the same time exhausting the community’s core experts and causing them to burn out.

Establishing Norms in conversations about culture: Moderators of communities of color work to create internal conversation spaces to share culture (what Communication scholars call “Counterpublics“)—as well as public facing spaces for cultural exchange with the much larger White population on Reddit. These communities face regular attacks from organized harassment campaigns, as well as influxes of attention when injustice is reported in the news.

Communities had some success using automated systems to handle these influxes of attention. Yet they also wonder why White allies are less active or visible participants—either to join the conversation or set norms around harassment. These findings are consistent with research elsewhere about White people avoiding conversations about race. In initial interviews, White allies have explained their silence through worries about saying the wrong thing, which may leave them feeling like they aren’t living up to their own values.

Supporting trustworthy learning spaces online requires constant attention and active moderation

Preventing burnout among moderators: One outcome when participants don’t speak up to make norms clear, is that community moderators may face a greater burden to review and enforce a community’s policies. This can lead moderators to burn out over time— an issue we have studied with Wikipedia communities.

All of the leaders mentioned the challenge of managing burnout. Supporting trustworthy learning spaces requires constant attention and active moderation, such as manually approving all questions and removing incorrect answers to questions. During large Q&A conversations, moderators work to protect guests from inundations of discriminatory comments (for example, when women scientists want to talk about the experiences of women in science). Insufficient support makes this workload difficult to manage, especially when recruiting new moderators is hard to manage while fighting so many fires. 

Community leaders also worry that strict moderation could maintain quality at the expense of inclusion, while failing to prevent problems. Strict, well-enforced rules have the advantage of ensuring that discussions in communities are high quality and trustworthy. However, they also raise the barrier of entry and discourage contributions, even from people who have the ability to provide high quality contributions. This intuition from moderators is consistent with research findings in Wiki communities, where high barriers to entry primarily deter high-quality contributions. Even worse—while removals, temporary, and permanent bans may be immediate fixes to rule violations, they don’t scaffold future good behavior.

A spiral sculpture in the shape of a flower, made from paper, entitled "Event Horizon," by Philip Chapman-Bell.
“Event Horizon” by Philip Chapman-Bell (CC BY-NC-SA 2.0)

Prompting Long-Term Improvements to Online Communities

After community leaders shared common challenges, we organized breakout groups to imagine measurable interventions for change—using CAT Lab’s experiment cards. Those cards become the basis of a longer conversation about possible research collaborations. Here’s what we discussed, with acknowledgments to moderators who were willing to share their usernames:

Encouraging quality contributions over time

Contributors: u/TheYellowRose, a moderator of a subreddit that discusses politics

Goal: Increase good faith contributions to communities

For culture and learning-focused communities, growth in participation is a good thing—it enables more people to learn while also expanding a community’s knowledge resources. But communities where people turn for knowledge also need that participation to be informed. Anecdotally, many learners mis-estimate their expertise, either under-valuing or over-valuing it.

Moderators were interested in learning how they might influence observers into active, good-faith learners and contributors, while decreasing forms of participation that are disruptive.

Interventions:

Moderators discussed interventions that CAT Lab has tested with other communities—making community norms more visible to newcomers. There was also a question about whether or not the content and framing of rules or guidelines could impact how people respond to them. Could moderators adjust rules to see how the rules impact participation?

Another moment when people encounter community norms is enforcement. Community leaders questioned whether or not particular approaches to enforcing rules would work better than others—for example, are public or private removal reasons more effective? Does it matter if explanations come from a named vs an anonymous moderator? Does the length of the explanation matter? What other ideas might help?

Connecting community questions to the social sciences: Early-stage interventions in a person’s relationship with a community are well-suited to recursive change since a person’s early perceptions and behaviors influence how they are treated by others, and what happens from there. We can imagine a range of interventions related to norms, reputation, and mindsets that might influence both how a newcomer behaves in a specific situation and also their subsequent experiences with the community.

As part of our grant with the Templeton World Charity Foundation, we’re interested to develop studies that test many competing theory-guided interventions to learn which is most effective. We’re exploring the possibility of using adaptive experiments that allow efficient comparison of these interventions. Since communities have the complementary goals of increasing participation and increasing the quality of people’s participation over time, we think this idea might be especially well-suited to adaptive experimentation.

Recruiting and training new moderators

Contributors: u/themeaningofhaste

Goal: recruiting, training, and supporting moderators to sustain flourishing communities

Leading and facilitating online communities requires a complex combination of organizing, diplomacy, judgment, and everyday cleanup. Many people take on this volunteer labor because they care about a community or worry about what would happen without their interventions. Because most people do moderation as volunteers and because it requires challenging social, emotional labor, the people most committed to it are also the most burnt out. 

That’s why community leaders were eager to discuss moderator recruitment, training, and resilience. Healthy communities depend on having large enough moderator teams that are good at cooperating and support the community well. In the workshop we discussed the idea that increasing moderators could reduce the chance that people would become overwhelmed by the workload, burn out and leave the team. Many communities are stuck in a cycle of burnout, held together by a small number of people who have persisted for many years with minimal to no support from the platforms that profit from a community’s work. Within those burnout cycles, leaders can struggle to know how to recruit, review, or train moderators to do well over time.

Intervention:

Toward that goal, we discussed research to understand burnout among moderators, how it changes over time, what kinds of experiences contribute to it, and how burnout might be mitigated by recruiting and training moderators. 

Recruitment practices vary by community. Some groups prefer to recruit people within the community—for example, culture and learning subreddits prefer to recruit from groups that have personal expertise or experience in the topic of the subreddit. And politics discussion groups sometimes need to find people from different political perspectives who can add balance to the community’s leadership while collaborating well with others.

How can communities describe this opportunity in a way that makes the social value and meaning of the role clear, while also being honest about some of the challenges? Leaders also wondered about where to do this outreach: when should they recruit within Reddit versus conducting recruitment drives among cultural and professional groups that aren’t on Reddit?

Once people apply, what application processes could inform community decisions when choosing new moderators? What kinds of experience and expertise should communities base their decisions on? How would communities define the “performance” of a given moderator in order to determine whether their recruitment approaches are successful over time?

Finally, we discussed interventions focused on training and supporting moderators once they commit to serving and are accepted by a community. How can moderators be supported to find meaning in this volunteer work, develop behaviors that encourage healthy communities, and manage the mental health risks of moderation?

Connecting community questions to the social sciences: Moderator recruitment and support connects to many areas of active social science inquiry that could profoundly benefit communities. At CAT Lab, we can see several points of connection that potential collaborators might be interested to explore:

  • Theories about recursive change that could be tested in early-stage interventions toward moderator well-being, resilience, and burnout over time
  • Theories in judgment and decision-making related to moderator selection and moderator training, since one role for moderators is to evaluate potential norm violations
  • Theories on skills, equity, and the relationship among moderation work, technical skills (in Reddit’s systems), and interpersonal skills (in community facilitation/moderation)

Improving first-contact

Contributors: u/Georgy_K_Zhukov

Goal: Improving newcomers’ first interaction with community governance

Community leaders also spoke about ideas for improving the first point of contact between community participants and moderators—something that everyone agreed could have long-term effects. Leaders explained a positive interaction can make users feel welcome in a community, and encourage them to contribute in ways that are both personally fulfilling and contribute to the community. However, most interactions between moderators and users are negative—they occur when a user breaks a rule and a moderator responds with a sanction, such as removing their comment or issuing a warning. How can communities create more welcoming first conversations or improve these “negative” moments of norm enforcement?

Intervention: Interacting with users can be time consuming. To save time, and improve consistency in interactions, Reddit’s moderator tools support the development of “macros”—pre-written messages that any moderator can program for their community and then apply sometimes dozens or even thousands of times a day. Different moderators may have different opinions about the content, and these workflows become entrenched: updating macros may also mean updating moderation approaches. Yet moderators don’t have access to reliable evidence on the effects of their interventions. 

In this group, we discussed experiments that could test these messaging approaches and their effects on the subsequent behavior on new users. We discussed possible outcomes on the quality and frequency of someone’s posts, their chance of violating community norms, and how much they go on to behave in ways that are supportive of others in the community.

Connecting community questions to the social sciences:

CAT Lab has already conducted a number of studies with communities about newcomer experiences in communities, especially focusing on theories of social norms. Going forward, we’re especially interested in study designs that could enable us to compare multiple interventions guided by different theories, inspired by the megastudy model, and perhaps tested with adaptive experiments. We look forward to continuing the conversation!

References