A box and pointer diagram showing the components of the CivilServant system. Components are divided into problem scoping & study design, data collection & storage, experiment deployment & management, and results analysis & reporting. Shows interaction with reddit, twitter, lumen, and 3rd-party moderation tools.
System architecture for the first version of the CivilServant infrastructure, circa 2016

Can citizen science help build a safer, kinder internet? Moderators of online communities, often volunteers, work hard to protect their communities from harassment, misinformation, trolls, and other online harms. But which policies and procedures actually work? And how can they be sure they’re having the desired effect?

CivilServant, a Free and Open-Source python app (GitHub), is a citizen science tool to help online community moderators and social media researchers answer these questions. Experiments are an effective way to test ideas about online behavior. Large companies often run two slightly different versions of an ad to test which gets more clicks, known as A/B testing. But these types of large-scale field experiments require infrastructure that doesn’t exist within most online platforms. CivilServant provides the infrastructure moderators need to perform their own field experiments. But instead of maximizing clicks, it helps identify which moderation techniques work best. For example, moderators might test if placing a “sticky comment” with the rules at the top of a discussion reduces the number of comments that need to be removed. In addition to automatically interacting with online content, CivilServant also implements research best-practices such as block-randomization. By lowering the barriers to doing citizen science, CivilServant makes it possible for more communities to test more ideas. And the more ideas get tested, the more evidence there is to guide online moderators.

At its core, CivilServant is an app for collecting real-time data from social media platforms, assigning activity to experimental treatment groups, and performing interventions. For each experiment, parameters are entered into a text-based YAML file. Moderators grant the desired level of permissions to a CivilServantBot account, which uses an API to interact with the platform. Data Monitoring Controllers then query activity at regular intervals and archive it in a MySQL database. Using experiment-specific Intervention Controllers, the app can run custom python functions, called hooks, when certain events occur. Hooks might respond to new posts, comment removals, or account suspension. Hooks can also randomly assign an account or event to an experimental condition and apply an intervention. In the sticky comment example, CivilServant identifies new posts, assigns them to control or treatment groups, and posts a sticky comment to posts in the treatment group. When the experiment is done, researchers analyze the archived data, evaluate how effective the intervention was, and report back to the community.

The CivilServant infrastructure has provided nearly continuous research for almost a decade. It has continued to work reliably during a period when many other research infrastructures have come under threat or faced severely-limiting technical restrictions. In 2017, FastCompany selected CivilServant as a finalist in the Innovation by Design awards. We’re grateful to our partner communities for co-designing a system that has become surprisingly resilient over time. 

Projects that have used the CivilServant infrastructure:

Scientific research supported by the CivilServant software has been published in journals and conferences across computer science and the social sciences, including the Proceedings of the National Academy of Sciences, Nature Scientific Reports, Computer-Supported Cooperative Work, and the Social Science Research Council.

Acknowledgments
CivilServant was originally developed by Dr. J. Nathan Matias for his doctoral dissertation, with contributions from Merry Mou. Several others have contributed to the maintenance and further development of CivilServant, including Eric Pennington, Edward L. Platt, Chris Willard, and Dan Taeyoung.

We are grateful to the many funders whose work has supported the platform, including the Templeton World Charity Foundation, Ford Foundation, MacArthur Foundation, Knight Foundation, the The Ethics & Governance of AI Fund, and the Mozilla Foundation.