Drawing of peoplee dancing around symbolic illustrations of a conversation

Wikimania is one of CAT Lab’s favorite events ever since we first attended in 2018. We’re always inspired by this annual gathering of the global movement to freely share the sum of all knowledge. We’re also privileged to collaborate with Wikipedia communities worldwide on research to help achieve that vision.

This year, with Wikimania yet again held online, we wanted to do something a bit more playful than our typical research idea workshops.  What kind of fun event could build social relationships, help people learn more about existing research about Wikipedia, and also highlight open questions that aren’t yet answered by science? There was only one answer:

Trivia!

To gather the information we needed for our trivia game, we collected a list of classic papers and new research about Wiki projects. While our goal going into planning the event was to provide a fun session for participants, we also learned trivia game development is a really fun way to organize a literature review and we hope to do more events like this in the future! We’ve included the complete list of studies we drew from at the end of this post for your reference.

We wanted the trivia game to be a fun learning opportunity. As a citizen/community science lab, we also wanted to help players learn to envision important questions that haven’t yet been answered. By inspiring the inner scientist in everyone, we hoped to start longer conversations about questions that could become future collaborations.

To achieve these goals, we organized the trivia game in the format of “two truths and a lie,” but used “two knowns and an unknown.” In each round, we showed players three statements and asked them which was an“unknown” statement about Wikipedia. Each unknown represents a practical, scientific question that hasn’t yet (as far as we can tell) been answered. After introducing CAT Lab and describing how the game was played, we formed teams by placing participants into Zoom breakout rooms. After a round of introductions, teams worked together to complete  a Google Form with the others in their group. We finished the session by reviewing the answers and invited participants to react in the chat.

We are especially grateful to the Wikimania tech team for supporting this interactive group session, and to Aparajita Bhandari for co-leading the session. We are also grateful to the Templeton World Charity Foundation for supporting our work with Wikipedia communities. Thanks!

If you’d like to play at home, the questions are listed below. Skip to the next section to see the answers and explanations!

The Questions

Question 1

  1. We can reliably identify the gender of registered Wiki users based on profile information. (Minguillón et al., 2021) 
  2. The reduction of mobility during COVID increased the volume of people seeking information on Wikipedia. However, once mobility returned to normal, the volume also returned to normal, but the kind of information people looked for did not. (Ribeiro et at., 2021). 
  3. Contributions from Tor users that slip though detection are similar to contributions from unregistered and new editors (Tran, 2019). 

Question 2 

  1. Wikipedia editors who spend more time monitoring Wikipedia for damaging content feel more emotionally drained but also more positive about their contributions compared to those who do less monitoring work. (Matias et al., 2020) 
  2. Requiring account registration to edit Wikia wikis reduced both the number of low-quality edits and the number of high quality edits, causing an overall decrease in quality, across 136 communities. (Hill & Shaw, 2021) 
  3. Data scientists have created a single reliable measure of information inequality on Wikipedia that works across languages and cultures. This measure can be used to monitor and improve how well Wiki Projects represent the world’s knowledge. (Beytía, 2020; Zia et al 2019) 

Question 3 

  1. As of 2021, the Wikimedia movement has found effective ways to increase the retention of editors from Asia, Africa, and Latin America year on year (Community Insights, 2021) 
  2. Edit-a-thon organizers have motivations beyond closing gaps in Wikipedia. For example, they also hope Edit-a-thons can build information literacy and foster community outside of Wikipedia. (March & Dasgupta, 2020). 
  3. While female music artists are underrepresented in the music industry, males are underrepresented on Wikipedia. (Wang et al., 2021) 

Question 4 

  1. While volunteer contributions to Wikipedia increased during COVID-19, larger Wikipedias grew more than small Wikipedias, which largely stayed at the same levels of participation. (Ruprechter et al.,  2021) 
  2. Financial support intended to incentivize contribution to Wikipedia does not lead to active participation across all Wikipedias. (Khatri et al., 2022) 
  3. Over the last decade, the coverage gap on Wikipedia between Europe and Africa, has reduced by roughly five times. (Dittus & Graham, 2022) 

Question 5 

  1. Sociocultural norms impact Indian women’s ability to contribute to Wikipedia in ways that have not been identified in other language Wikipedias. (Chakraborty & Hussain, 2022)  
  2. Recommender algorithms that suggest priorities to editors based on WikiData, are an effective way to fill knowledge gaps across Wikipedia.(Redi et al 2021) 
  3. A person’s biography is more likely to be available in languages common to the person’s nationality, ethnicity, and background. (Field at al., 2022) 

Question 6 

  1. Wikipedia is used to satisfy a variety of motivations, from looking up a topic that was referenced in media or conversation to wanting to learn something, with no dominant individual motivation. (Singer et al., 2017) 
  2. Biography pages of transgender women and non-binary people tend to be longer and available in more languages than comparison articles, indicating a possible glass ceiling effect in which there is a higher bar for transgender women and non-binary people to have a Wikipedia article. (Field & March, 2022). 
  3. Edit-a-thon organizers across language Wikipedias have motivations beyond closing gaps in Wikipedia. For example, they also hope Edit-a-thons can build information literacy and foster community outside of Wikipedia. (March & Dasgupta, 2020) 

Question 7 

  1. Globally, women are underrepresented as readers of Wikipedia. (Johnson et al., 2021). 
  2. On the weekends and late at night, Wikipedia readers are more likely to be led to Wikipedia by media coverage; on Fridays and Saturdays, they are more likely led by conversations. (Singer et al., 2017) 
  3. Data on behavior from Wikipedia can provide a reliable, indicator across languages of conflict and edit wars, including conflicts between bots. (Geiger & Halfaker, 2017) 

Question 8 

  1. Wikipedians and social scientists have found reliable ways to increase the visibility of notable women in Google results and the Wikipedia’s link structure by adding new biographies. (Langrock & González-Bailón, 2022) 
  2. Biographies about women who meet Wikipedia’s criteria for inclusion are more frequently considered non-notable and nominated for deletion compared to men’s biographies. (Tripodi, 2021) 
  3. Newcomers’ feelings of empowerment have increased since 2019. Especially among women and newcomers in East Asia. (Community Insights, 2021) 

Question 9 

  1. Most reported harassment on Wikipedia is perpetuated by multiple people over time (vs harassment perpetuated by one person once). (Wikimedia Harassment Report, 2015) 
  2. Wikipedia use among urban and rural users is similar. (Redi et al, 2021) 
  3. Most Wikidata items propagate to only a few language editions. (Valentim et al., 2021) 

Question 10

  1. There is a link between users’ circumstances and how they use Wikipedia. For example, people who use Wikipedia for work or school tend to access articles directly and spend more time reading, while those satisfying boredom use internal links and spend less time reading. (Singer et al 2017) 
  2. Recommender algorithms that suggest priorities to editors based on WikiData,are an effective way to fill knowledge gaps across Wikipedia 
  3. Adding a trust meter to Wikipedia pages influences to trust individual articles in ways that match their actual reliability— lower trust for unreliable articles and higher trust for reliable ones. (Kuznetsov et al 2022)

See the answers below!

A pile of Trivial Pursuit trivia cards of all different colours are splayed over the ground

The answers

And now for the part you’ve been waiting for—the answers! Below we’ve listed not only the answers, but also provide an explanation for why the unknown is the unknown and the percentage of groups

Question 1

The answer is a! “We can reliably identify the gender of registered Wiki users based on profile information. (Minguillón et al., 2021)”

Why is it unknown? While researchers have have found some better-than-random success identifying editor gender from editor user pages— enough to do some summary analysis of Spanish Wikipedia, automated tools can’t make judgments about the full spectrum of gender identities, making this area a continual unknown. This was a well known unknown! Of the 21 groups who submitted responses, 65% got it right!

Question 2

The answer is c! “Data scientists have created a single reliable measure of information inequality on Wikipedia that works across languages and cultures. This measure can be used to monitor and improve how well Wiki Projects represent the world’s knowledge. (Beytía, 2020; Zia et al 2019)”

Why is it unknown? While researchers have have found some better-than-random success identifying editor gender from editor user pages— enough to do some summary analysis of Spanish Wikipedia, automated tools can’t make judgments about the full spectrum of gender identities, making this area a continual unknown. This was an unknown, divided. While just over half of the 21 groups (52.6%) got the right answer, over a third (36.8%) thought that editor burnout was the unknown

Question 3

The answer is a! “As of 2021, the Wikimedia movement has found effective ways to increase the retention of editors from Asia, Africa, and Latin America year on year (Community Insights, 2021)”

Why is it unknown? While Wikipedia has been successful at attracting newcomers who live in Africa, Asia, and Latin America, as of the 2021 Community Insights report, there had been been no overall improvement in the geographic diversity of those tenured editors since 2019. Many researchers, including the Growth Team at the Wikimedia Foundation, have been testing ideas that can help. This one stumped our groups! Under half (42.1%) guessed the correct unknown while more (47.4%) guessed that research about female music artists was the unknown

Question 4 

The answer is b! “Financial support intended to incentivize contribution to Wikipedia does not lead to active participation across all Wikipedias. (Khatri et al., 2022)”

Why is it unknown? Qualitative research conducted with Indian language wikipedia has found that financial support intended to incentivize contribution did not lead to active participation. But it’s not clear whether a different approach to financial support would have worked or whether it would be effective elsewhere. Our participants were divided on this one—38.9% guessed b. And the same number guessed c!  

Question 5

The answer is b! “Recommender algorithms that suggest priorities to editors based on WikiData, are an effective way to fill knowledge gaps across Wikipedia. (Redi et al 2021)”

Why is it unknown? In A Taxonomy of Knowledge Gaps, the Wikimedia foundation suggested that recommender algorithms might help fill knowledge gaps, an area that’s an exciting area of experimentation by the movement and researchers. Exactly half of our participants guessed this one right! 

Question 6

The answer is c! “Edit-a-thon organizers across language Wikipedias have motivations beyond closing gaps in Wikipedia. For example, they also hope Edit-a-thons can build information literacy and foster community outside of Wikipedia. (March & Dasgupta, 2020)”

Why is it unknown? While many researchers have worked alongside organizers of Edit-a-tons in English Wikipedia, a recent paper by March & Dasgupta based on interviews with many edit-a-thon organizers strongly encouraged future researchers to work with non-English organizers to build a clearer picture about motivations in the majority world. This was a really tricky one: only 11.8% of participants guessed c! 47.1% guessed b and 41.2% guessed c. 

Question 7 

The answer is c! “Data on behavior from Wikipedia can provide a reliable, indicator across languages of conflict and edit wars, including conflicts between bots. (Geiger & Halfaker, 2017)”

Why is it unknown? While some researchers have created prototypes of fully automated metrics for conflict on Wikipedia, social scientists who looked closer at the context have found that what looks like conflict can sometimes actually be collaboration. The most popular guess for this question was b (41.2%) the rest of the guesses were divided between a and c with 29.4% of groups guessing each. 

Question 8

The answer is a! “Wikipedians and social scientists have found reliable ways to increase the visibility of notable women in Google results and the Wikipedia’s link structure by adding new biographies. (Langrock & González-Bailón, 2022)”

Why is it unknown? Research has found that while interventions can increase content about women, they don’t necessarily decrease gaps in infobox content and links that influence search results. While early research has investigated how Wikipedia influences search results, the best way to do that is not yet clear. Only 29.4% of our participants guessed this one correctly. The most popular guess for this question was c., with 47.1% of the guesses. 

Question 9

The answer is b! “Wikipedia use among urban and rural users is similar. (Redi et al, 2021)”

Why is it unknown? Survey research suggests that urban readers tend to be overrepresented among readers and contributors to Wikipedia. Therefore, less is known about the experiences of rural readers (or what might prevent people in rural areas from reading or accessing Wikipedia). This was our easiest questions, according to our participants. 64.7% guessed correctly (and none guessed option c)!

Question 10

The answer is b! Showing newcomers view counts on articles they edited causes them to see the value of their edits and contribute more to Wikipedia.  (Growth Team 2022 – contact them if you want to help!) 

Why is it unknown? Can we increase participation by showing newcomers how many people have viewed their articles and helping them see the value of their contributions? That’s an unknown- the Growth Team is testing this idea right now- so contact them if you want to test it with your language Wikipedia. We actually made a mistake with this one—one of the options was different on the form we shared with participants. So unfortunately, we don’t have stats for this one.

References

Knowns

  • Chakraborty, A., & Hussain, N. (2022). Documenting the gender gap in Indian Wikipedia communities: Findings from a qualitative pilot study. First Monday.
  • Community Insights 2021 Report. https://meta.wikimedia.org/wiki/Community_Insights/Community_Insights_2021_Report
  • Dittus, M., Graham, M (2022) The language geography of Wikipedia. Oxford Internet Institute and Whose Knowledge?
  • Field, A., Park, C. Y., Lin, K. Z., & Tsvetkov, Y. (2022, April). Controlled analyses of social biases in wikipedia bios. In Proceedings of the ACM Web Conference 2022 (pp. 2624-2635).
  • Hill, B. M., & Shaw, A. (2021). The Hidden Costs of Requiring Accounts: Quasi-Experimental Evidence From Peer Production. Communication Research, 48(6), 771-795.
  • Johnson, I., Lemmerich, F., Sáez-Trumper, D., West, R., Strohmaier, M., & Zia, L. (2021, May). Global gender differences in Wikipedia readership. In Proceedings of the International AAAI Conference on Web and Social Media (Vol. 15, pp. 254-265).
  • Kuznetsov, A., Novotny, M., Klein, J., Saez-Trumper, D., & Kittur, A. (2022, April). Templates and Trust-o-meters: Towards a widely deployable indicator of trust in Wikipedia. In CHI Conference on Human Factors in Computing Systems (pp. 1-17).
  • March, L., & Dasgupta, S. (2020). Wikipedia Edit-a-thons as Sites of Public Pedagogy. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2), 1-26.
  • Matias, J. N., Al-Kashif, R., Kamin, J., Klein, M., & Pennington, E. Volunteers Thanked Thousands of Wikipedia Editors to Learn the Effects of Receiving Thanks.
  • Ribeiro, M. H., Gligoric, K., Peyrard, M., Lemmerich, F., Strohmaier, M., & West, R. (2021, April). Sudden Attention Shifts on Wikipedia During the COVID-19 Crisis. In ICWSM (pp. 208-219)
  • Rodolfo Valentim, Giovanni Comarela, Souneil Park, Diego Saez-Trumper. 2021. Tracking Knowledge Propagation Across Wikipedia Languages. Proceedings of the Fifteenth International AAAI Conference on Web and Social Media (ICWSM ’21). Dataset.
  • Ruprechter, T., Horta Ribeiro, M., Santos, T., Lemmerich, F., Strohmaier, M., West, R., & Helic, D. (2021). Volunteer contributions to Wikipedia increased during COVID-19 mobility restrictions. Scientific reports, 11(1), 1-12.
  • Singer, P., Lemmerich, F., West, R., Zia, L., Wulczyn, E., Strohmaier, M., & Leskovec, J. (2017, April). Why we read Wikipedia. In Proceedings of the 26th international conference on world wide web (pp. 1591-1600).
  • Support and Safety Team. Harassment Survey 2015 Results Report. https://upload.wikimedia.org/wikipedia/commons/5/52/Harassment_Survey_2015_-_Results_Report.pdf
  • Tran, C., Champion, K., Forte, A., Hill, B. M., & Greenstadt, R. (2019). Tor users contributing to Wikipedia: Just like everybody else. arXiv preprint arXiv:1904.04324.
  • Tripodi, F. (2021). Ms. Categorized: Gender, notability, and inequality on Wikipedia. New Media & Society, 14614448211023772.
  • Wang, A., Pappu, A., & Cramer, H. (2021, May). Representation of Music Creators on Wikipedia, Differences in Gender and Genre. In ICWSM (pp. 764-775).

Unknowns

  • Beytía, P. (2020, April). The positioning matters: Estimating geographical bias in the multilingual record of biographies on wikipedia. In Companion Proceedings of the Web Conference 2020 (pp. 806-810).
  • Community Insights 2021 Report. https://meta.wikimedia.org/wiki/Community_Insights/Community_Insights_2021_Report
  • Geiger, R. S., & Halfaker, A. (2017). Operationalizing conflict and cooperation between automated software agents in wikipedia: A replication and expansion of’even good bots fight’. Proceedings of the ACM on Human-Computer Interaction, 1(CSCW), 1-33.
  • Khatri, S., Shaw, A., Dasgupta, S., & Hill, B. M. (2022, April). The social embeddedness of peer production: A comparative qualitative analysis of three Indian language Wikipedia editions. In CHI Conference on Human Factors in Computing Systems (pp. 1-18).
  • Langrock, I., & González-Bailón, S. (2022). The Gender Divide in Wikipedia: Quantifying and Assessing the Impact of Two Feminist Interventions. Journal of Communication, 72(3), 297-321.
  • March, L., & Dasgupta, S. (2020). Wikipedia Edit-a-thons as Sites of Public Pedagogy. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2), 1-26.
  • MediaWiki. Growth/Growth Team Updates. https://www.mediawiki.org/wiki/Growth/Growth_team_updates 
  • Minguillón, J., Meneses, J., Aibar, E., Ferran-Ferrer, N., & Fàbregues, S. (2021). Exploring the gender gap in the Spanish Wikipedia: Differences in engagement and editing practices. PLoS one, 16(2), e0246702.
  • Redi, M., Gerlach, M., Johnson, I., Morgan, J., & Zia, L. (2020). A taxonomy of knowledge gaps for wikimedia projects (second draft). arXiv preprint arXiv:2008.12314.