Does receiving a welcome message increase how many newcomers to Wikipedia make their first edit, and influence them to keep contributing over time? How might a personal offer of mentorship increase those effects, if they exist?
For many years, the French language Wikipedia has sent a welcome message to people who register new accounts. Even as Wikipedia as a whole has struggled to balance vandalism protection with newcomer support, the French community pioneered a scalable approach to hospitality and human contact.
Successful interventions on English Wikipedia have focused on already-active editors (Morgan & Halfaker 2017). In our collaboration, the French community hoped to achieve a more elusive goal (Narayan et al 2017): increasing the percentage of accounts that make their very first contribution. Roughly 565 new accounts are registered each day on French Wikipedia. Among those who haven’t received a welcome message, 37% of accounts go on to make at least one edit, and only 4% are still active contributors after four weeks.
successful interventions have focused on already-active editors; how can we support newcomers before they make their first edit?
To help the French Wikipedia community discover the effect of their welcome message, we worked with the community to design an experiment. The Citizens and Technology Lab (CAT Lab) works alongside communities to test ideas for change online. We do this by collaborating with communities to design new studies with community consent. We then use our research software to carry out those studies. This post is our report back to the French language Wikipedia community.
Many studies test a new intervention that hasn’t been tried before. To study something that the community was already doing, we created a control condition representing the newcomer experience without a welcome message. As people registered for new Wikipedia accounts, they were randomly assigned to one of the following conditions:
- In Treatment I, newcomers were sent a welcome message to their talk page using the same welcome message that French Wikipedia had used for the previous ___ years.
- In Intervention II, newcomers would be sent a slightly modified message with a prominent “contact me” button as further encouragement to contact mentors if they needed guidance.
- The Control condition, newcomers did not receive a welcome message
In a field experiment with 57,084 new accounts on French Wikipedia in early 2020, we did not detect any influence from the welcome message on the percentage of accounts that went on to make their first edit. We also did not find any reliable effect on the estimated number of minutes they contributed to Wikipedia over the next 84 days. Similarly, we did not find any significant differences between the “contact me” welcome message and the standard one.
What Happened in the Experiment
Because prior studies have found very small to no effects from welcome and orientation programs for new accounts, we designed a study that could observe small effect sizes. Our study included 57,084 participants, almost all of the newly-registered users to French Wikipedia between January 30, 2020 and May 10, 2020.
Following the community’s common practices, our study omitted users whose accounts were created for them, for example in an edit-a-thon, as well as accounts that are “autocreated” when an editor from another Wikipedia first logs into French Wikipedia. When we sent messages to accounts, we sent them to newcomers in conditions I and II within 3 minutes of their registration (the same time frame used by FR Wikipedia previous to the study). We then observed their contributions to Wikipedia over the next twelve weeks and asked them to take a survey (details here).
In any message-based study, not everyone receives the messages. Of the 38,056 newcomers we tried to send messages, 492 did not receive a message from us either because their account had already been blocked, the newcomer had elected to use a “flow” design for their talk page, which could not accept our automated message, or because they had already received a welcome message on their talk page from someone else. In a few cases, someone sent them the standard Treatment 1 message when our software would have sent the Treatment II message.
Some people erroneously received a welcome message. Of the 19,028 newcomers in the control condition, 934 received a welcome message from another source within the first three days of registration. The sender was sometimes one of FR.wiki’s automated bots, for example Salebot which alerts editors to errors and simultaneously sends a welcome message if one does not already exist. Other times, people received welcome messages from an especially supportive editor who manually sent them a welcome message.
After 84 days from registration, all newcomers in the study received an invitation, also on their talk page, to participate in a survey. A total of 854 people answered the survey, the lowest response rate on Wikipedia we have ever seen, at 1.5%.
Research Questions
After talking to community members, analyzing data, and surveying thousands of French Wikipedia newcomers, we decided to focus on two main questions (analysis plan, analysis plan supplement):
- Do the messages increase seven-day activation? This is the rate at which newly-registered accounts go on to make at least one edit in their first seven days. The base rate is 36.9%, for accounts that receive no message
- Do the messages increase the amount of time that accounts contribute to Wikipedia in their first 84 days? The base rate, measured in labor minutes, is 22 minutes, for accounts that receive no welcome message
We were interested to see if sending a welcome message had any effect on newcomer’s participation and whether adding the “contact me” button had an additional effect compared to the standard message. So for each outcome, we made two comparisons; one between Treatment I and the Control condition, and one between Treatment I and Treatment II.
In this post, we report an “intent to treat” estimate that looks at the effect of a newcomer being assigned to their treatment group rather than actually getting the message. Recall that not all newcomers received the correct treatment. In this model, we do not account for those who may have received a treatment other than the one assigned. We do conduct further modeling to account for who actually got the messages. Since the results were essentially the same, we report the simpler results here.
Results
First, we tested whether the welcome message would increase the rate of newcomers who go on to make at least one edit. We designed the study so that we would be able to detect differences of at 1.5 percentage points. Anything smaller than an increase from 36.9% to 38.4% would be undetectable, and we would have an 80% chance to observe anything larger.
In the intent-to-treat estimate, we neither saw an effect of receiving the welcome message on 7-day-activation, nor an increased effect of having a “contact me” button. If sending newly-registered accounts a welcome message increases seven-day activation rates at all, it would likely be an effect smaller than 1.5 percentage points, fewer than 8 people out of the 505 accounts that register every day.
We also tested whether the welcome message had an effect on the number of labor minutes people contribute to French Wikipedia over 12 weeks. In this analysis, when accounting for which participants actually received the message, we found that including the mentorship button caused participants to contribute 22% fewer minutes over 84 days on average. However, when we looked closely at the data, we confirmed that this difference was driven by two participants in the control group that contributed a very large amount of time. In a sensitivity analysis, we found that if you remove the randomization blocks that they appear in (12 participants in total), the number of labor minutes is not observably different.
| Group | 7 day activation rate | Incidence rate of Labor minutes over 84 days |
| Control | 36.9% | 22.8 |
| Welcome message | 37.1% | 21.3 |
| Welcome message + “contact me” | 37% | 20.3 |
What this means for French Language Wikipedia
When we first designed this study, some in the French Wikipedia community were worried that a study with a control group could reduce French Wikipedia’s retention rate. This was a legitimate concern. If the welcome message did have a substantial effect on newcomer retention, our experiment would reduce the number of contributors just to learn the effects of something the community already knew. Others responded with more uncertainty—what if the community was relying on something that worked less well than they thought? Our study would help the community put energy into the most effective ideas.
By discovering that the current welcome message does not have a measurable effect on newcomer activation, this study can help French Wikipedia community think about potentially more effective ideas. Here are some lessons for the community from our study.
Talk-page messages might be a poor way to reach the earliest newcomers, even if experienced contributors respond well to them. While Wikipedia does not provide data about how many people view a page, our survey results lead us to believe that very few people who register Wikipedia accounts go on to read the messages they receive. At the Wikimedia Foundation, the Growth Team has been re-thinking the new editor experience through designs like the Newcomer homepage.
Offers of mentorship might be more effective after people have attempted to contribute to Wikipedia. Prior studies, including our research on thank-you messages, have found significant effects from social support among contributors who have made at least several good-faith edits. It’s possible that this smaller group who have already contributed to Wikipedia are more interested in mentorship and are more able to benefit from it.
Finally, we hope that the French Wikipedia community will also be able to help us make sense of this result together.