| ppelberg |
| Nov 3 2025, 8:23 PM |
| F70292229: image.png |
| Nov 26 2025, 6:21 PM |
| F70292226: image.png |
| Nov 26 2025, 6:21 PM |
| F70292215: image.png |
| Nov 26 2025, 6:21 PM |
Description
This tasks holds the work of running a controlled experiment to evaluate the impact of enabling people on mobile web to edit any article section, regardless of which edit icon they first tap (T387175).
Deployment timing
| Milestone | Target completion date | Ticket | Responsible | Status | Notes |
|---|---|---|---|---|---|
| Implement UX | ✅ | T409990 | @Esanders | DONE | Team will consider further refinements after usability testing is completed |
| Draft Tech/News announcement | ✅ | T409112 | @Quiddity | DONE | |
| Review Tech/News announcement | ✅ | T409112 | @ppelberg | DONE | |
| Complete usability testing (1st Round) | ✅ | T410614 | @bmartinezcalvo | DONE | Run first round of usability test; Key decision: what (if any) UX issues will we address before start of A/B experiment |
| Complete usability testing (2nd Round) | ✅ | T410614 | @bmartinezcalvo | NOT NEEDED | This will only be run if/when we come to learn aspects of the initial test protocol or user experience needs revision |
| Publish usability testing findings | ✅ | T410614 | @bmartinezcalvo | DONE | |
| Finalize UX refinements (if any) | ✅ | T410614 | @ppelberg | DONE | We've prioritized one refinement: T411669 |
| QA instrumentation and experiment set-up | ✅ | T410800 | Experiment platform | DONE | |
| Complete button QA | ✅ | T410319 | Editing QA + @MNeisler | DONE | |
| Implement instrumentation and experiment set-up | ✅ | T410800 | @MNeisler | DONE | |
| Conduct pre-deployment QA of UX | ✅ | T411669 | Editing QA | DONE | |
| Start test | Thursday, 11 Dec | T409112 | Editing Engineering | DONE | |
| Complete test | ✅ | T409112 | Editing Engineering | DONE | |
| Turn off experiment | Editing Engineering | ||||
| Complete analysis | @MNeisler | ||||
| Decide on path forward | @ppelberg |
UX
| First section | Middle section | Last section |
Hypothesis
If we enable people on mobile web to edit any article section, regardless of which edit icon they first tap, then the newcomer mobile edit abandonment rate will decrease by 1% because they will be able to more more easily locate the content they tapped edit seeking to change.
Decision to be made
What – if any – adjustments to the mobile web {nav Edit] button UX need to be made before we can be confident all of the following are true?
- Newcomers and Junior Contributors are less likely to abandon edits they initiate using the visual editor on mobile because they will be able to more easily locate the content they tapped edit seeking to change.
- Newcomers and Junior Contributors will be more likely to publish a constructive edit b/c they will have an easier time locating the content they tapped edit seeking to change
- Newcomers and Junior Contributors who encounter the new {nav Edit] button UX are NOT more likely to make changes that are disruptive
KPIs
The main outcomes we are trying to impact through this feature. These are what we are primarily using for evaluating the hypothesis and deciding whether to deploy an intervention more widely.
| ID | Hypothesis | Metric description |
|---|---|---|
| KPI | If we enable people on mobile web to edit any article section, regardless of which edit icon they first tap, then the newcomer mobile edit abandonment rate will decrease by 4% because they will be able to more more easily locate the content they tapped edit seeking to change. | Proportion of all editing sessions by users with ≤100 cumulative edits or logged-out users that are started on a mobile web Wikipedia main namespace (user clicks that the edit button () that are not saved (). Overall and by edit init type (page or section). |
| KPI | Constructive edits by newcomers will increase because more people acting in good faith will have an easier time locating the content they tapped edit seeking to change | Proportion of all published edits[i] on mobile web by users with ≤100 cumulative edits or logged-out users that are constructive [1] |
| Curiosity #1 | Users spend less time in the editor before starting to type their edit because they will be able to locate the area of the article are intent on editing with less confusion and/or friction | Time between the editor loaded () and a user starting to type (). Overall and by edit init type (page or section). |
[1] Constructive edits = edits to pages in any Wikipedia main namespace that are not reverted within 48 hours of being published.
Guardrails
Used to make sure that the new checks presented are not negatively impacting an editor’s experience completing an edit or causing disruption on the wikis. The scenarios named in the chart below emerged through T325851.
| Guardrail Name | Metric description | Notes |
|---|---|---|
| Constructive edits decrease | Proportion of published edits that are reverted within 48 hours. | By making it easier for people tapping edit on mobile web to change larger portions of the article, we may reduce the friction to publishing destructive edits |
| Edit quality decreases | Proportion of published edits that are reverted within 48 hours. | |
| Edit completion rate drastically decreases | Proportion of all editing sessions by users with ≤100 cumulative edits that are started (user clicks the edit button () and are saved (). Overall and by edit init type (page or section). (This should be the inverse of edit abandonment rate metric) | |
| Editor load time increases | Time between the user clicking the edit button () and the editor being ready for input (). Overall and by edit init type (page or section). | |
| Edit abandonment rate increases | Proportion of all editing sessions by users with ≤100 cumulative edits that include at least one click to the new "Edit full page" button on mobile web and abandon the edit prior to editor being fully loaded. | People who tap the “Edit full page” button T409990 will introduce will not abandon before the full article is loaded in an editable state |
A/B Test: Decision Matrix
| ID | Scenario | Indicator(s) | Plan of Action |
|---|---|---|---|
| 1 | The new {nav Edit] button UX is disrupting, discouraging, or otherwise getting in the way of volunteers | ≥20% decrease in any of the the following metrics in edit sessions where the new {nav Edit] button UX is is activated relative to edit sessions where it is not: edit completion rate, editor load time, edit abandonment rate, revert rate | Pause scaling plans; investigate changes to the UX. |
| 2 | The new {nav Edit] button UX is increasing the likelihood that people will publish destructive edits | Increase in the proportion of published edits where the new {nav Edit] button UX was shown and are reverted within 48 hours relative to edits where it was not. | Pause scaling plans, Review edits to try to identify any patterns in abuse and propose changes to UX to mitigate them. |
| 3 | The new {nav Edit] button UX is effective at lowering the rate at which people abandon edits and the edits people publish are reverted at higher rates | Decrease in the proportion of all editing sessions by users with ≤100 cumulative edits or logged-out users that are started on a mobile web Wikipedia main namespace (user clicks that the edit button (action = ‘init’) that are not saved (action = ‘saveSuccess’) and the proportion of published edits that are reverted within 48 hours. | Pause scaling plans; analysis and manual review of reverted edits to understand why those edits are being reverted. |
| 4 | The new {nav Edit] button UX does not show a statistically significant impact on any KPI, curiosity, or guardrail metric | See metrics above | Move forward with scaling plans |
| 5 | The new {nav Edit] button UX is effective at increasing the likelihood that people publish constructive edits and/or decreasing the rate at which they abandon the edits they start without a meaningful change in any guardrail metrics | See above | Move forward with scaling plans |
Announcements
Experiment Start
Later this week, a controlled experiment will begin for editors on the 100 largest Wikipedias who are editing a section in the mobile website Visual editor. 50% of editors will notice a new "Edit full page" button that will enable them to expand their editing session to the whole page. The experiment will last ~4 weeks. This feature is intended to make it easier for people on mobile web to edit any article section, regardless of which section-edit icon they tapped to begin. Please visit T409112 to see more details about the project.
Related Objects
- Mentioned In
- T418530: [SPIKE] Decide whether the Suggestion Mode MVP controlled experiment will run on Test Kitchen
T417840: Re-analyze edit completion rate among logged in Junior Contributors (mobile web)
T415336: [SPIKE] Investigate the decrease in edit completion rate amongst logged-in users (test group) in the Mobile Section Editing Dead End Experiment
T416059: Deploy full-page mobile editing affordance as a default-on feature (all wikis)
T411044: Consider archiving the 'Section Editing (Mobile)' project
T412318: [SPIKE] Investigate feasibility of further narrowing test population
T411669: Adjust transition upon tapping "Edit full page"
T411657: Improve scannability and section orientation in full-page editing on mobile
T410614: Conduct usability study: [Mobile] Offer button that enables people to edit an entire article from section editing
T410803: Create data stream for mobile web section editing dead-end intervention
T410800: Finalize mobile web section editing dead-end experiment design
T409990: [Mobile] Offer button that enables people to edit an entire article from section editing - Mentioned Here
- T415336: [SPIKE] Investigate the decrease in edit completion rate amongst logged-in users (test group) in the Mobile Section Editing Dead End Experiment
T412318: [SPIKE] Investigate feasibility of further narrowing test population
T410319: Add event tracking to the full-page editing button T409990 will introduce
T410800: Finalize mobile web section editing dead-end experiment design
T411669: Adjust transition upon tapping "Edit full page"
T409990: [Mobile] Offer button that enables people to edit an entire article from section editing
T410614: Conduct usability study: [Mobile] Offer button that enables people to edit an entire article from section editing
T410799: Ready mobile web section editing dead-end intervention for Test Kitchen experiment
T325851: Conduct pre-mortem for Edit Check project
T387175: Improve discoverability of full-page editing on mobile
Event Timeline
Next step
Per this week's offline discussion, assigning this task over to @MNeisler who is going to:
- 1. Take first pass at drafting the measurement plan for evaluating the impact of the work we are doing in T387175
- 2. Once a first draft of the measurement plan is in a good place, learn - through a conversation with the Experiment Platform Team – which (if any) aspects of the measurement approach we're proposing is not compatible with what Test Kitchen currently offers
In T409112#11355415, @ppelberg wrote:Next step
Per this week's offline discussion, assigning this task over to @MNeisler who is going to:
- 1. Take first pass at drafting the measurement plan for evaluating the impact of the work we are doing in T387175
- 2. Once a first draft of the measurement plan is in a good place, learn - through a conversation with the Experiment Platform Team – which (if any) aspects of the measurement approach we're proposing is not compatible with what Test Kitchen currently offers
"1." is done (see: measurement plan) and "2" is in progress
Next steps
- @MNeisler to continue working on "2."
- @ppelberg to update this task description with what Megan proposed, and we converged on, in the measurement plan.
Tech News draft (WIP):
Later this week, some [Wikipedia? all?] editors using the mobile visualeditor who are editing a section in Visual editor will be shown a button that can expand their editing session to the whole page. This A/B test should enable editors to more easily access the rest of the article mid-edit and lower the edit-abandonment rate. [1]
In T409112#11410589, @Quiddity wrote...
Thank you, Nick. Some edits reflected in the draft below...
v2
Later this week, a controlled experiment affecting editors [100 Wikipedias] who are editing a section in the mobile Visual editor will begin. The experiment will last ~4 weeks. 50% of the editors in the experiment will not notice any difference in their experience and 50% of editors will notice a new button that will enable them to expand their editing session to the whole page. This interventions is meant to make it easier for people on mobile web to edit any article section, regardless of which edit icon they first tap. Please visit T409112 to see more details about the metrics we will be using to evaluate the impact of this intervention. [1]
Next step
@ppelberg: update task description to include contents of measurement plan
Tech News draft slightly adjusted/simplified, and added to Description. It will be ready for announcement in the 8th December edition.
We still need to get the correct link for the "100 Wikipedias", and potentially find a place onwiki to document the screenshot and a sentence explaining it. [I'll work on those aspects]
In T409112#11411611, @Quiddity wrote:Tech News draft slightly adjusted/simplified, and added to Description. It will be ready for announcement in the 8th December edition.
We still need to get the correct link for the "100 Wikipedias", and potentially find a place onwiki to document the screenshot and a sentence explaining it. [I'll work on those aspects]
Wonderful. All that you described sounds great. Thank you, Nick.
Per offline discussions with @MNeisler, we've settled on targeting a 1% decrease in edit abandonment rate. I've updated the task description in 409112#11449802 to reflect this.
Note: in parallel, we're going to investigate the feasibility of further limiting the population of people we examine through this experiment to people who we assume to be arriving with an edit in mind they are intent on making. This investigation will happen in T412318.
Findings (high-level)
- While the feature seems to create some friction for registered newcomers, it appears to be effective for both more experienced Junior Contributors (≤100 cumulative edits) and logged-out users completing constructive edits.
- The most concerning metric is the Section Edit Completion rate of logged-in users, which saw a statistically significant 5.8% decrease. Additionally, the time to first change increased by 13.1% for logged-in users and decreased -14.5% for logged-out users, suggesting the UI addition is slowing down the initial editing momentum for some subset of registered users and accelerating it for folks who are not logged in.
- Further investigation into section edit completion rate (T415336) indicates that the button is very helpful to users that directly engage with it. Logged-in users who click on the Edit Full Page Button are 55.6% more likely to complete their edit compared to those in the treatment group who don't click the new button.
- The new edit full page button also proved highly effective for users who started typing, increasing edit completion rates by 8.5% for those who started typing and achieving a 96.8% edit completion rate for those who engaged with the feature.
- The overall 5.8% decrease in edit completion rate is mostly due to logged-in users in the treatment group who abandon their edits before starting to type. This indicates that the new Edit Full Page button might be:
- Slightly distracting or
- Confusing to some newcomers or
- Deter low-intent editors before they start editing.
- We also confirmed statistically significant improvements in section edit constructive edits rate for logged-out users.
Findings (detailed)
The findings below are drawn from Test Kitchen Experiment Analytics.
| Metric | Logged-in | Logged-out | Notes |
|---|---|---|---|
| Constructive edit rate | -0.557%* | +2.599%* | There is 93.5% chance that edits by logged-out users shown the new Edit full page button are more constructive than edits in the control group. This does not meet the 95% threshold and it is a strong indicator that this feature increases the quality of edits by logged-out users. The slight decrease observed for logged-in users is not statistically significant (Only 37% chance to win) |
| Constructive edit rate (newer editors) | -0.359%* | +2.599%* | |
| Section constructive edit rate (newer editors) | -0.686%* | +3.513% | For section-initiated edits, there is a 95.7% chance that edits by logged-out users shown the new Edit full page button are more constructive than edits in the control group. Conversely, we did not see any statistically significant changes for logged-in users. |
| Section edit completion rate (newer editors) | -5.797% | -1.684%* | Edit completion rate investigation: T415336#11563723 |
| Time until first change (newer editors) | +13.121%* | -14.537% | Logged-out users spend less time starting to type after the editor loads when shown the new Edit full page button. We confirmed a -14.5% statistically significant decrease in the time until first change for logged-out users. |
= not statistically significant
Path forward
Based on these findings, we understand this experiment to fall in Scenario 5:
The new {nav Edit] button UX is effective at increasing the likelihood that people publish constructive edits and/or decreasing the rate at which they abandon the edits they start without a meaningful change in any guardrail metrics
In line with the above, we recommend moving forward with deployment to additional Wikipedias.
While we observed a slight decrease in overall completion rate – primarily driven by users abandoning before they begin typing – the feature proved highly effective for high-intent editors, significantly increasing their success rate.
We will continue to monitor the impact on edit completion and explore design iterations, such as adjusting button prominence or placement, to reduce initial distraction for newcomers while preserving the tool's utility for registered newcomers.
Moving to while we await feedback from team on proposed path forward. Feedback expected by EOD Tuesday, 17 Feb 2026.
