Description
This task is about identifying, as a team, the potential ways in which the Edit Check project could fail or otherwise not deliver the impact we're implementing it to have.
We will also use this task to determine how we will identify, and respond to, a scenario were it to arise.
Pre-mortem Link
Process
| Step | Description | Person/People Responsible | Status | Format |
|---|---|---|---|---|
| 1. | Identify and document potential failure scenarios | Editing-team | Done | Async |
| 2. | Cluster failure scenarios | @ppelberg | Done | Async |
| 3. | Estimate impact and likelihood of each failure scenario | Editing-team | Done | Sync/Async |
| 4. | Prioritize failure scenarios to plan for | Editing-team | Done | Async |
| 5. | Propose leading indicators (e.g. qualitative and quantitative) for prioritized scenario | @MNeisler | Done | Async |
| 6. | Propose a plan of action/mitigation strategy for each and Priority scenario | Editing-team | Done | |
| 7. | Relate each scenario to a planned analysis that will be defined in the experiment plan | @MNeisler | Done | Async |
Related Objects
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T265163 Create a system to encode best practices into editing experiences | |||
| Resolved | ppelberg | T331946 [RELEASE TICKET] Make Edit Check (references) available to all newcomers at all Wikipedias | |||
| Resolved | None | T331947 [RELEASE TICKET] Make Edit Check MVP available to all newcomers at Partner Wikipedias | |||
| Resolved | ppelberg | T338907 [RELEASE TICKET] Make Edit Check MVP available as Beta Feature (all wikis) | |||
| Resolved | None | T332646 [Deployment] Make Edit Check MVP (mobile + desktop) available for testing at test.wikipedia.org | |||
| Resolved | ppelberg | T325851 Conduct pre-mortem for Edit Check project |
- Mentioned In
- T409112: [MILESTONE] Run a controlled experiment to address section editing dead-end (mobile web)
T400101: [WE 1.1] Run a controlled experiment to evaluate impact of Reference Check at en.wiki
T399669: [WE 1.1.1] Run an A/B test to evaluate impact of Paste Check
T395166: Log when Tone Check was presented within an unpublished edit
T387918: [MILESTONE] Run an A/B test to evaluate impact of Tone Check
T379131: [A/B Test] Run an A/B test to evaluate impact of showing multiple Checks within a single edit
T352121: Conduct Pre-mortem for multi-check experience
T342930: [MILESTONE] Run an A/B test to evaluate Edit Check (references) impact - Mentioned Here
- E1630: Sabbatical
Event Timeline
Update: I've clustered the the failure scenarios that surfaced during the 1 March 2023 meeting the Editing Team had offline.
In T325851#8700640, @ppelberg wrote:Update: I've clustered the the failure scenarios that surfaced during the 1 March 2023 meeting the Editing Team had offline.
The tab of the Edit Check/Pre-Mortem sheet now reflects all of failure scenarios the team identified in the 1 March meeting I referenced in T325851#8700640 as well as the scenarios that were contributed asynchronously, before that meeting.
Update:
The Editing team has estimated the impact and likelihood of each failure scenario. All the scenarios in the Edit Check/Pre-Mortem sheet now have a priority.
I've proposed leading indicators (e.g. qualitative and quantitative) for each prioritized scenario and related each scenario to the final measurement plan. This included (1) adding a few new metrics to the measurement plan to account for any leading indicators not already identified and (2) also adding a note to each leading indicator in the pre-mortem sheet to identify the related experiment/analysis.
Next Steps:
The Editing team will review the prioritized leading indicators with @ppelberg once he's back and confirm a plan of action/mitigation strategy.
Propose a plan of action/mitigation strategy for each High and Medium Priority scenario
This is done; we've incorporated the "High" and "Medium" priority scenarios into the measurement plan by way of drafting a set of guardrails which can now be seen on-wiki in the table here: https://www.mediawiki.org/wiki/Edit_check#Evaluating_impact.