mirror of
https://github.com/jackyzha0/quartz.git
synced 2025-12-24 13:24:05 -06:00
53 lines
1.5 KiB
Markdown
53 lines
1.5 KiB
Markdown
---
|
|
title: "running-web-experiments"
|
|
aliases:
|
|
tags:
|
|
- info203
|
|
- scott-video
|
|
- lecture
|
|
---
|
|
|
|
randomly split traffic to website between two versions.
|
|
|
|
then collection metrics about each version
|
|
|
|
web makes it easier to try out something new, and test if it is effective.
|
|
|
|
|
|
small changes can have a significant effect.
|
|
|
|
|
|
application for more than just websites.
|
|
|
|
our expectations are often wrong.
|
|
small chagnes (e.g., wording on a button) can have large impacts
|
|
|
|
# Effective principles
|
|
- run experiments 50 50
|
|
- novice experimenters run 1% eperiments
|
|
- to detect an effect you need to expose a certain number of users to the treatment (baed on power calculations)
|
|
- fastest to run 50 50
|
|
- ramp up and auto abort
|
|
- start at 0%
|
|
- design may be disastrous
|
|
- bugs, other issues
|
|
- slowly increase until 50%
|
|
- pick a meaningful yardstick
|
|
- avoid measures things because they are easy to measure
|
|
- run it for long enough
|
|
- sometimes first use is not the same as what people are familiar with
|
|
- rules for random assignment
|
|
- consistent - same person should see same interface
|
|
- durable - truly and compelely random
|
|
- independent
|
|
- challenges and opportunities
|
|
- larger theories are still hard
|
|
- i.e., interpretation
|
|
- using multiple methods together helps
|
|
- whole is greater than sum of parts
|
|
- combine iwth eg. in person studies
|
|
- role on designer in online age
|
|
- creating alternatives
|
|
- people are often to sure of themselves
|
|
- rapid experimentation means the first release is (sometimes) less important
|
|
- fail fast |