Notes on IMVU Method of Experimentation

The New Science of Product Development

View more presentations from James Birchler

James is the VP of Engineering at IMBU, the very first Lean Startup founded by Eric Ries. IMVU experimented heavily on its way to tens of millions in annual revenue. In his talk, James describes how to build a culture and team of experimentation in your startup. You can watch the video and see the slides above.

Below are my notes on James Birchler's talk on experimentation at IMVU. I enjoyed his talk, particularly the segments that dealt with the specifics of IMVU's processes.

The Scientific Method is based on experimentation.

Core:

1. culture
   
2. technique
   
3. examples
   

Story of Copernicus

• heliocentrism
  
• experimental observation
  
• was met with violence; Church threatened him
  

Giordano Bruno (burned at stake)

Galileo

• scientific method
  
• put on house arrest
  

It was hard for these guys because the folks in charge didn't like hearing "bad news."

IMVU

• experiments
  
• share results freely, no matter who's pet project
  
• rapid iteration and learning
  

1. ask question
2. do research
3. form hypothesis
4. test (science?)
5. analyze data
6. conclusion
7. report results

And cycle back to beginning

Lean startup: build, measure, learn loop

1. talk to customers for use cases
   
2. form a hypothesis to test
   
3. write code, test on dev machine
   
4. test in production as QA/admin
   
5. roll out to a % of customers
   
6. analyze results, conclusion
   
7. share learning
   

Culture of experimentation:
Started out as simple as this:
if (setup_experiment(...) == "control") {
// do it the old way
} else {
// do it the new way
}

Then later built robust experiment management system

• Columns: experiment name, to which users (QA and admin only, 100%, 50%, etc.), status/close date
  

Also built tool to calculate statistical significance

• sample count: control and treatment
  
• mean: control and treatment
  
• variance: control and treatment
  
• p-value
  
• significance
  
• chance of occurring randomly
  
• table of user data with rows highlighted to show which validated or invalidated experiment
  

Embrace failure with exec team and whole company.

Embrace failure as an opportunity to learn.

Highest paid person's opinion (HiPPO) is not assumed to be correct.

Reference to paper "Practical guide to controlled experiments on the web."

Easy to screw up process at every stage

• use cases
  
• test hypotheses
  
• rinse, repeat
  

Don't ask customers what they want.

Instead, ask customers what they are trying to do.

Focus on use cases and not what they say they want.

What not to do: build what customers say they want.

Do this:

• start with use cases
  
• test hypotheses to learn best ways to fulfill them
  

It takes a lot of courage to balance experience, instinct, imagination, and experiment results.

"If you design a toaster oven and need to include directions for making toast, you have failed at designing a toaster oven." -Laura Klein

Don't just run a bunch of experiments without a strong point of view of what trying to learn.

Experiment design

• start with use cases
  
• have a specific hypothesis
  
• ask the right questions
  
• ask the right people
  
• ask enough people
  
• randomized control trial
  

experiment interpretation

• use the right p test
  
• don't rely only on p test
  
• don't confuse correlation with causation
• find psych or stats PhD's to help

Experiments are a great way to test hypotheses, not form them.

Product development

Build:

• 2-3 week sprints
  
• stop and adjust process each sprint
  
• agile and XP methods
  

Agile & XP methods:

• change, flexibility, iteration, continuous improvemnt
  
• Agile and XP support each other
• physical scrum board with post-its
• self-organizing teams, short sprints, daily stand-ups (15 min), no emails (just talk)
• clear roles and responsibilities: product owner, tech lead, visual designer, UED, QA
• stay flexible, don't be dogmatic
  

Measure:

• projects/stories completed
  
• time spent on tasks
  
• story points delivered
  
• unplanned vs. planned work completed
  
• how productive and happy do we feel
  

Learn:

• project postmortens
  
• sprint retrospectives
  
• 5 why's root cause analysis
  
• support open communication
  

Open communication:

• engineering project managers (just to assure engineering communicates ok)
  
• matrix management (managers manage people across teams to get slice across company)
  
• scrum of scrums (weekly uber-scrum mtg)
  
• team swaps (individuals switch teams liberally)
  
• open floor plan
  

Postmortems and retrospectives:

• meeting roles
  
• metrics
  
• action items
  
• all levels of organization must be in it
  

Meeting success:

• appoint a skilled facilitator
  
• foster communication and engagement (talk about elephant in room)
  
• table: days, project, story points
  
• whiteboard columns: misc., keep, stop (action item), start (action item)
  

5 Why's: fix root cause

Make the size of the fix commensurate with the size of the problem.

Handling interrupts

• Share interrupts among teams
  
• Create "interrupts" team; establish home team and rotate others through it
  
• Whenever someone commits code that fails battery of tests, interrupts team needs to fix the tests or the code.
  

Time tracking (used to do it more but now do it less):

• Focus on features and value to customers, not time
  
• Short planning meetings
  
• Caution: reduced ability to predict progress
  

Scrum 2.0: continuous planning

• based on successful process experiment
  
• just-in-time planning: kanban
  
• are we working on the right stuff?
  

User testing:

• pay $50 for 30 min of someone's time
  
• ask them to play your game, watch their reaction
  
• stop them after 15 min. and say they can go
  
• if they want to keep playing, then you're doing well
  

Keep your team together: embed designers, QA, data analyst, engineering and product in same location

Have a fixer: remove blocks, remain objective, team happiness = team productivity

Technical project reviews:

• organized by team tech lead: tech review of code, improve quality, reduce tech debt
  
• in-depth reviews of code
  
• all engineers welcome
  
• note follow-up action items
  
• prioritize actions and do them
  

Experiment they ran: small teams in a bare garage + no rules = big win

• break out of routine
  
• minimize process overhead
  
• focus on value to customers
  

Self-selecting teams: let people self-select to the teams they want to work on

Shake things up

• Switch to continuous planning
  
• Story points vs. Ideal Days