Cosi102

* **READING**: [A/B Testing Pitfalls](
https://newsletter.posthog.com/p/ab-testing-mistakes-i-learned-the?_bhlid=468968ca7029fc7ac413206d4e5b8e41803156a6)

*Reminder: Readings are your responsibility. You will be expected to come to class prepared, having read the material, and ready to participate in the discussion*

<slide_break></slide_break>

### Logistics
* Getting grading going. 
* New team and new Moodle...
* "Does not meet expectations", "Meets expectations", "Exceeds Expectations"
  
**Reading**

* At this point you *You will be expected to come to class prepared, having read the material, and ready to participate in the discussion"*
* You have read chapters 1-7
* Who has the book?
* Who has actually read those chapters (not skim, not while watching youtube)
* Let's discuss...

<slide_break></slide_break>

## Let's discuss

**IMVU Case Study**
* Context
  * Messaging is a feature of multiple apps
  * People develop habits around using one or two apps for messaging
  * Original concept was to create 3D avatars as an add-on to bridge all those apps
* What pain point was this supposed to address?
* What are "network effects"?
* What happened when IMVU first launched?
* Discuss why that happened

**Zappos Case Study**

**Dropbox Case Study**

<slide_break></slide_break>

<h2 id="takeaways-from-chapter-6---test">Takeaways from Chapter 6 - Test</h2>

<ul>
  <li>
    <p><strong>BIG IDEA</strong>: The Minimum Viable Product</p>
  </li>
  <li>Goal is to test articulated hypotheses</li>
  <li>MVP different from a prototype, beta test, mockup because of it’s explicit purpose</li>
  <li>Many types of MVP: video, concierge,</li>
  <li>Visionaries or early adopters accept expect a product that is incomplete. They want to be ahead</li>
  <li>Early adopters want to feel special: be the first one on the block with a product, or have an unfair advantage over other companies.</li>
  <li>If you leave out features, customer often fills in the blanks which may suggest missing features</li>
</ul>

<slide_break></slide_break>

<h3 id="summary-of-chapter-6">Summary of Chapter 6</h3>

<ul>
  <li>Introducing MVP: designed to test a hypothesis. Not necessarily a prototype.</li>
  <li>Visionaries or early adopters accept in fact expect a product that is incomplete. They want to be ahead. Early adopters want to feel special: be the first one on the block with a product, or have an unfair advantage over other companies.</li>
  <li>Also realize that if you leave out features, customer will often fill in the blanks in their mind giving you info about what feature to add or not.</li>
  <li>How can you test whether 10% of trial users will install, if you can’t even get trial users?</li>
  <li>MVP: any work beyond whats needed to learn / prove the hypothesis is waste!</li>
  <li>Food on the Table: I thought it an odd idea: delivering customized recipes based on specials in the local grocery store. An extended scenario of an MVP is described. Lessons:
    <ul>
      <li>While CEO was visiting customer to learn what recipes would be worth while, this very inefficient MVP is different from a restaurant where the owner is serving customers personally. In the former case, we are trying to learn. In the case of the restaurant, this is not a test, it is the real product.</li>
    </ul>
  </li>
  <li>Aardvark uses “wizard of oz testing” which to me sounds just like concierge!</li>
  <li>Is the MVP philosophy at odds with ‘quality’?
    <ul>
      <li>No. But don’t build quality, or beauty because you assume it is a customer requirement.</li>
      <li>Sometimes that’s not what matters initially.</li>
      <li>If you discover that your MVP is failing because people think its ugly or cannot figure out how to use it, that’s something else</li>
      <li>Remove any feature, process or effort that does not contribute directly to learning.</li>
    </ul>
  </li>
  <li>MVP thoughts
    <ul>
      <li>Customers don’t care how difficult or costly it is to provide something; and the thing they love most might be very minor in your eyes</li>
      <li>Role of courage: you need to dare to respond to what you learn even if it is not what you thought or intuited.</li>
    </ul>
  </li>
  <li>Commit to iterate
    <ul>
      <li>Even if the MVP fails decide up front that you will pivot and iterate</li>
    </ul>
  </li>
</ul>

<slide_break></slide_break>

<h3 id="discussion-questions-for-chapter-4-6">Discussion Questions for Chapter 4-6</h3>
<ul>
  <li>Preparation
    <ul>
      <li>Someone to present Zappos, Kodak, HP, VLS cases, CFPB,</li>
    </ul>
  </li>
  <li>What are some of the hypotheses that had to be tested in Zappos case? List them.
    <ul>
      <li>Are people willing to buy premium shoes online?</li>
      <li>What kinds of shoes, and prices do better?</li>
      <li>Will customers will appreciate a very liberal return policy over a big discount?</li>
      <li>What kind of photography is required? Will videos help sales?</li>
    </ul>
  </li>
  <li>What is a concierge product and in Zappos case?</li>
  <li>What are “Leap of Faith” Assumptions? What were they for Zappos? What were they for HP? What were they for Kodak? How are they different from other assumptions?
    <ul>
      <li>Those which if not true will cause the business to fail or need to be rethought. The riskiest assumptions. Crux of the vision.</li>
      <li>Zappos: People will buy shoes online</li>
      <li>HP: Employees have a desire to do volunteering</li>
      <li>Whats an MVP? What are the kinds of MVP? What would an MVP be for your project?
        <ul>
          <li>video</li>
          <li>web site</li>
          <li>sign up for beta waiting list</li>
          <li>concierge product</li>
          <li>mockup/prototype/3d rendering</li>
        </ul>
      </li>
    </ul>
  </li>
  <li>What does “Genchi Gembutsu” mean to you? “Go see for yourself”?</li>
  <li>Give an example of analysis paralysis</li>
  <li>Dropbox: Were there previous products? What was the key leap of faith value assumption? How could it be tested in an MVP?
    <ul>
      <li>yes there were lots. But they were all hard to use</li>
      <li>that if the product was seamless to use people would in fact flock to it</li>
      <li>hard to test in an mvp because it was difficult to prove seamlessness</li>
      <li>but then they made a video to fake the product</li>
      <li>added tons of people to the beta waiting list.</li>
    </ul>
  </li>
  <li>What did you think of Food on the Table service?
    <ul>
      <li>What about Craig’s List, eBay? Can you think of products that are kind of known for their roughness? First release of iPod? Original cordless phones, original cell phones? Original laptops?</li>
      <li>Seems very complicated and elaborate</li>
      <li>Is it solving a real problem?</li>
    </ul>
  </li>
  <li>What is the direct connection between the mvp and the hypotheses?
    <ul>
      <li>An mvp should clearly be designed to validate a hypothesis.</li>
    </ul>
  </li>
  <li>What are some MVP Challenges?
    <ul>
      <li>IP protection might be impeded if you reveal your product too early. You might loose your ability to patent</li>
      <li>Competitors might learn about your product and rip it off. In fact the biggest challenge is to be noticed at all! Ideas are cheap!</li>
      <li>Damaging the brand? You should be able to insulate yourself by making sure customers know this is a test or a alpha product, but worse comes to worse, give it a separate identity. Change the name!</li>
      <li>Bad for morale: team gets discouraged by bad news or failure to penetrate</li>
    </ul>
  </li>
  <li>Is the MVP approach at odds with a good quality? What are the risks and are they worth it?
    <ul>
      <li>give the company or the name a bad reputation?</li>
      <li>the product concept may be good but because of bad quality it is not appreciated?</li>
      <li>OTOH in the startup phase we don’t know who the user or the product is so how do we know what quality is?</li>
    </ul>
  </li>
  <li>Is the MVP approach workable for hardware products?
    <ul>
      <li>Why not? Because it costs too much to create an MVP</li>
      <li>No: MVP is not necessarily a prototype!</li>
    </ul>
  </li>
</ul>

<slide_break></slide_break>

<h2 id="takeaways-from-chapter-7---measure">Takeaways from Chapter 7 - Measure</h2>

<ul>
  <li>
    <p><strong>Big Idea</strong>: Engine of growth</p>
  </li>
  <li>Startup needs to measure (the right things)</li>
  <li>Actionable Metrics (vs. vanity metrics)</li>
  <li>Baseline -> Change something -> Measure change</li>
  <li>Growth engine: The things the startup does or has that it believes will lead to growth</li>
  <li>Cohort Analysis: Partition your user/customer base into cohorts based on time</li>
  <li>A/B Test: give different “treatments” to different groups and measure how they react</li>
</ul>

<slide_break></slide_break>

## Case Study

* Take a look at [projects from previous semesters](http://pitosalas.github.io/brandeisprojects/)
* Lets look at a few from last year

<slide_break></slide_break>

<h2 id="mvp-overview">MVP Overview</h2>

<ul>
  <li>“It’s not necessarily minimal nor a product nor viable!”</li>
  <li>MVP is not a perfectly descriptive term</li>
  <li>Smallest increment of work that will <em>test a hypothesis</em> (or more than one)</li>
  <li>You need to be able to say what hypothesis it will test and how you will judge the result.</li>
</ul>

<slide_break></slide_break>

<h3 id="what-forms-can-it-take">What forms can it take?</h3>

<ul>
  <li>The key is that it is <em>MINIMUM!</em></li>
  <li>MVP is always always related to a hypothesis (or more than one)</li>
  <li>Who and where do you show your MVP?
    <ul>
      <li>Product Reviews of other products might give you names</li>
      <li>Other feedback fora</li>
      <li>Think about: where do they congregate?</li>
      <li>Who are <em>they</em>?</li>
      <li>Find a way to observe them</li>
    </ul>
  </li>
</ul>

<div class="discussionbox"><strong>Discussion:</strong> Does MVP mean that quality is not important</div>

<slide_break></slide_break>

<ul>
  <li>Goal is to test articulated hypotheses</li>
  <li>MVP different from a prototype, beta test, mockup because of it’s explicit purpose</li>
  <li>Visionaries or early adopters accept expect a product that is incomplete. They want to be ahead</li>
  <li>Early adopters want to feel special: be the first one on the block with a product, or have an unfair advantage over other companies.</li>
  <li>If you leave out features, customer often fills in the blanks which may suggest missing features</li>
</ul>

<slide_break></slide_break>

<h3 id="types-of-mvps">Types of MVPs</h3>

<p>This is a A catalog of approaches to collecting “out of the building” information on the journey to product market fit. Click on the links to learn more about each approach.</p>

<slide_break></slide_break>

<table class="table table-striped table-sm table-responsive table-hover"><thead><tr><th>title</th><th>description</th></tr></thead><tr><td><a href="/content/topics/leanmvps/adwordstats.md/">Use Ad Tools for metrics</a></td><td>Facebook and Google both provide tools that can be used to do quanitative research</td></tr><tr><td><a href="/content/topics/leanmvps/concierge_mvp.md/">Concierge MVP</a></td><td>Simulate a product or service "by hand"</td></tr><tr><td><a href="/content/topics/leanmvps/elevator_pitch/">Elevator Pitch</a></td><td>Brief verbal or written summary of proposition</td></tr><tr><td><a href="/content/topics/leanmvps/expert_advice.md/">Expert Advice</a></td><td>Ask someone who has a track record of expertise in your area</td></tr><tr><td><a href="/content/topics/leanmvps/group_feedback.md/">Focus Group Feedback</a></td><td>Small group open discussion for feedback and ideas. Face to face lightly moderated meeting.</td></tr><tr><td><a href="/content/topics/leanmvps/landing_page.md/">Landing page</a></td><td>A single page web site, used to convey information and measure interest</td></tr><tr><td><a href="/content/topics/leanmvps/nonfeatureaspects.md/">Non-feature aspects</a></td><td>Test of pricing, naming, target markets etc</td></tr><tr><td><a href="/content/topics/leanmvps/online_survey.md/">Online Surveys</a></td><td>The simplest technique of all: an online survey</td></tr><tr><td><a href="/content/topics/leanmvps/p2p_interviews.md/">One on one interviews</a></td><td>One on one conversations / interviews</td></tr><tr><td><a href="/content/topics/leanmvps/paper_prototype.md/">Paper Prototype</a></td><td>Prototype or mockup of user experience, but on paper</td></tr><tr><td><a href="/content/topics/leanmvps/producthunt.md/">Product Proposal Sites</a></td><td>Many sites are available to test reaction to a product</td></tr><tr><td><a href="/content/topics/leanmvps/social_media.md/">Social media, etc.</a></td><td>Sending targeted questions using social media and other lists</td></tr><tr><td><a href="/content/topics/leanmvps/videos.md/">Video MVP</a></td><td>Videos, screen casts, animations and simulations to explain and get feedback</td></tr></table>

<slide_break></slide_break>

<h2 class="shadow p-3 bg-white rounded">Thank you. Questions?<img class="img-fluid w-100" src="https://picsum.photos/800/100.jpg" /> <small> (random Image from picsum.photos)</small></h2>