Lab 7: Great British Bake Off (A/B Test)¶

Welcome to Lab 7! This week’s lab will focus on A/B Testing using data from the popular British television show, The Great British Bake Off.

Helpful Resource:¶

• Python Reference

Recommended Readings:

• Error Probabilities

• A/B Testing

Lab Submission Deadline: Friday, March 20th at 5pm

Getting help on lab: Whenever you feel stuck or need some further clarification, find a GSI or tutor, and they’ll be happy to help!

As a reminder, here are the policies for getting full credit if you are in regular lab (Lab is worth 20% of your final grade):

• 80% of lab credit will be attendance-based. To receive attendance credit for lab, you must attend the full discussion portion (first hour) at which point the GSI will take attendance.

• The remaining 20% of credit will be awarded for submitting the programming-based assignment to Pensive by the deadline (5pm on Friday) with all test cases passing.

If you are in self-service lab, 100% of credit will be awarded for submitting the programming-based assignment to Pensive by the deadline (5pm on Friday) with all test cases passing.

Submission: Once you’re finished, run all cells besides the last one, select File > Save Notebook, and then execute the final cell. The result will contain a zip file that you can use to submit on Pensive.

Let’s begin by setting up the tests and imports by running the cell below.

0. Mid-Semester Survey¶

We would like you to fill out the mid-semester survey before proceeding any further in this assignment.

Please fill out the survey below and input the secret phrase that is shown at the end of the form when you submit. Please assign this phrase to mid_secret as a string in the cell below!

Find the survey here.

1. A/B Testing¶

A/B testing is a form of hypothesis testing that allows you to make comparisons between two distributions — distribution of group “A” and distribution of group “B”. We may also refer to an A/B test as a permutation test.

You’ll almost never be explicitly asked to perform an A/B test. Make sure you can identify situations where the test is appropriate and know how to correctly implement each step. Oftentimes, we use an A/B test to determine whether or not two samples came from the same underlying distribution.

Question 1.1. The following statements are the steps of an A/B hypothesis test presented in a random order:

1. Choose a test statistic (typically the difference in means between two categories)

2. Shuffle the labels of the original sample, find your simulated test statistic, and repeat many times

3. Find the value of the observed test statistic

4. Calculate the p-value based off your observed and simulated test statistics

5. Define a null and alternate hypothesis

6. Use the p-value and p-value cutoff to draw a conclusion about the null hypothesis

Assign ab_test_order to an array of integers that contains the correct order of an A/B test, where the first item of the array is the first step of an A/B test and the last item of the array is the last step of an A/B test.

Question 1.2. If the null hypothesis of an A/B test is correct, should the order of labels affect the differences in means between each group? Why do we shuffle labels in an A/B test? If you are in a lab section, confirm your answer with a neighbor or staff member before moving on.

Type your answer here, replacing this text.

2. The Great British Bake Off¶

“The Great British Bake Off (often abbreviated to Bake Off or GBBO) is a British television baking competition, produced by Love Productions, in which a group of amateur bakers compete against each other in a series of rounds, attempting to impress a group of judges with their baking skills” Wikipedia

For every week of the competition, the judges assign one contestant the title “Star Baker”. Ultimately, one winner is crowned every season. Using this information, we would like to investigate how winning Star Baker awards affects the odds of winning a season of the show.

Question 2.1. We want to know whether winning more Star Baker awards causes an increase in likelihood of winning a season. Why is it not sufficient to compare star baker rates for winners and losers?

Type your answer here, replacing this text.

Running an Experiment¶

We are going to run the following hypothesis test to determine the association between winning and number of Star Baker awards. The population we are examining is every contestant from seasons 2 through 11 of GBBO. We are going to use the following null and alternative hypotheses:

Null hypothesis: The distribution of Star Baker awards between contestants who won their season and contestants who did not win their season is the same.

Alternative hypothesis: Contestants who win their season of the show will win more Star Baker awards on average.

Our alternative hypothesis is related to our suspicion that contestants who win more Star Baker awards are more skilled, so they are more likely to win the season.

Question 2.2. Why is it appropriate to use an A/B test to test this hypothesis? What is the “A” group and what is the “B” group?

Type your answer here, replacing this text.

Check your answers with your neighbors or a staff member before you move on to the next section.

The bakers table below describes the number of star baker awards each contest won and whether or not they won their season (1 if they won, 0 if they did not win). The data was manually aggregated from Wikipedia for seasons 2-11 of the show. We randomized the order of rows as to not spoil the outcome of the show.

Question 2.3. Visualize the distribution of Star Baker awards for winners and non-winners as overlaid histograms. You should use the bins we provided.

Hint: You will want to use the group argument of tbl.hist. In order to produce several overlayed histograms based on unique values in a given column, we can do something like tbl.hist(..., group=<col_name>, bins=...). This will graph one histogram for each unique value in the specified column all on a single plot.

Question 2.4. We want to figure out if there is a difference between the distribution of Star Baker awards between winners and non winners.

What should the test statistic be? Which values of this test statistic support the null, and which values support the alternative? Assign test_option to the number corresponding to the correct test statistic.

1. Absolute value of the difference between the means between both groups; high values support the null

2. Absolute value of the difference between the means between both groups; low values support the null

3. Average Star Baker awards for winners - average Star Baker awards for non-winners; high values support the null

4. Average Star Baker awards for winners - average Star Baker awards for non-winners; low values support the null

Before moving on, confirm your answer with a peer or in the discussion forums.

Hint: You should think about what measures we use to describe a distribution.

Question 2.5. Create a new table called means that contains the mean number of star baker awards for bakers who did not win (won==0) and bakers that did win (won==1). The table should have the column names won and star baker awards mean.

Question 2.6. Set observed_difference to the observed test statistic using the means table.

Question 2.7. Given a table tbl (like bakers), a label column label_col, and a values column val_col, write a function that calculates the appropriate test statistic.

Hint: Inside the function, you will most likely want to make a new means table based on the function’s arguments. However, avoid using the original means table itself, since this assumes you will always use the same bakers table.

Hint: Make sure that you are taking the directionality of our alternative hypothesis into account.

When we run a simulation for A/B testing, we resample by shuffling the labels of the original sample. If the null hypothesis is true and the star baker award distributions are the same, we expect that the difference in mean star baker awards to not change when "won" labels are changed.

Question 2.8. Write a function simulate_and_test_statistic to compute one trial of our A/B test. Your function should run a simulation and return a test statistic.

Hint: Textbook chapter 12.1.4 can help!

Question 2.9. Simulate 5000 trials of our A/B test and store the test statistics in an array called differences.

Run the cell below to view a histogram of your simulated test statistics plotted with your observed test statistic as a red dot.

Question 2.10. Find the p-value for your test and assign it to empirical_p.

Question 2.11. Using a 5% p-value cutoff, draw a conclusion about the null and alternative hypotheses. Describe your findings using simple, non-technical language. What does your analysis tell you about the association between star baker awards and winning? What can you claim about causation from your statistical analysis? Confirm your answer with a peer, instructor or in the discussion forums.

Hint: See section 12.1.6

Type your answer here, replacing this text.

You’re done with lab!

Important submission information:

• Run all the tests and verify that they all pass

• Save from the File menu

• Run the final cell to generate the zip file

• Click the link to download the zip file

• Then, go to Pensive and submit the zip file to the corresponding assignment. The name of this assignment is “Lab XX Autograder”, where XX is the lab number -- 01, 02, 03, etc.

• If you finish early in Regular Lab, ask one of the staff members to check you off.

It is your responsibility to make sure your work is saved before running the last cell.

Pets of Data 8¶

Suki and Sandie hope you had fun doing this lab!

Congrats on completing Lab 7!

To double-check your work, the cell below will rerun all of the autograder tests.

Submission¶

Make sure you have run all cells in your notebook in order before running the cell below, so that all images/graphs appear in the output. The cell below will generate a zip file for you to submit. Please save before exporting!