CXL Institute Growth Marketing Minidegree. Part (5/12)
This week I just did the A/B Testing Mastery course. The instructor was Ton Wesseling, an expert with more than 20 years of experience working on Experimentation and A/B Testing.
A/B Testing mastery course:
I want to start by saying that this course is really dense and you need to be so concentrated to understand a lot of think. You need to think a lot so I recommend all of you to get a cup full of coffee, turn down the TV, Music, etc… grab your earphones and start the course in a silent environment.
The topic of A/B Testing is something that has been done for quite some time, many many years ago, but that started to have great importance and popularity in 2010, the data analysis along with the improvements and new internet products were intensifying the use of these tests to get better business results and improve the user experience in the websites.
If we want to make tests with a scientific and statistical basis and that our investments in digital business changes are minimal, that is, try to experiment to make mistakes as quickly as possible and know what things work and what things do not work so as not to waste resources unnecessarily and not waste time on it. If we find a good winner and if we find a good loser also because in this way we can know what not to do and not repeat errors.
— -
The course is divided into 3 main parts.
1. Planning the Test
2. Running the Test
3. Analyze the test results
We will be reviewing them one by one in more detail because each phase is super important and for example if we plan badly the test we may be testing a hypothesis that does not mean any benefit to the business but rather it is only a waste of time.
1. Planning the test:
This is the first phase and it must be taken into account that it must be built correctly because the other test faces depend 100% on the experiments being planned correctly. If we make a wrong hypothesis, the results we will obtain will not be very useful.
In order to pose a hypothesis correctly we must take things into account:
We need enough data to be able to perform the experiment. If we have at least 1000 conversions per month we can start to perform A/B tests.
We must decide which is the KPI we want to improve or monitor at the time of performing the test and get results.
We must understand the client’s needs to be able to pose the correct hypothesis and for this we must investigate all the stackeholders that use the product/service.
At the moment of creating the Hypothesis we think about it in the following way:
If I APPLY THIS, then THIS BEHAVORIAL CHANGE will happen, ( among THIS GROUP ), because of THIS REASON.
We must give priority to the experiments that have a greater impact on the business, but for this we must see the time and resources that it means to carry it out. Several small experiments may have a greater impact than a medium-size experiment.
These are the things we should consider when planning our experiments. This is very broad because I am leaving out all the details. If you want to see the details I recommend you to sign up for the course because I would really be about 7 days writing this post haha. And this blog is more than anything to tell my experience and summarize the content of the courses.
2. Execute the Test:
This is the second phase where we must execute and perform the test.
In general a good practice is only to perform a challgener 1 variable at a time.
It is recommended to use the Google Optimize tool that is free for the test. Although you can choose some paid if you prefer.
We must calculate the duration of the test. We can do it with the following calculator https://abtestguide.com/abtestsize
We have to monitor the progress of the tests as the time of the experiment progresses. If an experiment was not a winner we must cut and perform another with other variables. The possibility that that non-winning experiment is a winner if we repeat the experiment is almost nil.
3. Analyze the results:
To analyze the results it is recommended to use the Bayesian calculator https://abtestguide.com/bayesian/
Generally experiments are done with a significance of >90% which we would consider as a significant result and so we could choose a winner. The higher that percentage, the higher the probability that the winner will actually be a winner.
Apply the results to the business case. Show the impact this change can have on the business’ profits.
These are the 3 points that we must take into account as a summary.
— -
It is worth mentioning that probably not all of us will work directly on this, since we have different roles in the growth teams, but it is very important to understand all these concepts and processes in order to understand what is being done and discussed.
What if the methodology is the same for everyone and the quick experimentation will be important at the moment of fulfilling the business objectives.
— -
To finish, I want to mention how important this course has been for me. I started working as a Digital Product Manager recently and I have already applied several things I have learned in these courses, even though I have not taken even 15% of the total of the master degree. I hope to continue learning at CXL Institute.
Unfortunately, as it is the end of the year, I have not had much time to study and take the courses because at least in my country, at this time of year, people work more than 9 hours a day, which is unfortunate. But well that is the reality that we live to some where the mental health of the worker is not so important but that the only thing that matters to the companies is to make profitable their business without concerning the cost that implies. I hope someday to be able to go to work outside of my country to have a balance between my personal life and my work life. At the moment my working life is the one that consumes most of my time unfortunately.
See you next week!