.jpg)
Recently, an ABTest function for information delivery was designed and completed, just to summarize and sort out, so as to avoid everyone stepping on the pit.
Everyone has heard of the AB experiment, and it is more common in Internet companies. Before starting to talk about the content, let’s introduce the concept of AB: When you are doing product planning and design, you come up with a revision plan (let’s call it version B), but you are not sure about this. Whether the improvement works better than the older version (called Version A).
So I want to compare the effects of the two versions, so you assign 10% of the user group (assuming 100 people) to plan B and 90% of the traffic to plan A. After a week, check the data; if version B If the effect is better than version A, then prepare to go online. If the effect of version B is not good, then go offline or do the experiment again.
(hand-painted crowd diversion, please understand)
After the story is told, many people may think that it is very simple. Isn't it a single variable in the biochemistry you learned before. If you think so, then your AB experiment should be wrong.
1. Typical problems in AB experiments
Let's take a look at some of the core issues in AB:
How to divide the population, is it randomly divided or according to what rules to ensure the reasonable division of the population? (The diversion logic in the AB experiment);
The experimental results are out, how can I judge whether the results are credible or not (significant differences in the AB experiment);
The experimental results are out, the experimental group data is good, how can I judge whether it is really good (the first type of error in the AB experiment);
The experimental results are out, the experimental group data is poor, how can I judge whether it is really bad (the second type of error in the AB experiment);
The experimental results are out, there are many dimensions of data, how do I measure the experimental results (measurement indicators in the AB experiment);
The experimental results came out, but I always felt unreliable for a group of AB experiments (the AB group of the AB experiment, called the AA group and the AABB group).
Solving the above 6 problems logically country email list completed the planning of the AB experimental plan, and the rest is the product design.
2. How to divide the population reasonably
1. What is crowd division (diversion)
This question is the most critical diversion logic in the AB experiment.
The most important thing in the AB experiment is to ensure the variables. Yes, this is the same as the single-variable method in biochemistry. Except for the different points of your product to be tested, in fact, the population must be guaranteed the same. This does not mean the same flow or ratio. , but the same population characteristics.
For example: a BB cream manufacturer conducted an AB experiment with a total number of 1,200 people. No. 1 BB cream was put into a group of 600 girls (that is, 50% of crowd A was diverted), and No. 2 BB cream was put into another wave of 600 people. The group of men and boys (that is, 50% of Crowd B is divided), and the flow division is the same. The experimental results show that Crowd A is highly accepted. Conclusion: No. 1 BB Cream is more popular in the market!
Is this conclusion credible, not credible, because the characteristics of the people themselves are different, it has nothing to do with the flow rate of 1:1, the ratio of the flow rate will not affect the results of the AB experiment, and the acceptance of BB cream for boys is definitely not high (don't be horny). , say that boys get girlfriends and so on), this example illustrates the impact of population characteristics on the experimental results.
2. How to divide the population (diversion)
First of all, our company has a special crowd diversion system internally. The main default diversion algorithm is uid+hash factor calculation md5 modulo. According to the result, it is judged which experimental bucket it falls in. The system can be directly used, so it is not used here. Repeat.
This time, I will mainly talk about how to deal with companies that do not have a scientific diversion system?
Is also divided into online diversion and offline diversion. Online diversion refers to dividing according to the current real-time characteristics of the current online, and the result is more accurate; offline diversion refers to dividing according to the characteristics of T+1 or T+N (specific N is determined according to different businesses of different companies). The former is recommended when conditions permit.