1. Background

The data-driven growth methodology has helped companies like Facebook and Google maintain efficient business growth. In China, ByteDance is the company that values data-driven approaches most and has achieved tremendous success as a result. In the internet industry, using data to drive business growth has become a consensus, and data has become an indispensable and reliable means for company development. A/B testing is the core method of data-driven growth, effectively improving traffic conversion efficiency and product iteration efficiency. Currently, the company conducts many A/B experiments, but manual experimentation is inefficient, so we built an ABTest platform.

II. A/B Testing Introduction

A/B testing is a comparative analysis method that eliminates interference from other factors through traffic segmentation and random experiments, and monitors and tracks experimental effects to determine the feasibility and effectiveness of the strategy represented by the experiment.

  • Comparison: Through random segmentation of traffic, effective independent random experiments can be conducted, eliminating the influence of external conditions, making it more scientific.
  • Analysis: Through tracking and analyzing the actual effect data of experiments, the feasibility and effectiveness of experiments can be determined, making it more precise.

The following diagram shows a simple A/B testing process:

The current population is randomly divided into two parts. One group of users is shown page A, and the other group is shown page B. By observing the final conversion rate of each group, we can determine which format is better. The diagram clearly shows that page A has a higher conversion rate than page B, so we will ultimately adopt the page A solution for all users.

A/B Testing Process Diagram

Application Scenarios

A/B testing solves the problem of strategy optimization—finding the optimal strategy from multiple options. Common application scenarios include:

  • Gray release: Technology & algorithm iteration
  • Feature optimization: Interface modules, style, interaction methods, etc.
  • Content optimization: Promotional posters, landing pages, content modules, copywriting, etc.
  • Operations optimization: Operational strategies, communication scripts, etc.

Core Concept - Traffic Splitting

Referring to traffic usage rules from Google, Meituan, Youzan and other companies, we designed traffic splitting rules that fit our company's current stage.

We introduced two concepts: traffic domain and traffic layer:

  • Traffic domain: Traffic between traffic domains is independent and never intersects.
  • Traffic layer: Traffic between traffic layers is orthogonal. Traffic at the same level is the same traffic, just randomly shuffled when entering another layer, ensuring traffic orthogonality.
  • Traffic domains and layers can be nested infinitely, ultimately achieving traffic segmentation.

Traffic Splitting Rules Introduction

As shown above:

Domain A accounts for 80%, Domain B accounts for 20%. These two traffic portions are independent—a user will only be in one domain.

Layers 1, 2, 3, and 4 are all the same traffic, just randomly shuffled. The ordering of the 4 traffic layers is different. Experiment A uses Layer 1 traffic for testing. If Layer 2 is used for experiments, traffic is shuffled into Layer 2, and each experiment group in Layer 2 will test Experiment A at the same proportion. If two experiments affect the same result, they will interfere with each other. However, since the interference is at the same proportion, the results remain valid, but accurate quantitative analysis cannot be derived—only qualitative conclusions.

Experiment H is conducted in Domain B, which is purely independent with no other experiments, so the conclusion from Experiment H is an accurate quantitative analysis result.

Experiments E and D are conducted in Domain A3 and Domain A2 within Layer 4, so the traffic for these two experiments is completely independent, and their results don't interfere with each other. However, both experiments will be affected by experiments in Layers 1, 2, and 3.

Traffic domains and layers can be nested infinitely. If your traffic is large enough, you can split it infinitely to accommodate complex business A/B testing. This is the core traffic splitting rule of the ABTest platform.

Therefore, everyone can choose the corresponding traffic domain and traffic layer for A/B testing based on their business situation, achieving effective A/B testing, outputting accurate A/B testing conclusions, helping businesses find the right direction, and truly achieving data-driven growth.

III. ABTest Technical Solution

AB Testing System Technical Solution

As shown above, the AB testing system first configures experiments from the web backend, then the middle platform server performs traffic allocation, stores and records behavioral operations, then provides experiment configuration and traffic splitting services to the business layer, and finally the client requests the business and displays it to users.

Data Recovery: The client reports user behavior logs to LogHub, records log data in ODPS, calculates data metrics, and finally flows data back to data reports, displaying them in the AB testing platform.

Data Flow

The data flow is responsible for collecting ABTest system-related logs and producing offline data reports.

Core tracking information:

  • Experiment status: expStatus; Enum values: Pending 0, Deleted 1, Online 2, Offline 3, Full 4
  • Control type: expControlType; Enum value: Client