A/B test implementation checklist

Use this checklist to avoid common issues with statistical significance and unexpected test results.

Check your events

Check that all events are in the right format and are valid. For more information, see Validate your events.

Identify your users

Link click and conversion events with user profiles by assigning a unique identifier, userToken, to each user.

Handle backend search users

If you use backend search, since all searches appear to come from your server’s IP address, Algolia treats them as coming from a single user.

To differentiate between users, include the userToken or the user’s IP address when forwarding queries from your server to Algolia.

Check Personalization implementation

If you’re using Personalization, check it’s working as expected with the Personalization implementation help page on the Algolia dashboard. Your queries should have associated user tokens.

Personalization is available on the Build and Premium pricing plans.

Handle anonymous users

Sometimes, you may have a mix of anonymous and identified users. For example, you may have assigned an anonymous token to users who haven’t yet accepted cookie consent. Since anonymous users share the same user token, Algolia treats them as coming from a single user.

To avoid skewing results, turn off A/B testing for queries from anonymous users. For example, the following checks for an anonymous user token (null, undefined, or YOUR_ANONYMOUS_USER_TOKEN) and appends the { "[enableABTest](parameter_url('enableABTest'))": false } request option to the outgoing query.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// Set the query and get the current user token
const query = 'User search query';
const userToken = getUserToken();

// Clear the request options
let requestOptions = {};

// Is the user token anonymous?
if (userToken === null || userToken === undefined || userToken === 'YOUR_ANONYMOUS_USER_TOKEN') {
  // Disable A/B testing for this request
  requestOptions.enableABTest = false;
} else {
  // Set the user token to the current user token
  requestOptions.userToken = getUserToken();
}

// Perform the search
index.search(query, requestOptions)
  .then(result => {
    // Search results
    console.log(result);
  })
  .catch(error => {
    // Search errors
    console.error(error);
  });

Exclude internal searches

Exclude non-user searches (such as from a dashboard or internal page) from A/B testing by setting enableABTest to false.

Avoid overriding A/B test settings with query parameters

For accurate results, don’t change any query parameters you initially set in the A/B test. For example, you turn on Personalization for variant B and all queries. This wouldn’t give meaningful results because all users see the personalized experience, making it difficult to tell if variant B has any impact.

Manage bot traffic

Bots crawling your web pages and performing searches will generate outlier traffic. Although outliers are automatically removed from the A/B test results, you should still take steps to avoid them because they can affect the click-through rate.

To control and restrict bot activity and avoid skewing results:

• Set rate limits. Set API keys with rate limits to control the number of API calls per hour and IP address.
• Use HTTP referrers. However, don’t rely solely on referrer URLs sent through the referer or Origin HTTP headers to secure your data. Malicious actors can tamper with them, compromising your security, and some browsers, like Brave, don’t send these headers.
• Update robots.txt. Configure robots.txt to prevent bots from accessing your search pages.