Different Algorithms for 50-50 A/B Testing

Question

We are running A/B tests on web app customers, given a customerId. Each customer will see different web-feature designs. Trying to prevent usage of Feature Flags as its not currently setup yet in our system.

Initially we tried Even-Odd on CustomerId number, 50-50% ratio to test Feature 1. Example UserId 4 is even, 7 is odd. However, when testing another Feature 2, doing Even-Odd 50-50% would make , Feature 1 Groups to have a matching group with Feature 2, as they both share Same algorithm.

What is another mathematical algorithm method, to run a hash or 50-50% algorithm , so I can differentiate? We will have probably 10 Features to test, so need a way to add a parameter in the FeatureFlag Algorithm, and will track in a Document Table.

We are assigning groups with Javascript/Typescript btw.

Note: Groups should be steady and not random , eg Even-odd will give a consistent result.

Answer 1

You can reformulate your previous even/odd split as bit testing of the binary representation of the customer ID: for the first feature, you took the bit at the first position (the least significant bit) and assigned the groups according to its value.

You can then extend the same approach to define new groups so that you obtain splits that don't correlate with the previous splits: for the nth feature, take the bit at the nth position and assign the groups according to their values. This ensures that the groups are independent for every feature in a deterministic and reproducible way.

In Javascript it would be something like this:

const groupId = (customerId & (1 << featureNumber)) === 0 ? 0 : 1;

Where << is the bit shift operator and & is the bitwise and operator, and featureNumber is the order of the specific feature you are testing (starting at zero). The result of the bitwise-and is either 0 or (2 << feature_number). groupId would be either 0 or 1.

This approach, of course, is only valid if the number of customers is enough to fit the bits for all features, that is, at least 2048 ($=2^{11}$) customers.

One minor problem with this approach would be that the partitions will probably not lead to an exact 50/50 split, because your number of customers will probably not be an exact power of 2.

Answer 2

The below getFeatureAssignment function first generates a hash value for the customer ID using the hashCode function. It then performs a modulo operation on the hash value with the total number of partitions to get the partition number. Finally, it compares the partition number with the threshold value to determine whether the customer is assigned to the feature or not.

The hashCode function is used to generate a hash value for the customer ID. It iterates over each character in the string and generates a hash value using the djb2 algorithm. However you can replace this as needed.

function getFeatureAssignment(customerId: string, numPartitions: number, threshold: number): boolean {
  const hash = hashCode(customerId);
  const partition = hash % numPartitions;
  return partition / numPartitions < threshold;
}

function hashCode(str: string): number {
  let hash = 0;
  for (let i = 0; i < str.length; i++) {
      const char = str.charCodeAt(i);
      hash = ((hash << 5) - hash) + char;
      hash = hash & hash;
  }
  return hash;
}

This implementation ensures consistent group assignments for each feature and allows for easy addition of new features. The modulo operation with a prime number close to the total number of partitions ensures an even distribution of partitions.

The new groups would obtain splits that are independent of the previous splits for every feature in a deterministic and reproducible way. The threshold value can be adjusted for each feature to ensure that any slight imbalances in partition sizes are accounted for, without affecting the deterministic and reproducible nature of the group assignments.