A/B tests
An A/B test tries two to four versions of one channel's posts against each other, and tells you which did better. It sits under Performance in the menu and is part of Enterprise. On every plan below it is not in the menu.
What a test tries
A test runs on one channel's feed or custom feed: a source that makes a new post of the same template again and again, which is what versions can be compared over. A post list is not offered, because each of its posts chooses its own template.
Each version is a cut of the template and a brand, either of them the channel's own:
- Cuts of a template differ in how the clip is made: how it opens, its length, its layout.
- Brands differ in what the clip wears: a brand with its outro against one without, a soundtrack against silence, another logo or colour.
Two versions must not look alike: give each a different cut or a different brand. If the template has only one cut, the versions differ by brand.
How a test runs
While a test runs, every new post the feed makes for that channel goes out once, in one version: the version with the fewest posts so far, a tie drawn at random. The same content is never posted twice on one account. That would not be a fair test: each post gets its own chance from the platform's algorithm at its own moment, the second post reaches people who already saw the first, and platforms push down content that looks reposted.
So a test builds up over many posts. Each post is read at the same age (3 days after it went up, or 7) from the numbers reported for it, the way Analytics reads them. A test can only decide on numbers that were sent, so make sure your posts' numbers reach Reelwire.
The channel's own settings do not change while a test runs. The test lays its versions over them, post by post. Posts from the channel's other sources are not part of the test.
What decides
When you start a test you choose what decides it. The suggested measure depends on the platform:
| Platform | Suggested | Also possible |
|---|---|---|
| TikTok | Watched to the end | Share of the clip watched, watch time, engagement and shares per view, views |
| Instagram Reels | Share of the clip watched | Skipped in the first seconds (lower is better), and the others |
| YouTube, Facebook, LinkedIn | Share of the clip watched | Watch time, engagement and shares per view, views |
| Telegram | Views | Engagement and shares (forwards) per view |
The share watched and how many watched to the end tell far more about a version than views. One post in fifty travels, and a test decided on views is usually decided by whichever version happened to get that post. Views are still offered, and compared in a way that keeps one post from deciding alone.
If the versions differ in length, look at the watch time beside the share watched: a shorter clip is watched to a larger share without being watched longer.
Reading a test
The Result of a test is one of:
- Collecting: fewer than 20 posts of a version have numbers at the test's age yet.
- Too few reported: fewer than eight in ten of the posts old enough have numbers. The result waits, so the posts nobody reported cannot tilt it.
- The same: every version is within 5% of the best, either way. The difference is too small to matter, or there is none.
- Winner: one version is the best with a chance of at least 95%.
- No winner yet: none of the above. Keep it running; more posts narrow it down.
Most tests need weeks. At around 20 posts a week on a channel, 20 posts per version with numbers is a fortnight for two versions, and a small difference needs many more. "No winner yet" is a common and honest answer.
Details shows the versions side by side: how many posts each made and how many have numbers, the average, the difference from the first version with the range it lies in with 90% certainty, and the chance each version is the best. The range and the chance appear once every version has 20 posts with numbers.
Ending a test, and keeping a version
- End stops the test. New posts go back to the channel's own cut and brand. The test stays in the list, and its result keeps counting numbers still reported for its posts.
- Keep a version makes one version the channel's own, and ends the test if it runs: its cut becomes the channel's cut for that feed, and its brand becomes the channel's brand. The dialogue says what changes before anything does. The brand belongs to the channel, so its posts from every other source wear it too. You can keep any version, not only a winner.
- Delete, once a test has ended, takes it out of the list. Its posts and their numbers stay.
Keeping a version is the only time a test changes a channel. Everything else about a test is set on this screen.
Start an A/B test
Start an A/B test asks:
- Channel and feed. One test per feed at a time.
- The versions. Two to four, each with a name, a cut and a brand.
- How it is decided. The measure, and whether posts are read 3 or 7 days after they went up.
- Name. Optional. Left empty, the test is named after the channel and its versions.