In the Motiva Message Testing AI results, the rate chart plots the expected open or clickthrough rate over time of every email variation being tested. In the following example, there are four options plotted over 5 days:
It's a bit difficult to see in the screenshot, but there are lightly colored areas around each line in the chart. These areas represent the uncertainty interval for each option over time. Motiva believes the true rates will fall within these intervals 99% of the time.
In our app, to get a better look, hover over one of the lines in the chart to highlight that option's uncertainty interval. In the example below, we've hovered over the blue option. As you can see, the uncertainty interval gets smaller over time as Motiva gets more confident in its estimates.
On the 1st and 2nd days, the uncertainty interval overlaps the other options quite a bit. Towards the end of the campaign, as Motiva becomes more confident that the blue email is the best, the uncertainty interval only minimally overlaps the others.
The degree of overlap you see between the various options' intervals gives a visual indication of how confident Motiva is in the differences in performance.
โ
If intervals for two particular options overlap significantly, their performance is quite similar. If one email option outperforms all others with high confidence, the interval for that option will be higher and minimally overlap the others.ย
Motiva collects just enough evidence to be confident that a particular email option is not the best, thereby accelerating the learning process.
โ