What do these results mean? Picking a subject line winner
Touchstone virtual tests indicate how much better or worse your subject lines are likely to perform compared to average.
In the example above, a result of "85% Opens" means that based on the performance of similar subject lines, what was tested is likely to perform 85% better. Therefore, if your list average open rate is 12%, a result of "85% Opens" indicates that you could expect to achieve an open rate of 22.2% by using the subject line tested.
This is an indication. The more you use Touchstone, the more you will be able to get a feel for how the results relate to your own lists and segments. For even more precise results, you can sync historical performance data from your campaigns to Touchstone.
In general, Touchstone predictions fall within 1-2% of the true value. So yes, the tool is very accurate.
Testing on our own newsletter
We've tested the accuracy of Touchstone on our own newsletter: testing two subject lines within Touchstone, and then deploying them as A/B tests and seeing whether Touchstone's predicted winner actually performed better.
The basic premise behind the algorithm is relatively straightforward to explain. When you perform a search, here is what happens:
1. The algorithm finds all similar subject lines
Each subject line in our database is assigned a score describing how similar it is to your search, and the most similar subject lines are kept for further analysis.
2. The performance of these subject lines is analyzed
For each subject line that is similar to your search, we compare its performance against its sender's average performance. By always comparing the results of a subject line against a sender's average, we avoid the possibility of a few users with abnormally high or low open rates throwing off the results.
3. All results are combined
The performance results from all these subject lines is combined into a single score. This takes into account send volumes and the similarity score from Step 1 to optimize prediction accuracy.
4. Recommendations are generated
Finally, the algorithm tries dozens or hundreds of modified versions of your subject line (such as replacing "Good deals" by "Great deals") to find recommendations which will most improve your results.
Although it uses an extensive database with hundreds of thousands of subject lines, Touchstone may not always find subject lines which are similar to your search. If this happens to you, you have a few options:
1. If you have not uploaded your data onto the tool, do so. Touchstone will then be able to use your past subject lines in order to provide you with recommendations.
2. If you were searching only on your data, then perform a search on the entire Touchstone database by selecting a sector at the top of the screen.
3. If you have uploaded your data and were searching the entire database but still did not find any results, then your subject must simply be too unique for Touchstone to make an accurate analysis. However, do not despair: research shows that unique subject lines tend to be the most effective.
The short answer is that recommendations take at least a few seconds to compute. Our recommendation engine is very work-intensive and involves testing every combination of synonym and alternate word choice which are similar to your subject line.
For example, if you search for "Great deals", the recommendations engine will look for multiple subject lines such as "Awesome deals", "Great sales", and so on. You can easily imagine the huge number of alternatives our tool goes through before finding the best one. While you wait, however, imagine how long it would have taken had you needed to perform A/B tests on these dozens of combinations.
Some aspects of Touchstone will work in any language, and others are currently available only in American English.
Open, Click and Delivery Rates
The Touchstone search results of open rates, click rates and delivery rates will work in any language. When you test a subject line, Touchstone searches through its database (or through your user-uploaded database) for any subject lines which are similar to yours and analyse how well they performed.
Because language is complex, the word recommendation engine is currently available only in English.
Uploading subject line data is both very easy and immensely powerful. It lets you run Touchstone searches on your own data in order to analyse how your subscribers responded to the subject lines they received.
While searching the entire Touchstone database provides valuable insight regarding new subject line strategies you could be using, searching your own database provides insight on your current writing strategies, and helps you identify what works and what doesn't work. You should therefore ideally be using both approaches.
First, you need to download data from your ESP. At this stage, you want to create a spreadsheet with 1 row per mailing, with each row containing the subject line, mailing date, send volume, number of opens and number of clicks.
The specific procedure to download this data will vary by ESP and we are in the process of developing ESP-specific guides to help you out.
Step 2: Upload Data
Once you've gathered your data, log into your Touchstone account and follow the Upload Data link at the top of the screen.
Step 3: Identify Columns and Date Format
Finally, follow the instructions on the Upload page to upload your file and identify your columns. Play close attention to the "Date" field and ensure that it is in the correct format.