Computer Dunce Discovers
The "Taguchi Huchi!"
In this issue:
- My unbiased review of a powerful new FREE tool that makes optimizing your web pages easier than ever …
- Scientific online advertising for the technically challenged…
- Why reading this short article might be the most profitable thing you do this year …
- And more!
Dear Web Business Builder,
On Friday of last week, I decided to give Google’s brand-spanking-new Website Optimizer service a test drive. It’s a tool that allows you to conduct experiments with the various elements you’ve got on your web page to see which ones are best at getting visitors to take action.
Here’s my review …
Let’s say you’ve written several headlines for your web page, and you want to find out which headline gives you more sales. Or maybe you want to find out if a headline and image combination out pulls a straight text headline. Or perhaps you want to experiment with alternative leads, or calls to action etc.
Then Google’s new tool is for you.
Website Optimizer uses something known as multivariate testing. It’s a fancy shmancy name that simply means you can test multiple variables at the same time.
Here’s how it works…
You simply paste a few snippets of code that Google gives you onto the pages of your website where you want to test. Then you select the specific elements on that web page that you want to test (headline, lead, guarantee, image, whatever).
Next, you go into your HTML editor, and replace the existing element with a variation. You don’t need to save it or make any visible changes to your web page. You’re simply generating HTML code that corresponds with that new variation, so you can paste that bit of code into the Website Optimizer console.
Once that’s done — like magic — Google’s servers start inserting those variations into your web page in real time and rotating them. If you’ve chosen to test two elements at the same time, let’s say a headline, and a guarantee, then Google rotates all four combinations.
Visitor A sees headline 1 and guarantee 1… visitor B sees headline 1 and guarantee 2… visitor C sees headline 2 and guarantee 1… and visitor D sees headline 2 and guarantee 2, and so on. And all of this happens miraculously on just one web page!
Think of what you can do
with a tool like this …
I had a long copy web page running the equivalent of about 26 printed pages in length, with the first order link appearing on around page 21. I wanted to see how many people were making it all the way down to the order link and clicking through, and what I could do to increase that number.
Obviously the more people who clicked on that order link (provided I didn’t give them some other reason for doing so, other than to buy), the more sales the page would produce… With relative ease, I was able to find out how many were clicking through and begin experimenting to increase that number.
Here’s how I used Website Optimizer to do it.
Since the first decision somebody normally makes when they arrive at a website is whether or not they’re going to hang around and read it, I decided to start with the headline. People read the headline, and they decide if they’re going to continue. The more people who make a decision to do so, the more people will click on the order link.
So I created a whole new main headline, including deck copy, and added a pre-head.
On this particular web page, the first several hundred words after the headline are credibility copy followed by a number of fascinations and then a sub-head. Under this sub-head the salutation appears, and the sales letter begins.
I figured if someone makes it past the main headline, they would probably read at least some of this credibility copy, and then very naturally skim down the page until they hit this first sub-head. It felt like a decision point for me. So I decided to test different variations of that sub-head.
Here's what my console looked like when I was done setting up the test.
If you’re familiar with Adwords you can see how Website Optimizer is integrated nicely into the same console. What you should know though, is that you can use this powerful tool with any kind of traffic you like. Isn’t that cool?
You can see how 4 combinations resulted from testing two variations of my headline, and two variations of my sub-head. If I had chosen to test 3 elements, each one with two variations, the number of combinations would have jumped to 9.
The number of combinations is a multiple of the number of elements tested, and the number of variations per element. So if I had chosen to test 3 headline variations, and 3 sub-head variations, I would have again had 9 combinations. Website Optimizer figures it all out for you, and will let you test up to 40 combinations.
That’s a lot of combinations, which begs the question: How practical is multivariate testing? How long does it take to obtain meaningful results? Excellent questions. Google has answers …
And to be frank, this is one part of the tool that I’m going to be watching very closely. According to the user manual, Google uses something called “a full factorial design” to determine the length of an experiment, and will actually stop the experiment when a statistically valid conclusion is reached.
This duration is a function of the number of samples that have been measured, and the “spread” of the various elements. Spread meaning the relative difference in conversion performance between one element (or combination of elements) and another. The wider this spread, the more likely it is that the apparent winner is truly the winner.
My experience with Google's assessment of statistical validity isn’t entirely positive. With the Adwords serving optimization feature I feel they are far too quick to declare a winner, and so I turn this feature off religiously.
Hopefully they’ve got a better handle on statistical validity here, because once an experiment has been stopped, there is no way to turn it back on. You can effectively clear the data by copying the experiment with a new name, but you cannot continue running the same experiment to collect a larger sample of contiguous data.
Here’s what the console looked like after a few days …
The presentation of data is nice. In the far left column, you can see the various combinations you’re measuring (to see the actual pages you can click on the respective links, or see the names of each element simply by hovering over the link)… and proceeding to the right you’ve got all of the pertinent information you need to assess the results of your experiment. When you click on the “Page Sections” tab you see a similar presentation of data for each one of the elements in isolation.
As mentioned, Website Optimizer uses a full factorial design to determine the statistical validity of these tests.
What this means is that all of the variables you select will be tested as equally as possible for the duration of the test. Fractional factorial methods like Taguchi on the other hand, are designed to limit the number of combinations tested through some sort of ultra advanced predictive algorithm, thereby reducing the amount of traffic required to arrive at a “valid” test conclusion. Some swear by it. Others are skeptical. The folks at Google seem to be in the second camp.
Personally, I like to err on the conservative side, and run the tests for a good long time. Earlier in this experiment one of the challengers was showing a 35% improvement. To Google’s credit, the test continued to run. Even after 549 page views and 110 conversions, it wouldn’t surprise me in the least to see combination 2 (currently trailing the leader by 21%) bounce back. There’s just not near enough data here yet to draw any kind of a conclusion.
I realize this all sounds very complicated and time consuming, which leads me to this next well-considered question …
Why bother with
multivariate testing?
First of all, with Website Optimizer, it really isn’t much trouble to do. Google has done a great job of making it brain dead simple. It’s actually super easy to set up and run.
I’m butt stupid when it comes to computers and I had it all set-up and was running my first experiment in just a few hours — without even reading the manual from beginning to end. I’m allergic to reading manuals that way.
Second, I really believe this new tool actually simplifies the whole process of mapping out a test. I’ve done lots of simple split tests using services like 1ShoppingCart and Hypertracker that use links to split traffic between different pages. And having everything on one page like you do with Website Optimizer seems a lot easier to keep track of and manage.
Third, with multivariate testing, you have the ability to assess the interaction of the various elements you’re testing. That’s impossible to do with conventional split testing. With simple A/B, or A/n split testing, you can actually declare a winner in one split test, move on to the next test, and create a second winner that would have been more effective in combination with the loser of the previous test.
And fourth, once in a while you’ll come upon a real breakthrough. Even when you don’t, provided you run the tests for long periods of time so that you can be sure of their validity, you can add a number of small improvements together to create a big improvement over time.
Of course, my first Website Optimizer experiment has only been running for a short time. There may be more things to rave about I haven’t yet discovered. On the flip side …
One thing that Website Optimizer is not particularly good at is testing pages that are radically different from one another. In the case of doing a complete overhaul on the copy in an effort to create a new control (often the case), you’re better off using a more conventional method to obtain your control, and then start tweaking that new control with Website Optimizer.
Another downside is the ability to track only one action. It would be nice to be able to track more than one step in the sales funnel simultaneously. One example of this might be to judge the impact of squeeze page copy on both opt-ins AND sales. But at this point that doesn’t seem possible.
I tried to get around this limitation by using two different experiments, but was foiled. After creating a squeeze page experiment — which recorded a conversion when the visitor landed on the sales page — Website Optimizer refused to allow me to create a second experiment that would test various elements on that same sales page.
I thought that was a real bummer, but hey, it’s free. All in all, it’s one kick ass tool.
Mastering the art of Scientific Web Advertising
should be one of your top priorities right now …
Scientific testing, and kaizen, (gradually and orderly improvement) on your website are incredibly important.
Here’s why …
By continually testing, you’re almost certain to improve your lead and sales conversion rates. When that happens, it allows you to increase your traffic bids, and attract more prospects to your web site at break even.
More traffic at break even means you can complete your tests faster, thereby increasing the effectiveness of your sales process at warp speed. And then the whole virtuous cycle perpetuates itself.
As you test, you improve. As you improve, you can afford to pay more for traffic. When you can afford to pay more for traffic, you get more traffic. Pretty soon you’re building your customer base faster than anybody else in your niche — and dominating it!
All because you took the time to read Web Marketing Advisor and try Google’s new free service. Ya gotta love it!
Claude C. Hopkins would have killed for this.
So what’s your take on multivariate testing? Is Google wrong about Taguchi? Got questions about Website Optimizer? Or anything else to do with scientific web marketing? Your comments and questions are welcome.
Until next time, Good Selling!
Daniel Levis
Editor, The Web Marketing Advisor
THE TOTAL PACKAGE
Looking for resources related to this article? Try some of these.
Looking for more of Daniel’s articles? Check these out.
Looking for past issues of The Total Package? Click here for our archives.
P.S. For a limited time, you can now cram your hard drive full of control busting copy at a $100 savings with the Steal These Secrets Swipefile. Stop racking your brain needlessly for creative ideas when you can have a treasure trove of proven winning concepts at your fingertips – guaranteed to open the profit floodgates – or your money back! Check it out!
Daniel Levis is a top marketing consultant & direct response copywriter based in Toronto, Canada and publisher of the world famous copywriting anthology “Masters of Copywriting” featuring the selling wisdom of 44 of the “Top Money” marketing minds of all time, including Clayton Makepeace, Dan Kennedy, Joe Sugarman, John Carlton, Joe Vitale, Michel Fortin, Richard Armstrong and dozens more! For a FREE excerpt visit Sellingtohumannature.com
Related posts
3 Comments »
Join the Discussion!
Let us know what you think. Or ask us anything. Or offer your own sage advice.
The only rule: RESPECT THIS HOUSE! Postings that contain abusive language and/or personal attacks will be cheerfully VAPORIZED. One cross word and – POOF! – your well-thought-out post will be gone in a puff of smoke.
– Clayton
Comment by Andrew Cavanagh — May 2, 2007 @ 1:21 pm
Thank you so much for a cool post Daniel.
It may seem overly geeky but the split testing I\’ve done with clients has given me some of the biggest insights into writing effective web copy.
And when you\’re using split testing and multivariate testing you\’re staying on the cutting edge which is vital when you write online copy because the market is changing so fast.
Also to give more dimension to the value of split testing you could read the first couple of pages of this web site…
http://goto-pro.com
Which explains how split testing can help turn A $95 Weekly Income From Your Web Site Into $13,494 A Week Or MORE FAILING 50% Of The Time…
And as an aside nearly every component of that web page has been split tested many times.
Kindest regards,
Andrew Cavanagh
Comment by John — May 15, 2007 @ 3:29 am
Good article.
I use a tool called Kaizentrack to do taguchi testing.
The issue with multivariate testing is that it requires a lot of traffic (nice if you have it!), whereas taguchi methods are designed to get more information from fewer tests. Both work, they\\\’re just different. Taguchi is faster but might miss some obscure combinations, whereas Google\\\’s optimiser is more thorough but slower.
Whatever tool you use, testing is a must, and I totally agree that Claude would mortgaged his entire family to have the tools we have available today, often for free!
John Bower
Comment by Victoria Spence — November 12, 2008 @ 3:37 pm
y3nul61ydo6e5e1b