The Lead Singer Swap - How to A/B Test like a Rock Star

If you follow baseball, you’re familiar with the herculean slugger who also happens to be a clubhouse cancer.  If you follow business, you’ve heard of at least a few cases of visionary CEOs who are distrusted and often feared by his/her subordinates.  And if you follow music, the case of the gifted yet narcissistic musician is a familiar tale.

In 1994, a four-piece rock band out of California named Stone Temple Pilots (commonly known as STP) was suffering from this ailment.  On one end, they had released two of the most celebrated rock records of the decade, selling over ten million albums and captivating the music industry with their genre-crossing sound.  Yet, behind the scenes the band was a wreck.  Guitarists Dean and Robert DeLeo along with drummer Eric Kretz, despite writing the vast majority of the music for the band, were often overshadowed by the flamboyant lead singer Scott Weiland.  While a mesmerizing presence on stage, Weiland was both unreliable and antagonistic to his bandmates behind the curtains.

So the DeLeo brothers did something novel to the music world: They ran a test.  Rather than ousting Scott Weiland from STP, they began a second band, employing Kretz on drums and hiring an experienced lead singer, Dave Coutts. They named their side project “Talk Show” and prepared to simultaneously record, release and promote two albums, each with a different lead singer.

To marketing and product professionals, this style of test is commonly known as an A/B test.  In this type of test, two scenarios are tested that are identical with the exception of one variant.  Tests are then run for the variant against samples of the same audience.  For instance, a marketer may send the same email offer to sample groups using two different subject lines to test the open rates.  A user experience designer may place a button in two different locations on a website to test which gets more clicks.  As long as the samples for each test are large enough to provide statistically significant results, the results may be used to make important business decisions.  (Act-On - one of the leading marketing automation providers in the industry - detail other A/B best practices in a helpful eBook found on their website.)

To control the test the best they could, the DeLeo brothers wrote thirty songs together and randomly assigned half of the songs to one band and the remaining songs to the other.  They then recorded both albums under the same record label and toured with both bands.   In their A/B test, Weiland was test A and the Coutts was test B.   (Unfortunately, the audience couldn’t be partitioned for the test, providing one exception to the A/B test rule.)

The DeLeo brothers were banking on Talk Show success to allow them to disband STP and rid themselves of the infighting that was plaguing their career.  Unfortunately, the test results didn’t agree.  The album recorded with Coutts (self-titled as “Talk Show”) was a commercial and critical failure while STP’s record (“Tiny Music from the Vatican Gift Shop”) received higher marks from both critics and fans.  Listening to the records, it was immediately obvious why.  Weiland’s lyrics were eccentric and his vocals seemed to convey a different emotion in each track.  Coutts was mostly dry and vapid.

One of the cardinal rules around A/B testing is to listen to the data and not let personal bias influence conclusions.  The DeLeo brothers obliged.  They put an abrupt end to Talk Show and begrudgingly continued on with Stone Temple Pilots.  The band would slog through two more records before breaking up in 2002.  Stone Temple Pilots may not have been built for the long haul, but the data spoke for itself: Their band was most effective with Weiland at the helm.