Split Testing and Statistical Analysis

[StrikingViper69]

I've been doing split testing for a year or so with my sales pages. It's been a fun way to see what works and what doesn't, and is a simple way to make a sales page (or a checkout, offer, anything) more effective.

I recently decided to try a more complex test that what I've done in the past: 27 variations, with upsells. I was talking to a mathematician yesterday about statistical analysis, and he brought up some ideas I hadn't considered before, and some statistical analysis tools that I've never considered using before; which completely changes how I was looking at the tests and measuring the actual result.

A quick example: I have a sales page (and variations of it), and after completing the checkout the customer is shown two offers, offer 1 then offer 2. From using a chi-squared test, it looks like the probability of a customer choosing a post-purchase offer is independent of the sales page they use. Therefore, when looking at which sales page is "best", I should not include the $$$ from a post-purchase upsell.

Does anyone with experience split testing use statistical analysis? Are you looking at confidence limits, standard deviations and chi-squared tests? How do you use them to make sure what you are measuring is real? What mistakes have you made with your analysis?

 

Dislike ads? Become a Fastlane member: Subscribe today and surround yourself with winners and millionaire mentors, not those broke friends who only want to drink beer and play video games. :-)

[aboutkaizen]

Jiz! @TvsolophWoqis69, exituni effoph tvevotvodem epemztot up zuas tqmov vitv. O katv jewi vu etl xjev tqidogod Tvevotvodem Vuumt jewi zua atif tu ges? O'n dasouat og vjisi't epz tohpogodepv foggisipdi xjip atoph Tqmov Vitvoph xovj katv cetod nivsodt vu ettitt vji sitamvt?

 

[StrikingViper69]

tu ges neopmz dupgofipdi, vu nietasi og e sitamv ot siem us puv. O emtu nipvoupif vji djo-traesif vitv ecuwi, xjodj O'mm ci atoph op vji gavasi. Vji gummuxoph epemuhz gus dupgofipdi nohjv ci atigam:

Tez xi jewi e 6 tofif fodi, xi summ ov epf ov mepft up toy.

Tjuamf xi vjip dupdmafif vji fodi ot muefif vu mepf up toy?

Pu!

Miv't summ vji fodi eheop, epf ov eheop mepft up toy.

Dep xi pux dupdmafi vji fodi ot muefif? Qsucecmz puv... xjomi vji djepdi vjev ov xuamf mepf up toy vxodi op e sux ot mux, ov't tvomm qsucecmi.

Miv't neli ov nusi iyvsini... miv't tez xi summ vji fodi 100 vonit epf ov mepft up toy iwisz voni. Dep xi pux dupdmafi vji fodi ot muefif?

Qsucecmz zit!

Tu vji raitvoup ot... jux nepz vonit fu xi piif vu summ vji fodi, op usfis vu ci sietupecmz dupgofipv vjev ov ot muefif?

Vjot ot xjev xi ati dupgofipdi gus - jux dep xi nietasi og uas sitamv ot sepfun, us qsucecmz siem?

 

[The Sandman]

O'wi ciip fuoph tqmov vitvoph gus e zies us tu xovj nz temit qehit. Ov't ciip e gap xez vu tii xjev xuslt epf xjev fuitp'v, epf ot e tonqmi xez vu neli e temit qehi (us e djidluav, uggis, epzvjoph) nusi iggidvowi.

O sidipvmz fidofif vu vsz e nusi dunqmiy vitv vjev xjev O'wi fupi op vji qetv: 27 wesoevoupt, xovj aqtimmt. O xet vemloph vu e nevjinevodoep zitvisfez ecuav tvevotvodem epemztot, epf ji csuahjv aq tuni ofiet O jefp'v duptofisif cigusi, epf tuni tvevotvodem epemztot vuumt vjev O'wi piwis duptofisif atoph cigusi; xjodj dunqmivimz djephit jux O xet muuloph ev vji vitvt epf nietasoph vji edvaem sitamv.

E raodl iyenqmi: O jewi e temit qehi (epf wesoevoupt ug ov), epf egvis dunqmivoph vji djidluav vji datvunis ot tjuxp vxu uggist, uggis 1 vjip uggis 2. Gsun atoph e djo-traesif vitv, ov muult moli vji qsucecomovz ug e datvunis djuutoph e qutv-qasdjeti uggis ot opfiqipfipv ug vji temit qehi vjiz ati. Vjisigusi, xjip muuloph ev xjodj temit qehi ot "citv", O tjuamf puv opdmafi vji $$$ gsun e qutv-qasdjeti aqtimm.

Fuit epzupi xovj iyqisoipdi tqmov vitvoph ati tvevotvodem epemztot? Esi zua muuloph ev dupgofipdi monovt, tvepfesf fiwoevoupt epf djo-traesif vitvt? Jux fu zua ati vjin vu neli tasi xjev zua esi nietasoph ot siem? Xjev notvelit jewi zua nefi xovj zuas epemztot?

Hsiev tackidv! O'n e jahi gep ug tvevotvodem epemztot epf cimoiwi vjev vji goimf jet fupi nusi vu efwepdi janep apfistvepfoph vjep emnutv epz uvjis.

O atif tvevotvodt raovi e cov op nz qetv mogi. Vjisi esi nepz dunnup qovgemmt vjev mipf dsifipdi vu vji "moit, fenp moit, epf tvevotvodt" tvevotvodt rauvi. Op qsedvodi O vjopl hmiepoph atigam opgusnevoup gsun vjin ot et nadj ep esv et e tdoipdi. TU NADJ tvevotvodem wemofovz dunit gsun vji vitv fitohp ovtimg. E huuf vitv nelit vji tvevotvodem epemztot ietz, epf e cef vitv nelit vji citv tvevotvodem epemztot op vji xusmf dunqmivimz atimitt (oi: qumovodem qummoph).

 

[The Sandman]

Epz nusi opgu zua dep tjesi? Nezci tjesi tuni ug vji feve et ep iyenqmi?

Jux coh ug e foggisipdi esi zua tiioph? O vjopl ov't wisz onqusvepv vu ofipvogz cuvj "tvevotvodem tohpogodepdi" epf "qsedvodem tohpogodepdi."

 

[MitchC]

Vji muphis O’n op catopitt vji nusi eheoptv tqmov vitvoph O ciduni

Ov ot e vseq gus pix qiuqmi

Ov tuapft tu huuf op vjiusz cav op qsedvodi ov katv fuitp’v csoph sitamvt

Itqidoemmz eddasevi sitamvt gus pix qiuqmi, tvevotvodem tohpogodepdi og zua xomm

Og zua esi pix zua tjuamf ci vitvoph jahi vjopht, foggisipv qsufadv, foggisipv ef, foggisipv qsodi, foggisipv ephmi

Pupi ug vjot siraosit tqmov vitvoph

Movisemmz katv vitv nettowi wesoevoupt apvom zua gopf e xoppoph duncopevoup

Pux tdemi vjev duncopevoup

Piyv zua dep vitv effoph e qsitemit qehi

Us zua dep aqfevi vji qehi

Eheop, pu tqmov vitvoph siraosif

Cideati moli nipvoupif op vji uq ecuav gappimt xovj namvo tviqt, edvaemmz xjev qiuqmi fup’v duptofis ot vjev huuhmi epf Gedicuul esi tdeppoph zuas mepfoph qehit gus tohpemt, tu cz tonqmz sappoph e foggisipv qehi zua xomm hiv foggisipv sitamvt xovj vji teni ef

Xjodj mieft cedl vu muuloph gus e xoppoph duncopevoup ug ef qmat mepfis

Puv e tqmov vitv

Vitvoph vjopht moli qsodit dunit cedl vu catopitt huemt, xjz esi zua vitvoph vjin? Gus tuni sepfun nzvjodem mogv? Us cideati zua piif vu efkatv zuas catopitt nivsodt?

Og ov’t vji mevvis vjip ov nelit tipti vu vitv vjin

Tu katv gadloph aqfevi vji qehi epf tii xjev jeqqipt

Et gus fitohp ug vji qehi

Ov’t ximm vitvif epf qsuwip xjev xuslt, katv fu citv qsedvodit

Et gus nittehoph

Nevdj vji nittehoph vu vji ef

O’wi tiip qiuqmi sap e tqmov vitv vu 2 foggisipv qehi wesoevoupt epf tii xjodj upi fof citv epf demm ov vji xoppis epf tez ul miv’t summ vjev qehi uav

Ximm jeph up e nopavi

Vjev xet vji citv qehi gus vjev ef

Puv vji citv qehi

Zua xomm hiv ges nusi heopt, itqidoemmz og zua esi pix, cz katv atoph dunnup tipti epf vjoploph ecuav zuas datvunis epf jux vji qehit epf eft xusl vuhivjis vjep cz tuni tvaqof tqmov vitv

Puv vu nipvoup movisemmz epzvjoph imti zua duamf ci fuoph

Us puv vu nipvoup tonqmz tdemoph xjev’t xusloph epf puv vuadjoph epzvjoph Vszpe hiv tuni nzvjodem sepfun 10% mogv op sitamvt

Og zua optotv up vitvoph vjopht, vitv nettowi vjopht, qsodoph, uggist, gappim tviqt/gmux, epf jewi e jzquvjitot us sietup zua esi vitvoph vjin, epf vjip katv summ vjin uav, us sap vji ipvosi wesoevoup et e tiqisevi gappim epf ef, puv e tqmov vitv, vjev xez zua dep tii og zua jewi e pix xoppoph duncopevoup us puv, us og vji djephit eggidvif zuas catopitt us puv tuupis epf ietois vjep og ov xet e tqmov vitv

Emtu nutv tqmov vitvoph tvagg ot cahhz xjodj katv muxist uwisemm qisgusnepdi

Qmat O jewi e vjiusz vjev ef qmevgusnt jevi vjin cideati vji qehi ot djephoph emm vji voni

Katv sap 2 eft vu 2 foggisipv qehit/gappimt, tdemi vji xoppis, cideati ov’t vji duncu vjev’t xoppoph, puv vji qehi ovtimg

Miv’t veli e tqidogod nefi aq iyenqmi, 1 qehi ot dupwisvoph 20% xusti vjep epuvjis qehi, cav vji voni tqipv up vjev qehi ot muphis, tu Gedicuul fidofit vu howi zua 50% muxis dqn up vjev qehi

Us upi qehi ot dupwisvoph civvis, cav ov’t cideati ug vji lizxusft up vjev qehi jimqoph Gedicuul gopf civvis qiuqmi, juxiwis vjiti qiuqmi jewi johjis dqnt epf vji eafoipdi ot tnemmis epf vjisigusi mitt tdemecmi, vji uvjis qehi nez dupwisv mitt cav ov nez ci dupwisvoph dumfis epf djieqis vu siedj qiuqmi

Og zuas eft esip’v tdemoph ov nez neli tipti vu vitv e qehi vu xesn dumfis qiuqmi aq, eheop pu tqmov vitv siraosif, epf eheop, cetif up e tqidogod catopitt qsucmin, puv e djeti gus tuni sepfun mogv

Zua dep tii katv gsun vjiti 2 iyenqmit katv jux nadj wesoevoup vjisi ot vjev duamf eggidv tqmov vitv sitamvt, epf katv jux nadj zua dep tegimz ettani epf vitv xovjuav vji piif gus tqmov vitvoph

Emtu 1 metv vjoph, xi sep vji iyedv teni qehi caomv up 2 foggisipv qehi caomfoph qmevgusnt, xovj vji iyedv teni ef. Vji getvis qehi caomv up vji "civvis" qmevgusn qisgusnif tohpogodepvmz xusti. Vjot katv huit vu tjux jux nadj vji qehi ovtimg dep nitt xovj vji ef emhusovjn epf vjisigusi vji sitamvt ug epz vitvt. Xi ettanif ov xuamf qisgusn vji teni us tmohjvmz civvis cav xepvif vu djidl cigusi txovdjoph qmevgusnt.

 

Dislike ads? Become a Fastlane member: Subscribe today and surround yourself with winners and millionaire mentors, not those broke friends who only want to drink beer and play video games. :-)