It doesn’t matter what the goal of the testing was.

What I meant when I said we need more information is this: how were the tests done, was there any corrections for repeated tests with the same data, and so on.

The p-value and testing comment applies regardless: they give you zero information for or against either of your hypotheses.

That’s already been explained. Has to do with a marketing company. Steele in all his competence couldn’t even spell the name of the bank right.

I think it had to do with analyzing which versions of the same ad were more popular, analyzing click thru.

He gave an interview that described what he did, with AB testing of hundreds of versions of the same ad.

Ugh. It’s always distressing to hear of the “magic” of A/B testing as though it’s a new concept: the basic idea of comparing results with designed experiments is over 100 years old — there is nothing new here.

Also, as stated, that information provides zero information. This is a powerful idea, and the temptation to overuse it is high. Simply saying “hundreds” of tests were run is meaningless without information about the form of the individual tests, whether they were done based on predetermined questions that were decided upon for the data was collected or whether they were simply run after the data was collected to see what showed up. It should have been the first approach (although even then the chance of detecting “significant” results just by chance is huge): if it were the second approach the results are totally worthless.

This also ignores the fact that classical hypothesis testing, based on p-values, really provides no evidence for or against either of your hypotheses.

That cognitive scotoma of yours can be illustrated by the right hand panel here:

https://healthsaline.com/wp-content/uploads/2017/03/image-result-for-scotoma-768×406.jpeg