> Incrementality testing is a big part of marketing measurement at any reputable agency
In a huge number of cases these tests are run by people that don't have the statistical background to property run and understand these tests, in the remaining cases there is almost never follow through to ensure that the results of the test have persisted after the experiment.
People will say "oh this test shows 10% improvement, and this one does too! and this one! and this one!" But then you should see a nearly 50% improvement just because of ad spend and you almost never do. Nobody ever check that all the numbers add up, they just want the numbers that someone reported to feel like they are sound enough to hold up to scrutiny but scrutiny is never applied.
In a huge number of cases these tests are run by people that don't have the statistical background to property run and understand these tests, in the remaining cases there is almost never follow through to ensure that the results of the test have persisted after the experiment.
People will say "oh this test shows 10% improvement, and this one does too! and this one! and this one!" But then you should see a nearly 50% improvement just because of ad spend and you almost never do. Nobody ever check that all the numbers add up, they just want the numbers that someone reported to feel like they are sound enough to hold up to scrutiny but scrutiny is never applied.