Decisions about which ad to use are often based on an expert’s unaided judgments – or a committee's judgments - about which ads from a set will be most effective. By “unaided,” we mean that the judgments are not aided by formal evidence-based forecasting procedures.

One of the most powerful findings from forecasting research is that combining independent anonymous forecasts improves accuracy (Armstrong 2001). Structured combining procedures are likely to improve decisions over which ad to run. The primary shortcomings of combining expert opinions are that the procedure provides no advice on how to improve ads and, most important, there is a weak relationship between consensus and effectiveness.

Combining calls for predictions from 5 to 20 experts, the recommended procedure is to uses an average of the rankings with equal weights among the experts. The procedure is useful when expert predictions are expected to have some validity and especially when the experts draw upon different sources of knowledge.

The request for estimates (forecasts) should focus on effectiveness. We suggest: “Rank the effectiveness of the following set of ads. If you are uncertain, you can use equal ranks.”

One approach for combining would involve an administrator to solicit experts’ effectiveness rankings for a set of ads via email. Experts would understand that their judgments would remain anonymous. Another possibility is to take a time-out during a meeting and ask people to submit unsigned rankings. Alternatively, the Delphi procedure software could be used for this purpose by limiting the process to one round.

When relevant knowledge is distributed among experts, it helps to ask them to explain the reasons for their forecasts and to make this information available (anonymously) to the other experts in the panel. This implies using two or more rounds of the Delphi procedure.

Remember that while improvements are expected, such rankings are still subject to much uncertainty. For more effective procedures, use the AdPrin Audit and Testing Ads under the lectures on the Educational Materials page.