4.2 Experimental Setup

We aim to compare the effectiveness of the models listed in Table 4.1, each of which was introduced in either the preceding section or in Chapter 2. For all models, we use the resulting query model as estimated query part, ${\widehat{\theta }}_Q$, in Eq. 2.10. All of the models have a number of parameters in common. In this chapter, we focus on varying these parameters and observing the effect on retrieval effectiveness. We consider the following parameters: $\left|{\mathcal{V}}_Q\right|$, $\left|R\right|$, and ${\lambda }_Q$. See Section 3.4 for their descriptions. Some of the feedback models under investigation require additional parameter settings. For MBF, and NLLR (cf. Eqs. 2.16 and 4.3) we set ${\lambda }_R=0.15$ and ${\lambda }_R=0.5$ respectively. For PRM (cf. Eq. 2.27), we set $\mu =0$, which effectively results in RM-0 estimated on parsimonious document models: $$\begin{array}{rcll}P\left(t_1,\dots ,t_{\left|\mathcal{V}\right|}|\theta Q\right)& \propto & \prod _{i=1}^{\left|\mathcal{V}\right|}\frac{1}{\left|R\right|}\sum _{D\in R}P\left(t_i|{\widehat{\theta }}_D\right).& \text{(4.7)}\end{array}$$

In essence, Eq. 4.7 takes the middle ground between RM and MBF; it combines the estimation method of MBF with the document independence assumption of RM. For evaluation, we use the following diverse set of test collections

• TREC Robust 2004 (TREC-ROB-04),
• TREC Relevance Feedback 2008 (TREC-RF-2008 and TREC-PRF-08), and
• TREC Web 2009, using the Category B subset (TREC-WEB-09).

These collections were introduced in Section 3.3. The percentages and significance tests in the result tables in this chapter indicate the difference with respect to the baseline—we use a ‘*’ to indicate a significant difference, as detailed in Section 3.2.2. In the next section we consider retrieval effectiveness using pseudo relevance feedback and in Section 4.4 we turn to explicit relevance feedback.