5.4 Parameter Sensitivity Analysis

Both our conceptual language model and the relevance model have a number of parameters that need to be set, as introduced in Section 5.2.1. In this section we describe the optimal settings for each model and explore the sensitivity of the results to changes in the settings. Similar to related work (e.g., [98, 196, 354], we did not evaluate $\left|R\right|,\left|{\mathcal{V}}_Q\right|>10$. Even given this restriction, the obtained results are clear improvements and further improvements may be obtained with an even larger set of terms or documents.

Table 5.12 lists the optimal parameter settings for the relevance model per test collection and we observe that the setting of the optimal value for ${\lambda }_Q$ is dependent on the document collection. Table 5.13 lists the optimal parameter values for the conceptual language model. Again we observe that the optimal value for ${\lambda }_Q$ is dependent on the document collection. We zoom in on the sensitivity of the results of the conceptual language model towards the setting of ${\lambda }_Q$, by displaying the effect of varying ${\lambda }_Q$ on MAP (Figure 5.4a) and P5 (Figure 5.4b). We observe that the curves follow a similar pattern for the CLEF document collection and for both measures, with both maxima lying around ${\lambda }_Q=0.3$. The TREC-GEN-04 and TREC-GEN-05 topics—which both use the TREC 2004 document collection—follow a less similar pattern, although their maximum MAP scores have a similar corresponding ${\lambda }_Q$ value. The TREC-GEN-06 and the CLEF-DS-2007 topics show the largest relative improvement (both nearly 20% improvement over the query likelihood in terms of MAP, i.e., when ${\lambda }_Q=0$). We also observe that selecting the best value for ${\lambda }_Q$ based on the highest MAP scores does not necessarily lead to the highest score in terms of early precision. Interestingly, the TREC-GEN-06 topics reach roughly the same P5 scores for the query likelihood model as when we would only use the terms suggested by the conceptual language model.