5.1 Conceptual Language Models

Our goal is to utilize the knowledge captured using concepts from a concept language to enhance the estimation of the query model ${\theta }_Q$. To this end, we use the concepts as a pivot language in a double translation [169], similar to the method proposed by Berger and Lafferty [31] that was discussed in Section 2.3.1. The approach presented by French et al. [104] is also related to ours. They propose a heuristic method of associating terms with concepts. Our approach, however, utilizes the concepts that are associated with a query to find terms related to these concepts in order to estimate the expanded part of the query model, $P\left(t|{\widehat{\theta }}_Q\right)$ (cf. Eq. 2.10). Figure 5.1 shows a graphical representation of the dependencies of this process.

In words, first we translate the query $Q$ into a set of relevant concepts, $\mathcal{C}=\left\{c_1,\dots ,c_k\right\}$. Next, the vocabulary terms associated with the concepts are considered as possible terms to include in the query model and we marginalize out the concepts. More formally, we determine $$\begin{array}{rcll}P\left(t|{\widehat{\theta }}_Q\right)=\sum _{c\in \mathcal{C}}P\left(t|c\right)P\left(c|Q\right),& & & \text{(5.1)}\end{array}$$

where we assume that the probability of selecting a term is only dependent on the concept once we have selected that concept for the query.

Two components need to be estimated: $P\left(t|c\right)$, to which we refer as a generative concept model, and $P\left(c|Q\right)$, to which we refer as a conceptual query model. As to the former, we will need to associate terms with concepts in the concept language. While the concepts may be directly usable for retrieving documents [128, 302, 318], we associate each concept with a weighted set of most characteristic terms using a multinomial unigram model. To this end we consider the documents that are annotated with concept $c$ as bridges between the concept and terms, by representing concepts as multinomial distributions over terms, $P\left(t|c\right)$. Generative concept models will be detailed further in Section 5.1.2 below.

The second component—the conceptual query model $P\left(c|Q\right)$—is a distribution over concepts specific to the query. In some settings, concepts are provided with a query or as part of a query, see, e.g., the PubMed search interface [132], some early Text Retrieval Conference (TREC) adhoc tracks (6, 7, and 8 in particular), and the Initiative for the Evaluation of XML Retrieval (INEX) Entity Ranking track, where Wikipedia categories are used [87]. If this is not the case, however, we may leverage the document annotations to approximate this step: this is what we do in the next section.

5.1.1 Conceptual Query Modeling

We now turn to defining $P\left(c|Q\right)$, the conceptual query model. Contrary to the alternatives mentioned at the end of the previous section, in a typical IR setting concepts are not provided with a query and need to be inferred, estimated, or recognized [339, 358]. In this chapter, we formulate the estimation of concepts relevant to a query in a standard language modeling manner, by determining which concepts are most likely given documents relevant to the query. Alternatively, we could involve the end user and ask which documents, associated concepts, or terms are relevant. Since we do not have access to such assessments we use pseudo relevance methods. In recent work and using the same framework, different approaches of estimating a conceptual query model have been studied and it was concluded that using feedback documents is far more effective than using, e.g., string matching methods that try to recognize concepts in the query [318]. In the next chapter this finding is confirmed, albeit using a different setting and test collection.

Like Lavrenko and Croft [183], we view the process of obtaining a conceptual query model as a sampling process from a number of representative sources. The user has a notion of documents satisfying her information need, randomly selects one of these, and samples a concept from its representation. Hence, the conceptual query model is defined as follows: $$\begin{array}{rcll}P\left(c|Q\right)& =& \sum _{D\in R}P\left(c|D\right)P\left(D|Q\right).& \text{(5.2)}\end{array}$$

Here, $R$ is a set of pseudo relevant documents returned by an initial retrieval run using the textual query; $P\left(c|D\right)$ is the concept language model of the document, the estimation of which is discussed in the next section. We assume that the probability of observing a concept is independent of the query once we have selected a document given the query, i.e., $P\left(c|D,Q\right)=P\left(c|D\right)$. The term $P\left(D|Q\right)$ denotes the probability that document $D$ is chosen given $Q$, which is obtained using the retrieval scores, viz. Eq. 2.8.

We assume that pseudo relevant documents are a good source from which we can sample the conceptual query model. Indeed, manual inspection shows that they are annotated with many relevant concepts, but also that they, despite being related to the query, contain a lot of noise: some concepts occur in many documents and are not very informative. Sampling from the maximum likelihood estimate for these documents would thus result in very general conceptual query models. Therefore, to re-estimate the probability mass of the concepts in the sampling process, we use a parsimonious language model. In the next section we detail how re-estimation is performed.

Table 5.3 illustrates the difference between a maximum likelihood estimation and a parsimonious estimation. It shows the concepts (in this case MeSH terms) with the highest probability for topic 186 from the TREC Genomics 2006 test collection. The conceptual query model based on the parsimonious document models contains more specific—and thus more useful—concepts, such as PRESENILIN-1 and PRESENILIN-2. The model based on maximum likelihood estimates includes more general concepts such as HUMANS, which are relevant but too general to be useful for searching.

5.1.2 Generative Concept Models

Given Eq. 5.1, our goal is to arrive at a probability distribution $P\left(t|c\right)$ over vocabulary terms for each concept in the concept language used for annotating the documents. We determine the strength of the association between a term and a concept by looking at the annotations made by the trained annotators who have labeled the documents. In the end, this method defines the parameters of a generative language model for each concept: a generative concept model. We determine $P\left(t|c\right)$, i.e., the strength of association between a concept $c$ and a term $t$, by determining the probability of observing $t$ given $c$. Concepts that are used to annotate documents may have different characteristics from other parts of a document, such as title and content. Annotations are selected by human indexers from a concept language while the remaining content consists of free text. Since the terms that make up the document are “generated” using a different process than the concepts, we may assume that $t$ and $c$ are independent and identical samples given a document $D$ in (or with) which they occur. So, the probability of observing both $t$ and $c$ is: $$\begin{array}{rcll}P\left(t,c\right)=\sum _{D}P\left(D\right)P\left(c,t|D\right)=\sum _{D\in {\mathcal{D}}_C}P\left(D\right)P\left(t|D\right)P\left(c|D\right),& & & \text{(5.3)}\end{array}$$

where ${\mathcal{D}}_C$ denotes the set of documents annotated with concept $c$. We assume each document to have a uniform prior probability of being selected and obtain: $$\begin{array}{rcll}P\left(t|c\right)& =& \frac{P\left(t,c\right)}{P\left(c\right)}& \text{(5.4)}\\ & =& \frac{\sum _{D\in {\mathcal{D}}_C}P\left(D\right)P\left(t|D\right)P\left(c|D\right)}{P\left(c\right)}& \\ & \propto & \frac{1}{P\left(c\right)}\sum _{D\in {\mathcal{D}}_C}P\left(t|D\right)P\left(c|D\right).& \end{array}$$

Hence, it remains to define three terms: $P\left(c\right)$, $P\left(t|D\right)$, and $P\left(c|D\right)$. First, the term $P{\left(c\right)}^{-1}$ functions as a penalty for frequently occurring and thus relatively non-informative concepts. We estimate this term using MLE on the document collection: $$\begin{array}{rcll}P\left(c\right)=\frac{\sum _{D}n\left(c,D\right)}{\sum _{c^{\prime }}\sum _{D^{\prime }}n\left(c^{\prime },D^{\prime }\right)},& & & \text{(5.5)}\end{array}$$

where $n\left(c,D\right)$ is the number of times document $D$ is labeled with concept $c$ (which is typically 1).

Next we turn to $P\left(x|D\right)$, for $x\in \left\{t,c\right\}$. The size of these models (in terms of the number of words or the number of concepts that receive a non-zero probability) may be quite large, e.g., in the case of a large document collection or in the case of frequently occurring concepts. Moreover, as exemplified above, not all of the observed events (where events are either terms or concepts) are equally informative. Some may be common, whilst others may describe the general domain of the document. Earlier in the thesis, we have noted that it is common to consider each document as a mixture of document-specific and more general terms (cf. Eq. 2.5); we now generalize this statement to also include concepts. Further, given this assumption, we may update each document model by reducing the amount and probability mass of non-specific events. We do so by iteratively adjusting the individual probabilities in each document, based on a comparison with a large reference corpus such as the collection. More formally, we maximize the posterior probability of $D$ after observing $x$: $$\begin{array}{rcll}P\left(D|x\right)=\frac{{\lambda }_{x}P\left(x|D\right)}{\left(1-{\lambda }_x\right)P\left(x\right)+{\lambda }_{x}P\left(x|D\right)}.& & & \text{(5.6)}\end{array}$$

Note that ${\lambda }_x$ may be set differently for $D$ (Eq. 2.5) and $C$. For these estimations, we fix ${\lambda }_C={\lambda }_D=0.15$ based on [136, 211, 215]. We then apply the following EM algorithm until the estimates do not change significantly anymore: $$\begin{array}{rcll}\text{E-step:}& & e_x=P\left(D|x\right)=\frac{{\lambda }_{C}P\left(x|D\right)}{\left(1-{\lambda }_C\right)P\left(x\right)+{\lambda }_{C}P\left(x|D\right)}& \text{(5.7)}\\ & & & \\ \text{M-step:}& & P_C\left(x|D\right)=\frac{n\left(x,D\right)e_x}{\sum _{x^{\prime }}n\left(x^{\prime },D\right)e_{x^{\prime }}}.& \text{(5.8)}\end{array}$$

This updating mechanism enables more specific events, i.e., events that are not well-explained by the background model, to receive more probability mass, making the resulting document model more specific. After the EM algorithm has converged, we remove those events with a probability lower than a certain threshold $\delta$. Thus, the resulting document model for terms, $P\left(t|{\widehat{\theta }}_D\right)$, to be used as $P\left(t|D\right)$ in Eq. 5.4 is given by: $$\begin{array}{rcll}P\left(t|{\widehat{\theta }}_D\right)=\left\{\begin{array}{cc}{Z}_{D_t}\cdot P_C\left(t|D\right)\hfill & \text{if }t\in D\text{ and }{P}_C\left(t|D\right)>{\delta }_t\hfill \\ 0\hfill & \text{otherwise},\hfill \end{array}\right.& & & \text{(5.9)}\end{array}$$

where $Z_{D_t}$ is a document-specific normalization factor: $Z_{D_t}=1∕{\sum }_t{P}_C\left(t|D\right)$. Table 5.4 provides an example of the effects of applying Eq. 5.9 on a document from the CLEF-DS document collection (that will be introduced in Section 5.2). Similarly, the resulting document model for concepts, $P\left(c|{\widehat{\theta }}_D\right)$, to be used for $P\left(c|D\right)$ in Eq. 5.4, is given by: $$\begin{array}{rcll}P\left(c|{\widehat{\theta }}_D\right)=\left\{\begin{array}{cc}{Z}_{D_c}\cdot P_C\left(c|D\right)\hfill & \text{if }c\in D\text{ and }{P}_C\left(c|D\right)>{\delta }_c\hfill \\ 0\hfill & \text{otherwise},\hfill \end{array}\right.& & & \text{(5.10)}\end{array}$$

where $Z_{D_c}$ is a document-specific normalization factor: $Z_{D_c}=1∕{\sum }_c{P}_C\left(c|D\right)$. Table 5.3 provides an example of the effects of applying Eq. 5.10 on a topic from the TREC document collection (that will be introduced in Section 5.2). For the experiments in this chapter we fix ${\delta }_t={\delta }_c=0.01$.