Web search engines

The vector space model

• Documents represented as vectors in a multi-dimensional Euclidean space
• Each axis = a term (token)
• Coordinate of document d in direction of term t determined by:
• Term frequency TF(d,t)
• number of times term t occurs in document d, scaled in a variety of ways to normalize document length
• Inverse document frequency IDF(t)
• to scale down the coordinates of terms that occur in many documents

Term frequency

• . .
• Cornell SMART system uses a smoothed version

Inverse document frequency

• Given
• D is the document collection and is the set of documents containing t
• Formulae
• mostly dampened functions of
• SMART
• .

Vector space model

• Coordinate of document d in axis t
• .
• Transformed to in the TFIDF-space
• Query q
• Interpreted as a document
• Transformed to in the same TFIDF-space as d

Measures of proximity

• Distance measure
• Magnitude of the vector difference
• .
• Document vectors must be normalized to unit ( or ) length
• Else shorter documents dominate (since queries are short)
• Cosine similarity
• cosine of the angle between and
• Shorter documents are penalized

Relevance feedback

• Users learning how to modify queries
• Response list must have least some relevant documents
• Relevance feedback
• `correcting' the ranks to the user's taste
• automates the query refinement process
• Rocchio's method
• Folding-in user feedback
• To query vector
• Add a weighted sum of vectors for relevant documents D+
• Subtract a weighted sum of the irrelevant documents D-
• .