Question

What is the difference between stemming and lemmatizationand explain how this canaffect precision and recall with examples

Answer 1

Stemming and Lemmatization are much similar terms in the information retrieval system or in field of NLP(Natural Language Processing) for Text Normalization.

The Stemming is the process taking the relevant part of a word and removing the rest. This is actually removing the inflected parts of the word while retrieving with a query.

For example, Porter stemmer, the most used stemmer is used to remove inflections in natural language processing(NLP) applications.

So, if we use stemming to a query, we can increase recall by matching words against their other inflected parts.

Lemmatization is also a way to remove inflected portions from a word. Lemmatization relies on a lexical knowledge base like WordNet to obtain the correct base forms of words instead of simply removing the inflections.

For example WordNet lemmatizes the word "meanness" to "meaning".

Thus Lemmatization more precision than stemming, due to this special feature.

SKIP POINTERS:

Skip pointer is another mechanism used in information retrieval to skip the processing part when searching with a query. The two challenges are where to place the skip pointer and how to efficiently merging of result using skip pointers. But this skip pointer are not useful in the queries of the form X and Y because, we cant skip the processing here. Here we the search engine definitely have to visit every document ID in the list of posting of either x and y terms. Skip pointers skip several irrelevant steps to reduce the time of rerieval and to take only most important relevant data. Queries of the form X and Y means, there are two variables in the query connected with an "and". This means only data which include the information about both X and Y have to retrieve. To do so search engines have to check every dataset whether there is X and Y together.

For example if someone searched "Apple and Orange", the most compatible and relevant results should contain both apple and orange. If the skip pointer technique is used, the search engine may miss the connected data. Therefore skip pointer cannot be used here.