Question

(Building Index) Compute TFIDF scores for all words in all documents(Assume, there are any 5 text files in a Document/Folder) and build an inverted index using any of below technologies: i. Spark ii. Hive iii. Pig iv. Hbase Condition: Do not use any existing libraries to compute TFIDF. It has to be done from scratch.

Answer 1

TF-IDF stands for "Term Frequency, Inverse Document Frequency". It is a way to score the importance of words (or "terms") in a document based on how frequently they appear across multiple documents.

• f a word appears frequently in a document, it's important. Give the word a high score.
• But if a word appears in many documents, it's not a unique identifier. Give the word a low score.

Therefore, common words like "the" and "for", which appear in many documents, will be scaled down. Words that appear frequently in a single document will be scaled up.

import math
from textblob import TextBlob as tb

def tf(word, blob):
    return blob.words.count(word) / len(blob.words)

def n_containing(word, bloblist):
    return sum(1 for blob in bloblist if word in blob.words)

def idf(word, bloblist):
    return math.log(len(bloblist) / (1 + n_containing(word, bloblist)))

def tfidf(word, blob, bloblist):
    return tf(word, blob) * idf(word, bloblist)