Question

Some analysts have argued that Big Data is fundamentally about data “plumbing,” and not about insights, or deriving interesting patterns. It is argued that value (the fifth V) can just as easily be found in “small,” normal, or “weird” datasets (i.e., datasets that wouldn’t have been considered before). Do you agree with this? Can you think of small or novel datasets that would provide value as well, without requiring a full-fledged Hadoop setup?

Answer 1

The Answer is No, Big Data is a field that treats ways to Analyze and Systematically extract information from multiple Data sets which are complex to be dealt with by traditional data processing applications.

Big Data has various applications in many fields. Few of the contradictory factors of using normal datasets to deal with huge data are listed below:

1. Big Data mainly deals with Data Extraction, which comes from various sources like Logs, Audio devices, Video devices, Web pages, Data Bases, which is not possible to perform using normal Data sets. so, Big Data is Needed.

2. It is acceptable that the 5th V of Big data i.e., Value is found in normal Data sets. But it will not provide that many features that are present in Big Data Analytics. There are many Data Extraction tools present in the market but none of them provides as many functionalities as Big data provides.

3. Data Plumbing is also considered as one of the features of the big data. It provides a way to manage, process and store data efficiently, which further helps to easily access and process the Data. But Data plumbing is not the only feature of Big data, Apart from that it provides various techniques like Data Extraction, Data Transformation, Data Loading, Data virtualization, Data processing, etc. which is not provided by any other novel or small data sets.

4. Unlike other data sets that store information or data in a structured format. Big data processing tool Hadoop will store the data in a structured, semi-structured and unstructured format. so Big data tools are needed.

5. Data processing, Data manipulation, Data storing is done individually in Hadoop, while in normal data sets it will not follow this scenario.

6. Some of the Best data sets has proven a consistent, mature enough to perform big data operations. But still, these data sets are not appropriate for Real-time processings and they are not used yet.

7. Real-time applications where Data security is most needed which include security agencies, Authentication services, Restricted or confidential data services. In these cases, A Full-fledged system that provides all the necessary satisfying criteria is required to perform secure processing which is not possible by any normal data sets.

Finally, we can argue that Big Data is Essential and provides unique features, which is impossible for the other "small", "weird", "normal" Data sets to Perform. So, Big Data is Mandatory for such kind of applications.