Question

Examine and elaborate FIVE (5) major technical challenges encountered in Big Data application development.

Answer 1

No organization can function without data these days. With huge amounts of data being generated every second from business transactions, sales figures, customer logs, and stakeholders, data is the fuel that drives companies. All this data gets piled up in a huge data set that is referred to as Big Data.

Challenges:

1. Insufficient understanding and acceptance of big data

Companies fail in their Big Data initiatives due to insufficient understanding. Employees may not know what data is, its storage, processing, importance, and sources. Data professionals may know what is going on, but others may not have a clear picture.

For example, if employees do not understand the importance of data storage, they might not keep the backup of sensitive data. They might not use databases properly for storage. As a result, when this important data is required, it cannot be retrieved easily.

We can solve this by conducting Big Data workshops and seminars at companies for everyone. Basic training programs must be arranged for all the employees who are handling data regularly and are a part of the Big Data projects. A basic understanding of data concepts must be inculcated by all levels of the organization.

2. Data growth issues

The most obvious challenge associated with big data is simply storing and analyzing all that information. In its Digital Universe report, IDC estimates that the amount of information stored in the world's IT systems is doubling about every two years. By 2020, the total amount will be enough to fill a stack of tablets that reaches from the earth to the moon 6.6 times. And enterprises have responsibility or liability for about 85 percent of that information.

Much of that data is unstructured, meaning that it doesn't reside in a database. Documents, photos, audio, videos and other unstructured data can be difficult to search and analyze.

In order to handle these large data sets, companies are opting for modern techniques, such as compression, tiering, and deduplication. Compression is used for reducing the number of bits in the data, thus reducing its overall size. Deduplication is the process of removing duplicate and unwanted data from a data set.

3. Generating insights in a timely manner

Of course, organizations don't just want to store their big data — they want to use that big data to achieve business goals. According to the NewVantage Partners survey, the most common goals associated with big data projects included the following:

1. Decreasing expenses through operational cost efficiencies
2. Establishing a data-driven culture
3. Creating new avenues for innovation and disruption
4. Accelerating the speed with which new capabilities and services are deployed
5. Launching new product and service offerings

To achieve that speed, some organizations are looking to a new generation of ETL and analytics tools that dramatically reduce the time it takes to generate reports. They are investing in software with real-time analytics capabilities that allows them to respond to developments in the marketplace immediately.

4. Securing big data

Security is also a big concern for organizations with big data stores. After all, some big data stores can be attractive targets for hackers or advanced persistent threats (APTs).

However, most organizations seem to believe that their existing data security methods are sufficient for their big data needs as well. In the IDG survey, less than half of those surveyed (39 percent) said that they were using additional security measure for their big data repositories or analyses. Among those who do use additional measures, the most popular include identity and access control (59 percent), data encryption (52 percent) and data segregation (42 percent).

The precaution against your possible big data security challenges is putting security first. It is particularly important at the stage of designing your solution’s architecture. Because if you don’t get along with big data security from the very start, it’ll bite you when you least expect it.

5. Lack of data professionals

To run these modern technologies and Big Data tools, companies need skilled data professionals. These professionals will include data scientists, data analysts and data engineers who are experienced in working with the tools and making sense out of huge data sets.

Companies face a problem of lack of Big Data professionals. This is because data handling tools have evolved rapidly, but in most cases, the professionals have not. Actionable steps need to be taken in order to bridge this gap.

Companies are investing more money in the recruitment of skilled professionals. They also have to offer training programs to the existing staff to get the most out of them.

Another important step taken by organizations is the purchase of data analytics solutions that are powered by artificial intelligence/machine learning. These tools can be run by professionals who are not data science experts but have basic knowledge. This step helps companies to save a lot of money for recruitment.

upvote pls....