In: Computer Science
You have a table called scholar where there are two fields id and create date. You want to create an ETL process that loads the data from scholar table into the big data platform. What are the different things that you will put on a requirement document for the developer to write code and implement the solution?
Answer:
We dont hve any infromation about big data model in question so below document will have a generalized approach.
Requirement Document
1. Extraction: Since we have only 2 parameters. We will have a DB call according to the range of Ids which will fetch the created date filled for the records.
2. Load: We will have the extracted records in the form of the list and we will feed the list to the big data model(list is easily iterable and we have few fields to handle).
3.Transform: The big data model will give us the response against the fed data (which will be the analysis of the data). which will be in the form of dictionary and will be stored in the new DB table in the form of created date against the anlaysed data (created date{date time field}, model_response{dictionary}).
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
# Feel free to ask questions in the comment section
# Hit the like if my answer if it is worth helping to you.
# you can payback and help me with a like.