Question

Some modelers prefer to partition the data into three data sets (training/validation/test) vs. the more typical two data sets (training/validation). A test set can be used during the final modeling step to measure the expected prediction error in practice given that it has been totally separated from the modeling/validation process. Do you think it is important to partition the data into three data sets (training/validation/test) or just two (training/validation)? Justify your opinion by discussing the pros and cons of each partitioning process.

Answer 1

Training data set is the information used to train an algorithm The training data set includes both input data and the corresponding expected output. Based on this “ground-truth data, you can train an algorithm to apply technologies such as neural networks, to learn and produce complex results, so that it can make accurate decisions when later presented with new data.

Validation data set contains input and target information, which is new to the algorithm. You can use it to determine whether the algorithm can correctly identify relevant new examples. At this stage, you can discover new values that are impacting the algorithm’s processing. After validation, data scientists must often go back to the training data set to make the algorithm more precise and accurate.

Test data set is used after a lot of improvement and validation. It includes only input data, no corresponding output. The test set is used to assess how well your algorithm was trained, and to estimate model properties.

In machine learning, the study and construction of algorithms that can learn from and make predictions on data is a common task. Such algorithms work by making data-driven predictions or decisions, through building a mathematical model from input data. Well, most ML models are described by two sets of parameters. The 1st set consists in “regular” parameters that are “learned” through training. The other parameters, called hyper parameters or meta-parameters are parameters which values are set before the learning starts (think, for example, the learning rate, the regularization parameter, the number of layers or neurons in a layer for ANN etc.)

Obviously, different values for those parameters may lead to different (sometimes by a lot) generalization performance for our Machine Learning model therefore we need to identify a set of optimal values for them and this is done by training multiple models with different values for the hyper parameters (how to chose those values falls under the name of hyper parameter optimization.

Now, imagine you have you data and you need to run a supervised ML algorithm on it. You split the data into:

• training - this is the data for which your algorithm knows the “labels” and which you will feed it to the training process to build your model.
• test - this is a portion of the data that you keep hidden from your algorithm and only use it after the training takes places to compute some metrics that can give you a hint on how your algorithm behaves. For each item in you test data set you predict its “value” using the built model and compare against the real “value”.

Finally, the test data set is a data set used to provide an unbiased evaluation of a final model fit on the training data set. When the data in the test data set has never been used in training (for example in cross-validation), the test data set is also called a holdout data set.

A training data set is a data set of examples used for learning, that is to fit the parameters (e.g., weights) of, for example, a classifier.Most approaches that search through training data for empirical relationships tend to over fit the data, meaning that they can identify apparent relationships in the training data that do not hold in general.

A validation data set is a data set of examples used to tune the hyper parameters (i.e. the architecture) of a classifier. It is sometimes also called the development set or the "dev set". In artificial neural networks, a hyper parameter is, for example, the number of hidden units. It, as well as the testing set (as mentioned above), should follow the same probability distribution as the training data set.