Question

Define reliability and validity in your own words. Briefly describe the different types of reliability and validity, including why it is important to conduct a study that is reliable and valid.

Answer 1

Reliability and validity are two concepts in research that are important for defining and measuring bias and distortion. Reliability refers to the extent to which assessments are consistent. If a person takes the same personality test several times and always receives the same results, the test is said to be reliable.

Types of Reliability

1. Test-retest reliability – This is a measure of reliability obtained by administering the same test twice over a period of time to a group of individuals. The scores from Time 1 and Time 2 can then be correlated in order to evaluate the test for stability over time. For example:  A test designed to assess student learning in psychology given to a group of students twice, the second time after a gap of a week of the first test. The obtained correlation coefficient would indicate the stability of the scores.

2. Parallel forms reliability- This is a measure of reliability obtained by administering different versions of an assessment tool. Both versions must contain items that probe the same construct, skill, knowledge base, etc. to the same group of individuals. The scores from the two versions can then be correlated in order to evaluate the consistency of results across alternate versions. For example:  To evaluate the reliability of an intelligence assessment tool, a large set of items that all pertain to intelligence assessment tool is created and then randomly divided the questions into two sets, that would represent the parallel forms.

3. Inter-rater reliability-This is a measure of reliability used to assess the degree to which different judges or raters agree in their assessment decisions. Inter-rater reliability is useful because human observers do not interpret answers the same way. For example: Three professors are invited to judge the modern art paintings.

4. Internal consistency reliability – This is a measure of reliability used to evaluate the degree to which different test items that probe the same construct produce similar results.  It is of two types;

a.Average inter-item correlation - It is obtained by taking all of the items on a test that probe the same construct (e.g., reading comprehension), determining the correlation coefficient for each pair of items, and finally taking the average of all of these correlation coefficients. This final step yields the average inter-item correlation.

b.Split-half reliability- All the items of a test that are intended to probe the same area of knowledge are split in half in order to form two “sets” of items. The entire test is administered to a group of individuals, the total score for each “set” is computed, and finally, the split-half reliability is obtained by determining the correlation between the two totals “set” scores.

Validity refers to how well a test measures what it is purported to measure. If the results of the personality test claimed that a very shy person was, in fact, outgoing, the test would be invalid.

Types of Validity

1. Face Validity ascertains that the measure appears to be assessing the intended construct under study. It is easy to assess. Although this is not a very “scientific” type of validity, it may be an essential component in enlisting motivation of stakeholders. For example: If a measure of art appreciation is created all of the items should be related to the different components and types of art. If the questions are regarding historical time periods, with no reference to any artistic movement, stakeholders may not be motivated to give their best effort or invest in this measure because they do not believe it is a true assessment of art appreciation.

2. Construct Validity is used to ensure that the measure actually measures what it is intended to measure (i.e. the construct), and not other variables. Using a panel of “experts” familiar with the construct is a way in which this type of validity can be assessed. For example, A questionnaire to test the awareness among farmers is designed.The questions are written with complicated wording and phrasing. This can cause the test inadvertently becoming a test of reading comprehension, rather than a test of awareness among farmers. It is important that the measure is actually assessing the intended construct, rather than an extraneous factor.

3. Criterion-Related Validity is used to predict future or current performance. For example: If a biology program designed a measure to assess cumulative student learning throughout the major. The new measure could be correlated with a standardized test in this discipline, such as an ETS field test or the GRE subject test. The higher the correlation between the established measure and new measure, the more faith stakeholders can have in the new assessment tool.

4. Formative Validity - It is used to assess how well a measure is able to provide information to help improve the program under study. For example: When designing a rubric for biology one could assess student’s knowledge across the discipline. If the measure can provide information that students are lacking knowledge in a certain area, for instance, cell biology, then that assessment tool is providing meaningful information that can be used to improve the course or program requirements.

5. Sampling Validity- ensures that the measure covers the broad range of areas within the concept under study. For example: When designing an assessment of learning in the theatre department, it would not be sufficient to only cover issues related to acting. Other areas of theatre such as lighting, sound, functions of stage managers should all be included.

Reliability and validity are independent of each other. A measurement may be valid but not reliable, or reliable but not valid. For example, your bathroom scale was reset to read ten pounds lighter. The weight it reads will be reliable because it gives you the same weight every time you step on it but this result is not valid since it is not reading your actual weight.