Validity: Why Are Researchers Considered About Validity of Data?
Measurement: Assigning a numerical value to a feature of an object or occurrence so that it can be compared to other objects or occurrences
1. We use two major processes in measurement conceptualization and operationalization.
a. Conceptualization refers to taking an abstract construct and refining it by giving it a conceptual or theoretical definition. Conceptualization is the process of thinking through the various possible meanings of a construct.
b. Operationalization on the other hand refers to the process of moving from a construct’s conceptual definition to specific activities or measures that allow a researcher to observe it empirically.
c. For example, effectiveness of a campaign for a micro-credit scheme amongst a group of workers is an outcome that can vary from very effective to very ineffective. Effectiveness of a campaign is however a function of overall how appealing, easy to understand, relevant and persuasive the campaign is seen to be. Also, in case a campaign has high effectiveness it manifests itself in high signups for the program.
Validity and reliability are central constructs that help establish the truthfulness, credibility, and believability of findings.
Validity describes as the degree to which a research study measures what it intends to measure. For example, if the objective of the survey is to measure the disposition to avail of a micro-credit scheme amongst a group of workers; and the survey uses constructs that measure the intention to avail of the micro-credit then the data is said to be valid. There are two main types of validity for analysing the appropriateness and meaningfulness of data - internal and external validity. Internal validity refers to the validity of the measurement and test itself, whereas external validity refers to the ability to generalise the findings to the target population.
A closer look at validity:
Validity suggests truthfulness. It refers to how well an idea “fits” with actual reality. The absence of validity means that the fit between the ideas we use to analyze the social world and what occurs in the lived social world is poor. Validity measures how well an idea aligns with actual reality. E.g., Personality tests should measure a person’s “REAL” characteristics, and not the characteristics they want to show others. Validity means accuracy / correctness e.g., IQ test should measure mental ability, not exam-smartness.
The 4 main types of internal validity are:
There are 4 main types of validity used when assessing internal validity.
Criterion validity: This test of validity relies on some independent, outside verification (a.k.a. criterion).
Example: in personnel selection, we’re interested in predicting the individual’s future performance in a criterion or outcome measure (e.g., customer satisfaction) via his/her scores on a selection test (e.g., service orientation scale).
There are therefore two parts to criterion validity i.e. concurrent validity compares the results from a new measurement technique to those of a more established technique. In this case established technique is the customer satisfaction and the new data is the service orientation measure. Hence, concurrent validity would involve collecting service orientation data and customer satisfaction data. Also, correlation will be done between both these pieces of data.
The second validity is predictive validity-This is when the results obtained from measuring a construct can be accurately used to predict behaviour.
Content validity:
This refers to the extent to which the elements or facets of a measure reflects the construct it is intended to measure. This measures evidence that the content of the measure holistically reflects the construct it is intended to measure.
Content validity measurement encompasses besides the content or construct the following elements 1) measurement format (e.g., computer vs face-to-face), 2) response format (e.g., MCQ vs verbal responding),3) framing of the questions and simplicity, 4) administration
and scoring of the questionnaire.
For example, if we decide to study the construct of FOMO; and decide to use the 3 metrics of 1) I want to find out what others are doing, 2) I tend to compare my experiences with others, and I am afraid others will leave me out of their activities-
EXAMPLE OF HIGH CONTENT VALIDITY
Construct validity:
This measure evaluates the degree to which the latent construct which is not observable can account for variation in manifest variable which is observable e.g., anxiety is a latent construct. Thus, to use construct validity we need to define at the outset what are the latent and manifest variables. To elaborate: Stress is a latent construct; but it is manifested through fidgeting behavior, scores on anxiety and short temper scale etc. (see next slide).
To establish construct validity this measure evaluates using the convergent and discriminant validity. Convergent validity: looks at the association between related variables and means that the indicators of one construct “hang together,” or converge. Thus, in the earlier example if a person is stressed, they should also exhibit all 3 manifest variables of fidgeting behavior, and similar scores on anxiety and short temper scale.
Discriminant validity: Discriminant validity is the opposite of convergent validity and means that the indicators of one construct “hang together,” or converge, but also are negatively associated with opposing constructs. Hence, if a person is stressed, they should have low scores on metrics like happy, carefree etc.
This test of validity is used to reiterate the ‘truthfulness;’ of a construct that are derived using regression or SEM. For example, if we get a construct that risk taking is a function of being an extrovert, a leader and creative via a regression; than construct validity can be used to test the truthfulness of this framework.
EXAMPLE OF HIGH CONVERGENCE VALIDITY (CONSTRUCT VALIDITY) BUT LOW CONTENT VALIDITY
Threats to External Validity: External validity refers to how generalizable the findings are. Can the results be generalized to other people, places, circumstances, and periods of time? Threats to external validity are any factors within a study that reduce the generalizability (or generality) of the results. The four of the main threats to external validity are (a) selection biases; (b) presence of ambiguity in constructs or and confounding variables (c) divergence between the 'real world' versus the 'experimental world'.
What are the threats to internal validity? Factors that can affect internal validity
1. External events in the participant’s life that have led to a change in their mood etc.
2. Instrument led bias refers to a change in the measuring instrument due to incorrect phrasing or translation.
3. Interviewer led bias- errors in data capture
4. Breach of data collection protocol
5. Respondent fatigue and boredom.