Hi everyone, Hannah here and today I’m going to be doing a whistle stop tour through some psychometric concepts. For my previous blog submission, I discussed health outcomes – as my PhD is focused on how to use health-related quality of life data collected during dementia trials and studies. As part of my PhD, I have reviewed specific psychometric properties of certain health-related quality of life measures for use in dementia populations, and I plan on using psychometric techniques within my analyses. As I have previously explained, I am a community pharmacist my background, and learning these non-clinical academic concepts has been a learning curve for me. I am enjoying it, but I hope that this blog may be helpful to anyone else out there that may require a little introduction to the world of psychometrics (…just like I did!).
The world of psychometrics covers a wide range of concepts, but for the purpose of this blog, I will be mainly focusing on the concepts related to outcome development and validation. This is the area of psychometrics that I know best and feel the most confident reporting on.
But first of all, what is psychometrics… right? Psychometrics is the scientific field concerned with the measurement of subjective judgements using numerical scales, and the assessment of the measurement properties of said scales (totally relevant and related to health outcomes, and indeed quality of life assessment). The field of psychometrics was established over 100 years ago as a subdiscipline within psychology, defining the classic theories and methods for constructing measures, scaling responses, and evaluating measurement scales that have been produced to assess abstract psychological concepts such as personality, attitude, and other social/behavioural constructs.
The scientific association with psychometrics stems from its origins, rooted in rigorous laboratory methods used by psychophysicists in the mid-19th century when studying brain-behaviour relationships.
These methods were used to build formulas to model people’s subjective experience to physical stimuli such as sounds and sensation. The same psychometric methods were applied in the development of tests of intelligence, attitude, and personality over the past decades. These gold-standard scientific methods, borrowed from social sciences are now applied in health-related research – allowing policy and decision makers, clinicians, and researchers to determine whether a health outcome is a suitable measure that provides scientifically credible information.
To evaluate a measurement instrument, there are various tests of psychometric properties that can be conducted to establish whether the instrument meets the criteria to be considered a scientifically robust instrument. Following the development of a measure, at the field testing stage, there are 4 key psychometric properties to test: acceptability/feasibility, validity, reliability, and responsiveness.
- Acceptability and feasibility refer to the users experience of the measure. A questionnaire that is considered acceptable and feasible will produce minimal missing data, resulting in high quality data output.
- Validity refers to the extent to which a questionnaire measures that which it was intended to measure. As a result, there are 3 types of validity: construct, content and convergent validity. Measuring validity allows us to establish the degree of confidence that can be placed in the conclusions drawn from the questionnaires scores.
- Reliability refers to the extent to which a questionnaire is free from error. There are 3 key types of reliability: internal consistency, inter-rater reliability, and test-retest reliability. An instrument that is considered reliable is internally consistent and produces steady, repeatable scores.
- Responsiveness refers to the extent to which a measure is able to detect clinical changes, particularly those which are significant and occur over time.
Now that you’re introduced to the 4 key psychometric constructs related to outcome development and validity testing, I will now run through some examples of HOW these properties are tested to determine whether the instrument in question meets the standard criteria to be determined scientifically robust:
- Acceptability is tested through assessing the quality of the data via completeness of data and score distributions. The data can be considered of decent quality is the missing data for summary scores is below 5%, there is an even distribution of responses and there are minimal floor and ceiling effects observed for summary scores (generally <10%). Floor and ceiling effects occur when a significant proportion of the respondents score the best/ maximum or worst/minimum score – making the data more difficult to interpret
- The various types of validity have different methods for testing. Construct validity, for example, is demonstrated when there is evidence that the construct/idea of interest is being measured by the instrument and that its item scores sum together. This is assessed via internal consistency using Cronbach’s alpha and via correlation with other measures that capture the construct of interest
- Again, the various types of reliability have different methods of analysis. For example, inter-rater reliability is defined by the agreement between independent raters and can be measured via calculating the intra-class correlation coefficients; for which a coefficient of >0.7 is considered acceptable
- Finally, responsiveness is measured by the ability of the measure to detect clinically important changes over time. This is therefore assessed when longitudinal data is available and can use statistical tests such as t-tests, standardised response means and effect sizes. The significance between paired observations can be observed and compared with differences of expected magnitudes
Psychometric analysis is a valuable tool that can provide important insights into individual differences between measurable constructs. Through its rigorous testing and statistical procedures, psychometric analysis allows researchers to measure, compare, and evaluate various characteristics such as personality, attitudes, and quality of life. This information can be used to make informed decisions in various fields. While psychometric analysis has its limitations and challenges, it remains a crucial tool for advancing our understanding of human behaviour and cognition. As our understanding of psychometrics continues to evolve, we can expect to see new developments and applications in the years to come.
Thanks for reading and listening! Hannah.
Hannah Hussain
Author
Hannah Hussain [1] is a PhD Student in Health Economics at The University of Sheffield. As a proud third generation migrant and British-Asian, her career path has been linear and ever evolving, originally qualifying as a Pharmacist in Nottingham, then Health Economics in Birmingham. Her studies have opened a world into Psychology, Mental Health and other areas of health, and with that and personal influences she found her passion for dementia.