Brian O'Connor   UBCO Psychology   UBCO  

IRT Test Information and the Attenuation of Effect Sizes


The test information values that are produced in item response theory (IRT) analyses are probably ambiguous to everyone but IRT experts. Test information values are not on a zero-to-one scale, and the possible maximum values for test information vary greatly across measures. O'Connor (2018a) attempted to enhance interpretations of test information values by translating them into more familiar metrics. O'Connor(2018b) provided R code (1) for revealing the attenuation in effect sizes that occurs for combinations of constant test information values for two measures, and (2) for obtaining empirically-based estimates of the degree of attenuation that is caused by fluctuating levels of measurement error in specific datasets.

The R code for the analyses is now provided in a package named "IRTtestinfo". The package permits users to estimate the degree of effect size attenuation that occurred for their specific datasets. The degree attenuation is determined empirically rather than by formulas. One benefit of this method is that it preserves and takes into account the distributions of the user's sample-specific raw data values. The R code provides answers to these two questions: When there are two normally-distributed, correlated variables in a population, how much attenuation due to measurement error is caused by the fluctuating levels of test information for the measures and by the sample-specific variable distributions? What is the corrected-for-attenuation estimate of the correlation between the two variables?


O'Connor, B. P. (2018a). An illustration of the effects of fluctuations in test information on measurement error, the attenuation of effect sizes, and diagnostic reliability. Psychological Assessment, 30(8), 991-1003.

O'Connor, B. P. (2018b). Clarifications regarding test information and reliability, and new methods for estimating attenuation due to measurement error: Reply to Markon (2018). Psychological Assessment, 30(8), 1010-1012.



Run these commands to install and use the IRTtestinfo package:

install.packages("remotes", dependencies = TRUE) # install the remotes package

library(remotes) # load the remotes package

remotes::install_github("bpoconnor/IRTtestinfo") # use remotes to install IRTtestinfo from github

# if you get prompted about more recent versions of packages, select and choose to update them all

library(IRTtestinfo)

The IRTtestinfo package files are on GitHub

click here for the Reference manual for the IRTtestinfo package

Here is an illustration of the proportionate reductions in effect size that occur for various combinations of constant levels of IRT test information:

Logo

Brian P. O'Connor
Department of Psychology
University of British Columbia - Okanagan
Kelowna, British Columbia, Canada
brian.oconnor@ubc.ca