Reliability

Many details about reliability, its role, the indexes that measure it, etc., are known in the scientific community. Still, many other details are hardly known or applied. We try to advance the relevant use of reliability with several of our research projects.

See a presentation about some less known properties of reliability from the viewpoint of cognitive science.

The relation between reliability and the power of the hypothesis tests

The relation between reliability and power is complex. On the one hand, it is well known that reliability attenuates the measured correlation and, consequently, the power of the related hypothesis test. On the other hand, some robust cognitive (and other) phenomena are not reliable, termed the reliability paradox (e.g., Hedge, Powell, Sumner, 2018). This issue has been discussed and investigated in the literature. Here, I attempt to provide a more extensive, still accessible summary and explanation for the key relations between reliability and statistical power.

The manuscript is in preparation. Check back later for it.

Theoretical maximum of reliability

We often think about reliability as the result of some noise in the environment or in the participant that comes from occasional events or impulses. Still, according to some models, random noise is an inherent part of the processing. This latter description means an unavoidable noise in the measurement and, consequently, a theoretical maximum in the reliability. In the present project, it is investigated which factors influence the reliability and how. In addition, a software solution is provided to quantify the expected maximum possible reliability after some key parameters are estimated.

The manuscript is in preparation. Check back later for it.

The script is in preparation. Check back later for it.

Calculating reliability in CogStat

CogStat is a data and statistical analysis software that runs the analyses automatically and compiles an output that is easier to understand and interpret. In version 2.4, we add some initial support for reliability analyses and plan to improve them in later releases.

The example of Alternating Serial Reaction Time task

We investigated some fundamental properties of the ASRT task’s reliability.

Farkas, B. C., Krajcsi, A., Janacsek, K., & Nemeth, D. (2023). The complexity of measuring reliability in learning tasks: An illustration using the Alternating Serial Reaction Time Task. Behavior Research Methods, 1–17. https://doi.org/10.3758/s13428-022-02038-5. Or read the preprint.