To validate the performance of automated scoring algorithms for the Rey Auditory Verbal Learning Test (RAVLT), Letter Fluency Test, Rey-Osterrieth Complex Figure Test (ROCFT), and Clock Drawing Test implemented in the Philips IntelliSpace Cognition (ISC) digital test battery.

Traditional cognitive tests often require the clinician to manually score performance, for example, by evaluating drawings according to a standardized system or by judging the validity of verbal responses. This is a tedious process with room for error and subjectivity. We developed several scoring algorithms to automate and standardize the scoring process. We hypothesized that the scores provided by the algorithms would be practically equivalent to those given by expert human raters.

We randomly sampled 100 drawings (i.e., for Clock Drawing Test and ROCFT) or 100 audio recordings (i.e., for RAVLT and Letter Fluency) from a stratified sample consisting of 450 US-based healthy participants. The ROCFT and Clock drawings were scored by separate designated deep learning algorithms and six expert human raters. The audio recordings were scored by speech recognition algorithms and three expert human raters. For each test, we calculated agreement between human raters through intraclass correlations (ICC) as a benchmark for the algorithm. We tested equivalence between the human interrater ICC and a human-algorithm ICC, with equivalence defined as a maximum deviation of 0.1 points.

Human interrater ICCs were high, with values ranging from .85 for Clock Drawing Test to .99 for all RAVLT scores except the learning trial. All algorithm-human rater comparisons yielded ICCs well within the predefined range of practical equivalence with human rater scores.

The automated scoring algorithms implemented in the ISC test battery were shown to be reliable substitutes for manual scoring.