TY - GEN
T1 - Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy Levels
AU - Alonzo, Oliver
AU - Trussell, Jessica
AU - Watkins, Matthew
AU - Lee, Sooyeon
AU - Huenerfauth, Matt
N1 - Publisher Copyright:
© 2022 ACM.
PY - 2022/4/29
Y1 - 2022/4/29
N2 - Research has revealed benefits and interest among Deaf and Hard-of-Hearing (DHH) adults in reading-assistance tools powered by Automatic Text Simplification (ATS), a technology whose development benefits from evaluations by specific user groups. While prior work has provided guidance for evaluating text complexity among DHH adults, researchers lack guidance for evaluating the fluency of automatically simplified texts, which may contain errors from the simplification process. Thus, we conduct methodological research on the effectiveness of metrics (including reading speed; comprehension questions; and subjective judgements of understandability, readability, grammaticality, and system performance) for evaluating texts controlled to be at different levels of fluency, when measured among DHH participants at different literacy levels. Reading speed and grammaticality judgements effectively distinguished fluency levels among participants across literacy levels. Readability and understandability judgements, however, only worked among participants with higher literacy. Our findings provide methodological guidance for designing ATS evaluations with DHH participants.
AB - Research has revealed benefits and interest among Deaf and Hard-of-Hearing (DHH) adults in reading-assistance tools powered by Automatic Text Simplification (ATS), a technology whose development benefits from evaluations by specific user groups. While prior work has provided guidance for evaluating text complexity among DHH adults, researchers lack guidance for evaluating the fluency of automatically simplified texts, which may contain errors from the simplification process. Thus, we conduct methodological research on the effectiveness of metrics (including reading speed; comprehension questions; and subjective judgements of understandability, readability, grammaticality, and system performance) for evaluating texts controlled to be at different levels of fluency, when measured among DHH participants at different literacy levels. Reading speed and grammaticality judgements effectively distinguished fluency levels among participants across literacy levels. Readability and understandability judgements, however, only worked among participants with higher literacy. Our findings provide methodological guidance for designing ATS evaluations with DHH participants.
KW - accessibility
KW - automatic text simplification
KW - deaf and hard-of-hearing
KW - methodological research
UR - http://www.scopus.com/inward/record.url?scp=85130566084&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85130566084&partnerID=8YFLogxK
U2 - 10.1145/3491102.3517566
DO - 10.1145/3491102.3517566
M3 - Conference contribution
AN - SCOPUS:85130566084
T3 - Conference on Human Factors in Computing Systems - Proceedings
BT - CHI 2022 - Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
PB - Association for Computing Machinery
T2 - 2022 CHI Conference on Human Factors in Computing Systems, CHI 2022
Y2 - 30 April 2022 through 5 May 2022
ER -