Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy Levels

Oliver Alonzo, Jessica Trussell, Matthew Watkins, Sooyeon Lee, Matt Huenerfauth

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Scopus citations

Abstract

Research has revealed benefits and interest among Deaf and Hard-of-Hearing (DHH) adults in reading-assistance tools powered by Automatic Text Simplification (ATS), a technology whose development benefits from evaluations by specific user groups. While prior work has provided guidance for evaluating text complexity among DHH adults, researchers lack guidance for evaluating the fluency of automatically simplified texts, which may contain errors from the simplification process. Thus, we conduct methodological research on the effectiveness of metrics (including reading speed; comprehension questions; and subjective judgements of understandability, readability, grammaticality, and system performance) for evaluating texts controlled to be at different levels of fluency, when measured among DHH participants at different literacy levels. Reading speed and grammaticality judgements effectively distinguished fluency levels among participants across literacy levels. Readability and understandability judgements, however, only worked among participants with higher literacy. Our findings provide methodological guidance for designing ATS evaluations with DHH participants.

Original languageEnglish (US)
Title of host publicationCHI 2022 - Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450391573
DOIs
StatePublished - Apr 29 2022
Externally publishedYes
Event2022 CHI Conference on Human Factors in Computing Systems, CHI 2022 - Virtual, Online, United States
Duration: Apr 30 2022May 5 2022

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Conference

Conference2022 CHI Conference on Human Factors in Computing Systems, CHI 2022
Country/TerritoryUnited States
CityVirtual, Online
Period4/30/225/5/22

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design

Keywords

  • accessibility
  • automatic text simplification
  • deaf and hard-of-hearing
  • methodological research

Fingerprint

Dive into the research topics of 'Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy Levels'. Together they form a unique fingerprint.

Cite this