Support in the Moment: Benefits and use of video-span selection and search for sign-language video comprehension among ASL learners

Saad Hassan, Akhter Al Amin, Calua De Lacerda Pataca, Diego Navarro, Alexis Gordon, Sooyeon Lee, Matt Huenerfauth

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

As they develop comprehension skills, American Sign Language (ASL) learners often view challenging ASL videos, which may contain unfamiliar signs. Current dictionary tools require students to isolate a single sign they do not understand and input a search query, by selecting linguistic properties or by performing the sign into a webcam. Students may struggle with extracting and re-creating an unfamiliar sign, and they must leave the video-watching task to use an external dictionary tool. We investigate a technology that enables users, in the moment, i.e., while they are viewing a video, to select a span of one or more signs that they do not understand, to view dictionary results. We interviewed 14 American Sign Language (ASL) learners about their challenges in understanding ASL video and workarounds for unfamiliar vocabulary. We then conducted a comparative study and an in-depth analysis with 15 ASL learners to investigate the benefits of using video sub-spans for searching, and their interactions with a Wizard-of-Oz prototype during a video-comprehension task. Our findings revealed benefits of our tool in terms of quality of video translation produced and perceived workload to produce translations. Our in-depth analysis also revealed benefits of an integrated search tool and use of span-selection to constrain video play. These findings inform future designers of such systems, computer vision researchers working on the underlying sign matching technologies, and sign language educators.

Original languageEnglish (US)
Title of host publicationASSETS 2022 - Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450392587
DOIs
StatePublished - Oct 22 2022
Externally publishedYes
Event24th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2022 - Athens, Greece
Duration: Oct 23 2022Oct 26 2022

Publication series

NameASSETS 2022 - Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility

Conference

Conference24th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2022
Country/TerritoryGreece
CityAthens
Period10/23/2210/26/22

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design
  • Computer Networks and Communications
  • Hardware and Architecture
  • Human-Computer Interaction
  • Software

Keywords

  • ASL Learning
  • American Sign language
  • Continuous Signing
  • Integrated Search
  • Search Interface
  • Sign Language Learning
  • Sign Language Videos
  • Sign Languages
  • Sign Look-up
  • Video Selection

Fingerprint

Dive into the research topics of 'Support in the Moment: Benefits and use of video-span selection and search for sign-language video comprehension among ASL learners'. Together they form a unique fingerprint.

Cite this