Silent Speech Eyewear Interface: Silent Speech Recognition Method Using Eyewear and an Ear-Mounted Microphone with Infrared Distance Sensors.

Publisher: MDPI Country of Publication: Switzerland NLM ID: 101204366 Publication Model: Electronic Cited Medium: Internet ISSN: 1424-8220 (Electronic) Linking ISSN: 14248220 NLM ISO Abbreviation: Sensors (Basel) Subsets: MEDLINE

Original Publication: Basel, Switzerland : MDPI, c2000-

Wearable Electronic Devices* ; Speech*/physiology; Humans ; Ear/physiology ; Infrared Rays ; Smart Glasses ; Speech Recognition Software ; Machine Learning

As eyewear devices such as smart glasses become more common, it is important to provide input methods that can be used at all times for such situations and people. Silent speech interaction (SSI) has the potential to be useful as a hands-free input method for various situations and people, including those who have difficulty with voiced speech. However, previous methods have involved sensor devices that are difficult to use anytime and anywhere. We propose a method for SSI that involves using an eyewear device equipped with infrared distance sensors. The proposed method measures facial skin movements associated with speech from the infrared distance sensor mounted on an eyewear device and recognizes silent speech commands by applying machine learning to time series sensor data. The proposed method was applied to a prototype system including a sensor device consisting of eyewear and ear-mounted microphones to measure the movements of the cheek, jaw joint, and jaw. Evaluations 1 and 2 showed that five speech commands could be recognized with an F value of 0.90 and ten longer speech commands with an F value of 0.83. Evaluation 3 showed how the recognition accuracy changes with the combination of sensor points. Evaluation 4 examined whether the proposed method can be used for a larger number of speech commands with 21 commands by using deep learning LSTM and a combination of DTW and kNN. Evaluation 5 examined the recognition accuracy in some situations affecting recognition accuracy such as re-attaching devices and walking. These results show the feasibility of the proposed method for a simple hands-free input interface, such as with media players and voice assistants. Our study provides the first wearable sensing method that can easily apply SSI functions to eyewear devices.

Proc ACM Interact Mob Wearable Ubiquitous Technol. 2020 Jun;4(2):. (PMID: 34222759)

2023 Research grant KDDI Foundation; JP19K20330 Japan Society for the Promotion of Science

Keywords: eyewear; hands-free input; infrared distance sensor; silent speech interaction; skin motion sensing

Date Created: 20241127 Date Completed: 20241127 Latest Revision: 20241130

20241202

PMC11598394

10.3390/s24227368

39599144