Research & Innovation based on the research by Dr. Cara Stepp
Human-machine interfaces (HMIs) can utilize ones biosignals in order to perform tasks, typically those involving communication or mobility. For those who have lost normal user functions, HMIs can provide a means to restoring them. Many HMIs require visual feedback from the user causing their normal vision to become distracted. To resolve this problem, auditory feedback and facial expressions are being tested as ways to control HMI output.
A common biosignal used in HMIs is surface electromyography (sEMG) which is used by the HMI to convert muscle activity into interface commands. A study1 at the University of Technology Malaysia tested eleven facial gestures to determine which are most effective in controlling HMIs. Surface EMG signals were recorded from cranial muscles during a combination of gestures. A differentiation accuracy of 90% was found, proving that sEMG signals can be reliably used to operate HMIs
keywords: human-machine interfaces, electromyography, visual feedback, auditory feedback
Dr. Cara Stepp
Dr. Cara Stepp has worked extensively with HMIs and other rehabilitative devices used to restore ones daily functions. She has utilized EMG signals from the face and neck as a way to control an electrolarynx2. Furthermore, she has analyzed the effect of visual feedback training on auditory-motor performance where facial sEMG signals were used to control the HMI3. She continues to research the use of auditory feedback for HMI control.
Dr. Stepp and her colleagues performed a study4 to determine the extent to which categorical speech perception can be used to improve HMIs using auditory feedback via vowel synthesis. Users were separated into one of three test groups.
The first two controlled an HMI using auditory and text cues for specific American English vowels and a third used auditory cues that were vowel-like sounds, but not part of American English. Surface EMG signals were collected from the left and right orbicularis oris muscles using a Delsys Bagnoli™ EMG System. The sensors provided a high signal-to-noise ratio (SNR) input which could be utilized in visual and auditory feedback training. The sEMG signals provided real-time visual feedback during training allowing the subjects to identify the accuracy of their movements. Over time the visual cues were removed in order to identify the performance with the auditory feedback alone. The results showed that auditory feedback can be effective in HMI control when paired with established targets, although including both auditory and visual feedback can further improve HMI operation performance.
The ability to use auditory feedback in HMIs can provide users with control even without full visual function. Dr. Stepp concludes that HMIs can be effectively controlled through the use of auditory feedback if paired with established targets. However, auditory feedback alone still remains slightly less effective than the combination of auditory and visual feedback.
- Ali J, Dee-Uam C, Hamedi M, Ismail K, Pavaganum C, Salleh Sh-Hussain, Tan T, Yupapin P. Human facial neural activities and gesture recognition for machine-interfacing applications. International Journal of Nanomedicine. Vol. 6: 3461-72. 16 December 2011.
- Stepp, C.E.; Heaton, J.T.; Rolland, R.G.; Hillman, R.E., “Neck and Face Surface Electromyography for Prosthetic Voice Control After Total Laryngectomy,” Neural Systems and Rehabilitation Engineering, IEEE Transactions on , vol.17, no.2, pp.146,155, April 2009.
- Hands GL, Larson E, Stepp CE. Effects of augmentative visual training on audio-motor mapping, Human Movement Science, 12 February 2014, ISSN 0167-9457, http://dx.doi.org/10.1016/j.humov.2014.01.003.
- Larson E, Terry HP, Canevari MM, Stepp CE (2013) Categorical Vowel Perception Enhances the Effectiveness and Generalization of Auditory Feedback in Human-Machine-Interfaces. PLoS ONE 8(3): e59860. doi:10.1371/journal.pone.0059860.
Delsys EMG Systems
Delsys EMG Systems feature wired and wireless surface EMG sensors designed to produce a consistent EMG signal with low muscle crosstalk and reduced motion artifact.
The EMGworks® software acquires and analyzes EMG signals. EMGScript can determine maximum voluntary contraction percentage (%MVC) of each orbicularis oris muscle to map power activity to vowel targets for use in controlling HMIs.
The Study Parameters
HMIs were controlled using:
- Auditory and text cues of American English vowels
- Auditory cues of American English vowels only
- Auditory cues for Vowel-like sounds
The Study Focus
EMG providing high SNR input
EMG used as real-time biofeedback
Delsys Instruments Used in the Study
2 channel Bagnoli™ EMG System