IpsiHand: Direct Recoupling of Intention and Movement (Washington University in St. Louis)

Tile image shows complete prototype system laid out including 1)Emotiv EPOC EEG recording headset, 2)laptop, and 3)modified hand orthotic for grasp control

Sam Fok, Raphael Schwartz, Mark Wronkiewicz, Charles Holmes, Jessica Zhang, Nathan Brodell, Thane Somers (Washington University in St. Louis)

ABSTRACT

Stroke and traumatic brain injury (TBI) cause long-term, unilateral loss of motor control due to brain damage on the opposing (contralateral) side of the body. Conventional neurological therapies have been found ineffective in rehabilitating upper-limb function after stroke. Brain computer interfaces (BCIs), devices that tap directly into brain signals, show promise in providing rehabilitation but remain in research. Also, BCIs cannot work if the target signals have been eliminated due to injury. Therefore we present a novel BCI, the IpsiHand, which combines advances in neurophysiology, electronics, and rehabilitation. Recent studies show that during hand movement, the cortical hemisphere on the same (ipsilateral) side of the body as the hand also activates. IpsiHand uses electroencephalography (EEG) to record these signals and control a powered hand orthosis.  The undamaged hemisphere can then control both hands, and through neural plasticity IpsiHand will strengthen ipsilateral neural pathways to enhance ipsilateral motor control.

YouTube Preview Image

BACKGROUND and PROBLEM STATEMENT

Stroke and TBI combined are the leading cause of disability in the US, with around a million cases annually; this poses a significant cost to the economy and decreases quality of life for affected individuals. Half report trouble with hand movement, and conventional physical therapy produces little significant improvement after 3 months post injury [1]. Lasting disabilities result in a typical lifetime cost between $100k and $2M per patient, including inpatient care, rehabilitation, and follow-up [2] [3][4]. The most effective therapies have patients actively controlling their limb, which is not an option in cases of severe paralysis. While BCIs promise new hope for treatment, they remain in the research stage.  In addition, conventional BCIs cannot be applied to cases of brain injury since the classical motor signals in cortex contralateral to the target limb needed would be gone with the injury.

We address the problem of applying BCI technology to rehabilitation following stroke and TBI.

We developed a device, IpsiHand, for rehabilitation that synthesizes recent developments in neurophysiology, electronics, and physical therapy into a BCI hand orthosis.  A recent study found signals associated with hand movements in cortex ipsilateral to the hand. These signals were present in cortex anterior to ipsilateral primary motor cortex and at frequencies below 40Hz [5], which are accessible via EEG. To acquire these intent-to-move signals we used a non-invasive, low cost EEG consumer headset to record from cortex, controlling an orthosis which opens and closes a patient’s hand. Tactile and proprioceptive feedback provided from this device will facilitate neural plasticity, strengthening existing and developing new neural pathways ipsilateral to the affected hand that will ultimately restore motor control. Allowing the patient to regain hand control with their thoughts alone should also provide tremendous encouragement in the rehabilitation process.

Our objective is to directly recouple the intent-to-move a hand with hand motion in order to improve outcomes of recovery, reduce the lifetime cost of brain injury, and improve quality of life for those affected by stroke or TBI.

METHODS

IpsiHand integrates low-cost, commercial EEG acquisition hardware, signal processing software, and a modified orthosis to control hand grasping in real time (Figure 1).

Block diagram of data flow through IpsiHand system. Brainwave EEG data is acquired and digitized by an Emotiv EPOC headset. Digitized data is sent to a laptop for signal processing. BCI2000 and LabView software conduct spatial filtering, signal feature selection, and control signal generation from EEG data. Control signal transmitted via USB to mechanical actuator mounted on adjustable, prefabricated hand orthosis.

Figure 1- Block diagram of data flow through IpsiHand system. Brainwave EEG data is acquired and digitized by an Emotiv EPOC headset. Digitized data is sent to a laptop for signal processing. BCI2000 and LabView software conduct spatial filtering, signal feature selection, and control signal generation from EEG data. Control signal transmitted via USB to mechanical actuator mounted on an adjustable, prefabricated hand orthosis.

  1. Signal Acquisition: An Emotiv EPOCTM EEG headset records EEG signals from the scalp with 14 channels digitized at 128Hz, and transmits wirelessly to a computer.
  2. 2. Signal Processing and Control: EEG is the least invasive recording technique and practical for immediate application in the clinic, but EEG signals tend to be smaller and more spatially diffuse compared invasive recording techniques. Therefore, we must maximize the signal-to-noise ratio. We use a large Laplacian reference [6], which filters noise from wide areas of the scalp, detects signals specific to a particular brain areas, and makes IpsiHand resilient to electrode placement variations. On a computer, BCI2000, a software dedicated to BCI applications [7], extracts a control signal from the EEG data with spatial filtering, frequency analysis and feature selection, and statistical normalization. For optimal control, the user trains the algorithm by alternating between periods of attempted hand movement and periods of rest. After several trials, IpsiHand locates specific areas and brainwave frequencies that consistently changed in power between hand movement and rest conditions. The resulting control signal is sent to LabView software for conversion into actuator commands.
  3. Mechanical Actuation: A Becker Oregon TalonTM prefabricated orthosis (Figure 2) is fitted with a Firgell L16 linear actuator. Driven by signals from LabView, it flexes and extends the patient’s fingers for grasping. The orthosis was chosen for adjustable sizing, which is key in a clinical setting with a variety of patients. To mechanically prevent hyperextension or hyperflexion of the hand, The range of actuator motion is mapped only onto the natural range of finger joint rotation.
Procedure for modifying a Becker Oregon Talon prefabricated wrist driven hand orthosis (Model TAL100):  Wrist-hand linkage is removed, and replaced with Firgelli L16 linear actuator mounted on a modified plate linking the hand and forearm sections of the orthosis. This allows for powered control of hand grasp.

Figure 2 - Procedure for modifying a Becker Oregon Talon prefabricated wrist driven hand orthosis (Model TAL100): Wrist-hand linkage is removed, and replaced with Firgelli L16 linear actuator mounted on a modified plate linking the hand and forearm sections of the orthosis. This allows for powered control of hand grasp.

Our intended treatment plan is designed to integrate with current standards of clinical rehabilitation therapy. A typical intensive post-stroke treatment therapy consists of 36 sessions of 45 minutes over a period of 12 weeks [10]. Fitting IpsiHand is simple with Velcro® straps, and the EPOC headset self-fits. IpsiHand’s signal processing algorithm continually adapts to ensure full range of hand motion even with weak or moving signals. Each session will include of a 5 minute training period in which the algorithm adapts to the subject’s brainwaves.  The identified signal features are then used to control the orthosis during repetitive flexion-extension hand tasks. Patients that purchase IspiHand can benefit from increased therapy time with the device outside of normal therapy sessions.

RESULTS

IpsiHand was tested with three healthy subjects to verify the ability to use non-conventional signals from cortex on one side of the brain to control a hand on the same side of the body. We found that:

1.         Hand movement correlates with signals from the ipsilateral hemisphere (Figures 3 and 4).

2.         IpsiHand successfully uses EEG signals to move the hand (video above).

The identified frequencies and electrode locations used in the study are in Figures 3 and 4 and in our video. During the study, a cursor was controlled by the identified EEG signal feature, which was modulated when a subject moved, or imagined moving his hand. The modulated signal gave the subject control of 1D cursor movement, and the subject was tasked with moving the cursor to a target that randomly appeared on either side of a computer screen. Through 10 sets of trials with non-impaired individuals we were able to achieve an 81.3% success rate for this task. We expect that with optimization of our signal detection algorithms, we can achieve success rates upwards of 90% as it was not uncommon to see success rates upwards of 90%.

Left: EPOC headset electrode positions on head. Center: Correlation values between left hand movement condition and rest condition per channel and frequency bin.  Right: Frequency spectrum of signal of channel F3 showing changes in amplitude between left hand movement and rest

Figure 3- Left: Positions of EPOC headset electrodes. Center: Correlations between left hand movement condition and rest condition per electrode channel (y axis) and per frequency bin (x axis). Electrodes over the left hemisphere are on the lower half of the y axis and electrodes over the right hemisphere are on the upper half of the y axis. Bins with high correlation values can be used to predict whether the subject is moving the left hand or not. Right: Frequency spectrum of electrode channel F3 showing changes in amplitude between left hand movement and rest

Left: Colormap overlaid on head of correlation values between left hand movement condition and rest condition in 12Hz brainwaves. Shows high correlations around all electrodes over the frontal lobes. Right: Same colormap but of 22Hz brainwaves.  Shows high correlation values only around electrode F3.

Figure 4- Left: Colormap of correlation values between left hand movement and rest conditions in 12Hz brainwaves. Note high, bilateral correlations in frontal cortex electrodes. Right: Same colormap with 22Hz brainwaves. Note relatively high correlations seen only unilaterally in electrode F3. The strength of correlation at 12Hz allows IpsiHand to distinguish left hand movement versus rest, and the correlation at 22Hz in electrode F3 allows for distinction of left versus right hand movement.

DISCUSSION, COSTS, and IMPLICATIONS

Combining neurophysiology, electronics, and rehabilitation, IpsiHand offers more effective rehabilitation for stroke and TBI survivors even in severe cases of paralysis. In testing, IpsiHand was able to process EEG signals for real-time hand control with accuracy consistent with previous studies [8]. Recent evidence suggests that combining BCIs and orthotic devices induces neural plasticity and improves motor function [8]. IpsiHand combines this advance in rehabilitation with the discovery of motor related signals in the undamaged brain hemisphere. Furthermore, the potential for recovery is unhampered by the severity of neural pathway injury since IpsiHand circumvents the entire injured pathway and uses the brain’s plasticity to generate new ones.

If produced in volume, we estimate a cost of around $674 per unit (Table 1), which even when marked up for retail would be significantly cheaper than alternative devices.

Table showing estimated price of IpsiHand system per unit. Includes price accuracy and bulk discount.

Table 1 - Prototype cost and estimated cost for production including bulk discount. Bulk discounts determined with manufacturer correspondence. *Headset donated by Emotiv Systems; this is the retail price of a headset.

Signal acquisition, signal processing, and mechanical control methods are established, but synthesizing them with the new technique of ipsilateral cortex recording is potentially new intellectual property. Based on discussions with therapists at the Rehabilitation Institute of St. Louis, IpsiHand is unique and an improvement over existing robotic assist and plasticity-facilitating devices.  Using muscle signals rather than neural signals, Myomo of Neuro-robotic Systems® uses facilitates plasticity less directly. The Bioness H200 from Ness®, an electrical muscle stimulator, has an unwieldy number of parts, is expensive (~$6200), and is a passive-assist-device not designed to promote active attempts at hand control. Hand Mentor from Kinetic Muscles Inc. does emphasize treatment through interactive movement, but is also expensive, non-portable, and not an option for those without residual motor control. IpsiHand fulfills these shortcomings by tapping into plasticity directly to create new neural pathways in portable package applicable even in severe cases [9]. Allowing patients to regain hand control with their thoughts will provide tremendous encouragement to continue with a therapy. Combined with IpsiHand’s affordability and minimal requirements for therapist supervision, IpsiHand also makes in-home treatment a very practical possibility.

Based on therapist discussions, we will make improvements upon the prototype. Currently, a laptop processes the EEG signals to be used for orthosis movement. We plan to use a miniature single-board Gumstix® computer to provide a portable data processing on a package smaller than a stick of gum. Mounting the micro-computer and a battery pack to the orthosis would give our system complete portability and allow patients to go beyond rehabilitation and use IpsiHand as a replacement of daily hand function.

Acknowledgments

We would like to especially thank Dr. Eric Leuthardt, our faculty mentor, and David Bundy, our graduate student mentor, for their guidance. We would also like to thank Joanne Rasch and the Rehabilitation Institute of St. Louis for providing consumer feedback and expert opinion on current rehabilitation, and Professors Robert Morley and Joseph Klaesner for instruction during senior design. This work is supported in part by The National Collegiate Inventors and Innovators Alliance, the Washington University School of Engineering, and Emotiv Systems.

First Author: Sam Fok:  13196 Strawberry Way, St. Louis MO 63146, sbf3@cec.wustl.edu

References

[1] HS-Jorgensen, et_al,-”Outcome_and_time_course_of-recovery_in_stroke: Part_II:_Timecourse_of_recovery. The_Copenhagen_Stroke_Study,”-Archives-of-Physical-Medicine-and-Rehabilitation,-1995.
[2] D_Lloyd-Jones, et_al., “Heart_disease_and_stroke_statistics–2010 update: a_report_from_the_American_Heart_Association_Statistics_Committee and_Stroke_Statistics_Subcommittee,” Circulation, 2009.
[3] National_Institute_of_Neurolgical_Disorders_and_Stroke, “Interagency_head_injury_task_force_report,” National_Institute_of_Neurolgical_Disorders_and_Stroke, 1989.
[4] CE_Levy, et_al, “Functional_MRI_evidence_of_cortical_reorganization_in_upper-limb_stroke_hemiplegia treated_with_constraint-induced_movement_therapy,” Am_J_Phys_Med_Rehabil, 2001.
[5] KJ_Wineski et_al., “Unique_cortical_physiology_associated_with_ipsilateral_hand_movements and_neuroprosthetic_implications,” Stroke, 2009.
[6] DJ_McFarland, et_al, “Spatial_filter_selection_for_EEG-based_communication,” Electroencephalography_and_Clinical_Neurophysiology, vol. 103, 1997.
[7] G_Schalk, et_al, “BCI2000:_a_general-purpose_brain-computer_interface_(BCI)_system,” IEEE_Transactions_on_Biomedical_Engineering, 2004.
[8] DJ_McFarland, et_al, “Electroencephalographic_(EEG)_control_of_three-dimensional_movement,” J._Neural_Eng., vol. 7, 2010.
[8] D_Broetz, et_al, “Combination_of_brain-computer_interface_training_and_goal_directed_physical_therapy in_chronic_stroke: A_case_report,” J_Neurorehab_and_Neural_Repair, 2010.
[9] HI_Krebs, et_al., “A_Paradigm-Shift:_Rehabilitation_Robotics,” IEEE_Engineering_in_Medicine_and_Biology, vol. 7, 2008.
[10] A_Lo, et_al. “Robot-assisted_therapy_for_long-term_upper_limb_impairment_after_stroke,” The_New_England_Journal_of_Medicine, 2010.

 

avatar

, , , , , ,

5 Responses to IpsiHand: Direct Recoupling of Intention and Movement (Washington University in St. Louis)

  1. avatar
    Robb Greathouse April 29, 2011 at 7:20 pm #

    A friend has a brother with Lou Gehrig’s disease. We have been looking at robotic assists to help him feed himself.

    This looks like it could be a solution. How can we join this project or download it to be able to build a devise for him?

  2. avatar
    Sam Fok May 2, 2011 at 1:33 pm #

    Hi Robb, our device is still in the prototype phase and not yet consumer-ready, but we will surely announce when it is at that stage. A similar device for restoring hand control is the Broadened Horizons PowerGrip found here:
    http://www.broadenedhorizons.com/powergrip.htm
    This device uses muscle EMG signals rather than direct, brain EEG signals, but could suit the goal of restoring hand control.

    We only began working in earnest on the project since late 2010 and so hope to continue the rapid pace of development moving forward into the summer. This is a critical transition point for the project as about half of the original team is graduating and will be replaced by the about 20 newly recruited students.

    We’re certainly open to having outside assistance on the project if possible. For now, those interested should contact me at sbf3 [at] wustl.edu .

  3. avatar
    İbrahim Erkutlu June 22, 2011 at 5:35 pm #

    Hi Sam,

    I think there is a important deficiency in your experiment. As general rule, the left brain hemisphere controls the right side of the body such as arm, hand and legs etc. This is the main rule of the neuroscience because of the crossing of the corticospinal fibers at the brain stem. Therefore , your explanation about BCI controlled arm or hand control experiment does not support this universal reality… In addition to this, you had emphasized the ipsilateral side term repeatedly in your experiment . So how do you explain this confusion or conflict?

    Dr.İbrahim Erkutlu
    Neurosurgeon

  4. avatar
    Eric Leuthardt June 24, 2011 at 12:21 pm #

    As team mentor, maybe I can speak to this. I am neurosurgeon myself and a neuroscientist who studies motor physiology. Basically, it is true that the contralateral primary motor cortex is what is responsible for executing motor commands, and, when injured, people can not move that limb. Motor planning, however, is bilaterally represented in the frontal premotor cortex. Thus when one side is injured, the motor plan is still present, but unable to be executed. It is this unaffected area, ipsilateral to the affected limb (i.e. the uninjured hemisphere), that we are looking to strengthen links to. This ipsilateral activation has been shown to be associated with functional recovery in stroke. Below is an excerpt from the review with references.

    I hope this is helpful.

    Sincerely
    Eric Leuthardt

    Until recently, definitive electrophysiologic studies in humans to parse out the manner and extent to which ipsilateral cortex physiologically encodes hand movements have been limited(Shibasaki, 1975; Tarkka, 1990; Urbano, 1998). ECoG studies have helped to elucidate this phenomenon and extended BCIs capabilities in turn. Wisneski et al. utilized ECoG to more definitively define this physiology in six motor intact patients undergoing invasive monitoring. Electrocorticographic signals were recorded while the subjects engaged in specific ipsilateral or contralateral hand motor tasks. Ipsilateral hand movements were associated with electrophysiological changes that occurred in lower frequency spectra (average 37.5Hz), at distinct anatomic locations (most notably in premotor cortex), and earlier (by 160 ms) than changes associated with contralateral hand movements. Given that these cortical changes occurred earlier and were localized preferentially in premotor cortex compared to those associated with contralateral movements, the authors postulated that ipsilateral cortex is more associated with motor planning than its execution. Additionally, these changes were quite distinct from those changes associated with contralateral motor movements which were more dominantly associated with higher gamma rhythms (average 106.9 HZ).(Wisneski, 2008) In more recent works, the ipsilateral cortical signals associated with joystick movements represent specific motor kinematics (i.e. the direction of the joystick movement) (Ganguly, 2009; Sharma, 2009) Taken together, in normal motor intact human subjects, there appears to be cortical activity ipsilateral to the hand and arm movement that is distinct from activity associated with contralateral movements, associated with planning rather than execution, and encodes specific information about the motor movement.
    In the setting of stroke, premotor cortex appears to play a role in patients with poor functional recovery. Functional imaging has shown these severely affected patients to have increased activity in the premotor regions of their unaffected hemispheres (Weiller, 1992; Seitz, 1998). Incomplete recovery and its association with heightened ipsilateral activation may reflect the up-regulation of motor planning with an inability to execute or actuate the selected motor choice. In this situation, a BCI may provide a unique opportunity to aid in actuating the nascent premotor commands. By detecting the brain signals associated with these motor choices, the BCI may then convert these signals into machine commands that could control a robotic assist device that would allow for improved hand function (i.e., a robotic glove that opens and closes the hand). The BCI would allow the ipsilateral premotor cortex to bypass the physiological bottleneck determined by the small and variable percentage of uncrossed motor fibers. This new methodology would allow for restoration of function in chronically and severely affected subjects for whom methods of rehabilitation have not accomplished a sufficiently functional recovery. The earliest demonstration of ipsilateral derived control was published by Wisneski et al. 2008. Using electrocorticography, the group demonstrated that control comparable to contralateral derived control could be achieved by using the anatomic sites or the lower frequency amplitude changes distinctive to ipsilateral movements.(Wisneski, 2008)

    Ganguly, K., L. Secundo, G. Ranade, A. Orsborn, E. F. Chang, D. F. Dimitrov, J. D. Wallis, N. M. Barbaro, R. T. Knight and J. M. Carmena (2009). “Cortical representation of ipsilateral arm movements in monkey and man.” J Neurosci 29(41): 12948-56.

    Seitz, R. J., P. Hoflich, F. Binkofski, L. Tellmann, H. Herzog and H. J. Freund (1998). “Role of the premotor cortex in recovery from middle cerebral artery infarction.” Arch Neurol 55(8): 1081-8.

    Sharma, M., C. Gaona, J. Roland, N. Anderson, Z. Freudenberg and E. C. Leuthardt (2009). “Ipsilateral directional encoding of joystick movements in human cortex.” Conf Proc IEEE Eng Med Biol Soc 2009: 5502-5.

    Shibasaki, H. and M. Kato (1975). “Movement-associated cortical potentials with unilateral and bilateral simultaneous hand movement.” J Neurol 208(3): 191-9.

    Tarkka, I. M. and M. Hallett (1990). “Cortical topography of premotor and motor potentials preceding self-paced, voluntary movement of dominant and non-dominant hands.” Electroencephalogr Clin Neurophysiol 75(2): 36-43.

    Urbano, A., C. Babiloni, P. Onorati, F. Carducci, A. Ambrosini, L. Fattorini and F. Babiloni (1998). “Responses of human primary sensorimotor and supplementary motor areas to internally triggered unilateral and simultaneous bilateral one-digit movements. A high-resolution EEG study.” Eur J Neurosci 10(2): 765-70.

    Weiller, C., F. Chollet, K. J. Friston, R. J. Wise and R. S. Frackowiak (1992). “Functional reorganization of the brain in recovery from striatocapsular infarction in man.” Ann Neurol 31(5): 463-72.

    Wisneski, K. J., N. Anderson, G. Schalk, M. Smyth, D. Moran and E. C. Leuthardt (2008). “Unique cortical physiology associated with ipsilateral hand movements and neuroprosthetic implications.” Stroke 39(12): 3351-9. (abstract)

  5. avatar
    suleyman tosun beng(hons) December 21, 2011 at 8:07 am #

    hi sam as an engineer I am interested in these products and I wanted to now if i can buy these products from you.or should i contact individual manufacturer,?

Leave a Reply