The Institute for Human and Machine Cognition:

Training Wounded Soldiers to See and Walk with Their Tongues

Click here for full print version.

Five years ago, a transformation took place at an old public library building in Central Florida. The dark paneling, old bookshelves, and stained carpeting was replaced by sleek, modern furniture, skylights and whiteboard panels.

The only remodeled building in Florida to be awarded LEED gold level certification now houses researchers with the Institute for Human and Machine Cognition, or IHMC.

IHMC is not a company, but one of two research institutes, along with the Moffitt Cancer and Research Institute, associated with the State University System of Florida.

The institute’s scientists spend their days working on transformative research that changes the way humans use technology to interact with the world.

“One aspect of the work we do focuses on wounded service members,” says John Rogacki, associate director of IHMC and a former researcher with the Air Force, NASA and the University of Florida, “Due to advances in body armor, many of our men and women are coming home alive, but missing limbs or eyesight.”

IHMC works in three areas, mobility, sensory and cognitive assistance, to help injured service members, civilians, perform daily tasks so that they can reintegrate into society.

One application of Mobility Assistance is the development of passive exoskeletons that assist partially paralyzed people with using their limbs. An exoskeleton is a robotic device attached to the outside of the body. Passive means it is unpowered.

The robot doesn’t retrain the limb to move, it retrains the brain to send the right signals to the muscles.

NASA is interested in the use of powered exoskeletons to enable astronauts to maintain muscle and bone mass while in space. In this case, researchers with IHMS’s Blue Sky Program developed the Lunar Electric Rover, a powerful sort of mobile home for astronauts, which allows them to maneuver outside the space station, across all types of terrain, without the exhausting and time-consuming decompression and re-pressurizing necessary with current spacesuits.

Mobility Assistance and Sensory Substitution are used together to help people who have lost feeling in their extremities and cannot perceive the position of their limbs or feel when their limbs make contact with the ground or other obstacles.

Sensory Substitution involves placing a tactile pad on the torso, abdomen or even the tongue, which receives signals from sensors on the paralyzed limb. The person learns to perceive balance, touch and weight loading by feeling the stimulus on the receptor pad.

Just like a blind person learns to read with his fingertips, a paralyzed person can learn to feel her body weight with her tongue.

In fact, the technology can be used with loss of eyesight, caused by macular degeneration or trauma, as in the case many soldiers who have encountered improvised explosive devices. Instead of relying on the lenses in the eyes, data is relayed from a camera attached to glasses and transmitted to the interface on the tongue.

The brain can be trained to interpret data sent to the tongue to the point where a person can navigate obstacles around them and read an eye chart to the 20-60 level.

An example of Cognitive Assistance involves improving instrument displays in aircraft cockpits. In addition to focusing on their mission, pilots have to monitor many gauges and dials, which causes mental fatigue. Researchers at IHMC are developing a more human-centered interface that displays information in a more intuitive manner, allowing pilots to manage data with less mental effort.

Two additional examples of cognitive assistance are in cyber technology.

“The free world is constantly under attack, from everything to teenage kids sending viruses to professional cyber bank robbers to adversaries stealing military secrets,” says Rogacki, “Our country is putting a lot of resources into cyber security.”

In cyber space, events happen quickly; threats over a network come rapidly and with large amounts of data. Scientists are working on a program that highlights information in visual streams, allowing potential cyber attacks to be identified faster.

Companion Technology is another cognitive assistance application designed for Alzheimer’s and dementia patients.

A virtual pet dog can store information, such as family photos, interviews with family members or caregivers, phone numbers and medication schedules. The virtual pet learns faces, voices and routines and can understand the patient’s natural speech patterns. The virtual dog give reminders, initiate contact with caregivers, and even read to the patient.

For more on IHMC, visit