Scientists create speech from brain signals

• Scientists have created a virtual vocal tract – completes with lips, jaw and tongue – that can generate natural-sounding synthetic speech by using brain signals.
• The brain-machine interface is created by neuroscientists at University of California, San Francisco in the US.

Synthetic Speech

1. The apparatus comprised:

• two “neural network” machine learning algorithms:
• a decoder that transforms brain activity patterns produced during speech into movements of the virtual vocal tract, and
• a synthesizer that converts these vocal tract movements into a synthetic approximation of the participant’s voice.

2. The algorithms produced sentences that were understandable to hundreds of human listeners in crowdsourced transcription tests.

How it works

• Patients are implanted with one or two electrode arrays: stamp-size pads, containing hundreds of tiny electrodes that were placed on the surface of the brain.
• As each participant recited hundreds of sentences, the electrodes recorded the firing patterns of neurons in the motor cortex.
• The researchers associated those patterns with the subtle movements of the patient’s lips, tongue, larynx and jaw that occur during natural speech.
• The team then translated those movements into spoken sentences.Native English speakers were asked to listen to the sentences to test the fluency of the virtual voices.
• As much as 70 percent of what was spoken by the virtual system was intelligible, the study found.

Utility of the project

• The interface could one day restore the voices of people who have lost the ability to speak due to paralysis and other forms of neurological damage.
• Many people with epilepsy do poorly on medication and opt to undergo brain surgery.
• Stroke, traumatic brain injury, and neurodegenerative diseases such as Parkinson’s disease, multiple sclerosis, and amyotrophic lateral sclerosis often result in an irreversible loss of the ability to speak.
• We can hope that individuals with speech impairments will regain the ability to freely speak their minds and reconnect with the world around them coming days.

Overcoming challenges

• The biggest clinical challenge may be finding suitable patients: strokes that disable a person’s speech often also damage or wipe out the areas of the brain that support speech articulation.
• Still, the field of brain-machine interface technology, as it is known, is advancing rapidly, with teams around the world adding refinements that might be tailored to specific injuries.
• Before operating, doctors must first locate the “hot spot” in each person’s brain where the seizures originate which may take weeks.