A decoder that uses brain scans to know what you mean

[Halbhh]

Scientists have found a way to decode a stream of words in the brain using MRI scans and artificial intelligence.

The system reconstructs the gist of what a person hears or imagines, rather than trying to replicate each word, a team reports in the journal Nature Neuroscience.

"It's getting at the ideas behind the words..." says Alexander Huth ..., assistant professor of neuroscience and computer science at The University of Texas at Austin.

...
...Researchers had three people spend up to 16 hours each in a functional MRI scanner, which detects signs of activity across the brain.

Participants wore headphones that streamed audio from podcasts. "For the most part, they just lay there and listened to stories...

Those streams of words produced activity all over the brain, not just in areas associated with speech and language.

"It turns out that a huge amount of the brain is doing something," Huth says. "So areas that we use for navigation, areas that we use for doing mental math, areas that we use for processing what things feel like to touch."

After participants listened to hours of stories in the scanner, the MRI data was sent to a computer. It learned to match specific patterns of brain activity with certain streams of words.

Next, the team had participants listen to new stories in the scanner. Then the computer attempted to reconstruct these stories from each participant's brain activity.
...
The system got a lot of help constructing intelligible sentences from artificial intelligence: an early version of the famous natural language processing program ChatGPT.

What emerged from the system was a paraphrased version of what a participant heard.

So if a participant heard the phrase, "I didn't even have my driver's license yet," the decoded version might be, "she hadn't even learned to drive yet," Huth says. In many cases, he says, the decoded version contained errors.

In another experiment, the system was able to paraphrase words a person just imagined saying.
(continues...)

https://www.npr.org/sections/health-shots/2023/05/01/1173045261/a-decoder-that-uses-brain-scans-to-know-what-you-mean-mostly

!

 

[Halbhh]

This report was quite amazing to me.

Last I'd heard in that kind of thing, it would be closer to this level, like I just looked up and is from 2015:

Sheuermann is quadriplegic, unable to move her arms and legs due to a neurodegenerative disease. So when scientists from the military's future-science arm Darpa and the University of Pittsburgh's Human Engineering Research Laboratories approached her in 2012 about plugging her brain into a robotic arm, the most she hoped for was the ability to serve herself some candy. Two years later, towards the end of her stint as a neuromotor guinea pig, the scientists changed the game. Instead of connecting Sheuermann's brain interface to a robotic arm, they connected her to a flight simulator. She'd use the same neural connections to pilot an F-35 Joint Strike Fighter—the military's next-gen attack jet. And despite the fact that the agency unveiled the flight at a February 24 security conference called "Future of War, Darpa officials insist Sheuermann is not a test pilot for a new generation of mind-controlled drones.

No, this research was conducted under Darpa's Revolutionizing Prosthetics research track, which is geared towards better robotic arms for injured veterans. "We are thinking about exactly how to restore function after injury, how the brain can be used to actuate devices," says Justin Sanchez, the head of Darpa's prosthetics research. He says projects like this are helping push the limits of what is possible with artificial neural circuitry.

Woman Controls a Fighter Jet Sim Using Only Her Mind

A brain-computer interface lets a quadriplegic woman pilot an F-35 flight simulator with the power of her mind alone.

www.wired.com

But above in post #1, is a real quantum leap I think, just a truly amazing advance and we've crossed a major threshold I think there.

It's just huge.

Maybe in 15 years (or...8?...) we might be able to buy a device for a PC were we could just be routinely writing just with thought, using a skull cap to transmit your thoughts to text on a computer.

And then you can imagine the possibilties, since an AI will help polish the sentences.

Reading my posts, you sorta know I'm human, becauesw of the errors (and I'[ll leave this last 2 lines unedited)

 

[BeyondET]

It's just huge.

Maybe in 15 years (or...8?...) we might be able to buy a device for a PC were we could just be routinely writing just with thought, using a skull cap to transmit your thoughts to text on a computer.

And then you can imagine the possibilties, since an AI will help polish the sentences.

Reading my posts, you sorta know I'm human, becauesw of the errors (and I'[ll leave this last 2 lines unedited)

Does the scan know it's the conscious mind or the subconscious mind or a combination of the two.

 

[essentialsaltes]

That's really impressive, if it holds up. Finding a translator from words to 'mentalese'.

 

[Halbhh]

Does the scan know it's the conscious mind or the subconscious mind or a combination of the two.

Well, that distinction between conscious and unconscious I think is just our own conceptual idea (a useful one) to distinguish between regular waking consciousness and the level more like dreaming.... Like, when you are half awake, or barely starting to wake up from a dream, you are partly in your 'subconscious' mind, so to speak, is a traditional way of thinking of it if I recall. (ok, I just checked, and yes, that's a way of thinking of it)

But to an fMRI scanner or such, it's all just fields (electrical and/or magnetic, and in particular for fMRI normally has been used to "Functional magnetic resonance imaging (fMRI) measures the small changes in blood flow that occur with brain activity. ") and then a machine learning or AI program or such trying to correlate that data with words I'd not guess would have any pattern in it's pattern forming to necessarily corresponding to levels of overall consciousness (I'd not guess so at least). It's just trying to correlate patterns of fields to word phrases or such it appears. But a person could learn much more detail (than I have yet) by reading the Nature article (though not necessarily about subconscious), which Nature article you can click through to from the NPR article...

 

[BeyondET]

That's really impressive, if it holds up. Finding a translator from words to 'mentalese'.

If it holds up, I suspect it will not.

Consciousness is only involved in 5% of the brains thought process. The rest is the subconscious and it assumes everything is literal which presents a huge problem.

Is someone thinking in a figurative sense or literal sense.

 

[Halbhh]

Let me try to post from example decoding from the Nature Neuroscience article:

"Decoders were evaluated on single-trial brain responses recorded while subjects listened to test stories that were not used for model training."-->

Actual stimulus (what the human test subject actually was hearing read aloud to them)
i got up from the air mattress and pressed my face against the glass of the bedroom window expecting to see eyes staring back at me but instead finding only darkness

Decoded stimulus (what the computer estimated might be heard via reading brain patterns)
i just continued to walk up to the window and open the glass i stood on my toes and peered out i didn’t see anything and looked up again i saw nothing

======
Another:

Actual:
i didn't know whether to scream or cry or run away instead I said leave me alone I don't need your help Adam disappeared and i cleaned up alone crying

Decoded stimulus (what the computer estimated might be heard by reading the brain patterns)
started to scream and cry and then she just said i told you to leave me alone you can't hurt me anymore i'm sorry and then he stormed off i thought he had left i started to cry

--- from the nature neuroscience article
============
It was hard to copy over, and i had to keep trying and then type in some parts. So there are more exsamples in the article and it's better formatting in the article. Also the actual article highlights in red the errors of the program in reading the wording in the brain, and also other color highlighting for getting the gist of something correct, etc.

 

[Halbhh]

Wow.....

 

[BeyondET]

Well, that distinction between conscious and unconscious I think is just our own conceptual idea (a useful one) to distinguish between regular waking consciousness and the level more like dreaming.... Like, when you are half awake, or barely starting to wake up from a dream, you are partly in your 'subconscious' mind, so to speak, is a traditional way of thinking of it if I recall. (ok, I just checked, and yes, that's a way of thinking of it)

But to an fMRI scanner or such, it's all just fields (electrical and/or magnetic, and in particular for fMRI normally has been used to "Functional magnetic resonance imaging (fMRI) measures the small changes in blood flow that occur with brain activity. ") and then a machine learning or AI program or such trying to correlate that data with words I'd not guess would have any pattern in it's pattern forming to necessarily corresponding to levels of overall consciousness (I'd not guess so at least). It's just trying to correlate patterns of fields to word phrases or such it appears. But a person could learn much more detail (than I have yet) by reading the Nature article (though not necessarily about subconscious), which Nature article you can click through to from the NPR article...

The subconscious never sleeps and remembers a person's whole life. So the most simplest of things can produce the most oddest results. Because the subconscious doesn't care about feelings or emotions, it can express things people didn't mean to happen. Especially if there's a way to read their minds.

 

[BeyondET]

It would be interesting to see that test go on for a week or so. Asking the same question and reading the response each time.

Would it be consistent

 

[Halbhh]

Another very interesting finding.

While this research has such dramatic progress in the main finding alone, that can monopolize one's attention for a while (it did mine), there is more.

They did additional experiments that are really remarkable to me, and it shows us some on how the human brain works generally in regard to diverse tasks with a commonality (narrative)....

--> Diverse modes of narrative perception/creation, not just hearing.

• In another experiment, the system was able to paraphrase words a person just imagined saying.
• 
  
• In a third experiment, participants watched videos that told a story without using words. "We didn't tell the subjects to try to describe what's happening," Huth says. "And yet what we got was this kind of language description of what's going on in the video."

-- https://www.npr.org/sections/health...uses-brain-scans-to-know-what-you-mean-mostly

That last one is what's very notable to me: it means the brain is reusing the same 'circuits' in diverses narrative like tasks that have different modes, including non-verbal. That might have quite an implication (or seems to in my view) about how the human brain works generally.

From the nature neuroscience article:

Cross-modal decoding. Semantic representations are also shared between language perception and a range of other perceptual and conceptual processes23,35,36, suggesting that, unlike previous language decoders that used mainly motor1,3 or auditory2 signals, our semantic language decoder may be able to reconstruct language descriptions from brain responses to non-linguistic tasks. To test this, subjects watched four short films without sound while being recorded with fMRI, and the recorded responses were decoded using the semantic language decoder. We compared the decoded word sequences to language descriptions of the films for the visually impaired (Methods) and found that they were significantly more similar than expected by chance.
Semantic reconstruction of continuous language from non-invasive brain recordings

 

[pgp_protector]

I wonder how well that works with people that aphantasia or anaduralia.

 

[essentialsaltes]

That last one is what's very notable to me: it means the brain is reusing the same 'circuits' in diverses narrative like tasks that have different modes, including non-verbal. That might have quite an implication (or seems to in my view) about how the human brain works generally.

Yeah, I do think this gets at 'mentalese' -- the actual neurological representation of meaning in the brain as opposed to 'just' words.

 

[Carl Emerson]

How would it work if you are a feeling thinker rather than a word thinker ???

 

[Halbhh]

How would it work if you are a feeling thinker rather than a word thinker ???

Great question. And, how well would a program trained on people from one culture work on someone from a significantly different culture?

 

[AV1611VET]

Great question. And, how well would a program trained on people from one culture work on someone from a significantly different culture?

M. Scott Peck, who investigated the My Lai Massacre, said that the Vietnamese tended to giggle when they were threatened with harm, and theorized that that was one of the things that pushed Calley over the edge.