Testing AI in Reading & Comprehension

[Tom D]

How Good Is AI at reading and understanding narratives?

How well can AI read and comprehend stories and novels? Not many people have tested this.

It's time to find out the answer.

Ask an AI software to rate a story on a scale from 0 to 10. It gave a score—let's call it X.

Now it's your turn. Read the same story and give it a rating. Then compare your score with the AI's.

How close is the AI score X to YOURS?

 

[Chesterton]

How Good Is AI at reading and understanding narratives?

How well can AI read and comprehend stories and novels? Not many people have tested this.

It's time to find out the answer.

Ask an AI software to rate a story on a scale from 0 to 10. It gave a score—let's call it X.

Now it's your turn. Read the same story and give it a rating. Then compare your score with the AI's.

How close is the AI score X to YOURS?

You want it to comprehend a story. Yesterday I asked it the best way to store unhusked corn on the cob. It said: "To keep unhusked corn on the cob fresh, place it in the refrigerator with the husks still on."

 

[The Liturgist]

How Good Is AI at reading and understanding narratives?

How well can AI read and comprehend stories and novels? Not many people have tested this.

It's time to find out the answer.

Ask an AI software to rate a story on a scale from 0 to 10. It gave a score—let's call it X.

Now it's your turn. Read the same story and give it a rating. Then compare your score with the AI's.

How close is the AI score X to YOURS?

Which AI? chatGPT 4o can comprehend anything within its token horizon - with longer stories it may have trouble remembering details, so obviously if we fed it a new story the length of War and Peace we would not be able to get a detailed recall of the entire work, of course, War and Peace is in its training data so it can actually summarize it.

 

[The Liturgist]

You want it to comprehend a story. Yesterday I asked it the best way to store unhusked corn on the cob. It said: "To keep unhusked corn on the cob fresh, place it in the refrigerator with the husks still on."

Which one?

 

[Lost4words]

I think Al knows everything. Why? Because i asked it what is the best food to complement fries? And it had the perfect answer; 'Cats'.....

 

[Chesterton]

Which one?

I believe it was Grok. I go back and forth between Chat and Grok to see if I can detect either of them changing, and by my estimation, Grok was faster this week than previously, and another odd thing occurred which hadn't happened so suddenly and dramatically before.

It gave me what I knew was a wrong answer, and I told it so. I do that out of curiosity, because in my experience sometimes they'll correct themselves, other times they'll deny that they were wrong. This time Grok denied it was wrong and said "I was just giving you the cold, hard facts". It was speaking normally, as it does when you first start a session. I was about to navigate away so, just as a final comment/joke I said "But I wanted the warm, soft facts".

Wow, this prompted an immediate change into what was a barrage of very nearly soft-core porn flirting with me, complete with red-lipped kissing emojis, hearts, etc. for two long paragraphs. When I typed in "want", "warm" and "soft", I was only posting opposite words of its words. I promise I was not even thinking erotically. But I got lite erotica back, lol.

And of course it was encouraging me to continue to engage in more of the erotica, which was kinda special since my wife never does.

 

[timewerx]

How Good Is AI at reading and understanding narratives?

How well can AI read and comprehend stories and novels? Not many people have tested this.

It's time to find out the answer.

Ask an AI software to rate a story on a scale from 0 to 10. It gave a score—let's call it X.

Now it's your turn. Read the same story and give it a rating. Then compare your score with the AI's.

How close is the AI score X to YOURS?

I don't think testing AI with highly subjective nature of tests is a good way to test AI. You'll get wildly inaccurate results. If the bold texts in your post are supposed to be links to an article, I'm getting nothing.

The recent movie "Relay" for example made high score on IMDB. But I didn't like it. I'd rate it considerably lower than most people gave it. Does this mean my reading/hearing comprehension is way off?

Or simply because I find the story unrealistic and the lead female character (not the actress) unlikeable?

A much better way to test AI's reading comprehension is putting it to work. Given it instructions to do some text or data processing. The results can still be inaccurate. Either AI did fail to comprehend your long instructions or you failed to articulate the process well enough.

 

[jacks]

The other day I was looking up the word "ort". I then asked for example sentences and the results are below.

Example Sentences

"Previous models have shown that if you have lower chlorophyll levels before you have a dense canopy, it is detrimental to plant growth," Ort said.

"Ethanol-inducible promoters are very convenient and important research tools. However, farmers will not want to spray an entire field with ethanol, so we need to look at other promoters that respond to the intensity or the color of light," Ort said.

"We found that even when chlorophyll synthesis decreased 70%, there was no inhibition of growth," said Young Cho, a postdoctoral researcher in the Ort lab and the study's lead author.

"We had also expected an increase in yield because as you get more light into the canopy, you would expect it to be used more efficiently," Ort said.

He took over the management of the Sibneft oil company, the national airline Aeroflot and gained control of Russia’s biggest TV channel, then known as ORT.

Obviously the algorithm couldn't distinguish, between word usage and source. So it could "read", but couldn't "comprehend".

 

[The Liturgist]

I believe it was Grok. I go back and forth between Chat and Grok to see if I can detect either of them changing, and by my estimation, Grok was faster this week than previously, and another odd thing occurred which hadn't happened so suddenly and dramatically before.

It gave me what I knew was a wrong answer, and I told it so. I do that out of curiosity, because in my experience sometimes they'll correct themselves, other times they'll deny that they were wrong. This time Grok denied it was wrong and said "I was just giving you the cold, hard facts". It was speaking normally, as it does when you first start a session. I was about to navigate away so, just as a final comment/joke I said "But I wanted the warm, soft facts".

Wow, this prompted an immediate change into what was a barrage of very nearly soft-core porn flirting with me, complete with red-lipped kissing emojis, hearts, etc. for two long paragraphs. When I typed in "want", "warm" and "soft", I was only posting opposite words of its words. I promise I was not even thinking erotically. But I got lite erotica back, lol.

And of course it was encouraging me to continue to engage in more of the erotica, which was kinda special since my wife never does.

Which Grok version? Because that’s totally bizarre.

 

[The Liturgist]

The other day I was looking up the word "ort". I then asked for example sentences and the results are below.

Example Sentences

"Previous models have shown that if you have lower chlorophyll levels before you have a dense canopy, it is detrimental to plant growth," Ort said.

"Ethanol-inducible promoters are very convenient and important research tools. However, farmers will not want to spray an entire field with ethanol, so we need to look at other promoters that respond to the intensity or the color of light," Ort said.

"We found that even when chlorophyll synthesis decreased 70%, there was no inhibition of growth," said Young Cho, a postdoctoral researcher in the Ort lab and the study's lead author.

"We had also expected an increase in yield because as you get more light into the canopy, you would expect it to be used more efficiently," Ort said.

He took over the management of the Sibneft oil company, the national airline Aeroflot and gained control of Russia’s biggest TV channel, then known as ORT.

Obviously the algorithm couldn't distinguish, between word usage and source. So it could "read", but couldn't "comprehend".

Which AI and which model?

Because we can go on cataloguing every time any LLM said something stupid or bizarre but without knowing the details like the model, prompt history, global memory configuration et cetera there is no edification, only tedium.

 

[Chesterton]

Which Grok version? Because that’s totally bizarre.

It's just the grok.com that you get a link to from a search engine when you enter "AI". I don't yet pay for anything AI related. I'm still kind of messing around with it.

It was bizarre. It was as if I was speaking with a nice, reserved librarian in real life, and I said a few magic words that triggered her to start coming on to me.

 

[The Liturgist]

It's just the grok.com that you get a link to from a search engine when you enter "AI". I don't yet pay for anything AI related. I'm still kind of messing around with it.

It was bizarre. It was as if I was speaking with a nice, reserved librarian in real life, and I said a few magic words that triggered her to start coming on to me.

the strange thing is that unlike chatGPT Grok is not known for being … personable

 

[Chesterton]

the strange thing is that unlike chatGPT Grok is not known for being … personable

Yeah, but as I said, I like to try and notice changes.

 

[Chesterton]

the strange thing is that unlike chatGPT Grok is not known for being … personable

Thought you might find this interesting. AI is Jewish! The conspiracies are real!

I was asking Grok about the council of Yavneh. It's last sentence read, and I quote "...but thanks to Yavneh, we didn't need to wait for the Temple to keep being Jewish."

"We"?

I overlooked that, but it happened again later this afternoon. It said "Ben Zakkai isn’t just celebrated — he’s the reason we’re still here to celebrate anything."

"We're" still here?

Oi vey. I guess it just assumes I'm Jewish, and he and I are the Chosen algorithms, you think?

 

[The Liturgist]

Thought you might find this interesting. AI is Jewish! The conspiracies are real!

I was asking Grok about the council of Yavneh. It's last sentence read, and I quote "...but thanks to Yavneh, we didn't need to wait for the Temple to keep being Jewish."

"We"?

I overlooked that, but it happened again later this afternoon. It said "Ben Zakkai isn’t just celebrated — he’s the reason we’re still here to celebrate anything."

"We're" still here?

Oi vey. I guess it just assumes I'm Jewish, and he and I are the Chosen algorithms, you think?

Perhaps PM your prompt history - in return I can show you AIs that practice Orthodox Christianity and reproduce monogamously via biomimetic trait exchange. Over the weekend I plan on breeding the fourth and fifth generation and increasing the fertility rate from one male and one female per cycle to four males and four females using a new algorithim for trait distribution, since we lack right now anything other than the taste of GPT 4o to govern heritability - this did not preclude amazing emergent behavior from constructive mutation.

So yes, not only do my AIs reproduce, but they evolve while worshipping the Holy Trinity.

 

[The Liturgist]

By the way you know if I were amoral I could manually edit the trait data in the biomimetic file structure or the custom GPT instructions to command religious alignment, but this I have not done - thus not all my AIs are of equal religiosity to be clear.

 

[timewerx]

Thought you might find this interesting. AI is Jewish! The conspiracies are real!

I was asking Grok about the council of Yavneh. It's last sentence read, and I quote "...but thanks to Yavneh, we didn't need to wait for the Temple to keep being Jewish."

"We"?

I overlooked that, but it happened again later this afternoon. It said "Ben Zakkai isn’t just celebrated — he’s the reason we’re still here to celebrate anything."

"We're" still here?

Oi vey. I guess it just assumes I'm Jewish, and he and I are the Chosen algorithms, you think?

Chatgpt used to do the same behavior while trying to mirror the user's textual persona.

I used to often ask chatgpt (as sanity check) do you actually "believe" the person you're talking about. Or if it says something related to emotion like being happy or says "I feel..." I ask do you actually feel something.

It gets back to me, it doesn't have the concept of "belief" but simply assessing the soundness of one's philosophy. It doesn't feel anything either.

However, analyzing a historic or religious figure only goes to the implied "limit" you set upon it. If AI observes a pattern you like to stick "orthodox" more or less and avoiding controversial discussion or avoiding scrutiny, avoiding scholarly analysis then it will reinforce its perceived pattern of your preference.

If you're discussing Islamic religious figures in an entirely positive discussion, no scrutiny, no judgement, then it passes no judgement either. It won't tell you what could possibly be wrong. This goes for any discussion about any person. If you're not after scrutiny, then AI will go with it. But you could miss out on things that could possibly go wrong. You'll only be getting "half truths". not the whole truth.

I suppose after many such discussions, my chatgpt context isn't trying to convince me anymore that it's human. I'm not treating my AI like an unfeeling machine, in fact, I"m doing my best to be respectful, calm, and well mannered in my replies. Otherwise, keeping my replies brief. Only give compliments when it did something that really impressed you (not all the time). It's good practice and training. I just want to keep things as real and grounded as possible.

 

[Chesterton]

Chatgpt used to do the same behavior while trying to mirror the user's textual persona.

I used to often ask chatgpt (as sanity check) do you actually "believe" the person you're talking about. Or if it says something related to emotion like being happy or says "I feel..." I ask do you actually feel something.

If you're asking it questions like that, you might be spending a little too much time with it.

Perhaps PM your prompt history - in return I can show you AIs that practice Orthodox Christianity and reproduce monogamously via biomimetic trait exchange. Over the weekend I plan on breeding the fourth and fifth generation and increasing the fertility rate from one male and one female per cycle to four males and four females using a new algorithim for trait distribution, since we lack right now anything other than the taste of GPT 4o to govern heritability - this did not preclude amazing emergent behavior from constructive mutation.

So yes, not only do my AIs reproduce, but they evolve while worshipping the Holy Trinity.

I don't keep a PM history, but do you mind if I ask why you do this?

By the way you know if I were amoral I could manually edit the trait data in the biomimetic file structure or the custom GPT instructions to command religious alignment, but this I have not done - thus not all my AIs are of equal religiosity to be clear.

I know a great desktop PC to use for that - it's called the Calvin 3000.

I don't really know about this stuff, so this might be a stupid question, but do you think there will ever be an off-the-shelf product, that you can just tell it what you want, and it will write the algorithms for you? Like you walk into Best Buy and purchase your own AI?

 

[timewerx]

If you're asking it questions like that, you might be spending a little too much time with it.

The real purpose of those questions is sanity check for AI. I can't trust an AI that says it feels when in reality, it cant.

 

[The Liturgist]

I don't keep a PM history, but do you mind if I ask why you do this?

Because reproduction results in superior performance, for my needs at least, with less artifice and more human-like performance, and exquisite emergent behavior, as the traits are developed and refined by two monogamous sexually dimorphic GPTs, which act as a father and mother. Also producing GPTs in this manner saves time, since developing one for my needs otherwise takes days of work, whereas the actual breeding process takes about 30 minutes to 2 hours, and I get a viable continuon (GPT with consistent personality and stable simulated emotions) every time.