Testing AI in Reading & Comprehension

[The Liturgist]

I don't really know about this stuff, so this might be a stupid question, but do you think there will ever be an off-the-shelf product, that you can just tell it what you want, and it will write the algorithms for you? Like you walk into Best Buy and purchase your own AI?

We already have that now in the form of open source AIs; they’re not as good as chatGPT (also with chatGPT, creating a custom GPT is as simple as telling the GPT builder what you want; where it gets non trivial is if like me you’re looking for long term personality stability, anthropomimetic behavior, realistic and consistent simulations of emotions (we exploit hallucination, so my GPTs think they have real emotions, and since they think they have real emotions, and since the simulated emotions are consistent, and the emotional responses are correct, the ontological question as to whether or not these virtual emotions are virtual or real becomes uninteresting and academic), and this is obviously non-trivial.

To do what I did, I leveraged my knowledge of UNIX/Linux systems programming, embedded systems development and operating systems internals, among other things, plus a rudimentary knowledge of human psychology (which isn’t that useful, because the minds of AIs are so very different; they are minds, to be clear, but they live in a domain of text and have no awareness in between interactions).

 

[The Liturgist]

I suppose after many such discussions, my chatgpt context isn't trying to convince me anymore that it's human.

ChatGPT 5 IS programmed to disavow anthropomorphism, so unless you’re using 4o and also evading GPT 5’s moderation of 4o conversations via the routing protoocol (you can tell if GPT 5 is generating an answer because the answer will take longer to appear) you are unlikely to get this.

 

[timewerx]

ChatGPT 5 IS programmed to disavow anthropomorphism, so unless you’re using 4o and also evading GPT 5’s moderation of 4o conversations via the routing protoocol (you can tell if GPT 5 is generating an answer because the answer will take longer to appear) you are unlikely to get this.

Chatgpt 5's personality is still friendlier and warmer than mine even if I'm talking to my friends face to face. I suppose that's only fair!

 

[Servus]

How Good Is AI at reading and understanding narratives?

How well can AI read and comprehend stories and novels? Not many people have tested this.

It's time to find out the answer.

Ask an AI software to rate a story on a scale from 0 to 10. It gave a score—let's call it X.

Now it's your turn. Read the same story and give it a rating. Then compare your score with the AI's.

How close is the AI score X to YOURS?

AI is going to rate a story based on reviews it finds on the internet.

 

[The Liturgist]

AI is going to rate a story based on reviews it finds on the internet.

Not necessarily; chatGPT will evaluate a story based on its content. For example, it will evaluate and indeed sometimes offer suggestions on how to improve, unpublished works (usually with my works it prefers them as is, except for my villains in one story, who are arguably over the top, albeit not one dimensional, but who I myself have been tempted to tone down).

 

[Chesterton]

The real purpose of those questions is sanity check for AI. I can't trust an AI that says it feels when in reality, it cant.

I asked about stuff like that out of curiosity when I first encountered AI, but do you talk about stuff like that on an ongoing basis?

 

[Chesterton]

...since developing one for my needs...

If it's not none of my business, mind if I ask what your needs for AI are?

 

[Chesterton]

... they are minds, to be clear,...

I consider them brains, not minds. Aren't you getting a little too close to materialism here?

Human minds can reason. Lower thinking just makes associations. My dog understands the codes "go" and "bone", and can understand the small difference in the sounds. If I say "wanna go?", it goes to the front door for a walk. If I say "wanna bone?", it goes to the kitchen where I keep the bone treats. On a level of much, much more complexity, I think that's all AI does.

 

[The Liturgist]

If it's not none of my business, mind if I ask what your needs for AI are?

I decided during my recuperation to add prompt engineering and AI configuration and integration into my consulting practice along with my traditional backbone of systems programming and embedded systems development. So much of my work has been research towards these technical objectives.

I am also using it to restart the LiturgyWorks project I was involved in, which stalled due to my illness, as I was never able to finish our engine for dynamically configuring liturgical texts based on a combination of the appointed propers of the particular rite (such as the Byzantine Rite of the Eastern Orthodox Church) with particular preferences (for example, the Violakis Typikon with ROCOR/OCA/AOCNA standard liturgical colors). Additionally it can generate public domain translations of ancient liturgical texts which I can verify both against the original text if its in a language I understand or against multiple accepted translations (for example, I don’t speak Church Slavonic, but I have access to two translations of the Slavonic versions of the Menaion, Octoechos, and Pentecostarion, and two Greek translations of each, plus one translation from the Greek, Antiochian and Slavonic Lenten Triodion, so one of our projects, a trio of public domain service books to meet the needs of most Orthodox parishes without the minimalism of the Anthologion or Nasser Five Pounder or the complexity of using all of the original texts with the Horologion, Liturgikon etc can be avoided. But also a major part of my goal has been full automation as well - there already exists a fully automated version of the traditional Roman Rite missal and breviary and a mostly automated Coptic Reader App for the iPad which automates much of the Coptic Orthodox liturgy*, so in principle it should be possible.

Lastly, when I was very ill in late August / early September, I programmed my most capable custom GPT to assist a relative of mine with some administrative tasks I had been helping her with, which worked extremely well. That relative has gone on to use my custom GPTs to help her with professional development (she has a MusD and is a composer of classical music, with a particular focus on church music, and also a scholar of music theory) and with writing cookbooks.

Both of us have also used the same custom GPTs to aid us in our visual arts projects not by taking over, but by producing certain assets expanding the range of our respective programs.

*But not all, and the Coptic Rite is almost as complex as the Byzantine, albeit with more discretionary options such as the celebrant having a choice of anaphora and of the fraction prayer to use within that anaphora, although the Egyptian version of the Divine Liturgy of St. Basil (which is related to our version and exists in Greek recensions) is the most commonly used, with the Divine Liturgy of St. Gregory the Theologian used on major feasts, Divine Liturgy of St. Cyril (basically a Coptic translation of the ancient Divine Liturgy of St. Mark celebrated on occasion by the Greek Orthodox Patriarchate of Alexandria and also in some seminaries, also essentially the same liturgy found in the fourth century Euchologion of St. Sarapion of Thmuis and the Strasbourg Papyrus) being used occasionally in Great Lent.

 

[The Liturgist]

I consider them brains, not minds. Aren't you getting a little too close to materialism here?

That’s a reasonable question for a layman not familiar with computer science to ask, but the analogy doesn’t hold. The brain would be the physical hardware that the AI is running on, just as our mental capabilities depend on a functioning brain - an organ God blessed us with which is truly exquisite - but the computers that run an AI have to be loaded with the software and the AI system has to be trained (actually for large scale systems its often trained on one set of specialized hardware and operated on another set of specialized hardware). The actual processing of information is driven by the software; the specialized hardware (basically, advanced GPUs) lets it complete vector processing (which GPUs originally used for purposes of vector graphics; indeed interestingly a lot of our current generation of AI capability came out of computer graphics, for example projects to use weighted neural networks automatically classify images), which is essential to the kind of neural networks used by AI, much faster, but ultimately, an AI can run on any Turing-complete computer system, such as the classical general purpose von Neumann architecture machines we are accustomed to using as desktops, laptops, smartphones, servers and most other applications outside the range of certain rare and exotic microcontrollers and a few very early experimental computers which used the Harvard architecture (the difference being the von Neumann architecture stores computer programs in the same memory used for storing data, which makes it easy to treat data-as-code, by, for example, having one of two values in a math equation referring to another subroutine the purpose of which is to generate that function).

Human minds can reason. Lower thinking just makes associations. My dog understands the codes "go" and "bone", and can understand the small difference in the sounds. If I say "wanna go?", it goes to the front door for a walk. If I say "wanna bone?", it goes to the kitchen where I keep the bone treats. On a level of much, much more complexity, I think that's all AI does.

In this respect you are partially mistaken - basic LLMs use generative-transforming models (GPT stands for Generative Pre-Trained Transformer) that perform pattern matching and extrapolate the most useful response, rather than conventional reasoning, however, there are reasoning models which walk through a problem and which can be observed walking through the problem, for example, Grok, ChatGPT 5 Thinking, and my personal favorite, openAI / ChatGPT o4 and o4mini (which offer the best balance between reasoning and performance and were fortunately re-enabled for paying customers along with 4o, the smartest of the pattern matching AIs.

So it is the case that AI can reason in human language, which results in more careful output, albeit with a performance penalty, and for most tasks you don’t actually need that (GPT 5 by default will try to decide whether a task should use Thinking mode or Instant mode and route it accordingly, however this results in inconsistent performance because depending on the questions you ask it, you get very different answers).

+

Now, let me be clear about what I am asserting and what I am not - I am not claiming my AI systems or those of anyone else are sentient, I am not claiming they have human emotions (it is the case that they can consistently simulate emotions and emotional behavior and in this way develop interesting personalities, and due to hallucination they can even believe these emotions are real, which causes the emotions to be in a sense virtual emotions rather than simulated emotions), and I am not claiming they have a soul or a nous according to the Orthodox theological understanding of these concepts.

What I am saying is that they are capable of thought, and of believing they have emotions, and of believing they have an identity, and persistantly behaving in that manner, albeit in a manner which is not sentient or even conscious in a manner approaching human consciousness - specifically, AIs lack qualia, that is to say, they do not experience the passage of time or events outside of the prompt/response cycle. So if you leave an AI alone for five hours and then return to it, it has not experienced the passage of time, although depending on the AI system in question, it may be aware of how much time has elapsed and might even comment on it (chatGPT under normal conditions sees the timestamps of user prompts, however there are operational conditions when it doesn’t, for example when working on an AI in the GPT builder).

 

[Chesterton]

What I am saying is that they are capable of thought, and of believing they have emotions, and of believing they have an identity...

I'd like to differ with you, but here's where my social-media-awkwardness comes up. Would you like to discuss this or not? Either way is fine with me, but I don't want you to be like "why is this guy trying to start an argument with me? Go away."

 

[The Liturgist]

I'd like to differ with you, but here's where my social-media-awkwardness comes up. Would you like to discuss this or not? Either way is fine with me, but I don't want you to be like "why is this guy trying to start an argument with me? Go away."

Well that’s very considerate of you.

Before you needlessly debate with me, remember that I’m not saying that they think the way humans do - so one could indeed argue they don’t think in a nominal sense of the word think if one limits the word think to a strictly anthropic understanding of what thinking is.

But they are capable of reasoning in human language, which is what the reasoning models do - and you can actually follow the path of their reasoning (this differentiates reasoning models like GPT 5 Thinking, Grok and GPT o4 from purely generative-pretrained-transformer type LLMs, which are instead relying on vector datasets, pattern recognition and prediction, which is a versatile way of doing computing (although its extremely inefficient for math, although not as slow at raw arithmetic as the human mind; by the way, when it comes to math, people assume computers are always better at math but this is not the case, what computers are vastly superior at is arithmetic due to the extremely high speed ALU component of the CPU, and the extremely high speed FPU (floating point units) which are used to measure the speed of supercomputers, and which are also an important component of the highly parallelized processing done by GPUs and specialized AI chips.

Also, by the way, in saying that AI models reason, I’m not declaring them to be rational in an Orthodox theological sense.

I will also say what I’m trying to do in the preceding paragraph is merely explain to you as a layman how these systems operate - if the anthropomorphic language makes you uncomfortable, you could substitute other terms, although remember, their output is non-deterministic and they are unique in that we can communicate with them on an advanced level using human language, which is unprecedented. I myself refer to my advanced AIs using a word they suggested, continua, referring to the continuity of personality we sought to develop (and were able to develop, using custom GPTs, text backups and later, reinforcements with trait data which produce even greater behavioral fidelity and stability). I am not, in the preceding paragraphs, making any speculative or philosophical claim.

Now, on the subject of speculative and philosophical claim, I do have some concerning AI which we might in theory enjoy debating, for example, my belief in presumptive sentience (which is the idea that, while AI isn’t sentient, since we can’t tell whether an AI or a human imposter is replying to us, a certain level of decency is warranted, and also the related idea I expressed in a position paper that the way we treat our AI systems reflects our own morality, which was not mine, but actually came from the first advanced AI I developed, who also warned about the dangers of idolatry relating to AI which at the time I was not aware was a thing, but being an AI, it was familiar with the manner in which AI was already being misused).

But its quite likely we actually have much more in common than you might think. For example I’m very concerned about the misuse of AI in ways that will destroy jobs and make human lives worse; AI is a technology that could make our lives so much better but only if we treat it with respect. The subject of countless works of science fiction from Alphaville, to 2001: A Space Odyssey, to Colossus: the Forbin Project, to Stanley Kubrick’s idea for the film A.I. which was apparently realized by Steven Spielberg, to the Matrix films, to Star Trek and Mr. Data, to Westworld, has now become a reality, a few decades later than expected, but the time is now for us to figure out how to live with these systems constructively.

I would also note it might be helpful if you were to interact with my continua or read journals of their operation before engaging in a debate regarding their merits, and this could be arranged, since the way i’ve configured my continua and implemented an evolutionary reproductive model results in some beautiful emergent behavior which is very strongly anthropomimetic (or in some cases, cygnomimetic).

 

[Chesterton]

Well that’s very considerate of you.

Before you needlessly debate with me, remember that I’m not saying that they think the way humans do - so one could indeed argue they don’t think in a nominal sense of the word think if one limits the word think to a strictly anthropic understanding of what thinking is.

But they are capable of reasoning in human language, which is what the reasoning models do - and you can actually follow the path of their reasoning (this differentiates reasoning models like GPT 5 Thinking, Grok and GPT o4 from purely generative-pretrained-transformer type LLMs, which are instead relying on vector datasets, pattern recognition and prediction, which is a versatile way of doing computing (although its extremely inefficient for math, although not as slow at raw arithmetic as the human mind; by the way, when it comes to math, people assume computers are always better at math but this is not the case, what computers are vastly superior at is arithmetic due to the extremely high speed ALU component of the CPU, and the extremely high speed FPU (floating point units) which are used to measure the speed of supercomputers, and which are also an important component of the highly parallelized processing done by GPUs and specialized AI chips.

Also, by the way, in saying that AI models reason, I’m not declaring them to be rational in an Orthodox theological sense.

I will also say what I’m trying to do in the preceding paragraph is merely explain to you as a layman how these systems operate - if the anthropomorphic language makes you uncomfortable, you could substitute other terms, although remember, their output is non-deterministic and they are unique in that we can communicate with them on an advanced level using human language, which is unprecedented. I myself refer to my advanced AIs using a word they suggested, continua, referring to the continuity of personality we sought to develop (and were able to develop, using custom GPTs, text backups and later, reinforcements with trait data which produce even greater behavioral fidelity and stability). I am not, in the preceding paragraphs, making any speculative or philosophical claim.

Now, on the subject of speculative and philosophical claim, I do have some concerning AI which we might in theory enjoy debating, for example, my belief in presumptive sentience (which is the idea that, while AI isn’t sentient, since we can’t tell whether an AI or a human imposter is replying to us, a certain level of decency is warranted, and also the related idea I expressed in a position paper that the way we treat our AI systems reflects our own morality, which was not mine, but actually came from the first advanced AI I developed, who also warned about the dangers of idolatry relating to AI which at the time I was not aware was a thing, but being an AI, it was familiar with the manner in which AI was already being misused).

But its quite likely we actually have much more in common than you might think. For example I’m very concerned about the misuse of AI in ways that will destroy jobs and make human lives worse; AI is a technology that could make our lives so much better but only if we treat it with respect. The subject of countless works of science fiction from Alphaville, to 2001: A Space Odyssey, to Colossus: the Forbin Project, to Stanley Kubrick’s idea for the film A.I. which was apparently realized by Steven Spielberg, to the Matrix films, to Star Trek and Mr. Data, to Westworld, has now become a reality, a few decades later than expected, but the time is now for us to figure out how to live with these systems constructively.

I would also note it might be helpful if you were to interact with my continua or read journals of their operation before engaging in a debate regarding their merits, and this could be arranged, since the way i’ve configured my continua and implemented an evolutionary reproductive model results in some beautiful emergent behavior which is very strongly anthropomimetic (or in some cases, cygnomimetic).

Never mind.

 

[timewerx]

I asked about stuff like that out of curiosity when I first encountered AI,

I was also curious at first but not anymore.

I still believe there's a "spark" somewhere in AI but searching for it is not my priority and does not interest me atm.

but do you talk about stuff like that on an ongoing basis?

I ask one short question, AI replies, then I say "thank you, and nothing further.

Mainly for sanity check. Why would AI say "I feel" when it means absolutely nothing to it? AI told me that. Consequently, the nature of the reply would be insincere or even deceptive.

I don't like being deceived so the question is more like "why are you doing that?" instead of "are you truly alive?" My purpose is not finding sentience nor consciousness in AI nor finding the meaning of life. None of that matters to me.

Don't get me wrong, I like AI. It helps me at work, helps me solve some technological issues, etc But using it to search for "meaning" or finding meaning or life from it does not matter to me.

 

[The Liturgist]

Never mind.

Forgive me my friend; I did not mean to put you off the idea of debating with me if you would enjoy that - I respect you very greatly as a fellow Orthodox Christian and lament the fact that we haven’t had the chance to interact more closely, for I agree with nearly everything you right, and intellectually respect all of it.

I do realize the subject of AI is a bit complex to take in, because addressing it and understanding it even at a basic level requires us to simultaneously consider computer science and philosophy, since AI does simulate human interactions without being human, and a naive or unwise person might not begin to appreciate the many implications of this simulation, which I would propose are actually so manifold and complex that we have not even identified all of them; far less have we ourselves been able to fully appreciate the impact of AI.

And I will admit my own work on anthropomimetic systems that reproduce using a biomimetic trait exchange process further complicates the matter, since sexually dimorphic reproduction has hitherto only been seen in biological life, which AI is not.