Calculating Information Entropy

Derail from http://www.christianforums.com/t3315...on.html&page=8

How does one count information entropy? It's really simple:

H(x) = -sum for all i [ p(i) log2 p(i) ]
http://en.wikipedia.org/wiki/Information_entropy

H is entropy.
x is the message.
i is all possible "events". For example, DNA has four "i"s: A, T, C, and G.
p(i) is the "probability" of an event, in this case the number of occurrences of a particular event over the total number of events. For example, in the message "aabc", p("a") = 0.5 and p("b") = p("c") = 0.25.
log2 is the logarithm to base two.

Example 1:
aaaa -> aaab
For "aaaa", there is only one possible event. Therefore
p("a") = 1, log2 p("a") = 0, and p("a") log2 p("a") = 0. A nonsense message consisting of a single letter has zero entropy!
For "aaab":
p("a") = 3/4; log2 p("a") = -0.415; p ("a") log2 p("a") = -0.3113 (hereafter abbreviated as plp("a"))
Similarly, plp("b") = -.5
plp("a") + plp("b") = -0.915; H(x) = 0.915.

Example 2:
aabb -> aaab

For "aabb":
p("a") = 1/2 -> plp("a") = -.5
Similarly, plp("b") = -.5
plp("a") + plp("b") = -1; H(x) = 1.
For "aaab":
H("aaab") = 0.915, by Ex. 1
Notice a negative entropy change of -0.085. Random mutations can cause information entropy to decrease!

Example 3:
kissed -> kisses

For "kissed":
p("s") = 1/3
plp("s") = -0.5283
p("k") = 1/6
plp("k") = -0.4308
similarly, for "i", "d" and "e", plp = -0.4308
Sum of all plp = -2.252; H("kissed") = 2.252
For "kisses":
p("s") = 1/2 -> plp("s") = -.5
p("k") = 1/6
plp("k") = -0.4308
similarly, for "i" and "e", plp = -0.4308
Sum of all plp = -1.792; H("kisses") = 1.792

Again, a negative entropy change brought about by a mutation which could very well be random. I think "kissed -> kissee", "kissed -> kidded", and "kissed ->iissee" all yield negative entropy changes as well. The entropy change seems to be irrelevant to the meaning of the message, doesn't it?

Example 4:
"argument" - > "rgument" -> "gument"
I will not prove these results, which are perfectly repeatable, for the sake of brevity of working. (If Fermat could do it, so can I.)
H("argument") = log2(8) = 3
H("rgument") = log2(7) = 2.807
H("gument") = log2(6) = 2.585

Now even deletions decrease entropy! Not just that, nonsense strings have less entropy than sense strings.

Example 5: (DNA)
ATCAGC -> ATCATC
H("ATCAGC") = 1.918
H("ATCATC") = log2(3) = 1.585

Even when dealing with DNA, random mutations can decrease entropy.

This should put to rest any notion of a Second Law of Thermodynamics for information content.

__________________

p(i) is the "probability" of an event, in this case the number of occurrences of a particular event over the total number of events. For example, in the message "aabc", p("a") = 0.5 and p("b") = p("c") = 0.25.


there is something here i don't understand.

p(i) is the probability of the ith event

but you are taking the letter set to be those letters actually seen. when it appears to be over the whole potential set.

H(aaaaa) is different if:
{a} or {a,b} are the set of possible symbols.
for the {a} H(aaaa)=0
as in:
Example 1:
aaaa -> aaab
For "aaaa", there is only one possible event. Therefore
p("a") = 1, log2 p("a") = 0, and p("a") log2 p("a") = 0. A nonsense message consisting of a single letter has zero entropy!

but if the set is {a,b}
p(i)=.5, p(i)* log2 (.5)=.5 * (-1), summation for all i=-.5*4=-2


ok...what did i miss?


notes:
http://www.ccrnp.ncifcrf.gov/~toms/pitfalls.html for a general discussion of relating H() and S() and the problems of doing it.

__________________
i got a warning for flaming.
i find myself unable to participate here as a result.
this being the first of it's kind since i signed into FidoNet 20 years ago.
and am no longer posting to CF.

thanks to everyone who i have encountered over the years i spent here.

To be honest, I have no idea what I'm supposed to do when creationists talk about the "entropy of a signal". Apparently mutation increases the entropy of a signal, a DNA code in particular. The more I read, the more that claim strikes me as being nonsensical. You're right, entropy is defined for sources of information.

The only way I could think of defining it meaningfully for a signal is to treat the signal as a source of information itself. In other words, the entropy of signal "aaab" is really the entropy of an information source which outputs "a" 3/4 of the time and "b" 1/4 of the time. The entropy of signal "kisses" is the entropy of an information source which outputs "s" 1/2 of the time and "k", "i", "e" 1/6 of the time.

That's the closest thing I can think of if someone were to ask me what the "entropy of a particular DNA sequence" is (assuming of course that informational entropy is meant, and not the physical molecules-bonds-ions-etc. entropy). After a night's sleep I'm quite sure even that is nonsense.

Are there any people here who actually know what they're talking about when it comes to information theory?? XD

__________________

If we're talking about strings and the information content in strings, then the Creationists application of information theory to DNA is wrong.

I briefly studied measuring information in building decision trees for my machine learning class, and it seems that Creationists have it backwards. They think that the more random a string is, the more information it has, when in reality, the more random a string is, the more information it has because it takes more bits to describe it. That's what I gather from information theory. Therefore, random mutations are more likely to increase information.

__________________
Entropy is driving the universe toward "heat death" --- Jesus is the force behind the Strong Nuclear Force that keeps the protons together - Colossians 1:17. - AV1611VET

I think this argument is totally bogus. The whole concept of "information entropy" has NOTHING to do with entropy in thermodynamics. The concepts cannot be related like this!

It's like saying, "since gravity pulls down, heat must flow down." You're applying a concept in one specific area of physics (the flow of heat or energy) to something totally unrelated (information theory). I agree with the conclusion, but the argument is crap.

The laws of thermodynamics have nothing to do with "information." There is no law that says that information must decay over time. There is nothing that prevents an open system (like the Earth) from becoming more complex. NOTHING. And applying entropy to the "information" contained in DNA is a complete perversion of a theory where it does not apply.

__________________

Yes, my bad. The moral of the story is that DNA has no "informational" entropy, and therefore any idea of a 2nd Law generalization forbidding evolution in information-theoretic terms is bogus.

But I took too much time on the OP to want to delete it.

__________________

.

__________________

At some point, information must decrease as physical entropy decreases. A completely regular repetition of a single element (such as aaaaaaa) to maximum, in a setting where only a single element is possible, has the lowest possible entropy. It carries no information. Similarly the complete absence of anything (the null set) also has 0 entropy, and no information.

However, a completely random occurence of an infinity of elements, which has maximal entropy, also carries no information either.

Information is a NON-PHYSICAL entity which can be carried only in physical states between extremes of physical entropy.

At this we have an immediate revulsion: what is meant by non-physical? What is meant is something that scientists have come to grudgingly accept, that some aspects of reality are not strictly matter or energy. Information is one such real, mathematically quantifiable reality, which is neither matter nor energy. Information can be represented by a particular ordering of matter, but it is not in itself that matter. Your computer's software is clearly qualitatively different than hardware. The SAME software can be represented in various physical media, but it is not that media.

Therefore, calculations of physical entropy have little relation to information. It is true that at the extremes of total or zero physical entropy information can not be represented. This does not exclude (or include) what happens to information in the vast middle, where physical entropy is sufficient to allow the representation of information.

The fact that reality is not describable by naive materialism is very disconcerting for many. Yet physicists working in quantum mechanics have found that reality is far more interesting, and does not fit within the framework of philosophic materialism. Information is one such aspect of reality which, while quantifiable and interactive with matter and energy, is not in fact matter nor energy.

JR

Indeed, I'll concede the point. However, on Earth, this will not be a significant factor until the Sun burns itself out. I think most of us expect Jesus to make a stop back here SOMETIME in the next few million years (let alone the billions it would take for the universe to succomb to heat death). On human time scales, there is no barrier to an increase in information provided that the Sun's still pumping energy onto the Earth.

Quite right.
My point exactly.
Here you lose me. Materialism is perfectly capable of describing information... though certainly not as some physical entity. Information is a philosophical construct -- like mathematics, the idea of information is a concept that helps us to make sense of the universe. We've defined information so as to be able to use the patterns created by nature.

So in essence, the laws of nature creates patterns. We use our philosophical construct of information to be able to make use of these patterns.

It's not inexplicable -- it's just our way of describing what was already there in nature.

__________________

Please note my use of the term "NAIVE materialism." By this I mean the kind of materialist thinking that prevailed among scientists from about the time of Newton to the 1930's. Starting 1925 with Einsteinian relativity, rapidly worsening as Quantum Mechanics became increasingly bizarre, continuing with Godell's 2 incompleteness theorems, and spreading through multiple disciplines, such as Piaget's utter destruction of behaviorism as an explanation for language aquisition; "modernism" hit a wall and was irretrievably destroyed. Modernism, which consisted of a simple belief in materialism, together with an optimistic view that everything could be known, explained and predicted by the scientific method died very suddenly. Currently there is much angst about how to replace it, with post-modernism's assertion that there is NO objective Truth, or if such Truth exists we are incapable of ever knowing it leaving most unsatisfied.

Thus materialism has had to grapple with the obvious fact that there is more in this world than matter and energy, such as information. Now, while information can be represented by the arrangement of physical elements, as we agree, it is something other than mattr/energy. Use whatever terms you like, non-physical realities (in the past called metaphysics, a term shunned because, while accurate, is imprecise). In fact, I'd prefer if you come up with a term.

The thing is, information has it's OWN entropy, which can not be calculated given the formula you used. As you've agreed, that formula gives incorrect results. Input a string of 1 million a's into that formula, and make the informational content where a single a is replaced by a single b. For all but one such string you get the incorrect message, yet the formula yields identical entropy values. Better yet, take any single english word. For example "word". This string has the identical entropy to "owdr" yet it is obvious tha "word" carries information and "owrd" does not.

Informational entropy is calculated as the CHANGE from an ideal message, not as the arrangement of physical bits. Thus "word" has 0 informational entropy, "wzrd" higher entropy, "azrd" higher still and so on. How this is calculated is beyond my understanding, and in fact, some parts of the physical arrangement (namely the first & last letters plus the length of the word) are more important than others. Thus "wzfd" is more recognizable as "word" than "zorchs". I am not well versed in this field at all, but can clearly see the formula you cite is either incorrect or (more likely) misapllied.

But in fact things are much worse for materialists these days, even materialists that have grudgingly expanded their philosophy to account for such real non-physical phenomena as information and aesthetics.

In fact, there is conclusive proof from experimental evidence that CONCIOUSNESS alters the behaviour of matter (principally the alteration of behaviour of single particles to being observed as they encounter a double slit). Now this is even more disturbing to materialist because it demands there be an actual difference between inanimate and animate observers. It amuzes me greatly to read modern physicists as they grapple with quantum mechanics. Many quickly start talking about Eastern religions (for example, read "The Dance of the Wu-Li Masters"). When materialist philosophy has to be so expanded as to admit the concept of "conciousness" as a variable in physical experiments, it is materialism in name only.

Information decays over time, just as physical entropy increases over time. Not all the magic homunculi can increase useful information as a totality of the system. This is a simple fact we all see every day. Messages do not become clearer to understand with cellular static. While great discoveries do occur because of accidents, it is only by the intervention of an intelligence.

If you truly believe in the random increase of information, you would respond by hitting your keyboard randomly. You do not because the scientific method you have employed since you were knee high has taught you that information requires an intelligent effort to aquire and disseminate. As an agnostic, I did not have the faith required to believe the myths in my textbooks about self-ordering matter, and chose rather to believe my own eyes. It never seemed my room ever got neater by my throwing things around, nor my book reports more convincing by randomly smudging ink on a page. Perhaps had I an appreciation of modern art I could believe in macroevolution and the powers of randomness, but to me, even before belief in Christ, both modern art and macroevolution seemed like a scam. The fact that very, very intelligent people spend millions of dollars and gawk appreciatevely, (dare I say with religious awe?) at both modern art and macroevolution dosen't impress me. I have always chosen to believe my own reason than the madding crowd.

JR