In a long typical English text, the letter E occurs more frequently than any other letter, the letter Z is the least frequently letter, the letter Q is almost always followed by the letter U, the letter T occurs more frequently than any other consonant, and occurrence of a consonant generally implies that the following letter will be more likely to be a vowel than another consonant. Such a source producing dependent symbols from a set of 27 symbols, the 26 letters and a space, is considered ergodic, for if observed for a very long time, will (with probability 1) produce a sequence of source symbols that is typical. Messages that are highly structured usually convey less information per symbol, hence the decrease in its entropy.Ī prime example of a source producing non-independent symbols is the English language, which like any natural language has a statistical structure, as a letter in a multi-letter word may depend upon a number of preceding letters. The average information content per symbol emitting dependent symbols decreases as the message length increases (i.e., the average number of bits per symbol needed to represent a message decreases as the message length increases). It is important to note that in a DMS, symbols are generated in a statistically independent fashion, but most practical sources produce symbols that are statistically dependent, and statistical dependence lowers the amount of information.
Since we have H X 2 = 2 H X = 2 3 2 = 3 bits, (9.7) is satisfied.
0 kommentar(er)
