A class of small-deviation theorems for the relative entropy densities of arbitrary random field on the generalized Bethe tree are discussed by comparing the arbitrary measure with the Markov measure on the generalized Bethe tree. As corollaries, some Shannon-Mcmillan theorems for the arbitrary random field on the generalized Bethe tree, Markov chain field on the generalized Bethe tree are obtained.
1. Introduction and Lemma
Let be a tree which is infinite, connected and contains no circuits. Given any two vertices , there exists a unique path from to with distinct. The distance between and is defined to , the number of edges in the path connecting and . To index the vertices on , we first assign a vertex as the "root" and label it as . A vertex is said to be on the th level if the path linking it to the root has edges. The root is also said to be on the 0th level.
Let be a tree with root , and let be a sequence of positive integers. is said to be a generalized Bethe tree or a generalized Cayley tree if each vertex on the th level has branches to the th level. For example, when and (), is rooted Bethe tree on which each vertex has neighboring vertices (see Figure 1, ), and when (), is rooted Cayley tree on which each vertex has branches to the next level.
In the following, we always assume that is a generalized Bethe tree and denote by the subgraph of containing the vertices from level 0 (the root) to level . We use () to denote the th vertex at the th level and denote by the number of vertices in the subgraph . It is easy to see that, for ,
Let , , , where is a function defined on and taking values in , and let be the smallest Borel field containing all cylinder sets in . Let be the coordinate stochastic process defined on the measurable space ; that is, for any , define
Now we give a definition of Markov chain fields on the tree by using the cylinder distribution directly, which is a natural extension of the classical definition of Markov chains (see ).
Figure 1. Bethe tree .
Let . One has a strictly positive stochastic matrix on , a strictly positive distribution on , and a measure on . If
Then will be called a Markov chain field on the tree determined by the stochastic matrix and the distribution .
Let be an arbitrary probability measure defined as (1.3), denote
is called the entropy density on subgraph with respect to . If , then by (1.4), (1.5) we have
The convergence of in a sense ( convergence, convergence in probability, or almost sure convergence) is called the Shannon-McMillan theorem or the entropy theorem or the asymptotic equipartition property (AEP) in information theory. The Shannon-McMillan theorem on the Markov chain has been studied extensively (see [2, 3]). In the recent years, with the development of the information theory scholars get to study the Shannon-McMillan theorems for the random field on the tree graph (see ). The tree models have recently drawn increasing interest from specialists in physics, probability and information theory. Berger and Ye (see ) have studied the existence of entropy rate for G-invariant random fields. Recently, Ye and Berger (see ) have also studied the ergodic property and Shannon-McMillan theorem for PPG-invariant random fields on trees. But their results only relate to the convergence in probability. Yang et al. [7–9] have recently studied a.s. convergence of Shannon-McMillan theorems, the limit properties and the asymptotic equipartition property for Markov chains indexed by a homogeneous tree and the Cayley tree, respectively. Shi and Yang (see ) have investigated some limit properties of random transition probability for second-order Markov chains indexed by a tree.
In this paper, we study a class of Shannon-McMillan random approximation theorems for arbitrary random fields on the generalized Bethe tree by comparison between the arbitrary measure and Markov measure on the generalized Bethe tree. As corollaries, a class of Shannon-McMillan theorems for arbitrary random fields and the Markov chains field on the generalized Bethe tree are obtained. Finally, some limit properties for the expectation of the random conditional entropy are discussed.
Let and be two probability measures on , , and let be a positive-valued stochastic sequence such that
In particular, let , then
Proof (see ).
is called the sample relative entropy rate of relative to . is also called the asymptotic logarithmic likelihood ratio. By (1.9)
Hence can be look on as a type of measures of the deviation between the arbitrary random fields and the Markov chain fields on the generalized Bethe tree.
2. Main Results
Let be an arbitrary random field on the generalized Bethe tree. and are, respectively, defined as (1.5) and (1.10). Denote , the random conditional entropy of relative to on the measure , that is,
where is the natural logarithmic, is expectation with respect to the measure .
Let be the probability space we consider, an arbitrary constant. Define
We can obtain by (2.7), (2.8) that in the case ,
Therefore, , are a class of consistent distributions on . Let
then is a nonnegative supermartingale which converges almost surely (see ). By Doob's martingale convergence theorem we have
Hence by (1.3), (1.9), (2.9), and (2.11) we get
By (1.4), (2.8), and (2.11), we have
By (1.10), (2.2), (2.13), and (2.14) we have
By (2.15) we have
By the inequality
() and (2.16), (2.17), (2.3), we have in the case of ,
When , we get by (2.18)
Let , in the case , then it is obvious attains, at , its smallest value on the interval . We have
When , we select such that (). Hence for all , it follows from (2.19) that
It is easy to see that (2.20) also holds if from (2.21).
Analogously, when , it follows from (2.18) if ,
Setting in (2.14), by (2.14) we have
By (1.4), (1.5), (2.20), and (2.23), we obtain
Hence (2.4) follows from (2.25). By (1.4), (1.5), (1.10), (2.2), and (2.22), we have
Therefore (2.5) follows from (2.26). Set in (2.4) and (2.5), (2.6) holds naturally.
Let be the Markov chains field determined by the measure on the generalized Bethe tree , are, respectively, defined as (1.6) and (2.3), and is defined by (2.1). Then
We take , then . It implies that (2.2) always holds when . Therefore holds. Equation (2.27) follows from (2.3) and (2.6).
3. Some Shannon-McMillan Approximation Theorems on the Finite State Space
Let be an arbitrary random field which takes values in the alphabet on the generalized Bethe tree. , and are defined as (1.5), (1.10), and (2.2). Denote , . is defined as above. Then
Set we consider the function
Let thus . Accordingly it can be obtained that
By (2.3) and (3.5) we have
Therefore, (2.3) holds naturally. By (2.18) and (3.6) we have
In the case of , by (3.7) we have
Let , in the case , then it is obvious attains, at , its smallest value on the interval . That is
By the similar means of reasoning (2.21), it can be concluded that (3.9) also holds when . According to the methods of proving (2.4), (3.1) follows from (1.5), (2.23), and (3.9). Similarly, when , , by (3.7) we have
Imitating the proof of (2.5), (3.2) follows from (1.5), (1.10), (2.2), and (3.10).
Corollary 3.2 (see ).
Let be the Markov chains field determined by the measure on the generalized Bethe tree is defined as (1.6), and is defined as (2.1). Then
By (3.1) and (3.2) in Corollary 3.1, we obtain that when ,
Set , then . It implies (2.2) always holds when . Therefore holds. Equation(3.11) follows from (3.12).
Under the assumption of Corollary 3.1, if , then
It can be obtained that . holds if (see Gray 1990 ), therefore . Equation (3.13) follows from (3.12).
Let be a Markov chains field on the generalized Bethe tree with the initial distribution and the joint distribution with respect to the measure as follows:
where is a strictly positive stochastic matrix on , is a strictly positive distribution. Therefore, the entropy density of with respect to the measure is
Let the initial distribution and joint distribution of with respect to the measure be defined as (1.4) and (1.5), respectively.
We have the following conclusion.
Let be a Markov chains field on the generalized Bethe tree whose initial distribution and joint distribution with respect to the measure and are defined by (3.14), (3.15) and (1.4), (1.5), respectively. is defined as (3.16). If
Let in Corollary 3.1, and by (1.5), (3.15) we get (3.16). By the inequalities , , (3.17), and (1.10), we obtain
By (3.17) and (3.20) we have
It follows from (2.2) and (3.21) that ; therefore (3.18), (3.19) follow from (3.1), (3.2).
4. Some Limit Properties for Expectation of Random Conditional Entropy on the Finite State Space
Lemma 4.1 (see ).
Let be a Markov chains field defined on a Bethe tree , be the number of in the set of random variables . then for all ,
where is the stationary distribution determined by .
Let be a Markov chains field defined on a Bethe tree , and let be defined as above. Then
Noticing now , for all , , that therefore we have
Noticing that , by (4.3) we have
Equation(4.2) follows from (4.4).
Let be a Markov chains field defined on a Bethe tree , defined as above. Then
By the definition of and properties of conditional expectation, we have
Accordingly we have by (4.6)
Therefore (4.5) also holds.
The work is supported by the Project of Higher Schools' Natural Science Basic Research of Jiangsu Province of China (09KJD110002). The author would like to thank the referee for his valuable suggestions. Correspondence author: K. Wang, main research interest is strong limit theory in probability theory. D. Zong main research interest is intelligent algorithm.
Wen, L, Weiguo, Y: An extension of Shannon-McMillan theorem and some limit properties for nonhomogeneous Markov chains. Stochastic Processes and their Applications. 61(1), 129–145 (1996). Publisher Full Text
Berger, T, Ye, ZX: Entropic aspects of random fields on trees. IEEE Transactions on Information Theory. 36(5), 1006–1018 (1990). Publisher Full Text
Ye, Z, Berger, T: Ergodicity, regularity and asymptotic equipartition property of random fields on trees. Journal of Combinatorics, Information & System Sciences. 21(2), 157–184 (1996). PubMed Abstract | Publisher Full Text
Yang, W: Some limit properties for Markov chains indexed by a homogeneous tree. Statistics & Probability Letters. 65(3), 241–250 (2003). PubMed Abstract | Publisher Full Text | PubMed Central Full Text
Yang, W, Liu, W: Strong law of large numbers and Shannon-McMillan theorem for Markov chain fields on trees. IEEE Transactions on Information Theory. 48(1), 313–318 (2002). Publisher Full Text