Let’s refine and make clear the reason:
Now, let’s discover first the fee we pay to encode n-bit info into fewer bits relies on guaranteeing that every codeword shouldn’t be a prefix of one other. This imposes a price by lowering the variety of potential codewords.
Observe the desk
– If we use a code of size 0, we will solely encode 1 codeword, so we pay a price of 1 (no different codes are potential).
– For 1-bit knowledge, we pay a price of 1/2, as a result of selecting ‘0’ means half of the codewords will begin with ‘0’.
– For two-bit knowledge, the fee is 1/4, as selecting ‘01’ means 1 / 4 of the codewords begin with ‘01’.”
This clarification clarifies how the fee is decided primarily based on the prefix-free property of codes and the fraction of potential codewords sacrificed to make sure distinctive decodability.
So the fee for code of size L(x) is :
Now, one can intuitively determine that larger chance must be assigned to lower-cost codewords, and decrease chance to higher-cost codewords. To attenuate the anticipated price, we distribute chances in proportion to the inverse of the fee, i.e.:
You may confer with Colah’s blog for the proof, as it’s going to present a extra in-depth exploration of the subject.
Now,
or anticipated size as,
And L is outlined as entropy, Decrease entropy signifies larger certainty of the occasion.
Folks usually write entropy as :
utilizing the log property, log(1/a) = -log(a). Though the earlier expression offers true instinct. Now, let’s delve into the mathematical points of entropy, focusing particularly on Shannon Entropy (or entropy).
Shannon Entropy : Its identical as we outline entropy above utilizing info and code concept. Lets outline it extra mathematically.
The Shannon entropy, of a discrete random variable X is
the place x, is the one of many potential occasions from the random variable X
P(X=x) is the chance of the occasion x.
Generally case (not discrete), the Shannon entropy of a random variable X with distribution µ, with respect to a reference measure ρ , is
or
You would possibly surprise what’s reference measure and why a reference measure is important, particularly within the context of steady variables.
