![]() of bits needed to communicate the outcome”. Side note: Such equally probable instances where the number of possible outcomes is a power of 2 are realistically the places where we can apply the theoretical limit of “what I learnt, i.e. Figure 1 had a “1” bit entropy and Figure 2 has “2” bits and so on. When we are dealing with events containing equally probable outcomes, the measure should increase with the number of possible outcomes. We will go back to Figure 1 and Figure 2 examples to introduce another property for this measure. So similar to other measures like area, length, weight, etc., it is continuous and can be any number or fraction. But our refreshed definition of Entropy is a measure of “news” in the communication. Bits come in full - there’s no “half a bit”. While in a way, this might give us a theoretical limit on the number of bits needed to communicate the outcome - let’s not constrain ourselves with that as the basis of the definition. This uncertainty or information content is what is measured by Entropy. And a “1” for a fair coin toss outcome, meaning “I learnt a lot” or “I am most uncertain about” a fair coin toss output as it could either be heads or tails. It will be “0”, meaning “I learnt nothing” or “I was completely certain about the outcome”, for a biased coin that always turns up heads on tossing. We want it to be a quantitative measure of the information content aka uncertainty associated with the event. In other words, we want Entropy to represent how much “information content” is present in the outcome - however it is communicated to us. What we are really interested in, is not how many bits of data we need to communicate the outcomes, but rather it is“how much information we are really learning from what was communicated”, regardless of the actual physical bits used to communicate. Here’s where we need to adjust the way we defined Entropy a bit. Figure 4: Four not-equally-likely Possibilities Example
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |