Information Theory

The World and Theories and What This Means For Information Theory

These are book notes for Introduction to Information Theory by John R Pierce and part 1 of this series. Check the tag History of Information Theory for more blogs similar to this. Hear me rambling on and on about Information Theory here and if you want to check out more book summaries do check the tag Book Summary


  • Introduction to Information Theory by John R Pierce
  • Scientific Theory : The question it answers
  • Communication Theory — An Abstract yet a generalised theory

Introduction to Information Theory I got this audiobook on audible and fell in love with it. It was written in 1980’s and is narrated by Kyle Tait. I never thought I could listen to a book about a theory which is completely mathematical but man, I was wrong. The book just matches the same Shannon-fangirl energy level that I am at and it’s just perfect. John R Pierce describes information theory in the most beautiful way you can ever think of, is it art, is it magic, is it philosophy, is it a mathematical theory?

When we add to theory the word mathematical, with all its implications of rigor and magic, the attraction becomes almost irresistible. Perhaps if we learn a few formulae our problems of communication will be solved, and we shall become the masters of information rather than the slaves of misinformation.

John R Pierce

We start off by understanding the universal, bewildering notion of physics and of motion discussed by Aristotle.

“He included in the concept of motion the increase and decrease of that which can be increased or decreased, coming to and passing away, and also being built” John about Aristotle.

Though Aristotle’s concepts of motions were universal it was puzzling till we look at Newton’s laws. Newton’s laws are special definitions and are more generalised. This is where we taste our conflict. Universal, all encompassing theory which is abstract vs. a generalised but specific theory. There is always a tradeoff !

“Yet, they do little to answer many of the questions about motion which Aristotle considered. Newton’s laws solved the problem of motion as Newton defined it, not of motion in all the senses in which the word could be used in the Greek of the fourth century before our Lord or in the English of the twentieth century after” John Argues.

Let me point out something interesting which has been a thought lingering at the back of my head constantly. Newton’s laws solved the problem of motion as Newton defined it. We solve the problem we want to solve. The solution is never universal, but it’s a solution to the problem that we want it to work for. Do we alter the question to fit the solution, or do we solve for the question?

A valid scientific theory seldom if ever offers the solution to the pressing problems which we repeatedly state. It seldom supplies a sensible answer to our multitudinous questions. Rather than rationalising our ideas, it discards them entirely, or, rather, it leaves them as they were. It tells us in a fresh and new way what aspects of our experience can profitably be related and simply understood.In this book, it will be our endeavour to seek out the ideas concerning communication which can be so related and understood.
When the portions of our experience which can be related have been singled out, and when they have been related and understood, we have a theory concerning these matters

Let’s talk about where communication theory falls on this spectrum. It is a strongly mathematical and it is also generalised enough. It deals with problems of information in abstract way.

Also, we look at what is a bit. Did you know the term bit was coined by Shannon in his paper 1948 paper “A Mathematical Theory of Communication“. A bit is universal measure of amount of information in terms of choice of uncertainty. Sounds complicated? Don’t worry, we will cover all of these and break it down to it’s ELI5 version when we come to it.

  • Communication theory tells us how many bits of information can be sent per second over perfect and imperfect communication channels in terms of rather abstract descriptions of the properties of these channels.
  • Communication theory tells us how to measure the rate at which a message source, such as a speaker or a writer, generates information.
  • Communication theory tells us how to represent, or encode, messages from a particular message source efficiently for trans- mission over a particular sort of channel, such as an electrical circuit.
  • Communication theory tells us when we can avoid errors in transmission

Take Aways :

  1. A scientific theory will give tell you what aspects of your experience could be simply understood than giving an answer to the most pressing problem. Instead of something that solves a universal problem, it gives you different way to look at it.
  2. Communication theory or information theory is both abstract enough and also generalised. This makes the application space of this theory very wide.
  3. I don’t write good conclusions

Leave a Reply

%d bloggers like this: