Probability Theory: The Logic of Science

I have rarely learned so much from one book. This book is somewhat unusual among mathematical texts in that it is heavy on prose and (compared to other texts) light on equations. However, don't get the idea that it is any less rigorous! It simply focuses on precisely what most math books neglect: exhaustive explanation of the concepts...and to very good effect. Jaynes (and his editor) are possibly the most articulate writers of mathematics I've ever read. If you can read equations like English, you may not appreciate this. The rest of us will. Summarizing the content: The book very exhaustively demonstrates how Bayesian statistical approaches subsume rather than compete with "orthodox" (sampling theory-derived) statistics. Importantly, it begins by deriving the sum and product rules (which in other texts are typically presented as axioms) from "common sense" considerations. In other words, what is usually treated as "given" in other statistics texts is shown to, in fact, depend on even more fundamental (and, thus, indisputable) considerations of what constitutes rational plausible reasoning. This places the whole endeavor of statistics on firmer ground than any other text I've seen. The book is worth buying for the first few chapters alone, but it just gets better from there. Jaynes goes on to link Bayes rule to information-theoretic considerations and build up probability as an extended form of logic (as the title implies). In some cases this yields a new and deeper understanding of "orthodox statistical practice." In others it exposes (and explains) the absurdities of strictly frequentist approaches. Again, I have rarely learned so much from one book. One caveat: It does not at all require a statistics background, but, obviously, some of Jaynes (mildly polemical) discourse will, of course, be lost on you without it.

To me, this is the greatest book ever written on Statistics. I have studied statistics for the past 22 years and I have been teaching statistics for the past 10 years. I only got to know this book a couple of years ago. Many many conceptual issues that I have had in Statistics have been clarified from a careful study of this book. Jaynes had a deep understanding not only of Bayesian Statistics but also of Frequentist Statistics. Everything that he says about Frequentist "Orthodox" Statistics is correct (although often it took me many months to fully understand what he is saying). The ideas and messages of this book significantly differ from what is taught in pretty much all other statistics books. Here is one example, the Gaussian distribution is heavily used in statistical analysis. Most textbooks are pretty much apologetic about this overuse of the Gaussian distribution and struggle to suggest alternative methods. Jaynes, on the other hand, says (in Chapter 7) that the range of validity for the application of the Gaussian distribution in data analysis is actually "far wider that is usually supposed". A major highlight of the book is the focus on history. Very careful historical accounts are presented as to how the greats of the field (like Gauss, Laplace, Cox, Fisher etc) approached data analysis. This stuff again cannot be found in any other book in the field. I have been using this book heavily in pretty much anything I teach these days and, as a consequence, teaching statistics has been a much more pleasurable experience than before. Jaynes apparently originally wanted to write a sequel to this book focussing on more advanced applications. It is a pity that he passed away before he could write the sequel. I recommend readers to the outstanding books by MacKay and by von der Linden-Dose-von Toussaint for numerous interesting and nontrivial applications of Probability Theory (Bayesian Statistics) to Data Problems. I would also like to recommend (as sequels to reading Jaynes) the books of David Blower which clarify and complement the ideas of Jaynes. For readers interested in learning more about the various issues, pitfalls and shortcomings of Frequentist "Orthodox" statistics, I would like to recommend the collected works of Dev Basu.

I haven't finished reading this book yet, but the chapters I read so far gave me so much understanding of issues that are either obscure or absent in other probability and statistics books - but are of great practical importance - that I decided recommend it here. It is true Jaynes' style is caustic against positions that are contrary to his owns. But he is very convincing on the reasons he gives to pinpoint the big holes in the so called "orthodox" school of probability and statistics. Besides, the book is very lengthy, without being prolix, on its explanations, making it very pedagogical. Constrasting with that, nevertheless, Jaynes sometimes proposes examples that I believe only a mathematician or physicist with specific knowledge of the subject mentioned by the author will be able to follow. But those parts do not impact understanding of the main ideas. It must be noted also that "Probaility theory: the logic of science" is mainly a theory book. Its goal is to present probability as an extension of deductive logic. It only brings a small number of exercises. The best thing about this book, at least for me, is having a style that really makes me look forward reading the next page, something very rare for a technical book. In fact, the only other book I came across that had that virtue was the "Feynman Lectures on Physics".

One of the most important works of the 20th century (or any century) in both philosophy and physics, Jaynes' work lays the foundation for the physical ontology and epistemology of science. This book is the completion of what amounted to a lifetime of effort on Jaynes' part, dating back to the "Mobil Lectures" where he first laid out this approach to knowledge. It follows the world of Richard Cox, who demonstrated that Bayesian probability theory naturally follows from three simple axioms that also serve to establish the connection between evidence and plausible belief. In my opinion, this book is a required read for anyone who wishes to understand precisely how the scientific worldview is, in a mathematically defensible sense, the best possible worldview, the one that lets us optimally use evidence to develop an interlocked Bayesian network of evidence supported beliefs that can change and evolve as the evidence is accumulated. It also shows the critical connections between physics and statistical mechanics and Shannon's theorem in computational information theory, laying the foundation for a fair bit of modern physics as it demonstrates that physical entropy and information entropy are very much one and the same thing, from a certain point of view.

I really think everyone should read this book, and R. T. Cox's book. This books make plain the philosophical and mathematical reasons that all statistics should be Bayesian. It only looses a star, because the book is not complete. Jaynes died before he could finish it. One of his students prepared the book for publication. I wish he had finished it while he was at it.

Very thought provoking and interesting look at the subject. It is the voice of common sense in a subject area where many people adhere to strict frequentist inference and the traditional examples. While covering these same topics, this book makes notes and gives counter examples. It focuses on the idea of information relative to uncertainty. The author thoroughly criticizes and is careful about assumptions and critical of them, unlike many other books/authors. Many results will end up the same as deriving them with Kolmogorov, but the approach will be different. Note: The author writes long paragraphs describing problems and his logical approach to them. They are very important to the core understanding of his message and ideas. I would only recommend this for graduate level work.

Every scientist should read this book. It deals with the most fundamental question of experimental science: how do we assign plausibility based on data? What if we have very little data? What if we have none? It includes foundations of probability theory (introduced in a conversational and often funny tone), builds very slowly to its conclusions, and deals with most common criticisms of Bayesian analysis and MaxEnt methods, often based on misunderstandings. Also included are examples of applications, and places where Jaynes leaves the door open to further development. After all, this field is far from complete. This book is a manifesto. It embodies a sense of urgency and righteousness ever-present in scientific revolution. Such a sense is not misplaced here.

This is a classic and belongs on every serious data scientists bookshelf. It goes beyond just simply teaching statistics and focuses on the thinking one needs to acquire in order to understand underlying principles. As others have said, it is not for beginners since it is not a simplistic how to style reference. It requires concentrated effort to become familiar with the thought patterns one needs to develop to be a serious scientist. There are other books that provide the nuts and bolts of statistics, but this book provides a deep understanding of the usage of the various methods.