Some ramblings about Information Theory

Since Claude Shannon gave birth to Information Theory, we have seen how a number of subject has been influenced by his ideas. Information Theory is fundamentally based in the idea of entropy, a very well known, but not always well understood magnitude in Thermodynamics. 

I recently came across two interesting articles where Information Theory is applied: the creation of Maxwell’s Demon and the relation between information and living systems. 

In the first one researchers have confirmed that information comes with a cost, and the Second Law of Thermodynamics cannot be cheated. 

As interesting as confirming that information has an energetic cost, I find the second article more exciting. In it, Christoph Adami at Michigan State University proposes the idea that life is basically a matter of information. Adami’s idea is that a living system is a system that is not in thermodynamic equilibrium, but maintains itself in a state that differs from maximum entropy. This difference is the information that the living system contains. Somehow, the living system keeps this difference while it is alive. In his mathematical model, he found out that in systems close to equilibrium he can find molecules that can replicate themselves (living systems). Whenever the difference between the system and the environment is large, the probability of finding a living system is very small. 

This is not the first attempt to mathematically understand Biological problems. It reminded me of the application of Information Theory to Ecology by Margalef , or the attempts of understanding systems far from equilibrium by Prigogine, for example. 

Can we define life in such an abstract way? Is it so simple? 

Photo from Unicam.

 

From Supersymmetry to Schoenberg

Professor James Gates on Supersymmetry. The talk starts at around minute 12. He goes from symmetry concepts to the music of Arnold Schoenberg. What I found most interesting is the finding about doing calculus by means of graphs, or by coding the operations into binary code. They transform equations into what they called Adinkramat. Simply genius.

Quantum theory and information theory (again)

In a previous post we already talked about the relationship between quantum theory and information theory. In it, we saw how Lluís Masanes and collaborators derived quantum mechanics from a few postulates based on the properties that a unit of information should have. 

Today, I came across with two different articles that explore this relationship. In the first one, Stephanie Wehner and Esther Hanggi from the National University of Singapore’s Center for Quantum Technology, showed us that the uncertainty principle is intimately related to the second law of thermodynamics. (Note that thermodynamics are intrinsically related to information theory.) In particular, they saw how by loosening the uncertainty principle, they got more useful energy/information out of the system than they put into it thus violating the second law of thermodynamics. Since the violation of the second law is incompatible with the physics we know, this means that our ability to calculate the state of a particle with infinite accuracy (the uncertainty principle) is forbidden by the second law. Note that thermodynamics is related to the macroscopic state of a system, whereas quantum mechanics is related to its microstates. 

The second article I found interesting states that macroscopic systems cannot be quantum in nature, that is, we do not observe a superposition of states in the macroscopic world, but one only state. The author, Bolotin, states that the solution of the Schrodinger equation is just unsolvable for macroscopic objects. Bolotin says that the problem of solving Schrodinger equation is NP hard, and he shows, making a few calculations, that the computation time to solve this equation for a macroscopic state will either exceed the time of the universe, or the computation speed should be higher than the Plank time, where no state makes sense.

So the question here is, how does the universe compute its state? How does it go from quantum to macroscopic? 

I think the answer to that question goes again to the field of computational mechanics. In their article, Shalizi and Moore explain how Nature can be described in different levels of detail. They show how macroscopic states can have a higher predictability efficiency than the dynamics of their corresponding microstates. It all falls to information theory again. It their article, they define emergency of one description from the other, that is, a coarse grained version of the microstates, but with higher prediction efficiency than the other. 

It seems to me that the universe describes itself in only one way, it is only the way we look at it that separates between the different levels of description. As Shalizi and Moore put it: “for every question we ask It, Nature has a definite answer; but Nature has no preferred questions.”

Beauty and Truth

I recently read an article in Nature where they find that mathematicians react to what they think as beautiful equations in the same way somebody reacts to visual and musical beauty. It seems that the same region of the brain that is correlated to emotional responses to different kinds of beauty reacts when they see a specially beautiful equation. 

In particular, it seems that the famous Euler's formula is the most beautiful one. It relates the most important numbers in maths, that is, e, i, π, 1 and 0. The simplicity of it, and the whole meaning of this relationship is processed by the trained mind as beautiful. 

Long time ago I read the book "Why Beauty Is Truth: The History of Symmetry". In it, they explain why the sense of beauty (related very often to symmetry), drives mathematicians to find new algorithms and models. 

Maths is thought of as arid for most of the people. But, when you understand the rules, the logic of it, how one thing leads naturally to another, when you are trained in maths, you can experience it as a kind of art. 

Not surprisingly Srinivasa Ramanujan's infinite series for 1/π was found the ugliest. 

I don't find it specially beautiful either. But think about Ramanujan, an Indian mathematician that had no formal training, but an exceptional intuition for numbers. Beauty is somehow correlated not only to the individual components, but to the relationship between them in a piece of art. I guess in the mind of Ramanujan this formula correlated so many different parts of his knowledge that he probably found it very beautiful. 

I think everybody can relate to musical or visual beauty to some extent, but I guess also the trained artists or musicians experience this kind of beauty differently to the rest of the people. I remember I went once to an exhibition of the work by Paul Klee. A friend of mine explained to me a few things, and my whole experience changed, because I understood some things better. 

I think beauty is not only about emotion, but goes somewhere deeper as our understanding increases. As Tagore, another Indian said: "When our universe is in harmony with Man, the eternal, we know it as truth, we feel it as beauty". When we correlate to a stimulus as beautiful, we feel it as truth. For a mathematician, when they see truth in an equation, they experience it as beautiful. 

I think Plato would be very pleased with Zeki and collaborators' findings.

Conditional probability explained

Conditional probability was never so explicitly explained as in this visual application. Check this link. Via Microsiervos.

Curious things about infinity

This video reminded me of the book The Infinite Book, by John D. Barrow.

Yes, infinity is a very curious object. Aren´t you surprised?

The music of the spheres

It is said that the idea of the harmony of the spheres is attributed to Pythagoras, that the planets and stars moved according to mathematical equations, which corresponded to musical notes and thus produced a symphony. Well, Pythagoras was probably the first person to notice the remarkable relationship between music and mathematics. 

In another post I already mentioned the topic, but now I want to show you the work by Dmitri Tymoczko, which I fond through this link. In his works, which are published in Science, he describes harmony in a geometric way. He explains how the possible paths between cords are influenced by the symmetries of his mathematical constructions. 

So now we not only can listen to the music of the spheres, we also can see it!

Our experience of space and time

I found very interesting this reading about the discovery of the amplituhedron. It made me think about what we understand for time and space. 

Since the beginning of human curiosity we thought we knew what space and time is. We measured distances in rudimentary units such as cubits or stadiums, and the kept track of time with clepsydras or the sun's cycle. 

With these concepts Galileo started studying the relationship between space and time to represent movement. Descartes gave us a reference system, and Newton gave us laws for the movement of objects in these systems. Laws that apply both to celestial bodies and everyday objects. Our mind was prepared to accept three numbers for space and one for time. And then came Einstein's geometrical idea of curved space, and time and space were intrinsically connected. 

And then Quantum Mechanics makes us think that time is only an illusion, a derivation of the relationship between objects. 

The recent discovery of the amplituhedron for the calculation of the probabilities of outcomes of particle interactions changes our idea of time too. The change (time) arises from the change in the structure of this geometrical construct, not from the change in the object itself. 

We can see how the different geometrical descriptions of the natural laws change our interpretation of what is reality. Our everyday experience clashes against our mathematical representation of it. It does not mean that our experience is not valid. It only underlines the difficulties of describing Nature from the mathematical point of view. 

As our calculation power improves, we feel detached from reality. But, if we can measure reality in a better way, does not it mean that we understand it better?

A review of "Gödel, Escher, Bach: An Eternal Golden Braid"

I think the most interesting way (at least for me) of learning about the Gödel’s Incompleteness Theorems is by reading Douglas Hofstadter´s book "Gödel, Escher, Bach: An Eternal Golden Braid". 

Both content and form are carefully chosen, in such a way that you are introduced to very difficult concepts in a, let´s say, artistical way. And of course it is completely intentional. The structure of the book is, as it has been said, a counterpoint between Dialogues and Chapters.

The author makes a explicit analogue of Bach´s music in his book. And, I think, it shows the universality of the concepts he wants to show.

As a fugue, he first shows the main themes, and masterly proceeds to makes variations of it. He presents the concept of recursivity and isomorphism from an artistic point of view. He presents what he calls a "strange loop" in terms of music and painting. As he defines them it is a phenomenon that occurs whenever, by moving upwards (or downwards) through the levels of some hierarchical system, we unexpectedly find ourselves rightback where we started. You can find this in Bach´s music and in Escher paintings. 

He also explains that implicit in the concept of Strange Loops is the concept of infinity, and the conflict between the finite and the infinite in Bach´s and Escher´s works.

He introduces the main theme as the so-called Epimenides paradox, or liar paradox. Epimenides was a Cretan who made one immortal statement: "All Cretans are liars." What can be said about that statement? It is true or false? 

Godel´s idea was to use mathematical reasoning in exploring mathematical reasoning itself, and came up with the conclusion that "All consistent axiomatic formulations of number theory include undecidable propositions.

Hofstadter masterly shows the importance of strange loops in Gödel´s proof of his theorem. Once a formal system ask about its consistency and completeness, it cannot reach a conclusion. The equivalent of Epimenides paradox in mathematics it´s "This statement of number theory does not have any proof". Whereas the Epimenides statement creates a paradox since it is neither true nor false, the Gödel sentence is unprovable (inside its own formalization) but true. 

As Hofstadter concludes, Gödel’s Theorem shows that there are fundamental limitations to consistent formal systems with self-images. In particular, it cannot proves its own consistency. 

It has been used to prove that we cannot compute the human mind, because we would be incomplete. There is the argument against this of humans not being consistent, so we could be inconsistent Turing machines, and therefore computable. Read more about the computability of the mind and the Gödel´s theorems here.

 

I must say that it is not an easy book to read. You have to pay a lot of attention to it. But it is worth reading. 

As a mathematical representation of one of Bach´s compositions, here you have the "Crab canon", from his Musical Offering, with which Hofstadter starts his book. 

Omar Khayyam

I know Omar Khayyam from his Rubáiyát, or poems with 4 verses. I found them very beautiful, although a little bit sad. I found very striking the way he expressed his thoughts in such a modern way. He lived in the XII century, and his thoughts about religion and life were very up-to-date. Take these examples: 

You know, my Friends, with what a brave Carouse 

I made a Second Marriage in my house; 

Divorced old barren Reason from my Bed 

And took the Daughter of the Vine to Spouse. 

I sent my Soul through the Invisible, 

Some letter of that After-life to spell: 

And by and by my Soul return'd to me, 

And answer'd "I Myself am Heav'n and Hell:" 

So, some years after reading his poems, I was talking to a friend, and he told me that Omar Khayyam was also a very important mathematician and astronomer. I found out that Persia in those days was a decadent empire, and the detachment from religious beliefs and the immersion in hedonism was a common escape for intellectuals. 

It can be that, the reason of such decadent verses, but I also think that his point of view as a scientist could have influenced his world view. The observation of stars, the regular movements of planets and the moon, can be thought of as a divine signature, but the study of logic can only lead to a point where no explanation can be found. You want to explain the final cause of everything but that cannot be found with observation or logic. So that's why he divorced from Reason, and married the Daughter of Vine. He couldn't explain why we are here, the meaning of life, and some of his verses show this inability to explain the most asked questions in human history. Questions we have no explanation even now.

Earthquake music

The other they we had a visit from a high school student that wanted to study maths or physics. We were explaining him what we did. He said that he also liked music, so we came up with some music related entertainment: fractal music. 

There are some authors that play with physical models and convert it into music, like Xenakis, for example. We played a little bit, and we ended up creating a program that translates fractals into music. Here you have the result: 

We used the MATLAB scripts written by Ken Schutte, but with fractals. This fractal in particular has a Hurst exponent of 0.8. The timing is equal, but we can choose random or fractal timing too.

Then, at home, I thought that it would be cool to create a movie with an earthquake. The result is this: 

This is what the Lorca (Spain) 2011 earthquake would sound like. The sound is scaled, so this is not a real sound. But I thought it would be nice to see how the earthquake signal could be translated into sound. I am not the first one to think about it. You can look at NOAA , for example. 

Here you have the sound of seismic ambient noise, recorded at Almería (Spain), for a seismic campaign: 

The sound is also scaled.They are very different, don´t you think?

The movies and sounds are nothing special (I don´t think I will win an oscar), but I had fun making them. And I think that´s all that matter. 

Are we living in a simulation?

I was reading the other day about the Simulation Argument here. I found it very interesting.

 The question raised is: are we living in a simulation?

The concept was popularised by the film The Matrix. Are we living in a virtual world? The Simulation Argument states: One of these are true: (1) the human species is very likely to go extinct before reaching a “posthuman” stage; (2) any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof); (3) we are almost certainly living in a computer simulation.

The argument is very simple: if there is any stage when humanity can simulate the human mind, and a lot of people are dedicated to simulate human beings, it is very probable that we are a simulation.

There are many subtleties about this postulate. The first question would be: can we simulate a human mind? If we assume that our minds are only the result of our physical substrate, then, in theory, we could do that. I say in theory because the technological problems of simulating all our neurons and their interactions would be very difficult.

But, for the sake of the argument, let´s say that we can simulate a human brain. Then, what about the universe in which it is immersed? We should simulate it too in order for that mind not to realise it is not a human being. That seems a little bit difficult. In the first place we should know everything about our universe. Could we create a different universe that looks like ours without knowing our own?

Well, the question of the universe is important if we want to create a human mind that is free in its own universe. We could just create a life sequence where the individual thinks he or she is free, but that in fact is deterministic.

It that sense, there are different kind of simulations: the life of this person is completely simulated, or only their psychology is simulated, and they are actually free; you can simulate a population, instead of only one person, both with freedom or without it. In a free simulation with a population their minds would interact and could “discover” their world as we do, that is, creating an objective reality that is made of all the common things they experience. They could develop a science in their world.

But, would they realise they are simulations? Let´s go back to the simulation of the universe. Could we simulate our universe? There are different positions: some people like Wolfram believe that we are just an algorithm, a Cellular Automata, which rules we do not know yet. But, knowing those rules and the initial conditions, we would be able to simulate our universe. However, other people think that it is not possible (see my post “What does the Universe know?”). Would we reach a point where we find the “flaws” of The Matrix?

This also creates a problem for those who think about the simulation of specific lives, such as Einstein´s life, or Napoleon´s life, etc. The Simulation Argument would give the opportunity of having a holyday by living the life of some important person in History. For that we should know, as we said, the rules and initial conditions of the universe.

If the rules were known, the initial conditions could be calculated by an optimization method, by knowing the state of the universe at one point. That is not so “difficult”: we could use the holographic principle and measure the state of the universe in its boundary only.

And that is only if the universe is deterministic. What if it is not so? Could we simulate our History? Perhaps we would be able to create Histories that look alike, but with different details (like in The Foundation series, by Asimov). Or perhaps not.

The Simulation Argument raises a lot of questions, that I think are very important. The main one is: can a human mind be simulated? Well, since I believe that we are only a bunch of atoms (or quarks, or whatever), I think that “in theory” that could be possible. Only we do not have the technology necessary to do so. Then, would it be ethic? That is another question.

More questions come to my mind, but I think I will leave it there. I hope you found it interesting too.

Optimization methods based on nature

Today I will write about optimization methods based on biological concepts. But first I will introduce the concept of optimization, and why it is so important in science.

Science is based upon experiments and theories that explain the data obtained in those experiments. Those data are the observations we have of a particular phenomenon. They are measures of some physical quantity we can extract from the experiment. In general we try to correlate some physical quantities with others in the way of an equation, or physical law. These equations are composed by the observations and some parameters. For example, think about a straight line. The equation (physical law) would be: y=ax+b. That means that we have a series of measures of x and y, and we want to calculate the optimum parameters a and b that define that line.

It is important to note that we assume a physical law for the relationship between x and y. That is, our physical law would say that they are linearly correlated. The a and b parameters will describe other quantities related to the studied phenomenon. If our physical law is a straight line (more generally, if we have a linear problem), we have a procedure that calculates the parameters that define our data: the linear least squares fit. However, in Physics, and particularly in Geophysics, the equations are nonlinear. This poses a problem, because we usually want to find the models (parameters) that minimize the error between data and the equation (that is, linear least squares in linear fitting).

If the number of possible solutions is high (as usual in real problems), we cannot perform a systematic search. We need some algorithm that searches only in a small portion of the parameters' space. There exists a number of algorithms that can do that. Here we will talk about some of the ones that are based on biological concepts. As examples, I will explain Genetic Algorithms, Ant colonies, and Particle Swarm.

Genetic Algorithms are based on the idea that a solution (a set of parameters) is represented by a chromosome (each gene representing each parameter), and that a population evolves to the optimum (the best fit) following Darwinian rules. The operators that make the population evolve are the selection of the parents to be crossed, the crossover of the two solutions, mutation, and replacement of the individuals in the population. Each solution (chromosome) has a value that represents how well fitted the data are to said solution, and the individuals with better solutions have more opportunities to mate and therefore to pass on their information from generation to generation.

Another example of fitting algorithm is the Ant Colony Algorithm. In it, each agent (ant) creates a path that represents the solution. At first each ant will be wandering randomly, like in the natural world. Each trail will be marked with a pheromone trail. This makes the path attractive to other ants. If more ants follow this trail, it will be marked as well fitted. However, the pheromone trial evaporates with time. In that way, we will not end up always in the paths chosen by the first ants. The trail with higher pheromone content will be the solution.

Finally, we will explain the Particle Swarm Algorithm. In it, each solution is wandering through the parameters' space with certain velocity. This algorithm mimics the way a flock or a swarm behaves in the real world. In each iteration, each individual adjusts its movement along the parameters' space in function of the solution that is the leader in that iteration. The leader is, of course, the best fitted of all. The swarm as a whole changes its movement in function of the best solution found previously. Of course, there is a random component (in velocity) that prevents the search to end up in a local minimum.

It is very interesting to see how Biology can help in a purely mathematical problem like the optimization of functions. Of course, there are more Biology-based algorithms out there. I only mentioned the ones I know better. See Swarm Intelligence for other methods.

Kevin Bacon is the center of the Universe

Watching TV in UK I came across a commercial with Kevin Bacon making fun of his many connections. What started as a game (see here for a thorough explanation), made the theory of complex networks known for the general public. The Bacon number is the lower number of links an actor has with Kevin Bacon. For example, if an actor (or actress) played a role in the same film Bacon did, he or she has Bacon number 1. If he or she played a role with somebody who directly played with Kevin Bacon, he/she has Bacon number 2, and so on. Of course, Bacon has 0 as his Bacon number. You can check the Bacon number for all the actors and actresses in the Internet Movie Database (IMDb) in The Oracle of Bacon. It is striking to see that the higher number is only 9. So, it seems that everybody is very closely linked to Kevin Bacon. Thus the joke of he being the center of the Universe.

This concept is similar to the 6 degrees of separation popularized by the experiment of Milgram. This experiment was like this: you pick two people at random in the world (Milgram did it for the USA), and send a parcel to one of them, telling them that he or she has to reach the other person by direct links, sending the parcel to a known person that could be 'closer' (who knew the other person, or somebody who could know them) to the final receiver. He discovered that, in average, there were only 6 links of separation between sender and receiver. He called that the 'Small world' phenomenon. 

This closeness between the individuals in the world can also be seen in the natural world. The trophic chains, cellular biology, evolutionary relationships, internet, scientific citation networks, even seismicity (I have published some papers on this particular field). In complex networks, the most important concept is not the individual, but the nature of its relationship with the other individuals.

Complex networks studies are based on the mathematical graph theory. And graph theory started as a game too, proposed by the eminent mathematician Leonhard Euler: can you find a path that crosses all the Bridges of Königsberg only once? Euler solved it inventing graph theory in the process. 

It is important to analyze the relationships in a network. For example, it tells you who eats who in a trophic chain, or the main hubs in the internet. Why redundancy is important in the networks, so that a specific attack does not destroy the whole system. And many other applications. You can look it up in the internet, another network. Who said that playing games is not productive? 

As a curiosity, in the scientific publication network (who published with whom) the center of the Universe is Erdős. Even some people has Erdős-Bacon number!