On Hayek's "The Use of Knowledge in Society" (1945)
I think this paper is a libertarian touchstone because it doesn't get into a lot of specifics — which is always an issue for libertarianism.
Now look — I am fully on board with the idea of a market as an information processing system, and that Hayek is key in the development of that idea. However, his essay The Use of Knowledge in Society [pdf] is at best out of date and should be relegated to the history of economic thought. There are a lot of hints at something that could make sense given our modern knowledge of information theory and communication, but nothing that does on close examination — only assertion and speculation.
To use a physics analogy, Hayek is Bohr, not Heisenberg. The general thrust of what would become quantum mechanics was pioneered by Bohr, but “old quantum theory” (as it is referred to these days) was wrong despite getting the leading order energy levels of the Hydrogen atom correct. It would take Heisenberg (and Schrodinger, and Dirac, and ...) to nail down quantum mechanics. However, it seems very few people have genuinely taken Hayek as a new starting point for a re-invigoration of the field.
The state space
Hayek keenly identifies a huge source of invisible information with regard to any feasible economic system:
We need to remember only how much we have to learn in any occupation after we have completed our theoretical training, how big a part of our working life we spend learning particular jobs, and how valuable an asset in all walks of life is knowledge of people, of local conditions, and of special circumstances. To know of and put to use a machine not fully employed, or somebody’s skill which could be better utilized, or to be aware of a surplus stock which can be drawn upon during an interruption of supplies, is socially quite as useful as the knowledge of better alternative techniques. And the shipper who earns his living from using otherwise empty or half-filled journeys of tramp-steamers, or the estate agent whose whole knowledge is almost exclusively one of temporary opportunities, or the arbitrageur who gains from local differences of commodity prices, are all performing eminently useful functions based on special knowledge of circumstances of the fleeting moment not known to others. [...] the method by which such knowledge can be made as widely available as possible is precisely the problem to which we have to find an answer …
Part of the reason this information will always be invisible is change:
It is, perhaps, worth stressing that economic problems arise always and only in consequence of change. [...] In a competitive industry at any rate—and such an industry alone can serve as a test—the task of keeping cost from rising requires constant struggle, absorbing a great part of the energy of the manager.
I don't think anyone today would disagree with this — however, I do not believe anyone in 1945 would have disagreed with this either. Even the Soviet central planners knew the problem of controlling every decision was not feasible (hence the entire concept of “the commanding heights”), so Hayek is setting up a strawman. Regardless, there are so many small decisions made in an economy there is no feasible way to collect all of the information required. That is to say the economic state space is both large and dynamic on a scale too short to survey the entire space. In physics, this same problem is addressed by an emergent theory called thermodynamics — it depends not just on the “law of large numbers”, but emergent concepts like entropy and temperature.
Hayek says this, like five, is right out:
The comparative stability of the aggregates cannot, however, be accounted for—as the statisticians occasionally seem to be inclined to do—by the “law of large numbers” or the mutual compensation of random changes. The number of elements with which we have to deal is not large enough for such accidental forces to produce stability.
No evidence is presented for this assertion, and the fact that we haven't yet produced the macroeconomic theory for social systems analogous to thermodynamics — a “statistical economics” — does not mean it does not exist. In fact, I’ve been quite successful modeling the unemployment rate via exactly such an analog that I’ll get into more later.
It is the movement of this fine-grained information throughout the economic state space that Hayek says is crucial to understanding economics:
The various ways in which the knowledge on which people base their plans is communicated to them is the crucial problem for any theory explaining the economic process, and the problem of what is the best way of utilizing knowledge initially dispersed among all the people is at least one of the main problems of economic policy—or of designing an efficient economic system.
This the primary place where Hayek both hints at the real problem, but is misguided about the solution. The issue is that the reason this information is invisible at the level of the macroeconomy is because of its scale, not because it is not communicated. In a complex modern economy, it is simply too large a state space to be communicated. It has millions, if not billions, of dimensions at the agent scale. We will never figure out how it is communicated and because we cannot actually compute a centrally planned solution given an objective function we will never be able to prove optimality. As Cosma Shalizi puts it:
That planning is not a viable alternative to capitalism (as opposed to a tool within it) should disturb even capitalism’s most ardent partisans. It means that their system faces no competition, nor even any plausible threat of competition.
It is this misunderstanding of the problem he himself pointed out that leads to his completely baseless assertions that the impossible communication is in fact possible and that he knows how to do it.
The information problem
Hayek proposes that the price mechanism provides the solution to the information problem he identifies:
It is in this connection that what I have called the “economic calculus” proper helps us, at least by analogy, to see how this problem can be solved, and in fact is being solved, by the price system.
However, there is no particular evidence it does so except circumstantial. “See, incorporating price information into plans is also decentralized,” he says:
Fundamentally, in a system in which the knowledge of the relevant facts is dispersed among many people, prices can act to coördinate the separate actions of different people in the same way as subjective values help the individual to coördinate the parts of his plan.
Price decisions are decentralized, and the invisible knowledge is decentralized, therefore price decisions must communicate that knowledge. QED, right? But a few paragraphs earlier Hayek tells us:
This is, perhaps, also the point where I should briefly mention the fact that the sort of knowledge with which I have been concerned is knowledge of the kind which by its nature cannot enter into statistics and therefore cannot be conveyed to any central authority in statistical form. The statistics which such a central authority would have to use would have to be arrived at precisely by abstracting from minor differences between the things, by lumping together, as resources of one kind, items which differ as regards location, quality, and other particulars, in a way which may be very significant for the specific decision. It follows from this that central planning based on statistical information by its nature cannot take direct account of these circumstances of time and place and that the central planner will have to find some way or other in which the decisions depending on them can be left to the “man on the spot.”
This is exactly the same problem with prices — information is necessarily lost in aggregation whether into a statistic or a price. Hayek says as much:
Assume that somewhere in the world a new opportunity for the use of some raw material, say, tin, has arisen, or that one of the sources of supply of tin has been eliminated. It does not matter for our purpose—and it is very significant that it does not matter—which of these two causes has made tin more scarce. All that the users of tin need to know is that some of the tin they used to consume is now more profitably employed elsewhere and that, in consequence, they must economize tin. There is no need for the great majority of them even to know where the more urgent need has arisen, or in favor of what other needs they ought to husband the supply. [...] without the great majority of those instrumental in bringing about these substitutions knowing anything at all about the original cause of these changes.
The information lost in aggregating via statistics for the central planner in a large matrix calculation is the same scale of the information lost in aggregating prices since both yield comparable sets of numbers — the N prices are no better at capturing the M-dimensional invisible information behind each price than the N² matrix elements when M^N >> N. So when Hayek says
We must look at the price system as such a mechanism for communicating information if we want to understand its real function ...
we reach an impasse. Per Hayek, the price system is not communicating all the information, but rather some kind of coarse-grained aggregate of it. There is no evidence presented that the coarse graining that occurs is the optimal coarse graining — it is merely asserted that it is "relevant" information. However, a small set of real numbers cannot communicate the total amount of information in the multi-dimensional state space in a complex economy. Per Shannon, the proposed channel bandwidth is simply not big enough — and it never will be. And per Shalizi above we can never test optimality because the matrix problem is just too big to be solvable without a heroic dose of computational resources.
The fact that information is lost either way is why information equilibrium places both kinds of aggregation — prices (e.g. the price of bread) and aggregated statistics that act as abstract prices (e.g. the unemployment rate) — in the same framework1. There is merely a difference in how that information is collected2. And while there is no way to communicate that M-dimensional state space of invisible information across the entire network, there is a way to measure its flow.
Like the price of bread, the unemployment rate is a detector of information flow between supply and demand. Unfilled demand states (for bread or employees) need matching with available supply states (unused bread ovens, unemployed people) — and that matching requires detecting information flow, not information aggregation.
Millions of people making millions of economic decisions every second results in trillions of bits of information about the economic state space3 — and there is a real system where the full information about the state space could not possibly be communicated through a single dimensional real-number channel but nevertheless the ability to measure the flow allows that state space to be copied with extremely high fidelity.
That information flow — the difference in information content between two state spaces — can be measured by a one dimensional number which in turn can be used to reduce those differences in information to near zero. It is the Kullback-Leibler (KL) divergence used in the original implementation [pdf] of Generative Adversarial Networks (GAN) in machine learning. The KL divergence measures the differences between probability distributions, and the machine learning algorithm is able to use that "warmer, colder" detection of information flow to bring random noise into equilibrium with a Monet. GANs can be used as a “proof of concept” for how a price measures the differences (i.e. flow) between the (spatial, temporal) distributions of demand for bread and the supply of it and uses that information to match them.
Hayek first tells us there is a bunch of information that goes missing in statistical aggregates that needs to be communicated, but then turns around and says prices can communicate that information despite also dropping nearly all of it on the floor. “Don't worry,” he says. “It keeps the relevant information! Trust me!”
The initial insight is there — that there is an immeasurable amount of fine-grained information that is fundamentally inaccessible at the level of the macroeconomy due to the scale and dynamic nature of it. However, the suggestion prices magically capture the right information is no different than the assertion that planning the commanding heights is sufficient to run an economy. Saying that prices aggregate or communicate that fine-grained information is fundamentally wrong; saying that they detect the flow of that fine-grained information is plausible and there are real world examples we can point to. Regardless, the fine-grained information is still invisible at the macro scale.
I've used that idea of detecting information flow with both real-world and abstract prices relatively successfully — consolidating the ideas discussed here in the information equilibrium framework as applied to economics (see here for materials for my talk at UW econ including a Twitter talk). I've used information equilibrium to derive supply and demand and forecast the unemployment rate more accurately than several other models.
Information equilibrium or the GAN analogy also give us some insight to possible failure modes (non-ideal information transfer due to agents bunching up in the state space aka “the Helvetica scenario”), while Hayek only suggests the only worry is not enough price medicine (“a function which, of course, it fulfils less perfectly as prices grow more rigid”).
Your mileage may vary with regard to the usefulness of information equilibrium. However my primary point in using it as an example is that it is a concrete economic theory built on information flow in a social system. In contrast, there is no concreteness to Hayek's arguments even where they are not self-contradictory. It's almost as if he constructed the essay having already concluded the solution was the price system and set about collecting various assertions that supported that conclusion. Indeed The Use of Knowledge in Society came at a time when the conflict between central planning and decentralized prices was seeming to grant the edge to the former given the failure of the Great Depression and the rather stark turnaround in the Soviet Union from relative poverty to global superpower during World War II.
I think Hayek's essay is relevant background to modern economic thought, having influenced the field (mostly later on, via Milton Friedman). However it is probably better to know of the essay than to know the contents of the essay itself.
Writing
This is basically an aside; you can stop at the section above if your interest is just in technical content. However, I've been focusing on writing recently as part of my new science fiction ventures (both production and consumption) and can't help but read Hayek's essay purely as writing — which is awful. It is awful because it is purportedly a technical essay on a technical subject and so should be judged on clarity instead of, say, mellifluousness. Even then the essay conveys a kind of aristocratic detachment of someone who just enjoys hearing themselves talk:
"Though the problem with which I want primarily to deal in this paper is the problem of a rational economic organization, I shall in its course be led again and again to point to its close connections with certain methodological questions. Many of the points I wish to make are indeed conclusions toward which diverse paths of reasoning have unexpectedly converged."
TL;DR: Multiple lines of reasoning have led me to conclusions about rational economic organization.
Note that Hayek doesn’t really say what those diverse paths are through the course of the essay. It’s just a kind of “I’ve been thinking about this for awhile; trust me” vibe. And then there’s this:
"The various ways in which the knowledge on which people base their plans is communicated to them is the crucial problem for any theory explaining the economic process, and the problem of what is the best way of utilizing knowledge initially dispersed among all the people is at least one of the main problems of economic policy—or of designing an efficient economic system."
TL;DR: The main problem in economics is understanding how dispersed knowledge required for planning is communicated, and the main problem of economic policy is finding the best way to use that knowledge.
This is supposed to be a genuine statement of the thesis.
Certainly, when one is stumbling in the dark around a new idea the language is not always clear. I do think Hayek was genuinely stumbling on to a new idea relevant in the nascent information age — one that germinated too early, before the advent of information theory4.
I had the great joy of discovering someone who has both read and understood that I was making this exact point:
That guy [Jason Smith] is just confused. He doesn't even acknowledge that the price has to be paid. In his model, there is no difference between a price that has to be paid and one that doesn't have to be paid. → there is no concept of truthful revelation.
However, an important piece of invisible information is where transactions do not occur — the boundary of the economic state space. Therefore the price being paid cannot be the only information flow the price is detecting.
A market is a way to operate a decentralized asynchronous survey compared to e.g. the centralized synchronous BLS survey of households during the reference week to measure the unemployment rate. Similarly, prediction markets are basically asynchronous polls.
In particular, its edges where transactions do not happen — an even more invisible source of information!
Shannon's paper came out in 1948, a few years after Hayek's essay in 1945.