The Physics of Wall Street: a brief History of Predicting the Unpredictable

The Prediction Company 
•
149
investment firm on the backs of their personal checking accounts for-
ever. As the company’s one-year anniversary approached, in March of 
1992, the pressure was on to find a deal.
It is tempting to say that farmer, Packard, and their Prediction com-
pany collaborators “used chaos theory to predict the markets” or 
something along those lines. In fact, this is how their enterprise is usu-
ally characterized. But that isn’t quite right. farmer and Packard didn’t 
use chaos theory as a meteorologist or a physicist might. they didn’t 
do things such as attempt to find the fractal geometry underlying mar-
kets, or derive the deterministic laws that govern financial systems.
Instead, the fifteen years that farmer and Packard spent working on 
chaos theory gave them an unprecedented (by 1991 standards) under-
standing of how complex systems work, and the ability to use comput-
ers and mathematics in ways that someone trained in economics (or 
even in most areas of physics) would never have imagined possible. 
their experience with chaos theory helped them appreciate how regu-
lar patterns — patterns with real predictive power — could be masked 
by the appearance of randomness. their experience also showed them 
how to apply the right statistical measures to identify truly predictive 
patterns, how to test data against their models of market behavior, and 
finally how to figure out when those models were no longer doing their 
job. they were at ease with the statistical properties of fat-tailed dis-
tributions and wild randomness, which are characteristic of complex 
systems in physics as well as financial markets. this meant that they 
could easily apply some of Mandelbrot’s ideas for risk management in 
ways that people with more traditional economics training could not.
As far as the Prediction company was concerned, markets might 
be chaotic, or not. there might be various degrees of randomness in 
market behavior. Markets might be governed by simple laws, or by 
enormously complicated laws, or by laws that changed so fast that 
they might as well not have been there at all. What the Predictors were 
doing, rather, was trying to extract small amounts of information from 
a great deal of noise. It was a search for regularities of the same sort 
that lots of investors look for: how markets react to economic news 

---

like interest rates or employment numbers, how changes in one mar-
ket manifest themselves in others, how the performances of different 
industries are intertwined.
one strategy they used was something called statistical arbitrage, 
which works by betting that certain statistical properties of stocks will 
tend to return even if they disappear briefly. the classic example is 
pairs trading. Pairs trading works by observing that some companies’ 
stock prices are usually closely correlated. consider Pepsi and coca-
cola. virtually any news that isn’t company-specific is likely to affect 
Pepsi’s products in just the same way as coca-cola’s, which means 
that the two stock prices usually track one another. But changes in 
the two companies’ prices don’t always occur simultaneously, so some-
times the prices get out of whack compared to their long-term behav-
ior. If Pepsi goes up a little bit but coca-cola doesn’t, upsetting the 
usual relationship, you buy coca-cola and sell Pepsi because you have 
good reason to think that the two prices will soon revert to normal. 
farmer and Packard didn’t come up with pairs trading — it was largely 
pioneered in the 1980s at Morgan Stanley, by an astrophysicist named 
nunzio tartaglia and a computer scientist named Gerry Bamberger
— but they did bring a new level of rigor and sophistication to the 
identification and testing of the statistical relationships underlying the 
strategy.
this sophistication was purely a function of the tools that farmer 
and Packard were able to import from their days in physics. for in-
stance, as a physicist, Packard was at the very forefront of research in 
a variety of computer programs known as genetic algorithms. (An al-
gorithm is just a set of instructions that can be used to solve a particu-
lar problem.) Suppose you are trying to identify the ideal conditions 
under which to perform some experiment. A traditional approach 
might involve a long search for the perfect answer. this could take 
many forms, but it would be a direct attack. Genetic algorithms, on 
the other hand, approach such problems indirectly. You start with a 
whole bunch of would-be solutions, a wide variety of possible experi-
mental configurations, say, which then compete with one another, like 
animals vying for resources. the most successful possible solutions 
are then broken up and recombined in novel ways to produce a sec-
150 
•
t h e p h y s i c s o f wa l l s t r e e t

---