The Physics of Wall Street: a brief History of Predicting the Unpredictable

The Prediction Company 
•
151
ond generation of solutions, which are allowed to compete again. And 
so on. It’s survival of the fittest, where fitness is determined by some 
standard of optimality, such as how well an experiment would work 
under a given set of conditions. It turns out that in many cases, genetic 
algorithms find optimal or nearly optimal solutions to difficult physics 
problems very quickly.
Physicists in general, and farmer and Packard especially, have 
developed many kinds of optimization algorithms that, by different 
means, accomplish the same goals as genetic algorithms, with different 
algorithms carefully tailored to different tasks. these algorithms are 
pattern sleuths: they comb through data, testing millions of models at 
a time, searching for predictive signals.
But there’s nothing special about physics problems, as far as these 
algorithms are concerned. they can be applied to any number of dif-
ferent areas — including finance. Suppose you have discovered some 
strange statistical behavior relating the currency market for Japanese 
yen with the market for rice futures. It might seem sufficient to ob-
serve that if yen go up, then so do rice futures prices. You would then 
buy rice futures whenever you noticed yen ticking upward. or else, 
suppose you have an idea for a possible pairs trade, such as with Pepsi 
and coca-cola.
notice that in these cases, the basic strategy is clear. But there are 
all sorts of possibilities compatible with that basic strategy. to be per-
fectly scientific about the problem, you would want to figure out just 
how closely correlated the two prices are, and whether the degree of 
correlation varies with other market conditions. You would also want 
to think about how much rice to buy and how to time your purchase 
to be maximally certain that yen were really going up. But trying to 
come up with a way of relating all of these variables in an optimal 
way from scratch would be an enormously time-consuming and dif-
ficult process, and you could never be sure you’d gotten it right. In 
the meantime, your opportunity would pass. But if you used a genetic 
algorithm, you could let thousands of closely related models and trad-
ing strategies based on the supposed connection between yen and rice 
compete with one another. You would soon arrive at an optimal, or 
nearly optimal, strategy. this is a variety of forecasting, but it doesn’t 

---

require you to come up with some complete chaos-theoretic descrip-
tion of markets. It’s much more piecemeal than that.
Another one of the Prediction company’s ideas was to use many 
different models at once, each based on different simplified assump-
tions about the statistical properties of different assets. farmer and 
Packard developed algorithms that allowed the different models to 
“vote” on trades — and then they adopted a strategy only if their mod-
els were able to form a consensus that it would likely be successful. 
voting may not sound as if it has anything to do with physics, but the 
underlying idea comes right from farmer’s and Packard’s days study-
ing complex systems. Allowing many different models to vote identi-
fies which trading strategies are robust, in the sense that they aren’t 
sensitive to the special details of a particular model. there is a close 
connection between searching for robust strategies and searching for 
attractors in a complex system, since attractors are independent of ini-
tial conditions.
this kind of modeling, where one uses algorithmic methods to 
identify optimal strategies, is often called “black box” modeling in the 
financial industry. Black box models are very different from models 
like Black-Scholes and its predecessors, whose inner workings are not 
only transparent but often provide deep insights into why the mod-
els (should) work. Black box models are much more opaque, and as 
a result they are often scarier, especially to people who don’t under-
stand where they come from or why they should be trusted. Black box 
models were occasionally used before the Prediction company came 
along, but the Prediction company was one of the very first compa-
nies to build an entire business model based on them. It was a whole 
new way of thinking about trading.
Almost a year into the new company, the senior partners weren’t 
making any money. An investment firm needs something to invest. 
farmer, Packard, and McGill could only go so long without bringing 
home paychecks — and to make matters worse, they had been funding 
their team of graduate students and computer hackers out of their own 
pockets for eight months, since everyone had taken up residence at the 
152 
•
t h e p h y s i c s o f wa l l s t r e e t

---