From: John Conover <john@email.johncon.com>
Subject: A leap of faith with fractal analysis ...
Date: Fri, 16 Jan 1998 19:26:12 -0800
"... all this is civilization, the Sumerian God of Wisdom tells
Inanna of Uruk, who will take it back to her city, and thence give
it to the world. And, if you wish its ... [enigmatic]
... benefits, he goes on, you must take all its qualities,
...[including its disadvantages] ... without argument. All these
things I will give you, holy Inannna, but once taken, there can be
no dispute, and they can not be given back."
-Unknown poet; Uruk, Sumeria, circa
3000BC. (From the epic of Gilgamesh,
transcribed by Sin-leqiuninni around
1300BC, interpreted by the
anthropologist, Michael Wood.)
The Gilgamesh is the tale of the invention of civilization by the
ancient Sumerians, somewhat before the fourth millenia,
BC. Interestingly, it contains a theme that will be echoed by poets
through the ages when describing the enigma of civilization-the
formula for a fractal.
As with any application of abstraction, the quality of the abstraction
has to be evaluated. Historically, at least in mathematics, this was
done by determining the scalability of a theory-in short, how far can
its precision be extrapolated. In the more robust theories, for
example the quantum mechanics, scalability is preserved over many
orders of magnitude.
Fractal analysis is usually used to describe the evolution of a
complex system that operates under uncertainty. Equity values over
time, industrial markets, and complex (eg., iterated game-theoretic,)
economic phenomena are examples.
An argument can be made that if equity values are a complex system
that operates under uncertainty, and so are industrial markets, then
it would be reasonable to assume that the GDP, also, is a complex
system that operates under uncertainty. This argument is based on the
fact that fractals, which describe such things, are
self-similar. (What this means is that a fractal is made up of smaller
fractals, which in turn, are made up of even smaller fractals, and so
on, without end.) And, if that is true, then we should be able to
argue that the same is true for all the GDP's in the history of
civilization.
Such an extrapolation should be viewed with scepticism. Although we
have very precise data for the financial markets in this century, (in
point of fact, the US equity market tickers are some of the most
precise data we have all of science,) we don't really have good data
on the ecology/GDP of the cultural history-and the proposal is to make
a "leap of faith" over nine orders of magnitude! (That is, see if we
can present arguments that relate the US equity market ticker, by the
second, to the history of civilization, by the century.)
It should be regarded as sloppy science to do things like this, but
since I have received several questions on the subject, I'll attempt
it-on the qualification that the analysis be viewed with a significant
degree of scepticism.
Here is the proposal. We need to take the data from the US Equity
markets, by the second, and extrapolate that data to by the day, and
then analyze the fractal probabilities of that data. Then, we need to
take the US GDP data, by the year, and extrapolate that data to by the
day, and do likewise. Then we need to look at the historical record of
cultures/civilizations, and scheme up a way of scaling that data, by
the day.
We need a metric to compare how accurate our extrapolated
probabilities are. For this, I propose that we use the Shannon
probability, based on the day as a time unit. (The Shannon probability
is the simply likelihood that the value for the next day in a data set
will increase.) The reason I propose this is because we do not have
accurate data on the GDP 's of cultures in the historical record-but
we do have reasonably good data on how long cultures persisted, (ie.,
the run length of cultures; the ancient Sumerians were pretty
meticulous with their accounting, as were the Egyptians, the Romans,
the Greeks, the Mayans in Central America, the Chinese Dynasties, as
were the European countries since the Renaissance, etc.) Granted, the
historical perspective is subject to interpretation, but its the only
data we have. We ask only that the interpretations be axiomatically
consistent.
Here is the strategy. We will look at the distribution of run-lengths,
(ie., how long cultures/civilizations persisted,) and see if it fits a
known distribution, within reason. If the distribution assembles
itself into a reciprocal of the square root of time, (time being
measured in centuries,) then this will give some, but not a lot, of
credibility that civilization/culture is a fractal system that
operates under uncertainty. Since the US is in its second century, we
can use the probability that the run-length will continue in the
second century to compute the Shannon probability for the entire
century, extrapolate it to the day, and compare it with the
extrapolate data for the US equity markets and GDP, for this century.
The argument here is that if civilizations/cultures are fractal
systems, then they should start, and for the most part, grow for two
centuries, on average, then decay back to zero for two centuries, on
average. During the first two centuries, since the GDP/ecology is
expanding/growing, the Shannon probability will be greater than one
half. In the last two centuries, on average, the Shannon probability
will decrease, and be less than one half-being exactly 0.5, on
average, over the a average of four centuries.
Do we see a distribution of the run lengths of civilizations that is a
reciprocal of the square root of time? The data set is limited, (about
400 samples, far too small for analytical comfort,) but we do see
something that appears to be similar to what we expected, (The ancient
Egyptians won the duration lottery-about 30 centuries-which would have
a theoretical probability of about 18.3%. Note that we should have
found more, but our data set only extends only over a time interval of
60 centuries-about twice the duration of the ancient Egyptian
civilization.) The Romans were next, at 8 centuries, with most lasting
one century, or less. The average is about 3.6 centuries-which is
close to the 4 we expected. (Note, this type of distribution could not
be made by a system that operated on simpler statistical mechanisms; a
pure Gaussian distribution doesn't fit, since virtually all of the
distribution would be beyond the left three sigma limit of the curve,
instead of at 4 centuries; a random mechanism would have given a
"splatter" that have produced an equal probability of a civilization
lasting one century, or thirty. Not compelling arguments, but
interesting, nonetheless.)
So, it would appear that, although our data set size is too small to
justify any optimism, that, at least there is some, limited,
circumstantial evidence that culture/civilization evolves as a complex
system that operates under uncertainty.
Performing our "leap of faith," and calculating the Shannon
probability for the ecological growth in second century of a "typical"
culture/civilization, by assuming that the average growth in the
second century, will be reflected, on average, to all the constituent
fractals that make up the cultural/civilization fractal:
G(t) = sqrt (t)
where t is time, measured in centuries-and taking the derivative with
respect to time to get the marginal growth, M:
dG(t) 1
----- = ------------
dt 2 * sqrt (t)
and calculating this value at 1.5 centuries, (since we will be
comparing this value to the value for equity markets obtained in the
last half of the second century in the US,) and 2 centuries:
dG(t) 1
----- = -------------- = 0.4082483
dt 2 * sqrt (1.5)
dG(t) 1
----- = ------------ = 0.3535534
dt 2 * sqrt (2)
and average the two to get M = 0.3819008.
This means that there should have been a 38% growth, on the average,
in the second century of cultures that lasted longer than two
centuries. I now need to solve for the Shannon probability, by the
century, that would give this growth, ie., solve the equation:
ln (1 + M) = 1 + P ln (P) + (1 - P) ln (1 - P)
2 2 2
which gives P = 0.87859, for the second century of an average
culture/civilization. I now need to convert this value from a century
value, to a daily value. Knowing that:
R + 1
P = -----
2
where R is the root mean square of the century fluctuations:
0.75718 + 1
P = -----------
2
and, using since the daily fluctuations add root mean square to get
the century fluctuations, and there are 25,700 business days in a
century, (because I will be comparing it to equity data, taken by the
business day,) to get the root mean square of the daily fluctuations,
r:
r = 0.75718 / sqrt (25700) = 0.004723159
and the daily Shannon probability, p, would be:
r + 1
p = ----- = 0.5023616
2
For the analysis of the US GDP, the data came from
http://www.doc.gov/BudgetFY97/index.html, and is by year, 1940 to
1995, inclusive. Again, the data set is far too small. There is an
issue since the equity market numbers are not adjusted for inflation,
(and it is not clear what term "duration of civilization, adjusted for
inflation" would mean,) so I will derive the fractal statistics for
the US GDP in both real and non-adjusted dollars.
The average of the marginal increments of the US GDP is 0.082307, and
the root mean square of the marginal increments is 0.095762. From
this, the non-adjusted Shannon probability, by year, can be
calculated:
0.082307
-------- + 1
0.095762
P = ------------ = 0.9297477
2
and, since the daily fluctuations add root mean square to get the
annual fluctuations, and there are 257 business days in a year,
(because I will be comparing it to equity data, taken by the business
day) I need to calculate the Shannon probability of the non-inflation
adjusted US GDP by day, p:
0.082307
--------
0.095762 + 1
----------
sqrt (257) 0.05799607 + 1
p = -------------- = -------------- = 0.5268069
2 2
Since there are no reliable inflation numbers, (including the CPI,) by
the year, I will use the deflation numbers provided by the CBO, (which
are a running exponential,) and strike a curve through the inflation
adjusted GDP, calculate the growth, to derive the Shannon probability
for the inflation adjusted US GDP. In 1940 the inflation adjusted US
GDP was 831.735 billion. In 1995, it was 5,438.699 billion-a factor of
growth of 6.53898 in 55 years. Solving:
55
(x) = 6.53898
or:
55 * ln (x) = ln (6.53898)
and:
ln (6.53898)
------------
55
x = e = 1.034731
and solving the equation:
ln (x) = 1 + P ln (P) + (1 - P) ln (1 - P)
2 2 2
to get the Shannon probability of the inflation adjusted US GDP, by
the year, which is P = 0.6299. Again, using the fact that the daily
fluctuations add root mean square to get the annual fluctuations, and
there are 257 business days in a year, (because I will be comparing it
to equity data, taken by the business day) I need to calculate the
Shannon probability of the inflation adjusted US GDP by day, p:
R + 1
P = 0.6299 = -----
2
where R is the root mean square of the annual fluctuations, or:
R = 0.2598
and dividing R by the square root of 257 to get the root mean square
of the daily fluctuations in the inflation adjusted US GDP, r:
r = 0.01620588
and the daily Shannon probability, p, would be:
r + 1 0.01620588 + 1
p = ----- = -------------- = 0.5081029
2 2
For the analysis of the US equity markets, the data came from the
NYSE, by second, and was combined for all stocks, integrated and
sampled to directly convert the second ticker data to daily ticker
data, for the inclusive annual time interval, 1966 to 1996. The
average of the marginal increments of the NYSE is 0.000290, and the
0.00029
------- + 1
0.00851 0.03407756 + 1
p = ----------- = -------------- = 0.5170388
2 2
Here is how the Shannon probabilities, extrapolated over nine orders
of magnitude, compare:
non-inflation NYSE inflation adjusted duration of
adjusted US GDP US GDP civilizations
0.5268069 0.5170388 0.5081029 0.5023616
or about 0.5135776 +/- 2.33%
Which, depending on one's point of view, are reasonably close. We now
have to ask the question, "how reliable are these probabilities?" It
turns out that traditional statistical estimation techniques let us
down in fractal analysis, (we are hunting for the probability of a
probability that controls a system's evolution over a time interval.)
The assumption was that we were investigating the evolution of a
complex system that operates under uncertainty-ie., a fractal. But
there are whole families of fractals, and the paradigm assumption was
that we were dealing with the simplest fractal of all, fractional
Brownian motion. How valid was that assumption? What is the chance we
were misled and it is really some other kind of fractal? These are
complicated questions, and depend on the magnitude of the
probabilities under analysis.
The analysis of a system that is controlled by larger Shannon
probabilities can get by with smaller data sets. Smaller Shannon
probabilities require larger data set sizes, (for example, the
analysis of fractional Brownian motion with a Shannon probability of
exactly 50% requires an infinite data set size.) It is not trivial to
develop a methodology to evaluate how reliable our analysis is, but it
can be done. Without beleaguering the issue, here is how are
reliability stacks up for the metrics used in this analysis, where S
is the data set size, and P is the measured Shannon probability:
non-inflation NYSE inflation adjusted duration of
adjusted US GDP US GDP civilizations
S = 55 S = 46,260,00 S = 55 S = 1600
P = 0.9297477 P = 0.5170388 P = 0.6299 P = 0.87854
P' = 0.791092 P' = 0.515843 P' = 0.530797 P' = 0.841083
x = 85.08674% x = 99.76872% x = 84.26687% x = 95.73645%
Here, P' is the Shannon probability, after being adjusted for a finite
data set size, and x is the ratio of P' to P. Not surprisingly, the
mass of data provided by the NYSE provides the most reliable
probability. Also, not surprisingly, the GDP data is to be considered
the most unreliable, with the historical data being somewhere in
between, (don't forget that we still have to qualify the historical
data as being subject to interpretation.)
Our reliability estimate on these probabilities can only be regarded
as poor to fair, (which was a qualification that was made in the
beginning of the analysis.) However, we can state that there is some
evidence, albeit circumstantial, that we can extrapolate, or scale,
fractal analysis over nine orders of magnitude in the study of
ecology. The evidence, however, can not be regarded as compelling.
John
--
John Conover, john@email.johncon.com, http://www.johncon.com/