Help for the Statistics Page
1. Overview
The Statistics page is launched, in a new window, by clicking the "Statistics" link
on the Results page. A sample display is shown below.
The display is similar to the Summary Results display, except that the Statistics
page shows more data. When the page is first displayed, the Sort Order and Scaling
are taken from the "Preferences" bar
on the Results Page. The Statistics page also contains a control bar at the
top which lets you change the Sort Order and Scaling, and redisplay the page.
The data displayed on this page is described in the following sections.
2. "Format for Statistics" control bar
This control bar allows you to select the Sort Order and Scaling,
as follows:
- Sort Order (the order in which components are listed) can be:
- "Alphabetic", which is helpful if you want to look at a specific
system;
- "System profit/loss", which will list the systems with the
highest profit first; or
- "Individual profit/loss", which will list the
system-components with the highest profit first.
When the Sort Order is System or Individual, the sort values in the "Year 6"
column are highlighted.
- Scaling can be selected as either:
- "Percent", in which case the profit/loss and drawdown values are
displayed as a percentage of the Conservative or User account size; or
- "Dollars", in which case the profit/loss and drawdown values are
displayed in thousands of U.S.dollars ($K).
3. "Account Size $K" and "Weight" columns
These columns contain the same data as for the Summary Results display.
The Weight cannot be modified in the Statistics display.
4. "Months of Data /gap" column
This column shows two things: (1) The number of months of data
available; and (2) An indication of gaps in the data, if any. Gaps are
indicated as follows:
- A gap at the end of the data is displayed as, for example, "e2",
which would indicate that there is no data for the most recent 2 months.
A one-month end gap will be seen for components whose data has not yet
been entered for the most recent month. This is expected to be a common
occurrance.
- A gap in the middle of the data is displayed as, for example,
"g3", which would indicate that there is a 3-month gap somewhere
in the middle of the data. If there is more than one gap, the longest
will be indicated. You can view the Monthly Data display to
see where the gap(s) are. There should not be gaps in the middle of
the data, so this should seldom happen.
5. "Results Integrity" column
This column gives an indication of the integrity of the raw data. Unlike
Futures Truth, who run the trading systems on their own computers and produce their
own results, we get our results data from others, so we have no control over its
integrity. However, we do ask our data suppliers how they determine their results,
and this information is included in the 5-character "Results Integrity Code" as
follows:
- Character 1 indicates the basic nature of the data:
- A = Actual-trade data. This data is based on actual trades and is typically
certified, e.g. according to NFA standards. Typically this type of data is
supplied by a system-assist broker who also publishes the data on their
website.
- U = Uncertified-actual-trade data. This data is also based on actual
trades, but is not certified. Typically this type of data is supplied
privately by a system-assist broker or a system developer, but they do not
publish it on their website, so they have not gone to the trouble and
expense of certifying it.
- F = Hypothetical data for then-Fielded version of the system. When a
system has been released for a number of years, there may have been more
than one version. Results for the then-fielded version are obtained using
the version of the system that was in use at the time, so they are close to
actual trading as far as the system version is concerned.
Typically this type of data is supplied by the trading system developer.
- H = Hypothetical data for the current version of the system. For a new
system, this is the only type of hypothetical results available. For an
older system that has had more than one version, using the current version
might be less representative of the system's trading history, but might
provide a better estimate of the profitability and drawdown potential of
the current version.
Typically this type of data is supplied by the trading system developer.
- C = Computed mini, in which case the other characters are not used.
- Character 2 indicates the type of fills (for actual results) or whether slippage
and commission are included in the raw data (for hypothetical results):
- w = Worst-case results. Typically this data would be from a system-assist
broker who publishes the results on their website, and wants to ensure that
all their clients get at least as good a result as what appears on their
website.
- a = Average results. Many brokers use an averaging system whereby split
fills are averaged in such a way that all clients participating in a
particular trade receive nearly the same profit or loss.
- o = results from One particular account.
- s = estimated Slippage is included in the raw data (for hypothetical
results).
- c = estimated Slippage and Commission are included in the raw data (for
hypothetical results).
- p= Pieced data (for actual, or hypothetical results). This letter
indicates that there are two or more different types of data in the result
set. For example, the results for some months may be for the big contract,
while the results for other months are for the mini. The comment field (on
the Component Information page) should indicate how the data is pieced.
- Character 3 indicates how comprehensive the summary data used to compute the
results is:
- t = number of Trades is provided (for day-trader, swing-trader, or
position-trader).
- o = Open equity is included in the monthly profit or loss (for swing-trader
or position-trader).
- b = Both: the number of Trades is provided, and Open equity is included
in the monthly profit or loss (for swing-trader or position-trader).
- x = the number of trades is not given (for day-trader, swing-trader,
or position-trader).
- - = not applicable. This applies to a position-trader when summary data
is not used (i.e. when monthly results are calculated from the individual
trades), or to a computed mini.
- Character 4 indicates how readily available the results data is
(apart from Futures Examiner!):
- W = Website. The raw data is publicly available, typically on the broker's
or system vendor's website.
- P = Password required. The raw data is publicly available, but a username
and password is required. These can usually be obtained at no cost.
- R = by Request. The raw data can be obtained by requesting it from the
broker or vendor.
- N = Not available.
- Character 5 indicates whether data for individual trades is available:
- i = Individual trade data is available, typically including the date and
the entry and exit values. This type of data could be checked out by
an interested party to verify that the results are consistent with the
market action on the trade dates.
- x = Not available.
6. "DrawDown" column
This column shows the Maximum and Current end-of-month drawdown as for the Summary
Results display, except that the units will be either $K or percent, depending on
the current "Scaling" selection.
7. "Average trades /yr" column
This column shows the average number of trades per year, as for the Summary Results
display.
8. "Annual Profit or Loss, Max DrawDown" columns
These columns give, for each of the most recent six years, both the profit-or-loss
(PL) for that year, and the maximum end-of-month drawdown that occurred in that year.
"Year 1" is the oldest 12-month period, and "Year 6" means the most recent 12 months.
A
drawdown which begins in one 12-month period
(year) and continues into the following 12-month period is counted in both periods.
For example if the drawdown at the end of one year is $12K and the system
loses another $5K in the second year before it recovers, this would be
recorded as a $12K drawdown for the first year and a $17K drawdown for
the second year. Of course, if a subsequent drawdown in the second year
exceeds this amount, then the larger drawdown would be shown.
9. "Average" column
This column shows the average Annual Profit or Loss and the average maximum
annual drawdown. Both averages are taken over all the months of data available.
If there are any gaps in the data, the averages will be
affected accordingly (zero PL is used for a gap month). The methods in which
these numbers are calculated are described below.
For calculating average annual profit/loss, when there are 12 or less
months of data, the total profit/loss for those months is used as the average. I.e.
the profit/loss is not annualized. When there are greater than 12 months of
data, the total profit/loss is divided by the number of months and the result is
multiplied by 12. I.e. the total profit/loss is annualized.
Calculation of the average maximum annual drawdown is done somewhat
differently. Here, the maximum drawdowns for each year are simply averaged over the
number of years for which there is data, without regard for the number of
months of data. For example, when there are 16 months of data, there will be two
maximum drawdowns, and they are added together and the sum is divided by two.
10. "Sharpe" column
This column gives the
long-term and
one-year (short-term)
Sharpe
ratios. The Sharpe ratio is a measure of the
average monthly return in
excess of the risk-free return, relative to the
standard deviation of the
monthly returns. The Sharpe ratio is calculated as:
- (AvMPL - RfMPL) / StDevMPL, where
- AvMPL = average monthly PL of this SCD;
- RfMPL = Risk-free monthly PL for conservative or user account size
(based on 4%/year return); and
- StDevMPL = Standard Deviation of the monthly PLs for this SCD.
The risk-free return is the amount that could be earned
with zero risk, for example by investing in a Government bond.
We use 4% for the risk-free rate of return.
The standard deviation is a measure of how consistent
the values are in a set of numbers. For example in the set [5, 4, 6, 4, 6, 5] the
numbers are quite consistent so the standard deviation would be small,
whereas in the set [5, 0, 7, 3, 9, 2] the numbers are not consistent at all
and the standard deviation would be much larger. For the Sharpe ratio,
we are interested in the standard deviation of the monthly PL
values. If a "bell-shaped curve" (histogram) were drawn of the monthly PL values,
perhaps 68% of the area under the curve would be within plus and minus one standard
deviation of the average. For example, if the average were $1,000, and the
standard deviation were $600, then perhaps 68% of the area would be between $400 and
$1,600. We say "perhaps", because the monthly PL values would not likely follow a
"normal" distribution, on which the 68% number is based. In any case, if the
risk-free return in the above example was $100, then the Sharpe ratio would
be ($1,000 - $100) / $600 = 1.5.
The larger the Sharpe ratio the better (the more consistent the results).
The ratio will be negative if the average return is less than the
risk-free return. Some systems exhibit a Sharpe ratio of 0.5 or more, and
ratios above 1.0 are sometimes seen. For a long-term system, open profit
should be included in each month's PL data in order for the Sharpe ratio
to be meaningful. If there are less than 12 months of data, we do not
calculate the Sharpe ratio, because such a small number of data points might not
be statistically significant and could give misleading results.
The one-year (short-term) Sharpe ratio provides an indication of how
well a system has performed in the most recent 12 months. The calculation uses the
average monthly profit/loss in excess of the risk-free return for the most recent 12
months, divided by the standard deviation of monthly PLs over the same period.
See "Caution re One-year numbers" below.
11. "Sterling" column
This column gives the long-term and one-year (short-term)
Sterling ratios. The Sterling ratio is a measure, over the last 3
years, of the average annual return, relative to some measure of
drawdown.
We use the average yearly maximum drawdown, which may be more
representative of the drawdown characteristics of a system than the maximum drawdown,
which others may use. (Note that the annual maximum drawdown values are also given
in the Statistics page.) Also, we only use data for the most recent 3 years, while
others may use all the available data. We calculate the Sterling ratio as:
- (Av3yrPL ) / (Av3yrDD), where:
- Av3yrPL = average annual PL for the last 3 years; and
- Av3yrDD = average of the maximum end-of-month Drawdown for each of
the last 3 years.
The larger the Sterling ratio the better (the greater the return and the
smaller the drawdowns). The ratio will be negative if the average return is
negative. Some systems exhibit a Sterling ratio of 1.5 or more, and ratios
above 3 are sometimes seen.
As a drawdown measure, some sources specify 0.9 * AverageMaximumDD.
Our results are based on end-of-month data, so we use the "maximum
end-of-month drawdown", which will always be less than or equal to the
"maximum (instantaneous) drawdown", because the end-of-month equity high
is often less than the instantaneous equity high, and the end-of-month equity
low is often not as low as the instantaneous equity low.
When there are less than 24 months of data, we do not calculate the
Sterling ratio, because we feel there may not be enough drawdown data to be
statistically meaningful. When there are less than 36 months of data, the maximum
drawdown in the earliest (partial) year may not be representative because it will
contain fewer than 12 months of data. In this situation, we calculate the maximum
end-of-month drawdown for the earliest year based on however many months of data are
available; when the average is computed, we divide by ((months of data available) /
12) rather than 3. This in effect annualizes the drawdown for the earliest (partial)
year, which may may result in a Sterling ratio that is too small. (This approach
is different from the way the average maximum annual drawdown is calculated.)
The one-year (short-term) Sterling ratio is calculated to provide
an indication of how well a system has performed in the most recent 12 to 24-month
period. This calculation uses the total profit/loss over the most recent 12 months,
divided by the average of the maximum end-of-month drawdown during each of the most
recent two years.
12. "Sortino" column
This column gives the
long-term and
one-year (short-term)
Sortino
ratios. The Sortino ratio is a measure of the
average monthly return in
excess of the
risk-free return, relative to the
standard deviation
of the monthly disappointments, where "disappointment" means the amount by which
the return falls below the risk-free return. We use the term "standard deviation of
disappointments", because we find this to be more intuitively meaningful than
"downside deviation" which is commonly used. We calculate the Sortino ratio as:
- (AvMPL - RfMPL) / StDevMD, where
- MD = Monthly Disappointment, calculated as the amount by which a
month’s PL is below the risk-free return. E.g. if RfMPL = 100 and monthly
PL = -200 then the MD would be 300. If monthly PL = 100 or above then the
MD would be 0; and
- StdDevMD = standard deviation of the monthly disappointments.
The larger the Sortino ratio the better. The Sortino ratio will be larger
if the profit is high, and if the disappointments are small. For a given average
disappointment, the Sortino ratio would be better if there were many small
disappointments, rather than a few large disappointments (see the examples
below). The ratio will be negative if the average return is below the risk-free
return. Some systems exhibit a Sortino ratio of 1 or more, and ratios above 2 may
be seen.
If there are less than 24 months of data, we do not calculate the
Sortino ratio, because we feel that there may not be enough "disappointment" data
to be statistically meaningful.
When "average" and "standard deviation" of the disappointments are mentioned,
the calculations include the zero values. For example, for disappointments of
1.5, 1.5, 0, 0, 0, 0 the average is 0.5 and the standard deviation is 0.8;
for disappointments of 1, 1, 1, 0, 0, 0 the average is again 0.5 but the standard
deviation is 0.5, which is significantly smaller than in the first example.
The one-year (short-term) Sortino ratio is calculated, to provide
an indication of how well a system has performed in the most recent 12 to 24-month
period. This calculation uses the average monthly profit/loss in excess of the
risk-free return over the most recent 12 months, divided by the standard deviation
of the monthly disappointments over the most recent 24 months.
13. "Figure of Merit" column
This column gives the
long-term and
one-year (short-term)
Figure of
Merit. This Figure of Merit is our attempt to express the overall quality of a
component in a single figure. Many different approaches could be taken for this.
The calculation that we use is based on the average profit relative to account
size; the consistency of returns (Sharpe ratio); and the drawdown (Sterling ratio).
We do not calculate the Figure of Merit when there are less than
24 months of
data. The Figure of Merit is calculated as:
- Fig of Merit = AvYPL% * (normalized Sharpe ratio) * (normalized Sterling
ratio), where
- AvYPL% is the average annual profit/loss as a percentage
of the conservative or user account size.
- Normalized ratio is within the range of 0.707 (for a very low ratio)
to 1.414 (for a very high ratio), where 1.06 corresponds with a fairly
good ratio. See below for details.
The Sharpe and Sterling ratios are normalized in such a way that the Figure of
Merit will be between half and double the average annual percentage profit/loss.
The larger the Figure of Merit the better. The Figure of Merit will be larger if
the profit was high, the returns were consistent, and the drawdowns were small. If
the average annual percentage profit/loss is negative, the Figure of Merit will be
zero. If there are less than 24 months of data, the Figure of Merit is not
calculated.
The normalized ratio is calculated as:
- Normalized Ratio = 0.707 * ( 2 - 1 / ( 1 + Ratio/Pivot ) ), where
- Ratio is the Sharpe or Sterling ratio. If the ratio is negative,
a value of zero is used.
- Pivot is a fairly good value for the ratio (e.g. 66 percentile),
chosen from the range of values observed for our data. Current
values for the Pivot are 0.3 for the Sharpe ratio, and 1.0 for the Sterling
ratio.
| Effect of Normalizing function |
| Ratio |
Situation |
Normalized Ratio |
| Zero |
Worst possible ratio |
0.707 |
| Pivot / 3 |
Low ratio |
0.885 |
| Pivot |
Fairly Good ratio |
1.06 |
| 3 * Pivot |
High ratio |
1.237 |
| Infinity |
Best possible ratio |
1.414 |
The one-year (short-term) Figure of Merit is calculated to provide
an indication of how well a system has performed over the most recent 12 to 24-month
period. It is calculated as the percentage annual return over the most recent 12
months, multiplied by the normalized one-year Sharpe ratio and the normalized one-year
Sterling ratio.
14. Caution re One-year numbers
The one-year Sharpe, Sterling, and Sortino ratios and Figure of Merit are
calculated to provide an indication of how well the systems have performed
in the most recent 12 to 24-month period. For these calculations, we use data for
the most recent 12 months in the numerator, to provide a responsive indicator.
However, 24-month data is used for the denominators of the Sterling and Sortino
ratios, because we feel that 12-month data would not be meaningful enough. All the
one-year indicators use small numbers of data values, so their statistical
significance is not high, and accordingly they should be used with caution.
15. Note re Sharpe, Sterling, and Sortino ratios
These ratios can be defined in more than one way, so different sources could report
different values from the same set of result data. We have chosen simple definitions
which capture the essence of the ratios, while avoiding annualization and other
complications which are included in some definitions. Many definitions for these
ratios can be found on the internet, for example see
hedgefund.net.
16. "Single-Chart" column
This column contains a "
View" link that you can click to launch, in a new
window, a
single-component chart of a component.
Page last updated 2006-Apr-02
Caution:
Commodity trading involves substantial risk of LOSS, and is not
appropriate for everyone. Past performance is not necessarily indicative of
future results. Do not trade with funds you can not afford to LOSE!!
Disclaimer:
Information on this website is provided for educational and
informational purposes only. We do not endorse or recommend any particular
system(s) or broker(s). We endeavour to make the information as
accurate and meaningful as possible. However, we assume no liability for
the accuracy or integrity of data, charts, or other information on this
website, or for any use the user may make of this data. Much of the data
presented on this website is based on data obtained from other sources,
and we cannot guarantee its accuracy or integrity. Errors can and will occur.
If you notice any errors or other problems, please inform us as soon as
possible.
