Kalman Filter (02) – S&P 500 and Dow Jones Pairs Trading

Our previous article on Kalman filter gave us a simple linear regression output. The model is designed to handle noisy data, but not on streaming data and time adaptive parameters. This post we will study more general usage of the Kalman filter.

Simple Pairs Trading

From the linear regression result between S&P 500 and Dow Jons ETFs, a simple strategy can be easily derived. The regression parameters and give us not just an equation to evaluate the relative value of the price pair, but also confidence ranges for divergent signals. Here we can modify the linear relationship

to a two-sigma event detector below

When

we might just long the S&P 500 ETF (SPY) and short the Doe Jons ETF (DIA) and wait for them to converge, and vice versa for the case when

Moreover, we close the position when the prices converge within the range. We provide a sample code below to test the strategy from the year 2000 to 2015. The divergent event is capture by using the confidence interval calculated from the ordinary least-squares fitting results from the `statesmodel` module.

parameters	coef	std err	t value	[95.0% Conf. Int.]
	-2.0651	0.385	-5.364	[-2.820, -1.310]
	0.8903	0.003	264.258	[0.884, 0.897]

Python code implementation is included in the IPython Notebook file. Let’s check the outcome:

There are two difficulties with the strategy.

• The linear regression is derived from the whole history of data. It is impossible for us to use the linear regressed pair to trade from the beginning.
• From our last finding, the parameters should not be constant.

We expect to resolve the issues by Kalman filter.

Kalman Filter

The full model of a Kalman filter can be characterized as

and

where

• are the latent vector,
• are the observation vector,
• are the transition matrices,
• are the observation matrices,
• are the transition white noise following the distribution ,
• are the measurement white noise following the distribution .

By letting be an identity matrices, , , and , we have the time-adaptive linear regression model between the two prices.

So how this Kalman Filter improve the model?

One of the advantages using Kalman Filter is to find the latent factor distributions. The inferences on these parameters can give us more thoughts on building a mean-reverting trading strategy.

The noise affects latent variable distribution. We cannot directly observe , so we start with an initial conjecture, then continuously modify mean and variance. The is observation noise. It simply indicates that, based on the information on distribution, how the errors between the estimates and actual values behave. If is large, then in our case the linear regression might not be a good model to capture the relationship between S&P 500 ETF and Dow Jones ETF. We will make an assumption on and .

Okay, now what do we have? We can observe the prices from the market. We assume the linear relationship between the prices dynamics. Time adaptive parameters need an updating mechanism taking in streaming data. Kalman filter has another advantage for online optimization. It can be accomplished by using

filter_update function

from pykalman module.

For pairs trading with Kalman filter, we need to decide the two timings:

1. When to buy and sell?
2. When to close?

The updated state variance in Kalman filter provides a hint on this part since it indicates upper and lower bounds of the difference between DIA and SPY prices derived from the model. Each iteration Kalman filter outputs the updated mean and variance of , which is estimated at time . The estimated statistics are computed through Kalman gain. Details can be found in many literature. Then we simply enter the positions if the price difference is large that certain standard divation of the price deviation. To achieve it, we compute the estimation error variance

and use it to identify the divergent event. That is, we trade the pair when where controls the number of standard deviation.

By setting , we have

Improved Kalman Filter Pairs Trading

We can increase the accuracy of the linear regression prediction by estimating the changing speed and acceleration of future regression parameters. It is a common method used in signal processing. Through Taylor expansion, we have

where denote the speed and acceleration of the latent factor at time . To plug it into our Kalman filter, we can simply take and

However, the measurement matrices is defined as

to maintain the same simple linear regression form.

Just to make the model simpler, we set and apply the trick to the first order on and . The key parts of the code and the result are shown below.

Creating Kalman Filter

[sourcecode language=”python” light=”true” wraplines=”false” collapse=”false”]
# Create a Kalman Filter Object
kf = KalmanFilter(n_dim_obs=1,
n_dim_state=4,
transition_matrices=A,
transition_covariance=Q,
observation_covariance=R,
observation_matrices=array_SPY)
kf = kf.em(df_data.ix[‘1999’][‘DIA’].values)

# Linear Regression Outputs
alpha = kf.initial_state_mean[3]
beta = kf.initial_state_mean[0]
[/sourcecode]

Kalman Filter Update

[sourcecode language=”python” light=”true” wraplines=”false” collapse=”false”]
for row in df_data.ix[‘2000’:].iterrows():
price_spy = row[1][‘SPY’]
price_dia = row[1][‘DIA’]

# Kalman filter update
updated_mean, updated_variance = kf.filter_update(filtered_state_mean=updated_mean,
filtered_state_covariance=updated_variance,
observation_matrix=np.expand_dims(np.array([price_spy, 0, 1, 0]), 0),
observation=np.array([price_dia])
)
[/sourcecode]

Partial rule of buying and selling: to long SPY and short DIA we check if

Trading (Long SPY, Short DIA)

[sourcecode language=”python” light=”true” wraplines=”false” collapse=”false”]
if (price_dia > price_spy * beta + alpha + 1*P and
not long_spy_short_dia): # Long SPY-DIA Spread
if short_spy_long_dia:
total_wealth += shares_spy * (price_spy – cost_spy) + shares_dia * (price_dia – cost_dia)
short_spy_long_dia = False
shares_spy = total_wealth / price_spy
shares_dia = -1 * total_wealth / price_dia
cost_spy = price_spy
cost_dia = price_dia
long_spy_short_dia = True
[/sourcecode]

Performance is improved.

What’s Next?

We have shown how Kalman filter can used for pairs trading between S&P 500 ETF and Dow Jons ETF. We double the Sharpe ratio by implementing a second-order time adaptive linear regression based on Kalman filter and Taylor expansion. In our next topic on Kalman filter, we will examine the -asset pairs trading and probably non-linear Kalman filter.