Packaging the agent-based model Python code for friends and colleagues

In this post I will walk through packaging the agent-based model Python code with Anaconda’s conda build scripts. The big idea is to make the simulation apparatus accessible to folks with some Python chops and reasonable facility with the Jupyter Notebook, IPython, or command line environment. I will put all of the relevant links at the end of the post. There are many variations on how to build conda packages. I will discuss how I did it and leave the remaining variations to the documentation. For example, I will build from a GitHub repository; but there are other ways to do the build without a GitHub account.

Step 1: Tag a release in GitHub

I tagged the master branch with version 0.0.1b. You can see the result here.

Step 2: Write a setup.py file and put it in the root.

You can see the directory structure on my GitHub page for pyziabm. The file makes use of setup, which is imported from setuptools. Many of the keywords are optional, but Conda requires the “packages” keyword to do the build. Rummaging around the setup documentation could prove useful if you plan to do more complicated builds – with Cython, for instance.

from setuptools import setup

setup(
   name='pyziabm',
   version='0.1.0b',
   description='Zero Intelligence Agent-Based Model of Modern Limit Order Book',
   author='Charles Collver',
   author_email='jackbenny@always39.com',
   packages=['pyziabm'],
   url='https://github.com/JackBenny39/pyziabm',
   license='BSD3'
)

 

Step 3: Create a local build directory with three files: meta.yaml, build.sh, bld.bat.

I named the build directory with the package name (pyziabm), but it doesn’t matter. The meta.yaml file specifies the information Conda requires to run the build. I chose to ensure that the user has all of the necessary packages to create hdf5 files by adding them to the requirements: run: specs. The only test is to ensure that the package imports upon completion. You can run other tests as well. I chose to run them (manually) in Eclipse/PyDev before pushing to GitHub, merging and tagging.

package:

  name: pyziabm
  version: "0.1.0b"

source:

  git_rev: v0.1.0b
  git_url: https://github.com/JackBenny39/pyziabm.git

requirements:

  build:

    - python

  run:

    - python
    - numpy
    - h5py
    - pytables
    - pandas

test:

  imports:

    - pyziabm

about:

  home: https://github.com/JackBenny39/pyziabm
  license: BSD3
  license_file: LICENSE

 

The build scripts, bld.bat and build.sh, run setup.py install with some flags. See the install flags link below for more details.

build.sh and bld.bat

$PYTHON setup.py install --single-version-externally-managed --record=record.txt

%PYTHON% setup.py install --single-version-externally-managed --record=record.txt
if errorlevel 1 exit 1

 

Step 4: Navigate to the local build directory and run the build

This step typically produces a lot of output to the screen. While it is an interesting read, the important thing to check is if the build was actually successful! If it was, the output will contain the full path to the package. The output package file is usually buried somewhere in your anaconda directory structure. Copy that path, you will need it later.

pyziabm:~$ conda-build .

The dot is not a period. You can run conda build from another (enclosing) directory by specifying the full path.

Step 5: Install the new package and run it through its paces

somepath:~$ conda install --use-local pyziabm

 

Step 6: Push the package up to the conda cloud

This step requires you to have an account with Anaconda. It is free and painless. It is also unnecessary. You can just as easily distribute the package (via email attachment, for example) and have your friends conda install from local. Assuming you are determined to make your package available in the conda cloud and have created an account, then:

~$ anaconda login

~$ anaconda upload that/long/path/you/copied/pyziabm-0.1.0b-py36_0.tar.bz2

 

Step 7: Repeat steps 3 – 6 for other operating systems

I built from Windows then repeated from my Mac and an EC2-Ubuntu instance on AWS. The documentation claims you can use the “conda convert” command to generate the package for other OS, but it didn’t work for me.

The packages I built and uploaded to the conda cloud are available at: https://anaconda.org/dennycrane/pyziabm

As always, feel free to leave a comment or ask questions.

Useful links:

Tagging a Release in GitHub: https://help.github.com/articles/creating-releases/

distutils.setup keywords: https://setuptools.readthedocs.io/en/latest/setuptools.html#basic-use

install flags: https://setuptools.readthedocs.io/en/latest/setuptools.html#command-reference

Conda build: https://conda.io/docs/user-guide/tasks/build-packages/index.html

Collecting the agent-based model simulation output results with Python

This is the final post of the coding project in support of replicating the results in my Tick Pilot Agent-Based Modeling paper. The previous post developed the Runner class for running a single simulation. The final step is to write a wrapper that imports the Runner and runs a set of related simulations and then collects output data to be used as inputs for the charts and tables in the paper. First I will walk through the wrapper code. Then I will provide some details for running the entire set of 4,800 simulations on AWS EC2 Ubuntu servers.

The wrapper strategy is to generate a temporary hdf5 file to hold a set of tables containing results for one simulation. This data is then aggregated and summarized (i.e., munged) and stored in a bunch of lists at the end of each simulation. After the final simulation, these interim results are stored as tables in a summary hdf5 file. These summary tables contain the inputs to the charts and tables of the paper. This strategy will become clear as we walk through it. The full code is available on GitHub as runwrapper2017mpi_r4.py. First, the imports:

import random
import time
import numpy as np
import pandas as pd

from pyziabm.runner2017mpi_r4 import Runner

Pandas and numpy help with the munging and hdf5 file manipulations. Numpy and random are used to generate seeds for the random numbers. Time is an optional import for timing the individual simulations. The final import is the Runner class. Runner generates an hdf5 file with 4 tables: trades, mmp, tob, and orders. After each individual simulation, these tables are aggregated to summary results and stored to collector lists: participation_collector, position_collector, profit_collector, spread_collector, canceltrade_collector, by_mm_collector, and returns_collector.

Participation

The first function reads the trades table into a pandas DataFrame, creates the summaries, and appends the summary as a dict to the participation_collector.

def participation_to_list(h5in, outlist):
    trade_df = pd.read_hdf(h5in, 'trades')
    trade_df = trade_df.assign(trader_id = trade_df.resting_order_id.str.split('_').str[0])
    lt_df = pd.DataFrame(trade_df.groupby(['trader_id']).quantity.count())
    lt_df.rename(columns={'quantity': 'trade'}, inplace=True)
    if 'p999999' in lt_df.index:
        lt_df.drop('p999999', inplace=True)
    ltsum_df = pd.DataFrame(trade_df.groupby(['trader_id']).quantity.sum())
    ltsum_df.rename(columns={'quantity': 'trade_vol'}, inplace=True)
    ltsum_df = ltsum_df.assign(Participation = 100*ltsum_df.trade_vol/ltsum_df.trade_vol.sum())
    providers = ltsum_df.index.unique()
    market_makers = [x for x in providers if x.startswith('m')]
    market_makers.append('j0')
    ltsum_df = ltsum_df.ix[market_makers]
    part_dict = {'MCRun': j, 'MM_Participation': ltsum_df.loc['m0', 'Participation']}
    if 'j0' in providers:
        part_dict.update({'PJ_Participation': ltsum_df.loc['j0', 'Participation']})
    outlist.append(part_dict)

This strategy is repeated with varying levels of pandas munging in the functions to follow.

Position

def position_to_list(h5in, outlist):
    mmcf_df = pd.read_hdf(h5in, 'mmp')
    market_makers = mmcf_df.mmid.unique()
    for mm in market_makers:
        pos_dict = {}
        pos_dict['MCRun'] = j
        pos_dict['MarketMaker'] = mm
        pos_dict['Min'] =  mmcf_df[mmcf_df.mmid == mm].position.min()
        pos_dict['Max'] =  mmcf_df[mmcf_df.mmid == mm].position.max()
        outlist.append(pos_dict)

Profit

def profit_to_list(h5in, outlist):
    trade_df = pd.read_hdf(h5in, 'trades')
    trade_df = trade_df.assign(trader_id = trade_df.resting_order_id.str.split('_').str[0])
    buy_trades = trade_df[trade_df.side=='buy']
    buy_trades = buy_trades.assign(BuyCashFlow = buy_trades.price*buy_trades.quantity)
    buy_trades = buy_trades.assign(BuyVol = buy_trades.groupby('trader_id').quantity.cumsum(),
                                   CumulBuyCF = buy_trades.groupby('trader_id').BuyCashFlow.cumsum()
                                  )
    buy_trades.rename(columns={'timestamp': 'buytimestamp'}, inplace=True)
    sell_trades = trade_df[trade_df.side=='sell']
    sell_trades = sell_trades.assign(SellCashFlow = -sell_trades.price*sell_trades.quantity)
    sell_trades = sell_trades.assign(SellVol = sell_trades.groupby('trader_id').quantity.cumsum(),
                                     CumulSellCF = sell_trades.groupby('trader_id').SellCashFlow.cumsum()
                                    )
    sell_trades.rename(columns={'timestamp': 'selltimestamp'}, inplace=True)
    buy_trades = buy_trades[['trader_id', 'BuyVol', 'CumulBuyCF', 'buytimestamp']]
    sell_trades = sell_trades[['trader_id', 'SellVol', 'CumulSellCF', 'selltimestamp']]
    cash_flow = pd.merge(buy_trades, sell_trades, left_on=['trader_id', 'BuyVol'], right_on=['trader_id', 'SellVol'])
    cash_flow = cash_flow.assign(NetCashFlow = cash_flow.CumulBuyCF + cash_flow.CumulSellCF)
    temp_df = cash_flow.groupby('trader_id')['NetCashFlow', 'BuyVol'].last()
    temp_df = temp_df.assign(NetCFPerShare = temp_df.NetCashFlow/temp_df.BuyVol)
    temp_df = temp_df[['NetCashFlow', 'NetCFPerShare']]
    outlist.append(temp_df)

Spread

def spread_to_list(h5in, outlist):
    indf = pd.read_hdf(h5in, 'tob')
    indf = indf.assign(spread = indf.best_ask - indf.best_bid)
    last_df = indf.groupby('timestamp').last()
    last_df = last_df.loc[50:]
    spread_dict = {'MCRun': j, 'Min': last_df.spread.min(), 'Max': last_df.spread.max(), 'Median': last_df.spread.median(),
                   'Mean': last_df.spread.mean()}
    outlist.append(spread_dict)

Trades and Returns

def tradesrets_to_list(h5in, outlist):
    indf = pd.read_hdf(h5in, 'trades')
    trades = indf.price.count()
    minprice = indf.price.min()
    maxprice = indf.price.max()
    
    indf = indf.assign(ret = 100*indf.price.pct_change())
    indf = indf.assign(abs_ret = np.abs(indf.ret))
    lags = []
    autocorr = []
    abs_autocorr = []
    for i in range(1,51):
        ac = indf.ret.autocorr(lag = i)
        aac = indf.abs_ret.autocorr(lag = i)
        lags.append(i)
        autocorr.append(ac)
        abs_autocorr.append(aac)
    ar_df = pd.DataFrame({'lag': lags, 'autocorrelation': autocorr, 'autocorrelation_abs': abs_autocorr})
    ar_df.set_index('lag', inplace=True)
    clustering_constant = np.abs(ar_df.autocorrelation_abs.sum()/ar_df.autocorrelation.sum())
    
    returns_dict = {'Trades': trades, 'MinPrice': minprice, 'MaxPrice': maxprice, 'ClusteringConstant': clustering_constant,
                    'MeanRet': indf.ret.mean(), 'StdRet': indf.ret.std(), 'SkewRet': indf.ret.skew(),
                    'KurtosisRet': indf.ret.kurtosis(), 'MCRun': j}
    outlist.append(returns_dict)

Cancel-to-Trade and By Market Maker

def canceltrade_to_list(h5in, outlist1, outlist2):
    order_df = pd.read_hdf(h5in, 'orders')
    order_df = order_df.assign(trader_id = order_df.order_id.str.split('_').str[0])
    lpsum_df = order_df.groupby(['trader_id','type']).quantity.sum().unstack(level=-1)
    lpsum_df.rename(columns={'add': 'add_vol', 'cancel': 'cancel_vol'}, inplace=True)
    
    trade_df = pd.read_hdf(h5in, 'trades')
    trade_df = trade_df.assign(trader_id = trade_df.resting_order_id.str.split('_').str[0])
    ltsum_df = pd.DataFrame(trade_df.groupby(['trader_id']).quantity.sum())
    ltsum_df.rename(columns={'quantity': 'trade_vol'}, inplace=True)
    
    both_sum = pd.merge(lpsum_df, ltsum_df, how='right', left_index=True, right_index=True)
    both_sum = both_sum.assign(trade_order_vol_pct = 100*both_sum['trade_vol']/both_sum['add_vol'],
                               cancel_order_vol_pct = 100*both_sum['cancel_vol']/both_sum['add_vol'],
                               cancel_trade_vol = both_sum['cancel_vol']/both_sum['trade_vol']
                              )
    total_dict = {}
    total_dict['total_trade_to_order_vol'] = 100*both_sum.trade_vol.sum()/both_sum.add_vol.sum()
    total_dict['total_cancel_to_trade_vol'] = both_sum.cancel_vol.sum()/both_sum.trade_vol.sum()
    total_dict['MCRun'] = j
    outlist1.append(total_dict)
    
    traders = both_sum.index.unique()
    market_makers = [x for x in traders if (x.startswith('m') or x.startswith('j'))]
    for mm in market_makers:
        cto_dict = {}
        temp = both_sum.loc[mm, :]
        cto_dict['MCRun'] = j
        cto_dict['MarketMaker'] = mm
        cto_dict['CancelToTrade'] = temp['cancel_vol']/temp['trade_vol']
        cto_dict['TradeToOrderPct'] =  100*temp['trade_vol']/temp['add_vol']
        outlist2.append(cto_dict)

The final function is called after all of the simulations have run. It loads the interim collector lists into pandas DataFrames and saves each DataFrame as a table in a summary hdf5 file. There are lots of ways to do this. Pandas is convenient. A final note regarding the future of the ABM test bed: HDF5 is written in and compatible with C++. When the test bed code (i.e., all of the code in the previous posts up to and including the Runner) is re-written in C++, this last wrapper file will not require any changes to the bookkeeping functions.

def lists_to_h5(participation_list, position_list, profit_list, spread_list, canceltrade_list, by_mm_list, returns_list, h5out):
    participation_df = pd.DataFrame(participation_list)
    participation_df.set_index('MCRun', inplace=True)
    participation_df.to_hdf(h5out, 'participation', append=True, format='table', complevel=5, complib='blosc')
    
    position_df = pd.DataFrame(position_list)
    position_df.to_hdf(h5out, 'position', append=True, format='table', complevel=5, complib='blosc')
    
    profit_df = pd.concat(profit_list)
    profit_df.to_hdf(h5out, 'profit', append=True, format='table', complevel=5, complib='blosc')
    
    spread_df = pd.DataFrame(spread_list)
    spread_df.set_index('MCRun', inplace=True)
    spread_df.to_hdf(h5out, 'spread', append=True, format='table', complevel=5, complib='blosc')
    
    returns_df = pd.DataFrame(returns_list)
    returns_df.set_index('MCRun', inplace=True)
    returns_df.to_hdf(h5out, 'returns', append=True, format='table', complevel=5, complib='blosc')
    
    cancel_trade_df = pd.DataFrame(canceltrade_list)
    cancel_trade_df.to_hdf(h5out, 'cancel_trade', append=True, format='table', complevel=5, complib='blosc')

    by_mm_df = pd.DataFrame(by_mm_list)
    by_mm_df.to_hdf(h5out, 'by_mm', append=True, format='table', complevel=5, complib='blosc')

User inputs are declared before running the loop. Most of these are fixed for the results portrayed in the paper. The empty collectors are created and the variable inputs are specified. In this case, whether the run includes a penny jumper (pj), the penny jumper alpha (alpha_pj), the trial number (trial_no) and the number of simulations are specified by the user. I chose to fix the mpi in this file and change it in the Runner, but mpi could be user-specified as well. The final input is the specification of the final summary hdf5 file. I alter this last input when I switch to the executable version of this file.

participation_collector = []
position_collector = []
profit_collector = []
spread_collector = []
canceltrade_collector = []
by_mm_collector = []
returns_collector = []

# User inputs
#num_mms=1
#mm_maxq=1
#mm_quotes=12
#mm_quote_range=60
#mm_delta=0.05
#num_takers=100
#taker_maxq=1
#num_providers=38
#provider_maxq=1
#q_provide=0.5
#alpha=0.0375
#mu=0.001
#delta=0.025
#lambda0=100
#wn=0.001
#c_lambda=5.0
#run_steps=100000
#mpi=1
#h5filename='test.h5'  
alpha_pj = 0.001
pj = False
trial_no = 801
end = 101

h5_out = 'C:\\path\\to\\h5 files\\Trial %d\\ABMSmallCapSum.h5' % trial_no

The final step of the code is to run the simulations in a loop. For each simulation in the loop:

  1. the random seeds are set,
  2. the interim hdf5 file is specified (note the trial number and the simulation run number),
  3. the simulation is run conditional on the value of pj,
  4. the results are summarized and stored to lists,
  5. the interim hdf5 file is removed (optional here, but not in the unix version),
  6. the run time is reported

After the loop is run, the final hdf5 file is created.

start = time.time()
print(start)       
for j in range(1, end):
    random.seed(j)
    np.random.seed(j)
    h5_file = 'C:\\Path\\to\\h5 files\\Trial %d\\smallcap_%d.h5' % (trial_no, j)
    if pj:
        market1 = Runner(alpha_pj=alpha_pj, h5filename=h5_file)
    else:
        market1 = Runner(h5filename=h5_file)
    
    participation_to_list(market1.h5filename, participation_collector)
    position_to_list(market1.h5filename, position_collector)
    profit_to_list(market1.h5filename, profit_collector)
    spread_to_list(market1.h5filename, spread_collector)
    canceltrade_to_list(market1.h5filename, canceltrade_collector, by_mm_collector)
    tradesrets_to_list(market1.h5filename, returns_collector)
#    os.remove(market1.h5filename)
    
    print('Run %d:  %.2f minutes' % (j, (time.time() - start)/60))
    start = time.time()

lists_to_h5(participation_collector, position_collector, profit_collector, spread_collector, canceltrade_collector, by_mm_collector, returns_collector, h5_out)

The unix version of this file requires making the file executable, telling the OS where to find the python interpreter, specifying unique output hdf5 filenames, and allowing for the user-specified inputs as arguments to the script. The differences are portrayed in this last code snippet:

Code Block 11: executable, sys & os, use of sys.argv

#!/home/username/anaconda3/bin/python3

import os
import random
import sys
import time
import numpy as np
import pandas as pd

from pyziabm.runner2017mpi_r4 import Runner

...

alpha_pj = float(sys.argv[3])
pj = int(sys.argv[2])
trial_no = int(sys.argv[1])
end = 101

h5_out = '/home/username/h5/ABMSmallCapSum_%d.h5' % trial_no

You should be able to run the executable file (repeatedly) and use the final hdf5 files as inputs to a bunch of Jupyter Notebooks to generate the exact results portrayed in the paper. In the next post I will detail the steps for making a conda package from the pyziabm files and discuss the minor changes required to run the package import with the wrapper file.

Coding the agent-based model simulation loop with Python

This blog continues the coding project in support of replicating the results in my Tick Pilot Agent-Based Modeling paper. The first and second blogs created and tested the limit order book. The third and fourth blogs created and tested the traders. The next step is to pull the book and traders together and run a simulation. The strategy is designed to enable a user to install a package, import the package, and instantiate from a Runner class. For example, from the command line:

~$ conda install pyziabm

The user might have to specify the conda repo or download and install from local. See the Tick Pilot ABM project website for more details. Then from IPython or a Jupyter Notebook:

import pyziabm as pzi
pzi.Runner()

This command would run the simulation with a set of defaults and store some results in a table in an hdf5 file. The defaults are all keywords. The user can change the defaults by calling Runner with the keywords updated – in the spirit of how matplotlib gets things done:

pzi.Runner(mpi=1, h5filename='test2.h5', pj=True, alpha_pj=0.01)

The full code is available on GitHub as runner2017mpi_r4.py. As usual, the first step is to import some python packages. The traders and the orderbook were designed to be imported by the simulation module. We will import those as well.

import random
import numpy as np
import pandas as pd

from pyziabm.orderbook3 import Orderbook
from pyziabm.trader2017_r3 import Provider, Provider5, Taker, MarketMaker, MarketMaker5, PennyJumper

The __init__() method does all of the work in four major steps: create some useful attributes for later use, create the traders, orderbook and information environment, set up and run the simulation, and save some output. The first portion of __init__() demonstrates the keyword strategy and creates some attributes.

    def __init__(self, prime1=20, num_mms=1, mm_maxq=1, mm_quotes=12, mm_quote_range=60, mm_delta=0.025, 
                 num_takers=50, taker_maxq=1, num_providers=38, provider_maxq=1, q_provide=0.5,
                 alpha=0.0375, mu=0.001, delta=0.025, lambda0=100, wn=0.001, c_lambda=1.0, run_steps=100000,
                 mpi=5, h5filename='test.h5', pj=False, alpha_pj=0):
        self.alpha_pj = alpha_pj
        self.q_provide = q_provide
        self.lambda0 = lambda0
        self.run_steps = run_steps+1
        self.h5filename = h5filename

The second portion creates the traders and their arrival intervals, the order book and the information environment.

        self.t_delta_t, self.taker_array = self.make_taker_array(taker_maxq, num_takers, mu)
        self.t_delta_p, self.provider_array = self.make_provider_array(provider_maxq, num_providers, delta, mpi, alpha)
        self.t_delta_m, self.marketmaker_array = self.make_marketmaker_array(mm_maxq, num_mms, mm_quotes, mm_quote_range, mm_delta, mpi)
        self.pennyjumper = self.make_pennyjumper(mpi)
        self.exchange = Orderbook()
        self.q_take, self.lambda_t = self.make_q_take(wn, c_lambda)
        self.trader_dict = self.make_traders(num_takers, num_providers, num_mms)

The final portion prepares and runs the simulation and then saves output.

        self.seed_orderbook()
        self.make_setup(prime1)
        if pj:
            self.run_mcsPJ(prime1)
        else:
            self.run_mcs(prime1)
        self.exchange.trade_book_to_h5(h5filename)
        self.out_to_h5()

We will take each of these steps in order and I will provide a brief overview of what’s going on in each of the methods. See the Tick Pilot Agent-Based Modeling paper for further details and a full description of the agents and the simulation strategy.

The Traders

The make_taker_array(…) method creates the taker agents and their arrival intervals. The first three lines of code determine the trade size (size = 1 in the paper). The fourth line creates the random arrival intervals and the fifth and sixth lines prepare and create the Taker instances and store them in a numpy array. The arrival intervals are permanently associated with specific Taker instances via numpy arrays. We will make use of this later.

    def make_taker_array(self, maxq, num_takers, mu):
        default_arr = np.array([1, 5, 10, 25, 50])
        actual_arr = default_arr[default_arr<=maxq]
        taker_size = np.random.choice(actual_arr, num_takers)
        t_delta_t = np.floor(np.random.exponential(1/mu, num_takers)+1)*taker_size
        takers_list = ['t%i' % i for i in range(num_takers)]
        takers = np.array([Taker(t,i) for t,i in zip(takers_list,taker_size)])
        return t_delta_t, takers

The make_provider_array(…) method follows a similar strategy while using an if block to specify whether the provider should use a unit (penny) pricing increment or a 5 unit increment.

    def make_provider_array(self, maxq, num_providers, delta, mpi, alpha):
        default_arr = np.array([1, 5, 10, 25, 50])
        actual_arr = default_arr[default_arr<=maxq]
        provider_size = np.random.choice(actual_arr, num_providers)
        t_delta_p = np.floor(np.random.exponential(1/alpha, num_providers)+1)*provider_size
        providers_list = ['p%i' % i for i in range(num_providers)]
        if mpi==1:
            providers = np.array([Provider(p,i,mpi,delta) for p,i in zip(providers_list,provider_size)])
        else:
            providers = np.array([Provider5(p,i,mpi,delta) for p,i in zip(providers_list,provider_size)])
        return t_delta_p, providers

The make_marketmaker_array(…) method also follows the same strategy. The market maker arrival interval is the same as the trade size. In the paper, the single market maker has a trade size of one and therefore appears once every simulation step.

    def make_marketmaker_array(self, maxq, num_mms, mm_quotes, mm_quote_range, mm_delta, mpi):
        default_arr = np.array([1, 5, 10, 25, 50])
        actual_arr = default_arr[default_arr<=maxq]
        provider_size = np.random.choice(actual_arr, num_mms)
        t_delta_m = maxq
        marketmakers_list = ['m%i' % i for i in range(num_mms)]
        if mpi==1:
            marketmakers = np.array([MarketMaker(p,i,mpi,mm_delta,mm_quotes,mm_quote_range) for p,i in zip(marketmakers_list,provider_size)])
        else:
            marketmakers = np.array([MarketMaker5(p,i,mpi,mm_delta,mm_quotes,mm_quote_range) for p,i in zip(marketmakers_list,provider_size)])
        return t_delta_m, marketmakers

The make_pennyjumper(…) method merely returns the single instance of the Penny Jumper.

    def make_pennyjumper(self, mpi):
        return PennyJumper('j0', 1, mpi)

The Information Environment

The information environment includes a vector, q_take, that determines the probability a taker will submit a buy order and a vector, lambda_t, that serves as a parameter for a method that modifies the exponential distribution from which the Providers choose their prices.

    def make_q_take(self, s, c_lambda):
        noise = np.random.rand(2,self.run_steps)
        qt_take = np.empty_like(noise)
        qt_take[:,0] = 0.5
        for i in range(1,self.run_steps):
            qt_take[:,i] = qt_take[:,i-1] + (noise[:,i-1]>qt_take[:,i-1])*s - (noise[:,i-1]<qt_take[:,i-1])*s
        lambda_t = -self.lambda0*(1 + (np.abs(qt_take[1] - 0.5)/np.sqrt(np.mean(np.square(qt_take[0] - 0.5))))*c_lambda)
        return qt_take[1], lambda_t

Preparing the Orderbook

Preparing the order book for the simulation involves seeding the book with one ask order and one bid order and then priming the book for twenty steps with just the Providers participating. The seed_orderbook(…) method accomplishes the seeding. The make_setup(…) method calls make_providers(…) to prime the book. For each time step, make_setup(…) calls make_providers(…) and loops through the returned active Providers: the Provider processes the top-of-book signal; the Exchange (orderbook) processes the Provider order and then updates the top-of-book, which serves as an input for the next step through the list of active Providers. make_providers(…) uses np.remainder() on the arrival interval vector to determine which Providers are active in any particular step. We will re-use this strategy in the main simulation loop to follow.

    def seed_orderbook(self):
        seed_provider = Provider('p999999', 1, 5, 0.05)
        self.trader_dict.update({'p999999': seed_provider})
        ba = random.choice(range(1000005, 1002001, 5))
        bb = random.choice(range(997995, 999996, 5))
        qask = {'order_id': 'p999999_a', 'timestamp': 0, 'type': 'add', 'quantity': 1, 'side': 'sell',
              'price': ba, 'exid': 99999999}
        qbid = {'order_id': 'p999999_b', 'timestamp': 0, 'type': 'add', 'quantity': 1, 'side': 'buy',
              'price': bb, 'exid': 99999999}
        seed_provider.local_book['p999999_a'] = qask
        self.exchange.add_order_to_book(qask)
        self.exchange.order_history.append(qask)
        seed_provider.local_book['p999999_b'] = qbid
        self.exchange.add_order_to_book(qbid)
        self.exchange.order_history.append(qbid)

    def make_setup(self, prime1):
        top_of_book = self.exchange.report_top_of_book(0)
        for current_time in range(1, prime1):
            for p in self.make_providers(current_time):
                p.process_signal(current_time, top_of_book, self.q_provide, -self.lambda0)
                self.exchange.process_order(p.quote_collector[-1])
                top_of_book = self.exchange.report_top_of_book(current_time)

    def make_providers(self, step):
        providers = self.provider_array[np.remainder(step, self.t_delta_p)==0]
        np.random.shuffle(providers)
        return providers

Running the Simulation

The run_mcs(…) method steps through the remaining time, calling make_both(…) to determine which traders will participate in the time step and to randomize them (a misnomer, should name the method make_all(…)). A series of actions are specified as a function of trader type. Providers and MarketMakers add orders if their arrival interval matches the time step (that’s what “if row[1]:” determines) and potentially cancel orders regardless of whether their interval matches the time step or not. Takers add orders, too. A make_traders(…) method creates a dictionary of trader objects and their ids, thereby enabling liquidity provider lookup when a taker takes liquidity. This facilitates sending confirm messages to the liquidity providers when one of their resting orders is hit. The final block of code stores some of the larger history objects to an hdf5 file and resets the containers to empty.

    def run_mcs(self, prime1):
        top_of_book = self.exchange.report_top_of_book(prime1)
        for current_time in range(prime1, self.run_steps):
            for row in self.make_both(current_time):
                if row[0].trader_type == 'Provider':
                    if row[1]:
                        row[0].process_signal(current_time, top_of_book, self.q_provide, self.lambda_t[current_time])
                        self.exchange.process_order(row[0].quote_collector[-1])
                        top_of_book = self.exchange.report_top_of_book(current_time)
                    row[0].bulk_cancel(current_time)
                    if row[0].cancel_collector:
                        for c in row[0].cancel_collector:
                            self.exchange.process_order(c)
                            if self.exchange.confirm_modify_collector:
                                row[0].confirm_cancel_local(self.exchange.confirm_modify_collector[0])
                        top_of_book = self.exchange.report_top_of_book(current_time)
                elif row[0].trader_type == 'MarketMaker':
                    if row[1]:
                        row[0].process_signal(current_time, top_of_book, self.q_provide)
                        for q in row[0].quote_collector:
                            self.exchange.process_order(q)
                        top_of_book = self.exchange.report_top_of_book(current_time)
                    row[0].bulk_cancel(current_time)
                    if row[0].cancel_collector:
                        for c in row[0].cancel_collector:
                            self.exchange.process_order(c)
                            if self.exchange.confirm_modify_collector:
                                row[0].confirm_cancel_local(self.exchange.confirm_modify_collector[0])
                        top_of_book = self.exchange.report_top_of_book(current_time)
                else:
                    row[0].process_signal(current_time, self.q_take[current_time])
                    self.exchange.process_order(row[0].quote_collector[-1])
                    if self.exchange.traded:
                        for c in self.exchange.confirm_trade_collector:
                            trader = self.trader_dict[c['trader']]
                            trader.confirm_trade_local(c)
                        top_of_book = self.exchange.report_top_of_book(current_time)
            if not np.remainder(current_time, 2000):
                self.exchange.order_history_to_h5(self.h5filename)
                self.exchange.sip_to_h5(self.h5filename)

    def make_both(self, step):
        providers_mask = np.remainder(step, self.t_delta_p)==0
        takers_mask = np.remainder(step, self.t_delta_t)==0
        marketmakers_mask = np.remainder(step, self.t_delta_m)==0
        providers = np.vstack((self.provider_array, providers_mask)).T
        takers = np.vstack((self.taker_array, takers_mask)).T
        marketmakers = np.vstack((self.marketmaker_array, marketmakers_mask)).T
        traders = np.vstack((providers, marketmakers, takers[takers_mask]))
        np.random.shuffle(traders)
        return traders

    def make_traders(self, num_takers, num_providers, num_mms):
        takers_dict = dict(zip(['t%i' % i for i in range(num_takers)], list(self.taker_array)))
        providers_dict = dict(zip(['p%i' % i for i in range(num_providers)], list(self.provider_array)))
        takers_dict.update(providers_dict)
        marketmakers_dict = dict(zip(['m%i' % i for i in range(num_mms)], list(self.marketmaker_array)))
        takers_dict.update(marketmakers_dict)
        if self.alpha_pj > 0:
            takers_dict.update({'j0': self.pennyjumper})
        return takers_dict

The run_mcsPJ(…) method has an additional block of code at the end of each step through the traders. This code determines whether a PennyJumper will be active after a trader shows up. If so, the PennyJumper has an opportunity to add and/or cancel orders. Note that the PennyJumper can participate zero, one, or many times during each time step.

    def run_mcsPJ(self, prime1):

...

                if random.uniform(0,1) < self.alpha_pj:
                    self.pennyjumper.process_signal(current_time, top_of_book, self.q_take[current_time])
                    if self.pennyjumper.cancel_collector:
                        for c in self.pennyjumper.cancel_collector:
                            self.exchange.process_order(c)
                    if self.pennyjumper.quote_collector:
                        for q in self.pennyjumper.quote_collector:
                            self.exchange.process_order(q)
                    top_of_book = self.exchange.report_top_of_book(current_time)

...

The final step saves some results.

    def qtake_to_h5(self):
        temp_df = pd.DataFrame({'qt_take': self.q_take, 'lambda_t': self.lambda_t})
        temp_df.to_hdf(self.h5filename, 'qtl', append=True, format='table', complevel=5, complib='blosc')
        
    def mm_profitability_to_h5(self):
        for m in self.marketmaker_array:
            temp_df = pd.DataFrame(m.cash_flow_collector)
            temp_df.to_hdf(self.h5filename, 'mmp', append=True, format='table', complevel=5, complib='blosc')
            
    def out_to_h5(self):
        self.qtake_to_h5()
        self.mm_profitability_to_h5()

If you have comments or suggestions, feel free to post them. Coming up are posts describing the wrapper file to replicate the results in the Working Paper and some notes on a simple Conda build process for creating a package.

Unit testing the zero-intelligence traders with Python

In this post I will walk through unit testing the zero-intelligence trader classes and methods discussed in Coding some zero-intelligence traders with Python. Again, I will use the unittest module from the Python Standard Library. See Unit testing a simple limit order book with Python for a brief discussion of the unit testing choices available to you in the Python biosphere.

The full code and directory structure is available in my GitHub repo. You might find it helpful to have the actual Trader code handy when walking through the tests. As usual, we will create a separate test module, called testTrader2017_r3.py, and import the necessary support modules.

import random
import unittest

import numpy as np

from pyziabm.trader2017_r3 import Provider, Provider5, Taker, MarketMaker, MarketMaker5, PennyJumper

Next we declare the test class and define the setUp() fixture. In setUp() we instantiate the traders and some quotes for use with the tests to follow.

class TestTrader(unittest.TestCase):
    '''
    Five classes:
    1. ZITrader is a base class with make_add_quote()
    2. Taker inherits from ZITrader with process_signal()
    3. Provider inherits from ZITrader with make_cancel_quote(), confirm_cancel_local(), confirm_trade_local(), 
       process_signal(), choose_price_from_exp() and bulk_cancel().
    4. MarketMaker inherits from Provider with confirm_trade_local(), cumulate_cashflow() and process_signal().
    5. PennyJumper inherits from ZITrader with make_cancel_quote(), confirm_trade_local() and process_signal().
       PennyJumper has an ask_quote dictionary and bid_quote dictionary which can be None.
    '''

    def setUp(self):
        self.p1 = Provider('p1', 1, 1, 0.05)
        self.p5 = Provider5('p5', 1, 5, 0.05)
        self.t1 = Taker('t1', 1)
        self.m1 = MarketMaker('m1', 1, 1, 0.05, 12, 60)
        self.m5 = MarketMaker5('m5', 1, 5, 0.05, 12, 60)
        self.j1 = PennyJumper('j1', 1, 5)
        self.q1 = {'order_id': 'p1_1', 'timestamp': 1, 'type': 'add', 'quantity': 1, 'side': 'buy',
                   'price': 125}
        self.q2 = {'order_id': 'p1_2', 'timestamp': 2, 'type': 'add', 'quantity': 5, 'side': 'buy',
                   'price': 125}
        self.q3 = {'order_id': 'p1_3', 'timestamp': 3, 'type': 'add', 'quantity': 1, 'side': 'buy',
                   'price': 124}
        self.q4 = {'order_id': 'p1_4', 'timestamp': 4, 'type': 'add', 'quantity': 1, 'side': 'buy',
                   'price': 123}
        self.q5 = {'order_id': 'p1_5', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'buy',
                   'price': 122}
        self.q6 = {'order_id': 'p1_6', 'timestamp': 6, 'type': 'add', 'quantity': 1, 'side': 'sell',
                   'price': 126}
        self.q7 = {'order_id': 'p1_7', 'timestamp': 7, 'type': 'add', 'quantity': 5, 'side': 'sell',
                   'price': 127}
        self.q8 = {'order_id': 'p1_8', 'timestamp': 8, 'type': 'add', 'quantity': 1, 'side': 'sell',
                   'price': 128}
        self.q9 = {'order_id': 'p1_9', 'timestamp': 9, 'type': 'add', 'quantity': 1, 'side': 'sell',
                   'price': 129}
        self.q10 = {'order_id': 'p1_10', 'timestamp': 10, 'type': 'add', 'quantity': 1, 'side': 'sell',
                    'price': 130}

ZITrader is a base class with one responsibility: make the quote. This quote is the only required means of communication from the trader to the order book. All traders ultimately inherit the _make_add_quote() method from ZITrader. Though the test is very simple, I include it to remind myself if/when I make changes to the quote structure. The testing strategy for the trader instances establishes the state of the trader before calling the specific method (if necessary), creates any needed inputs, calls the method, and finally, tests that the output matches what is expected.

    def test_make_add_quote(self):
        time = 1
        quantity = 1
        side = 'sell'
        price = 125
        q = self.p1._make_add_quote(time, quantity, side, price)
        expected = {'order_id': 'p1_1', 'timestamp': 1, 'type': 'add', 'quantity': 1, 'side': 'sell', 
                    'price': 125}
        self.assertDictEqual(q, expected)

The Taker class inherits directly from ZITrader. It overrides the __repr__() method to include the trader_type attribute in the test. This public attribute is used in the simulation loop to determine which type of trader (in this case, a taker) is currently active. The Taker class also adds a process_signal() method, which is tested with some random seeds to ensure we get a buy order and a sell order.

    def test_repr_Taker(self):
        #print('Provider: {0}, Taker: {1}'.format(self.p1, self.t1))
        self.assertEqual('Taker: Trader(t1, 1, Taker)', 'Taker: {0}'.format(self.t1))
        
    def test_process_signal_Taker(self):
        '''
        Generates a quote object (dict) and appends to quote_collector
        '''
        time = 1
        q_taker = 0.5
        low_ru_seed = 1
        hi_ru_seed = 10
        self.assertFalse(self.t1.quote_collector)
        random.seed(low_ru_seed)
        self.t1.process_signal(time, q_taker)
        self.assertEqual(len(self.t1.quote_collector), 1)
        self.assertEqual(self.t1.quote_collector[0]['side'], 'buy')
        self.assertEqual(self.t1.quote_collector[0]['price'], 2000000)
        random.seed(hi_ru_seed)
        self.t1.process_signal(time, q_taker)
        self.assertEqual(len(self.t1.quote_collector), 1)
        self.assertEqual(self.t1.quote_collector[0]['side'], 'sell')
        self.assertEqual(self.t1.quote_collector[0]['price'], 0)

The Provider class also inherits directly from ZITrader and overrides the __repr__() method to include the trader_type attribute.

    def test_repr_Provider(self):
        #print('Provider: {0}, Taker: {1}'.format(self.p1, self.t1))
        self.assertEqual('Provider: Trader(p1, 1, Provider)', 'Provider: {0}'.format(self.p1))
        self.assertEqual('Provider: Trader(p5, 1, Provider)', 'Provider: {0}'.format(self.p5))

The remaining tests address three activities: confirming messages from the order book, canceling outstanding orders and generating add orders. The confirm tests both add some orders to the trader’s local book, call the method, and then test for the expected impact on the local book.

    def test_confirm_cancel_local_Provider(self):
        self.p1.local_book[self.q1['order_id']] = self.q1
        self.p1.local_book[self.q2['order_id']] = self.q2
        self.assertEqual(len(self.p1.local_book), 2)
        q = self.p1._make_cancel_quote(self.q1, 2)
        self.p1.confirm_cancel_local(q)
        self.assertEqual(len(self.p1.local_book), 1)
        expected = {self.q2['order_id']: self.q2}
        self.assertDictEqual(self.p1.local_book, expected)

    def test_confirm_trade_local_Provider(self):
        '''
        Test Provider for full and partial trade
        '''
        # Provider
        self.p1.local_book[self.q1['order_id']] = self.q1
        self.p1.local_book[self.q2['order_id']] = self.q2
        # trade full quantity of q1
        trade1 = {'timestamp': 2, 'trader': 'p1', 'order_id': 'p1_1', 'quantity': 1, 'side': 'buy', 'price': 2000000}
        self.assertEqual(len(self.p1.local_book), 2)
        self.p1.confirm_trade_local(trade1)
        self.assertEqual(len(self.p1.local_book), 1)
        expected = {self.q2['order_id']: self.q2}
        self.assertDictEqual(self.p1.local_book, expected)
        # trade partial quantity of q2
        trade2 = {'timestamp': 3, 'trader': 'p1', 'order_id': 'p1_2', 'quantity': 2, 'side': 'buy', 'price': 2000000}
        self.p1.confirm_trade_local(trade2)
        self.assertEqual(len(self.p1.local_book), 1)
        expected = {'order_id': 'p1_2', 'timestamp': 2, 'type': 'add', 'quantity': 3, 'side': 'buy', 
                    'price': 125}
        self.assertDictEqual(self.p1.local_book.get(trade2['order_id']), expected)

The cancel tests include a simple test for generating a cancel order and some tests for randomly canceling orders on the local book. The inline comments provide more details.

    def test_make_cancel_quote_Provider(self):
        q = self.p1._make_cancel_quote(self.q1, 2)
        expected = {'order_id': 'p1_1', 'timestamp': 2, 'type': 'cancel', 'quantity': 1, 'side': 'buy', 
                    'price': 125}
        self.assertDictEqual(q, expected)

    def test_bulk_cancel_Provider(self):
        '''
        Put 10 orders in the book, use random seed to determine which orders are cancelled,
        test for cancelled orders in the queue
        '''
        self.assertFalse(self.p1.local_book)
        self.assertFalse(self.p1.cancel_collector)
        self.p1.local_book[self.q1['order_id']] = self.q1
        self.p1.local_book[self.q2['order_id']] = self.q2
        self.p1.local_book[self.q3['order_id']] = self.q3
        self.p1.local_book[self.q4['order_id']] = self.q4
        self.p1.local_book[self.q5['order_id']] = self.q5
        self.p1.local_book[self.q6['order_id']] = self.q6
        self.p1.local_book[self.q7['order_id']] = self.q7
        self.p1.local_book[self.q8['order_id']] = self.q8
        self.p1.local_book[self.q9['order_id']] = self.q9
        self.p1.local_book[self.q10['order_id']] = self.q10
        self.assertEqual(len(self.p1.local_book), 10)
        self.assertFalse(self.p1.cancel_collector)
        # np.random seed = 8 generates 1 position less than 0.025 from np.random.ranf: 5
        np.random.seed(8)
        self.p1._delta = 0.025
        self.p1.bulk_cancel(11)
        self.assertEqual(len(self.p1.cancel_collector), 1)
        # np.random seed = 7 generates 2 positions less than 0.1 from np.random.ranf: 0, 7
        np.random.seed(7)
        self.p1._delta = 0.1
        self.p1.bulk_cancel(12)
        self.assertEqual(len(self.p1.cancel_collector), 2)
        # np.random seed = 6 generates 0 position less than 0.025 from np.random.ranf
        np.random.seed(6)
        self.p1._delta = 0.025
        self.p1.bulk_cancel(12)
        self.assertFalse(self.p1.cancel_collector)

The final Provider tests check the methods for creating a new add quote. test_choose_price_from_exp() tests for choosing a price consistent with the proper side of the book and for the correct pricing increment, which in this case is either 1 or 5. test_process_signal() checks for the random choice of buy or sell.

    def test_choose_price_from_exp(self):
        # mpi == 1
        sell_price = self.p1._choose_price_from_exp('bid', 75000, -100)
        self.assertLess(sell_price, 75000)
        buy_price = self.p1._choose_price_from_exp('ask', 25000, -100)
        self.assertGreater(buy_price, 25000)
        self.assertEqual(np.remainder(buy_price,self.p1._mpi),0)
        self.assertEqual(np.remainder(sell_price,self.p1._mpi),0)
        # mpi == 5        
        sell_price = self.p5._choose_price_from_exp('bid', 75000, -100)
        self.assertLess(sell_price, 75000)
        buy_price = self.p5._choose_price_from_exp('ask', 25000, -100)
        self.assertGreater(buy_price, 25000)
        self.assertEqual(np.remainder(buy_price,self.p5._mpi),0)
        self.assertEqual(np.remainder(sell_price,self.p5._mpi),0)
            
    def test_process_signal_Provider(self):
        time = 1
        q_provider = 0.5
        low_ru_seed = 1
        hi_ru_seed = 10
        tob_price = {'best_bid': 25000, 'best_ask': 75000}
        self.assertFalse(self.p1.quote_collector)
        self.assertFalse(self.p1.local_book)
        np.random.seed(low_ru_seed)
        self.p1.process_signal(time, tob_price, q_provider, -100)
        self.assertEqual(len(self.p1.quote_collector), 1)
        self.assertEqual(self.p1.quote_collector[0]['side'], 'buy')
        self.assertEqual(len(self.p1.local_book), 1)
        np.random.seed(hi_ru_seed)
        self.p1.process_signal(time, tob_price, q_provider, -100)
        self.assertEqual(len(self.p1.quote_collector), 1)
        self.assertEqual(self.p1.quote_collector[0]['side'], 'sell')
        self.assertEqual(len(self.p1.local_book), 2)

The MarketMaker class inherits from Provider and overrides the trader_type attribute and the confirm_trade_local() method. The MarketMaker version of confirm_trade_local() does some extra bookkeeping to track the market maker’s current position in shares. The _cumulate_cash_flow() method is called from confirm_trade_local() and stores the market maker’s current cash position.

    def test_repr_MM(self):
        self.assertEqual('MarketMaker: Trader(m1, 1, MarketMaker, 12)', 'MarketMaker: {0}'.format(self.m1))
        self.assertEqual('MarketMaker: Trader(m5, 1, MarketMaker, 12)', 'MarketMaker: {0}'.format(self.m5))
        
    def test_confirm_trade_local_MM(self):
        '''
        Test Market Maker for full and partial trade
        '''
        # MarketMaker buys
        self.m1.local_book[self.q1['order_id']] = self.q1
        self.m1.local_book[self.q2['order_id']] = self.q2
        # trade full quantity of q1
        trade1 = {'timestamp': 2, 'trader': 'p1', 'order_id': 'p1_1', 'quantity': 1, 'side': 'buy', 'price': 2000000}
        self.assertEqual(len(self.m1.local_book), 2)
        self.m1.confirm_trade_local(trade1)
        self.assertEqual(len(self.m1.local_book), 1)
        self.assertEqual(self.m1._position, 1)
        expected = {self.q2['order_id']: self.q2}
        self.assertDictEqual(self.m1.local_book, expected)
        # trade partial quantity of q2
        trade2 = {'timestamp': 3, 'trader': 'p1', 'order_id': 'p1_2', 'quantity': 2, 'side': 'buy', 'price': 2000000}
        self.m1.confirm_trade_local(trade2)
        self.assertEqual(len(self.m1.local_book), 1)
        self.assertEqual(self.m1._position, 3)
        expected = {'order_id': 'p1_2', 'timestamp': 2, 'type': 'add', 'quantity': 3, 'side': 'buy', 
                    'price': 125}
        self.assertDictEqual(self.m1.local_book.get(trade2['order_id']), expected) 
        
        # MarketMaker sells
        self.setUp()
        self.m1.local_book[self.q6['order_id']] = self.q6
        self.m1.local_book[self.q7['order_id']] = self.q7
        # trade full quantity of q6
        trade1 = {'timestamp': 6, 'trader': 'p1', 'order_id': 'p1_6', 'quantity': 1, 'side': 'sell', 'price': 0}
        self.assertEqual(len(self.m1.local_book), 2)
        self.m1.confirm_trade_local(trade1)
        self.assertEqual(len(self.m1.local_book), 1)
        self.assertEqual(self.m1._position, -1)
        expected = {self.q7['order_id']: self.q7}
        self.assertDictEqual(self.m1.local_book, expected)
        # trade partial quantity of q7
        trade2 = {'timestamp': 7, 'trader': 'p1', 'order_id': 'p1_7', 'quantity': 2, 'side': 'sell', 'price': 0}
        self.m1.confirm_trade_local(trade2)
        self.assertEqual(len(self.m1.local_book), 1)
        self.assertEqual(self.m1._position, -3)
        expected = {'order_id': 'p1_7', 'timestamp': 7, 'type': 'add', 'quantity': 3, 'side': 'sell', 
                    'price': 127}
        self.assertDictEqual(self.m1.local_book.get(trade2['order_id']), expected) 
        
    def test_cumulate_cashflow_MM(self):
        self.assertFalse(self.m1.cash_flow_collector)
        expected = {'mmid': 'm1', 'timestamp': 10, 'cash_flow': 0, 'position': 0}
        self.m1._cumulate_cashflow(10)
        self.assertDictEqual(self.m1.cash_flow_collector[0], expected)

The MarketMaker process_signal() method is tested for two possible states of the top of book: either the best price has only one share available or it has more than one share available. The method is also tested for buys and sells in a manner similar to the Provider tests. test_process_signal MM1_12() tests for a one tick price increment and the addition of 12 quotes within a pre-specified price range. The test introduces the subtest() method of unittest.TestCase within a context (i.e., with self.subtest()). To run in the for loop, the argument after the equal sign must match the loop argument. The main benefits of this strategy are that it reduces repetitive code and the complete loop will run whether the individual tests pass or not. See Doug Hellman’s Python Module of the Week for more details and a different example.

    def test_process_signal_MM1_12(self):
        time = 1
        q_provider = 0.5
        low_ru_seed = 1
        hi_ru_seed = 10
        # size > 1: market maker matches best price
        tob1 = {'best_bid': 25000, 'best_ask': 75000, 'bid_size': 10, 'ask_size': 10}
        self.assertFalse(self.m1.quote_collector)
        self.assertFalse(self.m1.local_book)
        random.seed(low_ru_seed)
        self.m1.process_signal(time, tob1, q_provider)
        self.assertEqual(len(self.m1.quote_collector), 12)
        self.assertEqual(self.m1.quote_collector[0]['side'], 'buy')
        for i in range(len(self.m1.quote_collector)):
            with self.subTest(i=i):
                self.assertLessEqual(self.m1.quote_collector[i]['price'], 25000)
                self.assertGreaterEqual(self.m1.quote_collector[i]['price'], 24941)
                self.assertTrue(self.m1.quote_collector[i]['price'] in range(24941, 25001))
        self.assertEqual(len(self.m1.local_book), 12)
        random.seed(hi_ru_seed)
        self.m1.process_signal(time, tob1, q_provider)
        self.assertEqual(len(self.m1.quote_collector), 12)
        self.assertEqual(self.m1.quote_collector[0]['side'], 'sell')
        for i in range(len(self.m1.quote_collector)):
            with self.subTest(i=i):
                self.assertLessEqual(self.m1.quote_collector[i]['price'], 75060)
                self.assertGreaterEqual(self.m1.quote_collector[i]['price'], 75000)
                self.assertTrue(self.m1.quote_collector[i]['price'] in range(75000, 75061))
        self.assertEqual(len(self.m1.local_book), 24)
        # size == 1: market maker adds liquidity one point behind
        self.setUp()
        tob2 = {'best_bid': 25000, 'best_ask': 75000, 'bid_size': 1, 'ask_size': 1}
        self.assertFalse(self.m1.quote_collector)
        self.assertFalse(self.m1.local_book)
        np.random.seed(low_ru_seed)
        self.m1.process_signal(time, tob2, q_provider)
        self.assertEqual(len(self.m1.quote_collector), 12)
        self.assertEqual(self.m1.quote_collector[0]['side'], 'buy')
        for i in range(len(self.m1.quote_collector)):
            with self.subTest(i=i):
                self.assertLessEqual(self.m1.quote_collector[i]['price'], 24999)
                self.assertGreaterEqual(self.m1.quote_collector[i]['price'], 24940)
                self.assertTrue(self.m1.quote_collector[i]['price'] in range(24940, 25000))
        self.assertEqual(len(self.m1.local_book), 12)
        np.random.seed(hi_ru_seed)
        self.m1.process_signal(time, tob2, q_provider)
        self.assertEqual(len(self.m1.quote_collector), 12)
        self.assertEqual(self.m1.quote_collector[0]['side'], 'sell')
        for i in range(len(self.m1.quote_collector)):
            with self.subTest(i=i):
                self.assertLessEqual(self.m1.quote_collector[i]['price'], 75060)
                self.assertGreaterEqual(self.m1.quote_collector[i]['price'], 75001)
                self.assertTrue(self.m1.quote_collector[i]['price'] in range(75001, 75061))
        self.assertEqual(len(self.m1.local_book), 24)

test_process_signal MM5_12() tests for a five tick price increment.

    def test_process_signal_MM5_12(self):
        time = 1
        q_provider = 0.5
        low_ru_seed = 1
        hi_ru_seed = 10
        # size > 1: market maker matches best price
        tob1 = {'best_bid': 25000, 'best_ask': 75000, 'bid_size': 10, 'ask_size': 10}
        self.assertFalse(self.m5.quote_collector)
        self.assertFalse(self.m5.local_book)
        random.seed(low_ru_seed)
        self.m5.process_signal(time, tob1, q_provider)
        self.assertEqual(len(self.m5.quote_collector), 12)
        self.assertEqual(self.m5.quote_collector[0]['side'], 'buy')
        for i in range(len(self.m5.quote_collector)):
            with self.subTest(i=i):
                self.assertLessEqual(self.m5.quote_collector[i]['price'], 25000)
                self.assertGreaterEqual(self.m5.quote_collector[i]['price'], 24935)
                self.assertTrue(self.m5.quote_collector[i]['price'] in range(24935, 25001, 5))
        self.assertEqual(len(self.m5.local_book), 12)
        random.seed(hi_ru_seed)
        self.m5.process_signal(time, tob1, q_provider)
        self.assertEqual(len(self.m5.quote_collector), 12)
        self.assertEqual(self.m5.quote_collector[0]['side'], 'sell')
        for i in range(len(self.m5.quote_collector)):
            with self.subTest(i=i):
                self.assertLessEqual(self.m5.quote_collector[i]['price'], 75065)
                self.assertGreaterEqual(self.m5.quote_collector[i]['price'], 75000)
                self.assertTrue(self.m5.quote_collector[i]['price'] in range(75000, 75066, 5))
        self.assertEqual(len(self.m5.local_book), 24)
        # size == 1: market maker adds liquidity one point behind
        self.setUp()
        tob2 = {'best_bid': 25000, 'best_ask': 75000, 'bid_size': 1, 'ask_size': 1}
        self.assertFalse(self.m5.quote_collector)
        self.assertFalse(self.m5.local_book)
        np.random.seed(low_ru_seed)
        self.m5.process_signal(time, tob2, q_provider)
        self.assertEqual(len(self.m5.quote_collector), 12)
        self.assertEqual(self.m5.quote_collector[0]['side'], 'buy')
        for i in range(len(self.m5.quote_collector)):
            with self.subTest(i=i):
                self.assertLessEqual(self.m5.quote_collector[i]['price'], 24995)
                self.assertGreaterEqual(self.m5.quote_collector[i]['price'], 24930)
                self.assertTrue(self.m5.quote_collector[i]['price'] in range(24930, 24995))
        self.assertEqual(len(self.m5.local_book), 12)
        np.random.seed(hi_ru_seed)
        self.m5.process_signal(time, tob2, q_provider)
        self.assertEqual(len(self.m5.quote_collector), 12)
        self.assertEqual(self.m5.quote_collector[0]['side'], 'sell')
        for i in range(len(self.m5.quote_collector)):
            with self.subTest(i=i):
                self.assertLessEqual(self.m5.quote_collector[i]['price'], 75065)
                self.assertGreaterEqual(self.m5.quote_collector[i]['price'], 75005)
                self.assertTrue(self.m5.quote_collector[i]['price'] in range(75005, 75065, 5))
        self.assertEqual(len(self.m5.local_book), 24)

The PennyJumper class inherits from ZITrader and implements its own confirm_trade_local() and process_signal() methods in addition to acquiring its own trader_type attribute. The test_confirm_trade_localPJ() method first populates the j1 instance _bid_quote and _ask_quote attributes so that the instance has two orders outstanding. Then it tests that the confirm_trade_local() method removes the two orders.

    def test_repr_PJ(self):
        self.assertEqual('PennyJumper: Trader(j1, 1, 5, PennyJumper)', 'PennyJumper: {0}'.format(self.j1))
        
    def test_confirm_trade_local_PJ(self):
        # PennyJumper book
        self.j1._bid_quote = {'order_id': 'j1_1', 'timestamp': 1, 'type': 'add', 'quantity': 1, 'side': 'buy',
                              'price': 125}
        self.j1._ask_quote = {'order_id': 'j1_6', 'timestamp': 6, 'type': 'add', 'quantity': 1, 'side': 'sell',
                              'price': 126}
        # trade at the bid
        trade1 = {'timestamp': 2, 'trader': 'j1', 'order_id': 'j1_1', 'quantity': 1, 'side': 'buy', 'price': 0}
        self.assertTrue(self.j1._bid_quote)
        self.j1.confirm_trade_local(trade1)
        self.assertFalse(self.j1._bid_quote)
        # trade at the ask
        trade2 = {'timestamp': 12, 'trader': 'j1', 'order_id': 'j1_6', 'quantity': 1, 'side': 'sell', 'price': 2000000}
        self.assertTrue(self.j1._ask_quote)
        self.j1.confirm_trade_local(trade2)
        self.assertFalse(self.j1._ask_quote)

The test_process_signalPJ() method tests combinations of the state of the top of the book and the penny jumper’s position relative to that state. The top of book spread can either be at the minimum price increment or larger than the minimum price increment. The penny jumper can be out of the market, alone at the inside, not alone at the inside, or behind the book.

Spread > MPI and penny jumper not in market: test add one bid and one ask.

    def test_process_signal_PJ(self):
        # spread > mpi
        tob = {'bid_size': 5, 'best_bid': 999990, 'best_ask': 1000005, 'ask_size': 5}
        # PJ book empty
        self.j1._bid_quote = None
        self.j1._ask_quote = None
        # random.seed = 1 generates random.uniform(0,1) = 0.13 then .85
        # jump the bid by 1, then jump the ask by 1
        random.seed(1)
        self.j1.process_signal(5, tob, 0.5)
        self.assertDictEqual(self.j1._bid_quote, {'order_id': 'j1_1', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'buy',
                                                 'price': 999995})
        tob = {'bid_size': 1, 'best_bid': 999995, 'best_ask': 1000005, 'ask_size': 5}
        self.j1.process_signal(6, tob, 0.5)
        self.assertDictEqual(self.j1._ask_quote, {'order_id': 'j1_2', 'timestamp': 6, 'type': 'add', 'quantity': 1, 'side': 'sell',
                                                 'price': 1000000})

Spread > MPI and penny jumper alone at the inside: test do nothing.

        # PJ alone at tob
        tob = {'bid_size': 1, 'best_bid': 999995, 'best_ask': 1000000, 'ask_size': 1}
        # nothing happens
        self.j1.process_signal(7, tob, 0.5)
        self.assertDictEqual(self.j1._bid_quote, {'order_id': 'j1_1', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'buy',
                                                 'price': 999995})
        self.assertDictEqual(self.j1._ask_quote, {'order_id': 'j1_2', 'timestamp': 6, 'type': 'add', 'quantity': 1, 'side': 'sell',
                                                 'price': 1000000})

Spread > MPI and penny jumper behind the book: test cancel outstanding orders, add new orders.

        # PJ bid and ask behind the book
        tob = {'bid_size': 1, 'best_bid': 999990, 'best_ask': 1000005, 'ask_size': 1}
        self.j1._bid_quote = {'order_id': 'j1_1', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'buy',
                             'price': 999985}
        self.j1._ask_quote = {'order_id': 'j1_2', 'timestamp': 6, 'type': 'add', 'quantity': 1, 'side': 'sell',
                             'price': 1000010}
        # random.seed = 1 generates random.uniform(0,1) = 0.13 then .85
        # jump the bid by 1, then jump the ask by 1; cancel old quotes
        random.seed(1)
        self.j1.process_signal(10, tob, 0.5)
        self.assertDictEqual(self.j1._bid_quote, {'order_id': 'j1_3', 'timestamp': 10, 'type': 'add', 'quantity': 1, 'side': 'buy',
                                                 'price': 999995})
        self.assertDictEqual(self.j1.cancel_collector[0], {'order_id': 'j1_1', 'timestamp': 10, 'type': 'cancel', 'quantity': 1, 'side': 'buy',
                                                            'price': 999985})
        self.assertDictEqual(self.j1.quote_collector[0], self.j1._bid_quote)
        self.j1.process_signal(11, tob, 0.5)
        self.assertDictEqual(self.j1._ask_quote, {'order_id': 'j1_4', 'timestamp': 11, 'type': 'add', 'quantity': 1, 'side': 'sell',
                                                 'price': 1000000})
        self.assertDictEqual(self.j1.cancel_collector[0], {'order_id': 'j1_2', 'timestamp': 11, 'type': 'cancel', 'quantity': 1, 'side': 'sell',
                                                           'price': 1000010})
        self.assertDictEqual(self.j1.quote_collector[0],self.j1._ask_quote)

Spread > MPI and penny jumper not alone at the inside: test cancel outstanding orders, add new orders.

        # PJ not alone at the inside
        tob = {'bid_size': 5, 'best_bid': 999990, 'best_ask': 1000010, 'ask_size': 5}
        self.j1._bid_quote = {'order_id': 'j1_1', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'buy',
                             'price': 999990}
        self.j1._ask_quote = {'order_id': 'j1_2', 'timestamp': 6, 'type': 'add', 'quantity': 1, 'side': 'sell',
                             'price': 1000010}
        # random.seed = 1 generates random.uniform(0,1) = 0.13 then .85
        # jump the bid by 1, then jump the ask by 1; cancel old quotes
        random.seed(1)
        self.j1.process_signal(12, tob, 0.5)
        self.assertDictEqual(self.j1._bid_quote, {'order_id': 'j1_5', 'timestamp': 12, 'type': 'add', 'quantity': 1, 'side': 'buy',
                                                 'price': 999995})
        self.assertDictEqual(self.j1.cancel_collector[0], {'order_id': 'j1_1', 'timestamp': 12, 'type': 'cancel', 'quantity': 1, 'side': 'buy',
                                                           'price': 999990})
        self.assertDictEqual(self.j1.quote_collector[0], self.j1._bid_quote)
        self.j1.process_signal(13, tob, 0.5)
        self.assertDictEqual(self.j1._ask_quote, {'order_id': 'j1_6', 'timestamp': 13, 'type': 'add', 'quantity': 1, 'side': 'sell',
                                                 'price': 1000005})
        self.assertDictEqual(self.j1.cancel_collector[0], {'order_id': 'j1_2', 'timestamp': 13, 'type': 'cancel', 'quantity': 1, 'side': 'sell',
                                                           'price': 1000010})
        self.assertDictEqual(self.j1.quote_collector[0],self.j1._ask_quote)

Spread == MPI and penny jumper alone at the inside: test do nothing.

        # spread at mpi, PJ alone at nbbo
        tob = {'bid_size': 1, 'best_bid': 999995, 'best_ask': 1000000, 'ask_size': 1}
        self.j1._bid_quote = {'order_id': 'j1_1', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'buy',
                             'price': 999995}
        self.j1._ask_quote = {'order_id': 'j1_2', 'timestamp': 6, 'type': 'add', 'quantity': 1, 'side': 'sell',
                             'price': 1000000}
        random.seed(1)
        self.j1.process_signal(14, tob, 0.5)
        self.assertDictEqual(self.j1._bid_quote, {'order_id': 'j1_1', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'buy',
                                                 'price': 999995})
        self.assertFalse(self.j1.cancel_collector)
        self.assertFalse(self.j1.quote_collector)
        self.j1.process_signal(15, tob, 0.5)
        self.assertDictEqual(self.j1._ask_quote, {'order_id': 'j1_2', 'timestamp': 6, 'type': 'add', 'quantity': 1, 'side': 'sell',
                                                 'price': 1000000})
        self.assertFalse(self.j1.cancel_collector)
        self.assertFalse(self.j1.quote_collector)

Spread == MPI and penny jumper behind the book: test cancel both.

        # PJ bid and ask behind the book
        self.j1._bid_quote = {'order_id': 'j1_1', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'buy',
                             'price': 999990}
        self.j1._ask_quote = {'order_id': 'j1_2', 'timestamp': 6, 'type': 'add', 'quantity': 1, 'side': 'sell',
                             'price': 1000010}
        # random.seed = 1 generates random.uniform(0,1) = 0.13 then .85
        # cancel bid and ask
        random.seed(1)
        self.assertTrue(self.j1._bid_quote)
        self.assertTrue(self.j1._ask_quote)
        self.j1.process_signal(16, tob, 0.5)
        self.assertFalse(self.j1._bid_quote)
        self.assertFalse(self.j1._ask_quote)
        self.assertDictEqual(self.j1.cancel_collector[0], {'order_id': 'j1_1', 'timestamp': 16, 'type': 'cancel', 'quantity': 1, 'side': 'buy',
                                                           'price': 999990})
        self.assertDictEqual(self.j1.cancel_collector[1], {'order_id': 'j1_2', 'timestamp': 16, 'type': 'cancel', 'quantity': 1, 'side': 'sell',
                                                           'price': 1000010})
        self.assertFalse(self.j1.quote_collector)

As usual, the tests are much longer than the code to be tested. If you have comments or suggestions, feel free to post them. Coming up are posts describing the simulation loop and a wrapper file to replicate the results in the Working Paper.

Coding some zero-intelligence traders with Python

This blog continues the coding project in support of replicating the results in my Tick Pilot Agent-Based Modeling paper. The first blog introduced the limit order book and the second described unit testing the order book. The next step is coding up some zero-intelligence traders – traders who follow simple rules and act randomly. The strategy employs object-oriented programming to enforce a single channel of communication from the trader to the order book and to reuse code whenever possible. There are several classes representing four basic trader types:

  1. The ZITrader class is the base class from which the others inherit. This class defines the communication mechanism (i.e., the order) and enforces the idea that this is the only way a trader can send messages to the order book.
  2. Two simple liquidity provider classes: Provider and Provider5, with Provider inheriting from ZITrader and Provider5 inheriting from Provider.
  3. Two market maker classes: MarketMaker and MarketMaker5, with MarketMaker inheriting from Provider and MarketMaker5 inheriting from MarketMaker.
  4. A Taker class inherits from ZITrader.
  5. A PennyJumper class inherits from ZITrader.

The full code is available on GitHub as trader2017_r3.py. As usual, the first step is to import some python modules.

import random
import numpy as np

The base class is ZITrader. It contains the _make_add_quote() method and some supporting infrastructure for making and storing quotes.

class ZITrader(object):
    '''
    ZITrader generates quotes (dicts) based on mechanical probabilities.
    
    A general base class for specific trader types.
    Public attributes: quote_collector
    Public methods: none
    '''

    def __init__(self, name, maxq):
        '''
        Initialize ZITrader with some base class attributes and a method
        
        quote_collector is a public container for carrying quotes to the exchange
        '''
        self._trader_id = name # trader id
        self._max_quantity = maxq
        self.quote_collector = []
        self._quote_sequence = 0
        
    def __repr__(self):
        return 'Trader({0}, {1})'.format(self._trader_id, self._max_quantity)
        
    def _make_add_quote(self, time, quantity, side, price):
        '''Make one add quote (dict)'''
        self._quote_sequence += 1
        order_id = '%s_%d' % (self._trader_id, self._quote_sequence)
        return {'order_id': order_id, 'timestamp': time, 'type': 'add', 'quantity': quantity, 
                'side': side, 'price': price}

In this simple model, I chose to make the quote a Python dict. In a more complex model, the actual quote could have been an instance of a separate class with ‘.’ access to the attributes.

Instances of the Provider class must be capable of several activities: generating inputs to the _make_add_quote() method, receiving confirmation messages from the order book, and canceling outstanding orders.

class Provider(ZITrader):
    '''
    Provider generates quotes (dicts) based on make probability.
    
    Subclass of ZITrader
    Public attributes: trader_type, quote_collector (from ZITrader), cancel_collector, local_book
    Public methods: confirm_cancel_local, confirm_trade_local, process_signal, bulk_cancel
    '''

    def __init__(self, name, maxq, mpi, delta):
        '''Provider has own mpi and delta; a local_book to track outstanding orders and a 
        cancel_collector to convey cancel messages to the exchange.
        '''
        ZITrader.__init__(self, name, maxq)
        self.trader_type = 'Provider'
        self._mpi = mpi
        self._delta = delta
        self.local_book = {}
        self.cancel_collector = []
                
    def __repr__(self):
        return 'Trader({0}, {1}, {2})'.format(self._trader_id, self._max_quantity, self.trader_type)
    
    def _make_cancel_quote(self, q, time):
        return {'type': 'cancel', 'timestamp': time, 'order_id': q['order_id'], 'quantity': q['quantity'],
                'side': q['side'], 'price': q['price']}
        
    def confirm_cancel_local(self, cancel_dict):
        del self.local_book[cancel_dict['order_id']]

    def confirm_trade_local(self, confirm):
        to_modify = self.local_book.get(confirm['order_id'], "WTF???")
        if confirm['quantity'] == to_modify['quantity']:
            self.confirm_cancel_local(to_modify)
        else:
            self.local_book[confirm['order_id']]['quantity'] -= confirm['quantity']
            
    def bulk_cancel(self, time):
        '''bulk_cancel cancels _delta percent of outstanding orders'''
        self.cancel_collector.clear()
        lob = len(self.local_book)
        if lob > 0:
            order_keys = list(self.local_book.keys())
            orders_to_delete = np.random.ranf(lob)
            for idx in range(lob):
                if orders_to_delete[idx] < self._delta:
                    self.cancel_collector.append(self._make_cancel_quote(self.local_book.get(order_keys[idx]), time))

    def process_signal(self, time, qsignal, q_provider, lambda_t):
        '''Provider buys or sells with probability related to q_provide'''
        self.quote_collector.clear()
        if np.random.uniform(0,1) < q_provider:
            price = self._choose_price_from_exp('bid', qsignal['best_ask'], lambda_t)
            side = 'buy'
        else:
            price = self._choose_price_from_exp('ask', qsignal['best_bid'], lambda_t)
            side = 'sell'
        q = self._make_add_quote(time, self._max_quantity, side, price)
        self.local_book[q['order_id']] = q
        self.quote_collector.append(q)            
      
    def _choose_price_from_exp(self, side, inside_price, lambda_t):
        '''Prices chosen from an exponential distribution'''
        # make pricing explicit for now. Logic scales for other mpi.
        plug = np.int(lambda_t*np.log(np.random.rand()))
        if side == 'bid':
            #price = np.int(5*np.floor((inside_price-1-plug)/5))
            price = inside_price-1-plug
        else:
            #price = np.int(5*np.ceil((inside_price+1+plug)/5))
            price = inside_price+1+plug
        return price

The process_signal() method randomly chooses whether to buy or sell and calls _choose_price_from_exp() to establish the price. Then it does some bookkeeping by adding the quote (dict) to the individual trader’s local book and to the quote_collector, a list that conveys the message to the order book. _choose_price_from_exp() randomly selects a price increment (plug) from an exponential distribution and then computes a price based on the distance from the best price on the opposite side of the market. This design has a purpose: Providers never cross the spread! bulk_cancel() and _make_cancel_quote() randomly select some quotes on the local book to be canceled and pass the cancel messages on to the exchange in the cancel_collector list. The two remaining methods, confirm_cancel_local() and confirm_trade_local() receive messages from the order book and modify the local book appropriately.

One of the contributions of the Working Paper is quantifying the impact of increasing the minimum pricing increment from one to five ticks on some market quality measures and market maker profitability. Provider5 enforces pricing on a five tick grid by employing its own _choose_price_from_exp() method.

class Provider5(Provider):
    '''
    Provider5 generates quotes (dicts) based on make probability.
    
    Subclass of Provider
    '''

    def __init__(self, name, maxq, mpi, delta):
        '''Provider has own mpi and delta; a local_book to track outstanding orders and a 
        cancel_collector to convey cancel messages to the exchange.
        '''
        Provider.__init__(self, name, maxq, mpi, delta)

    def _choose_price_from_exp(self, side, inside_price, lambda_t):
        '''Prices chosen from an exponential distribution'''
        # make pricing explicit for now. Logic scales for other mpi.
        plug = np.int(lambda_t*np.log(np.random.rand()))
        if side == 'bid':
            price = np.int(5*np.floor((inside_price-1-plug)/5))
        else:
            price = np.int(5*np.ceil((inside_price+1+plug)/5))
        return price

The MarketMaker is a special type of liquidity provider that inherits most of its behavior from Provider. Instances of the MarketMaker class submit multiple orders with prices chosen from uniform distribution. process_signal() accomplishes this by choosing a number of prices (_num_quotes) from a range of prices defined by the best price and the upper/lower limit (_quote_range). One of the research questions in the Working Paper focused on market maker profitability. Instances of MarketMaker track profitability by adding a few attributes (_position, _cash_flow, and cash_flow_collector), overriding the confirm_trade_local() method, and employing the _cumulate_cashflow() helper method.

class MarketMaker(Provider):
    '''
    MarketMaker generates a series of quotes near the inside (dicts) based on make probability.
    
    Subclass of Provider
    Public attributes: trader_type, quote_collector (from ZITrader), cancel_collector (from Provider),
    cash_flow_collector
    Public methods: confirm_cancel_local (from Provider), confirm_trade_local, process_signal 
    '''

    def __init__(self, name, maxq, mpi, delta, num_quotes, quote_range):
        '''_num_quotes and _quote_range determine the depth of MM quoting;
        _position and _cashflow are stored MM metrics
        '''
        Provider.__init__(self, name, maxq, mpi, delta)
        self.trader_type = 'MarketMaker'
        self._num_quotes = num_quotes
        self._quote_range = quote_range
        self._position = 0
        self._cash_flow = 0
        self.cash_flow_collector = []
                      
    def __repr__(self):
        return 'Trader({0}, {1}, {2}, {3})'.format(self._trader_id, self._max_quantity, self.trader_type, self._num_quotes)
            
    def confirm_trade_local(self, confirm):
        '''Modify _cash_flow and _position; update the local_book'''
        if confirm['side'] == 'buy':
            self._cash_flow -= confirm['price']*confirm['quantity']
            self._position += confirm['quantity']
        else:
            self._cash_flow += confirm['price']*confirm['quantity']
            self._position -= confirm['quantity']
        to_modify = self.local_book.get(confirm['order_id'], "WTF???")
        if confirm['quantity'] == to_modify['quantity']:
            self.confirm_cancel_local(to_modify)
        else:
            self.local_book[confirm['order_id']]['quantity'] -= confirm['quantity']
        self._cumulate_cashflow(confirm['timestamp'])
         
    def _cumulate_cashflow(self, timestamp):
        self.cash_flow_collector.append({'mmid': self._trader_id, 'timestamp': timestamp, 'cash_flow': self._cash_flow,
                                         'position': self._position})
            
    def process_signal(self, time, qsignal, q_provider):
        '''
        MM chooses prices from a grid determined by the best prevailing prices.
        MM never joins the best price if it has size=1.
        ''' 
        # make pricing explicit for now. Logic scales for other mpi and quote ranges.
        self.quote_collector.clear()
        if random.uniform(0,1) < q_provider:
            max_bid_price = qsignal['best_bid'] if qsignal['bid_size'] > 1 else qsignal['best_bid']-self._mpi
            prices = np.random.choice(range(max_bid_price-self._quote_range+1, max_bid_price+1, self._mpi), size=self._num_quotes)
            side = 'buy'
        else:
            min_ask_price = qsignal['best_ask'] if qsignal['ask_size'] > 1 else qsignal['best_ask']+self._mpi
            prices = np.random.choice(range(min_ask_price, min_ask_price+self._quote_range, self._mpi), size=self._num_quotes)
            side = 'sell'
        for price in prices:
            q = self._make_add_quote(time, self._max_quantity, side, price)
            self.local_book[q['order_id']] = q
            self.quote_collector.append(q)

MarketMaker5 enforces pricing on a five tick grid by overriding the MarketMaker process_signal() method. Two new attributes, _p5ask and _p5bid, assign probabilities to the discrete uniform price grid, thereby establishing the idea that the true reservation prices are still formulated on a one-tick grid. See the Working Paper for more details.

class MarketMaker5(MarketMaker):
    '''
    MarketMaker5 generates a series of quotes near the inside (dicts) based on make probability.
    
    Subclass of MarketMaker
    Public methods: process_signal 
    '''
    
    def __init__(self, name, maxq, mpi, delta, num_quotes, quote_range):
        '''
        _num_quotes and _quote_range determine the depth of MM quoting;
        _position and _cashflow are stored MM metrics
        '''
        MarketMaker.__init__(self, name, maxq, mpi, delta, num_quotes, quote_range)
        self._p5ask = [1/20, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/30]
        self._p5bid = [1/30, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/12, 1/20]
               
    def process_signal(self, time, qsignal, q_provider):
        '''
        MM chooses prices from a grid determined by the best prevailing prices.
        MM never joins the best price if it has size=1.
        ''' 
        # make pricing explicit for now. Logic scales for other mpi and quote ranges.
        self.quote_collector.clear()
        if random.uniform(0,1) < q_provider:
            max_bid_price = qsignal['best_bid'] if qsignal['bid_size'] > 1 else qsignal['best_bid']-self._mpi
            prices = np.random.choice(range(max_bid_price-self._quote_range, max_bid_price+1, self._mpi), size=self._num_quotes, p=self._p5bid)
            side = 'buy'
        else:
            min_ask_price = qsignal['best_ask'] if qsignal['ask_size'] > 1 else qsignal['best_ask']+self._mpi
            prices = np.random.choice(range(min_ask_price, min_ask_price+self._quote_range+1, self._mpi), size=self._num_quotes, p=self._p5ask)
            side = 'sell'
        for price in prices:
            q = self._make_add_quote(time, self._max_quantity, side, price)
            self.local_book[q['order_id']] = q
            self.quote_collector.append(q)

The PennyJumper is also a liquidity provider, but with a simple rule: either be alone at the best (inside) price or leave the market. The PennyJumper can have a maximum of two quotes outstanding: _ask_quote and _bid_quote. The quoting rule is implemented in its own process_signal() method. After clearing out the collectors, the method checks for the existence of an available price at the inside. If there is one, then the PennyJumper randomly chooses the side of the quote. It then checks whether it is alone at the best price and cancels if not. If the PennyJumper has no quote (self._bid_quote is None, for example) then the PennyJumper adds a new quote to establish the best inside price. If there is no available price at the inside (i.e., the spread is equal to the minimum price increment), then the PennyJumper checks whether it is alone at the inside and cancels if not.

class PennyJumper(ZITrader):
    '''
    PennyJumper jumps in front of best quotes when possible
    
    Subclass of ZITrader
    Public attributes: trader_type, quote_collector (from ZITrader), cancel_collector
    Public methods: confirm_trade_local (from ZITrader)
    '''
    
    def __init__(self, name, maxq, mpi):
        '''
        Initialize PennyJumper
        
        cancel_collector is a public container for carrying cancel messages to the exchange
        PennyJumper tracks private _ask_quote and _bid_quote to determine whether it is alone
        at the inside or not.
        '''
        ZITrader.__init__(self, name, maxq)
        self.trader_type = 'PennyJumper'
        self._mpi = mpi
        self.cancel_collector = []
        self._ask_quote = None
        self._bid_quote = None
        
    def __repr__(self):
        return 'Trader({0}, {1}, {2}, {3})'.format(self._trader_id, self._max_quantity, self._mpi, self.trader_type)
    
    def _make_cancel_quote(self, q, time):
        return {'type': 'cancel', 'timestamp': time, 'order_id': q['order_id'], 'quantity': q['quantity'],
                'side': q['side'], 'price': q['price']}

    def confirm_trade_local(self, confirm):
        '''PJ has at most one bid and one ask outstanding - if it executes, set price None'''
        if confirm['side'] == 'buy':
            self._bid_quote = None
        else:
            self._ask_quote = None
            
    def process_signal(self, time, qsignal, q_taker):
        '''PJ determines if it is alone at the inside, cancels if not and replaces if there is an available price 
        point inside the current quotes.
        '''
        self.quote_collector.clear()
        self.cancel_collector.clear()
        if qsignal['best_ask'] - qsignal['best_bid'] > self._mpi:
            # q_taker > 0.5 implies greater probability of a buy order; PJ jumps the bid
            if random.uniform(0,1) < q_taker:
                if self._bid_quote: # check if not alone at the bid
                    if self._bid_quote['price'] < qsignal['best_bid'] or self._bid_quote['quantity'] < qsignal['bid_size']:
                        self.cancel_collector.append(self._make_cancel_quote(self._bid_quote, time))
                        self._bid_quote = None
                if not self._bid_quote:
                    price = qsignal['best_bid'] + self._mpi
                    side = 'buy'
                    q = self._make_add_quote(time, self._max_quantity, side, price)
                    self.quote_collector.append(q)
                    self._bid_quote = q
            else:
                if self._ask_quote: # check if not alone at the ask
                    if self._ask_quote['price'] > qsignal['best_ask'] or self._ask_quote['quantity'] < qsignal['ask_size']:
                        self.cancel_collector.append(self._make_cancel_quote(self._ask_quote, time))
                        self._ask_quote = None
                if not self._ask_quote:
                    price = qsignal['best_ask'] - self._mpi
                    side = 'sell'
                    q = self._make_add_quote(time, self._max_quantity, side, price)
                    self.quote_collector.append(q)
                    self._ask_quote = q
        else: # spread = mpi
            if self._bid_quote: # check if not alone at the bid
                if self._bid_quote['price'] < qsignal['best_bid'] or self._bid_quote['quantity'] < qsignal['bid_size']:
                    self.cancel_collector.append(self._make_cancel_quote(self._bid_quote, time))
                    self._bid_quote = None
            if self._ask_quote: # check if not alone at the ask
                if self._ask_quote['price'] > qsignal['best_ask'] or self._ask_quote['quantity'] < qsignal['ask_size']:
                    self.cancel_collector.append(self._make_cancel_quote(self._ask_quote, time))
                    self._ask_quote = None

The Taker is the only liquidity taker in this model. process_signal() randomly chooses whether to buy or sell then makes an add quote guaranteed to cross the spread and take liquidity by choosing a price equal to zero for sells and 2,000,000 for buys.

class Taker(ZITrader):
    '''
    Taker generates quotes (dicts) based on take probability.
        
    Subclass of ZITrader
    Public attributes: trader_type, quote_collector (from ZITrader)
    Public methods: process_signal 
    '''

    def __init__(self, name, maxq):
        ZITrader.__init__(self, name, maxq)
        self.trader_type = 'Taker'
        
    def __repr__(self):
        return 'Trader({0}, {1}, {2})'.format(self._trader_id, self._max_quantity, self.trader_type)
        
    def process_signal(self, time, q_taker):
        '''Taker buys or sells with 50% probability.'''
        self.quote_collector.clear()
        if random.uniform(0,1) < q_taker: # q_taker > 0.5 implies greater probability of a buy order
            price = 2000000 # agent buys at max price (or better)
            side = 'buy'
        else:
            price = 0 # agent sells at min price (or better)
            side = 'sell'
        q = self._make_add_quote(time, self._max_quantity, side, price)
        self.quote_collector.append(q)

Additional trader types can be added to this model. The only necessity is that they generate the information required for _make_add_quote(). You may also want to augment your traders with additional bookkeeping tasks. If so, you can employ or modify the strategy demonstrated in MarketMaker. If you choose to make new liquidity providers, then adding the ability to cancel outstanding orders and track their own trading activity might also be useful. If you have comments or suggestions, please post them. The current WordPress settings require me to approve the first comment from a specific source. After that, you are free to comment without further approval. Coming up are posts describing the Trader unit tests followed by a post or two on the simulation loop.

Unit testing a simple limit order book with Python

This post is a follow-up to the previous post on building a simple limit order book with Python. After that original post, I learned that there were some quirks in how the WordPress editor handles the “less than” symbol in code blocks. That post has been updated and I will continue to monitor it for any more corruption.

I will walk through unit testing the orderbook methods with the unittest module. There are a variety of python testing alternatives. I chose unittest for two reasons: 1.) unittest is included in the standard library; and 2.) unittest works well with the Eclipse/PyDev Integrated Development Environment (IDE). Many Python aristocrats (Pythonistocrats?) have adopted Pytest because it is easier to implement with automated build processes. See for example the pandas documentation for a discussion of how they incorporate Pytest into their continuous integration services. I find the pandas documentation a very helpful resource for learning how to manage the test-build-ship process from GitHub.

Nearly every post on code testing will admonish you to implement test-driven development. But I admit that for me the combination of the orderbook and the tests is more like test-enhanced development: I would write the basics of the method, then test, and then repeat if necessary. To satisfy the rule on writing about testing, I recommend you use test-driven development!

Testing begins with creating a separate module, importing the module/class to be tested and unittest, and defining a test class that inherits from unittest.TestCase. The full code and directory structure is available in my GitHub repo. You might find it helpful to have the actual Orderbook code handy when walking through the tests.

from pyziabm.orderbook3 import Orderbook
import unittest

class TestOrderbook(unittest.TestCase):

There is a special method in unittest called setUp(). This method is called every time a test method is called. We will use it to provide a clean Orderbook instance and a set of known orders to each test.

    def setUp(self):
        '''
        setUp creates the Orderbook instance and a set of orders
        '''
        self.ex1 = Orderbook()
        self.q1_buy = {'order_id': 't1_1', 'timestamp': 2, 'type': 'add', 'quantity': 1, 'side': 'buy',
                       'price': 50}
        self.q2_buy = {'order_id': 't1_2', 'timestamp': 3, 'type': 'add', 'quantity': 1, 'side': 'buy',
                       'price': 50}
        self.q3_buy = {'order_id': 't10_1', 'timestamp': 4, 'type': 'add', 'quantity': 3, 'side': 'buy',
                       'price': 49}
        self.q4_buy = {'order_id': 't11_1', 'timestamp': 5, 'type': 'add', 'quantity': 3, 'side': 'buy',
                       'price': 47}
        self.q1_sell = {'order_id': 't1_3', 'timestamp': 2, 'type': 'add', 'quantity': 1, 'side': 'sell',
                        'price': 52}
        self.q2_sell = {'order_id': 't1_4', 'timestamp': 3, 'type': 'add', 'quantity': 1, 'side': 'sell',
                        'price': 52}
        self.q3_sell = {'order_id': 't10_2', 'timestamp': 4, 'type': 'add', 'quantity': 3, 'side': 'sell',
                        'price': 53}
        self.q4_sell = {'order_id': 't11_2', 'timestamp': 5, 'type': 'add', 'quantity': 3, 'side': 'sell',
                        'price': 55}

The testing strategy for the Orderbook instance (self.ex1) is similar for all of the tests: establish the state of the orderbook before calling the orderbook method (if necessary), create any needed inputs, call the method, and finally, test that the output matches what is expected. All test methods must begin with the word ‘test’. test_add_order_to_history() is a simple example of the strategy.

    def test_add_order_to_history(self):
        '''
        add_order_to_history() impacts the order_history list
        '''
        h1 = {'order_id': 't1_5', 'timestamp': 4, 'type': 'add', 'quantity': 5, 'side': 'sell', 'price': 55}
        self.assertFalse(self.ex1.order_history)
        h1['exid'] = 1
        self.ex1._add_order_to_history(h1)
        self.assertDictEqual(h1, self.ex1.order_history[0])

In this test, h1 is the input dict. The next line asserts that the order history list is empty. After appending the exchange order id to the dict, we call the method with h1. Finally, we assert that the modified h1 dict from the test method matches the first (and only) dict in the exchange order history list. If the assertions pass (return True), then the test will pass. In PyDev, the tests are run from a menu. In the console area, it will return something like:

Finding files... done.
Importing test modules ... done.
----------------------------------------------------------------------
Ran 1 tests in 0.000s

OK

You can also run the test module from a shell: python –m unittest testOrderbook3.

test_add_order_to_book() follows a similar strategy. First we check that the price list and the book dict are both empty (because setUp() was just called). Then we add one order to the bid book with self.ex1.add_order_to_book(self.q1_buy) and test whether actual and expected are the same with simple assertions: assertTrue, assertEqual, asserDictEqual. We then add another order to check whether the incrementing portion of add_order_to_book is working correctly. Finally, we repeat the process for sell orders.

    def test_add_order_to_book(self):
        '''
        add_order_to_book() impacts _bid_book and _bid_book_prices or _ask_book and _ask_book_prices
        Add two buy orders, then two sell orders
        '''
        # 2 buy orders
        self.assertFalse(self.ex1._bid_book_prices)
        self.assertFalse(self.ex1._bid_book)
        self.ex1.add_order_to_book(self.q1_buy)
        self.assertTrue(50 in self.ex1._bid_book_prices)
        self.assertTrue(50 in self.ex1._bid_book.keys())
        self.assertEqual(self.ex1._bid_book[50]['num_orders'], 1)
        self.assertEqual(self.ex1._bid_book[50]['size'], 1)
        self.assertEqual(self.ex1._bid_book[50]['order_ids'][0], self.q1_buy['order_id'])
        self.assertDictEqual(self.ex1._bid_book[50]['orders'][self.q1_buy['order_id']], self.q1_buy)
        self.ex1.add_order_to_book(self.q2_buy)
        self.assertEqual(self.ex1._bid_book[50]['num_orders'], 2)
        self.assertEqual(self.ex1._bid_book[50]['size'], 2)
        self.assertEqual(self.ex1._bid_book[50]['order_ids'][1], self.q2_buy['order_id'])
        self.assertDictEqual(self.ex1._bid_book[50]['orders'][self.q2_buy['order_id']], self.q2_buy)
        # 2 sell orders
        self.assertFalse(self.ex1._ask_book_prices)
        self.assertFalse(self.ex1._ask_book)
        self.ex1.add_order_to_book(self.q1_sell)
        self.assertTrue(52 in self.ex1._ask_book_prices)
        self.assertTrue(52 in self.ex1._ask_book.keys())
        self.assertEqual(self.ex1._ask_book[52]['num_orders'], 1)
        self.assertEqual(self.ex1._ask_book[52]['size'], 1)
        self.assertEqual(self.ex1._ask_book[52]['order_ids'][0], self.q1_sell['order_id'])
        self.assertDictEqual(self.ex1._ask_book[52]['orders'][self.q1_sell['order_id']], self.q1_sell)
        self.ex1.add_order_to_book(self.q2_sell)
        self.assertEqual(self.ex1._ask_book[52]['num_orders'], 2)
        self.assertEqual(self.ex1._ask_book[52]['size'], 2)
        self.assertEqual(self.ex1._ask_book[52]['order_ids'][1], self.q2_sell['order_id'])
        self.assertDictEqual(self.ex1._ask_book[52]['orders'][self.q2_sell['order_id']], self.q2_sell)

test_remove_order() first adds two orders and checks the state of the book. Then it removes the two orders and checks the orderbook state after each removal. Finally, it also checks that removing an order that is not there causes no harm. The process is then repeated for sell orders.

    def test_remove_order(self):
        '''
        _remove_order() impacts _bid_book and _bid_book_prices or _ask_book and _ask_book_prices
        Add two  orders, remove the second order twice
        '''
        # buy orders
        self.ex1.add_order_to_book(self.q1_buy)
        self.ex1.add_order_to_book(self.q2_buy)
        self.assertTrue(50 in self.ex1._bid_book_prices)
        self.assertTrue(50 in self.ex1._bid_book.keys())
        self.assertEqual(self.ex1._bid_book[50]['num_orders'], 2)
        self.assertEqual(self.ex1._bid_book[50]['size'], 2)
        self.assertEqual(len(self.ex1._bid_book[50]['order_ids']), 2)
        # remove first order
        self.ex1._remove_order('buy', 50, 't1_1')
        self.assertEqual(self.ex1._bid_book[50]['num_orders'], 1)
        self.assertEqual(self.ex1._bid_book[50]['size'], 1)
        self.assertEqual(len(self.ex1._bid_book[50]['order_ids']), 1)
        self.assertFalse('t1_1' in self.ex1._bid_book[50]['orders'].keys())
        self.assertTrue(50 in self.ex1._bid_book_prices)
        # remove second order
        self.ex1._remove_order('buy', 50, 't1_2')
        self.assertFalse(self.ex1._bid_book_prices)
        self.assertEqual(self.ex1._bid_book[50]['num_orders'], 0)
        self.assertEqual(self.ex1._bid_book[50]['size'], 0)
        self.assertEqual(len(self.ex1._bid_book[50]['order_ids']), 0)
        self.assertFalse('t1_2' in self.ex1._bid_book[50]['orders'].keys())
        self.assertFalse(50 in self.ex1._bid_book_prices)
        # remove second order again
        self.ex1._remove_order('buy', 50, 't1_2')
        self.assertFalse(self.ex1._bid_book_prices)
        self.assertEqual(self.ex1._bid_book[50]['num_orders'], 0)
        self.assertEqual(self.ex1._bid_book[50]['size'], 0)
        self.assertEqual(len(self.ex1._bid_book[50]['order_ids']), 0)
        self.assertFalse('t1_2' in self.ex1._bid_book[50]['orders'].keys())
        # sell orders
        self.ex1.add_order_to_book(self.q1_sell)
        self.ex1.add_order_to_book(self.q2_sell)
        self.assertTrue(52 in self.ex1._ask_book_prices)
        self.assertTrue(52 in self.ex1._ask_book.keys())
        self.assertEqual(self.ex1._ask_book[52]['num_orders'], 2)
        self.assertEqual(self.ex1._ask_book[52]['size'], 2)
        self.assertEqual(len(self.ex1._ask_book[52]['order_ids']), 2)
        # remove first order
        self.ex1._remove_order('sell', 52, 't1_3')
        self.assertEqual(self.ex1._ask_book[52]['num_orders'], 1)
        self.assertEqual(self.ex1._ask_book[52]['size'], 1)
        self.assertEqual(len(self.ex1._ask_book[52]['order_ids']), 1)
        self.assertFalse('t1_1' in self.ex1._ask_book[52]['orders'].keys())
        self.assertTrue(52 in self.ex1._ask_book_prices)
        # remove second order
        self.ex1._remove_order('sell', 52, 't1_4')
        self.assertFalse(self.ex1._ask_book_prices)
        self.assertEqual(self.ex1._ask_book[52]['num_orders'], 0)
        self.assertEqual(self.ex1._ask_book[52]['size'], 0)
        self.assertEqual(len(self.ex1._ask_book[52]['order_ids']), 0)
        self.assertFalse('t1_2' in self.ex1._ask_book[52]['orders'].keys())
        self.assertFalse(52 in self.ex1._ask_book_prices)
        # remove second order again
        self.ex1._remove_order('sell', 52, 't1_4')
        self.assertFalse(self.ex1._ask_book_prices)
        self.assertEqual(self.ex1._ask_book[52]['num_orders'], 0)
        self.assertEqual(self.ex1._ask_book[52]['size'], 0)
        self.assertEqual(len(self.ex1._ask_book[52]['order_ids']), 0)
        self.assertFalse('t1_2' in self.ex1._ask_book[52]['orders'].keys())

In modern limit order book markets, some order modifications do not generally result in loss of time priority. Reducing the limit order quantity is one example of this type of modification. test_modify_order() begins by adding an order with quantity of 2 to a clean orderbook and then tests for a reduction in quantity and finally tests for removal when quantity becomes zero. The tests are repeated for sell orders.

    def test_modify_order(self):
        '''
        _modify_order() primarily impacts _bid_book or _ask_book
        _modify_order() could impact _bid_book_prices or _ask_book_prices if the order results
        in removing the full quantity with a call to _remove_order()
        Add 1 order, remove partial, then remainder
        '''
        # Buy order
        q1 = {'order_id': 't1_1', 'timestamp': 5, 'type': 'add', 'quantity': 2, 'side': 'buy',
              'price': 50}
        self.ex1.add_order_to_book(q1)
        self.assertEqual(self.ex1._bid_book[50]['size'], 2)
        # remove 1
        self.ex1._modify_order('buy', 1, 't1_1', 50)
        self.assertEqual(self.ex1._bid_book[50]['size'], 1)
        self.assertEqual(self.ex1._bid_book[50]['orders']['t1_1']['quantity'], 1)
        self.assertTrue(self.ex1._bid_book_prices)
        # remove remainder
        self.ex1._modify_order('buy', 1, 't1_1', 50)
        self.assertFalse(self.ex1._bid_book_prices)
        self.assertEqual(self.ex1._bid_book[50]['num_orders'], 0)
        self.assertEqual(self.ex1._bid_book[50]['size'], 0)
        self.assertFalse('t1_1' in self.ex1._bid_book[50]['orders'].keys())
        # Sell order
        q2 = {'order_id': 't1_1', 'timestamp': 5, 'type': 'add', 'quantity': 2, 'side': 'sell',
              'price': 50}
        self.ex1.add_order_to_book(q2)
        self.assertEqual(self.ex1._ask_book[50]['size'], 2)
        # remove 1
        self.ex1._modify_order('sell', 1, 't1_1', 50)
        self.assertEqual(self.ex1._ask_book[50]['size'], 1)
        self.assertEqual(self.ex1._ask_book[50]['orders']['t1_1']['quantity'], 1)
        self.assertTrue(self.ex1._ask_book_prices)
        # remove remainder
        self.ex1._modify_order('sell', 1, 't1_1', 50)
        self.assertFalse(self.ex1._ask_book_prices)
        self.assertEqual(self.ex1._ask_book[50]['num_orders'], 0)
        self.assertEqual(self.ex1._ask_book[50]['size'], 0)
        self.assertFalse('t1_1' in self.ex1._ask_book[50]['orders'].keys())

test_add_trade_to_book(), test_confirm_trade() and test_confirm_modify() all test whether a python dict has been added to a list. These tests follow the same strategy as test_add_order_to_history().

    def test_add_trade_to_book(self):
        '''
        add_trade_to_book() impacts trade_book
        Check trade book empty, add a trade, check non-empty, verify dict equality
        '''
        t1 = dict(resting_order_id='t1_1', resting_timestamp=2, incoming_order_id='t2_1',
                  timestamp=5, price=50, quantity=1, side='buy')
        self.assertFalse(self.ex1.trade_book)
        self.ex1._add_trade_to_book('t1_1', 2, 't2_1', 5, 50, 1, 'buy')
        self.assertTrue(self.ex1.trade_book)
        self.assertDictEqual(t1, self.ex1.trade_book[0])

    def test_confirm_trade(self):
        '''
        confirm_trade() impacts confirm_trade_collector
        Check confirm trade collector empty, add a trade, check non-empty, verify dict equality
        '''
        t2 = dict(timestamp=5, trader='t3', order_id='t3_1', quantity=1,
                  side='sell', price=50)
        self.assertFalse(self.ex1.confirm_trade_collector)
        self.ex1._confirm_trade(5, 'sell', 1, 't3_1', 50)
        self.assertTrue(self.ex1.confirm_trade_collector)
        self.assertDictEqual(t2, self.ex1.confirm_trade_collector[0])

    def test_confirm_modify(self):
        '''
        confirm_modify() impacts confirm_modify_collector
        Check confirm modify collector empty, add a trade, check non-empty, verify dict equality
        '''
        m1 = dict(timestamp=7, trader='t5', order_id='t5_10', quantity=5, side='buy')
        self.assertFalse(self.ex1.confirm_modify_collector)
        self.ex1._confirm_modify(7, 'buy', 5, 't5_10')
        self.assertTrue(self.ex1.confirm_modify_collector)
        self.assertDictEqual(m1, self.ex1.confirm_modify_collector[0])

In the Orderbook instance, process_order() potentially relies upon _match_trade(). Testing these independently is difficult. I decided to test process_order() with a simple trade quantity of 1 and then test for proper matching (i.e., “walking the book”) with quantities > 1 in _match_trade(). test_process_order() seeds each side of the orderbook with 2 orders with the same price, then tests the impact of marketable buy and sell orders with quantity 1. It then tests for adding, canceling and modifying some orders. See the docstring and inline comments for more details.

    def test_process_order(self):
        '''
        process_order() impacts confirm_modify_collector, traded indicator, order_history,
        _bid_book and _bid_book_prices or _ask_book and _ask_book_prices.
        process_order() is a traffic manager. An order is either an add order or not. If it is an add order,
        it is either priced to go directly to the book or is sent to match_trade (which is tested below). If it
        is not an add order, it is either modified or cancelled. To test, we will add some buy and sell orders,
        then test for trades, cancels and modifies. process_order() also resets some object collectors.
        '''
        self.q2_buy['quantity'] = 2
        self.q2_sell['quantity'] = 2

        self.assertEqual(len(self.ex1._ask_book_prices), 0)
        self.assertEqual(len(self.ex1._bid_book_prices), 0)
        self.assertFalse(self.ex1.confirm_modify_collector)
        self.assertFalse(self.ex1.order_history)
        self.assertFalse(self.ex1.traded)
        # seed order book
        self.ex1.add_order_to_book(self.q1_buy)
        self.ex1.add_order_to_book(self.q1_sell)
        # process new orders
        self.ex1.process_order(self.q2_buy)
        self.ex1.process_order(self.q2_sell)
        self.assertEqual(len(self.ex1._ask_book_prices), 1)
        self.assertEqual(len(self.ex1._bid_book_prices), 1)
        self.assertEqual(len(self.ex1.order_history), 2)
        # marketable sell takes out 1 share
        q3_sell = {'order_id': 't3_1', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'sell',
                   'price': 0}
        self.ex1.process_order(q3_sell)
        self.assertEqual(len(self.ex1.order_history), 3)
        self.assertEqual(self.ex1._bid_book[50]['num_orders'], 1)
        self.assertEqual(self.ex1._bid_book[50]['size'], 2)
        self.assertTrue(self.ex1.traded)
        # marketable buy takes out 1 share
        q3_buy = {'order_id': 't3_2', 'timestamp': 5, 'type': 'add', 'quantity': 1, 'side': 'buy',
                  'price': 10000}
        self.ex1.process_order(q3_buy)
        self.assertEqual(len(self.ex1.order_history), 4)
        self.assertEqual(self.ex1._ask_book[52]['num_orders'], 1)
        self.assertEqual(self.ex1._ask_book[52]['size'], 2)
        self.assertTrue(self.ex1.traded)
        # add/cancel buy order
        q4_buy = {'order_id': 't4_1', 'timestamp': 10, 'type': 'add', 'quantity': 1, 'side': 'buy',
                  'price': 48}
        self.ex1.process_order(q4_buy)
        self.assertEqual(len(self.ex1.order_history), 5)
        self.assertEqual(len(self.ex1._bid_book_prices), 2)
        self.assertEqual(self.ex1._bid_book[48]['num_orders'], 1)
        self.assertEqual(self.ex1._bid_book[48]['size'], 1)
        self.assertFalse(self.ex1.traded)
        q4_cancel1 = {'order_id': 't4_1', 'timestamp': 10, 'type': 'cancel', 'quantity': 1, 'side': 'buy',
                     'price': 48}
        self.ex1.process_order(q4_cancel1)
        self.assertEqual(len(self.ex1.order_history), 6)
        self.assertEqual(len(self.ex1._bid_book_prices), 1)
        self.assertFalse(self.ex1.traded)
        # add/cancel sell order
        q4_sell = {'order_id': 't4_2', 'timestamp': 10, 'type': 'add', 'quantity': 1, 'side': 'sell',
                  'price': 54}
        self.ex1.process_order(q4_sell)
        self.assertEqual(len(self.ex1.order_history), 7)
        self.assertEqual(len(self.ex1._ask_book_prices), 2)
        self.assertEqual(self.ex1._ask_book[54]['num_orders'], 1)
        self.assertEqual(self.ex1._ask_book[54]['size'], 1)
        self.assertFalse(self.ex1.traded)
        q4_cancel2 = {'order_id': 't4_2', 'timestamp': 10, 'type': 'cancel', 'quantity': 1, 'side': 'sell',
                     'price': 54}
        self.ex1.process_order(q4_cancel2)
        self.assertEqual(len(self.ex1.order_history), 8)
        self.assertEqual(len(self.ex1._ask_book_prices), 1)
        self.assertFalse(self.ex1.traded)
        # add/modify buy order
        q5_buy = {'order_id': 't5_1', 'timestamp': 10, 'type': 'add', 'quantity': 5, 'side': 'buy',
                  'price': 48}
        self.ex1.process_order(q5_buy)
        self.assertEqual(len(self.ex1.order_history), 9)
        self.assertEqual(len(self.ex1._bid_book_prices), 2)
        self.assertEqual(self.ex1._bid_book[48]['num_orders'], 1)
        self.assertEqual(self.ex1._bid_book[48]['size'], 5)
        q5_modify1 = {'order_id': 't5_1', 'timestamp': 10, 'type': 'modify', 'quantity': 2, 'side': 'buy',
                     'price': 48}
        self.ex1.process_order(q5_modify1)
        self.assertEqual(len(self.ex1.order_history), 10)
        self.assertEqual(len(self.ex1._bid_book_prices), 2)
        self.assertEqual(self.ex1._bid_book[48]['size'], 3)
        self.assertEqual(self.ex1._bid_book[48]['orders']['t5_1']['quantity'], 3)
        self.assertEqual(len(self.ex1.confirm_modify_collector), 1)
        self.assertFalse(self.ex1.traded)
        # add/modify sell order
        q5_sell = {'order_id': 't5_1', 'timestamp': 10, 'type': 'add', 'quantity': 5, 'side': 'sell',
                  'price': 54}
        self.ex1.process_order(q5_sell)
        self.assertEqual(len(self.ex1.order_history), 11)
        self.assertEqual(len(self.ex1._ask_book_prices), 2)
        self.assertEqual(self.ex1._ask_book[54]['num_orders'], 1)
        self.assertEqual(self.ex1._ask_book[54]['size'], 5)
        q5_modify2 = {'order_id': 't5_1', 'timestamp': 10, 'type': 'modify', 'quantity': 2, 'side': 'sell',
                     'price': 54}
        self.ex1.process_order(q5_modify2)
        self.assertEqual(len(self.ex1.order_history), 12)
        self.assertEqual(len(self.ex1._ask_book_prices), 2)
        self.assertEqual(self.ex1._ask_book[54]['size'], 3)
        self.assertEqual(self.ex1._ask_book[54]['orders']['t5_1']['quantity'], 3)
        self.assertEqual(len(self.ex1.confirm_modify_collector), 1)
        self.assertFalse(self.ex1.traded)

For _match_trade(), we will test buys and sells separately. The logic is the same and is documented in the comments. In general, the tests check for partial executions (one incoming order with a quantity less than the quantity available at that price), walking the book (incoming order priced to remove more than one order from the book – possibly at different prices), and making a new market. First some sell orders:

    def test_match_trade_sell(self):
        '''
        An incoming order can:
        1. take out part of an order,
        2. take out an entire price level,
        3. if priced, take out a price level and make a new inside market.
        '''
        # seed order book
        self.ex1.add_order_to_book(self.q1_buy)
        self.ex1.add_order_to_book(self.q1_sell)
        # process new orders
        self.ex1.process_order(self.q2_buy)
        self.ex1.process_order(self.q2_sell)
        self.ex1.process_order(self.q3_buy)
        self.ex1.process_order(self.q3_sell)
        self.ex1.process_order(self.q4_buy)
        self.ex1.process_order(self.q4_sell)
        # The book: bids: 2@50, 3@49, 3@47 ; asks: 2@52, 3@53, 3@55
        self.assertEqual(self.ex1._bid_book[47]['size'], 3)
        self.assertEqual(self.ex1._bid_book[49]['size'], 3)
        self.assertEqual(self.ex1._bid_book[50]['size'], 2)
        self.assertEqual(self.ex1._ask_book[52]['size'], 2)
        self.assertEqual(self.ex1._ask_book[53]['size'], 3)
        self.assertEqual(self.ex1._ask_book[55]['size'], 3)
        #self.assertFalse(self.ex1.sip_collector)
        # market sell order takes out part of first best bid
        q1 = {'order_id': 't100_1', 'timestamp': 10, 'type': 'add', 'quantity': 1, 'side': 'sell',
              'price': 0}
        self.ex1.process_order(q1)
        self.assertEqual(self.ex1._bid_book[50]['size'], 1)
        self.assertTrue(50 in self.ex1._bid_book_prices)
        self.assertEqual(self.ex1._bid_book[49]['size'], 3)
        self.assertEqual(self.ex1._bid_book[47]['size'], 3)
        self.assertEqual(self.ex1._bid_book[50]['orders'][self.ex1._bid_book[50]['order_ids'][0]]['quantity'], 1)
        #self.assertEqual(len(self.ex1.sip_collector), 1)
        # market sell order takes out remainder first best bid and all of the next level
        self.assertEqual(len(self.ex1._bid_book_prices), 3)
        q2 = {'order_id': 't100_2', 'timestamp': 11, 'type': 'add', 'quantity': 4, 'side': 'sell',
              'price': 0}
        self.ex1.process_order(q2)
        self.assertEqual(len(self.ex1._bid_book_prices), 1)
        self.assertFalse(50 in self.ex1._bid_book_prices)
        self.assertFalse(49 in self.ex1._bid_book_prices)
        self.assertTrue(47 in self.ex1._bid_book_prices)
        #self.assertEqual(len(self.ex1.sip_collector), 3)
        # make new market
        q3 = {'order_id': 't101_1', 'timestamp': 12, 'type': 'add', 'quantity': 2, 'side': 'buy',
              'price': 48}
        q4 = {'order_id': 't102_1', 'timestamp': 13, 'type': 'add', 'quantity': 3, 'side': 'sell',
              'price': 48}
        self.ex1.process_order(q3)
        self.assertEqual(len(self.ex1._bid_book_prices), 2)
        self.assertTrue(48 in self.ex1._bid_book_prices)
        self.assertTrue(47 in self.ex1._bid_book_prices)
        self.assertEqual(self.ex1._bid_book_prices[-1], 48)
        self.assertEqual(self.ex1._bid_book_prices[-2], 47)
        # sip_collector does not reset until new trade at new time
        #self.assertEqual(len(self.ex1.sip_collector), 3)
        self.ex1.process_order(q4)
        self.assertEqual(len(self.ex1._bid_book_prices), 1)
        self.assertFalse(48 in self.ex1._bid_book_prices)
        self.assertTrue(47 in self.ex1._bid_book_prices)
        self.assertEqual(len(self.ex1._ask_book_prices), 4)
        self.assertTrue(48 in self.ex1._ask_book_prices)
        self.assertEqual(self.ex1._ask_book_prices[0], 48)
        self.assertEqual(self.ex1._bid_book_prices[-1], 47)
        #self.assertEqual(len(self.ex1.sip_collector), 1)

Then some buy orders:

    def test_match_trade_buy(self):
        '''
        An incoming order can:
        1. take out part of an order,
        2. take out an entire price level,
        3. if priced, take out a price level and make a new inside market.
        '''
        # seed order book
        self.ex1.add_order_to_book(self.q1_buy)
        self.ex1.add_order_to_book(self.q1_sell)
        # process new orders
        self.ex1.process_order(self.q2_buy)
        self.ex1.process_order(self.q2_sell)
        self.ex1.process_order(self.q3_buy)
        self.ex1.process_order(self.q3_sell)
        self.ex1.process_order(self.q4_buy)
        self.ex1.process_order(self.q4_sell)
        # The book: bids: 2@50, 3@49, 3@47 ; asks: 2@52, 3@53, 3@55
        self.assertEqual(self.ex1._bid_book[47]['size'], 3)
        self.assertEqual(self.ex1._bid_book[49]['size'], 3)
        self.assertEqual(self.ex1._bid_book[50]['size'], 2)
        self.assertEqual(self.ex1._ask_book[52]['size'], 2)
        self.assertEqual(self.ex1._ask_book[53]['size'], 3)
        self.assertEqual(self.ex1._ask_book[55]['size'], 3)
        # market buy order takes out part of first best ask
        q1 = {'order_id': 't100_1', 'timestamp': 10, 'type': 'add', 'quantity': 1, 'side': 'buy',
              'price': 100000}
        self.ex1.process_order(q1)
        self.assertEqual(self.ex1._ask_book[52]['size'], 1)
        self.assertTrue(52 in self.ex1._ask_book_prices)
        self.assertEqual(self.ex1._ask_book[53]['size'], 3)
        self.assertEqual(self.ex1._ask_book[55]['size'], 3)
        self.assertEqual(self.ex1._ask_book[52]['orders'][self.ex1._ask_book[52]['order_ids'][0]]['quantity'], 1)
        # market buy order takes out remainder first best ask and all of the next level
        self.assertEqual(len(self.ex1._ask_book_prices), 3)
        q2 = {'order_id': 't100_2', 'timestamp': 11, 'type': 'add', 'quantity': 4, 'side': 'buy',
              'price': 100000}
        self.ex1.process_order(q2)
        self.assertEqual(len(self.ex1._ask_book_prices), 1)
        self.assertFalse(52 in self.ex1._ask_book_prices)
        self.assertFalse(53 in self.ex1._ask_book_prices)
        self.assertTrue(55 in self.ex1._ask_book_prices)
        # make new market
        q3 = {'order_id': 't101_1', 'timestamp': 12, 'type': 'add', 'quantity': 2, 'side': 'sell',
              'price': 54}
        q4 = {'order_id': 't102_1', 'timestamp': 13, 'type': 'add', 'quantity': 3, 'side': 'buy',
              'price': 54}
        self.ex1.process_order(q3)
        self.assertEqual(len(self.ex1._ask_book_prices), 2)
        self.assertTrue(55 in self.ex1._ask_book_prices)
        self.assertTrue(54 in self.ex1._ask_book_prices)
        self.assertEqual(self.ex1._ask_book_prices[0], 54)
        self.assertEqual(self.ex1._ask_book_prices[1], 55)
        self.ex1.process_order(q4)
        self.assertEqual(len(self.ex1._ask_book_prices), 1)
        self.assertFalse(54 in self.ex1._ask_book_prices)
        self.assertTrue(55 in self.ex1._ask_book_prices)
        self.assertEqual(len(self.ex1._bid_book_prices), 4)
        self.assertTrue(54 in self.ex1._bid_book_prices)
        self.assertEqual(self.ex1._ask_book_prices[0], 55)
        self.assertEqual(self.ex1._bid_book_prices[-1], 54)

There is also an additional test for market collapse:

    def test_market_collapse(self):
        '''
        At setup(), there is 8 total bid size and 8 total ask size
        A trade for 8 or more should collapse the market
        '''
        print('Market Collapse Tests to stdout:\n')
        # seed order book
        self.ex1.add_order_to_book(self.q1_buy)
        self.ex1.add_order_to_book(self.q1_sell)
        # process new orders
        self.ex1.process_order(self.q2_buy)
        self.ex1.process_order(self.q2_sell)
        self.ex1.process_order(self.q3_buy)
        self.ex1.process_order(self.q3_sell)
        self.ex1.process_order(self.q4_buy)
        self.ex1.process_order(self.q4_sell)
        # The book: bids: 2@50, 3@49, 3@47 ; asks: 2@52, 3@53, 3@55
        # market buy order takes out part of the asks: no collapse
        q1 = {'order_id': 't100_1', 'timestamp': 10, 'type': 'add', 'quantity': 4, 'side': 'buy',
              'price': 100000}
        self.ex1.process_order(q1)
        # next market buy order takes out the asks: market collapse
        q2 = {'order_id': 't100_2', 'timestamp': 10, 'type': 'add', 'quantity': 5, 'side': 'buy',
              'price': 100000}
        self.ex1.process_order(q2)
        # market sell order takes out part of the bids: no collapse
        q3 = {'order_id': 't100_3', 'timestamp': 10, 'type': 'add', 'quantity': 4, 'side': 'sell',
              'price': 0}
        self.ex1.process_order(q3)
        # next market sell order takes out the asks: market collapse
        q4 = {'order_id': 't100_4', 'timestamp': 10, 'type': 'add', 'quantity': 5, 'side': 'sell',
              'price': 0}
        self.ex1.process_order(q4)

This test is mostly for internal checking. Many combinations of inputs will necessarily result in market collapse (i.e., exhaustion of all orders on one side of the book). Check out the Preis et al. (2006) and Preis et al. (2007) references in the Bibliography for more details. The final test checks for posting the top-of-book:

    def test_report_top_of_book(self):
        '''
        At setup(), top of book has 2 to sell at 52 and 2 to buy at 50
        at time = 3
        '''
        self.ex1.add_order_to_book(self.q1_buy)
        self.ex1.add_order_to_book(self.q2_buy)
        self.ex1.add_order_to_book(self.q1_sell)
        self.ex1.add_order_to_book(self.q2_sell)
        tob_check = {'timestamp': 5, 'best_bid': 50, 'best_ask': 52, 'bid_size': 2, 'ask_size': 2}
        self.ex1.report_top_of_book(5)
        self.assertDictEqual(self.ex1._sip_collector[0], tob_check)

That’s it for testing the Orderbook class and associated methods. As usual, the tests are more than twice as long as the actual code to be tested. Future posts will cover the @unittest.skip() decorator and how to run tests in a loop. If you have comments or suggestions, feel free to post them. The current WordPress settings require me to approve the first comment from a specific source. After that, you are free to comment without further approval. Coming up are posts describing the Trader classes and associated tests followed by a post or two on the simulation loop.

Coding a simple limit order book with Python

I will walk through designing and coding a simple two-sided continuous auction limit order book using an object-oriented approach with Python 3. The full code is available on GitHub as orderbook3.py. The limit order book will have attributes and methods. Private attributes (things the order book has) and methods (things the order book does) are not used outside of the class declaration. These are denoted with a leading underscore. Public attributes and methods are called from other modules that import the orderbook. These have no leading underscore.

The first step is to import some python modules we will need within the class: bisect and pandas. We will use bisect.insort to maintain two ordered lists of prices – thereby maintaining price priority in both the bid and ask queue and pandas to facilitate permanent results storage in hdf5 files.

import bisect
import pandas as pd 

Next, we declare the Orderbook class and initialize the attributes. For now, we will skip the documentation comments.

class Orderbook(object):

    def __init__(self):
        self.order_history = []
        self._bid_book = {}
        self._bid_book_prices = []
        self._ask_book = {}
        self._ask_book_prices = []
        self.confirm_modify_collector = []
        self.confirm_trade_collector = []
        self._sip_collector = []
        self.trade_book = []
        self._order_index = 0
        self.traded = False 

First, Orderbook inherits from object – the default in Python 3. order_history is a list of all of the orders sequenced by arrival time. This is used to reconstruct the orderbook after the simulation is run. _bid_book_prices and _ask_book_prices are lists of existing prices in ascending order. The sorted order is established by using bisect.insort. The prices act as pointers to the two books: _bid_book and _ask_book. An example of the _ask_book_prices:

[998, 999, 1000, 1001, 1005, … , 1010]

And an example of the _ask_book:

{998: {‘num_orders’: 2, ‘size’: 5, ‘order_ids’: [id1, id2], ‘orders’: {id1: {order for id1}, id2: {order for id2}}, 999: …}

confirm_modify_collector and confirm_trade_collector are public lists that carry messages (dictionaries) to the traders. _sip_collector is a private list of dictionaries containing best bid and ask prices along with their associated sizes for each discrete event. This top-of-book information is provided to traders via a public Orderbook method called in the looping logic contained in a separate module. trade_book is a list of dictionaries containing details for each trade. _order_index is used to generate unique incremented order ids and traded is a public boolean attribute used in the simulation looping logic to determine if a trade occurred or not. Much of this will become clearer as we introduce the Orderbook methods.

There are three major types of methods in Orderbook. The actual order processing and matching is done by a public process_order method and a private _match_trade method, respectively. Several methods are helper functions called from these two main order processing methods. The remaining methods prepare and save some important data for processing after the simulation is run.

_add_order_to_history adds a unique order index to an existing order (dict) and appends the modified order to the order_history list.

    def _add_order_to_history(self, order):
        '''Add an order (dict) to order_history'''
        hist_order = {'order_id': order['order_id'], 'timestamp': order['timestamp'], 'type': order['type'],
                      'quantity': order['quantity'], 'side': order['side'], 'price': order['price']}
        self._order_index += 1
        hist_order['exid'] = self._order_index
        self.order_history.append(hist_order)

Here we can see how simple the order book really is. To extend this orderbook, we could add reserve or hidden features to the order. We would also have to modify the bookkeeping and matching logic that follows. Note that hist_order is just a hand-written copy of the order parameter. This is much faster than using copy.deepcopy().

add_order_to_book performs all of the book maintenance for incoming orders that do not result in a full trade (i.e., either the order is not priced to trade or the size is not fully exhausted if it is priced to trade).

    def add_order_to_book(self, order):
        '''
        Use insort to maintain on ordered list of prices which serve as pointers
        to the orders.
        '''
        book_order = {'order_id': order['order_id'], 'timestamp': order['timestamp'], 'type': order['type'],
                      'quantity': order['quantity'], 'side': order['side'], 'price': order['price']}
        if order['side'] == 'buy':
            book_prices = self._bid_book_prices
            book = self._bid_book
        else:
            book_prices = self._ask_book_prices
            book = self._ask_book
        if order['price'] in book_prices:
            book[order['price']]['num_orders'] += 1
            book[order['price']]['size'] += order['quantity']
            book[order['price']]['order_ids'].append(order['order_id'])
            book[order['price']]['orders'][order['order_id']] = book_order
        else:
            bisect.insort(book_prices, order['price'])
            book[order['price']] = {'num_orders': 1, 'size': order['quantity'], 'order_ids': [order['order_id']],
                                    'orders': {order['order_id']: book_order}}

Again, the incoming order is copied to a new dict object. The order side determines which book we are using: bid or ask. Then we check if the order price is in the list of book_prices – a very expensive task that would take longer if we were to check for “not in” prices. If the order price is already in the book, the order book is updated with the new information. If not, a new price is inserted in the proper sorted slot and the book is (re-)established for the new price.

_remove_order removes an order from the order book and removes the price from the price list if removal results in an empty book for that price. Maintaining a list of valid prices instead of merely keeping all of the prices (with some prices pointing to empty books) speeds up the trade matching algorithm.

    def _remove_order(self, order_side, order_price, order_id):
        '''Pop the order_id; if  order_id exists, updates the book.'''
        if order_side == 'buy':
            book_prices = self._bid_book_prices
            book = self._bid_book
        else:
            book_prices = self._ask_book_prices
            book = self._ask_book
        is_order = book[order_price]['orders'].pop(order_id, None)
        if is_order:
            book[order_price]['num_orders'] -= 1
            book[order_price]['size'] -= is_order['quantity']
            book[order_price]['order_ids'].remove(is_order['order_id'])
            if book[order_price]['num_orders'] == 0:
                book_prices.remove(order_price)

_modify_order behaves similarly, but also checks if the modify actually results in removal.

    def _modify_order(self, order_side, order_quantity, order_id, order_price):
        '''Modify order quantity; if quantity is 0, removes the order.'''
        book = self._bid_book if order_side == 'buy' else self._ask_book        
        if order_quantity < book[order_price]['orders'][order_id]['quantity']:
            book[order_price]['size'] -= order_quantity
            book[order_price]['orders'][order_id]['quantity'] -= order_quantity
        else:
            self._remove_order(order_side, order_price, order_id)

_add_trade_to_book is a helper function that facilitates trade bookkeeping.

    def _add_trade_to_book(self, resting_order_id, resting_timestamp, incoming_order_id, timestamp, price, quantity, side):
        '''Add trades (dicts) to the trade_book list.'''
        self.trade_book.append({'resting_order_id': resting_order_id, 'resting_timestamp': resting_timestamp, 
                                'incoming_order_id': incoming_order_id, 'timestamp': timestamp, 'price': price,
                                'quantity': quantity, 'side': side})

_confirm_trade and _confirm_modify are helper functions that append trade or modify messages to a list that is conveyed to the traders.

    def _confirm_trade(self, timestamp, order_side, order_quantity, order_id, order_price):
        '''Add trade confirmation to confirm_trade_collector list.'''
        trader = order_id.partition('_')[0]
        self.confirm_trade_collector.append({'timestamp': timestamp, 'trader': trader, 'order_id': order_id, 
                                             'quantity': order_quantity, 'side': order_side, 'price': order_price})
    
    def _confirm_modify(self, timestamp, order_side, order_quantity, order_id):
        '''Add modify confirmation to confirm_modify_collector list.'''
        trader = order_id.partition('_')[0]
        self.confirm_modify_collector.append({'timestamp': timestamp, 'trader': trader, 'order_id': order_id, 
                                              'quantity': order_quantity, 'side': order_side})

process_order determines whether an incoming order results in a match with a resting order or not.

    def process_order(self, order):
        '''Check for a trade (match); if so call _match_trade, otherwise modify book(s).'''
        self.confirm_modify_collector.clear()
        self.traded = False
        self._add_order_to_history(order)
        if order['type'] == 'add':
            if order['side'] == 'buy':
                if order['price'] >= self._ask_book_prices[0]:
                    self._match_trade(order)
                else:
                    self.add_order_to_book(order)
            else: #order['side'] == 'sell'
                if order['price'] <= self._bid_book_prices[-1]:
                    self._match_trade(order)
                else:
                    self.add_order_to_book(order)
        else:
            book_prices = self._bid_book_prices if order['side'] == 'buy' else self._ask_book_prices
            if order['price'] in book_prices:
                book = self._bid_book if order['side'] == 'buy' else self._ask_book
                if order['order_id'] in book[order['price']]['orders']:
                    self._confirm_modify(order['timestamp'], order['side'], order['quantity'], order['order_id'])
                    if order['type'] == 'cancel':
                        self._remove_order(order['side'], order['price'], order['order_id'])
                    else: #order['type'] == 'modify'
                        self._modify_order(order['side'], order['quantity'], order['order_id'], order['price'])

It does some bookkeeping then checks the type of order. If it is an add order, it results in a trade if it is priced to match an existing order. This is assessed by checking the order price against the best bid (_bid_book_prices[-1]) or best ask (_ask_book_prices[0]). If it is not an add order, then it must be a cancel or modify and the order book is updated and messages are created for the trader.

_match_trade enforces price-time priority for matching incoming orders against resting orders.

    def _match_trade(self, order):
        '''Match orders to generate trades, update books.'''
        self.traded = True
        self.confirm_trade_collector.clear()
        if order['side'] == 'buy':
            book_prices = self._ask_book_prices
            book = self._ask_book
            remainder = order['quantity']
            while remainder > 0:
                if book_prices:
                    price = book_prices[0]
                    if order['price'] >= price:
                        book_order_id = book[price]['order_ids'][0]
                        book_order = book[price]['orders'][book_order_id]
                        if remainder >= book_order['quantity']:
                            self._confirm_trade(order['timestamp'], book_order['side'], book_order['quantity'], book_order['order_id'], book_order['price'])
                            self._add_trade_to_book(book_order['order_id'], book_order['timestamp'], order['order_id'], order['timestamp'], book_order['price'], 
                                                    book_order['quantity'], order['side'])
                            self._remove_order(book_order['side'], book_order['price'], book_order['order_id'])
                            remainder -= book_order['quantity']
                        else:
                            self._confirm_trade(order['timestamp'], book_order['side'], remainder, book_order['order_id'], book_order['price'])
                            self._add_trade_to_book(book_order['order_id'], book_order['timestamp'], order['order_id'], order['timestamp'], book_order['price'],
                                                    remainder, order['side'])
                            self._modify_order(book_order['side'], remainder, book_order['order_id'], book_order['price'])
                            break
                    else:
                        order['quantity'] = remainder
                        self.add_order_to_book(order)
                        break
                else:
                    print('Ask Market Collapse with order {0}'.format(order))
                    break
        else: #order['side'] =='sell'
            book_prices = self._bid_book_prices
            book = self._bid_book
            remainder = order['quantity']
            while remainder > 0:
                if book_prices:
                    price = book_prices[-1]
                    if order['price'] <= price:
                        book_order_id = book[price]['order_ids'][0]
                        book_order = book[price]['orders'][book_order_id] 
                        if remainder >= book_order['quantity']:
                            self._confirm_trade(order['timestamp'], book_order['side'], book_order['quantity'], book_order['order_id'], book_order['price'])
                            self._add_trade_to_book(book_order['order_id'], book_order['timestamp'], order['order_id'], order['timestamp'], book_order['price'],
                                                    book_order['quantity'], order['side'])
                            self._remove_order(book_order['side'], book_order['price'], book_order['order_id'])
                            remainder -= book_order['quantity']
                        else:
                            self._confirm_trade(order['timestamp'], book_order['side'], remainder, book_order['order_id'], book_order['price'])
                            self._add_trade_to_book(book_order['order_id'], book_order['timestamp'], order['order_id'], order['timestamp'], book_order['price'],
                                                    remainder, order['side'])
                            self._modify_order(book_order['side'], remainder, book_order['order_id'], book_order['price'])
                            break
                    else:
                        order['quantity'] = remainder
                        self.add_order_to_book(order)
                        break
                else:
                    print('Bid Market Collapse with order {0}'.format(order))
                    break

It does a little bookkeeping then checks whether the incoming order is a buy or sell. The “while” loops ensure price priority by checking for the best price pointer (price = book_prices[0], for example), then ensures time priority by walking through the resting orders in the order of arrival for each price (book_order_id = book[price][‘order_ids’][0]; book_order = book[price][‘orders’][book_order_id]). The remaining portions of the while loop check if the remaining order size is greater than the size available for the current best price and behaves accordingly.

Three helper functions facilitate saving data to an hdf5 for use after the simulation has ended.

    def order_history_to_h5(self, filename):
        '''Append order history to an h5 file, clear the order_history'''
        temp_df = pd.DataFrame(self.order_history)
        temp_df.to_hdf(filename, 'orders', append=True, format='table', complevel=5, complib='blosc', 
                       min_itemsize={'order_id': 12}) 
        self.order_history.clear()
        
    def trade_book_to_h5(self, filename):
        '''Append trade_book to an h5 file, clear the trade_book'''
        temp_df = pd.DataFrame(self.trade_book)
        temp_df.to_hdf(filename, 'trades', append=True, format='table', complevel=5, complib='blosc', 
                       min_itemsize={'resting_order_id': 12, 'incoming_order_id': 12}) 
        self.trade_book.clear()
        
    def sip_to_h5(self, filename):
        '''Append _sip_collector to an h5 file, clear the _sip_collector'''
        temp_df = pd.DataFrame(self._sip_collector)
        temp_df.to_hdf(filename, 'tob', append=True, format='table', complevel=5, complib='blosc')
        self._sip_collector.clear()

The final function is a public method for conveying the top of book information to the traders.

    def report_top_of_book(self, now_time):
        '''Update the top-of-book prices and sizes'''
        best_bid_price = self._bid_book_prices[-1]
        best_bid_size = self._bid_book[best_bid_price]['size']   
        best_ask_price = self._ask_book_prices[0]
        best_ask_size = self._ask_book[best_ask_price]['size']
        tob = {'timestamp': now_time, 'best_bid': best_bid_price, 'best_ask': best_ask_price, 'bid_size': best_bid_size, 'ask_size': best_ask_size}
        self._sip_collector.append(tob)
        return tob

That is it! Easy? Maybe not the first time. Creating order book code is an iterative process, even when a lot of planning and forethought is applied and even with a lot of prior knowledge about how order books are actually created by professional trading firms. The logic here can be extended to include more order information like hidden or iceberg orders. The order processing, trade matching and bookkeeping would have to be updated as well. Adding more functionality like pegged or sliding features would require considerable modification to the order processing and trade matching algorithms. But it can be done! And finally, the basic organization of this order book module can be applied to other matching mechanisms like auctions or dealer markets. Simulations will always require a module or set of functions to determine which agents traded and the prices the agents received.

Next posts will cover unit testing and designing various trader agents with Python.

[2/12/2018: Updated to properly format “less than” in code blocks.]