Coding the agent-based model simulation loop with Python

This blog continues the coding project in support of replicating the results in my Tick Pilot Agent-Based Modeling paper. The first and second blogs created and tested the limit order book. The third and fourth blogs created and tested the traders. The next step is to pull the book and traders together and run a simulation. The strategy is designed to enable a user to install a package, import the package, and instantiate from a Runner class. For example, from the command line:

~$ conda install pyziabm

The user might have to specify the conda repo or download and install from local. See the Tick Pilot ABM project website for more details. Then from IPython or a Jupyter Notebook:

import pyziabm as pzi
pzi.Runner()

This command would run the simulation with a set of defaults and store some results in a table in an hdf5 file. The defaults are all keywords. The user can change the defaults by calling Runner with the keywords updated – in the spirit of how matplotlib gets things done:

pzi.Runner(mpi=1, h5filename='test2.h5', pj=True, alpha_pj=0.01)

The full code is available on GitHub as runner2017mpi_r4.py. As usual, the first step is to import some python packages. The traders and the orderbook were designed to be imported by the simulation module. We will import those as well.

import random
import numpy as np
import pandas as pd

from pyziabm.orderbook3 import Orderbook
from pyziabm.trader2017_r3 import Provider, Provider5, Taker, MarketMaker, MarketMaker5, PennyJumper

The __init__() method does all of the work in four major steps: create some useful attributes for later use, create the traders, orderbook and information environment, set up and run the simulation, and save some output. The first portion of __init__() demonstrates the keyword strategy and creates some attributes.

    def __init__(self, prime1=20, num_mms=1, mm_maxq=1, mm_quotes=12, mm_quote_range=60, mm_delta=0.025, 
                 num_takers=50, taker_maxq=1, num_providers=38, provider_maxq=1, q_provide=0.5,
                 alpha=0.0375, mu=0.001, delta=0.025, lambda0=100, wn=0.001, c_lambda=1.0, run_steps=100000,
                 mpi=5, h5filename='test.h5', pj=False, alpha_pj=0):
        self.alpha_pj = alpha_pj
        self.q_provide = q_provide
        self.lambda0 = lambda0
        self.run_steps = run_steps+1
        self.h5filename = h5filename

The second portion creates the traders and their arrival intervals, the order book and the information environment.

        self.t_delta_t, self.taker_array = self.make_taker_array(taker_maxq, num_takers, mu)
        self.t_delta_p, self.provider_array = self.make_provider_array(provider_maxq, num_providers, delta, mpi, alpha)
        self.t_delta_m, self.marketmaker_array = self.make_marketmaker_array(mm_maxq, num_mms, mm_quotes, mm_quote_range, mm_delta, mpi)
        self.pennyjumper = self.make_pennyjumper(mpi)
        self.exchange = Orderbook()
        self.q_take, self.lambda_t = self.make_q_take(wn, c_lambda)
        self.trader_dict = self.make_traders(num_takers, num_providers, num_mms)

The final portion prepares and runs the simulation and then saves output.

        self.seed_orderbook()
        self.make_setup(prime1)
        if pj:
            self.run_mcsPJ(prime1)
        else:
            self.run_mcs(prime1)
        self.exchange.trade_book_to_h5(h5filename)
        self.out_to_h5()

We will take each of these steps in order and I will provide a brief overview of what’s going on in each of the methods. See the Tick Pilot Agent-Based Modeling paper for further details and a full description of the agents and the simulation strategy.

The Traders

The make_taker_array(…) method creates the taker agents and their arrival intervals. The first three lines of code determine the trade size (size = 1 in the paper). The fourth line creates the random arrival intervals and the fifth and sixth lines prepare and create the Taker instances and store them in a numpy array. The arrival intervals are permanently associated with specific Taker instances via numpy arrays. We will make use of this later.

    def make_taker_array(self, maxq, num_takers, mu):
        default_arr = np.array([1, 5, 10, 25, 50])
        actual_arr = default_arr[default_arr<=maxq]
        taker_size = np.random.choice(actual_arr, num_takers)
        t_delta_t = np.floor(np.random.exponential(1/mu, num_takers)+1)*taker_size
        takers_list = ['t%i' % i for i in range(num_takers)]
        takers = np.array([Taker(t,i) for t,i in zip(takers_list,taker_size)])
        return t_delta_t, takers

The make_provider_array(…) method follows a similar strategy while using an if block to specify whether the provider should use a unit (penny) pricing increment or a 5 unit increment.

    def make_provider_array(self, maxq, num_providers, delta, mpi, alpha):
        default_arr = np.array([1, 5, 10, 25, 50])
        actual_arr = default_arr[default_arr<=maxq]
        provider_size = np.random.choice(actual_arr, num_providers)
        t_delta_p = np.floor(np.random.exponential(1/alpha, num_providers)+1)*provider_size
        providers_list = ['p%i' % i for i in range(num_providers)]
        if mpi==1:
            providers = np.array([Provider(p,i,mpi,delta) for p,i in zip(providers_list,provider_size)])
        else:
            providers = np.array([Provider5(p,i,mpi,delta) for p,i in zip(providers_list,provider_size)])
        return t_delta_p, providers

The make_marketmaker_array(…) method also follows the same strategy. The market maker arrival interval is the same as the trade size. In the paper, the single market maker has a trade size of one and therefore appears once every simulation step.

    def make_marketmaker_array(self, maxq, num_mms, mm_quotes, mm_quote_range, mm_delta, mpi):
        default_arr = np.array([1, 5, 10, 25, 50])
        actual_arr = default_arr[default_arr<=maxq]
        provider_size = np.random.choice(actual_arr, num_mms)
        t_delta_m = maxq
        marketmakers_list = ['m%i' % i for i in range(num_mms)]
        if mpi==1:
            marketmakers = np.array([MarketMaker(p,i,mpi,mm_delta,mm_quotes,mm_quote_range) for p,i in zip(marketmakers_list,provider_size)])
        else:
            marketmakers = np.array([MarketMaker5(p,i,mpi,mm_delta,mm_quotes,mm_quote_range) for p,i in zip(marketmakers_list,provider_size)])
        return t_delta_m, marketmakers

The make_pennyjumper(…) method merely returns the single instance of the Penny Jumper.

    def make_pennyjumper(self, mpi):
        return PennyJumper('j0', 1, mpi)

The Information Environment

The information environment includes a vector, q_take, that determines the probability a taker will submit a buy order and a vector, lambda_t, that serves as a parameter for a method that modifies the exponential distribution from which the Providers choose their prices.

    def make_q_take(self, s, c_lambda):
        noise = np.random.rand(2,self.run_steps)
        qt_take = np.empty_like(noise)
        qt_take[:,0] = 0.5
        for i in range(1,self.run_steps):
            qt_take[:,i] = qt_take[:,i-1] + (noise[:,i-1]>qt_take[:,i-1])*s - (noise[:,i-1]<qt_take[:,i-1])*s
        lambda_t = -self.lambda0*(1 + (np.abs(qt_take[1] - 0.5)/np.sqrt(np.mean(np.square(qt_take[0] - 0.5))))*c_lambda)
        return qt_take[1], lambda_t

Preparing the Orderbook

Preparing the order book for the simulation involves seeding the book with one ask order and one bid order and then priming the book for twenty steps with just the Providers participating. The seed_orderbook(…) method accomplishes the seeding. The make_setup(…) method calls make_providers(…) to prime the book. For each time step, make_setup(…) calls make_providers(…) and loops through the returned active Providers: the Provider processes the top-of-book signal; the Exchange (orderbook) processes the Provider order and then updates the top-of-book, which serves as an input for the next step through the list of active Providers. make_providers(…) uses np.remainder() on the arrival interval vector to determine which Providers are active in any particular step. We will re-use this strategy in the main simulation loop to follow.

    def seed_orderbook(self):
        seed_provider = Provider('p999999', 1, 5, 0.05)
        self.trader_dict.update({'p999999': seed_provider})
        ba = random.choice(range(1000005, 1002001, 5))
        bb = random.choice(range(997995, 999996, 5))
        qask = {'order_id': 'p999999_a', 'timestamp': 0, 'type': 'add', 'quantity': 1, 'side': 'sell',
              'price': ba, 'exid': 99999999}
        qbid = {'order_id': 'p999999_b', 'timestamp': 0, 'type': 'add', 'quantity': 1, 'side': 'buy',
              'price': bb, 'exid': 99999999}
        seed_provider.local_book['p999999_a'] = qask
        self.exchange.add_order_to_book(qask)
        self.exchange.order_history.append(qask)
        seed_provider.local_book['p999999_b'] = qbid
        self.exchange.add_order_to_book(qbid)
        self.exchange.order_history.append(qbid)

    def make_setup(self, prime1):
        top_of_book = self.exchange.report_top_of_book(0)
        for current_time in range(1, prime1):
            for p in self.make_providers(current_time):
                p.process_signal(current_time, top_of_book, self.q_provide, -self.lambda0)
                self.exchange.process_order(p.quote_collector[-1])
                top_of_book = self.exchange.report_top_of_book(current_time)

    def make_providers(self, step):
        providers = self.provider_array[np.remainder(step, self.t_delta_p)==0]
        np.random.shuffle(providers)
        return providers

Running the Simulation

The run_mcs(…) method steps through the remaining time, calling make_both(…) to determine which traders will participate in the time step and to randomize them (a misnomer, should name the method make_all(…)). A series of actions are specified as a function of trader type. Providers and MarketMakers add orders if their arrival interval matches the time step (that’s what “if row[1]:” determines) and potentially cancel orders regardless of whether their interval matches the time step or not. Takers add orders, too. A make_traders(…) method creates a dictionary of trader objects and their ids, thereby enabling liquidity provider lookup when a taker takes liquidity. This facilitates sending confirm messages to the liquidity providers when one of their resting orders is hit. The final block of code stores some of the larger history objects to an hdf5 file and resets the containers to empty.

    def run_mcs(self, prime1):
        top_of_book = self.exchange.report_top_of_book(prime1)
        for current_time in range(prime1, self.run_steps):
            for row in self.make_both(current_time):
                if row[0].trader_type == 'Provider':
                    if row[1]:
                        row[0].process_signal(current_time, top_of_book, self.q_provide, self.lambda_t[current_time])
                        self.exchange.process_order(row[0].quote_collector[-1])
                        top_of_book = self.exchange.report_top_of_book(current_time)
                    row[0].bulk_cancel(current_time)
                    if row[0].cancel_collector:
                        for c in row[0].cancel_collector:
                            self.exchange.process_order(c)
                            if self.exchange.confirm_modify_collector:
                                row[0].confirm_cancel_local(self.exchange.confirm_modify_collector[0])
                        top_of_book = self.exchange.report_top_of_book(current_time)
                elif row[0].trader_type == 'MarketMaker':
                    if row[1]:
                        row[0].process_signal(current_time, top_of_book, self.q_provide)
                        for q in row[0].quote_collector:
                            self.exchange.process_order(q)
                        top_of_book = self.exchange.report_top_of_book(current_time)
                    row[0].bulk_cancel(current_time)
                    if row[0].cancel_collector:
                        for c in row[0].cancel_collector:
                            self.exchange.process_order(c)
                            if self.exchange.confirm_modify_collector:
                                row[0].confirm_cancel_local(self.exchange.confirm_modify_collector[0])
                        top_of_book = self.exchange.report_top_of_book(current_time)
                else:
                    row[0].process_signal(current_time, self.q_take[current_time])
                    self.exchange.process_order(row[0].quote_collector[-1])
                    if self.exchange.traded:
                        for c in self.exchange.confirm_trade_collector:
                            trader = self.trader_dict[c['trader']]
                            trader.confirm_trade_local(c)
                        top_of_book = self.exchange.report_top_of_book(current_time)
            if not np.remainder(current_time, 2000):
                self.exchange.order_history_to_h5(self.h5filename)
                self.exchange.sip_to_h5(self.h5filename)

    def make_both(self, step):
        providers_mask = np.remainder(step, self.t_delta_p)==0
        takers_mask = np.remainder(step, self.t_delta_t)==0
        marketmakers_mask = np.remainder(step, self.t_delta_m)==0
        providers = np.vstack((self.provider_array, providers_mask)).T
        takers = np.vstack((self.taker_array, takers_mask)).T
        marketmakers = np.vstack((self.marketmaker_array, marketmakers_mask)).T
        traders = np.vstack((providers, marketmakers, takers[takers_mask]))
        np.random.shuffle(traders)
        return traders

    def make_traders(self, num_takers, num_providers, num_mms):
        takers_dict = dict(zip(['t%i' % i for i in range(num_takers)], list(self.taker_array)))
        providers_dict = dict(zip(['p%i' % i for i in range(num_providers)], list(self.provider_array)))
        takers_dict.update(providers_dict)
        marketmakers_dict = dict(zip(['m%i' % i for i in range(num_mms)], list(self.marketmaker_array)))
        takers_dict.update(marketmakers_dict)
        if self.alpha_pj > 0:
            takers_dict.update({'j0': self.pennyjumper})
        return takers_dict

The run_mcsPJ(…) method has an additional block of code at the end of each step through the traders. This code determines whether a PennyJumper will be active after a trader shows up. If so, the PennyJumper has an opportunity to add and/or cancel orders. Note that the PennyJumper can participate zero, one, or many times during each time step.

    def run_mcsPJ(self, prime1):

...

                if random.uniform(0,1) < self.alpha_pj:
                    self.pennyjumper.process_signal(current_time, top_of_book, self.q_take[current_time])
                    if self.pennyjumper.cancel_collector:
                        for c in self.pennyjumper.cancel_collector:
                            self.exchange.process_order(c)
                    if self.pennyjumper.quote_collector:
                        for q in self.pennyjumper.quote_collector:
                            self.exchange.process_order(q)
                    top_of_book = self.exchange.report_top_of_book(current_time)

...

The final step saves some results.

    def qtake_to_h5(self):
        temp_df = pd.DataFrame({'qt_take': self.q_take, 'lambda_t': self.lambda_t})
        temp_df.to_hdf(self.h5filename, 'qtl', append=True, format='table', complevel=5, complib='blosc')
        
    def mm_profitability_to_h5(self):
        for m in self.marketmaker_array:
            temp_df = pd.DataFrame(m.cash_flow_collector)
            temp_df.to_hdf(self.h5filename, 'mmp', append=True, format='table', complevel=5, complib='blosc')
            
    def out_to_h5(self):
        self.qtake_to_h5()
        self.mm_profitability_to_h5()

If you have comments or suggestions, feel free to post them. Coming up are posts describing the wrapper file to replicate the results in the Working Paper and some notes on a simple Conda build process for creating a package.

Published by

Chuck Collver

Quant, Programmer, Data Scientist, Developer

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s