Tuesday, February 28, 2012

Weird stuff in high frequency markets

On the left is a graph from a really neat paper, "Low-Latency Trading" by Joel Hasbrouck and Gideon Saar (2011). You're looking at the flow of "messages"--limit orders placed or canceled--on the NASDAQ.  The x axis is time, modulo 10 seconds. So, you're looking at the typical flow of messages over any 10 second time interval.

As you can see, there is a big crush of messages on the top of the second, which rapidly tails off in the milliseconds following the even second. There is a second surge between 500 and 600 milliseconds.

Evidently, lots of computer programs reach out and look at the markets once per second, or once per half second. The programs clocks are tightly synchronized to the exchange's clock, so if you program a computer "go look once per second," it's likely to go look exactly on the second (or half second). The result is a flurry of activity on the even second.

 It's likely the even-second traders are what Joel and Gideon call "Agency traders." They're trying to buy or sell a given quantity, but spread it out to avoid price impact. Their on-the-second activity spawns a flurry of responses from the high frequency traders, whose computers monitor markets constantly.

There's a natural question: Is this an accident, or is there intentional "on the second" bunching? You can see that a programmer who didn't think about it would check once per second, not realizing that means exactly on the top of the second. But sometimes there is more liquidity when we all agree to meet at the same time. Volume has always been higher at the open and close.  Joel and Gideon show the pattern lasted from 2007 to 2008, so was not an obvious short-term programming bug.  (Do notice the vertical scale however. The range is from 9 to 13, not 0 to 13.) I'd be curious to know if it's still going on.

Here's another one, found by one of my students on nanex.net here. (Teaching has many benefits when the students know more about markets than you do!).

You're looking at bids, asks, and (white dot) trades in the natural gas futures markets. From nanex:

On June 8, 2011, starting at 19:39 Eastern Time, trade prices began oscillating almost harmonically along with the depth of book. However, prices rose as bid were executed, and prices declined when offers were executed .....price oscillates from low to high when trades are executing against the highest bid price level. After reaching a peak, prices then move down as trades execute against the highest ask price level. This is completely opposite of normal market behavior....It's almost as if someone is executing a new algorithm that has it's buying/selling signals crossed. Most disturbing to us is the high volume violent sell off that affects not only the natural gas market, but all the other trading instruments related to it.
I'm generally give efficient markets the benefit of doutbt, but it's hard not to suspect that some programming bugs are working against each other here. It's hard enough to debug a program to work alone, but when 17 programs work against each other all sorts of interesting weirdness can spill out. I am reminded of work in game theory in which computer programs fight out the prisoner's dilemma and all sorts of weird stuff erupts. If so, this will settle down, but it may take a while.

The Economist reports an interesting related story.
ON FEBRUARY 3RD 2010, at 1.26.28 pm, an automated trading system operated by a high-frequency trader (HFT) called Infinium Capital Management malfunctioned. Over the next three seconds it entered 6,767 individual orders to buy light sweet crude oil futures... Enough of those orders were filled to send the market jolting upwards.
A NYMEX business-conduct panel investigated what happened that day.... Infinium had finished writing the algorithm only the day before it introduced it to the market, and had tested it for only a couple of hours in a simulated trading environment to see how it would perform. .... When the algorithm started its frenetic buying spree, the measures designed to shut it down automatically did not work. One was supposed to turn the system off if a maximum order size was breached, but because the machine was placing lots of small orders rather than a single big one the shut-down was not triggered. The other measure was meant to prevent Infinium from selling or buying more than a certain number of contracts, but because of an error in the way the rogue algorithm had been written, this, too, failed to spot a problem. ..
High frequency trading presents a lot of interesting puzzles. The Booth faculty lunchroom has hosted some interesting discussions: "what possible social use is it to have price discovery in a microsecond instead of a millisecond?" "I don't know, but there's a theorem that says if it's profitable it's socially beneficial." "Not if there are externalities" "Ok, where's the externality?" At which point we all agree we don't know what the heck is going on.

There is also the more prosaic question whether high frequency traders "provide liquidity" and thus are in some sense beneficial to markets, or if they are somehow making markets worse. A question for another day (there is some interesting new research).

There are lots of reports of how profitable it is. But high frequency trading is a zero sum game. Anything you do in milliseconds can only talk to another computer. By definition, they can't all be making money off each other. 


  1. High Frequency is a zero sum game if all hf traders are trading with each other. But since HF as a whole is trading with other low frequency traders, it is a high expected value game.

  2. "I am reminded of work in game theory in which computer programs fight out the prisoner's dilemma and all sorts of weird stuff erupts."

    What like cooperation being the dominant strategy? Why is that "weird"?

    Timothy Watson

    1. I believe he might be referring to the "Iterated Prisoners Dilemma Competition," which is a yearly programming/game theory challenge. I think it was Bob Axelrod who started it.

      Anyways, tit-for-tat won every years for two decades. Recently a group brilliantly beat the system by encoding a signal through which their programs could locate eachother in the game and cooperate, building up one program to be the best.

      It's a pretty impressive little project.

  3. This comment has been removed by the author.

  4. HFT Is not a zero-sum game:



    1. Those posts seem only to support the argument that HFT helps ensure market efficiency; I think most economists would agree with that.

    2. you read those articles wrong if you believe they exemplify efficient markets. No one in the real trading world cares what an "Economist" thinks, They're laws aren't real laws, they fit their views and most don't put their money where their mouth is. Go back and read more or re-read them because you completely missed the point being made.

  5. As a LFT-buysider, anything that adds liquidity is good for me. Any behavior by poorly programmed HFTs is good because I can take advantage of their 'unusual' sells/buys in securities I own.

    As with any other industry the -EV programs and their backers will go bust and disappear.

    Anyone who argues that the milliseconds we've all become used to is okay, but microseconds is not, needs to find a real job.

  6. Are flash crashes a negative externality or an opportunity for "slow" traders? If trades can get cancelled after the fact, does this mean that there's no opportunity and so it is an externality?

  7. @morgan, with the correct software, you'll never even see it happen. The best thing to do is track it and find instances where the probably is high for something to go wrong then have your trigger ready. This paper from Nanex, the makers of the software I use for HFT arbitrage, co-authored paper about what you are asking. Hopefully this will help you buddy, good luck.


  8. here's more evidence:





  9. I think it's less of a software issue, but an issue of market structure. When some traders can arbitrage for profit without risk against other traders orders, there is a problem. That's the case on the SEC side of the market. It's so fragmented that HFT traders in dark pools et al can take advantage.

    On the CFTC side of the market, their advantage comes from co-location. Since they are faster, and only a fixed amount can co-locate, there are negative externalities that occur in the trading community.

    Themis Trading and I have been blogging about this for a long time.

  10. "The Booth faculty lunchroom has hosted some interesting discussions: "what possible social use is it to have price discovery in a microsecond instead of a millisecond?" "I don't know, but there's a theorem that says if it's profitable it's socially beneficial.""

    Perhaps you should ask IT people with knowledge of complex information infrastructure.

    Here is what I believe is going on: Toddlers playing with matches and dynamite. It's all fun and games until someone else (the taxpayer) loses their hand. It may not look like it, but I am making a serious comment.

    In a normal IT infrastructure you try (but often don't succeed, even if you try hard) to make components work together and align to a common cause; achieving company goals (messages, buffers & queues, cpu, memory, throughput, latency).

    In a trading infrastructure you try to make components work together to have an as fast as possible beam to a single point (the market) and basically wage the equivalent of a full scale electronic Distributed Denial of Service attack by different botnets.

    All these algorithms are clumsily trying to play on each other (like the fish eats the fish eats the fish) and the market curves generating these types of higher harmonics disintegrating into chaotic behavior. This has nothing anymore to do with the 'efficiency' of markets where true value is priced but in a perversion of it. The only thing that will remain if unchecked is a wasteland roamed by algorithms.

  11. Is there an alternative market which those who ultimately pay the costs of HFT, the gas producers and consumers, could choose if they find the burden imposed by HFT on this market excessive? The market is undoubtedly better than burdensome bilateral contracts, but what alternative is there between the two? If there is an alternate market, and producers and consumers choose to return to this one, they have voted with their therms. If there is no such alternate, and HFT is truly deleterious, there is an opportunity for someone to make a slower, cheaper market.

  12. Suppose that, absent HFT, adverse-selection is a substantial cost associated with price quotes in limit order markets. HFT entry might then increase welfare as rational investors 'use' them to reduce such cost. Boyan Jovanovic and I develop this argument more formally in: http://goo.gl/8ap47.

  13. is it related with brownian motion


Comments are welcome. Keep it short, polite, and on topic.

Thanks to a few abusers I am now moderating comments. I welcome thoughtful disagreement. I will block comments with insulting or abusive language. I'm also blocking totally inane comments. Try to make some sense. I am much more likely to allow critical comments if you have the honesty and courage to use your real name.