[R-SIG-Finance] Question for IB TWS users [long]

[Bill Pippin]

Mel,

With respect connecting to the IB tws, where you are able to open a socket,
and write to it, but do not receive the handshake response in time:

> I tried to wrote a little function establishing a socketConnection ...

> connectToTWS = function()
> {
        ...
> res = readLines(con, 1);    # would like to get something here !
> print(res);                 # but always character(0)
        ...
> }

If we grant that the design approach you take is feasible, about which more
later, then you need a timeout.  It looks like your code has a race condition;
you are likely to read from the socket before results have arrived.  If
you are determined to take this approach, you may want to insert sleep
delays here and there, at least for debugging purposes.

________________________________________________________________________________

Summary of what follows:

    * The IB tws api is not, in my opinion, convenient for synchronous use
      by a controlling R process

    * Another system, the trading shim, provides a command interpreter input
      interface, though message responses from the tws may be out of order,
      so that it can not achieve a true RPC interface.

    * The two ways I see to make the IB tws api accessible from R involve
      either:

        + limiting ourselves to synchronous requests only (e.g., history),
          and accepting high latencies, awkward error-handling, and clumsy
          timeouts

          OR

        + providing full scheduler task loop logic on the client side,
          mirroring similar code in the shim.

    * The trading shim can be used to support either of the above approaches.

    * I'm interested in knowing if anyone on this list has comments on
      how the shim should be integrated with R.

________________________________________________________________________________

Off topic warning:

*** Much of the following may or may not be of interest to most R users,
*** and in case that be true, I suggest an alternative list near the end
*** of this post.

Now, about the design.  Viewing the IB servers and external markets as
upstream, the problem is how to achieve downstream control of the tws
interface.

It would certainly be convenient to have synchronous control of the IB tws,
in effect a remote procedure call interface, though unfortunately, I believe
this to be infeasible.

After all, the tick data streams are asynchronous, and the delays for history
data can be worst case unpleasantly long.   If you want to combine such
queries and subscriptions with orders, the various latencies are all over
the map, and in any case orders themselves are fundamentally async; you may
be able to shoehorn the common case into a sync inteface, but it's the
uncommon case that will eat you alive.  And finally, how do you integrate error
handling and timeouts?

Ulimately, I believe you need another process between the client code and
the IB tws, one that simplifies the IB tws protocol, and converts between
text for the downstream, and the binary tws api.  I also accept that, for
the general case, the resulting text outupt must be processed by downstream
programs that are themselves able to handle asynchronous events, such as
timeouts and error messages.

We're working on such a system, the trading-shim, and to make a long story
short, it boils down to a database-augmented command intepreter.  The database
works to simplify the command protocol; the text commands from the downstream
can be short and simple, since they can use database keys to stand for
contracts, order line items, and the like.  The command interpreter also
converts the binary IB tws api to a text stream, so that standard filters
become applicable.

Please consider joining the ts-general at trading-shim.org mailing list if
you are interested, see http://www.trading-shim.org/mailman/listinfo
for details.  The trading-shim is GPL, and available from:

    http://www.trading-shim.com/

________________________________________________________________________________

Now, back to R, and about tying R and the shim together.  I believe it's
well worth doing, but our resources, and in particular time, are very much
finite.

One possible design might involve yet another program, a downstream
script that drives both R and the shim.  After all, command interpreters are
designed to (surprise) accept commands, and both systems are command
interpreters.  Note that we haven't yet implemented this kind of R:shim
system.

The downstream script would poll on sockets for the two interpreters, e.g.,
with a bsd-style select.  It would send commands to the shim to subscribe
to or query for data, capture the resulting market data and history
information, store that data or otherwise provide it to the R process, and 
ask for analysis as needed.

I'm interested in what the R experts on this list think about the problem
of tying the two interpreters together.  The issue I see is that any
client program has to poll, handle async events, and in practice provide
a task loop.  I've implemented such logic in the shim itself, but I suspect
that it is often an unpleasant surprise to developers of downstream clients
that they must do the same.

For those who prefer to treat the shim process as subordinate to R, there
would an be upfront time cost while the shim read in its database and connected
to the IB tws, and then commands could be written to the shim, and results
read back.  The initial connection would necessarily be static state; due
to limitations of the IB tws, fast connect-disconnect cycles are a bad idea.

***NOTE:*** The code to understand the resulting output is nontrivial.  There
are on the order of 80 or so command, message, request, and comment events
that can be echoed to the downstream at the highest level of verbosity.  This
includes in particular a lot of tws api requests and message types, and it all
adds up.  The shim commands themselves are very simple, and basic clients can
just ignore most lines of message text, filtering by prefix.

By the way, for the simple case of history queries, and where latency and
reliability are not critical issues, synchrony is straight forward; start
the shim, write history queries to it, and read the results from the text
messages file.

One problem here is upstream history farm disconnects and resource limits;
it's important for downstream clients to watch for and understand error
messages, as well as to react to timeouts when the upstream chooses to
ignore the query because load limits were exceeded.

Thanks,

Bill Pippin