[Bioc-devel] BrowserViz and sub-protocols

[Tim Triche, Jr.]

Kitware seems to have had good experiences with WAMP, which is a good sign, given the complexity of their typical visualization projects (stemming from old HPC projects in fluid dynamics, astrophysics, realtime anatomical reconstruction, etc). Thus WAMP seems like a great way to leverage the experiences of an open source focused, medium sized shop known for doing highly technical work with some similar aims to Paul's pkg. 

Avro's IDL becomes onerous for large projects rather fast.  The lack of inheritance can be excruciating when it is pushed beyond its typical scope. Best not to poke that sleeping dog ;-)

--t

> On Apr 6, 2015, at 9:35 AM, Michael Lawrence <lawrence.michael at gene.com> wrote:
> 
> Sounds like a good plan. It does seem like there is a trend away from type
> formalism in application development. Everything is essentially JSON. Have
> you checked out WAMP? It seems very similar to your solution; it might pay
> off to stick with standards, even if it takes more up-front investment.
> It's JSON-based, so it doesn't solve the high-level type problem. Very few
> serialization formats do. Avro has a nice IDL, but that's out of the hadoop
> world and would be foreign to the web. There is someting to be said for
> simplicity, but the incurred technical debt can become burdensome.
> 
> 
>> On Mon, Apr 6, 2015 at 8:39 AM, Paul Shannon <pshannon at fredhutch.org> wrote:
>> 
>> Hi Michael,
>> 
>> Thanks for the link to the subprotocol negotiaion discussion in "High
>> Performance Browser Networking".  There's a number of things in that
>> discussion of which I was not aware.  Very helpful.
>> 
>> There may be real merit -- that is, a good trade-off between some added
>> complexity and extra clarity -- to adding a mandatory content-type subfield
>> to the payload or to the message.  Or adding full sub-protocol negotiation
>> as the reference you supplied discusses.
>> 
>> For the first release of BrowserViz (and its demo subclass,
>> BrowserVizDemo, along with RCyjs) I would like to keep things as simple as
>> possible.  By which I mean:  provide (as we do now) the mechanism by which
>> content-type can be specified, without promulgating the policy that every
>> message must have that payload subfield.
>> 
>> In practice I have found it quick and easy in both Javascript and R to
>> examine the incoming 4-field message (cmd, status, callback, payload):
>> 
>> 1) Is the cmd field something I can respond to?  (Do I have a handler
>> registered for that cmd?)
>> 2) Does the status indicate trouble? A request?
>> 3) If 1 & 2 suggest proceeding, then  names(msg$payload) can help me
>> figure out if the incoming data is structured as I expect.
>> 
>> One could reasonably object that this is all "programming by (ad hoc)
>> agreement", rather than by transparent, self-describing, clearly
>> negotiatied data and exchange standards.
>> 
>> In order to arrive at such standards, to figure out how formal we should
>> go, let's experiment for a release cycle.  This could be accomplished
>> without any changes to BrowserViz by using the status field
>> (status="summarizedExperimentRequest"), or by always using two fields in
>> every message payload (payload$contentType, payload$content).  If a few
>> people collaborated on (let's say) a SummarizedExperiment browser viz
>> webapp, those people could agree on how to do this, try it out, perhaps
>> discover a few sub-varieties of content-type are actually needed.  Then,
>> when things have settled down, we could explore adding some formality to
>> the process.
>> 
>> How does that strike you?
>> 
>> Thanks for the discussion!
>> 
>> - Paul
>> 
>> 
>> 
>> On Apr 3, 2015, at 3:11 PM, Michael Lawrence <lawrence.michael at gene.com>
>> wrote:
>> 
>>> The high-level type issue is sort of discussed here:
>>> 
>> http://chimera.labs.oreilly.com/books/1230000000545/ch17.html#_subprotocol_negotiation
>>> 
>>> What about extending your protocol so that the payload consists of two
>> fields:
>>> content-type and content, where the content-type adheres to the media
>> type specification? This is analogous to how every S4 object has a class
>> attribute.
>>> 
>>> Thanks for the discussion!
>>> 
>>> On Fri, Apr 3, 2015 at 2:57 PM, Paul Shannon <
>> paul.thurmond.shannon at gmail.com> wrote:
>>> Hi Michael,
>>> 
>>> Thanks for the clarification.   You make a good point about caching:
>> nobody wants to have to reinvent and re-engineer that!   If too many
>> features like that become important, then the simplicity of websockets will
>> have an attendant cost.
>>> 
>>> With regard to more formalism around the payload:  I can imagine that
>> there will be circumstances in which that is essential.  A
>> SummarizedExperiment browser would be very useful, and would clearly
>> benefit from a standard payload structure.
>>> 
>>> I figure, however, that that's above my pay-grade ;).   I leave that up
>> to people like you.  And I stand ready to add any features such formalized
>> structures might require.  I'd like to think (I may be naive) that the
>> architects of SummarizedExperiment and heavy users of it could devise and
>> negotiate some standard JSON representation of the class, translators to
>> and from, which all could then be used by BrowserViz without BrowserViz
>> needing to know it's there.
>>> 
>>> The "binary JSON" format may be useful in some circumstances:
>> http://bjson.org
>>> 
>>> - Paul
>>> 
>>> On Apr 3, 2015, at 2:43 PM, Michael Lawrence <lawrence.michael at gene.com>
>> wrote:
>>> 
>>>> 
>>>> 
>>>> On Fri, Apr 3, 2015 at 2:00 PM, Paul Shannon <
>> paul.thurmond.shannon at gmail.com> wrote:
>>>> Hi Michael,
>>>> 
>>>> Great to get your response, comments and questions.  Answers attempted
>> below.
>>>> 
>>>> I think our overriding difference lies in our contrasting experience of
>> complexity.  I have come to see websockets as minimal, simple, fast and
>> flexible, whereas you see them quite differently.  I would like to
>> understand your views on this; I could be overlooking somce important and
>> maybe costly complicating features, perhaps because of my fondness for
>> other simplifying features.
>>>> 
>>>> 
>>>> I was just referring to potential complexity. If one attempted to
>> reimplement the useful features of HTTP (like caching), complexity would be
>> introduced into application code, while you really want that complexity in
>> the protocol implementation. HTTP seems most appropriate for when the
>> server is acting as the data model. Outside of that, I see the benefits of
>> web sockets.
>>>> 
>>>> I think you missed my question about RPC. Also, any thoughts as to more
>> formalism around the payload? We obviously have complex data structures,
>> and it would be nice to communicate semantics somehow to the web browser.
>> For example, could there be some convention for representing a
>> SummarizedExperiment? Could a payload contain the equivalent of a media
>> type that an R/JS library could understand to marshal objects? Could there
>> be some way to query for the types of payload a command supports?
>>>> 
>>>> I've seen stuff like WAMP, but they seem to lack the ability to declare
>> high-level types. Maybe that's just out of style?
>>>> 
>>>> 
>>>> More below...
>>>> 
>>>> I look forward to hearing back from you.
>>>> 
>>>> - Paul
>>>> 
>>>> On Apr 3, 2015, at 1:45 PM, Michael Lawrence <lawrence.michael at gene.com>
>> wrote:
>>>> 
>>>>> Thanks to Val's excellent newsletter, I've had my first glance at
>>>>> BrowserViz. I'm glad to see something that is more flexible and
>> low-level
>>>>> than e.g. shiny.
>>>>> 
>>>>> I'm curious about the motivation behind web sockets. I guess any
>>>>> application with an R-driven web UI actually has two UIs: the R
>> console and
>>>>> the browser. But what if the R session is headless, or if there is no
>> need
>>>>> for commands in R to affect the browser? Then the web socket layer
>> brings
>>>>> mostly unneeded complexity.
>>>> 
>>>> I see websockets (like TCP sockets) as musch simpler than HTTP.  No
>> headers,
>>>> no explicit server and explicit client (once the connection is open).
>>>> Could you explain the complexity you see?
>>>> 
>>>>> An interesting comparison to BrowserViz is not
>>>>> shiny but OpenCPU. It's purely HTTP-based and still manages to maintain
>>>>> state (not sure how efficiently). I guess one advantage of web sockets
>> is
>>>>> that one can program imperatively instead of declaratively on the
>> server,
>>>>> i.e., the server can send a command to show a popup in response to some
>>>>> event, instead of returning a "declaration" that the popup should be
>> shown.
>>>> 
>>>> Exactly!
>>>> 
>>>>> 
>>>>> So essentially web sockets are more natural for implementing
>> server-side
>>>>> controllers (think MVC), instead of just the data model, but man, it's
>> a
>>>>> shame to lose the features of HTTP.
>>>> 
>>>> I'll confess:  I set out to -shed- the features of HTTP.  Isn't it a
>> protocol designed for serving up web pages on demand?  Not for fast,
>> lighweight peer-to-peer communications?
>>>> 
>>>> 
>>>>> 
>>>>> Ultimately, I think we want web apps that are easy to develop and
>> maintain,
>>>>> and run equally well from either a useR's session or a remote client
>>>>> communicating to a dedicated, headless server. Is the generality of
>>>>> websockets worth the complexity?
>>>> 
>>>>> 
>>>>> As an aside, it would seem relatively straight-forward to implement a
>>>>> simple bi-directional RPC mechanism between R and JS using the standard
>>>>> protocol (i.e., hide details like the callback). Does that sound
>> reasonable?
>>>>> 
>>>>> I was also a bit surprised about the need to copy/paste the JS
>> boilerplate.
>>>>> Certainly there must be javascript frameworks with a more elegant
>> solution
>>>>> to extensibility.
>>>> 
>>>> You are correct on this, and I only need to correct the vignette,
>> written many weeks ago.
>>>> I now provide "BrowserViz.jx", a simple Javascript module.  It is used
>> by both
>>>> BrowserVizDemo and RCyjs.  Sorry to have sent out an obsolete vignette.
>>>> 
>>>>> 
>>>>>     [[alternative HTML version deleted]]
>>>>> 
>>>>> _______________________________________________
>>>>> Bioc-devel at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
>    [[alternative HTML version deleted]]
> 
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel