[Bioc-devel] BrowserViz and sub-protocols
Tim Triche, Jr.
tim.triche at gmail.com
Tue Apr 7 03:10:31 CEST 2015
Protobuf begat Avro (more or less)
https://avro.apache.org/docs/current/#compare
--t
> On Apr 6, 2015, at 5:08 PM, Dan Tenenbaum <dtenenba at fredhutch.org> wrote:
>
>
>
> ----- Original Message -----
>> From: "Michael Lawrence" <lawrence.michael at gene.com>
>> To: "Tim Triche, Jr." <tim.triche at gmail.com>
>> Cc: "Michael Lawrence" <lawrence.michael at gene.com>, bioc-devel at r-project.org, "Paul Shannon" <pshannon at fredhutch.org>
>> Sent: Monday, April 6, 2015 4:48:41 PM
>> Subject: Re: [Bioc-devel] BrowserViz and sub-protocols
>>
>> On Mon, Apr 6, 2015 at 10:05 AM, Tim Triche, Jr.
>> <tim.triche at gmail.com>
>> wrote:
>>
>>> Kitware seems to have had good experiences with WAMP, which is a
>>> good
>>> sign, given the complexity of their typical visualization projects
>>> (stemming from old HPC projects in fluid dynamics, astrophysics,
>>> realtime
>>> anatomical reconstruction, etc). Thus WAMP seems like a great way
>>> to
>>> leverage the experiences of an open source focused, medium sized
>>> shop known
>>> for doing highly technical work with some similar aims to Paul's
>>> pkg.
>>>
>>> Avro's IDL becomes onerous for large projects rather fast. The
>>> lack of
>>> inheritance can be excruciating when it is pushed beyond its
>>> typical scope.
>>> Best not to poke that sleeping dog ;-)
>> Hmm, thanks for pointing that out. Is there no good solution for
>> serializing objects, outside of simple lists and maps?
>
> Protocol buffers?
>
> https://github.com/dcodeIO/ProtoBuf.js/tree/master/examples/websocket
>
> Dan
>
>
>>
>>> --t
>>>
>>>> On Apr 6, 2015, at 9:35 AM, Michael Lawrence
>>>> <lawrence.michael at gene.com>
>>> wrote:
>>>>
>>>> Sounds like a good plan. It does seem like there is a trend away
>>>> from
>>> type
>>>> formalism in application development. Everything is essentially
>>>> JSON.
>>> Have
>>>> you checked out WAMP? It seems very similar to your solution; it
>>>> might
>>> pay
>>>> off to stick with standards, even if it takes more up-front
>>>> investment.
>>>> It's JSON-based, so it doesn't solve the high-level type problem.
>>>> Very
>>> few
>>>> serialization formats do. Avro has a nice IDL, but that's out of
>>>> the
>>> hadoop
>>>> world and would be foreign to the web. There is someting to be
>>>> said for
>>>> simplicity, but the incurred technical debt can become
>>>> burdensome.
>>>>
>>>>
>>>>> On Mon, Apr 6, 2015 at 8:39 AM, Paul Shannon
>>>>> <pshannon at fredhutch.org>
>>> wrote:
>>>>>
>>>>> Hi Michael,
>>>>>
>>>>> Thanks for the link to the subprotocol negotiaion discussion in
>>>>> "High
>>>>> Performance Browser Networking". There's a number of things in
>>>>> that
>>>>> discussion of which I was not aware. Very helpful.
>>>>>
>>>>> There may be real merit -- that is, a good trade-off between
>>>>> some added
>>>>> complexity and extra clarity -- to adding a mandatory
>>>>> content-type
>>> subfield
>>>>> to the payload or to the message. Or adding full sub-protocol
>>> negotiation
>>>>> as the reference you supplied discusses.
>>>>>
>>>>> For the first release of BrowserViz (and its demo subclass,
>>>>> BrowserVizDemo, along with RCyjs) I would like to keep things as
>>>>> simple
>>> as
>>>>> possible. By which I mean: provide (as we do now) the
>>>>> mechanism by
>>> which
>>>>> content-type can be specified, without promulgating the policy
>>>>> that
>>> every
>>>>> message must have that payload subfield.
>>>>>
>>>>> In practice I have found it quick and easy in both Javascript
>>>>> and R to
>>>>> examine the incoming 4-field message (cmd, status, callback,
>>>>> payload):
>>>>>
>>>>> 1) Is the cmd field something I can respond to? (Do I have a
>>>>> handler
>>>>> registered for that cmd?)
>>>>> 2) Does the status indicate trouble? A request?
>>>>> 3) If 1 & 2 suggest proceeding, then names(msg$payload) can
>>>>> help me
>>>>> figure out if the incoming data is structured as I expect.
>>>>>
>>>>> One could reasonably object that this is all "programming by (ad
>>>>> hoc)
>>>>> agreement", rather than by transparent, self-describing, clearly
>>>>> negotiatied data and exchange standards.
>>>>>
>>>>> In order to arrive at such standards, to figure out how formal
>>>>> we should
>>>>> go, let's experiment for a release cycle. This could be
>>>>> accomplished
>>>>> without any changes to BrowserViz by using the status field
>>>>> (status="summarizedExperimentRequest"), or by always using two
>>>>> fields in
>>>>> every message payload (payload$contentType, payload$content).
>>>>> If a few
>>>>> people collaborated on (let's say) a SummarizedExperiment
>>>>> browser viz
>>>>> webapp, those people could agree on how to do this, try it out,
>>>>> perhaps
>>>>> discover a few sub-varieties of content-type are actually
>>>>> needed. Then,
>>>>> when things have settled down, we could explore adding some
>>>>> formality to
>>>>> the process.
>>>>>
>>>>> How does that strike you?
>>>>>
>>>>> Thanks for the discussion!
>>>>>
>>>>> - Paul
>>>>>
>>>>>
>>>>>
>>>>> On Apr 3, 2015, at 3:11 PM, Michael Lawrence
>>>>> <lawrence.michael at gene.com
>>>>
>>>>> wrote:
>>>>>
>>>>>> The high-level type issue is sort of discussed here:
>>> http://chimera.labs.oreilly.com/books/1230000000545/ch17.html#_subprotocol_negotiation
>>>>>>
>>>>>> What about extending your protocol so that the payload consists
>>>>>> of two
>>>>> fields:
>>>>>> content-type and content, where the content-type adheres to the
>>>>>> media
>>>>> type specification? This is analogous to how every S4 object has
>>>>> a class
>>>>> attribute.
>>>>>>
>>>>>> Thanks for the discussion!
>>>>>>
>>>>>>> On Fri, Apr 3, 2015 at 2:57 PM, Paul Shannon <
>>>>>> paul.thurmond.shannon at gmail.com> wrote:
>>>>>> Hi Michael,
>>>>>>
>>>>>> Thanks for the clarification. You make a good point about
>>>>>> caching:
>>>>> nobody wants to have to reinvent and re-engineer that! If too
>>>>> many
>>>>> features like that become important, then the simplicity of
>>>>> websockets
>>> will
>>>>> have an attendant cost.
>>>>>>
>>>>>> With regard to more formalism around the payload: I can
>>>>>> imagine that
>>>>> there will be circumstances in which that is essential. A
>>>>> SummarizedExperiment browser would be very useful, and would
>>>>> clearly
>>>>> benefit from a standard payload structure.
>>>>>>
>>>>>> I figure, however, that that's above my pay-grade ;). I leave
>>>>>> that up
>>>>> to people like you. And I stand ready to add any features such
>>> formalized
>>>>> structures might require. I'd like to think (I may be naive)
>>>>> that the
>>>>> architects of SummarizedExperiment and heavy users of it could
>>>>> devise
>>> and
>>>>> negotiate some standard JSON representation of the class,
>>>>> translators to
>>>>> and from, which all could then be used by BrowserViz without
>>>>> BrowserViz
>>>>> needing to know it's there.
>>>>>>
>>>>>> The "binary JSON" format may be useful in some circumstances:
>>>>> http://bjson.org
>>>>>>
>>>>>> - Paul
>>>>>>
>>>>>> On Apr 3, 2015, at 2:43 PM, Michael Lawrence <
>>> lawrence.michael at gene.com>
>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Apr 3, 2015 at 2:00 PM, Paul Shannon <
>>>>> paul.thurmond.shannon at gmail.com> wrote:
>>>>>>> Hi Michael,
>>>>>>>
>>>>>>> Great to get your response, comments and questions. Answers
>>>>>>> attempted
>>>>> below.
>>>>>>>
>>>>>>> I think our overriding difference lies in our contrasting
>>>>>>> experience
>>> of
>>>>> complexity. I have come to see websockets as minimal, simple,
>>>>> fast and
>>>>> flexible, whereas you see them quite differently. I would like
>>>>> to
>>>>> understand your views on this; I could be overlooking somce
>>>>> important
>>> and
>>>>> maybe costly complicating features, perhaps because of my
>>>>> fondness for
>>>>> other simplifying features.
>>>>>>>
>>>>>>>
>>>>>>> I was just referring to potential complexity. If one attempted
>>>>>>> to
>>>>> reimplement the useful features of HTTP (like caching),
>>>>> complexity
>>> would be
>>>>> introduced into application code, while you really want that
>>>>> complexity
>>> in
>>>>> the protocol implementation. HTTP seems most appropriate for
>>>>> when the
>>>>> server is acting as the data model. Outside of that, I see the
>>>>> benefits
>>> of
>>>>> web sockets.
>>>>>>>
>>>>>>> I think you missed my question about RPC. Also, any thoughts
>>>>>>> as to
>>> more
>>>>> formalism around the payload? We obviously have complex data
>>>>> structures,
>>>>> and it would be nice to communicate semantics somehow to the web
>>> browser.
>>>>> For example, could there be some convention for representing a
>>>>> SummarizedExperiment? Could a payload contain the equivalent of
>>>>> a media
>>>>> type that an R/JS library could understand to marshal objects?
>>>>> Could
>>> there
>>>>> be some way to query for the types of payload a command
>>>>> supports?
>>>>>>>
>>>>>>> I've seen stuff like WAMP, but they seem to lack the ability
>>>>>>> to
>>> declare
>>>>> high-level types. Maybe that's just out of style?
>>>>>>>
>>>>>>>
>>>>>>> More below...
>>>>>>>
>>>>>>> I look forward to hearing back from you.
>>>>>>>
>>>>>>> - Paul
>>>>>>>
>>>>>>> On Apr 3, 2015, at 1:45 PM, Michael Lawrence <
>>> lawrence.michael at gene.com>
>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks to Val's excellent newsletter, I've had my first
>>>>>>>> glance at
>>>>>>>> BrowserViz. I'm glad to see something that is more flexible
>>>>>>>> and
>>>>> low-level
>>>>>>>> than e.g. shiny.
>>>>>>>>
>>>>>>>> I'm curious about the motivation behind web sockets. I guess
>>>>>>>> any
>>>>>>>> application with an R-driven web UI actually has two UIs: the
>>>>>>>> R
>>>>> console and
>>>>>>>> the browser. But what if the R session is headless, or if
>>>>>>>> there is no
>>>>> need
>>>>>>>> for commands in R to affect the browser? Then the web socket
>>>>>>>> layer
>>>>> brings
>>>>>>>> mostly unneeded complexity.
>>>>>>>
>>>>>>> I see websockets (like TCP sockets) as musch simpler than
>>>>>>> HTTP. No
>>>>> headers,
>>>>>>> no explicit server and explicit client (once the connection is
>>>>>>> open).
>>>>>>> Could you explain the complexity you see?
>>>>>>>
>>>>>>>> An interesting comparison to BrowserViz is not
>>>>>>>> shiny but OpenCPU. It's purely HTTP-based and still manages
>>>>>>>> to
>>> maintain
>>>>>>>> state (not sure how efficiently). I guess one advantage of
>>>>>>>> web
>>> sockets
>>>>> is
>>>>>>>> that one can program imperatively instead of declaratively on
>>>>>>>> the
>>>>> server,
>>>>>>>> i.e., the server can send a command to show a popup in
>>>>>>>> response to
>>> some
>>>>>>>> event, instead of returning a "declaration" that the popup
>>>>>>>> should be
>>>>> shown.
>>>>>>>
>>>>>>> Exactly!
>>>>>>>
>>>>>>>>
>>>>>>>> So essentially web sockets are more natural for implementing
>>>>> server-side
>>>>>>>> controllers (think MVC), instead of just the data model, but
>>>>>>>> man,
>>> it's
>>>>> a
>>>>>>>> shame to lose the features of HTTP.
>>>>>>>
>>>>>>> I'll confess: I set out to -shed- the features of HTTP.
>>>>>>> Isn't it a
>>>>> protocol designed for serving up web pages on demand? Not for
>>>>> fast,
>>>>> lighweight peer-to-peer communications?
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Ultimately, I think we want web apps that are easy to develop
>>>>>>>> and
>>>>> maintain,
>>>>>>>> and run equally well from either a useR's session or a remote
>>>>>>>> client
>>>>>>>> communicating to a dedicated, headless server. Is the
>>>>>>>> generality of
>>>>>>>> websockets worth the complexity?
>>>>>>>
>>>>>>>>
>>>>>>>> As an aside, it would seem relatively straight-forward to
>>>>>>>> implement a
>>>>>>>> simple bi-directional RPC mechanism between R and JS using
>>>>>>>> the
>>> standard
>>>>>>>> protocol (i.e., hide details like the callback). Does that
>>>>>>>> sound
>>>>> reasonable?
>>>>>>>>
>>>>>>>> I was also a bit surprised about the need to copy/paste the
>>>>>>>> JS
>>>>> boilerplate.
>>>>>>>> Certainly there must be javascript frameworks with a more
>>>>>>>> elegant
>>>>> solution
>>>>>>>> to extensibility.
>>>>>>>
>>>>>>> You are correct on this, and I only need to correct the
>>>>>>> vignette,
>>>>> written many weeks ago.
>>>>>>> I now provide "BrowserViz.jx", a simple Javascript module. It
>>>>>>> is used
>>>>> by both
>>>>>>> BrowserVizDemo and RCyjs. Sorry to have sent out an obsolete
>>> vignette.
>>>>>>>
>>>>>>>>
>>>>>>>> [[alternative HTML version deleted]]
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Bioc-devel at r-project.org mailing list
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>>>
>>>> [[alternative HTML version deleted]]
>>>>
>>>> _______________________________________________
>>>> Bioc-devel at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>> [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> Bioc-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
More information about the Bioc-devel
mailing list