[Bioc-devel] BrowserViz and sub-protocols

Tim Triche, Jr. tim.triche at gmail.com
Tue Apr 7 03:10:41 CEST 2015 

Bingo

--t

> On Apr 6, 2015, at 5:32 PM, Michael Lawrence <lawrence.michael at gene.com> wrote:
> 
> Worth considering, but my understanding is that protocol buffers do not directly embed their type information. So while there is a schema corresponding to the buffer, the application needs to know which. Presumably one would need some sort of IDL on top of protobufs to handle that.
> 
> There's also this one: https://capnproto.org/.
> 
> It has a very rich schema/IDL language (including inheritance), and the RPC protocol has full support for lazy evaluation (pipeline promises), and supports incremental reads, with mmap, etc. Just seems a little out there. By the guy who designed protobufs v2.
> 
> 
>> On Mon, Apr 6, 2015 at 5:08 PM, Dan Tenenbaum <dtenenba at fredhutch.org> wrote:
>> 
>> 
>> ----- Original Message -----
>> > From: "Michael Lawrence" <lawrence.michael at gene.com>
>> > To: "Tim Triche, Jr." <tim.triche at gmail.com>
>> > Cc: "Michael Lawrence" <lawrence.michael at gene.com>, bioc-devel at r-project.org, "Paul Shannon" <pshannon at fredhutch.org>
>> > Sent: Monday, April 6, 2015 4:48:41 PM
>> > Subject: Re: [Bioc-devel] BrowserViz and sub-protocols
>> >
>> > On Mon, Apr 6, 2015 at 10:05 AM, Tim Triche, Jr.
>> > <tim.triche at gmail.com>
>> > wrote:
>> >
>> > > Kitware seems to have had good experiences with WAMP, which is a
>> > > good
>> > > sign, given the complexity of their typical visualization projects
>> > > (stemming from old HPC projects in fluid dynamics, astrophysics,
>> > > realtime
>> > > anatomical reconstruction, etc). Thus WAMP seems like a great way
>> > > to
>> > > leverage the experiences of an open source focused, medium sized
>> > > shop known
>> > > for doing highly technical work with some similar aims to Paul's
>> > > pkg.
>> > >
>> > > Avro's IDL becomes onerous for large projects rather fast.  The
>> > > lack of
>> > > inheritance can be excruciating when it is pushed beyond its
>> > > typical scope.
>> > > Best not to poke that sleeping dog ;-)
>> > >
>> > >
>> > Hmm, thanks for pointing that out. Is there no good solution for
>> > serializing objects, outside of simple lists and maps?
>> >
>> 
>> Protocol buffers?
>> 
>> https://github.com/dcodeIO/ProtoBuf.js/tree/master/examples/websocket
>> 
>> Dan
>> 
>> 
>> >
>> > > --t
>> > >
>> > > > On Apr 6, 2015, at 9:35 AM, Michael Lawrence
>> > > > <lawrence.michael at gene.com>
>> > > wrote:
>> > > >
>> > > > Sounds like a good plan. It does seem like there is a trend away
>> > > > from
>> > > type
>> > > > formalism in application development. Everything is essentially
>> > > > JSON.
>> > > Have
>> > > > you checked out WAMP? It seems very similar to your solution; it
>> > > > might
>> > > pay
>> > > > off to stick with standards, even if it takes more up-front
>> > > > investment.
>> > > > It's JSON-based, so it doesn't solve the high-level type problem.
>> > > > Very
>> > > few
>> > > > serialization formats do. Avro has a nice IDL, but that's out of
>> > > > the
>> > > hadoop
>> > > > world and would be foreign to the web. There is someting to be
>> > > > said for
>> > > > simplicity, but the incurred technical debt can become
>> > > > burdensome.
>> > > >
>> > > >
>> > > >> On Mon, Apr 6, 2015 at 8:39 AM, Paul Shannon
>> > > >> <pshannon at fredhutch.org>
>> > > wrote:
>> > > >>
>> > > >> Hi Michael,
>> > > >>
>> > > >> Thanks for the link to the subprotocol negotiaion discussion in
>> > > >> "High
>> > > >> Performance Browser Networking".  There's a number of things in
>> > > >> that
>> > > >> discussion of which I was not aware.  Very helpful.
>> > > >>
>> > > >> There may be real merit -- that is, a good trade-off between
>> > > >> some added
>> > > >> complexity and extra clarity -- to adding a mandatory
>> > > >> content-type
>> > > subfield
>> > > >> to the payload or to the message.  Or adding full sub-protocol
>> > > negotiation
>> > > >> as the reference you supplied discusses.
>> > > >>
>> > > >> For the first release of BrowserViz (and its demo subclass,
>> > > >> BrowserVizDemo, along with RCyjs) I would like to keep things as
>> > > >> simple
>> > > as
>> > > >> possible.  By which I mean:  provide (as we do now) the
>> > > >> mechanism by
>> > > which
>> > > >> content-type can be specified, without promulgating the policy
>> > > >> that
>> > > every
>> > > >> message must have that payload subfield.
>> > > >>
>> > > >> In practice I have found it quick and easy in both Javascript
>> > > >> and R to
>> > > >> examine the incoming 4-field message (cmd, status, callback,
>> > > >> payload):
>> > > >>
>> > > >> 1) Is the cmd field something I can respond to?  (Do I have a
>> > > >> handler
>> > > >> registered for that cmd?)
>> > > >> 2) Does the status indicate trouble? A request?
>> > > >> 3) If 1 & 2 suggest proceeding, then  names(msg$payload) can
>> > > >> help me
>> > > >> figure out if the incoming data is structured as I expect.
>> > > >>
>> > > >> One could reasonably object that this is all "programming by (ad
>> > > >> hoc)
>> > > >> agreement", rather than by transparent, self-describing, clearly
>> > > >> negotiatied data and exchange standards.
>> > > >>
>> > > >> In order to arrive at such standards, to figure out how formal
>> > > >> we should
>> > > >> go, let's experiment for a release cycle.  This could be
>> > > >> accomplished
>> > > >> without any changes to BrowserViz by using the status field
>> > > >> (status="summarizedExperimentRequest"), or by always using two
>> > > >> fields in
>> > > >> every message payload (payload$contentType, payload$content).
>> > > >>  If a few
>> > > >> people collaborated on (let's say) a SummarizedExperiment
>> > > >> browser viz
>> > > >> webapp, those people could agree on how to do this, try it out,
>> > > >> perhaps
>> > > >> discover a few sub-varieties of content-type are actually
>> > > >> needed.  Then,
>> > > >> when things have settled down, we could explore adding some
>> > > >> formality to
>> > > >> the process.
>> > > >>
>> > > >> How does that strike you?
>> > > >>
>> > > >> Thanks for the discussion!
>> > > >>
>> > > >> - Paul
>> > > >>
>> > > >>
>> > > >>
>> > > >> On Apr 3, 2015, at 3:11 PM, Michael Lawrence
>> > > >> <lawrence.michael at gene.com
>> > > >
>> > > >> wrote:
>> > > >>
>> > > >>> The high-level type issue is sort of discussed here:
>> > > >>>
>> > > >>
>> > > http://chimera.labs.oreilly.com/books/1230000000545/ch17.html#_subprotocol_negotiation
>> > > >>>
>> > > >>> What about extending your protocol so that the payload consists
>> > > >>> of two
>> > > >> fields:
>> > > >>> content-type and content, where the content-type adheres to the
>> > > >>> media
>> > > >> type specification? This is analogous to how every S4 object has
>> > > >> a class
>> > > >> attribute.
>> > > >>>
>> > > >>> Thanks for the discussion!
>> > > >>>
>> > > >>> On Fri, Apr 3, 2015 at 2:57 PM, Paul Shannon <
>> > > >> paul.thurmond.shannon at gmail.com> wrote:
>> > > >>> Hi Michael,
>> > > >>>
>> > > >>> Thanks for the clarification.   You make a good point about
>> > > >>> caching:
>> > > >> nobody wants to have to reinvent and re-engineer that!   If too
>> > > >> many
>> > > >> features like that become important, then the simplicity of
>> > > >> websockets
>> > > will
>> > > >> have an attendant cost.
>> > > >>>
>> > > >>> With regard to more formalism around the payload:  I can
>> > > >>> imagine that
>> > > >> there will be circumstances in which that is essential.  A
>> > > >> SummarizedExperiment browser would be very useful, and would
>> > > >> clearly
>> > > >> benefit from a standard payload structure.
>> > > >>>
>> > > >>> I figure, however, that that's above my pay-grade ;).   I leave
>> > > >>> that up
>> > > >> to people like you.  And I stand ready to add any features such
>> > > formalized
>> > > >> structures might require.  I'd like to think (I may be naive)
>> > > >> that the
>> > > >> architects of SummarizedExperiment and heavy users of it could
>> > > >> devise
>> > > and
>> > > >> negotiate some standard JSON representation of the class,
>> > > >> translators to
>> > > >> and from, which all could then be used by BrowserViz without
>> > > >> BrowserViz
>> > > >> needing to know it's there.
>> > > >>>
>> > > >>> The "binary JSON" format may be useful in some circumstances:
>> > > >> http://bjson.org
>> > > >>>
>> > > >>> - Paul
>> > > >>>
>> > > >>> On Apr 3, 2015, at 2:43 PM, Michael Lawrence <
>> > > lawrence.michael at gene.com>
>> > > >> wrote:
>> > > >>>
>> > > >>>>
>> > > >>>>
>> > > >>>> On Fri, Apr 3, 2015 at 2:00 PM, Paul Shannon <
>> > > >> paul.thurmond.shannon at gmail.com> wrote:
>> > > >>>> Hi Michael,
>> > > >>>>
>> > > >>>> Great to get your response, comments and questions.  Answers
>> > > >>>> attempted
>> > > >> below.
>> > > >>>>
>> > > >>>> I think our overriding difference lies in our contrasting
>> > > >>>> experience
>> > > of
>> > > >> complexity.  I have come to see websockets as minimal, simple,
>> > > >> fast and
>> > > >> flexible, whereas you see them quite differently.  I would like
>> > > >> to
>> > > >> understand your views on this; I could be overlooking somce
>> > > >> important
>> > > and
>> > > >> maybe costly complicating features, perhaps because of my
>> > > >> fondness for
>> > > >> other simplifying features.
>> > > >>>>
>> > > >>>>
>> > > >>>> I was just referring to potential complexity. If one attempted
>> > > >>>> to
>> > > >> reimplement the useful features of HTTP (like caching),
>> > > >> complexity
>> > > would be
>> > > >> introduced into application code, while you really want that
>> > > >> complexity
>> > > in
>> > > >> the protocol implementation. HTTP seems most appropriate for
>> > > >> when the
>> > > >> server is acting as the data model. Outside of that, I see the
>> > > >> benefits
>> > > of
>> > > >> web sockets.
>> > > >>>>
>> > > >>>> I think you missed my question about RPC. Also, any thoughts
>> > > >>>> as to
>> > > more
>> > > >> formalism around the payload? We obviously have complex data
>> > > >> structures,
>> > > >> and it would be nice to communicate semantics somehow to the web
>> > > browser.
>> > > >> For example, could there be some convention for representing a
>> > > >> SummarizedExperiment? Could a payload contain the equivalent of
>> > > >> a media
>> > > >> type that an R/JS library could understand to marshal objects?
>> > > >> Could
>> > > there
>> > > >> be some way to query for the types of payload a command
>> > > >> supports?
>> > > >>>>
>> > > >>>> I've seen stuff like WAMP, but they seem to lack the ability
>> > > >>>> to
>> > > declare
>> > > >> high-level types. Maybe that's just out of style?
>> > > >>>>
>> > > >>>>
>> > > >>>> More below...
>> > > >>>>
>> > > >>>> I look forward to hearing back from you.
>> > > >>>>
>> > > >>>> - Paul
>> > > >>>>
>> > > >>>> On Apr 3, 2015, at 1:45 PM, Michael Lawrence <
>> > > lawrence.michael at gene.com>
>> > > >> wrote:
>> > > >>>>
>> > > >>>>> Thanks to Val's excellent newsletter, I've had my first
>> > > >>>>> glance at
>> > > >>>>> BrowserViz. I'm glad to see something that is more flexible
>> > > >>>>> and
>> > > >> low-level
>> > > >>>>> than e.g. shiny.
>> > > >>>>>
>> > > >>>>> I'm curious about the motivation behind web sockets. I guess
>> > > >>>>> any
>> > > >>>>> application with an R-driven web UI actually has two UIs: the
>> > > >>>>> R
>> > > >> console and
>> > > >>>>> the browser. But what if the R session is headless, or if
>> > > >>>>> there is no
>> > > >> need
>> > > >>>>> for commands in R to affect the browser? Then the web socket
>> > > >>>>> layer
>> > > >> brings
>> > > >>>>> mostly unneeded complexity.
>> > > >>>>
>> > > >>>> I see websockets (like TCP sockets) as musch simpler than
>> > > >>>> HTTP.  No
>> > > >> headers,
>> > > >>>> no explicit server and explicit client (once the connection is
>> > > >>>> open).
>> > > >>>> Could you explain the complexity you see?
>> > > >>>>
>> > > >>>>> An interesting comparison to BrowserViz is not
>> > > >>>>> shiny but OpenCPU. It's purely HTTP-based and still manages
>> > > >>>>> to
>> > > maintain
>> > > >>>>> state (not sure how efficiently). I guess one advantage of
>> > > >>>>> web
>> > > sockets
>> > > >> is
>> > > >>>>> that one can program imperatively instead of declaratively on
>> > > >>>>> the
>> > > >> server,
>> > > >>>>> i.e., the server can send a command to show a popup in
>> > > >>>>> response to
>> > > some
>> > > >>>>> event, instead of returning a "declaration" that the popup
>> > > >>>>> should be
>> > > >> shown.
>> > > >>>>
>> > > >>>> Exactly!
>> > > >>>>
>> > > >>>>>
>> > > >>>>> So essentially web sockets are more natural for implementing
>> > > >> server-side
>> > > >>>>> controllers (think MVC), instead of just the data model, but
>> > > >>>>> man,
>> > > it's
>> > > >> a
>> > > >>>>> shame to lose the features of HTTP.
>> > > >>>>
>> > > >>>> I'll confess:  I set out to -shed- the features of HTTP.
>> > > >>>>  Isn't it a
>> > > >> protocol designed for serving up web pages on demand?  Not for
>> > > >> fast,
>> > > >> lighweight peer-to-peer communications?
>> > > >>>>
>> > > >>>>
>> > > >>>>>
>> > > >>>>> Ultimately, I think we want web apps that are easy to develop
>> > > >>>>> and
>> > > >> maintain,
>> > > >>>>> and run equally well from either a useR's session or a remote
>> > > >>>>> client
>> > > >>>>> communicating to a dedicated, headless server. Is the
>> > > >>>>> generality of
>> > > >>>>> websockets worth the complexity?
>> > > >>>>
>> > > >>>>>
>> > > >>>>> As an aside, it would seem relatively straight-forward to
>> > > >>>>> implement a
>> > > >>>>> simple bi-directional RPC mechanism between R and JS using
>> > > >>>>> the
>> > > standard
>> > > >>>>> protocol (i.e., hide details like the callback). Does that
>> > > >>>>> sound
>> > > >> reasonable?
>> > > >>>>>
>> > > >>>>> I was also a bit surprised about the need to copy/paste the
>> > > >>>>> JS
>> > > >> boilerplate.
>> > > >>>>> Certainly there must be javascript frameworks with a more
>> > > >>>>> elegant
>> > > >> solution
>> > > >>>>> to extensibility.
>> > > >>>>
>> > > >>>> You are correct on this, and I only need to correct the
>> > > >>>> vignette,
>> > > >> written many weeks ago.
>> > > >>>> I now provide "BrowserViz.jx", a simple Javascript module.  It
>> > > >>>> is used
>> > > >> by both
>> > > >>>> BrowserVizDemo and RCyjs.  Sorry to have sent out an obsolete
>> > > vignette.
>> > > >>>>
>> > > >>>>>
>> > > >>>>>     [[alternative HTML version deleted]]
>> > > >>>>>
>> > > >>>>> _______________________________________________
>> > > >>>>> Bioc-devel at r-project.org mailing list
>> > > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> > > >>>>
>> > > >>>>
>> > > >>>
>> > > >>>
>> > > >>
>> > > >>
>> > > >
>> > > >    [[alternative HTML version deleted]]
>> > > >
>> > > > _______________________________________________
>> > > > Bioc-devel at r-project.org mailing list
>> > > > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> > >
>> >
>> >       [[alternative HTML version deleted]]
>> >
>> > _______________________________________________
>> > Bioc-devel at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
>> >
> 

	[[alternative HTML version deleted]]

More information about the Bioc-devel mailing list