[Bioc-devel] BrowserViz and sub-protocols

Michael Lawrence lawrence.michael at gene.com
Tue Apr 7 02:32:38 CEST 2015 

Worth considering, but my understanding is that protocol buffers do not
directly embed their type information. So while there is a schema
corresponding to the buffer, the application needs to know which.
Presumably one would need some sort of IDL on top of protobufs to handle
that.

There's also this one: https://capnproto.org/.

It has a very rich schema/IDL language (including inheritance), and the RPC
protocol has full support for lazy evaluation (pipeline promises), and
supports incremental reads, with mmap, etc. Just seems a little out there.
By the guy who designed protobufs v2.


On Mon, Apr 6, 2015 at 5:08 PM, Dan Tenenbaum <dtenenba at fredhutch.org>
wrote:

>
>
> ----- Original Message -----
> > From: "Michael Lawrence" <lawrence.michael at gene.com>
> > To: "Tim Triche, Jr." <tim.triche at gmail.com>
> > Cc: "Michael Lawrence" <lawrence.michael at gene.com>,
> bioc-devel at r-project.org, "Paul Shannon" <pshannon at fredhutch.org>
> > Sent: Monday, April 6, 2015 4:48:41 PM
> > Subject: Re: [Bioc-devel] BrowserViz and sub-protocols
> >
> > On Mon, Apr 6, 2015 at 10:05 AM, Tim Triche, Jr.
> > <tim.triche at gmail.com>
> > wrote:
> >
> > > Kitware seems to have had good experiences with WAMP, which is a
> > > good
> > > sign, given the complexity of their typical visualization projects
> > > (stemming from old HPC projects in fluid dynamics, astrophysics,
> > > realtime
> > > anatomical reconstruction, etc). Thus WAMP seems like a great way
> > > to
> > > leverage the experiences of an open source focused, medium sized
> > > shop known
> > > for doing highly technical work with some similar aims to Paul's
> > > pkg.
> > >
> > > Avro's IDL becomes onerous for large projects rather fast.  The
> > > lack of
> > > inheritance can be excruciating when it is pushed beyond its
> > > typical scope.
> > > Best not to poke that sleeping dog ;-)
> > >
> > >
> > Hmm, thanks for pointing that out. Is there no good solution for
> > serializing objects, outside of simple lists and maps?
> >
>
> Protocol buffers?
>
> https://github.com/dcodeIO/ProtoBuf.js/tree/master/examples/websocket
>
> Dan
>
>
> >
> > > --t
> > >
> > > > On Apr 6, 2015, at 9:35 AM, Michael Lawrence
> > > > <lawrence.michael at gene.com>
> > > wrote:
> > > >
> > > > Sounds like a good plan. It does seem like there is a trend away
> > > > from
> > > type
> > > > formalism in application development. Everything is essentially
> > > > JSON.
> > > Have
> > > > you checked out WAMP? It seems very similar to your solution; it
> > > > might
> > > pay
> > > > off to stick with standards, even if it takes more up-front
> > > > investment.
> > > > It's JSON-based, so it doesn't solve the high-level type problem.
> > > > Very
> > > few
> > > > serialization formats do. Avro has a nice IDL, but that's out of
> > > > the
> > > hadoop
> > > > world and would be foreign to the web. There is someting to be
> > > > said for
> > > > simplicity, but the incurred technical debt can become
> > > > burdensome.
> > > >
> > > >
> > > >> On Mon, Apr 6, 2015 at 8:39 AM, Paul Shannon
> > > >> <pshannon at fredhutch.org>
> > > wrote:
> > > >>
> > > >> Hi Michael,
> > > >>
> > > >> Thanks for the link to the subprotocol negotiaion discussion in
> > > >> "High
> > > >> Performance Browser Networking".  There's a number of things in
> > > >> that
> > > >> discussion of which I was not aware.  Very helpful.
> > > >>
> > > >> There may be real merit -- that is, a good trade-off between
> > > >> some added
> > > >> complexity and extra clarity -- to adding a mandatory
> > > >> content-type
> > > subfield
> > > >> to the payload or to the message.  Or adding full sub-protocol
> > > negotiation
> > > >> as the reference you supplied discusses.
> > > >>
> > > >> For the first release of BrowserViz (and its demo subclass,
> > > >> BrowserVizDemo, along with RCyjs) I would like to keep things as
> > > >> simple
> > > as
> > > >> possible.  By which I mean:  provide (as we do now) the
> > > >> mechanism by
> > > which
> > > >> content-type can be specified, without promulgating the policy
> > > >> that
> > > every
> > > >> message must have that payload subfield.
> > > >>
> > > >> In practice I have found it quick and easy in both Javascript
> > > >> and R to
> > > >> examine the incoming 4-field message (cmd, status, callback,
> > > >> payload):
> > > >>
> > > >> 1) Is the cmd field something I can respond to?  (Do I have a
> > > >> handler
> > > >> registered for that cmd?)
> > > >> 2) Does the status indicate trouble? A request?
> > > >> 3) If 1 & 2 suggest proceeding, then  names(msg$payload) can
> > > >> help me
> > > >> figure out if the incoming data is structured as I expect.
> > > >>
> > > >> One could reasonably object that this is all "programming by (ad
> > > >> hoc)
> > > >> agreement", rather than by transparent, self-describing, clearly
> > > >> negotiatied data and exchange standards.
> > > >>
> > > >> In order to arrive at such standards, to figure out how formal
> > > >> we should
> > > >> go, let's experiment for a release cycle.  This could be
> > > >> accomplished
> > > >> without any changes to BrowserViz by using the status field
> > > >> (status="summarizedExperimentRequest"), or by always using two
> > > >> fields in
> > > >> every message payload (payload$contentType, payload$content).
> > > >>  If a few
> > > >> people collaborated on (let's say) a SummarizedExperiment
> > > >> browser viz
> > > >> webapp, those people could agree on how to do this, try it out,
> > > >> perhaps
> > > >> discover a few sub-varieties of content-type are actually
> > > >> needed.  Then,
> > > >> when things have settled down, we could explore adding some
> > > >> formality to
> > > >> the process.
> > > >>
> > > >> How does that strike you?
> > > >>
> > > >> Thanks for the discussion!
> > > >>
> > > >> - Paul
> > > >>
> > > >>
> > > >>
> > > >> On Apr 3, 2015, at 3:11 PM, Michael Lawrence
> > > >> <lawrence.michael at gene.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> The high-level type issue is sort of discussed here:
> > > >>>
> > > >>
> > >
> http://chimera.labs.oreilly.com/books/1230000000545/ch17.html#_subprotocol_negotiation
> > > >>>
> > > >>> What about extending your protocol so that the payload consists
> > > >>> of two
> > > >> fields:
> > > >>> content-type and content, where the content-type adheres to the
> > > >>> media
> > > >> type specification? This is analogous to how every S4 object has
> > > >> a class
> > > >> attribute.
> > > >>>
> > > >>> Thanks for the discussion!
> > > >>>
> > > >>> On Fri, Apr 3, 2015 at 2:57 PM, Paul Shannon <
> > > >> paul.thurmond.shannon at gmail.com> wrote:
> > > >>> Hi Michael,
> > > >>>
> > > >>> Thanks for the clarification.   You make a good point about
> > > >>> caching:
> > > >> nobody wants to have to reinvent and re-engineer that!   If too
> > > >> many
> > > >> features like that become important, then the simplicity of
> > > >> websockets
> > > will
> > > >> have an attendant cost.
> > > >>>
> > > >>> With regard to more formalism around the payload:  I can
> > > >>> imagine that
> > > >> there will be circumstances in which that is essential.  A
> > > >> SummarizedExperiment browser would be very useful, and would
> > > >> clearly
> > > >> benefit from a standard payload structure.
> > > >>>
> > > >>> I figure, however, that that's above my pay-grade ;).   I leave
> > > >>> that up
> > > >> to people like you.  And I stand ready to add any features such
> > > formalized
> > > >> structures might require.  I'd like to think (I may be naive)
> > > >> that the
> > > >> architects of SummarizedExperiment and heavy users of it could
> > > >> devise
> > > and
> > > >> negotiate some standard JSON representation of the class,
> > > >> translators to
> > > >> and from, which all could then be used by BrowserViz without
> > > >> BrowserViz
> > > >> needing to know it's there.
> > > >>>
> > > >>> The "binary JSON" format may be useful in some circumstances:
> > > >> http://bjson.org
> > > >>>
> > > >>> - Paul
> > > >>>
> > > >>> On Apr 3, 2015, at 2:43 PM, Michael Lawrence <
> > > lawrence.michael at gene.com>
> > > >> wrote:
> > > >>>
> > > >>>>
> > > >>>>
> > > >>>> On Fri, Apr 3, 2015 at 2:00 PM, Paul Shannon <
> > > >> paul.thurmond.shannon at gmail.com> wrote:
> > > >>>> Hi Michael,
> > > >>>>
> > > >>>> Great to get your response, comments and questions.  Answers
> > > >>>> attempted
> > > >> below.
> > > >>>>
> > > >>>> I think our overriding difference lies in our contrasting
> > > >>>> experience
> > > of
> > > >> complexity.  I have come to see websockets as minimal, simple,
> > > >> fast and
> > > >> flexible, whereas you see them quite differently.  I would like
> > > >> to
> > > >> understand your views on this; I could be overlooking somce
> > > >> important
> > > and
> > > >> maybe costly complicating features, perhaps because of my
> > > >> fondness for
> > > >> other simplifying features.
> > > >>>>
> > > >>>>
> > > >>>> I was just referring to potential complexity. If one attempted
> > > >>>> to
> > > >> reimplement the useful features of HTTP (like caching),
> > > >> complexity
> > > would be
> > > >> introduced into application code, while you really want that
> > > >> complexity
> > > in
> > > >> the protocol implementation. HTTP seems most appropriate for
> > > >> when the
> > > >> server is acting as the data model. Outside of that, I see the
> > > >> benefits
> > > of
> > > >> web sockets.
> > > >>>>
> > > >>>> I think you missed my question about RPC. Also, any thoughts
> > > >>>> as to
> > > more
> > > >> formalism around the payload? We obviously have complex data
> > > >> structures,
> > > >> and it would be nice to communicate semantics somehow to the web
> > > browser.
> > > >> For example, could there be some convention for representing a
> > > >> SummarizedExperiment? Could a payload contain the equivalent of
> > > >> a media
> > > >> type that an R/JS library could understand to marshal objects?
> > > >> Could
> > > there
> > > >> be some way to query for the types of payload a command
> > > >> supports?
> > > >>>>
> > > >>>> I've seen stuff like WAMP, but they seem to lack the ability
> > > >>>> to
> > > declare
> > > >> high-level types. Maybe that's just out of style?
> > > >>>>
> > > >>>>
> > > >>>> More below...
> > > >>>>
> > > >>>> I look forward to hearing back from you.
> > > >>>>
> > > >>>> - Paul
> > > >>>>
> > > >>>> On Apr 3, 2015, at 1:45 PM, Michael Lawrence <
> > > lawrence.michael at gene.com>
> > > >> wrote:
> > > >>>>
> > > >>>>> Thanks to Val's excellent newsletter, I've had my first
> > > >>>>> glance at
> > > >>>>> BrowserViz. I'm glad to see something that is more flexible
> > > >>>>> and
> > > >> low-level
> > > >>>>> than e.g. shiny.
> > > >>>>>
> > > >>>>> I'm curious about the motivation behind web sockets. I guess
> > > >>>>> any
> > > >>>>> application with an R-driven web UI actually has two UIs: the
> > > >>>>> R
> > > >> console and
> > > >>>>> the browser. But what if the R session is headless, or if
> > > >>>>> there is no
> > > >> need
> > > >>>>> for commands in R to affect the browser? Then the web socket
> > > >>>>> layer
> > > >> brings
> > > >>>>> mostly unneeded complexity.
> > > >>>>
> > > >>>> I see websockets (like TCP sockets) as musch simpler than
> > > >>>> HTTP.  No
> > > >> headers,
> > > >>>> no explicit server and explicit client (once the connection is
> > > >>>> open).
> > > >>>> Could you explain the complexity you see?
> > > >>>>
> > > >>>>> An interesting comparison to BrowserViz is not
> > > >>>>> shiny but OpenCPU. It's purely HTTP-based and still manages
> > > >>>>> to
> > > maintain
> > > >>>>> state (not sure how efficiently). I guess one advantage of
> > > >>>>> web
> > > sockets
> > > >> is
> > > >>>>> that one can program imperatively instead of declaratively on
> > > >>>>> the
> > > >> server,
> > > >>>>> i.e., the server can send a command to show a popup in
> > > >>>>> response to
> > > some
> > > >>>>> event, instead of returning a "declaration" that the popup
> > > >>>>> should be
> > > >> shown.
> > > >>>>
> > > >>>> Exactly!
> > > >>>>
> > > >>>>>
> > > >>>>> So essentially web sockets are more natural for implementing
> > > >> server-side
> > > >>>>> controllers (think MVC), instead of just the data model, but
> > > >>>>> man,
> > > it's
> > > >> a
> > > >>>>> shame to lose the features of HTTP.
> > > >>>>
> > > >>>> I'll confess:  I set out to -shed- the features of HTTP.
> > > >>>>  Isn't it a
> > > >> protocol designed for serving up web pages on demand?  Not for
> > > >> fast,
> > > >> lighweight peer-to-peer communications?
> > > >>>>
> > > >>>>
> > > >>>>>
> > > >>>>> Ultimately, I think we want web apps that are easy to develop
> > > >>>>> and
> > > >> maintain,
> > > >>>>> and run equally well from either a useR's session or a remote
> > > >>>>> client
> > > >>>>> communicating to a dedicated, headless server. Is the
> > > >>>>> generality of
> > > >>>>> websockets worth the complexity?
> > > >>>>
> > > >>>>>
> > > >>>>> As an aside, it would seem relatively straight-forward to
> > > >>>>> implement a
> > > >>>>> simple bi-directional RPC mechanism between R and JS using
> > > >>>>> the
> > > standard
> > > >>>>> protocol (i.e., hide details like the callback). Does that
> > > >>>>> sound
> > > >> reasonable?
> > > >>>>>
> > > >>>>> I was also a bit surprised about the need to copy/paste the
> > > >>>>> JS
> > > >> boilerplate.
> > > >>>>> Certainly there must be javascript frameworks with a more
> > > >>>>> elegant
> > > >> solution
> > > >>>>> to extensibility.
> > > >>>>
> > > >>>> You are correct on this, and I only need to correct the
> > > >>>> vignette,
> > > >> written many weeks ago.
> > > >>>> I now provide "BrowserViz.jx", a simple Javascript module.  It
> > > >>>> is used
> > > >> by both
> > > >>>> BrowserVizDemo and RCyjs.  Sorry to have sent out an obsolete
> > > vignette.
> > > >>>>
> > > >>>>>
> > > >>>>>     [[alternative HTML version deleted]]
> > > >>>>>
> > > >>>>> _______________________________________________
> > > >>>>> Bioc-devel at r-project.org mailing list
> > > >>>>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> > > >>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >
> > > >    [[alternative HTML version deleted]]
> > > >
> > > > _______________________________________________
> > > > Bioc-devel at r-project.org mailing list
> > > > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> > >
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > Bioc-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/bioc-devel
> >
>

	[[alternative HTML version deleted]]

More information about the Bioc-devel mailing list