[BioC] flowCore: logicle transformation of flow cytometry data

[Josef Spidlen]

Hi Nishant,
> Now that you mention there are issues with the implementation of flowCores
> logicle transform  it would be great if you could provide more information
> regarding what exactly is incorrect here so that the issue can be corrected.
I believe that there is a minor issue related to parameterization and 
maybe the use of natural vs. decadic logs in the code. When you use 
logicle with say d=5 and r=the range of your data, then the max of your 
data after the transformation should be d, i.e, 5. Your implementation 
seems to stretch it to 5*ln(10). Looking at the actual values, I wasn't 
able to match it to my implementation even when I set the range to 
5*ln(10). So there may be issues with my implementation, your 
implementation, or both of them :-)... or maybe just with matching the 
parameters... Anyways, I guess I'll get back to you offline with more 
details... And I guess I should get my implementation validated before 
blaming you :-)

> Has the results/software used for the Gating ML standards unit tests for
> the logicle transformation been verified by Parks and Moore to be correct
> so that everyone has a gold standard that has been verified to be correct
> from the original authors of the transformation?
Not yet but that is the plan and they agreed to doing so. I gave them 
all the necessary bits on September 22 and we have a call with them 
scheduled for October 27. These are busy people but I hope to hear back 
from them by than.
> Also what is the level of compliance to the standards definition for this
> transformation amongst the currently available flow cytometry software.
I guess zero. But some of them are working on it and trying to refactor 
their own code. At this point, third party software is usually 
implementing some kind of logicle transformation that does something 
reasonable to the data but it is not fully compliant. My understanding 
is that they are usually trying to set the parameters automatically 
based on the data but it is not even easy for them to understand their 
own code. This is particularly due to external programmers having 
implemented some functionality in their software and particularly due to 
additional optimization and 'magic' with the data making it nicer and 
faster for the end users (e.g., combining the transformation with 
compensation or applying some smoothing to avoid instrumentation related 
artifacts, such as the fence effect with low values). Some of these 
software vendors are actually going to meet with the authors of the 
transformation during the CDW workshop in Asilomar next week with the 
purpose of sorting this out.

Also, Gating-ML 2.0 (the one with Logicle included) has not been 
released yet. I believe the Logicle transformation in there is kind of 
fixed by now (this took like 8 months to accomplish) but some other 
features will likely be added before Gating-ML 2.0 is released (e.g., BD 
requests the quad gates to be added).

> 3) Future updates will be made to flowCore to
> a. Add the new parameter "A" defined for the transform.
> b. Move all the inputs to the decade scale...
That sounds great... so maybe once you have that, I'll also have my 
implementation validated (and fixed eventually) and than we could 
properly compare it to the updated implementation in flowCore. This 
should help us resolve all outstanding issues and if we can match, it 
would provide additional validation for the both of us.

Thanks again for doing this!

Cheers,
Josef

-- 
Josef Spidlen, Ph.D.
Research Associate, Terry Fox Laboratory, BC Cancer Agency
675 West 10th Avenue, V5Z 1L3 Vancouver, BC, Canada
Tel: +1 (604) 675-8000 x 7755