[BioC] An issue in flowCore-package (?)

[Florian Hahne]

The problem is that the $DATATYPE keyword is broken in the FCS3 file.  
Keyword value pairs are supposed to be separated by a particular  
character according to the FCS specifications ('/' here), and if you  
take a closer look at the file you will notice that there are a bunch  
of spaces before $DATATYPE. flowCore correctly reads this keyword as
[8]  
"                                                                                                     $ 
DATATYPE"
which of course will cause the software to break when attempting to  
access it by name. If this is a common problem for Partec files I can  
hard-code a fix for this this into our read.FCS function, issuing a  
warning that the $DATATYPE keyword is invalid but that we try to guess  
the correct value. However, I will need the full FCS file in order to  
test this, the truncated header information is not sufficient. There  
might be other issues with these file given the fact that they  
couldn't even get the trivial keywords right...
Florian



On 04.08.2009, at 00:14, Kyösti Karttunen wrote:

> Hi Florian,
> Thanks for prompt reply!
>
>> I am slightly worried about the fact that your fcs files don't  
>> contain
>> a $DATATYPE keyword,...
> Hmmm.. ???
> "Key"-keywords were also the first things I checked earlier.
> That is why I mailed the first lines of the header of a (ver3) fcs  
> file
> already. And, I Can see the $DATATYPE keyword in both fcs2 and fcs3- 
> headers,
> although in different places. In FCS2 it is in position 60
> and in FCS3 (near) position 260.
> Anyhow, attaced examples of the fcs-headers of same data
> in both fcs2- and fcs3-formats.
> Kyösti
>
>
> On Thu, 30 Jul 2009 21:00:37 +0300, Florian Hahne <fhahne at fhcrc.org>  
> wrote:
>
>> Hi Kyoesti,
>> I am slightly worried about the fact that your fcs files don't  
>> contain
>> a $DATATYPE keyword, which is mandatory according to the FCS specs.
>> That said, fhe fact that some instrument manufacturers don't adhere  
>> to
>> this standard is no news, and we have invested some efforts to allow
>> for these inconsistencies during file import. If you could send one  
>> of
>> the offending FCS files I will take a look and see what can be done.
>> Florian
>>
>>
>> On 29.07.2009, at 07:30, Kyösti Karttunen wrote:
>>
>>> Dear Florian
>>> Our lab is using flow cytometry the first years
>>> for environmental and industrial cases.
>>> My post here is to "take care" of the data handling issues.
>>>
>>> Having used R some time, and after visiting
>>> the bioconductor-pages I found the nice packages
>>> of you and your teams.
>>>
>>>> However, the FCS 3.0 standard includes some options
>>>> that are not yet implemented in this function.
>>>> If you need extensions, please let me know. )
>>>
>>> Anyhow, learnig the trying them, I run into an issue(?) :
>>>
>>> We have a Partec FCM-unit using Flomax (ver 2.56) software.
>>> I am not able to read its the FCS3 -files
>>> with read.FCS() of flowCore.
>>>
>>> An example (pifi is FCS3-file),
>>> selected lines:
>>>
>>>> pix <- read.FCS(pifi, which.lines(100), transformation = FALSE)
>>> Error in readFCSgetPar(x, "$DATATYPE") :
>>> Parameter(s) $DATATYPE not contained in 'x'
>>>
>>> # ... then I read only some header params:
>>>> pixh2 <- read.FCSheader(pifi, keyword=c("FCSversion", "$BEGINDATA",
>>>> "$ENDTEXT", "$DATATYPE", "$PAR") )
>>>> pixh2
>>>
>>> $`F:/Data/pimike/2009/090227/01_1_654195.FCS`
>>> FCSversion $BEGINDATA   $ENDTEXT       <NA>       $PAR
>>>      "3"     "6421"     "3413"         NA       "14"
>>>
>>> # And here the corrensponding data file (in hex & ascii):
>>>
>>> 46 43 53 33  2E 30 20 20  20 20 20 20  20 20 20 20  FCS3.0
>>> 35 38 20 20  20 20 33 34  31 33 20 20  20 20 36 34  58    3413    64
>>> 32 31 20 37  33 37 37 35  33 33 20 20  20 20 33 34  21 7377533    34
>>> 31 34 20 20  20 20 36 34  32 30 2F 24  42 45 47 49  14    6420/$BEGI
>>> 4E 41 4E 41  4C 59 53 49  53 2F 33 34  31 34 2F 24  NANALYSIS/3414/$
>>> 42 45 47 49  4E 44 41 54  41 2F 36 34  32 31 2F 24  BEGINDATA/6421/$
>>> 42 45 47 49  4E 54 45 58  54 2F 35 38  2F 24 45 4E  BEGINTEXT/58/$EN
>>> 44 41 4E 41  4C 59 53 49  53 2F 36 34  32 30 2F 24  DANALYSIS/6420/$
>>> 45 4E 44 44  41 54 41 2F  37 33 37 37  35 33 33 2F  ENDDATA/7377533/
>>> 24 45 4E 44  54 45 58 54  2F 33 34 31  33 2F 20 20  $ENDTEXT/3413/
>>> 20 20 20 20  20 20 20 20  20 20 20 20  20 20 20 20
>>> 20 20 20 20  20 20 20 20  20 20 20 20  20 20 20 20
>>> 20 20 20 20  20 20 20 20  20 20 20 20  20 20 20 20
>>> 20 20 20 20  20 20 20 20  20 20 20 20  20 20 20 20
>>> 20 20 20 20  20 20 20 20  20 20 20 20  20 20 20 20
>>> 20 20 20 20  20 20 20 20  20 20 20 20  20 20 20 20
>>> 20 20 20 24  44 41 54 41  54 59 50 45  2F 49 2F 24     $DATATYPE/I/$
>>> 50 41 52 2F  31 34 2F 24  4D 4F 44 45  2F 4C 2F 24  PAR/14/$MODE/L/$
>>> 50 31 42 2F  31 36 2F 24  50 32 42 2F  31 36 2F 24  P1B/16/$P2B/16/
>>> :
>>> : etc
>>> :
>>>
>>>
>>> Reading FCS2-data by read.FCS() seems to work fine:
>>>
>>>> pixh2 <- read.FCSheader(pifi22, keyword=c("FCSversion",
>>>> "$BEGINDATA", "$ENDTEXT", "$DATATYPE", "$PAR", "$MODE") )
>>>> pixh2
>>> $`F:/Data/pimike/2009/090227/02_1_654290--FCS2.FCS`
>>> FCSversion       <NA>       <NA>  $DATATYPE       $PAR      $MODE
>>>      "2"         NA         NA        "I"       "14"        "L"
>>>
>>> We Could convert files manually, but changing hundreds of files
>>> is not so tempting ;)
>>>
>>> Usually there are ways to handle things.
>>> Anyhow, I would appreciate your comment on this issue.
>>>
>>> And, good luck for your work!
>>> Best regards,
>>> Kyösti Karttunen
>>> research scientist / physics
>>> University of Oulu
>>> Measurement and Sensor Lab (in Kajaani)
>>> Finland
> <PartecFCS2and3headerExamples.zip>