[R] laf_open_fwf

Fri Aug 9 16:43:31 CEST 2013

Jan,

Many thanks for your suggestion! The code runs perfectly fine on the test set. Applying it to the complete data set, however, results in the following error:

> while (TRUE) {
+              lines <- readLines(con, encoding='LATIN1')
+              if (length(lines) == 0) break
+              lines <- sprintf("%-238s", lines)
+              writeLines(lines, out, useBytes=TRUE) }
Error: cannot allocate vector of size 23.2 Mb

Best Regard

Christian Kamenik
Project Manager

Federal Department of the Environment, Transport, Energy and Communications DETEC  
Federal Roads Office FEDRO
Division Road Traffic
Road Accident Statistics

Mailing Address: 3003 Bern
Location: Weltpoststrasse 5, 3015 Bern

Tel +41 31 323 14 89 
Fax +41 31 323 43 21

christian.kamenik at astra.admin.ch
www.astra.admin.ch

-----Ursprüngliche Nachricht-----
Von: Jan van der Laan [mailto:rhelp at eoos.dds.nl] 
Gesendet: Freitag, 9. August 2013 10:01
An: Kamenik Christian ASTRA
Betreff: Re: AW: AW: [R] laf_open_fwf

Christian,

It seems some of the lines in your file have additional characters at the end causing the line lengths to vary. The only way I could think of is to first add whitespace to the shorter lines to make all line lengths equal:

# Add whitespace to the end of the lines to make all lines the same length con <- file("testdata.txt", "rt") out <- file("testdata_2.txt", "wt") while (TRUE) {
   lines <- readLines(con, n=1E5)
   if (length(lines) == 0) break
   lines <- sprintf("%-238s", lines)
   writeLines(lines, out, useBytes=TRUE) }
close(con)
close(out)

I am then able to read you test file using LaF:

library(LaF)

column_widths <- c(3, 28, 4, 30, 28, 6, 3, 30, 10, 26, 25, 30, 2, 5, 5) column_types <- rep("string", length(column_widths)) column_types[c(1, 3, 7)] <- "integer"

laf <- laf_open_fwf("testdata_2.txt", column_types = column_types, column_widths = column_widths)

HTH,
Jan

christian.kamenik at astra.admin.ch schreef:

> Hello Jan
>
> I attached an example. Any help is highly appreciated!
>
> Kind Regard
>
> Christian Kamenik
> Project Manager
>
> Federal Department of the Environment, Transport, Energy and 
> Communications DETEC Federal Roads Office FEDRO Division Road Traffic 
> Road Accident Statistics
>
> Mailing Address: 3003 Bern
> Location: Weltpoststrasse 5, 3015 Bern
>
> Tel +41 31 323 14 89
> Fax +41 31 323 43 21
>
> christian.kamenik at astra.admin.ch
> www.astra.admin.ch
> -----Ursprüngliche Nachricht-----
> Von: Jan van der Laan [mailto:rhelp at eoos.dds.nl]
> Gesendet: Donnerstag, 8. August 2013 13:58
> An: r-help at r-project.org
> Cc: Kamenik Christian ASTRA
> Betreff: Re: AW: [R] laf_open_fwf
>
>
> Without example data it is difficult to give suggestions on how you 
> might read this file.
>
> Are you sure your file is fixed width? Sometimes columns are neatly 
> aligned using whitespace (tabs/spaces). In that case you could use 
> read.table with the default settings.
>
> Another possibility might be that the file is encoded in utf8. I 
> expect that reading it in assuming another encoding (such as latin1) 
> would lead to varying line sizes. Although I would expect the lengths 
> to be larger than the sum of your column widths (as one symbol can be 
> larger than one byte).
>
> Jan
>
>
>
> christian.kamenik at astra.admin.ch schreef:
>
>> Dear Jan
>>
>> Many thanks for your help. In fact, all lines are shorter than my 
>> column width...
>>
>> my.column.widths:	238
>> range(nchar(lines)):	235 237
>>
>> So, it seems I have an inconsistent file structure...
>> I guess there is no way to handle this in an automated way?
>>
>> Best Regard
>>
>> Christian Kamenik
>> Project Manager
>>
>> Federal Department of the Environment, Transport, Energy and 
>> Communications DETEC Federal Roads Office FEDRO Division Road Traffic 
>> Road Accident Statistics
>>
>> Mailing Address: 3003 Bern
>> Location: Weltpoststrasse 5, 3015 Bern
>>
>> Tel +41 31 323 14 89
>> Fax +41 31 323 43 21
>>
>> christian.kamenik at astra.admin.ch
>> www.astra.admin.ch
>> -----Ursprüngliche Nachricht-----
>> Von: Jan van der Laan [mailto:rhelp at eoos.dds.nl]
>> Gesendet: Mittwoch, 7. August 2013 20:57
>> An: r-help at r-project.org
>> Cc: Kamenik Christian ASTRA
>> Betreff: Re: [R] laf_open_fwf
>>
>> Dear Christian,
>>
>> Well... it shouldn't normally do that. The only way I can currently 
>> think of that might cause this problem is that the file has \r\n\r\n, 
>> which would mean that every line is followed by an empty line.
>>
>> Another cause might be (although I would not really expect the 
>> results you see) that the sum of your column widths is larger than 
>> the actual with of the line.
>>
>> You can check your line lengths using:
>>
>> lines <- readLines(my.filename)
>> nchar(lines)
>>
>> Each line should have the same length and be equal to (or at least 
>> larger than) sum(my.column.widths)
>>
>> If this is not the problem: would it be possible that you send me a 
>> small part of your file so that I could try to reproduce the problem?
>> Or if you cannot share your data: replace the actual values with 
>> nonsense values.
>>
>> Regards,
>> Jan
>>
>> PS I read your mail by chance as I am not a regular r-help reader.
>> When you have specific LaF problems it is better to also cc me 
>> directly.
>>
>> On 08/06/2013 12:35 PM, christian.kamenik at astra.admin.ch wrote:
>>> Dear all
>>>
>>> I was trying the (fairly new) LaF package, and came across the 
>>> following problem:
>>>
>>> I opened a connection to a fixed width ASCII file using 
>>> laf_open_fwf(my.filename, my.column_types, my.column_widths,
>>> my.column_names)
>>>
>>> When looking at the data, it turned out that \n (newline) and \r 
>>> (carriage return) were considered as characters, thus destroying the 
>>> structure in my data (the second column does not include any
>>> numbers):
>>>
>>>> my.data[1565:1575,1:3]
>>>
>>>     MF_FARZ1              Fahrzeugarttext MF_MARKE
>>> 1     \n043 Landwirt. Traktor                2140
>>> 2     \n043 Landwirt. Traktor                6206
>>> 3     \n001 Personenwagen                    2026
>>> 4     \n001 Personenwagen                    2026
>>> 5    \r\n00 1Personenwagen                    404
>>> 6    \r\n02 0Gesellschaftswagen               710
>>> 7    \r\n00 1Personenwagen                    505
>>> 8    \r\n00 1Personenwagen                    505
>>> 9    \r\n00 1Personenwagen                    301
>>> 10   \r\n00 1Personenwagen                    553
>>> 11   \r\n04 3Landwirt. Traktor                257
>>>
>>> I am working on Windows 7 32-bit.
>>>
>>> Any help would be highly appreciated.
>>>
>>> Best Regard
>>>
>>> Christian Kamenik
>>> Project Manager
>>>
>>> Federal Department of the Environment, Transport, Energy and 
>>> Communications DETEC Federal Roads Office FEDRO Division Road 
>>> Traffic Road Accident Statistics
>>>
>>> Mailing Address: 3003 Bern
>>> Location: Weltpoststrasse 5, 3015 Bern
>>>
>>> Tel +41 31 323 14 89
>>> Fax +41 31 323 43 21
>>>
>>> christian.kamenik at astra.admin.ch<mailto:christian.kamenik at astra.admin.
>>> ch> www.astra.admin.ch<http://www.astra.admin.ch/>
>>>
>>>
>>> 	[[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>