---
title: "Getting started with raisr"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with raisr}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## What the RAIS is

The RAIS (*Relação Anual de Informações Sociais*) is the annual census of
formal employment in Brazil: every employer declares, for each year, the
employment relationships it held (with admission and separation dates,
occupation, wages month by month, worker characteristics) and the
establishments themselves. The Ministry of Labour and Employment publishes
the public, non-identified microdata on the PDET FTP server, one folder per
reference year:

```
ftp://ftp.mtps.gov.br/pdet/microdados/RAIS/
    1985/ ... 2017/      one file per state (PE2017.7z, ...) + ESTB2017.7z
    2018/ ... 2025/      RAIS_VINC_PUB_<REGION>.7z (7 regional files) + RAIS_ESTAB_PUB.7z
    2023 Parcial/        preliminary edition of a year, when one was published
    2023/Legado/         the previous files, kept when a year is re-published
```

The regional files of the employment relationships (`vinculos`) are large:
about 130 MB compressed for the North, 600 MB for the Northeast and more
than 1 GB for Sao Paulo, each with tens of millions of records.

`raisr` does three things with these files:

1. **lists** what is on the server and resolves which archive holds a state;
2. **downloads** the archives into an idempotent local cache;
3. **reads** them as a stream, filtering by state and selecting columns
   *before* anything is kept in memory.

The third point is what makes the package useful on an ordinary laptop: one
state of the Northeast is a fraction of its regional file, and you never
hold the rest in memory.

## Installation

```{r, eval = FALSE}
# From CRAN (when available):
install.packages("raisr")

# Development version:
# remotes::install_github("StrategicProjects/raisr")
```

The package reads `.7z` archives through the `archive` package, which needs
`libarchive`. It is bundled on Windows and macOS binaries; on Linux install
`libarchive-dev` (Debian/Ubuntu) or `libarchive-devel` (Fedora) first.

## Sample archives ship with the package

Every function that reads data can be tried offline with the small archives
in `inst/extdata`, which keep the exact format of the Ministry's files
(Latin-1 text, original headers, separator and decimal mark of each
generation) for a handful of records from Pernambuco and Bahia.

```{r}
library(raisr)

f <- system.file("extdata", "2024", "RAIS_VINC_PUB_NORDESTE_sample.7z", package = "raisr")
x <- rais_read(f, verbose = FALSE)
x
```

Column names come back normalized (accents removed, lower case, `_` between
words); codes are integers, wages are numbers, classification codes with
leading zeros (CNAE, CBO) stay character. Two columns are added by the
package: `rais_year` (taken from the folder or the file name) and
`rais_type` (`"vinculos"` or `"estabelecimentos"`).

```{r}
rais_layout()[, c("column", "original", "type")]
```

## Reading one state

The employment files have no state column: the state is the first two
digits of the establishment's municipality code, and `rais_read()` filters
on that. Pass the IBGE code or the two-letter abbreviation, and optionally
the columns you need. Both filters are applied chunk by chunk while the
archive is being decompressed.

```{r}
pe <- rais_read(
  f,
  uf = "PE",
  columns = c("municipio", "cnae_20_subclasse", "vinculo_ativo_31_12",
              "mes_admissao", "mes_desligamento", "vl_remun_dezembro_nom"),
  verbose = FALSE
)
pe
```

## Stock, admissions, separations and payroll

`vinculo_ativo_31_12` is `1` for a relationship active on 31 December, which
is what the Ministry calls the employment **stock**. `mes_admissao` and
`mes_desligamento` are `0` when the movement did not happen in the year.
`rais_stock()` turns that into the usual figures, by `rais_year` plus any
grouping columns you ask for:

```{r}
rais_stock(pe, by = "municipio")
```

## Establishments

The establishments file has one row per establishment with the stock of
relationships (`qtd_vinculos_ativos`), activity and size codes and, from
2018, the postal code. It does have a `uf` column, but the state filter
works the same way.

```{r}
e <- system.file("extdata", "2023", "RAIS_ESTAB_PUB_sample.7z", package = "raisr")
rais_read(e, uf = 26, columns = c("municipio", "cnae_20_subclasse", "qtd_vinculos_ativos"),
          verbose = FALSE)
```

## Working with the server

Which archive holds a state is decided offline by `rais_files()`:

```{r}
rais_files(2024, uf = "PE")
rais_files(2017, uf = c("PE", "BA"), type = c("vinculos", "estabelecimentos"))
```

The functions below need network access to `ftp.mtps.gov.br` (port 21 and
the passive-mode data ports). They are not evaluated in this vignette.

```{r, eval = FALSE}
# Everything published for 2024, with sizes and dates
rais_available(year = 2024)

# Download the NORDESTE regional file of 2024 into the cache (about 600 MB)
rais_download(2024, uf = "PE")

# Download and read in one go: Pernambuco, three years, a few columns
pe <- rais_fetch(2022:2024, uf = "PE",
                 columns = c("municipio", "vinculo_ativo_31_12", "vl_remun_dezembro_nom"))

# Stock and December payroll by municipality and year
rais_stock(pe, by = "municipio")
```

By default the cache lives under `tempdir()` and disappears with the R
session. For repeated work set a persistent location once:

```{r, eval = FALSE}
Sys.setenv(RAISR_CACHE_DIR = "~/dados/rais")
rais_cache_list()
```

See `vignette("streaming-and-cache")` for the details of how files are read
and cached, and for the two header generations of the files.