Large files, raw microdata and provenance

Renato Prado Siqueira

2026-08-24

Overview

Record-level health data can be much larger than TABNET aggregates. The package provides two complementary workflows:

Both workflows support discovery, local caching, selected columns, standardized schemas and provenance.

Traditional DBC/DBF microdata

The raw microdata catalog covers SIM, SINASC and SIH/SUS:

microdados_catalogo()

microdados_arquivos(
  sistema = "sih",
  ano = 2024,
  mes = 1,
  uf = "AC"
)

Select only the columns required by the analysis and start with a small number of records:

admissions <- sih_microdados(
  ano = 2024,
  mes = 1,
  uf = "AC",
  colunas = c(
    "MUNIC_RES", "DT_INTER", "DIAG_PRINC", "VAL_TOT"
  ),
  n_max = 1000,
  normalizar = TRUE
)

deaths <- sim_microdados(
  ano = 2023,
  uf = "RR",
  colunas = c("CODMUNRES", "DTOBITO", "CAUSABAS"),
  n_max = 1000,
  normalizar = TRUE
)

births <- sinasc_microdados(
  ano = 2023,
  uf = "RR",
  colunas = c("CODMUNRES", "DTNASC", "SEXO", "PESO"),
  n_max = 1000,
  normalizar = TRUE
)

DBC files are decoded directly and read through the same public interface as DBF files.

Inspect schemas before reading

head(datasus_dicionario("sih"), 10)
#>   sistema tipo      campo           campo_padronizado
#> 1     sih   RD      n_aih                         aih
#> 2     sih   RD       cnes                        cnes
#> 3     sih   RD  munic_res codigo_municipio_residencia
#> 4     sih   RD   dt_inter             data_internacao
#> 5     sih   RD   dt_saida                  data_saida
#> 6     sih   RD diag_princ diagnostico_principal_cid10
#> 7     sih   RD    val_tot                 valor_total
#> 8     sih   RD       sexo                        sexo
#> 9     sih   RD      idade                  idade_anos
#>                               descricao    classe formato
#> 1                  N<U+00FA>mero da AIH character        
#> 2                    C<U+00F3>digo CNES character        
#> 3 Munic<U+00ED>pio de resid<U+00EA>ncia character        
#> 4      Data de interna<U+00E7><U+00E3>o      date  %Y%m%d
#> 5                  Data de sa<U+00ED>da      date  %Y%m%d
#> 6   Diagn<U+00F3>stico principal CID-10 character        
#> 7                  Valor total aprovado   numeric        
#> 8                                  Sexo character        
#> 9                         Idade em anos   numeric

After a read, validate the critical analysis fields:

datasus_validar_esquema(
  admissions,
  sistema = "sih",
  campos = c(
    "codigo_municipio_residencia",
    "data_internacao",
    "diagnostico_principal_cid10",
    "valor_total"
  ),
  estrito = TRUE
)

Selected OpenDataSUS columns

For ordinary files, opendatasus_ler() can select columns while parsing:

resources <- opendatasus_recursos("arboviroses-dengue")
csv_id <- resources$id[resources$formato == "CSV"][1]

sample <- opendatasus_ler(
  "arboviroses-dengue",
  recurso = csv_id,
  colunas = c("DT_NOTIFIC", "SG_UF", "ID_MUNICIP"),
  n_max = 1000
)

Convenience wrappers should be preferred when available because they encode the dataset’s partition rules and curated schema.

Process CSV files in bounded memory

opendatasus_processar() calls a function for each block instead of retaining the entire dataset:

resources <- opendatasus_recursos(
  "notificacoes-de-sindrome-gripal-leve-2020"
)
ms_id <- resources$id[
  resources$formato == "CSV" & grepl("^Dados MS", resources$nome)
][1]

processed <- opendatasus_processar(
  "notificacoes-de-sindrome-gripal-leve-2020",
  recurso = ms_id,
  ano = NULL,
  colunas = c("municipioIBGE", "resultadoTeste"),
  tamanho_bloco = 50000,
  sistema = "sindrome_gripal",
  FUN = function(dados, posicao, arquivo) {
    data.frame(
      arquivo = arquivo,
      bloco_inicial = posicao,
      registros = nrow(dados),
      positivos = sum(
        dados$resultado_teste == "Positivo",
        na.rm = TRUE
      )
    )
  }
)

processed$linhas
processed$blocos
processed$resultados

The callback receives the standardized block when sistema is supplied. Physical parts are processed in catalog order and are listed in processed$arquivos.

Cache and provenance

Downloads are written atomically. Cached files are reused unless atualizar = TRUE is requested:

first <- ocupacao_hospitalar(
  ano = 2022,
  n_max = 1000,
  cache = TRUE
)

source <- datasus_proveniencia(first)
str(source)

The provenance record identifies the official URL, resource, portal update time, local path, download time and MD5 checksum. Multipart reads retain a record for each physical file.

Practical strategy

For large resources:

  1. inspect resource metadata and partitions;
  2. select the smallest useful year, month and state;
  3. provide colunas before increasing n_max;
  4. use opendatasus_processar() for reductions that do not need all rows in memory;
  5. validate critical fields with datasus_validar_esquema();
  6. retain provenance with the analytical output.

n_max limits parsed rows, but a remote file may still need to be downloaded in full before parsing. Cache reuse prevents that transfer from being repeated.