BOLDNODE

BOLDNODE is an R package that offers functionality to efficiently explore BOLD dataset releases (https://boldsystems.org/data/data-packages/) in the Barcode Core Data Model (BCDM) format locally (for more information on BCDM please visit its GitHub repo https://github.com/boldsystems-central/BCDM). It uses a DuckDB back end to query parquet files directly in R, enabling fast searches even on systems with limited RAM. Data collection is optimized through customized chunk sizes and configurable system pause intervals.

The package also allows seamless conversion of search results into standard R data structures without collecting the data in memory for downstream analyses:

1. occurrence matrix (biodiversity and ecology related analyses)
2. sf (spatial data analyses)
3. DNAStringset (phylogenetic data analyses)
4. fasta (phylogenetic data analyses in third-party tools or in R)

User manual

The user manual for the package can be downloaded from the following link: (https://github.com/boldsystems-central/BOLDconnectR_examples/blob/main/BOLD.NODE_v0.0.3.pdf)

Installation

The package can be installed using devtools::install_github function from the devtools package in R (which first needs to be installed).

devtools::install_github("https://github.com/sameerpadhye/BOLDNODE.git")

Downloading Data Packages

Users need to log into BOLD (https://bench.boldsystems.org/index.php/Login/page?destination=MAS_Management_UserConsole) to download datasets in parquet format. Users can then directly use the file as input for the search and vocabulary functions.

BOLDNODE has 12 functions:

1. bcdm_field_names
2. bcdm_field_values
3. bold_parquet_search
4. bold_search_collect
5. get_bin_consensus
6. get_bin_reps
7. get_concise_summary
8. bcdm_to_dnastringset
9. bcdm_to_dwc
10. bcdm_to_fasta
11. bcdm_to_occmatrix
12. bcdm_to_sf

Note Function 8: bcdm_to_dnastringset requires the package Biostrings to be installed and imported in the R session beforehand. It can be installed using BiocManager package.

# if (!requireNamespace("BiocManager", quietly=TRUE))
#
# install.packages("BiocManager")
#
# BiocManager::install("Biostrings")

Workflow for search and collect

A typical workflow for exploring and BOLD data (Steps in italics are optional but useful in some instances):

1. bcdm_field_values and bcdm_field_values (Provide the names of different BCDM fields and unique terms present in a particular field, making it easier for exploring bold_parquet_search search parameters).
2. bold_parquet_search (Searches the dataset based on the user criteria and prints the number of records available).
3. bold_search_collect(Collects the output of the bold_parquet_search in memory for downstream exploration/analyses).
4. Optional: Transform the searched data into a fasta or sf or DNAStringset or an occurrence matrix for downstream analyses (the bcdm_to functions).
5. Optional: Generate concise summary and/or BIN-centric summaries of the searched data based on various parameters, e.g., taxonomy, attributions, sequence meta data (the get_ functions).

1. Get the vocabulary for specific fields

This function can be used to get unique values of some of the categorical fields (e.g. institutes) to make searches easier.

# parquet_file<-'path where the parquet file from BOLD is downloaded'

# vocab.data <- bcdm_field_values(parquet_file,specific.cols = c("country.ocean"))

2. Search the dataset

This function lets users search by more than 10 different search parameters including taxonomy (species, genus, family etc.), geography (country.ocean, province, region), ids (processid, sampleid) and more. The user can simply input the search query directly as an argument (e.g. Canada as the geography argument or Coleoptera as the taxonomy argument) with the function searching the query in the relevant fields internally. The search result is a tbl_sql object that is used for collecting or transforming the data.

# parquet_file<-'path where the parquet file from BOLD is downloaded'

# 1 Taxonomy

# bold_search_taxonomy <- bold_parquet_search(input.parquet=parquet_file,
# taxonomy = c("Odonata","Poecilia"))

# 2 Geography

# 2a without specifying a geographic scope
# bold_search_geography <- bold_parquet_search(input.parquet=parquet_file,
# taxonomy= c("Panthera pardus"),
# scope.geography = 'any',
# geography = c("India"))

# 2b specifying 'country.ocean' as the geographic scope (so that the search will only get the 
# records where India is the assigned country)
# bold_search_geography <- bold_parquet_search(input.parquet=parquet_file,
# taxonomy= c("Panthera pardus"),
# scope.geography = 'country.ocean',
# geography = c("India"))


# 3 Combination of many search criteria

# bold_search_combination <- bold_parquet_search(
# input.parquet=parquet_file,
# taxonomy= "Coleoptera",
# geography = "Canada",
# marker = "COI-5P",
# basecount = c(500, 660))

3. Collect the searched data

The searched data can be collected in memory using this function. Please note Some queries (e.g., all “Diptera”) may return very large datasets. Always check the printed message in the console (shows the total records in the search) after search and before collecting data to ensure you don’t exceed the available RAM.

# Collect data (no export)

# bold_search_geography <- bold_parquet_search(input.parquet=parquet_file,
# taxonomy = c("Panthera pardus"),
# geography = c("India"))

# collected_data<-bold_search_collect(
# bold_search_geography,
# chunk.size = 50000,
# export = FALSE)

# Collect data (with parquet export)

# collected_data_w_export<-bold_search_collect(
# bold_search_geography,
# chunk.size = 50000,
# export = TRUE,
# export.type = "parquet",
# output.path = userdefinedpath)

The get_ functionality

The get_ functions provide multiple ways of summarizing the data. The get_concise_summary function gives a succinct summary of the searched data. For each Barcode Index Number (BIN) in the search data, get_bin_consensus returns consensus taxonomy, while get_bin_reps selects a sample of representative records based on provided criteria.

get_bin_reps

Returns a dataset with one or more representative record(s) from each BIN based on specified criteria.

# bold_search <- bold_parquet_search(
# input.parquet = parquet_file,
#  taxonomy = "Araneae",
# geography = "Canada")

# Select one representative per BIN from the searched data based on sequence length of 658 basepairs
# bin_tax_reps <- get_bin_reps(
# bold.search.res = bold_search,
# criteria = list(seq_length = 658)
# )

get_concise_summary

Returns a concise summary of the searched data. Search results include total records, total countries, amplicon length range and many more.

#  Search the data
# bold_search <- bold_parquet_search(
# input.parquet=parquet_file,
# taxonomy = "Coleoptera",
# geography = "Canada",
# marker = "COI-5P",
# basecount = c(500, 660))

#  Get concise summary of the data
# bold_summary <- get_concise_summary(bold_search)

The bcdm_to_ functionality

The bcdm_to_* functions convert the output of bold_parquet_search into objects compatible with packages such as vegan, msa, DECIPHER, terra, and geodata, enabling seamless interoperability with the broader R ecosystem.

bcdm_to_fasta

Creates a fasta file with customized headers of the searched data. This can be exported locally for any downstream analytical pipelines in third party tools.

#  Search the data

# bold_search <- bold_parquet_search(
# input.parquet=parquet_file,
# taxonomy = "Coleoptera",
# geography = "Canada",
# marker = "COI-5P",
# basecount = c(500, 660))

# Get fasta

# bcdm_to_fasta(
# bold_search,
# output.file = "trial.fas",
# fas.header = c("bin_uri", "processid"))

bcdm_to_sf

Generates a sf object of the searched data for any downstream spatial data analyses.

#  Search the data

# bold_search <- bold_parquet_search(
# input.parquet=parquet_file,
# taxonomy = "Coleoptera",
# geography = "Canada",
# marker = "COI-5P",
# basecount = c(500, 660))

# Get sf

# sf_res <- bcdm_to_sf(bold_search, chunk.size = 100000)

bcdm_to_occmatrix

Creates an occurrence matrix from the searched data based on the taxon.rank, taxon.name (optional) and the site.cat.

#  Search the data
# bold_search <- bold_parquet_search(
# input.parquet=parquet_file,
# taxonomy = "Coleoptera",
# geography = "Canada",
# marker = "COI-5P",
# basecount = c(500, 660))

#  Get occurrence data
# occurrence_data <- bcdm_to_occmatrix(
# bold_search,
# taxon.rank = "family",
# site.cat = "region")

bcdm_to_dnastringset

Generates a DNAStringSet (Biostrings object) object of the searched data for any downstream sequence alignment and tree generation with custom headers. The library Biostrings has to be installed and imported before using this function.

#  Search the data
# bold_search <- bold_parquet_search(
# input.parquet=parquet_file,
# taxonomy = "Coleoptera",
# geography = "Canada",
# marker = "COI-5P",
# basecount = c(500, 660))

#  Get DNAStringSet
# bold.dnastringset<-bcdm_to_dnastringset(bold_search,
# marker="COI-5P",
# cols_for_seq_names = c("processid","family"))

Benchmarks

It takes roughly 10 minutes to collect ~10M records on a i7 2.8GHZ 16GB RAM machine

The package is under active development and the functionality is subject to change

Funding

This work was funded by the New Frontiers in Research Fund (NFRF) - Transformation 2020