Assumptions

• Table broken into records (normally by new lines), and records broken into fields (normally by delimiter). But you can’t parse into lines and then into fields because end of a record you need to understand escaping and quoting. This makes the readr C++ interface fundamentally field based.

• Data is fundamentally rectangular, and usually stored in record-major format (input may be ragged but output will always be rectangular dataset).

• Always know number (and type) of fields; don’t always know number of rows. Field types are known in advance (either guessed from first 100 rows, or supplied by user). For large files, cost of determining types is negligible compared to parsing; small files are so quick anyway that a little additional overhead doesn’t matter.

• Key to good performance is to avoid copying and allocation. The C++ API is designed so that we never need to copy data, except at the last possible moment to unescape strings.

Main components

There are four main components:

• A source provides an iterator-based interface to a file, string or raw vector.

  In C++, these are SourceFile, SourceString, SourceRaw, etc. Each source has a corresponding R-representation usually generated by datasource(). Source::create() instantiates the approach class from the R representation.

• A token is an iterator that points to a single value in source. A token also contains metadata about the location of the value (e.g. the row and col, needed for informative error message) and optionally, an unescape method. The unescape method is used if the tokenizer detects escapes (and hence the memory allocated by the source can’t be used directly).

• A tokeniser converts a stream of characters from a source into a stream of tokens. Tokenizers are typically written in DFA style. This is a bit more verbose than informal parsing, but it makes it much easier to verify correctness.

• Field collectors take a stream of tokens, parsing each token and storing it an R vector.

  There is one collector for each column type: CollectorLogical, CollectorInteger, CollectorDouble etc. On the R side, these are represented by col_logical, col_integer(), col_double() etc. Collector::create() dynamically creates a Collector subclass from an R list.

Each component is described in more detail below.

# Create the C++ object from the R spec
TokenizerPtr tokenizer = Tokenizer::create(tokenizerSpec);

# Initialise it with a source
tokenizer->tokenize(source->begin(), source->end());

# Call nextToken until there are no tokens left 
for (Token t = tokenizer->nextToken(); t.type() != TOKEN_EOF; t = tokenizer->nextToken());