Working with files on disk.
Taking the advantage of multicore machines.
This vignette demonstrates some advanced features of the text2vec package: how to read large collections of text stored on disk rather than in memory, and how to let text2vec functions use multiple cores.
In many cases, you will have a corpus of texts which are too large to fit in memory. This section demonstrates how to use text2vec to vectorize large collections of text stored in files.
Imagine we have a collection of movie reviews stored in multiple text files on disk. For this vignette, we will create files on disk using the movie_review dataset:
library(text2vec)
library(magrittr)
data("movie_review")
# remove all internal EOL to simplify reading
movie_review$review = gsub(pattern = '\n', replacement = ' ',
x = movie_review$review, fixed = TRUE)
N_FILES = 10
CHUNK_LEN = nrow(movie_review) / N_FILES
files = sapply(1:N_FILES, function(x) tempfile())
chunks = split(movie_review, rep(1:N_FILES,
each = nrow(movie_review) / N_FILES ))
for (i in 1:N_FILES ) {
write.table(chunks[[i]], files[[i]], quote = T, row.names = F,
col.names = T, sep = '|')
}
# Note what the moview review data looks like
str(movie_review, strict.width = 'cut')## 'data.frame': 5000 obs. of 3 variables:
## $ id : chr "5814_8" "2381_9" "7759_3" "3630_4" ...
## $ sentiment: int 1 1 0 0 1 1 0 0 0 1 ...
## $ review : chr "With all this stuff going down at the moment with MJ"..The text2vec provides functions to easily work with files. You need to follow a few steps.
ifiles() function.reader() function to ifiles() that can read those files. You can use a function from base R or any other package to read plain text, XML, or other files and convert them to text. The text2vec package doesn’t handle the reading itself. reader function should return NAMED character vector:
idsfilename + line_number (assuming that each line is a separate document)itoken() function.Let’s see how it works:
library(data.table)
reader = function(x, ...) {
# read
chunk = data.table::fread(x, header = T, sep = '|')
# select column with review
res = chunk$review
# assign ids to reviews
names(res) = chunk$id
res
}
# create iterator over files
it_files = ifiles(files, reader = reader)
# create iterator over tokens from files iterator
it_tokens = itoken(it_files, preprocess_function = tolower,
tokenizer = word_tokenizer, progessbar = FALSE)
vocab = create_vocabulary(it_tokens)##
|
|====== | 10%
|
|============= | 20%
|
|==================== | 30%
|
|========================== | 40%
|
|================================ | 50%
|
|======================================= | 60%
|
|============================================== | 70%
|
|==================================================== | 80%
|
|========================================================== | 90%
|
|=================================================================| 100%Now are able to construct DTM:
dtm = create_dtm(it_tokens, vectorizer = vocab_vectorizer(vocab))##
|
|====== | 10%
|
|============= | 20%
|
|==================== | 30%
|
|========================== | 40%
|
|================================ | 50%
|
|======================================= | 60%
|
|============================================== | 70%
|
|==================================================== | 80%
|
|========================================================== | 90%
|
|=================================================================| 100%str(dtm, list.len = 5)## Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
## ..@ i : int [1:746105] 4998 4996 4996 4995 4993 4992 4992 4992 49..
## ..@ p : int [1:47107] 0 1 2 3 4 5 6 7 8 9 ...
## ..@ Dim : int [1:2] 5000 47106
## ..@ Dimnames:List of 2
## .. ..$ : chr [1:5000] "5814_8" "2381_9" "7759_3" "3630_4" ...
## .. ..$ : chr [1:47106] "Eamonn" "Link" "rumpled" "portfolio" ...
## ..@ x : num [1:746105] 1 1 1 1 1 1 1 1 1 1 ...
## .. [list output truncated]Note that the DTM has document ids. They are inherited from the document names we assigned in reader function. This is a convenient way to assign document IDs when working with files.
Fall back to auto-generated ids. Lets see how text2vec would handle the cases when user didn’t provide document ids:
for (i in 1:N_FILES ) {
write.table(chunks[[i]][["review"]], files[[i]], quote = T, row.names = F,
col.names = T, sep = '|')
}
# read with default reader - readLines
it_files = ifiles(files)
# create iterator over tokens from files iterator
it_tokens = itoken(it_files, preprocess_function = tolower,
tokenizer = word_tokenizer, progessbar = FALSE)
dtm = create_dtm(it_tokens, vectorizer = hash_vectorizer())##
|
|====== | 10%
|
|============= | 20%
|
|==================== | 30%
|
|========================== | 40%
|
|================================ | 50%
|
|======================================= | 60%
|
|============================================== | 70%
|
|==================================================== | 80%
|
|========================================================== | 90%
|
|=================================================================| 100%str(dtm, list.len = 5)## Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
## ..@ i : int [1:745785] 819 3105 3348 4900 4972 172 1013 3717 4032..
## ..@ p : int [1:262145] 0 0 0 0 0 0 0 0 0 1 ...
## ..@ Dim : int [1:2] 5010 262144
## ..@ Dimnames:List of 2
## .. ..$ : chr [1:5010] "fileb61e230ee5e5_1" "fileb61e230ee5e5_2" "fileb"..
## .. ..$ : chr(0)
## ..@ x : num [1:745785] 1 1 1 1 1 1 1 1 2 1 ...
## .. [list output truncated]For many tasks text2vec allows to take the advantage of multicore machines. The functions create_dtm(), create_tcm(), and create_vocabulary() are good example. In contrast to GloVe fitting which uses low-level thread parallelism via RcppParallel, these functions use standard high-level R parallelizatin provided by the foreach package. They are flexible and can use diffrent parallel backends, such as doParallel() or doRedis(). But remember that such high-level parallelism might involve significant overhead.
The user must do two things manually to take advantage of a multicore machine:
itoken iterators.Here is simple example:
N_WORKERS = 4
library(doParallel)
# register parallel backend
registerDoParallel(N_WORKERS)
# prepare splits
# "jobs" is a list of itoken iterators!
N_SPLITS = 4
jobs = files %>%
split_into(N_SPLITS) %>%
lapply(ifiles, reader = reader) %>%
# Worth to set chunks_number to 1 because we already splitted input
lapply(itoken, chunks_number = 1, preprocess_function = tolower,
tokenizer = word_tokenizer, progessbar = FALSE)
# Alternatively when data is in memory we can perform splite in the following way:
#
# review_chunks = split_into(movie_review$review, N_SPLITS)
# review_ids = split_into(movie_review$id, N_SPLITS)
#
# jobs = Map(function(doc, ids) {
# itoken(iterable = doc, ids = ids, preprocess_function = tolower,
# tokenizer = word_tokenizer, chunks_number = 1, progessbar = FALSE)
# }, review_chunks, review_ids)
# Now all below function calls will benefit from multicore machines
# Each job will be evaluated in separate process
# vocabulary creation
vocab = create_vocabulary(jobs)
# DTM vocabulary vectorization
v_vectorizer = vocab_vectorizer(vocab)
vocab_dtm_parallel = create_dtm(jobs, vectorizer = v_vectorizer)
# DTM hash vectorization
h_vectorizer = hash_vectorizer()
hash_dtm_parallel = create_dtm(jobs, vectorizer = h_vectorizer)
# co-ocurence statistics
tcm_vectorizer = vocab_vectorizer(vocab, grow_dtm = FALSE, skip_grams_window = 5)
tcm_parallel = create_tcm(jobs, vectorizer = tcm_vectorizer)