寻求可替代H2O中h2o.word2vec功能的R语言工具包

Hey there! I’ve run into the exact same issue before when moving from H2O to a restricted production environment. Here are a few solid R-native alternatives that can replicate the functionality of h2o.word2vec (loading pre-trained GloVe embeddings) and h2o.transform (generating average-pooled document vectors) without needing H2O at all:

text2vec is a go-to for text vectorization tasks in R, and it handles pre-trained embeddings smoothly. This is my top recommendation for most cases:

# Install if needed
install.packages("text2vec")
library(text2vec)

# 1. Load your pre-trained GloVe file (adjust the path to your actual file)
glove_path <- "path/to/glove.6B.300d.txt"
glove_raw <- read.delim(glove_path, header = FALSE, stringsAsFactors = FALSE, quote = "", sep = " ", row.names = 1)
glove_matrix <- as.matrix(glove_raw)

# 2. Assume your `tokenized_words` is a list where each element is a tokenized document
#    e.g., tokenized_words <- list(c("hello", "world"), c("data", "science"))

# 3. Prepare the token iterator and vocabulary
text_iterator <- itoken(tokenized_words, progressbar = FALSE)
vocab <- create_vocabulary(text_iterator)
vocab <- prune_vocabulary(vocab, term_count_min = 1) # Keep all terms present in your data
vectorizer <- vocab_vectorizer(vocab)

# 4. Generate document vectors with average pooling (matches H2O's aggregate_method = "AVERAGE")
doc_term_matrix <- create_dtm(text_iterator, vectorizer)
# Multiply DTM by embedding matrix, then normalize by document length to get averages
doc_vecs <- doc_term_matrix %*% glove_matrix[rownames(doc_term_matrix), ]
doc_lengths <- rowSums(doc_term_matrix)
doc_vecs <- doc_vecs / doc_lengths
# Handle empty documents to avoid NA values
doc_vecs[is.na(doc_vecs)] <- 0

If you’re already working with quanteda for text processing, it can integrate pre-trained embeddings seamlessly:

install.packages("quanteda")
library(quanteda)

# Load and format GloVe embeddings
glove_path <- "path/to/glove.6B.300d.txt"
glove_text <- readtext(glove_path)
glove_tokens <- tokens(glove_text$text, split = " ", remove_separators = FALSE)
glove_df <- data.frame(
  term = sapply(glove_tokens, function(x) x[1]),
  matrix(sapply(glove_tokens, function(x) as.numeric(x[-1])), ncol = 300, byrow = TRUE),
  stringsAsFactors = FALSE
)
rownames(glove_df) <- glove_df$term
glove_matrix <- as.matrix(glove_df[, -1])

# Convert your tokenized data to quanteda tokens
docs_tokens <- tokens(tokenized_words)

# Generate average-pooled document vectors
doc_dfm <- dfm(docs_tokens)
# Filter to only terms present in GloVe
common_terms <- intersect(featnames(doc_dfm), rownames(glove_matrix))
doc_dfm_filtered <- doc_dfm[, common_terms]
# Calculate average vectors
doc_vecs <- doc_dfm_filtered %*% glove_matrix[common_terms, ]
doc_lengths <- rowSums(doc_dfm_filtered)
doc_vecs <- doc_vecs / doc_lengths
doc_vecs[is.na(doc_vecs)] <- 0

If your production environment has strict package restrictions, you can do this entirely with base R:

# Load GloVe embeddings into a named matrix
glove_path <- "path/to/glove.6B.300d.txt"
glove_lines <- readLines(glove_path)
glove_list <- lapply(glove_lines, function(line) {
  parts <- strsplit(line, " ")[[1]]
  list(term = parts[1], vec = as.numeric(parts[-1]))
})
glove_matrix <- do.call(rbind, lapply(glove_list, function(x) x$vec))
rownames(glove_matrix) <- sapply(glove_list, function(x) x$term)

# Generate average document vectors directly from your tokenized list
doc_vecs <- t(sapply(tokenized_words, function(tokens) {
  # Keep only tokens that exist in GloVe
  valid_tokens <- tokens[tokens %in% rownames(glove_matrix)]
  # Return zero vector if no valid tokens
  if (length(valid_tokens) == 0) return(rep(0, 300))
  # Calculate average of the embeddings
  colMeans(glove_matrix[valid_tokens, ])
}))

All these solutions will give you document vectors identical in purpose to what you got from H2O—just pick the one that fits your production environment’s constraints best!

内容的提问来源于stack exchange，提问作者A. Prok