如何在R语言中使用Keras的pad_sequences函数填充文本序列？

Great question! Let's walk through completing your text preprocessing pipeline step by step, including the padding part you're asking about. We'll build on the code you already started and make sure everything works seamlessly with your sample dataset.

Step 1: Finish Setting Up the Tokenizer #

First, let's complete the tokenizer fitting process you started. You had initialized the tokenizer correctly, just needed to finish training it on your text data:

# Your original dataset
dat <- data.frame(
  id=c("1","2","3","4","5"),
  text=as.character(c("hello","hello you","hello duck","Dogs and cats","hello cats, ducks and dogs")),
  stringsAsFactors = F
)

# Load required libraries
library(keras)
library(dplyr)

# Initialize tokenizer (your existing code was spot-on here)
tok <- keras::text_tokenizer(
  num_words = 10, 
  lower = TRUE, 
  split = " ", 
  char_level = FALSE
)

# Fit tokenizer to your text data to build the word index
keras::fit_text_tokenizer(tok, dat$text)

Step 2: Convert Text to Integer Sequences #

Next, we turn each text string into a sequence of integers using the tokenizer's learned word mappings:

# Convert raw text to integer sequences
text_sequences <- keras::texts_to_sequences(tok, dat$text)

# Let's inspect the sequences to see how words are mapped
print(text_sequences)

You'll see output like this, where each unique word is replaced by its assigned index:

[[1]]
[1] 1

[[2]]
[1] 1 2

[[3]]
[1] 1 3

[[4]]
[1] 4 5 6

[[5]]
[1] 1 6 7 5 4

Step 3: Pad Sequences to Uniform Length #

This is the critical step you were missing! Keras models require input data to have consistent dimensions, so we use pad_sequences to make all sequences the same length. You can customize padding and truncation behavior to fit your needs:

# Calculate the length of the longest sequence in your data
max_seq_length <- max(sapply(text_sequences, length))

# Pad sequences to match the longest sequence length
padded_sequences <- keras::pad_sequences(
  text_sequences,
  maxlen = max_seq_length,  # Use longest sequence length or a fixed value
  padding = "post",         # Add padding zeros at the end of sequences
  truncating = "post"       # Truncate extra tokens from the end if sequences are too long
)

# View the final padded sequences
print(padded_sequences)

The result will be a neat matrix where every row has the same length, with 0s filling the padding gaps:

[,1] [,2] [,3] [,4] [,5]
[1,]    1    0    0    0    0
[2,]    1    2    0    0    0
[3,]    1    3    0    0    0
[4,]    4    5    6    0    0
[5,]    1    6    7    5    4

Key Parameter Breakdown #

• maxlen: Set this to the length you want all sequences to be. Using the longest sequence length (as we did) preserves all your data, but you can also use a fixed number if you need to limit input size for longer texts.
• padding: "pre" adds zeros at the start (common for recurrent models), while "post" adds them at the end (easier to interpret visually).
• truncating: If sequences exceed maxlen, this determines whether to cut tokens from the start ("pre") or end ("post").

Full Runnable Code #

Here's the complete pipeline that takes your raw text to padded sequences ready for a Keras classification model:

# Load libraries
library(keras)
library(dplyr)

# Your dataset
dat <- data.frame(
  id=c("1","2","3","4","5"),
  text=as.character(c("hello","hello you","hello duck","Dogs and cats","hello cats, ducks and dogs")),
  stringsAsFactors = F
)

# Initialize and fit tokenizer
tok <- keras::text_tokenizer(num_words = 10, lower = TRUE, split = " ", char_level = FALSE)
keras::fit_text_tokenizer(tok, dat$text)

# Convert text to sequences
text_sequences <- keras::texts_to_sequences(tok, dat$text)

# Pad sequences to uniform length
max_seq_length <- max(sapply(text_sequences, length))
padded_sequences <- keras::pad_sequences(
  text_sequences,
  maxlen = max_seq_length,
  padding = "post",
  truncating = "post"
)

# Final output ready for model input
padded_sequences

Now you can feed padded_sequences directly into the input layer of your Keras text classification model!

内容的提问来源于stack exchange，提问作者captcoma