基于PySpark或Koalas实现类Pandas的大规模数据正则匹配优化方案咨询

Got it, let's tackle this problem step by step. Since you're dealing with large datasets, both PySpark and Koalas are excellent choices—Koalas will feel familiar if you’re coming from Pandas, while PySpark gives you full control over Spark’s distributed optimizations. Let’s break down equivalent implementations for both, plus regex optimization tips to boost performance.

Koalas Implementation #

Koalas mirrors the Pandas API almost exactly, so your transition will be smooth. The main adjustments are avoiding inplace=True (Spark DataFrames are immutable) and optimizing regex patterns to cut down on redundant data passes.

import databricks.koalas as ks
import re

# Load your dataset into a Koalas DataFrame (adjust input method as needed)
ks_dataset = ks.read_csv("your_data_source.csv")

# Build the truncated details column with optimized regex steps
ks_dataset['details_trunc'] = (
    ks_dataset['details']
    # Combine 3 numeric-based replacements into one pass to save time
    .str.replace(r'[0-9]+GB? |[0-9]+MB?P?S? |[0-9]+\s?mins? ', '', regex=True, flags=re.IGNORECASE)
    # Remove content wrapped in parentheses
    .str.replace(r"\(.*\)", "", regex=True)
    # Split on $ and keep the first segment
    .str.split("$").str[0]
    # Split on - and keep the first segment
    .str.split("-").str[0]
    # Remove standalone numeric values
    .str.replace(r"\b[0-9]+\b", "", regex=True)
    # Split on 'fr', 'ends', ':' and keep the first segment each time
    .str.split('fr').str[0]
    .str.split('ends').str[0]
    .str.split(':').str[0]
    # Trim leading/trailing whitespace
    .str.strip()
)

# Standardize Apple/Play Store entries
ks_dataset['details_trunc'] = ks_dataset['details_trunc'].str.replace(
    r'Apple App Store.*$', 'Apple App Store', regex=True
)
ks_dataset['details_trunc'] = ks_dataset['details_trunc'].str.replace(
    r'Google Play.*$', 'Google Play', regex=True
)

# Replace empty strings and nulls with 'NA'
ks_dataset['details_trunc'] = ks_dataset['details_trunc'].fillna('NA')
ks_dataset['details_trunc'] = ks_dataset['details_trunc'].replace('', 'NA')

PySpark Implementation #

For full access to Spark’s distributed processing power, use native PySpark DataFrame functions. We’ll chain together built-in functions to replicate your Pandas logic without slow Python UDFs.

from pyspark.sql import SparkSession
from pyspark.sql.functions import regexp_replace, split, element_at, trim, when, col
import re

# Initialize Spark session (tweak configs for your cluster if needed)
spark = SparkSession.builder.appName("DetailsTruncation").getOrCreate()

# Load your dataset into a PySpark DataFrame
spark_dataset = spark.read.csv("your_data_source.csv", header=True, inferSchema=True)

# Build the truncated details column with optimized steps
spark_dataset = spark_dataset.withColumn(
    "details_trunc",
    trim(
        element_at(
            split(
                regexp_replace(
                    regexp_replace(
                        element_at(
                            split(
                                element_at(
                                    split(
                                        col("details"),
                                        r"\$"  # Split on $
                                    ),
                                    1  # Take first segment
                                ),
                                r"-"  # Split on -
                            ),
                            1  # Take first segment
                        ),
                        # Combine numeric patterns into one replacement
                        r'[0-9]+GB? |[0-9]+MB?P?S? |[0-9]+\s?mins? ', '', flags=re.IGNORECASE
                    ),
                    r"\(.*\)", ""  # Remove parenthetical content
                ),
                r'fr|ends|:',  # Split on 3 delimiters in one pass
                1  # Take first segment
            )
        )
    )
)

# Standardize Apple/Play Store entries
spark_dataset = spark_dataset.withColumn(
    "details_trunc",
    when(
        col("details_trunc").rlike(r'Apple App Store.*$'),
        'Apple App Store'
    ).when(
        col("details_trunc").rlike(r'Google Play.*$'),
        'Google Play'
    ).otherwise(col("details_trunc"))
)

# Replace empty strings and nulls with 'NA'
spark_dataset = spark_dataset.withColumn(
    "details_trunc",
    when(
        col("details_trunc").isNull() | (col("details_trunc") == ""),
        'NA'
    ).otherwise(col("details_trunc"))
)

# Verify results with your sample data
spark_dataset.select("details", "details_trunc", "Class").show(truncate=False)

Key Regex Optimization Tips #

To speed up processing for large datasets:

• Combine regex patterns: Instead of running 3 separate numeric replacements, we merged them into one. Each replacement requires a pass over the data—fewer passes mean better performance.
• Split on multiple delimiters at once: In the PySpark example, we split on fr|ends|: in a single call instead of three separate splits, reducing data processing steps.
• Avoid Python UDFs: Built-in Spark functions are optimized for distributed processing and far faster than custom Python UDFs.
• Consider regexp_extract: If you can define a pattern that directly extracts your target text (instead of removing unwanted parts), regexp_extract can be more efficient. For example: regexp_extract(col("details"), r'^([^\d\$\-:()]+)', 1) (adjust based on your exact data patterns).

内容的提问来源于stack exchange，提问作者T3J45