如何在PySpark DataFrame中用正则表达式替换开头0为91

Got it, let's tackle this problem—replacing the leading 0 in your phone column with 91 in PySpark is straightforward once you know the right function to use, similar to how you did it in pandas but with a slight syntax shift.

Core Approach: Use regexp_replace #

PySpark's regexp_replace function (from pyspark.sql.functions) is the direct equivalent to pandas' replace(regex=True) for regex-based string substitutions. It lets you target specific patterns (like a leading 0) and replace them with your desired value.

Full Example Code #

Here's how to implement this end-to-end, including handling the example DataFrame you provided:

from pyspark.sql import SparkSession
from pyspark.sql.functions import regexp_replace, col
from pyspark.sql.types import StringType

# Initialize Spark session
spark = SparkSession.builder.appName("PhoneNumberFormatting").getOrCreate()

# Create your sample DataFrame
sample_data = [(1, "088976854667"), (2, "089706790002")]
df = spark.createDataFrame(sample_data, ["id", "phone"])

# Optional: If your phone column is numeric (not string), cast it to string first
# df = df.withColumn("phone", col("phone").cast(StringType()))

# Replace leading 0 with 91
transformed_df = df.withColumn(
    "phone",
    regexp_replace(col("phone"), "^0", "91")  # ^0 targets ONLY the leading 0
)

# View the result
transformed_df.show(truncate=False)

Output You'll Get #

+---+-------------+
|id |phone        |
+---+-------------+
|1  |9188976854667|
|2  |9189706790002|
+---+-------------+

Matching Your Pandas Workflow #

Your pandas code filters rows with underscores first, then replaces the leading 0. If you need that filter step in PySpark too, add a filter call using rlike (PySpark's regex match operator):

# Filter rows where phone contains underscores, then replace leading 0
filtered_transformed_df = df.filter(col("phone").rlike("_")) \
    .withColumn("phone", regexp_replace(col("phone"), "^0", "91"))

Key Notes #

• The regex ^0 is critical here: it only matches the 0 at the start of the string, so you won't accidentally replace 0s in the middle of the phone number.
• Always ensure your phone column is a string type before running regex operations—numeric columns won't work with regexp_replace.

内容的提问来源于stack exchange，提问作者Tilo