如何使用PySpark移除字符串内部所有空格？解决DataFrame表头及字段值空格处理无效问题并实现通用函数

Got it, let's fix this properly! You're right that your current code only handles leading/trailing spaces in values and ignores both header spaces and spaces inside string values. Here's a reusable, generic function that covers all your requirements:

import pyspark.sql.functions as F

def clean_dataframe_spaces(df):
    # Step 1: Clean column names - remove all spaces (leading, trailing, and internal)
    cleaned_columns = [F.col(name).alias(name.replace(" ", "")) for name in df.columns]
    df_cleaned = df.select(*cleaned_columns)
    
    # Step 2: Clean string values - remove leading/trailing AND internal spaces
    for col_name in df_cleaned.columns:
        # Skip non-string columns to avoid unnecessary operations
        if df_cleaned.schema[col_name].dataType.typeName() == "string":
            # First trim edge spaces, then strip all internal whitespace
            df_cleaned = df_cleaned.withColumn(
                col_name,
                F.regexp_replace(F.trim(F.col(col_name)), "\\s+", "")
            )
    
    return df_cleaned

Let's break down what this function does: #

• Header Cleaning: We iterate through every column name, replacing every space (no matter where it is) with an empty string. So a column like City Name becomes CityName, and Population turns into Population.
• Value Cleaning: For each string column, we first use trim() to eliminate leading/trailing spaces, then use regexp_replace() with \s+ (which matches one or more whitespace characters) to wipe out all internal spaces. Non-string columns stay untouched to avoid errors.

How to use it with your data: #

# Read your original dataset
df1 = spark.read.csv('/content/drive/MyDrive/BigData2021/Lecture23/datasets/cities.csv', header=True, inferSchema=True)

# Apply the cleaning function
df_cleaned = clean_dataframe_spaces(df1)

# Verify the result
df_cleaned.show()

Optional adjustments for different needs: #

• If you'd rather replace header spaces with underscores (e.g., City Name → City_Name) instead of removing them, tweak the column cleaning line to:

  cleaned_columns = [F.col(name).alias(name.replace(" ", "_")) for name in df.columns]
  

• If you only want to remove leading/trailing spaces in values (and keep internal spaces), simplify the value cleaning step to just F.trim(F.col(col_name)).

内容的提问来源于stack exchange，提问作者sasaii