PySpark写入CSV时移除Decimal列的.00小数后缀方法问询

Got it, let's solve this problem! The core issue here is that converting DecimalType directly to StringType doesn't handle the .00 suffix removal automatically—we need to add conditional logic to format values based on whether their decimal portion is zero.

Approach 1: Single Column Handling with Built-in Functions #

Instead of using a UDF (which can have performance overhead), we can leverage PySpark's built-in functions to conditionally format the Decimal column:

from pyspark.sql import functions as F
from pyspark.sql.types import DecimalType

# Assume your DataFrame has a DecimalType column named 'amount'
df_formatted = df.withColumn(
    "amount",
    F.when(
        # Check if decimal part is 0
        F.col("amount") % 1 == 0,
        # Convert to integer first to drop .00, then to string
        F.col("amount").cast("integer").cast("string")
    ).otherwise(
        # Keep the original decimal string if there's a non-zero decimal part
        F.col("amount").cast("string")
    )
)

# Write the formatted DataFrame to CSV
df_formatted.write.csv("your/output/path.csv", header=True, mode="overwrite")

How This Works: #

• F.col("amount") % 1 == 0: Checks if the decimal portion of the value is exactly 0 (e.g., 100.00 % 1 = 0, 150.50 % 1 = 0.50)
• If true: Casting to integer drops the .00 suffix, then casting back to string gives us the clean integer value
• If false: Directly casting to string preserves the original decimal format

Approach 2: Batch Process All DecimalType Columns #

If you have multiple Decimal columns in your DataFrame, you can automate the formatting for all of them:

from pyspark.sql import functions as F
from pyspark.sql.types import DecimalType

# Get all column names with DecimalType
decimal_columns = [col.name for col in df.schema if isinstance(col.dataType, DecimalType)]

# Iterate over each Decimal column and apply formatting
for col_name in decimal_columns:
    df = df.withColumn(
        col_name,
        F.when(
            # Handle null values first to avoid errors
            F.col(col_name).isNotNull() & (F.col(col_name) % 1 == 0),
            F.col(col_name).cast("integer").cast("string")
        ).when(
            F.col(col_name).isNotNull(),
            F.col(col_name).cast("string")
        ).otherwise(
            # Keep null values as-is
            F.lit(None)
        )
    )

# Write to CSV
df.write.csv("your/output/path.csv", header=True, mode="overwrite")

Example Verification #

Let's say your input DataFrame looks like this:

After applying the formatting, the output DataFrame will be:

Writing this to CSV will produce exactly the formatted values you need, with .00 suffixes removed only when applicable.

内容的提问来源于stack exchange，提问作者Usman Azhar