PySpark（Databricks）读取最大日期文件夹文件问题求助

Hey there! Let's fix this problem together. It sounds like you're trying to dig into nested month/date folders under xyz and pull files from the most recent one (max month, then max date in that month). Here's a reliable Python approach that should work, plus tips to avoid common pitfalls:

Step-by-Step Explanation #

First, we need to:

1. Scan the xyz directory to find all month folders, then pick the latest one.
2. Within that month folder, scan all date folders and pick the latest date.
3. Finally, access and read files from that target folder.

Working Code #

import os
from datetime import datetime

def get_latest_date_folder(base_dir):
    # Filter out only directories (ignore files directly in base_dir)
    month_folders = [folder for folder in os.listdir(base_dir) 
                     if os.path.isdir(os.path.join(base_dir, folder))]
    
    if not month_folders:
        raise ValueError("No month folders found in your xyz directory!")
    
    # Parse month names/numbers to sort correctly (adjust format to match your folders)
    def parse_month(month_str):
        # Example formats: 
        # - Numeric months: "%m" (for '05' = May)
        # - Full month names: "%B" (for 'May')
        # - Year-month: "%Y-%m" (for '2024-05')
        try:
            return datetime.strptime(month_str, "%m")  # Update this line for your folder format
        except ValueError:
            return datetime.strptime(month_str, "%B")
    
    # Sort months from newest to oldest and pick the top one
    sorted_months = sorted(month_folders, key=parse_month, reverse=True)
    latest_month_path = os.path.join(base_dir, sorted_months[0])
    
    # Now repeat for date folders inside the latest month
    date_folders = [folder for folder in os.listdir(latest_month_path) 
                    if os.path.isdir(os.path.join(latest_month_path, folder))]
    
    if not date_folders:
        raise ValueError("No date folders found in the latest month directory!")
    
    def parse_date(date_str):
        # Example formats:
        # - Day only: "%d" (for '07' = 7th)
        # - Month-day: "%m-%d" (for '05-07')
        # - Full date: "%Y-%m-%d" (for '2024-05-07')
        # Combine with latest month if needed (adjust below)
        return datetime.strptime(f"{sorted_months[0]}-{date_str}", "%m-%d")  # Update format
    
    sorted_dates = sorted(date_folders, key=parse_date, reverse=True)
    latest_folder_path = os.path.join(latest_month_path, sorted_dates[0])
    
    return latest_folder_path

# --------------------------
# Use the function here
# --------------------------
xyz_directory = "/path/to/your/xyz/folder"  # Replace with your actual path
try:
    target_folder = get_latest_date_folder(xyz_directory)
    print(f"Successfully located the latest folder: {target_folder}")
    
    # Read files in the target folder
    for filename in os.listdir(target_folder):
        file_full_path = os.path.join(target_folder, filename)
        if os.path.isfile(file_full_path):
            print(f"Processing file: {file_full_path}")
            # Add your file-reading logic here!
            # Example:
            # with open(file_full_path, "r") as file:
            #     content = file.read()
            #     # Do something with content
except Exception as e:
    print(f"Oops, hit an error: {str(e)}")

Key Tips to Avoid Issues #

• Match Your Folder Format: The biggest mistake here is using the wrong date format string. If your month folders are named like 2024-05 (year-month), update the parse_month function to use "%Y-%m" instead of "%m". Same for dates—if they're full dates like 2024-05-07, adjust parse_date accordingly.
• Filter Out Files: Always make sure you're only looking at directories (not loose files) with os.path.isdir(). This prevents errors from random files sitting in the xyz or month folders.
• Handle Edge Cases: The code includes checks for empty directories (no months/dates found) so you get clear error messages instead of cryptic crashes.

内容的提问来源于stack exchange，提问作者aamirmalik124