Python内部函数调用异常：gw2_etl执行无实际效果求助

问题现状 #

调用gw2_etl(url)仅返回'Success!'，未执行任何内部逻辑（无文件生成、无MongoDB写入），且无报错信息。

核心问题与修复步骤 #

1. 主函数未触发内部流程 #

原gw2_etl仅定义了三个子函数，但未调用它们，直接返回成功。需在主函数内按爬取→处理→入库的顺序调用子函数，并传递必要参数。

2. 修复log_scrape函数的逻辑缺陷 #

• 移除错误递归：函数末尾return log_scrape()会无限递归，应改为返回生成的JSON文件路径。
• 修正文件写入逻辑：循环内return jsonFile会导致仅写入第一行就退出，需将return移至循环外。
• 补全路径分支：try块处理bossTag == 'vg'时未闭合，且未处理其他Boss标签，需完善路径分支避免pathName未定义。
• 返回正确结果：函数需返回生成的JSON文件完整路径，供后续store_data读取。

3. 修复store_data函数的逻辑缺陷 #

• 调整return位置：原代码中return stats_dict在try块内，导致后续生成JSON/CSV的代码永远无法执行，需将文件生成逻辑移至return之前。
• 修正路径处理：jsonFile.split('\\\\')在Windows环境下易出错，改用os.path模块处理路径更可靠。
• 移除错误递归：函数末尾return store_data()无意义，应删除。
• 优化异常处理：sys.exit()会直接终止程序，改为打印错误并返回None，避免程序崩溃。

4. 修复mongo_connect函数的逻辑缺陷 #

• 移除错误递归：函数末尾return mongo_connect()会无限递归，应改为返回插入结果或成功标识。
• 优化连接管理：用上下文管理器with管理MongoDB连接，确保连接自动关闭。

修复后的完整代码示例 #

import requests
from bs4 import BeautifulSoup
import re
import json
import pandas as pd
import pymongo
import sys
import os

def gw2_etl(url):
    def log_scrape(url):
        HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246'}
        response = requests.get(url=url, headers=HEADERS)
        soup = BeautifulSoup(response.content, 'html.parser')

        data = soup.find_all('script')[8]
        dataString = data.text.rstrip()
        logData = re.findall(r'{.*}', dataString)

        try:
            urlLines = url.split('/')
            if len(urlLines) < 5:
                bossName = urlLines[3]
            elif len(urlLines) == 5:
                bossName = urlLines[4]
        except Exception as e:
            return f'Error: {str(e)}'
        
        tag = bossName.split('_')
        bossTag = tag[1] if len(tag) >=2 else ''

        # 完善Boss路径分支（示例仅保留vg，需补充其他Boss逻辑）
        pathName = ''
        try:
            if bossTag == 'vg':
                pathName = 'ETL\\EXTRACT_00\\Web Scraping\\Boss_data\\Wing_1\\Valley_Guardian'
            # 补充其他Boss的pathName逻辑...
            else:
                raise ValueError(f'Unsupported boss tag: {bossTag}')
        except Exception as e:
            return f'Error: {str(e)}'
        
        # 确保目录存在
        os.makedirs(pathName, exist_ok=True)
        json_file_path = f'{pathName}\\{bossName}.json'
        
        with open(json_file_path, 'w') as f:
            for line in logData:
                f.write(line)
        
        return json_file_path

    def store_data(json_file_path):
        try:
            with open(json_file_path) as f:
                data = json.load(f)
        except Exception as e:
            print(f'Error loading JSON: {str(e)}')
            return None
        
        # 用os.path处理路径更可靠
        posSp = os.path.basename(json_file_path)
        bossTag = posSp.split('_')
        nameTag = ''
        if len(bossTag) >=2:
            nameTag = bossTag[1].split('.')[0]
        else:
            print('Error: Invalid boss filename format')
            return None

        # Players Data:
        player_group = []
        player_acc = []
        player_names = []
        player_classes = []

        for player in data.get('players', []):
            player_group.append(player.get('group', ''))
            player_acc.append(player.get('acc', ''))
            player_names.append(player.get('name', ''))
            player_classes.append(player.get('profession', ''))
        
        stats_dict = None
        try:
            if nameTag == 'vg':
                player_dps1 = []
                player_dps2 = []
                player_dps3 = []

                # Phase_1
                phase1 = data['phases'][1]['dpsStats']
                phase1_time_raw = data['phases'][1]['duration']
                phase1_time = round(phase1_time_raw/1000,1)
                for dps in phase1:
                    dps1_raw = dps[0]
                    player_dps1.append(round(dps1_raw/phase1_time,2))

                # Phase_2
                phase2 = data['phases'][6]['dpsStats']
                phase2_time_raw = data['phases'][6]['duration']
                phase2_time = round(phase2_time_raw/1000,1)
                for dps in phase2:
                    dps2_raw = dps[0]
                    player_dps2.append(round(dps2_raw/phase2_time,2))

                # Phase_3
                phase3 = data['phases'][12]['dpsStats']
                phase3_time_raw = data['phases'][12]['duration']
                phase3_time = round(phase3_time_raw/1000,1)
                for dps in phase3:
                    dps3_raw = dps[0]
                    player_dps3.append(round(dps3_raw/phase3_time,2))

                stats_dict = {
                    'players':{
                        'group': player_group,
                        'account': player_acc,
                        'names': player_names,
                        'profession': player_classes,
                        'phase_1_dps': player_dps1,
                        'phase_2_dps': player_dps2,
                        'phase_3_dps': player_dps3
                    }
                }

                df = pd.DataFrame(stats_dict['players'], columns=['group','account','names','profession','phase_1_dps','phase_2_dps','phase_3_dps'])

                # 生成JSON和CSV文件（移至return之前）
                pathName = 'ETL\\TRANSFORM_01\\Players_info'
                os.makedirs(pathName, exist_ok=True)
                
                jsonString = json.dumps(stats_dict)
                with open(f"{pathName}\\{nameTag}_player_stats.json", 'w') as f:
                    f.write(jsonString)
                
                df.to_csv(f"{pathName}\\{nameTag}_player_stats.csv", index=True)

        except Exception as e:
            print(f'Error processing data: {str(e)}')
            return None
        
        return stats_dict

    def mongo_connect(stats_dict):
        if not stats_dict:
            print('Error: No data to insert')
            return False
        
        try:
            with pymongo.MongoClient('mongodb://localhost:27017/') as client:
                db = client['GW2_SRS']
                collection = db['players_info']
                mongo_insert = collection.insert_one(stats_dict)
                print(f'Data inserted with ID: {mongo_insert.inserted_id}')
                return True
        except Exception as e:
            print(f'Connection failed: {str(e)}')
            return False

    # 主流程：依次调用子函数
    json_file = log_scrape(url)
    if isinstance(json_file, str) and not json_file.startswith('Error'):
        stats = store_data(json_file)
        if stats:
            mongo_connect(stats)
    return 'Success!'

验证要点 #

1. 调用gw2_etl(目标URL)后，检查对应目录是否生成JSON/CSV文件。
2. 登录MongoDB客户端，查看GW2_SRS.players_info集合是否有新数据插入。
3. 测试异常场景（如无效URL、不支持的Boss标签），确保错误信息正常输出且程序不崩溃。

内容的提问来源于stack exchange，提问作者Icharo-tb