如何用Pandas将字典数组转换为数据集并按session_id分组

可以使用纯Python代码实现按session_id分组，并统计每个唯一条目出现的次数，保留attributes、filters等JSON格式数据不变。以下是具体实现：

代码实现 #

import json
import pprint

# 原始数据
original_data = [
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'red'}, 'duration': 25, 'attributes': [{'Occasion': 'Casual'}, {'Ideal For': 'Women'}, {'Heel Height': '0 inch'}], 'product': 'V4STYLES Loafers', 'price': 1049},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'blue'}, 'duration': 25, 'attributes': [{'Occasion': 'Casual'}, {'Ideal For': 'Women'}, {'Heel Height': '0 inch'}], 'product': 'V4STYLES Loafers', 'price': 1049},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'blue'}, 'duration': 25, 'attributes': [{'Occasion': 'Casual'}, {'Ideal For': 'Women'}, {'Heel Height': '0 inch'}], 'product': 'V4STYLES Loafers', 'price': 1049},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'blue'}, 'duration': 25, 'attributes': [{'Occasion': 'Casual'}, {'Ideal For': 'Women'}, {'Heel Height': '0 inch'}], 'product': 'V4STYLES Loafers', 'price': 1049},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'blue'}, 'duration': 25, 'attributes': [{'Occasion': 'Casual, Ethnic, Formal, Party'}, {'Ideal For': 'Women'}, {'Weight': '200 g (per single Shoe) - Weight of the product may vary depending on size.'}], 'product': 'dilli bazaaar Bellies, Corporate Casuals, Casuals', 'price': 349},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'blue'}, 'duration': 25, 'attributes': [{'Ideal For': "Women's"}, {'Occasion': 'Casual, Party'}, {'Pattern': 'Self Design'}], 'product': "F&S Casual, Party Short Sleeve Self Design Women's Top", 'price': 421},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'blue'}, 'duration': 25, 'attributes': [{'Ideal For': "Women's"}, {'Occasion': 'Casual, Party'}, {'Pattern': 'Self Design'}], 'product': "F&S Casual, Party Short Sleeve Self Design Women's Top", 'price': 421},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Base Material': 'Brass'}, {'Brand': 'Vendee Fashion'}, {'Model Number': 'VD8636'}], 'product': 'Vendee Fashion Brass Jewel Set', 'price': 829},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Base Material': 'Brass'}, {'Brand': 'Vendee Fashion'}, {'Model Number': 'VD8636'}], 'product': 'Vendee Fashion Brass Jewel Set', 'price': 829},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Base Material': 'Brass'}, {'Brand': 'Vendee Fashion'}, {'Model Number': 'VD8636'}], 'product': 'Vendee Fashion Brass Jewel Set', 'price': 829},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Base Material': 'Brass'}, {'Brand': 'Vendee Fashion'}, {'Model Number': 'VD8636'}], 'product': 'Vendee Fashion Brass Jewel Set', 'price': 829},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Occasion': 'Casual'}, {'Ideal For': 'Women'}, {'Tip Shape': 'Round'}], 'product': 'Willy Winkies Black Boots', 'price': 999},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Occasion': 'Casual'}, {'Ideal For': 'Women'}, {'Tip Shape': 'Round'}], 'product': 'Willy Winkies Black Boots', 'price': 999},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Occasion': 'Casual'}, {'Ideal For': 'Women'}, {'Tip Shape': 'Round'}], 'product': 'Willy Winkies Black Boots', 'price': 999},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Ideal For': 'Women'}, {'Occasion': 'Casual'}, {'Weight': '500.00000000 g (per single Shoe) - Weight of the product may vary depending on size.'}], 'product': 'CatBird Boots', 'price': 999},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Ideal For': 'Women'}, {'Occasion': 'Casual'}, {'Weight': '500.00000000 g (per single Shoe) - Weight of the product may vary depending on size.'}], 'product': 'CatBird Boots', 'price': 999},
    {'session_id': 'fc4867a5-cff2-4210-9ddf-9e5db35187ec', 'filters': {'color': 'green'}, 'duration': 25, 'attributes': [{'Ideal For': 'Women'}, {'Occasion': 'Casual'}, {'Weight': '500.00000000 g (per single Shoe) - Weight of the product may vary depending on size.'}], 'product': 'CatBird Boots', 'price': 999}
]

# 初始化分组结果
grouped_result = {}

for item in original_data:
    session_id = item['session_id']
    # 提取除session_id外的内容作为唯一标识的基础
    content = {key: value for key, value in item.items() if key != 'session_id'}
    # 将内容转换为排序后的JSON字符串，确保相同结构的内容生成相同的键
    content_key = json.dumps(content, sort_keys=True)
    
    # 如果session_id不在结果中，创建新条目
    if session_id not in grouped_result:
        grouped_result[session_id] = []
    
    # 检查当前内容是否已存在于该session的列表中
    matched = False
    for entry in grouped_result[session_id]:
        if entry['content_key'] == content_key:
            entry['count'] += 1
            matched = True
            break
    
    # 如果不存在，添加新的统计条目
    if not matched:
        grouped_result[session_id].append({
            'content': content,
            'count': 1,
            'content_key': content_key
        })

# 整理最终结果，移除临时的content_key
final_output = {}
for sid, entries in grouped_result.items():
    final_output[sid] = [{'content': e['content'], 'count': e['count']} for e in entries]

# 打印格式化后的结果
pprint.pprint(final_output)

代码说明 #

1. 分组逻辑：以session_id为键，将同一会话下的条目归类到一起。
2. 唯一标识生成：使用json.dumps()将每个条目的内容（排除session_id）转换为排序后的JSON字符串，确保结构相同的条目生成相同的键，避免重复统计。
3. 计数统计：遍历每个条目，若该条目内容已在当前会话中存在则计数加1，否则新增一条统计记录。
4. 结果整理：移除临时生成的content_key，保留原始内容和计数。

输出示例 #

运行上述代码后，你会得到如下结构的结果（以目标会话为例）：

{
    'fc4867a5-cff2-4210-9ddf-9e5db35187ec': [
        {
            'content': {
                'attributes': [{'Occasion': 'Casual'}, {'Ideal For': 'Women'}, {'Heel Height': '0 inch'}],
                'duration': 25,
                'filters': {'color': 'red'},
                'price': 1049,
                'product': 'V4STYLES Loafers'
            },
            'count': 1
        },
        {
            'content': {
                'attributes': [{'Occasion': 'Casual'}, {'Ideal For': 'Women'}, {'Heel Height': '0 inch'}],
                'duration': 25,
                'filters': {'color': 'blue'},
                'price': 1049,
                'product': 'V4STYLES Loafers'
            },
            'count': 3
        },
        # 其余条目省略...
    ]
}

内容的提问来源于stack exchange，提问作者DaniD