导入Twitter数据时遭遇JSONDecodeError错误，寻求解决方案

Problem Description #

I'm trying to fetch Twitter data using Tweepy, but when I try to load the saved JSON file, I get a JSONDecodeError: Extra data error. Here's my code:

import os
import tweepy
import pandas as pd
import json

access_token = "wwwwwww"
access_token_secret = "xxxxxxxx"
consumer_key = "yyyyyyyyy"
consumer_secret = "zzzzzzzz"

# Pass OAuth details to tweepy's OAuth handler
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

tweet_list = []

class MyStreamListener(tweepy.StreamListener):
    def __init__(self, api=None):
        super(MyStreamListener, self).__init__()
        self.num_tweets = 0
        self.file = open("tweets.txt", "w")

    def on_status(self, status):
        tweet = status._json
        self.file.write(json.dumps(tweet) + '\n')
        tweet_list.append(status)
        self.num_tweets += 1
        if self.num_tweets < 100:
            return True
        else:
            return False
        self.file.close()

# Initialize Stream listener
l = MyStreamListener()

# Create your Stream object with authentication
stream = tweepy.Stream(auth, l)

# Filter Twitter Streams to capture data by the keywords:
stream.filter(track= ('clinton', 'trump', 'sanders', 'cruz'))

# String of path to file: tweets_data_path
tweets_data_path = "tweets.txt"

# Initialize empty list to store tweets: tweets_data
tweets_data = []

# Open connection to file
tweets_file = open(tweets_data_path, "r")

# Read in tweets and store in list: tweets_data
for line in tweets_file:
    tweet = json.loads(line)
    tweets_data.append(tweet)

# Close connection to file
tweets_file.close()

# Print the keys of the first tweet dict
print(tweets_data[0].keys())

And here's the error I'm getting:

Traceback (most recent call last):
  File "C:/Users/servo/AppData/Local/Programs/Python/Python38/tweeter.py", line 53, in <module>
    tweet = json.loads(line)
  File "C:\Users\servo\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "C:\Users\servo\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 3258 (char 3257)

I've checked the official docs and StackOverflow but haven't found a working fix, so I'm asking for help here.

Solution #

Let's break down the issue and fix it step by step:

1. Root Cause: File Isn't Being Closed Properly

Look at your on_status method in MyStreamListener:

def on_status(self, status):
    # ... existing code ...
    if self.num_tweets < 100:
        return True
    else:
        return False
    self.file.close()  # This line NEVER runs!

Once you hit return False, the function exits immediately, so self.file.close() is never executed. This leaves the file handle open, which can cause incomplete writes or corrupted data in your tweets.txt file — that's exactly why you're getting the JSON decode error.

2. Fix the File Handling

We can fix this in two reliable ways:

• Option 1: Use Stream Listener Lifecycle Methods to Close the File
  Override the on_stop and on_error methods to ensure the file gets closed when the stream stops (either after hitting 100 tweets or on an error):

  class MyStreamListener(tweepy.StreamListener):
      def __init__(self, api=None):
          super(MyStreamListener, self).__init__()
          self.num_tweets = 0
          self.file = open("tweets.txt", "w")
  
      def on_status(self, status):
          tweet = status._json
          self.file.write(json.dumps(tweet) + '\n')
          tweet_list.append(status)
          self.num_tweets += 1
          # Simplify the condition to make logic clearer
          return self.num_tweets < 100
  
      def on_stop(self):
          self.file.close()
          print("Stream stopped successfully, file closed.")
  
      # Handle errors to avoid leaving the file open unexpectedly
      def on_error(self, status_code):
          self.file.close()
          print(f"Error {status_code} encountered, file closed.")
          return False  # Stop the stream on error
  

• Option 2: Explicitly Close the File Before Stopping the Stream
  Adjust the on_status method to close the file right before returning False:

  def on_status(self, status):
      tweet = status._json
      self.file.write(json.dumps(tweet) + '\n')
      tweet_list.append(status)
      self.num_tweets += 1
      if self.num_tweets >= 100:
          self.file.close()
          return False
      return True
  

3. Add Error Handling When Reading JSON

Even with proper file closing, you might encounter empty lines or malformed JSON entries. Add a try-except block to skip problematic lines instead of crashing:

# Read in tweets and store in list: tweets_data
for line in tweets_file:
    line = line.strip()  # Remove extra whitespace/newlines
    if not line:
        continue  # Skip empty lines
    try:
        tweet = json.loads(line)
        tweets_data.append(tweet)
    except json.JSONDecodeError as e:
        print(f"Skipping malformed JSON line: {e}")
        continue

4. Verify the File Content

After fixing the code, open tweets.txt and double-check that each line is a single, valid JSON object. If you still see issues, print out the problematic line to debug further.

内容的提问来源于stack exchange，提问作者Washington Muniz