Twitter API数据返回检查及StreamListener入库异常问题咨询

Hey there, let's walk through the common pitfalls in your current code and get your tweet data saving properly to the SQL database. Here are the key issues and fixes:

1. Unhandled None Value for Location #

Not every tweet includes location data—all_data["place"] might be None if the user didn't enable location tagging or the tweet has no geolocation info. Trying to access ["name"] on a None value will throw a KeyError and crash your listener.

Fix: Add a check to handle missing location data gracefully:

Location = all_data["place"]["name"] if all_data.get("place") is not None else "Unknown"

2. Database Transactions Not Committed #

Your code runs c.execute() but never commits the transaction to the database. By default, most SQL connections don't auto-commit, so your data is only in the transaction buffer, not permanently saved.

Fix: Add a commit call after inserting (make sure you have your connection object accessible, e.g., conn):

c.execute("INSERT INTO Tweets (Username, Tweet, Location) VALUES (?,?,?)", (Username, Tweet, Location))
conn.commit()  # Don't forget this line!

3. Missing Error Handling #

If something goes wrong (invalid JSON data, database connection drop, missing keys), your listener will stop entirely. You need to catch exceptions to keep the stream running.

Fix: Wrap your logic in a try-except block and log errors:

import json
import logging

logging.basicConfig(level=logging.INFO)

class StdOutListener(StreamListener):
    def on_data(self, data):
        try:
            all_data = json.loads(data)
            # Get username safely
            Username = all_data["user"]["screen_name"]
            # Handle extended tweets to get full text (avoids truncation)
            Tweet = all_data.get("extended_tweet", {}).get("full_text", all_data["text"])
            # Handle missing location
            Location = all_data["place"]["name"] if all_data.get("place") is not None else "Unknown"
            
            c.execute("INSERT INTO Tweets (Username, Tweet, Location) VALUES (?,?,?)", (Username, Tweet, Location))
            conn.commit()
            logging.info(f"Saved tweet from @{Username}")
        except Exception as e:
            logging.error(f"Error processing tweet: {str(e)}")
            # Return True to keep the stream running after an error
            return True

4. Minor: PEP8 Naming Style #

While not a functional issue, Python conventions use snake_case for variables (e.g., username instead of Username). Adjusting this makes your code more readable for other developers.

内容的提问来源于stack exchange，提问作者A.Conroy