使用Tweepy抓取推文时出现IndexError: list index out of range的问题排查

The problem boils down to this: when you're fetching tweets going back a full year, eventually you'll exhaust all the tweets the user posted before your startDate. At that point, the Twitter API returns an empty list for tmpTweets—but your code still tries to access tmpTweets[-1] (the last element of the list) in the while loop condition. Since the list is empty, this throws an IndexError.

For recent timeframes (like the first week of February), you don't hit this issue because you stop fetching tweets before you run out of them. But when going back a year, you reach the end of the user's tweet history, leading to the empty list error.

The Simple Fix #

Update your while loop condition to first check if tmpTweets isn't empty before trying to access its last element. Here's the corrected loop:

# Fetch tweets
tweets = []
tmpTweets = api.user_timeline(user)
for tweet in tmpTweets:
    if tweet.created_at < endDate and tweet.created_at > startDate:
        tweets.append(tweet)

# Modified loop condition adds a check for non-empty tmpTweets
while tmpTweets and tmpTweets[-1].created_at > startDate:
    print('Fetched up to', tmpTweets[-1].created_at - timezoneShift, '...fetching more')
    tmpTweets = api.user_timeline(user, max_id = tmpTweets[-1].id-1)
    for tweet in tmpTweets:
        if tweet.created_at < endDate and tweet.created_at > startDate:
            tweets.append(tweet)

The tmpTweets check in the condition ensures that if the API returns an empty list, the loop exits immediately instead of trying to access a non-existent index.

Bonus Tips for Robustness #

• Handle Rate Limits: Twitter's API limits how many requests you can make in a window. Add error handling for tweepy.RateLimitError to pause and retry when you hit the limit.
• Fix Timezone Logic: Your fixed 5-hour shift doesn't account for daylight saving time. Use the pytz library to work with proper timezones (e.g., America/New_York) instead of manual offsets.
• Optimize Filtering: Instead of fetching all tweets and filtering later, see if you can use API parameters to narrow down tweets by date (though note that some date parameters are deprecated in newer Tweepy versions—check the latest docs for alternatives).

内容的提问来源于stack exchange，提问作者svadhisthana