Python新手使用Beautiful Soup爬取课程信息时遇无返回结果问题求助

Hey there! As a fellow Python learner who’s stumbled through plenty of "why isn’t Beautiful Soup returning anything?" moments, let’s work through your problem step by step. You’re trying to scrape course info (title, time, enrolled students, instructor) based on credit hours, but right now BS4 is giving you nothing—here’s what to check:

1. First, Verify Your HTTP Request Is Actually Fetching Content #

Most empty result issues start here. When you use requests.get() to grab the page, make sure:

• You’re getting a successful response: Check response.status_code—it should be 200. If it’s 403/404, that’s a red flag (the site might be blocking your request).
• The page content isn’t empty: Print response.text to see if you’re actually getting the HTML you see in your browser. A lot of sites block unauthenticated requests, so add a basic User-Agent header to mimic a real browser:

  import requests
  from bs4 import BeautifulSoup
  
  headers = {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
  }
  url = "your-course-page-url-here"
  response = requests.get(url, headers=headers)
  print(response.status_code)  # Should return 200 if successful
  print(response.text[:500])  # Check the first 500 characters of the page
  

• Also, if the course data loads dynamically (via JavaScript), requests won’t catch it—you might need tools like Selenium or Playwright to render the page fully first.

2. Double-Check Your Tag Selection Logic #

You mentioned targeting a "course table tag" with sub-tables per course—chances are your selector is off. Here’s how to fix that:

• Use your browser’s DevTools (F12) to inspect the actual HTML structure. Right-click the course element and select "Inspect" to see the real tag names, classes, or IDs (sometimes sites use dynamic class names you won’t guess).
• Try using more specific selectors. Instead of just soup.find("table"), use something like:

  # Replace with your actual selector from DevTools
  course_container = soup.find("div", class_="course-list-wrapper")
  if course_container:
      courses = course_container.find_all("div", class_="course-item")
      print(f"Found {len(courses)} courses!")
  else:
      print("Couldn't find the course container—double-check your selector!")
  

3. Once You Have the Courses, Extract Data & Filter by Credits #

Once you’ve successfully grabbed the course elements, loop through them to pull the info you need, then filter by the user’s input credit hours:

user_credit = int(input("Enter course credit hours: "))

for course in courses:
    # Adjust these selectors to match your page's actual HTML
    title = course.find("h3", class_="course-title").get_text(strip=True)
    time = course.find("span", class_="course-time").get_text(strip=True)
    enrolled = course.find("span", class_="enrolled-count").get_text(strip=True)
    instructor = course.find("span", class_="instructor-name").get_text(strip=True)
    # Extract credits and convert to integer for comparison
    credits = int(course.find("span", class_="course-credits").get_text(strip=True))
    
    if credits == user_credit:
        print(f"\nCourse: {title}")
        print(f"Time: {time}")
        print(f"Enrolled Students: {enrolled}")
        print(f"Instructor: {instructor}")

Quick Tips to Avoid Headaches #

• Always print intermediate results (like course_container or individual course elements) to see what BS4 is actually picking up.
• If classes have spaces (e.g., class="course card"), use class_="course card" or a CSS selector like soup.select(".course.card").
• Handle missing fields gracefully—add checks like if course.find("span", class_="enrolled-count"): before calling .get_text() to avoid errors.

Hope this helps you get past the empty results and finish your assignment! 😊

内容的提问来源于stack exchange，提问作者Huizhi Wang