贷款模拟器Web爬取需求：获取TAEG及核心还款数据

Hey there! Let's tackle this loan simulator scraping challenge together. Since you couldn't find a direct JSON endpoint to pull data from, here are a few practical, actionable approaches you can try:

1. Simulate User Interaction with Browser Automation #

This is the most straightforward way to handle dynamically rendered pages like this one. Tools like Selenium or Playwright let you mimic real user actions (adjusting sliders, selecting terms) and extract data once the page updates.

Step-by-Step Workflow: #

• Launch a browser instance and load the target page
• Target the loan amount input/slider and update it to your desired values
• Switch between different loan term (meses) options
• Wait for the page to refresh the calculated values (use explicit waits to avoid race conditions)
• Extract all four required data points from their respective page elements
• Loop through all your desired amount/term combinations to collect full dataset

Example Snippet (Python + Selenium): #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait, Select
from selenium.webdriver.support import expected_conditions as EC

# Initialize browser (Chrome in this case)
driver = webdriver.Chrome()
driver.get("https://m-pt-funnel.younited-credit.com/initialproposal")

# Example: Set loan amount (adjust selector to match the actual input ID/class)
amount_input = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "loanAmount"))
)
amount_input.clear()
amount_input.send_keys("5000")

# Example: Select 12-month term (adjust selector for your page's term dropdown)
term_dropdown = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "loanTerm"))
)
select_term = Select(term_dropdown)
select_term.select_by_value("12")

# Extract core data (update selectors based on actual page HTML)
taeg_value = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".taeg-percentage"))
).text

monthly_payment = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".orange-monthly-payment"))
).text

term_months = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".term-display"))
).text

total_due = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, "//*[contains(text(), 'MONTANTE TOTAL DEVIDO')]/following-sibling::span"))
).text

# Print results (or store in a dataset)
print(f"TAEG: {taeg_value} | Monthly Payment: {monthly_payment} | Term: {term_months} | Total Due: {total_due}")

# Clean up
driver.quit()

Note: Use your browser's DevTools (F12) to inspect elements and get accurate selectors for inputs and data fields.

2. Intercept Hidden API Requests #

Sometimes data is loaded via AJAX/Fetch calls that aren't obvious at first glance. Here's how to find them:

• Open the page and launch Chrome DevTools (F12) → go to the Network tab
• Adjust the loan amount or term, then watch for new XHR/Fetch requests in the network log
• If you spot a request that returns loan calculation data, copy its URL, headers, and parameters
• Use a library like requests to replicate these calls directly, avoiding full browser automation

If the request uses signed/encrypted parameters, you'll need to analyze the front-end JS code (via the Sources tab in DevTools) to figure out how those parameters are generated.

3. Reverse-Engineer Frontend Calculation Logic #

In some cases, the loan values are calculated entirely client-side with JavaScript. To leverage this:

• Use DevTools' Sources tab to search for keywords like TAEG or MONTANTE TOTAL DEVIDO
• Locate the JS functions that handle these calculations
• Translate that logic into your preferred programming language (Python, etc.)
• Input your desired amount/term combinations directly into your translated code to generate results

This method is the fastest once you crack the logic, but requires basic JavaScript reverse-engineering skills.

Important Notes: #

• Always check the site's robots.txt and terms of service to ensure you're scraping responsibly
• Avoid making too many rapid requests—add delays to prevent getting your IP blocked
• Keep an eye on page updates: if the site changes its HTML structure or JS logic, you'll need to adjust your scraper

内容的提问来源于stack exchange，提问作者brunosm