如何使用Python 3解码Windows-1252编码的HTML字符串

Hey there, let's break down how to tackle that weird string you're seeing in the HTML. First, we'll get that element extracted, then figure out how to decode its content.

1. Extracting the Obfuscated String #

Since it's not a standard HTML tag or properly formatted comment (missing the -- for valid comments), BeautifulSoup might parse it oddly. Here are two reliable ways to pull it out:

Option 1: Regex Directly on Raw HTML Content #

Sometimes skipping the parser and using regex is simpler, especially if the parser mangles the unusual element:

import requests
import re

url = 'xxxxxx'
webpage = requests.get(url, verify=False)
# Adjust the regex if the string has variable parts
pattern = re.compile(r'<!sh6dnzerw9bef91nf0n2p6drlmxdadeulbyz24ho3kt\s+\w+\s+\w+\s+\w+>')
match = pattern.search(webpage.text)
if match:
    obfuscated_str = match.group()
    print("Extracted string:", obfuscated_str)

Option 2: Dig Through BeautifulSoup Nodes #

If you want to stick with BeautifulSoup, iterate through all nodes to find the target content:

from bs4 import BeautifulSoup

soup = BeautifulSoup(webpage.content, 'html.parser')
for node in soup.descendants:
    if hasattr(node, 'string') and node.string is not None:
        if 'sh6dnzerw9bef91nf0n2p6drlmxdadeulbyz24ho3kt' in node.string:
            print("Found obfuscated content:", node.string)

2. Decoding the Random Strings #

Those jumbled class-like sequences are almost certainly encoded content. Let's try common decoding tricks:

Try ROT13 (Simple Caesar Cipher) #

This is a super common lightweight obfuscation. Test it on one of the strings:

import codecs

test_str = 'sh6dnzerw9bef91nf0n2p6drlmxdadeulbyz24ho3kt'
rot13_result = codecs.encode(test_str, 'rot_13')
print("ROT13 Decoded:", rot13_result)

Reverse the String #

Sometimes flipping the string reveals readable text:

reversed_str = test_str[::-1]
print("Reversed string:", reversed_str)

Check for Modified Base64 #

Sites often tweak Base64 (swap +// for -/_ to avoid URL issues). Try this:

import base64

# Adjust replacement characters if needed based on site's pattern
modified_base64 = test_str.replace('-', '+').replace('_', '/')
try:
    decoded = base64.b64decode(modified_base64).decode('utf-8')
    print("Modified Base64 Decoded:", decoded)
except:
    print("Not modified Base64 content")

Look for JavaScript Decoding #

Chances are, the site uses client-side JS to decode these strings. Here's how to handle that:

1. Use your browser's dev tools to search for parts of the obfuscated string in the page's JavaScript files to find the decoding function.
2. Use Selenium to run the decoding function and get the result:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get(url)
# Replace with the actual decoding function you found in the site's JS
decoded_content = driver.execute_script("return decodeObfuscated('" + test_str + "');")
print("Decoded via JavaScript:", decoded_content)
driver.quit()

Quick Anti-Scraping Note #

Sites use this kind of obfuscation to block scrapers, so make sure you're complying with their robots.txt and terms of service. Also, verify=False in requests disables SSL verification—only use that if you trust the site and understand the security risks.

内容的提问来源于stack exchange，提问作者BabaLiam