Bash中使用hxselect命令：如何将输出存入数组对应索引？

问题描述 #

现有Bash爬虫脚本抓取房产网页数据时，所有价格被存入prices数组的同一个索引，无法按网页房源顺序依次存入prices[0]、prices[1]等对应位置，其他数组也存在同样问题。

问题根源 #

使用hxselect提取房源列表时，默认会将所有匹配的<li>元素输出为连续的文本块，没有分隔符。后续while IFS= read -r announcement循环按换行符分割输入，导致整个房源列表被当成单一行读取，循环仅执行一次——所有价格被一次性提取并添加到数组的同一个元素中。

解决方案 #

修改hxselect的调用方式，为每个匹配的房源元素添加换行分隔符，确保while循环能逐个读取单条房源数据：

修改后的关键代码片段 #

# 修改提取announcements的命令，添加-s '\n'分隔每个li元素，并过滤空行
announcements=$(echo "$html_content" | hxselect 'li.nd-list__item.in-searchLayoutListItem' -s '\n' | grep -v '^$')

完整修改后的脚本 #

#!/bin/bash

# Define variables for URL and browser
sGCitta="fucecchio"
sGTypo="sale-houses"
sGDomain="real-estate"
url="https://www.$sGDomain.it/$sGTypo/$sGCitta"

# Get HTML content of the page
html_content=$(curl -s -L "$url")

# Use html-xml-utils to extract announcements, add newline separator and filter empty lines
announcements=$(echo "$html_content" | hxselect 'li.nd-list__item.in-searchLayoutListItem' -s '\n' | grep -v '^$')

# Connect to SQLite database
db_file="immo.db"

# Initialize arrays for prices, links, descriptions, sizes, and auction
prices=()
links=()
descriptions=()
sizes=()
auctions=()

# Loop through the announcements
while IFS= read -r announcement; do
    # Extract price
    price=$(echo "$announcement" | hxselect 'div.in-listingCardPrice span' -c | grep -oP '(?<=€ )[0-9,.]+')

    # Extract link
    link=$(echo "$announcement" | hxselect 'a.in-listingCardTitle' -s '\n' | grep -o 'href="[^"]*"' | sed 's/^href="//' | sed 's/"$//')

    # Extract description
    description=$(echo "$announcement" | hxselect 'a.in-listingCardTitle' -s '\n' | grep -oP '(?<=title=")[^"]+')

    # Extract size
    size=$(echo "$announcement" | hxselect 'div.in-listingCardFeatureList__item:nth-of-type(2) span' -c | grep -oP '[0-9]+')

    # Check if the description contains the word "auction"
    if [[ "$description" =~ "auction" ]]; then
        auction=1
    else
        auction=0
    fi

    # Add data to the arrays
    prices+=("$price")
    links+=("$link")
    descriptions+=("$description")
    sizes+=("$size")
    auctions+=("$auction")

done <<< "$announcements"

# Print the sizes of the arrays
echo "Size of the prices array: ${#prices[@]}"
echo "Size of the links array: ${#links[@]}"
echo "Size of the descriptions array: ${#descriptions[@]}"
echo "Size of the sizes array: ${#sizes[@]}"
echo "Size of the auctions array: ${#auctions[@]}"

# Insert data into the SQLite database
for ((i = 0; i < ${#prices[@]}; i++)); do
    sqlite3 "$db_file" "INSERT INTO $sGDomain (price, link, description, size, auction) VALUES ('${prices[i]}', '${links[i]}', '${descriptions[i]}', '${sizes[i]}', '${auctions[i]}')"
done

说明 #

• -s '\n'参数让hxselect在每个匹配的<li>元素后添加换行符，确保每条房源独立成一行
• grep -v '^$'过滤掉可能产生的空行，避免循环读取空内容导致数组出现空元素

内容的提问来源于stack exchange，提问作者antonioleng