如何用Selenium提取ADP招聘站点的职位详情链接?
问题:ADP旗下招聘站点无法提取职位详情链接
- 常规爬取招聘信息时,可通过以下方式提取职位详情链接:
link = job.find_element(By.TAG_NAME, 'a') link = link.get_attribute('href') - 但ADP旗下的招聘站点(例如Metrea的招聘页面)不存在带
href属性的<a>标签,导致无法通过常规方式获取链接。目前已能爬取职位名称、地点、发布日期等信息,但始终无法提取职位详情链接。考虑过点击每个职位后获取当前页面链接,但不确定是否有更优方案,特此求助。
现有爬取代码
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.chrome.service import Service from selenium.webdriver.chrome.options import Options from webdriver_manager.chrome import ChromeDriverManager from urllib.parse import urljoin from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC def scrape_metrea(): # Setup chrome options chrome_options = Options() chrome_options.add_argument("--headless") # Ensure GUI is off chrome_options.add_argument("--no-sandbox") chrome_options.add_argument("--disable-dev-shm-usage") # Set up the Chrome webdriver service webdriver_service = Service(ChromeDriverManager().install()) # Start the browser driver = webdriver.Chrome(service=webdriver_service, options=chrome_options) # Open the webpage driver.get('https://workforcenow.adp.com/mascsr/default/mdf/recruitment/recruitment.html?cid=39bc5efc-a576-4a4d-83eb-32e95d4e96c8&ccId=19000101_000001&lang=en_US&selectedMenuKey=CurrentOpenings') # Set implicit wait driver.implicitly_wait(2) # Wait up to 10 seconds for elements to be found try: # Get the dynamic content job_listings = driver.find_elements(By.CLASS_NAME, 'current-openings-item') jobs_list = [] for job in job_listings: location = '' job_info = {} title = job.find_element(By.CSS_SELECTOR, 'span.current-opening-title').text location = job.find_element(By.CSS_SELECTOR, 'label.current-opening-location-item').text date = job.find_element(By.CSS_SELECTOR, 'span.current-opening-post-date').text job_info['Company'] = 'Metrea' job_info['Job Title'] = title job_info['Job Link'] = None job_info['Location'] = location job_info['Date Posted'] = date job_info['ID'] = None job_info['Category'] = None job_info['Job Type'] = None jobs_list.append(job_info) #NO LINK CANT FIND OUT HOW TO PULL except: jobs_list = [] message = {} message['Message'] = 'Error Scraping Company Metrea' jobs_list.append(message) print('Error scraping metrea') print('scraping Metrea') # Close the browser driver.quit() return jobs_list
内容的提问来源于stack exchange,提问作者Annalise Wang
相关产品推荐
相关产品推荐

