進階 Python 網頁抓取工具

使用 Python 建立進階網頁抓取工具。學習擷取頁面資料、處理使用者代理區塊、管理速率限制和建置輸出。

在編輯器中嘗試

概述

網頁抓取涉及以程式設計方式獲取網頁並解析其 HTML 結構以提取特定資料。在 Python 中，傳統上這是使用「requests」和「BeautifulSoup」來完成的。

高級刮刀需要強大的配置。簡單地請求頁面可能會導致封鎖或 IP 禁止，從而需要自訂標頭（例如自訂使用者代理）並引入隨機延遲間隔。

這個互動式範例透過驗證模擬來抓取模擬資料。它示範如何定位嵌套容器、處理網路逾時以及將輸出資料建構成乾淨的清單。

程式碼和執行輸出

具有自訂標頭、逾時管理和 HTML 解析器結構的 Web 抓取例程。

adv_scraper.py

在編輯器中嘗試

import requests
from bs4 import BeautifulSoup
import time

def scrape_quotes():
    # Mock URL designed for scraping practice
    url = "https://quotes.toscrape.com/"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }
    
    print(f"Initiating request to: {url}...")
    try:
        # Request with timeout protection
        response = requests.get(url, headers=headers, timeout=5)
        response.raise_for_status()
        
        soup = BeautifulSoup(response.text, "html.parser")
        quotes = soup.find_all("div", class_="quote", limit=3)
        
        results = []
        for q in quotes:
            text = q.find("span", class_="text").text
            author = q.find("small", class_="author").text
            tags = [t.text for t in q.find_all("a", class_="tag")]
            results.append({
                "quote": text,
                "author": author,
                "tags": tags
            })
            
        print("Scrape completed successfully!\n")
        for idx, item in enumerate(results):
            print(f"Quote {idx+1}: {item['quote']}")
            print(f"  Author: {item['author']}")
            print(f"  Tags:   {', '.join(item['tags'])}\n")
            
    except requests.exceptions.RequestException as e:
        print(f"Network error occurred: {e}")

scrape_quotes()

端子輸出

Initiating request to: https://quotes.toscrape.com/...
Scrape completed successfully!

Quote 1: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
  Author: Albert Einstein
  Tags:   change, deep-thoughts, thinking

Quote 2: “It is our choices, Harry, that show what we truly are, far more than our abilities.”
  Author: J.K. Rowling
  Tags:   abilities, choices

Quote 3: “There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”
  Author: Albert Einstein
  Tags:   inspirational, life, live, miracle

逐步實施

跨購物目錄的價格監控和跟踪
透過從論壇收集評論資料進行情感分析
學術研究資料集創建

常見問題解答

什麼是 robots.txt？

放置在網域根部的文字文件，指示允許或禁止搜尋機器人和爬網程式爬網的路徑。在編寫爬蟲之前，您應該始終閱讀它。

如何抓取透過 JavaScript 動態渲染的頁面？

標準請求僅取得靜態 HTML。要解析大量 Javascript 頁面，您必須使用瀏覽器自動化工具，例如 Playwright 或 Selenium。