高级 Python 网页抓取工具

使用 Python 构建高级网络抓取工具。学习提取页面数据、处理用户代理块、管理速率限制和构建输出。

在编辑器中尝试

概述

网页抓取涉及以编程方式获取网页并解析其 HTML 结构以提取特定数据。在 Python 中，传统上这是使用“requests”和“BeautifulSoup”来完成的。

高级刮刀需要强大的配置。简单地请求页面可能会导致阻止或 IP 禁止，从而需要自定义标头（例如自定义用户代理）并引入随机延迟间隔。

这个交互式示例通过验证模拟抓取模拟数据。它演示了如何定位嵌套容器、处理网络超时以及将输出数据构造为干净的列表。

代码和执行输出

具有自定义标头、超时管理和 HTML 解析器结构的 Web 抓取例程。

adv_scraper.py

在编辑器中尝试

import requests
from bs4 import BeautifulSoup
import time

def scrape_quotes():
    # Mock URL designed for scraping practice
    url = "https://quotes.toscrape.com/"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }
    
    print(f"Initiating request to: {url}...")
    try:
        # Request with timeout protection
        response = requests.get(url, headers=headers, timeout=5)
        response.raise_for_status()
        
        soup = BeautifulSoup(response.text, "html.parser")
        quotes = soup.find_all("div", class_="quote", limit=3)
        
        results = []
        for q in quotes:
            text = q.find("span", class_="text").text
            author = q.find("small", class_="author").text
            tags = [t.text for t in q.find_all("a", class_="tag")]
            results.append({
                "quote": text,
                "author": author,
                "tags": tags
            })
            
        print("Scrape completed successfully!\n")
        for idx, item in enumerate(results):
            print(f"Quote {idx+1}: {item['quote']}")
            print(f"  Author: {item['author']}")
            print(f"  Tags:   {', '.join(item['tags'])}\n")
            
    except requests.exceptions.RequestException as e:
        print(f"Network error occurred: {e}")

scrape_quotes()

端子输出

Initiating request to: https://quotes.toscrape.com/...
Scrape completed successfully!

Quote 1: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
  Author: Albert Einstein
  Tags:   change, deep-thoughts, thinking

Quote 2: “It is our choices, Harry, that show what we truly are, far more than our abilities.”
  Author: J.K. Rowling
  Tags:   abilities, choices

Quote 3: “There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”
  Author: Albert Einstein
  Tags:   inspirational, life, live, miracle

逐步实施

跨购物目录的价格监控和跟踪
通过从论坛收集评论数据进行情感分析
学术研究数据集创建

常见问题解答

什么是 robots.txt？

放置在域根部的文本文件，指示允许或禁止搜索机器人和爬网程序爬网的路径。在编写爬虫之前，您应该始终阅读它。

如何抓取通过 JavaScript 动态渲染的页面？

标准请求仅获取静态 HTML。要解析大量 Javascript 页面，您必须使用浏览器自动化工具，例如 Playwright 或 Selenium。