Web scraping using Python has become one of the most efficient ways to collect data from websites at scale. Whether you're building price trackers, monitoring competitors, gathering market intelligence, or training AI models, Python offers powerful libraries and tools that simplify the entire scraping process.

Many modern scraping projects rely on rotating proxy infrastructure to avoid bans and maintain reliable data collection. If you're new to proxies, check out:

If you're just getting started or want to improve your existing scraping setup, this guide covers everything from beginner to advanced concepts.

Web scraping is the process of automatically extracting data from websites.

Instead of manually copying information from web pages, scraping tools collect the data programmatically and organize it into structured formats such as:

Python is the most popular programming language for web scraping because it is:

Python also has excellent scraping libraries that handle:

You can also explore our detailed comparison of the best tools available today: 

Best Python Web Scraping Libraries in 2026

Most scraping projects follow this process:

Website → Python Script → Parsed Data → CSV/Database


Here’s a beginner-friendly scraping example:

importrequests
frombs4importBeautifulSoup

url="<https://example.com/products>"

headers= {
"User-Agent":"Mozilla/5.0"
}

response=requests.get(url,headers=headers)

soup=BeautifulSoup(response.text,"html.parser")

products=soup.find_all("h2")

forproductinproducts:
print(product.text)


Modern websites often load data dynamically using JavaScript.

fromplaywright.sync_apiimportsync_playwright

withsync_playwright()asp:
browser=p.chromium.launch()
page=browser.new_page()

page.goto("<https://example.com>")

print(page.title())

browser.close()


Websites use anti-bot systems to stop scrapers.

Proxy rotation distributes requests across multiple IP addresses.

Residential and mobile proxies are especially effective for scraping.

Learn more about how rotating IPs help prevent bans: 

Choosing the right proxy type matters for scraping performance and detection resistance: 

Always include realistic browser headers:

headers= {
"User-Agent":"Mozilla/5.0",
"Accept-Language":"en-US,en;q=0.9"
}


Reliable proxy infrastructure is essential for scaling scraping projects and avoiding bans:

importrequests

proxies= {
"http":"<http://username:password@proxy-ip>:port",
"https":"<http://username:password@proxy-ip>:port"
}

response=requests.get(
"<https://httpbin.org/ip>",
proxies=proxies
)

print(response.text)


CAPTCHAs are one of the biggest scraping obstacles.

The best strategy is usually prevention rather than solving.

Parsing Data with XPath and CSS Selectors

Scrapers rely on selectors to find page elements.

importcsv

withopen("products.csv","w")asfile:
writer=csv.writer(file)

writer.writerow(["Name"])

writer.writerow(["Example Product"])


importjson

data= {
"product":"Example"
}

withopen("data.json","w")asfile:
json.dump(data,file)


Speed up scraping using asynchronous requests.

Headless browsers run without opening a visible UI.

Tools like Playwright help reduce detection.

Many websites load data through hidden APIs.

Inspect network requests in DevTools to discover:

API scraping is usually faster and more stable than HTML parsing.

Proxyrack

Company

Web Scraping Using Python: Complete Beginner-to-Advanced Guide

Web Scraping Using Python

This tutorial walks an AI coding agent through building a 

 for Naver search results. The browser is used 

 to fetch a snapshot; all parser development runs against saved HTML on disk.

 Structured JSON (products, blogs, videos, web, images, dictionary, related keywords)

flowchart LR
A[api.py + browser] -->|fetch once| B[index.html]

B --> C[split_html.py]

C --> D[html_pieces/]

D --> E[parser/ modules]

B --> E

E --> F[output.json]

Layer                 -                   Role

 -                                  Safe rendered HTML fetch via headless Brave (CDP)

models/ -                                Pydantic schemas (Product, BlogPost, SearchResult, …)

split_html.py  -                      Splits DOM sections for incremental parser work

parser/  -                                 Section-specific BeautifulSoup parsers

  -                                 CLI orchestrator — parse local HTML → JSON

cd /path/to/naver
uv sync          # or: pip install -e .


Dependencies: beautifulsoup4, pydantic, browser stack (httpx, websocket-client, psutil).

 Only fetch when index.html is missing. Never re-fetch during parser iteration.

# First time: fetch search results (recommended — rich parseable data)
python fetch_html.py --query shoes

# Or fetch homepage only
python fetch_html.py --home

# Force re-fetch (rare)
python fetch_html.py --query shoes --force

 After this step, close the browser loop. All further work uses index.html.

Naver search pages are large (~13k+ lines). Split by major sections before writing selectors.

This creates html_pieces/ with files like:

*_powerlink.html 파워링크 ads (#power_link_body)

*_shopping_price.html 네이버 가격비교 (#shp_tre_root)

*_plus_store.html 네이버플러스 스토어 (#shs_lis_root)

*_related_keywords.html 연관 검색어 sidebar

html_pieces/manifest.json maps section names → filenames.

 Open one piece at a time in the editor, identify stable selectors, implement the matching parser module, run 

All structured output lives in Pydantic models:

Product(
    title="발편한 가벼운 신발 포레스트 메리노울 운동화",
    price=116900,
    discount_rate=20,
    store_name="르무통",
    source_section="plus_store",  # or powerlink | price_comparison
    badges=["플러스세일", "공식"],
)


 #shp_tre_root, #shs_lis_root product cards

 [data-meta-ssuid="review"] cafe/blog posts

 Inline video + 네이버 클립 carousel

 [data-meta-area="ldc_btm"] + Merriam-Webster web block

 Price parsing (117,900원, 2.8만), URL helpers

naver_parser.py Orchestrator + JSON export

Step 5 — Run the scraper (local HTML only)

# Default: parse full index.html, auto-split if needed
python main.py

# Parse from pre-split pieces (faster iteration)
python main.py --pieces

# Re-split then parse
python main.py --split

# Custom output path
python main.py --output results/shoes.json

Query: 'shoes'
Products: 22
Blog posts: 4
Videos: 22
Images: 7
Web results: 9
Dictionary: 2
Related keywords: 6
Saved JSON -> output.json

{
  "meta": {
    "query": "shoes",
    "search_url": "https://search.naver.com/search.naver?query=shoes",
    "tab": "전체"
  },
  "products": [ ... ],
  "blog_posts": [ ... ],
  "videos": [ ... ],
  "images": [ ... ],
  "web_results": [ ... ],
  "dictionary": [ ... ],
  "related_keywords": [ ... ]
}

python -c "
import json
d = json.load(open('output.json'))
assert d['meta']['query']
assert len(d['products']) > 0
print('OK:', {k: len(v) if isinstance(v, list) else v for k, v in d.items()})
"


#power_link_body ul.lst_type > li.lst
.lnk_tit          /* title */
.link_desc        /* description */
a.site            /* store name */


Shopping cards (price comparison & plus store)

#shp_tre_root li.q86P3e7M
#shs_lis_root li.K70iQ12A
span.Q9_4wzl0      /* price */
span.rjwcz7fY      /* discount % */
span.mlLzqQ3t      /* review score */
a.iMhVFYLc         /* store name */


[data-meta-ssuid="web"] .fds-web-doc-root
a[data-heatmap-target=".link"] span.sds-comps-text-type-headline1


[data-meta-area="ldc_btm"] mark
[data-meta-area="ldc_btm"] [data-audioid]
span.kTlmTmRATGOguYiV24_u   /* Korean meaning */


.related_srch .lst_related_srch li.item a.keyword


naver/
├── api.py                 # Browser HTML fetch (use once)
├── fetch_html.py          # CLI: save index.html
├── split_html.py          # CLI: index.html → html_pieces/
├── main.py                # CLI: parse → output.json
├── constants.py           # Paths, URLs, browser config
├── index.html             # Saved snapshot (do not re-fetch)
├── html_pieces/           # Split DOM chunks + manifest.json
├── output.json            # Parsed result
├── models/
│   ├── listing.py
│   ├── search_meta.py
│   └── browser.py
└── parser/
    ├── naver_parser.py
    ├── meta.py
    ├── powerlink.py
    ├── shopping.py
    ├── ugc.py
    ├── videos.py
    ├── web.py
    ├── images.py
    ├── dictionary.py
    ├── related.py
    └── utils.py

Building a Naver Search HTML Parser (Agent Tutorial)

How to Scrape Naver.com

Python has become the go-to language for web scraping thanks to its simplicity, huge ecosystem, and powerful automation libraries. Whether you are extracting product data, monitoring competitors, collecting market research, or building AI datasets, choosing the right Python web scraping library can make a massive difference in performance and scalability.

In this guide, we compare the best Python web scraping libraries in 2026, including BeautifulSoup, Scrapy, Selenium, Playwright, HTTPX, and Requests. We will also cover their strengths, limitations, and which use cases they are best suited for.

A Python web scraping library helps developers extract data from websites automatically. These libraries can:

Modern web scraping often requires handling anti-bot protections, CAPTCHAs, 

, and rate limits. This is why many developers combine scraping libraries with 

BeautifulSoup is one of the most popular Python web scraping libraries for parsing HTML and XML documents. It is lightweight, beginner-friendly, and perfect for small to medium scraping projects.

frombs4importBeautifulSoup
importrequests

url="<https://example.com>"

response=requests.get(url)

soup=BeautifulSoup(response.text,"html.parser")

title=soup.title.text

print(title)


BeautifulSoup is often paired with the Requests library for simple scraping workflows.

Scrapy is a powerful scraping framework designed for high-performance crawling and data extraction. It is widely used for enterprise-level scraping projects and large datasets.

importscrapy

classQuotesSpider(scrapy.Spider):
name="quotes"

start_urls= [
"<https://quotes.toscrape.com>",
    ]

defparse(self,response):
forquoteinresponse.css("div.quote"):
yield {
"text":quote.css("span.text::text").get(),
            }


Scrapy becomes even more powerful when combined with rotating residential proxies to avoid IP bans during large scraping operations.

Selenium is a browser automation framework capable of interacting with dynamic websites and JavaScript-heavy applications.

Unlike basic HTTP request libraries, Selenium controls a real browser.

fromseleniumimportwebdriver

driver=webdriver.Chrome()

driver.get("<https://example.com>")

print(driver.title)

driver.quit()


Selenium is still popular, but many developers are moving toward Playwright for modern scraping projects.

Playwright has quickly become one of the best Python web scraping tools for handling modern websites. It supports Chromium, Firefox, and WebKit while offering better performance and stealth capabilities than Selenium.

fromplaywright.sync_apiimportsync_playwright

withsync_playwright()asp:
browser=p.chromium.launch()

page=browser.new_page()

page.goto("<https://example.com>")

print(page.title())

browser.close()


Playwright is currently one of the best options for scraping websites protected by advanced anti-bot systems.

HTTPX is a modern HTTP client for Python that supports both synchronous and asynchronous requests. It is becoming increasingly popular among developers building high-performance scraping systems.

importhttpx
importasyncio

asyncdeffetch():
asyncwithhttpx.AsyncClient()asclient:
response=awaitclient.get("<https://example.com>")
print(response.status_code)

asyncio.run(fetch())


HTTPX is an excellent replacement for Requests when performance and concurrency matter.

Requests remains one of the most widely used Python libraries thanks to its simplicity and readability.

importrequests

response=requests.get("<https://example.com>")

print(response.status_code)


Although Requests is simple, it still powers countless production scraping systems worldwide.

Which Python Web Scraping Library Should You Choose?

The best Python scraping library depends entirely on your project requirements.

Web scraping is powerful—but if you’ve spent any time doing it, you’ve likely run into frustrating HTTP errors that stop your scripts in their tracks.

Among the most common are status code 403, 502, 521, 522, and 499. These errors aren’t random—they’re signals that something in your request, infrastructure, or behavior is triggering protections or failures.

HTTP status codes are responses from a server indicating the result of your request. When scraping, these codes often reflect anti-bot defenses, rate limits, or infrastructure issues rather than simple technical failures.

If you’re new to scraping fundamentals, it’s worth understanding how modern scraping works in practice—especially with real-world examples and code. You can explore this in our guide 

Understanding them is the difference between:

) means the server understands your request—but refuses to authorize it.

Advanced detection methods like TCP/IP fingerprinting are increasingly used to identify scrapers. If you want to understand how that works, this breakdown is useful - 

TCP OS Fingerprinting: How Websites Detect Automated Requests

 means one server received an invalid response from another server (usually upstream).

If you're optimizing scraping efficiency and cost, this article ‘

Why Unmetered Proxies Are Cheaper (Even With a Lower Success Rate)

’ helps explain infrastructure trade-offs:

A 521 error typically comes from services like Cloudflare and means the origin server is refusing connections.

A 522 error occurs when the server takes too long to respond.

 means the client (your scraper) closed the connection before the server responded.

The Real Solution: Proxies + Smart Scraping

Fixing these errors isn’t about patching one issue—it’s about making your scraper look like a real user.

If you're evaluating proxy reliability and accuracy, especially for IP intelligence, this article adds useful context: '

Why residential IP intelligence services are highly inaccurate

HTTP errors during scraping aren’t random—they’re feedback.

Once you understand this, the solution becomes clear:

better infrastructure, smarter requests, and proper IP management.

If you build your scraper with these principles in mind, you won’t just fix errors—you’ll prevent them entirely.

How to Fix HTTP Errors (403, 502, 521, 522, 499) When Scraping

Cybercrime is on track to cost the world $11.88 trillion in 2026, with the trajectory pointing toward $19.71 trillion by 2030. The threat industrialized over 2025: payment volumes fell as attack volumes climbed to record highs. AI lowered the barrier to entry for new attackers. Manufacturing absorbed the largest year-over-year ransomware surge. Third-party breaches doubled.

This article distills the key findings from our full Global Cybercrime Report 2026, which combines forecast modeling, third-party primary research, and Proxyrack's internal calendar 2025 KYC enforcement data into a single picture of where the threat is heading.

1. Global cybercrime cost is estimated to reach $11.88 trillion in 2026

, up from $10.5 trillion in 2025, on a path to a projected $19.71 trillion by 2030. That figure approximates the GDP of China and exceeds the combined GDP of Germany, the UK, and India.

2. The 10 highest-risk countries cluster across emerging markets

 in Latin America, Africa, and the Middle East. The 2026 update places Myanmar, Haiti, and the DR Congo at the top, reflecting Basel AML Index 2025 data. Nordic countries continue to dominate the safest end.

3. Ransomware payments fell roughly 8% to an estimated $820 million in 2025

 (Chainalysis), but the median payment per victim climbed 368%. Attack volume reached the highest level on record. Q1 2026 trackers show activity holding at the elevated 2025 plateau.

4. Manufacturing absorbed a 61% surge in ransomware activity

 in 2025. Healthcare took 290+ ransomware incidents at provider organizations. SMBs faced disproportionate exposure: 88% of ransomware-affected organizations had fewer than 1,000 employees (Sophos).

5. AI lowered the skill floor for cybercriminals

 rather than raising the ceiling. The FunkSec ransomware group, whose developers told researchers they were not coders, used generative AI to ship working malware and reportedly claimed 113 victims by March 2025.

6. Proxyrack internal KYC enforcement data

 for calendar 2025 shows the United States accounted for the largest single share of suspended accounts (26.32%), followed by a tier of 6 countries at 5.26% each (Brazil, Hong Kong, Indonesia, Cambodia, Nigeria, Poland). The dataset reflects Proxyrack's platform enforcement activity and is not a measurement of global cybercrime distribution.

Global Cybercrime Cost: $11.88 Trillion in 2026

Cybercrime is estimated to cost the world $11.88 trillion in 2026, growing at roughly 13% compound annually through 2030. The forecast comes from Proxyrack's internal model, which uses Cybersecurity Ventures historical baseline data only (2015-2023) and applies growth-rate inputs derived from Verizon DBIR breach prevalence trends, IBM Cost of a Data Breach annual averages, and Chainalysis crypto-crime payment volumes.

The 2025 actual landed within 4% of our prior-year projection, which means the model is calibrated against ground truth. Three structural shifts amplify the trajectory: AI-augmented attack tooling has expanded the attacker pool, the Initial Access Broker economy now functions as a 30-day leading indicator for ransomware activity, and third-party breaches doubled to 30% of all breaches in 2025. For deeper analysis on direct breach costs, see our 

The 2026 update places Myanmar at the top of the risk list with a Cybercrime Risk Score of 8.43, followed by Haiti at 8.12 and the Democratic Republic of the Congo at 7.94. Venezuela, Panama, Costa Rica, Chile, Saudi Arabia, Uruguay, and the UAE round out the top 10. The shift at the top reflects the Basel AML Index 2025 update, which elevated Myanmar, Haiti, and the DRC to the highest-risk tier for financial crime vulnerability. For the previous year's rankings, see our 

Finland, Iceland, and San Marino take the safest spots. Nordic countries continue to dominate the safest end of the rankings, with mandatory cybersecurity legislation, deep public-private cooperation, and high digital infrastructure investment per capita all reinforcing each other.

The Proxyrack Cybercrime Risk Score combines five international indices weighted equally: the Basel AML Index 2025 (14th Public Edition), the Cybersecurity Exposure Index (held at 2020 because no newer edition exists), the National Cyber Security Index (live data), the Digital Development Level (live data), and the ITU Global Cybersecurity Index v5 (held at 2024 because v6 has not been released). Each index is normalized to a 0-10 scale before averaging. The methodology is unchanged from the 2025 Proxyrack report to preserve year-over-year comparability.

Ransomware in 2026: Industry Restructured, Did Not Weaken

Ransomware payments fell roughly 8% to an estimated $820 million in 2025 (Chainalysis), the second consecutive annual decline. At the same time, attack volume reached the highest level on record. The two facts together describe an industry that did not weaken. It restructured around a smaller pool of paying victims, a fragmented supplier base, and an AI-enabled new class of low-effort operators.

70 active ransomware groups tracked in Q1 2026

 (Ransomware.live), up from 67 a year earlier and roughly double the count from 2022. New groups keep arriving even as established names get disrupted by law enforcement.

Top 10 groups captured 73% of payment volume

 (Chainalysis 2026 Crypto Crime Report). The long tail of 60+ smaller groups absorbs the rest, often through volume-over-value pricing.

 from 2023 to 2024 ($175,000 to $814,000) and continued rising in 2025. The smaller pool of organizations that still pay are paying significantly more per incident.

Manufacturing absorbed a 61% year-over-year surge

 in ransomware activity. The Akira, Qilin, and Play groups dominated the segment.

Public incidents anchored the year. The Marks & Spencer breach in April and May 2025, attributed to Scattered Spider, disrupted UK retail operations. The Jaguar Land Rover incident has been reported in industry coverage to have inflicted multi-billion-dollar damages. The DaVita ransomware breach is reported to have exposed approximately 2.7 million patient records.

AI-Powered Cybercrime: The Skill Floor Collapse

AI cybercrime is the dominant cybersecurity narrative of 2026, but the popular framing has it backwards. The risk is not that AI created a small number of more dangerous attackers. The risk is that AI eliminated the apprenticeship that previously kept low-skill actors out of the cybercrime market, creating a much larger pool of mediocre but functional attackers operating at higher volume.

FunkSec is the case study. The group launched in late 2024 with developers who told researchers they were not coders. Using generative AI tools to produce Rust-based malware, they shipped a functional ransomware payload in weeks. By March 2025, FunkSec had reportedly claimed 113 victims across 10 countries, ranking among the top groups by victim count for that quarter despite the small team size and limited capital.

IBM's 2025 study of 600 breached organizations found that 

16% of breaches involved AI-assisted attacks

, most commonly in the phishing and impersonation categories. The percentage is rising quarter over quarter according to IBM's tracking through Q1 2026.

The defensive side of the AI shift is more encouraging. IBM found that organizations using AI tools in security operations cut their breach lifecycle by 80 days and saved nearly 

 compared to organizations without AI in the SOC.

Supply Chain Attacks: The Largest Single-Year Shift

Third-party breaches doubled in 2025 to 30% of all breaches, the largest single-year shift in attack vector composition Verizon has recorded in the DBIR's history. Vendor risk management still receives a small fraction of enterprise security spend despite this exposure. For the broader picture of which industries are most exposed, see our 

The increase reflects two shifts. Organizations have continued to outsource more operations to specialized vendors. Attackers have learned that compromising one well-positioned vendor can cascade access to dozens or hundreds of downstream organizations.

Proxyrack Internal Data: KYC Enforcement Record for Calendar 2025

Proxyrack operated a Know Your Customer (KYC) verification process throughout calendar 2025 as one layer of its anti-abuse stack. The dataset below captures all account suspensions executed under that process during the full 2025 calendar year. Total suspensions: 76 accounts.

The United States accounted for the largest single share at 26.32% of suspensions. A second tier of six countries (Brazil, Hong Kong, Indonesia, Cambodia, Nigeria, Poland) each accounted for 5.26%. A third tier of four countries (India, Netherlands, Pakistan, Vietnam) each accounted for 2.63%. The remaining 24 countries each accounted for 1.32%.

Important caveats. The figures represent share of total suspensions by country, not in-country suspension rates. The dataset is small (76 total accounts) and reflects activity on the Proxyrack platform during one calendar year. It is not a measurement of global cybercrime distribution.

Proxyrack's KYC verification process was retired in late 2025 as the company consolidated its anti-abuse stack around its 

, which provides ongoing real-time enforcement at the device, network, and session layers.

What This Means for Security Teams in 2026

The defining ransomware shift of 2025 is not in attack count or payment dollars. It is in the supply chain. Initial Access Brokers, AI-enabled small operators like FunkSec, and the splintering of LockBit-class groups into a long tail of low-skill teams have changed the threat model. The old defensive playbook assumed a small number of well-resourced attackers. The 2026 reality is a much larger pool of lower-skill operators working at significantly higher volume, supported by increasingly accessible credential and access markets.

- Katy Salgado, Operations Manager, Proxyrack

Treat the Initial Access Broker market as a leading indicator, not a downstream artifact. When IAB listings spike in your sector, ransomware activity in the same sector spikes 30 days later. Build that into your threat intelligence cadence.

Build vendor security review into the same priority tier as identity and access. Third-party breaches at 30% of all breaches mean the supply chain is now a primary attack surface, not a secondary concern.

Design defensive architecture for the high-volume low-skill AI-enabled attacker, not just for the sophisticated operator. Detection logic optimized for nation-state TTPs will miss the FunkSec-class segment because there is nothing distinctive to detect.

Download the full Global Cybercrime Report 2026

The 2026 cybercrime cost forecast is produced by Proxyrack's internal model using Cybersecurity Ventures historical baseline data only and growth-rate inputs derived from current Verizon DBIR, IBM, and Chainalysis data. The country rankings methodology is unchanged from the 2025 Proxyrack report to preserve year-over-year comparability. The KYC suspension dataset reflects Proxyrack platform enforcement activity for calendar 2025 only.

Primary external sources cited in this article and the full report include the Chainalysis 2026 Crypto Crime Report, Verizon DBIR 2025, Sophos State of Ransomware 2025, IBM Cost of a Data Breach Report 2025, Halcyon 2026 Power Rankings, IPinfo + AbuseIPDB joint research at RSA Conference 2026, GreyNoise 2026 Invisible Army Report, Google Cloud Threat Intelligence IPIDEA Disruption (January 2026), the Basel AML Index 2025, the e-Governance Academy National Cyber Security Index, the ITU Global Cybersecurity Index v5, and others. The complete source list is in the full report.

This article summarizes findings from the Global Cybercrime Report 2026 (8,700 words, 5 charts, 24 primary sources). The full report covers 11 sections including phishing economics, cryptojacking trends, government action, NIS2 enforcement, and our complete methodology.

 (PDF, 8,700 words, 5 charts, 24 sources). 

Global Cybercrime Report 2026: Key Findings

A beginner-friendly guide — no programming knowledge needed. Just follow the steps.

 as backup when the website blocks automated requests.

sudo apt update
sudo apt install python3.12 python3.12-venv


 Open a terminal (Command Prompt on Windows, Terminal on Mac/Linux) and type:

 If you install Brave via the terminal, note where it's installed. The default is usually 

. If it's somewhere else, you'll update one line of code later.