Scrape Ebay for items

A step-by-step guide to scraping eBay search results into structured item data.

What We're Building

We'll create a small program that includes:

1. An eBay search page parser - turns search HTML into structured listing data

2. An eBay search page fetcher - visits eBay home, then fetches a search page for a keyword

3. An eBay search result saver - stores cookies/session and parsed items for the next run

4. A CLI - run the parser on local HTML, or fetch + parse a live search

Sample HTML: sample.html  ·  Session file: data/ebay_session.json

Local-first rule: Fetch HTML once and save it. Build and test the parser against that saved file - do not re-fetch while iterating on selectors.

Architecture Overview

scraper.py + httpxhome then search → sample.html / live HTML

↓
ebay_parse.py
↓

search_parser.py
↓
models/search.py

session_store.py
↓
data/ebay_session.json

↓
EbaySearchData

Step 1: Install Python 3.12

Python Installation

Windows:

1. Go to python.org/downloads

2. Click the yellow button that says "Download Python 3.12.x"

3. Run the downloaded .exe file

4. IMPORTANT: Check the box that says "Add Python to PATH" at the bottom

5. Click "Install Now" and wait for it to finish

macOS:

1. Go to python.org/downloads

2. Click the yellow button for "Download Python 3.12.x"

3. Open the downloaded .pkg file and follow the installer

Ubuntu / Debian Linux:

sudo apt update
sudo apt install python3.12 python3.12-venv

Fedora Linux:

sudo dnf install python3.12

Verify it worked:

Open a terminal and type:

python3.12 --version

You should see:

Python 3.12.x

Step 2: Install uv (Package Manager)

Install uv

uv is a fast tool that installs Python libraries. We use it instead of pip.

Windows (PowerShell):

powershell -c "irm <https://astral.sh/uv/install.ps1> | iex"

macOS / Linux:

curl -LsSf <https://astral.sh/uv/install.sh> | sh
source $HOME/.local/bin/env

After installing, close and reopen your terminal (or run the source line above to activate immediately).

Verify:

uv --version

Should show something like

uv 0.x.x

Linux only - pyright dependency:

pyright ships with a bundled Node binary that requires

libatomic

, which is absent on minimal Ubuntu/Debian installs. If

uv run pyright

fails with

libatomic.so.1: cannot open shared object file

, install it:

sudo apt install libatomic1

Step 3: Install Dependencies

Let uv install everything

Open your terminal, navigate to the project folder, then run:

cd ebay
uv sync

This reads pyproject.toml and installs:

• httpx - makes web requests

• beautifulsoup4 + lxml - parses HTML

• pydantic - handles data cleanly

Dev tools (installed automatically): pytest, ruff, pyright (strict mode).

Quality checks:

uv run pytest          # all tests must pass
uv run ruff check .    # lint
uv run pyright         # strict type check

Step 4: Parse Offline

Run the parser against local HTML

The parser works directly on HTML. You only need a search-results HTML file (like sample.html). No live request is required for this step.

uv run python main.py --html sample.html

# Show more rows
uv run python main.py --html sample.html --limit 10

# Optional label stored on the output object
uv run python main.py --html sample.html --keyword "shoes men" --limit 10

Args:

• -html - path to a local eBay search HTML file (defaults to sample.html)

• -limit - how many listings to print in the terminal (default: 5)

• -keyword - optional label only for the output data object; it is not used to fetch or filter the HTML

What happens:

1. Read the HTML file from disk

2. parse_ebay_search_page() finds ul.srp-results > li.s-card cards

3. Print item count and the first few listings

Expected Output

Parsed from local file: sample.html
============================================================
Keyword: None
Source: .../sample.html
Items found: 60
------------------------------------------------------------
- 168520328526 | $999.99 | Off White Virgil Abloh Air Jordan 1 High VAA ...
  <https://www.ebay.com/itm/168520328526>
  Pre-Owned | Buy It Now | +$396.57 delivery | United States
------------------------------------------------------------

Step 5: Inspect the Fixture

Probe the saved HTML before changing selectors

uv run python -c "
from bs4 import BeautifulSoup
html = open('sample.html').read()
soup = BeautifulSoup(html, 'lxml')
cards = soup.select('ul.srp-results > li.s-card')
print('cards:', len(cards))
title = cards[0].select_one('.s-card__title .su-styled-text.primary')
print('title:', title.get_text(strip=True)[:80])
print('price:', cards[0].select_one('.s-card__price').get_text(strip=True))
"

Key DOM areas in sample.html

Workflow: Open sample.html, search for stable classes like s-card__price, update parsers/selectors.py, run tests.

Step 6: Live Search

Fetch and parse a search page

uv run python main.py --search "shoes men"

uv run python main.py --search "shoes men" --fresh

Args:

• -search - keyword to search on eBay; visits home, then fetches that search page and parses the listings

• -fresh - ignore saved cookies/session and start a new session before searching

What happens:

1. Load cookies from data/ebay_session.json if present

2. GET eBay home (warm session like a normal browser)

3. Wait at least 4 seconds

4. GET one search URL for the keyword

5. Parse listings into EbaySearchData

6. Save cookies + items back to data/ebay_session.json

7. If that fails: clear session, retry once, then stop

keyword → home GET → wait >= 4s → search GET → parse_ebay_search_page() → EbaySearchData

Request limits: max 1 search per process run, minimum 4s between requests, stop on captcha/challenge markers, no pagination.

Fetch a fresh HTML fixture

uv run python -m scripts.fetch_fixture "shoes men"
# writes tests/fixtures/shoes_men_search.html and updates session store

Step 7: Verify Output

Run tests

uv run pytest -v

Quick sanity check

uv run python -c "
from ebay_parse import parse_ebay_search_page
html = open('sample.html').read()
data = parse_ebay_search_page(html, keyword='shoes men')
assert len(data.items) >= 50
assert data.items[0].item_id
assert data.items[0].price
print('OK:', len(data.items), 'items |', data.items[0].title[:50])
"

Key Project Files

pyproject.toml

[project]
name = "ebay-com"
version = "0.1.0"
description = "eBay product and review scraper"
requires-python = ">=3.12"
dependencies = [
    "beautifulsoup4>=4.15.0",
    "httpx>=0.28.1",
    "lxml>=6.1.1",
    "pydantic>=2.13.4",
]

[dependency-groups]
dev = ["pyright>=1.1.410", "pytest>=9.0.3", "ruff>=0.15.16"]

ebay_parse.py

from models.search import EbaySearchData
from parsers.search_parser import parse_search_results

def parse_ebay_search_page(
    html: str,
    *,
    keyword: str | None = None,
    source_url: str | None = None,
) -> EbaySearchData:
    return parse_search_results(html, keyword=keyword, source_url=source_url)

parsers/selectors.py (excerpt)

SRP_RESULTS_LIST = "ul.srp-results"
SRP_ITEM_CARD = "li.s-card"
CARD_TITLE = ".s-card__title .su-styled-text.primary"
CARD_LINK = "a.s-card__link"
CARD_PRICE = ".s-card__price"
CARD_IMAGE = "img.s-card__image"
CARD_SUBTITLE = ".s-card__subtitle"

All selectors live in one file. When eBay changes markup, update parsers/selectors.py first - not scattered across parser modules.

Output Data Shape

Structured output lives in models/search.py:

EbaySearchData(
    keyword="shoes men",
    items=[
        SearchResultItem(
            item_id="168520328526",
            title="Off White Virgil Abloh Air Jordan 1 High VAA ...",
            url="<https://www.ebay.com/itm/168520328526>",
            price="$999.99",
            shipping="+$396.57 delivery",
            condition="Pre-Owned",
            location="United States",
            buy_format="Buy It Now",
            watchers="261 watchers",
        ),
        # ~60 listings on sample.html
    ],
)

Export to JSON: data.model_dump(mode="json")

Parser Iteration Loop

Improve parsers without fetching again

1. Read sample.html - search for the field that failed

2. Add or update the selector in parsers/selectors.py

3. Edit search_parser.py or text_utils.py

4. Run uv run pytest tests/test_search_parser.py -v

5. Run uv run pyright and uv run ruff check .

6. Repeat until tests pass and field values look correct

Do not edit the fixture during parser iteration unless eBay markup genuinely changed and you need a fresh snapshot via scripts/fetch_fixture.py.

Troubleshooting

Common Problems & Fixes

Design Rules

1. Never fetch during parser iteration - only read sample.html

2. Use scraper.py for live fetches - single place for headers, delay, and session reuse

3. Keep models in models/ - no inline dicts in parser code

4. Keep selectors in parsers/selectors.py - one file to update when eBay changes markup

5. Prefer stable classes like s-card__* over hashed CSS

6. Strict typing required - pyright mode is strict

7. One search per live run - do not loop requests or paginate

8. Stop on challenge pages - do not retry in a tight loop

Final File Checklist

Your folder should look like this:

ebay/
├── index.html # This guide
├── TUTORIAL.md
├── README.md
├── sample.html # Search HTML fixture
├── api.py # Re-exports EbayScraper
├── scraper.py # Live search fetch + parse
├── session_store.py # Cookies + items cache
├── ebay_parse.py # parse_ebay_search_page(html)
├── constants.py
├── main.py # Offline / live CLI
├── pyproject.toml
├── scripts/
│ └── fetch_fixture.py # Download one search HTML fixture
├── models/
│ └── search.py # EbaySearchData, SearchResultItem
├── parsers/
│ ├── selectors.py # * Update selectors here
│ ├── text_utils.py # Title / attribute helpers
│ ├── url_utils.py # Item URL helpers
│ └── search_parser.py # Search DOM → EbaySearchData
├── data/
│ └── .gitkeep # ebay_session.json at runtime
└── tests/
├── conftest.py
├── test_search_parser.py
├── test_url_utils.py
└── test_session_store.py

You're done!

Parse local HTML with:

uv run python main.py --html sample.html --limit 10

Live search with:

uv run python main.py --search "shoes men"

Verify with:

uv run pytest

Questions? Double-check file names and paths - 95% of issues are a typo or a missing sample.html.