A beginner-friendly guide for Linux. No programming knowledge needed - just follow each step in order. Written for a fresh server (or a normal desktop) with nothing installed.

. Every tool gets its own install step below. Do the steps 

 and do not skip any. The four most common mistakes are:

 - only text and numbers. It works on a server with no screen.

. It already contains the whole program and a saved example (

) so you can try it before going online. 

 is a window where you type commands instead of clicking buttons. Every command in this guide is typed here, then you press 

If you are on a server (a droplet/VPS), you are already inside a terminal after you connect. Skip to the check below.

It prints the folder you are currently in. If it prints a path like 

Step 2: Install the Basic Tools (unzip and curl)

 installer. Neither is installed by default on a fresh Linux system, so install them now.

sudo apt update
sudo apt install -y unzip curl

Each should print version information. If either says 

, run the install command again and read the error message.

Step 3: Get the Project (extract into a folder named 

. If you just double-click "Extract Here," the files (

, ...) get dumped loose into the current folder, and later 

# 1. Go to where the uploaded zip lives (usually your home folder)
cd ~

# 2. Create the destination folder
mkdir -p youtube

# 3. Extract the zip's contents INTO that folder (-d means destination)
unzip -o youtube.zip -d youtube

# 4. Enter the folder
cd youtube

# 5. Confirm the files are here
ls

Already extracted the files loose somewhere?

mkdir -p ~/youtube
mv api.py constants.py fetch_sample.py main.py pyproject.toml \
   youtube_parse.py uv.lock sample.json index.html TUTORIAL.md \
   models data ~/youtube/
cd ~/youtube

Errors about files that are "not found" are fine if those files are already in place.

 the zip is not in your home folder. Find it with 

 into the folder it printed before running 

From now on, every command is run while inside 

 problem (and means you never install Python by hand)

 is a fast tool that installs the Python libraries this project needs - including 

, so you do not install Python - and you should not try to.

curl -LsSf https://astral.sh/uv/install.sh | sh

 (installed in Step 2) to download the installer and run it. It puts 

 so it can find the new program. (Or run 

, add the folder to your PATH, then reload it:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
uv --version

Step 5: Install yt-dlp (into the project's own 

Should print a date-like version such as 

. The apt version is often months out of date and stops working with YouTube. The project's copy in 

 YouTube changes often. If fetching suddenly stops working, update yt-dlp first:

 go back to Step 4. If it fails with network/download errors, check your internet connection and run 

libatomic.so.1: cannot open shared object file

First run - read the saved example (no internet used)

Using cached /home/<you>/youtube/sample.json (55119 bytes)

============================================================
VIDEO: Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)
============================================================
  url        : https://www.youtube.com/watch?v=dQw4w9WgXcQ
  video id   : dQw4w9WgXcQ
  channel    : Rick Astley  (4,550,000 subs)
  likes      : 19,490,351   views: 1,828,540,610
  uploaded   : 20091025   duration: 213s
  desc       : The official video for “Never Gonna Give You Up” by Rick Astley. ...
  tags       : rick astley, Never Gonna Give You Up, nggyu, ...

  SUBTITLES (6 tracks)
    en       manual   89 lines  English
    de-DE    manual   75 lines  German (Germany)
    ja       manual   83 lines  Japanese
    pt-BR    manual   69 lines  Portuguese (Brazil)
    es-419   manual   69 lines  Spanish (Latin America)
    en       auto     52 lines  English (Original)

  TOP COMMENTS (20)
     1. @grustache4569  (105,000 likes)
        I Rick rolled my entire French class because we had to study a city ...
     2. @candycake9531  (292,000 likes)
        Is everyone going to ignore the fact that Rick Astley looks like ...
  ...

============================================================
Saved full result to /home/<you>/youtube/data/dQw4w9WgXcQ.json
Done.

 subtitles were uploaded by the video's owner (one per language). 

 is YouTube's automatic speech-to-text, in the language spoken in the video. YouTube's machine translations into ~150 other languages are skipped on purpose.

The comments are in YouTube's "Top comments" order

 - the same order you see on the website - not sorted by number of likes. A pinned comment is marked 

Copy the video's link from your browser's address bar, then:

uv run python main.py --url "https://www.youtube.com/watch?v=jNQXAC9IVRw" --fetch

) and live links work too. It takes about 5 seconds and prints its progress:

Fetching metadata and top comments for https://www.youtube.com/watch?v=jNQXAC9IVRw ...
  got 20 comments
  subtitle en (manual): 440 bytes
  subtitle de (manual): 338 bytes
Backed up previous sample.json to sample.json.bak
Saved 16272 bytes to /home/<you>/youtube/sample.json

...followed by the same summary as Step 6.

, the program stops and tells you there is no downloaded copy of that video. That is on purpose - it will not show you the wrong video's data. Just add 

 Every run saves the complete result - full description, every subtitle (as text and as a 

 subtitle file's contents), and all comments - to 

. Open it with any text editor, or view it with:

Proxyrack

Company

How to Scrape a Youtube Video (likes count, descriptions, subtitles and top comments)

, and no browser. Every tool gets its own install step below. Do the steps 

 and do not skip any. The five most common mistakes are:

TikTok builds its search page with JavaScript, so the raw page has no results in it. We use a real browser controlled through Chrome DevTools Protocol (CDP). The browser loads the page, scrolls, and grabs the finished HTML. After that first fetch, everything runs offline against the saved file.

 (invisible) for the final fetch, but headless 

headed init run under a virtual screen called Xvfb (Step 5 + Step 8A)

. That headed run invokes TikTok's JavaScript to verify your session and cookies — it fails with 

. Only after it fails do you run the headless fetch (Step 8B), which succeeds on the second try with about 

. It already contains the whole program - including the parser, the API, and the command-line tool. 

You do not copy code from any other project and you do not type the code by hand.

 printing nothing is normal - the default headless browser does not need a screen.

# 1. Go to where the uploaded zip lives (usually your home folder)
cd ~

# 2. Create the destination folder
mkdir -p tiktok

# 3. Extract the zip's contents INTO that folder (-d means destination)
unzip -o tiktok.zip -d tiktok

# 4. Enter the folder
cd tiktok

# 5. Confirm the files are here
ls

mkdir -p ~/tiktok
mv api.py constants.py fetch_sample.py main.py pyproject.toml \
   tiktok_parse.py uv.lock sample.html index.html TUTORIAL.md \
   browser models data ~/tiktok/
cd ~/tiktok

 is a fast tool that installs the Python libraries this project needs. It also 

, so you do not install Python - and you should not try to. The 

 apt package does not exist on Ubuntu 22.04 / 26.04 or Debian 12 / 13, so trying to install it just fails. 

The program controls a real browser through a debugging port. Either browser works.

 This scraper was built and verified with 

. It is the recommended browser and the default path (

Brave (recommended) - official repository:

sudo apt update
sudo apt install -y curl gpg

sudo curl -fsSLo /usr/share/keyrings/brave-browser-archive-keyring.gpg \
  https://brave-browser-apt-release.s3.brave.com/brave-browser-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/brave-browser-archive-keyring.gpg] \
https://brave-browser-apt-release.s3.brave.com/ stable main" \
  | sudo tee /etc/apt/sources.list.d/brave-browser-release.list

sudo apt update
sudo apt install -y brave-browser

sudo apt install -y wget
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install -y ./google-chrome-stable_current_amd64.deb

. If Chrome is installed some other way, that path may not exist.

If it prints a different path, you will edit one line of 

 later (the file is in the reference section).

MUST-DO - required for Step 8. Headless fetching is impossible without this.

 Even though the final fetch is headless, the mandatory first run (Step 8A) is headed and needs a virtual screen on a server. Install it now:

sudo apt install -y xvfb
which xvfb-run
xvfb-run --help

Make sure you are in the project folder first, then sync:

 again. The first run downloads everything.

First run - parse the saved page (no browser)

 the browser never opens unless you ask with 

Using cached /home/<you>/tiktok/sample.html (533498 bytes)

============================================================
SEARCH: 'new trend'  (12 videos)
============================================================

   1. @eyochico (grimes)
      url      : https://www.tiktok.com/@eyochico/video/7585351064929553677
      video id : 7585351064929553677
      desc     : holy duo #foryou #adrianalima #meganfox #mogging #targetaudience
      likes    : 237800   comments: -   shares: -   plays: -
      posted   : 2025-12-19
      hashtags : #foryou #adrianalima #meganfox #mogging #targetaudience
      cover    : https://p16-common-sign.tiktokcdn.com/tos-useast5-p-0068-tx/...

============================================================
Done.

The number of videos changes from one capture to the next.

8A headed-init (must fail) → 8B headless retry

 line always prints the real number. The 

 (@eyochico) across samples of this query.

# A different query always needs --fetch — redo 8A then 8B:
xvfb-run -a uv run python main.py --query "dance challenge" --fetch --headed
# expect 0 videos (init), then:
uv run python main.py --query "dance challenge" --fetch
# if 0 videos, run the same headless command once more (max ~11/12)

, the program stops and tells you there is no downloaded cache for it. That is on purpose - it will not show you the wrong query's data.

Step 8: Live Fetch (get fresh results) — do 8A first, then 8B

8A headed-under-Xvfb first (it MUST fail)

. Running headless before 8A will not work.

How to Scrape TikTok Search Results

A beginner-friendly guide. No programming knowledge needed - just follow each step.

Instagram blocks normal HTTP requests, so we use a real browser controlled through Chrome DevTools Protocol (CDP). The browser opens, loads the page, waits for it to render, and grabs the HTML. After that first fetch, everything runs offline against the saved file.

sudo apt update
sudo apt install python3.12 python3.12-venv


Open a terminal (Command Prompt on Windows, Terminal on Mac/Linux) and type:

 installed. Either works - the program controls it through a debugging port.

 If you install Brave via the terminal, note where it ends up. The default is usually 

. If it's somewhere else, you'll update one line of code later.

 is a fast tool that installs Python libraries. We use it instead of 

powershell -c "irm <https://astral.sh/uv/install.ps1> | iex"


curl -LsSf <https://astral.sh/uv/install.sh> | sh


Step 4: Create the Project Folder and Files

Open your terminal and run these commands 

# Create the main folder
mkdir instagram-scraper
cd instagram-scraper

# Create subfolders
mkdir browser
mkdir models
mkdir data


[project]
name = "instagram-com"
version = "0.1.0"
description = "Instagram profile and post scraper using Brave CDP browser"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "beautifulsoup4>=4.15.0",
    "httpx>=0.28.1",
    "lxml>=6.1.1",
    "psutil>=7.0.0",
    "pydantic>=2.13.4",
    "websocket-client>=1.9.0",
]

[tool.ruff.lint]
fixable = ["ALL"]
select = ["I", "B", "E"]

[tool.pyright]
typeCheckingMode = "strict"
venvPath = "."
venv = ".venv"


from pathlib import Path

# Paths
HOME = Path.home()
BASE_DIR = Path(__file__).parent

# Browser location - CHANGE THIS if your browser is elsewhere
DEBUG_BROWSER_PATH = "/usr/bin/brave-browser"

# Profile folder (cookies and settings are saved here automatically)
USER_PROFILE_DIR = BASE_DIR / "browser" / "browser_profile"

# Browser window size
WIN_W = 720
WIN_H = 760

# Instagram
INSTAGRAM_HOME = "<https://www.instagram.com/>"
SAMPLE_HTML_PATH = BASE_DIR / "sample.html"


Change the browser path to something like:

DEBUG_BROWSER_PATH = "C:\\Program Files\\BraveSoftware\\Brave-Browser\\Application\\brave.exe"


DEBUG_BROWSER_PATH = "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser"


from subprocess import Popen

import websocket
from pydantic import BaseModel, Field

class DebuggerTab(BaseModel):
    id: str = ""
    url: str = ""
    type: str = ""
    title: str = ""
    description: str = ""
    debug_url: str = Field(alias="webSocketDebuggerUrl")
    devtools_frontend_url: str = Field(alias="devtoolsFrontendUrl")

class BrowserDebuggerInfo(BaseModel):
    tabs: list[DebuggerTab] = []
    last_updates: float = -1

class Browser:
    def __init__(self) -> None:
        self.ws: websocket.WebSocket = websocket.WebSocket()
        self.ws_url: str = ""
        self.__msg_id: int = -1
        self.debugger_info: BrowserDebuggerInfo = BrowserDebuggerInfo()
        self.authorization_token: str = ""
        self.process: Popen[bytes] | None = None
        self.process_id: int = -1

    def connect(self) -> None:
        if not self.ws_url:
            self.ws_url = self.debugger_info.tabs[0].debug_url
        self.ws.connect(self.ws_url)

    def next_id(self) -> int:
        self.__msg_id += 1
        return self.__msg_id

    @property
    def msg_id(self):
        return self.__msg_id


from pydantic import BaseModel, Field

class BioLink(BaseModel):
    title: str = ""
    url: str = ""
    link_type: str = ""

class ProfilePost(BaseModel):
    shortcode: str
    post_url: str
    display_uri: str | None = None
    media_type: int = 1  # 1=photo, 2=video, 8=carousel
    is_video: bool = False
    is_carousel: bool = False
    carousel_media_count: int | None = None
    caption: str | None = None
    accessibility_caption: str | None = None
    owner_username: str | None = None

class InstagramProfileData(BaseModel):
    pk: str | None = None
    username: str | None = None
    full_name: str | None = None
    biography: str | None = None
    profile_pic_url: str | None = None
    is_private: bool | None = None
    is_verified: bool | None = None
    follower_count: int | None = None
    following_count: int | None = None
    bio_links: list[BioLink] = Field(default_factory=list)
    posts: list[ProfilePost] = Field(default_factory=list)


These files let the program open and control the browser

import os
import subprocess
import time

import httpx

from constants import (
    DEBUG_BROWSER_PATH,
    USER_PROFILE_DIR,
    WIN_H,
    WIN_W,
)
from models.browser import Browser, DebuggerTab

def spawn_debug_browser(debug_port: int, headless: bool = False):
    user_data_dir = USER_PROFILE_DIR
    if not user_data_dir.exists():
        os.makedirs(user_data_dir, exist_ok=True)

    cmd = [
        DEBUG_BROWSER_PATH,
        "--no-sandbox",
        f"--remote-debugging-port={debug_port}",
        "--remote-allow-origins=*",
        "--no-first-run",
        "--no-default-browser-check",
        f"--window-size={WIN_W},{WIN_H}",
        "--window-position=0,0",
        "--disable-popup-blocking",
        f"--user-data-dir={user_data_dir}",
        "--disable-component-update",
        "--metrics-recording-only",
        "--disable-gpu",
        "--disable-dev-shm-usage",
    ]
    if headless:
        cmd += ["--headless"]
    process = subprocess.Popen(
        cmd,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    )

    return process.pid, process

def attach_debugger(browser: Browser, debug_port: int):
    while True:
        try:
            response = httpx.get(f"<http://127.0.0.1>:{debug_port}/json")
            response.raise_for_status()
            browser.debugger_info.tabs = [DebuggerTab(**i) for i in response.json()]
            browser.debugger_info.last_updates = time.time()
            break
        except httpx.ConnectError:
            print("Waiting for browser to be ready ...")
            time.sleep(2)


import json
import time
from typing import Any

from models.browser import Browser

def wait_for_page_load(browser: Browser, timeout: int = 30) -> None:
    start_time = time.time()
    while time.time() - start_time < timeout:
        try:
            response = json.loads(browser.ws.recv())
            if (
                "method" in response
                and response["method"] == "Page.frameStoppedLoading"
            ):
                print("Frame stopped loading")
                return
        except Exception as e:
            print(f"Error waiting for load: {e}")
            break
    print(f"Timeout after {timeout}s waiting for page load")

def browser_send_command(
    browser: Browser,
    method: str,
    params: dict[str, str | int | float | list[Any]] | None = None,
    wait_for_load_event: bool = False,
):
    message = {"id": browser.next_id(), "method": method, "params": params or {}}
    browser.ws.send(json.dumps(message))

    while True:
        response = json.loads(browser.ws.recv())
        if response.get("id") == browser.msg_id:
            break

    if "error" in response:
        raise ValueError(
            f"CDP command failed: {response['error']['message']} "
            f"(code {response['error']['code']})"
        )

    result = response.get("result")

    if wait_for_load_event and method == "Page.navigate":
        wait_for_page_load(browser, timeout=30)

    return result

def browser_init_domains(
    browser: Browser,
    domains: list[str] = [
        "Page.enable",
        "Runtime.enable",
        "DOM.enable",
        "Network.enable",
        "Log.enable",
    ],
):
    for domain in domains:
        browser_send_command(browser, domain, {})

def browser_open_url(
    browser: Browser,
    url: str,
    wait_for_load_event: bool = False,
) -> None:
    browser.ws_url = url
    browser_send_command(browser, "Page.enable", {})
    browser_send_command(
        browser, "Page.navigate", {"url": url},
        wait_for_load_event=wait_for_load_event,
    )

def dom_js_execute(
    browser: Browser,
    js: str,
    options: dict[str, str | bool | int] = {},
):
    result = browser_send_command(
        browser, "Runtime.evaluate", {"expression": js, **options}
    )
    return result.get("result", {})

def dom_element_html(
    browser: Browser,
    selector: str,
    *,
    outer_html: bool = False,
) -> str:
    script = f'document.querySelector("{selector}").innerHTML'
    if outer_html:
        script = f'document.querySelector("{selector}").outerHTML'
    res = dom_js_execute(browser, script)
    return res.get("value", "")


import atexit
import time

import psutil

from browser.actions import (
    browser_init_domains,
    browser_open_url,
    dom_element_html,
)
from browser.utils import attach_debugger, spawn_debug_browser
from models.browser import Browser

def debug_setup_phase(browser: Browser):
    if not browser.ws.connected:
        browser.connect()
        return

def initialize_cpd_domains(browser: Browser):
    br = browser
    browser_init_domains(br)

def get_browser_instance(debug_port: int = 9999, headless: bool = False):
    browser = Browser()
    debug_port = 9999
    browser.process_id, browser.process = spawn_debug_browser(debug_port, headless)
    attach_debugger(browser, debug_port)
    cdp_initialized = False
    while browser.process and browser.process.poll() is None:
        if not browser.ws.connected:
            debug_setup_phase(browser)
            continue

        if not cdp_initialized:
            initialize_cpd_domains(browser)
            cdp_initialized = True
            continue

        if browser.debugger_info.last_updates == -1:
            attach_debugger(browser, debug_port)
            cdp_initialized = False
            continue
        break
    atexit.register(lambda: kill_browser(browser))
    return browser

def kill_browser(browser: Browser):
    process = browser.process
    if process and process.poll() is None:
        try:
            parent = psutil.Process(process.pid)
            children = parent.children(recursive=True)
            for child in children:
                child.terminate()
            _, alive = psutil.wait_procs(children, timeout=3)
            for child in alive:
                child.kill()
            process.terminate()
            process.wait(timeout=3)
        except psutil.NoSuchProcess:
            pass
        except Exception as e:
            print(f"Error killing process: {e}")
            process.terminate()
            time.sleep(2)
            if process.poll() is None:
                process.kill()

def render_html(
    browser: Browser,
    visit_url: str,
    timeout_for_page_load: int = 5,
) -> str:
    browser_open_url(browser, visit_url)
    time.sleep(timeout_for_page_load)
    return dom_element_html(browser, "body", outer_html=True)


These files handle fetching and parsing Instagram data

import json

from bs4 import BeautifulSoup

from models.profile import BioLink, InstagramProfileData, ProfilePost

INSTAGRAM_BASE = "<https://www.instagram.com>"

MEDIA_VIDEO = 2
MEDIA_CAROUSEL = 8

def _extract_xig_user(html: str, *, field_marker: str) -> dict | None:
    soup = BeautifulSoup(html, "lxml")
    for script in soup.find_all("script", type="application/json"):
        txt = script.string or ""
        if "xig_user_by_username" not in txt or field_marker not in txt:
            continue
        try:
            data = json.loads(txt)
            node = (
                data["require"][0][3][0]
                ["__bbox"]["require"][0][3][1]
                ["__bbox"]["result"]["data"]
                ["xig_user_by_username"]
            )
            if field_marker in node:
                return node
        except (KeyError, IndexError, json.JSONDecodeError, TypeError):
            continue
    return None

def _parse_bio_links(raw: list) -> list[BioLink]:
    out = []
    for item in raw or []:
        out.append(BioLink(
            title=item.get("title") or "",
            url=item.get("url") or "",
            link_type=item.get("link_type") or "",
        ))
    return out

def _parse_posts(posts_user: dict) -> list[ProfilePost]:
    connection = posts_user.get("polaris_ordered_timeline_connection") or {}
    edges = connection.get("edges") or []
    out = []
    for edge in edges:
        node = edge.get("node") or {}
        code = node.get("code") or ""
        if not code:
            continue
        media_type = node.get("media_type") or 1
        caption_obj = node.get("caption") or {}
        caption_text = (
            caption_obj.get("text")
            if isinstance(caption_obj, dict) else None
        )
        owner = node.get("user") or {}
        out.append(ProfilePost(
            shortcode=code,
            post_url=f"{INSTAGRAM_BASE}/p/{code}/",
            display_uri=node.get("display_uri"),
            media_type=media_type,
            is_video=media_type == MEDIA_VIDEO,
            is_carousel=media_type == MEDIA_CAROUSEL,
            carousel_media_count=node.get("carousel_media_count"),
            caption=caption_text,
            accessibility_caption=node.get("accessibility_caption"),
            owner_username=owner.get("username"),
        ))
    return out

def parse_profile_page(html: str) -> InstagramProfileData:
    profile_user = _extract_xig_user(html, field_marker="follower_count")
    posts_user = _extract_xig_user(
        html, field_marker="polaris_ordered_timeline_connection"
    )

    if profile_user is None and posts_user is None:
        return InstagramProfileData()

    u = profile_user or {}
    posts = _parse_posts(posts_user) if posts_user else []

    return InstagramProfileData(
        pk=str(u.get("pk") or "") or None,
        username=u.get("username"),
        full_name=u.get("full_name"),
        biography=u.get("biography"),
        profile_pic_url=u.get("profile_pic_url"),
        is_private=u.get("is_private"),
        is_verified=u.get("is_verified"),
        follower_count=u.get("follower_count"),
        following_count=u.get("following_count"),
        bio_links=_parse_bio_links(u.get("bio_links") or []),
        posts=posts,
    )


from pathlib import Path

from browser.browser import get_browser_instance, kill_browser, render_html
from instagram_parse import parse_profile_page
from models.profile import InstagramProfileData

INSTAGRAM_BASE = "<https://www.instagram.com>"
RENDER_TIMEOUT = 15

class InstagramClient:
    def __init__(self) -> None:
        self._browser = None

    def _ensure_browser(self) -> None:
        if self._browser is None:
            print("Spawning Brave browser ...")
            self._browser = get_browser_instance(9999)

    def fetch_profile_html(self, username: str) -> str:
        url = f"{INSTAGRAM_BASE}/{username}/"
        self._ensure_browser()
        assert self._browser is not None
        print(f"Fetching {url} ...")
        return render_html(self._browser, url, timeout_for_page_load=RENDER_TIMEOUT)

    def scrape_profile(
        self,
        username: str,
        *,
        html_path: Path | None = None,
    ) -> InstagramProfileData:
        if html_path and html_path.exists():
            html = html_path.read_text(encoding="utf-8")
        else:
            html = self.fetch_profile_html(username)
            if html_path:
                html_path.write_text(html, encoding="utf-8")
                print(f"Saved {len(html)} bytes to {html_path}")

        return parse_profile_page(html)

    def close(self) -> None:
        if self._browser:
            kill_browser(self._browser)
            self._browser = None

    def __enter__(self) -> "InstagramClient":
        return self

    def __exit__(self, exc_type, exc, tb) -> None:
        self.close()


import argparse

from api import InstagramClient
from constants import SAMPLE_HTML_PATH

DEFAULT_USERNAME = "cristiano"

def print_separator(char: str = "=", length: int = 50) -> None:
    print(char * length)

def main() -> None:
    parser = argparse.ArgumentParser(description="Instagram profile scraper")
    parser.add_argument(
        "--username",
        default=DEFAULT_USERNAME,
        help=f"Instagram username to scrape (default: {DEFAULT_USERNAME})",
    )
    parser.add_argument(
        "--fetch",
        action="store_true",
        help="Force a live browser fetch even if sample.html exists",
    )
    args = parser.parse_args()

    html_path = SAMPLE_HTML_PATH

    if args.fetch and html_path.exists():
        print(f"Removing cached {html_path} for fresh fetch ...")
        html_path.unlink()

    if html_path.exists():
        print(f"Using cached {html_path} ({html_path.stat().st_size} bytes)")
    else:
        print("sample.html not found - will fetch from browser.")

    with InstagramClient() as client:
        data = client.scrape_profile(args.username, html_path=html_path)

    print_separator()
    print("PROFILE")
    print_separator()
    print(f"  username    : {data.username}")
    print(f"  full name   : {data.full_name}")
    print(f"  bio         : {data.biography}")
    print(f"  private     : {data.is_private}")
    print(f"  verified    : {data.is_verified}")
    print(f"  followers   : {data.follower_count}")
    print(f"  following   : {data.following_count}")
    print(f"  profile pic : {(data.profile_pic_url or '')[:70]}...")
    if data.bio_links:
        print("  bio links   :")
        for lnk in data.bio_links:
            print(f"    [{lnk.title}] {lnk.url}")

    print()
    print_separator()
    print(f"POSTS ({len(data.posts)})")
    print_separator()
    for i, post in enumerate(data.posts, 1):
        kind = "carousel" if post.is_carousel else (
            "video" if post.is_video else "photo"
        )
        print(f"\n  {i}. [{kind}] {post.post_url}")
        if post.caption:
            print(f"     caption  : {post.caption[:80]}")
        if post.accessibility_caption:
            print(f"     alt text : {post.accessibility_caption}")
        if post.carousel_media_count:
            print(f"     slides   : {post.carousel_media_count}")

    print()
    print_separator()
    print("Done.")

if __name__ == "__main__":
    main()


from constants import SAMPLE_HTML_PATH
from browser.browser import get_browser_instance, kill_browser, render_html

PROFILE_URL = "<https://www.instagram.com/cristiano/>"
RENDER_TIMEOUT = 15

def main():
    if SAMPLE_HTML_PATH.exists():
        size = SAMPLE_HTML_PATH.stat().st_size
        print(f"sample.html already exists ({size} bytes). Delete it to re-fetch.")
        return

    print("Spawning Brave browser ...")
    browser = get_browser_instance(9999)

    print(f"Navigating to {PROFILE_URL} ...")
    html = render_html(browser, PROFILE_URL, timeout_for_page_load=RENDER_TIMEOUT)

    print(f"Got {len(html)} bytes of HTML")
    SAMPLE_HTML_PATH.write_text(html, encoding="utf-8")
    print(f"Saved to {SAMPLE_HTML_PATH}")

    kill_browser(browser)
    print("Browser closed.")

if __name__ == "__main__":
    main()


How to scrape Instagram (profile & posts) step-by-step Guide

Scrape Instagram Profiles

A step-by-step guide to scraping eBay search results into structured item data.

We'll create a small program that includes:

 Fetch HTML once and save it. Build and test the parser against that saved file - do not re-fetch while iterating on selectors.

curl -LsSf <https://astral.sh/uv/install.sh> | sh
source $HOME/.local/bin/env


pyright ships with a bundled Node binary that requires

, which is absent on minimal Ubuntu/Debian installs. If

libatomic.so.1: cannot open shared object file


Open your terminal, navigate to the project folder, then run:

uv run pytest          # all tests must pass
uv run ruff check .    # lint
uv run pyright         # strict type check


The parser works directly on HTML. You only need a search-results HTML file (like 

). No live request is required for this step.

uv run python main.py --html sample.html

# Show more rows
uv run python main.py --html sample.html --limit 10

# Optional label stored on the output object
uv run python main.py --html sample.html --keyword "shoes men" --limit 10


Parsed from local file: sample.html
============================================================
Keyword: None
Source: .../sample.html
Items found: 60
------------------------------------------------------------
- 168520328526 | $999.99 | Off White Virgil Abloh Air Jordan 1 High VAA ...
  <https://www.ebay.com/itm/168520328526>
  Pre-Owned | Buy It Now | +$396.57 delivery | United States
------------------------------------------------------------


Probe the saved HTML before changing selectors

uv run python -c "
from bs4 import BeautifulSoup
html = open('sample.html').read()
soup = BeautifulSoup(html, 'lxml')
cards = soup.select('ul.srp-results > li.s-card')
print('cards:', len(cards))
title = cards[0].select_one('.s-card__title .su-styled-text.primary')
print('title:', title.get_text(strip=True)[:80])
print('price:', cards[0].select_one('.s-card__price').get_text(strip=True))
"


uv run python main.py --search "shoes men"

uv run python main.py --search "shoes men" --fresh


keyword → home GET → wait >= 4s → search GET → parse_ebay_search_page() → EbaySearchData

 max 1 search per process run, minimum 4s between requests, stop on captcha/challenge markers, no pagination.

uv run python -m scripts.fetch_fixture "shoes men"
# writes tests/fixtures/shoes_men_search.html and updates session store


How to scrape eBay

A step-by-step guide to building a local-first HTML parser for Amazon product pages and embedded reviews.

 HTML is fetched once and saved to disk. All parser development runs against that saved file — never re-fetch during iteration.

→ Live HTML fetch via httpx (no browser unless Amazon blocks requests)

 → CLI to download and save HTML fixture

 → BeautifulSoup parsers + shared selectors and text helpers

→ Demo CLI — accepts a product URL, parses fixture (or live) and prints summary

 → Offline pytest suite against saved HTML

 pyright ships with a bundled Node binary that requires 

, which is absent on minimal Ubuntu/Debian installs. If 

uv run pytest # all tests must pass uv run ruff check . # lint uv run pyright # strict type check

 Only fetch when the fixture is missing or Amazon markup has changed. Never re-fetch during parser iteration.

# Default sample product (Nike Balaclava)
 uv run python -m scripts.fetch_fixture

# Any product or product-reviews URL (ASIN is extracted automatically)
 uv run python -m scripts.fetch_fixture "<https://www.amazon.com/dp/B001GAOTSW>"

# Custom output path
 uv run python -m scripts.fetch_fixture "<https://www.amazon.com/dp/B0959JT4PV>" \
  --output tests/fixtures/b0959jt4pv_product.html


 page often redirects to Amazon Sign-In over plain HTTP. Reviews are parsed from the 

embedded review block on the product page

Probe the saved HTML before writing selectors

Amazon product pages are large (~2 MB). Inspect the saved HTML locally:

uv run python -c "
from bs4 import BeautifulSoup
html = open('tests/fixtures/b0959jt4pv_product.html').read()
soup = BeautifulSoup(html, 'lxml')
print('title:', soup.select_one('#productTitle').get_text(strip=True)[:60])
print('reviews:', len(soup.select('[data-hook=\"review\"]')))
"


data-hook='average-star-rating'] product_parser.py

a[aria-label] inside histogram widget product_parser.py

#detailBullets_feature_div li product_parser.py

[data-hook='reviewTitle'] review_parser.py

[data-hook='reviewText'] review_parser.py

 attributes (stable across redesigns), add selectors to 

uv run python main.py "<https://www.amazon.com/dp/B0959JT4PV>"


Parsed from local fixture: b0959jt4pv_product.html
============================================================
Title: Nike unisex-adult mens Balaclava
ASIN: B0959JT4PV
Brand: Nike
Rating: 4.7 (8302 reviews)
...
Reviews parsed from page: 13
- 5.0★ | Great Fit and Quality – Highly Recommend
  I recently bought this Nike ski mask...


uv run python -c "
from api import AmazonClient
url = '<https://www.amazon.com/Nike-Hyperwarm-Hydropull-Hood-Balaclava/dp/B0959JT4PV>'
with AmazonClient() as client:
    data = client.scrape_product(url)
print(data.product.title, '|', len(data.reviews), 'reviews')
"


product URL → normalize_product_url() → httpx GET → parse_amazon_page() → AmazonProductData

How to Scrape Amazon Product Pages and Reviews

How to scrape Amazon product pages and reviews

Web scraping using Python has become one of the most efficient ways to collect data from websites at scale. Whether you're building price trackers, monitoring competitors, gathering market intelligence, or training AI models, Python offers powerful libraries and tools that simplify the entire scraping process.

Many modern scraping projects rely on rotating proxy infrastructure to avoid bans and maintain reliable data collection. If you're new to proxies, check out:

If you're just getting started or want to improve your existing scraping setup, this guide covers everything from beginner to advanced concepts.

Web scraping is the process of automatically extracting data from websites.

Instead of manually copying information from web pages, scraping tools collect the data programmatically and organize it into structured formats such as:

Python is the most popular programming language for web scraping because it is:

Python also has excellent scraping libraries that handle:

You can also explore our detailed comparison of the best tools available today: 

Best Python Web Scraping Libraries in 2026

Most scraping projects follow this process:

Website → Python Script → Parsed Data → CSV/Database


Here’s a beginner-friendly scraping example:

importrequests
frombs4importBeautifulSoup

url="<https://example.com/products>"

headers= {
"User-Agent":"Mozilla/5.0"
}

response=requests.get(url,headers=headers)

soup=BeautifulSoup(response.text,"html.parser")

products=soup.find_all("h2")

forproductinproducts:
print(product.text)


Modern websites often load data dynamically using JavaScript.

fromplaywright.sync_apiimportsync_playwright

withsync_playwright()asp:
browser=p.chromium.launch()
page=browser.new_page()

page.goto("<https://example.com>")

print(page.title())

browser.close()


Websites use anti-bot systems to stop scrapers.

Proxy rotation distributes requests across multiple IP addresses.

Residential and mobile proxies are especially effective for scraping.

Learn more about how rotating IPs help prevent bans: 

Choosing the right proxy type matters for scraping performance and detection resistance: 

Always include realistic browser headers:

headers= {
"User-Agent":"Mozilla/5.0",
"Accept-Language":"en-US,en;q=0.9"
}


Reliable proxy infrastructure is essential for scaling scraping projects and avoiding bans: