How to scrape Instagram (profile & posts) step-by-step Guide

Scrape Instagram Profiles

A beginner-friendly guide. No programming knowledge needed - just follow each step.

What We're Building

A small program that:

1. Opens a real browser (Brave or Chrome) and visits an Instagram profile

2. Saves the page HTML locally

3. Pulls out the profile info (name, bio, followers, posts) and prints it

Instagram blocks normal HTTP requests, so we use a real browser controlled through Chrome DevTools Protocol (CDP). The browser opens, loads the page, waits for it to render, and grabs the HTML. After that first fetch, everything runs offline against the saved file.

Step 1: Install Python 3.12

Python Installation

Windows:

1. Go to python.org/downloads

2. Click the yellow button that says "Download Python 3.12.x"

3. Run the downloaded .exe file

4. IMPORTANT: Check the box that says "Add Python to PATH" at the bottom

5. Click "Install Now" and wait for it to finish

macOS:

1. Go to python.org/downloads

2. Click the yellow button for "Download Python 3.12.x"

3. Open the downloaded .pkg file and follow the installer

Ubuntu / Debian Linux:

sudo apt update
sudo apt install python3.12 python3.12-venv

Fedora Linux:

sudo dnf install python3.12

Verify it worked:

Open a terminal (Command Prompt on Windows, Terminal on Mac/Linux) and type:

python3.12 --version

You should see:

Python 3.12.x

Step 2: Install a Browser

Choose one browser

We need Brave or Google Chrome installed. Either works - the program controls it through a debugging port.

Brave Browser (recommended):

1. Go to brave.com/download

2. Download and install like any other program

Google Chrome:

1. Go to google.com/chrome

2. Download and install

Important for Linux users: If you install Brave via the terminal, note where it ends up. The default is usually /usr/bin/brave-browser. If it's somewhere else, you'll update one line of code later.

Step 3: Install uv (Package Manager)

Install uv

uv is a fast tool that installs Python libraries. We use it instead of pip.

Windows (PowerShell):

powershell -c "irm <https://astral.sh/uv/install.ps1> | iex"

macOS / Linux:

curl -LsSf <https://astral.sh/uv/install.sh> | sh

After installing, close and reopen your terminal.

Verify:

uv --version

Should show something like

uv 0.x.x

Step 4: Create the Project Folder and Files

Set up the project

Open your terminal and run these commands one at a time:

# Create the main folder
mkdir instagram-scraper
cd instagram-scraper

# Create subfolders
mkdir browser
mkdir models
mkdir data

Now create the following files. Copy-paste each one exactly.

File 1: pyproject.toml

[project]
name = "instagram-com"
version = "0.1.0"
description = "Instagram profile and post scraper using Brave CDP browser"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "beautifulsoup4>=4.15.0",
    "httpx>=0.28.1",
    "lxml>=6.1.1",
    "psutil>=7.0.0",
    "pydantic>=2.13.4",
    "websocket-client>=1.9.0",
]

[tool.ruff.lint]
fixable = ["ALL"]
select = ["I", "B", "E"]

[tool.pyright]
typeCheckingMode = "strict"
venvPath = "."
venv = ".venv"

File 2: constants.py

from pathlib import Path

# Paths
HOME = Path.home()
BASE_DIR = Path(__file__).parent

# Browser location - CHANGE THIS if your browser is elsewhere
DEBUG_BROWSER_PATH = "/usr/bin/brave-browser"

# Profile folder (cookies and settings are saved here automatically)
USER_PROFILE_DIR = BASE_DIR / "browser" / "browser_profile"

# Browser window size
WIN_W = 720
WIN_H = 760

# Instagram
INSTAGRAM_HOME = "<https://www.instagram.com/>"
SAMPLE_HTML_PATH = BASE_DIR / "sample.html"

For Windows users:

Change the browser path to something like:

DEBUG_BROWSER_PATH = "C:\\Program Files\\BraveSoftware\\Brave-Browser\\Application\\brave.exe"

For macOS users:

DEBUG_BROWSER_PATH = "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser"

If using Chrome, replace

brave

with

chrome

in the path.

File 3: models/__init__.py

File 4: models/browser.py

from subprocess import Popen

import websocket
from pydantic import BaseModel, Field

class DebuggerTab(BaseModel):
    id: str = ""
    url: str = ""
    type: str = ""
    title: str = ""
    description: str = ""
    debug_url: str = Field(alias="webSocketDebuggerUrl")
    devtools_frontend_url: str = Field(alias="devtoolsFrontendUrl")

class BrowserDebuggerInfo(BaseModel):
    tabs: list[DebuggerTab] = []
    last_updates: float = -1

class Browser:
    def __init__(self) -> None:
        self.ws: websocket.WebSocket = websocket.WebSocket()
        self.ws_url: str = ""
        self.__msg_id: int = -1
        self.debugger_info: BrowserDebuggerInfo = BrowserDebuggerInfo()
        self.authorization_token: str = ""
        self.process: Popen[bytes] | None = None
        self.process_id: int = -1

    def connect(self) -> None:
        if not self.ws_url:
            self.ws_url = self.debugger_info.tabs[0].debug_url
        self.ws.connect(self.ws_url)

    def next_id(self) -> int:
        self.__msg_id += 1
        return self.__msg_id

    @property
    def msg_id(self):
        return self.__msg_id

File 5: models/profile.py

from pydantic import BaseModel, Field

class BioLink(BaseModel):
    title: str = ""
    url: str = ""
    link_type: str = ""

class ProfilePost(BaseModel):
    shortcode: str
    post_url: str
    display_uri: str | None = None
    media_type: int = 1  # 1=photo, 2=video, 8=carousel
    is_video: bool = False
    is_carousel: bool = False
    carousel_media_count: int | None = None
    caption: str | None = None
    accessibility_caption: str | None = None
    owner_username: str | None = None

class InstagramProfileData(BaseModel):
    pk: str | None = None
    username: str | None = None
    full_name: str | None = None
    biography: str | None = None
    profile_pic_url: str | None = None
    is_private: bool | None = None
    is_verified: bool | None = None
    follower_count: int | None = None
    following_count: int | None = None
    bio_links: list[BioLink] = Field(default_factory=list)
    posts: list[ProfilePost] = Field(default_factory=list)

Step 5: Add the Browser Control Files

These files let the program open and control the browser

Create each file below inside the browser/ folder.

browser/__init__.py

browser/utils.py

import os
import subprocess
import time

import httpx

from constants import (
    DEBUG_BROWSER_PATH,
    USER_PROFILE_DIR,
    WIN_H,
    WIN_W,
)
from models.browser import Browser, DebuggerTab

def spawn_debug_browser(debug_port: int, headless: bool = False):
    user_data_dir = USER_PROFILE_DIR
    if not user_data_dir.exists():
        os.makedirs(user_data_dir, exist_ok=True)

    cmd = [
        DEBUG_BROWSER_PATH,
        "--no-sandbox",
        f"--remote-debugging-port={debug_port}",
        "--remote-allow-origins=*",
        "--no-first-run",
        "--no-default-browser-check",
        f"--window-size={WIN_W},{WIN_H}",
        "--window-position=0,0",
        "--disable-popup-blocking",
        f"--user-data-dir={user_data_dir}",
        "--disable-component-update",
        "--metrics-recording-only",
        "--disable-gpu",
        "--disable-dev-shm-usage",
    ]
    if headless:
        cmd += ["--headless"]
    process = subprocess.Popen(
        cmd,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    )

    return process.pid, process

def attach_debugger(browser: Browser, debug_port: int):
    while True:
        try:
            response = httpx.get(f"<http://127.0.0.1>:{debug_port}/json")
            response.raise_for_status()
            browser.debugger_info.tabs = [DebuggerTab(**i) for i in response.json()]
            browser.debugger_info.last_updates = time.time()
            break
        except httpx.ConnectError:
            print("Waiting for browser to be ready ...")
            time.sleep(2)

browser/actions.py

import json
import time
from typing import Any

from models.browser import Browser

def wait_for_page_load(browser: Browser, timeout: int = 30) -> None:
    start_time = time.time()
    while time.time() - start_time < timeout:
        try:
            response = json.loads(browser.ws.recv())
            if (
                "method" in response
                and response["method"] == "Page.frameStoppedLoading"
            ):
                print("Frame stopped loading")
                return
        except Exception as e:
            print(f"Error waiting for load: {e}")
            break
    print(f"Timeout after {timeout}s waiting for page load")

def browser_send_command(
    browser: Browser,
    method: str,
    params: dict[str, str | int | float | list[Any]] | None = None,
    wait_for_load_event: bool = False,
):
    message = {"id": browser.next_id(), "method": method, "params": params or {}}
    browser.ws.send(json.dumps(message))

    while True:
        response = json.loads(browser.ws.recv())
        if response.get("id") == browser.msg_id:
            break

    if "error" in response:
        raise ValueError(
            f"CDP command failed: {response['error']['message']} "
            f"(code {response['error']['code']})"
        )

    result = response.get("result")

    if wait_for_load_event and method == "Page.navigate":
        wait_for_page_load(browser, timeout=30)

    return result

def browser_init_domains(
    browser: Browser,
    domains: list[str] = [
        "Page.enable",
        "Runtime.enable",
        "DOM.enable",
        "Network.enable",
        "Log.enable",
    ],
):
    for domain in domains:
        browser_send_command(browser, domain, {})

def browser_open_url(
    browser: Browser,
    url: str,
    wait_for_load_event: bool = False,
) -> None:
    browser.ws_url = url
    browser_send_command(browser, "Page.enable", {})
    browser_send_command(
        browser, "Page.navigate", {"url": url},
        wait_for_load_event=wait_for_load_event,
    )

def dom_js_execute(
    browser: Browser,
    js: str,
    options: dict[str, str | bool | int] = {},
):
    result = browser_send_command(
        browser, "Runtime.evaluate", {"expression": js, **options}
    )
    return result.get("result", {})

def dom_element_html(
    browser: Browser,
    selector: str,
    *,
    outer_html: bool = False,
) -> str:
    script = f'document.querySelector("{selector}").innerHTML'
    if outer_html:
        script = f'document.querySelector("{selector}").outerHTML'
    res = dom_js_execute(browser, script)
    return res.get("value", "")

browser/browser.py

import atexit
import time

import psutil

from browser.actions import (
    browser_init_domains,
    browser_open_url,
    dom_element_html,
)
from browser.utils import attach_debugger, spawn_debug_browser
from models.browser import Browser

def debug_setup_phase(browser: Browser):
    if not browser.ws.connected:
        browser.connect()
        return

def initialize_cpd_domains(browser: Browser):
    br = browser
    browser_init_domains(br)

def get_browser_instance(debug_port: int = 9999, headless: bool = False):
    browser = Browser()
    debug_port = 9999
    browser.process_id, browser.process = spawn_debug_browser(debug_port, headless)
    attach_debugger(browser, debug_port)
    cdp_initialized = False
    while browser.process and browser.process.poll() is None:
        if not browser.ws.connected:
            debug_setup_phase(browser)
            continue

        if not cdp_initialized:
            initialize_cpd_domains(browser)
            cdp_initialized = True
            continue

        if browser.debugger_info.last_updates == -1:
            attach_debugger(browser, debug_port)
            cdp_initialized = False
            continue
        break
    atexit.register(lambda: kill_browser(browser))
    return browser

def kill_browser(browser: Browser):
    process = browser.process
    if process and process.poll() is None:
        try:
            parent = psutil.Process(process.pid)
            children = parent.children(recursive=True)
            for child in children:
                child.terminate()
            _, alive = psutil.wait_procs(children, timeout=3)
            for child in alive:
                child.kill()
            process.terminate()
            process.wait(timeout=3)
        except psutil.NoSuchProcess:
            pass
        except Exception as e:
            print(f"Error killing process: {e}")
            process.terminate()
            time.sleep(2)
            if process.poll() is None:
                process.kill()

def render_html(
    browser: Browser,
    visit_url: str,
    timeout_for_page_load: int = 5,
) -> str:
    browser_open_url(browser, visit_url)
    time.sleep(timeout_for_page_load)
    return dom_element_html(browser, "body", outer_html=True)

Step 6: Add the Scraper Files

These files handle fetching and parsing Instagram data

Create each file in the project root.

instagram_parse.py

import json

from bs4 import BeautifulSoup

from models.profile import BioLink, InstagramProfileData, ProfilePost

INSTAGRAM_BASE = "<https://www.instagram.com>"

MEDIA_VIDEO = 2
MEDIA_CAROUSEL = 8

def _extract_xig_user(html: str, *, field_marker: str) -> dict | None:
    soup = BeautifulSoup(html, "lxml")
    for script in soup.find_all("script", type="application/json"):
        txt = script.string or ""
        if "xig_user_by_username" not in txt or field_marker not in txt:
            continue
        try:
            data = json.loads(txt)
            node = (
                data["require"][0][3][0]
                ["__bbox"]["require"][0][3][1]
                ["__bbox"]["result"]["data"]
                ["xig_user_by_username"]
            )
            if field_marker in node:
                return node
        except (KeyError, IndexError, json.JSONDecodeError, TypeError):
            continue
    return None

def _parse_bio_links(raw: list) -> list[BioLink]:
    out = []
    for item in raw or []:
        out.append(BioLink(
            title=item.get("title") or "",
            url=item.get("url") or "",
            link_type=item.get("link_type") or "",
        ))
    return out

def _parse_posts(posts_user: dict) -> list[ProfilePost]:
    connection = posts_user.get("polaris_ordered_timeline_connection") or {}
    edges = connection.get("edges") or []
    out = []
    for edge in edges:
        node = edge.get("node") or {}
        code = node.get("code") or ""
        if not code:
            continue
        media_type = node.get("media_type") or 1
        caption_obj = node.get("caption") or {}
        caption_text = (
            caption_obj.get("text")
            if isinstance(caption_obj, dict) else None
        )
        owner = node.get("user") or {}
        out.append(ProfilePost(
            shortcode=code,
            post_url=f"{INSTAGRAM_BASE}/p/{code}/",
            display_uri=node.get("display_uri"),
            media_type=media_type,
            is_video=media_type == MEDIA_VIDEO,
            is_carousel=media_type == MEDIA_CAROUSEL,
            carousel_media_count=node.get("carousel_media_count"),
            caption=caption_text,
            accessibility_caption=node.get("accessibility_caption"),
            owner_username=owner.get("username"),
        ))
    return out

def parse_profile_page(html: str) -> InstagramProfileData:
    profile_user = _extract_xig_user(html, field_marker="follower_count")
    posts_user = _extract_xig_user(
        html, field_marker="polaris_ordered_timeline_connection"
    )

    if profile_user is None and posts_user is None:
        return InstagramProfileData()

    u = profile_user or {}
    posts = _parse_posts(posts_user) if posts_user else []

    return InstagramProfileData(
        pk=str(u.get("pk") or "") or None,
        username=u.get("username"),
        full_name=u.get("full_name"),
        biography=u.get("biography"),
        profile_pic_url=u.get("profile_pic_url"),
        is_private=u.get("is_private"),
        is_verified=u.get("is_verified"),
        follower_count=u.get("follower_count"),
        following_count=u.get("following_count"),
        bio_links=_parse_bio_links(u.get("bio_links") or []),
        posts=posts,
    )

api.py

from pathlib import Path

from browser.browser import get_browser_instance, kill_browser, render_html
from instagram_parse import parse_profile_page
from models.profile import InstagramProfileData

INSTAGRAM_BASE = "<https://www.instagram.com>"
RENDER_TIMEOUT = 15

class InstagramClient:
    def __init__(self) -> None:
        self._browser = None

    def _ensure_browser(self) -> None:
        if self._browser is None:
            print("Spawning Brave browser ...")
            self._browser = get_browser_instance(9999)

    def fetch_profile_html(self, username: str) -> str:
        url = f"{INSTAGRAM_BASE}/{username}/"
        self._ensure_browser()
        assert self._browser is not None
        print(f"Fetching {url} ...")
        return render_html(self._browser, url, timeout_for_page_load=RENDER_TIMEOUT)

    def scrape_profile(
        self,
        username: str,
        *,
        html_path: Path | None = None,
    ) -> InstagramProfileData:
        if html_path and html_path.exists():
            html = html_path.read_text(encoding="utf-8")
        else:
            html = self.fetch_profile_html(username)
            if html_path:
                html_path.write_text(html, encoding="utf-8")
                print(f"Saved {len(html)} bytes to {html_path}")

        return parse_profile_page(html)

    def close(self) -> None:
        if self._browser:
            kill_browser(self._browser)
            self._browser = None

    def __enter__(self) -> "InstagramClient":
        return self

    def __exit__(self, exc_type, exc, tb) -> None:
        self.close()

main.py

import argparse

from api import InstagramClient
from constants import SAMPLE_HTML_PATH

DEFAULT_USERNAME = "cristiano"

def print_separator(char: str = "=", length: int = 50) -> None:
    print(char * length)

def main() -> None:
    parser = argparse.ArgumentParser(description="Instagram profile scraper")
    parser.add_argument(
        "--username",
        default=DEFAULT_USERNAME,
        help=f"Instagram username to scrape (default: {DEFAULT_USERNAME})",
    )
    parser.add_argument(
        "--fetch",
        action="store_true",
        help="Force a live browser fetch even if sample.html exists",
    )
    args = parser.parse_args()

    html_path = SAMPLE_HTML_PATH

    if args.fetch and html_path.exists():
        print(f"Removing cached {html_path} for fresh fetch ...")
        html_path.unlink()

    if html_path.exists():
        print(f"Using cached {html_path} ({html_path.stat().st_size} bytes)")
    else:
        print("sample.html not found - will fetch from browser.")

    with InstagramClient() as client:
        data = client.scrape_profile(args.username, html_path=html_path)

    print_separator()
    print("PROFILE")
    print_separator()
    print(f"  username    : {data.username}")
    print(f"  full name   : {data.full_name}")
    print(f"  bio         : {data.biography}")
    print(f"  private     : {data.is_private}")
    print(f"  verified    : {data.is_verified}")
    print(f"  followers   : {data.follower_count}")
    print(f"  following   : {data.following_count}")
    print(f"  profile pic : {(data.profile_pic_url or '')[:70]}...")
    if data.bio_links:
        print("  bio links   :")
        for lnk in data.bio_links:
            print(f"    [{lnk.title}] {lnk.url}")

    print()
    print_separator()
    print(f"POSTS ({len(data.posts)})")
    print_separator()
    for i, post in enumerate(data.posts, 1):
        kind = "carousel" if post.is_carousel else (
            "video" if post.is_video else "photo"
        )
        print(f"\n  {i}. [{kind}] {post.post_url}")
        if post.caption:
            print(f"     caption  : {post.caption[:80]}")
        if post.accessibility_caption:
            print(f"     alt text : {post.accessibility_caption}")
        if post.carousel_media_count:
            print(f"     slides   : {post.carousel_media_count}")

    print()
    print_separator()
    print("Done.")

if __name__ == "__main__":
    main()

fetch_sample.py (optional - one-shot fetcher)

from constants import SAMPLE_HTML_PATH
from browser.browser import get_browser_instance, kill_browser, render_html

PROFILE_URL = "<https://www.instagram.com/cristiano/>"
RENDER_TIMEOUT = 15

def main():
    if SAMPLE_HTML_PATH.exists():
        size = SAMPLE_HTML_PATH.stat().st_size
        print(f"sample.html already exists ({size} bytes). Delete it to re-fetch.")
        return

    print("Spawning Brave browser ...")
    browser = get_browser_instance(9999)

    print(f"Navigating to {PROFILE_URL} ...")
    html = render_html(browser, PROFILE_URL, timeout_for_page_load=RENDER_TIMEOUT)

    print(f"Got {len(html)} bytes of HTML")
    SAMPLE_HTML_PATH.write_text(html, encoding="utf-8")
    print(f"Saved to {SAMPLE_HTML_PATH}")

    kill_browser(browser)
    print("Browser closed.")

if __name__ == "__main__":
    main()

Step 7: Install Dependencies

Let uv install everything

Make sure you're in the instagram-scraper folder, then run:

uv sync

This reads pyproject.toml and installs:

• beautifulsoup4 + lxml - parses HTML

• httpx - talks to the browser's debug port

• pydantic - handles data cleanly

• psutil - manages the browser process

• websocket-client - sends commands to the browser

You should see: Resolved X packages in Xms followed by installation messages. If you see errors, double-check that you're in the right folder.

Linux only - pyright dependency:

pyright ships with a bundled Node binary that requires

libatomic

, which is absent on minimal Ubuntu/Debian installs. If

uv run pyright

fails with

libatomic.so.1: cannot open shared object file

, install it:

sudo apt install libatomic1

Step 8: Run the Scraper

First run - fetch the profile

In your terminal (still in the instagram-scraper folder):

uv run python main.py

What happens on the first run:

1. uv creates a virtual environment (one-time setup)

2. No sample.html exists yet, so a Brave/Chrome browser window opens

3. The browser navigates to the Instagram profile and waits 15 seconds for it to render

4. The HTML is saved as sample.html

5. The parser reads the local file and prints profile + posts

6. The browser closes

Every run after that: The browser never opens. The program reads the saved sample.html and parses it instantly.

Expected Output

Using cached .../sample.html (615236 bytes)
==================================================
PROFILE
==================================================
  username    : cristiano
  full name   : Cristiano Ronaldo
  bio         : Hi
  private     : False
  verified    : False
  followers   : 85
  following   : 87
  profile pic : <https://scontent.cdninstagram.com/v/t51.82787-19/567143892_1>...
  bio links   :
    [protfolio] hrbl.me/CR7Pro2col
    [linkedin] <https://www.linkedin.com/in/cristiano/>
    [github] <https://github.com/cristiano>

==================================================
POSTS (3)
==================================================

  1. [carousel] <https://www.instagram.com/voguemagazine/p/DcGauVjJtS8/>
     caption  : ...
     alt text : Photo by Cristiano Ronaldo on May 15, 2026.
     slides   : 2

  2. [carousel] <https://www.instagram.com/voguemagazine/p/DcGauVjJtS8/>
     alt text : Photo by Cristiano Ronaldo on January 15, 2026.
     slides   : 3

  3. [carousel] <https://www.instagram.com/voguemagazine/p/DcGauVjJtS8/>
     alt text : Photo by Cristiano Ronaldo on November 23, 2025.
     slides   : 5

==================================================
Done.

Want to scrape a different profile?

# Delete the cached file and fetch someone else
rm sample.html
uv run python main.py --username someoneelse

# Or use --fetch to force a re-fetch
uv run python main.py --fetch --username someoneelse

Troubleshooting

Common Problems and Fixes

"Command not found: uv"

Close and reopen your terminal. If it still doesn't work, restart your computer.

"python3.12: command not found"

On Windows, try python instead of python3.12. On Mac, make sure you installed from python.org (not the built-in one).

Browser doesn't open / "connection refused"

Check constants.py. Make sure DEBUG_BROWSER_PATH points to where your browser is actually installed. For Windows, use double backslashes: C:\\Program Files\\...

"No module named 'models'"

Make sure you created the models/__init__.py file (it can be empty). Same for browser/__init__.py.

Browser opens but nothing happens

Wait up to 30 seconds. The first run is slow because it creates a browser profile from scratch. After that it's faster.

Instagram shows a login page in sample.html

Instagram sometimes requires login for profiles. Try a different, public profile. The parser will still extract what it can from og tags, but the full data (follower count, posts) may not be there.

sample.html is empty or very small

The page might not have finished rendering. Delete sample.html and try again. If it keeps happening, increase RENDER_TIMEOUT in api.py from 15 to 25.

Final File Checklist

Your folder should look exactly like this:

instagram-scraper/
├── pyproject.toml
├── constants.py
├── api.py
├── instagram_parse.py
├── fetch_sample.py
├── main.py
├── sample.html (created on first run)
├── browser/
│ ├── init.py
│ ├── browser.py
│ ├── actions.py
│ └── utils.py
├── models/
│ ├── init.py
│ ├── browser.py
│ └── profile.py
└── data/
└── .gitkeep

You're done!

If every file is in place and you ran

uv sync

, the scraper should work. Run it anytime with:

cd instagram-scraper
uv run python main.py

Questions? Double-check file names and paths - 95% of issues are a typo or a missing file.