Scraping GitHub with Firecrawl

Updated Feb 2026

Firecrawl can extract repository data, README content, issue discussions, and documentation from GitHub. This guide covers structured extraction, search, crawling, and batch operations for GitHub content.

Installation

npm install @mendable/firecrawl-js zod

# Or with Python
pip install firecrawl-py

JSON Mode: Structured Repository Data

Define a schema to extract validated repository metadata from any GitHub repo page.

import FirecrawlApp from "@mendable/firecrawl-js";
import { z } from "zod";

const app = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY });

const repoSchema = z.object({
  name: z.string().describe("Repository name"),
  description: z.string().describe("Repository description"),
  stars: z.string().describe("Star count"),
  forks: z.string().describe("Fork count"),
  language: z.string().describe("Primary programming language"),
  topics: z.array(z.string()).describe("Repository topics/tags"),
  lastUpdated: z.string().optional().describe("Last commit or update date"),
  license: z.string().optional().describe("License type"),
  openIssues: z.string().optional().describe("Number of open issues"),
});

const result = await app.scrapeUrl(
  "https://github.com/mendableai/firecrawl",
  {
    formats: ["json"],
    jsonOptions: {
      schema: repoSchema,
    },
  }
);

console.log(result.json);
// {
//   name: "firecrawl",
//   description: "Turn entire websites into LLM-ready markdown or structured data",
//   stars: "30,000+",
//   forks: "2,500+",
//   language: "TypeScript",
//   topics: ["web-scraping", "llm", "ai", "markdown"],
//   license: "AGPL-3.0",
//   openIssues: "120"
// }

Search: Find Repositories

Search for GitHub repositories using site-scoped queries.

const results = await app.search("machine learning site:github.com", {
  limit: 10,
  scrapeOptions: {
    formats: ["markdown"],
  },
});

for (const result of results.data) {
  console.log(`${result.title}`);
  console.log(`  ${result.url}`);
  console.log(`  ${result.description}\n`);
}

Search Tips

Query Pattern	What It Finds
"react component library" site:github.com	Repos matching the phrase
"awesome list" machine learning site:github.com	Curated resource lists
site:github.com/issues "bug report"	Open issues across repos
"README" web scraping site:github.com	READMEs mentioning web scraping

Scrape: Single Page Content

Extract a repository page, README, or issue as clean markdown.

Repository README

const result = await app.scrapeUrl(
  "https://github.com/mendableai/firecrawl",
  {
    formats: ["markdown"],
  }
);

console.log(result.markdown);
// Full README content as clean markdown

Issue or Discussion

const issue = await app.scrapeUrl(
  "https://github.com/mendableai/firecrawl/issues/1",
  {
    formats: ["markdown"],
  }
);

console.log(issue.markdown);
// Issue title, body, and comments as markdown

Specific File

const file = await app.scrapeUrl(
  "https://github.com/mendableai/firecrawl/blob/main/package.json",
  {
    formats: ["markdown"],
  }
);

console.log(file.markdown);

Map: Discover Repository URLs

Use Map to discover all navigable URLs within a repository -- documentation pages, directories, files, issues, and more. Map returns URLs only, without content.

const urls = await app.mapUrl(
  "https://github.com/mendableai/firecrawl"
);

console.log(`Found ${urls.links.length} URLs`);

// Filter to documentation pages
const docUrls = urls.links.filter((url) =>
  url.includes("/docs/") || url.includes("/wiki/")
);

// Filter to issues
const issueUrls = urls.links.filter((url) =>
  url.includes("/issues/")
);

console.log(`Docs: ${docUrls.length}, Issues: ${issueUrls.length}`);

Crawl: Multi-Page Extraction

Crawl documentation sites, wikis, or entire repositories.

// Crawl a GitHub wiki
const wikiResult = await app.crawlUrl(
  "https://github.com/mendableai/firecrawl/wiki",
  {
    limit: 30,
    scrapeOptions: {
      formats: ["markdown"],
    },
  }
);

for (const page of wikiResult.data) {
  console.log(`Page: ${page.metadata.title}`);
  console.log(`URL: ${page.metadata.sourceURL}`);
  console.log(`Content length: ${page.markdown.length} chars\n`);
}

Crawl GitHub Pages Documentation

Many projects host documentation on GitHub Pages:

const docsResult = await app.crawlUrl(
  "https://docs.example.com",  // GitHub Pages URL
  {
    limit: 50,
    includePaths: ["/docs/*", "/guides/*"],
    scrapeOptions: {
      formats: ["markdown"],
    },
  }
);

Batch Scrape: Multiple Repositories

Extract structured data from multiple repositories simultaneously.

const repoUrls = [
  "https://github.com/vercel/next.js",
  "https://github.com/facebook/react",
  "https://github.com/sveltejs/svelte",
  "https://github.com/vuejs/vue",
];

const batchResult = await app.batchScrapeUrls(repoUrls, {
  formats: ["json"],
  jsonOptions: {
    schema: repoSchema,
  },
});

console.log(`Completed: ${batchResult.completed}/${batchResult.total}`);

for (const repo of batchResult.data) {
  const { name, stars, language } = repo.json;
  console.log(`${name}: ${stars} stars (${language})`);
}

Extracting Issue Data

Create a schema tailored for GitHub issue extraction.

const issueSchema = z.object({
  title: z.string().describe("Issue title"),
  author: z.string().describe("Issue author username"),
  status: z.string().describe("Open or Closed"),
  labels: z.array(z.string()).describe("Issue labels"),
  body: z.string().describe("Issue body text"),
  commentCount: z.string().optional().describe("Number of comments"),
  createdAt: z.string().optional().describe("Creation date"),
});

const issueResult = await app.scrapeUrl(
  "https://github.com/mendableai/firecrawl/issues/1",
  {
    formats: ["json"],
    jsonOptions: {
      schema: issueSchema,
    },
  }
);

console.log(issueResult.json);

Python Example

from firecrawl import FirecrawlApp
from pydantic import BaseModel
from typing import Optional

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

class Repository(BaseModel):
    name: str
    description: str
    stars: str
    forks: str
    language: str
    topics: list[str]
    license: Optional[str] = None

result = app.scrape_url(
    "https://github.com/mendableai/firecrawl",
    params={
        "formats": ["json"],
        "jsonOptions": {
            "schema": Repository.model_json_schema(),
        },
    },
)

repo = result["json"]
print(f"{repo['name']}: {repo['stars']} stars ({repo['language']})")

Use Cases

Use Case	Method	Notes
Tech stack research	Search + JSON Scrape	Find and analyze repos by technology
Documentation ingestion	Crawl	Feed docs into RAG / knowledge base
Competitor analysis	Batch Scrape with repo schema	Compare star counts, activity, features
Issue tracking	Scrape with issue schema	Monitor issues across multiple repos
README collection	Batch Scrape as markdown	Aggregate README content for analysis

TIP

GitHub rate-limits unauthenticated requests. Firecrawl handles rendering and retries, but for best results when scraping many pages, use Batch Scrape instead of sequential individual scrapes.

Resources

• Firecrawl Scrape Docs
• Firecrawl Search Docs
• Firecrawl Batch Scrape