Scraping Wikipedia with Firecrawl

Updated Feb 2026

Firecrawl extracts clean, structured content from Wikipedia articles including infobox data, article text, references, and category information. This guide covers all extraction methods.

Installation

npm install @mendable/firecrawl-js zod

# Or with Python
pip install firecrawl-py

JSON Mode: Structured Article Extraction

Use a Zod schema to extract specific structured fields from Wikipedia articles. This is particularly useful for extracting infobox data.

Example: Programming Language Infobox

import FirecrawlApp from "@mendable/firecrawl-js";
import { z } from "zod";

const app = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY });

const languageSchema = z.object({
  name: z.string().describe("Programming language name"),
  paradigm: z.string().describe("Programming paradigm(s)"),
  designedBy: z.string().describe("Creator / designer"),
  firstAppeared: z.string().describe("Year of first appearance"),
  typingDiscipline: z.string().describe("Typing discipline (static, dynamic, etc.)"),
  stableRelease: z.string().optional().describe("Latest stable release version"),
  website: z.string().optional().describe("Official website URL"),
  influencedBy: z.array(z.string()).optional().describe("Languages that influenced this one"),
  influenced: z.array(z.string()).optional().describe("Languages influenced by this one"),
});

const result = await app.scrapeUrl(
  "https://en.wikipedia.org/wiki/Python_(programming_language)",
  {
    formats: ["json"],
    jsonOptions: {
      schema: languageSchema,
    },
  }
);

console.log(result.json);
// {
//   name: "Python",
//   paradigm: "Multi-paradigm: object-oriented, procedural, functional, structured, reflective",
//   designedBy: "Guido van Rossum",
//   firstAppeared: "1991",
//   typingDiscipline: "Duck, dynamic, strong",
//   stableRelease: "3.13.2",
//   website: "https://www.python.org",
//   influencedBy: ["ABC", "C", "Haskell", "Lisp", "Modula-3"],
//   influenced: ["Boo", "Cobra", "Go", "JavaScript", "Julia", "Ruby", "Swift"]
// }

Example: Person Infobox

const personSchema = z.object({
  name: z.string().describe("Full name"),
  born: z.string().describe("Birth date and location"),
  nationality: z.string().optional().describe("Nationality"),
  occupation: z.string().describe("Occupation or profession"),
  knownFor: z.array(z.string()).describe("Notable achievements or associations"),
  awards: z.array(z.string()).optional().describe("Awards received"),
  education: z.string().optional().describe("Education background"),
  summary: z.string().describe("First paragraph summary of the article"),
});

const result = await app.scrapeUrl(
  "https://en.wikipedia.org/wiki/Alan_Turing",
  {
    formats: ["json"],
    jsonOptions: {
      schema: personSchema,
    },
  }
);

console.log(result.json);

Example: Company Infobox

const companySchema = z.object({
  name: z.string().describe("Company name"),
  industry: z.string().describe("Industry sector"),
  founded: z.string().describe("Founding date and location"),
  founders: z.array(z.string()).describe("Founder names"),
  headquarters: z.string().describe("Headquarters location"),
  revenue: z.string().optional().describe("Annual revenue"),
  employees: z.string().optional().describe("Number of employees"),
  products: z.array(z.string()).optional().describe("Key products or services"),
  website: z.string().optional().describe("Official website"),
});

const result = await app.scrapeUrl(
  "https://en.wikipedia.org/wiki/OpenAI",
  {
    formats: ["json"],
    jsonOptions: {
      schema: companySchema,
    },
  }
);

Search: Find Articles

Search Wikipedia programmatically using site-scoped queries.

const results = await app.search(
  "artificial intelligence history site:en.wikipedia.org",
  {
    limit: 10,
    scrapeOptions: {
      formats: ["markdown"],
    },
  }
);

for (const result of results.data) {
  console.log(`${result.title}`);
  console.log(`  ${result.url}\n`);
}

Scrape: Full Article as Markdown

Extract a complete Wikipedia article as clean markdown -- ideal for feeding into LLMs or knowledge bases.

const result = await app.scrapeUrl(
  "https://en.wikipedia.org/wiki/Machine_learning",
  {
    formats: ["markdown"],
  }
);

console.log(result.markdown);
// Clean article text with headings, lists, and links preserved
// Wikipedia's sidebar, navigation, and footer elements are stripped

Extract with Main Content Only

const result = await app.scrapeUrl(
  "https://en.wikipedia.org/wiki/Machine_learning",
  {
    formats: ["markdown", "links"],
  }
);

console.log(`Content length: ${result.markdown.length} characters`);
console.log(`Links found: ${result.links.length}`);

Map: Discover Related URLs

Use Map to discover all URLs linked from a Wikipedia portal or category page.

const urls = await app.mapUrl(
  "https://en.wikipedia.org/wiki/Portal:Technology"
);

console.log(`Found ${urls.links.length} URLs`);

// Filter to article pages (exclude talk, user, special pages)
const articleUrls = urls.links.filter((url) =>
  url.includes("/wiki/") &&
  !url.includes(":") // Excludes namespaced pages like Talk:, User:, etc.
);

console.log(`Article URLs: ${articleUrls.length}`);

Crawl: Multi-Article Extraction

Crawl from a portal or category page to collect multiple related articles.

const crawlResult = await app.crawlUrl(
  "https://en.wikipedia.org/wiki/Portal:Science",
  {
    limit: 20,
    scrapeOptions: {
      formats: ["markdown"],
    },
  }
);

for (const page of crawlResult.data) {
  console.log(`Article: ${page.metadata.title}`);
  console.log(`URL: ${page.metadata.sourceURL}`);
  console.log(`Length: ${page.markdown.length} chars\n`);
}

Batch Scrape: Multiple Articles in Parallel

Process a list of known Wikipedia article URLs simultaneously.

const articleUrls = [
  "https://en.wikipedia.org/wiki/Artificial_intelligence",
  "https://en.wikipedia.org/wiki/Machine_learning",
  "https://en.wikipedia.org/wiki/Deep_learning",
  "https://en.wikipedia.org/wiki/Natural_language_processing",
  "https://en.wikipedia.org/wiki/Computer_vision",
];

const batchResult = await app.batchScrapeUrls(articleUrls, {
  formats: ["json"],
  jsonOptions: {
    schema: z.object({
      title: z.string(),
      summary: z.string().describe("First two paragraphs of the article"),
      keyTopics: z.array(z.string()).describe("Main topics covered"),
      seeAlso: z.array(z.string()).optional().describe("See Also section links"),
    }),
  },
});

for (const article of batchResult.data) {
  console.log(`${article.json.title}`);
  console.log(`  Topics: ${article.json.keyTopics.join(", ")}\n`);
}

Python Example

from firecrawl import FirecrawlApp
from pydantic import BaseModel
from typing import Optional

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

class WikiArticle(BaseModel):
    title: str
    summary: str
    key_topics: list[str]
    see_also: list[str] = []

result = app.scrape_url(
    "https://en.wikipedia.org/wiki/Web_scraping",
    params={
        "formats": ["json"],
        "jsonOptions": {
            "schema": WikiArticle.model_json_schema(),
        },
    },
)

article = result["json"]
print(f"{article['title']}")
print(f"Topics: {', '.join(article['key_topics'])}")

Use Cases

Use Case	Method	Notes
Knowledge graph construction	Batch Scrape + JSON schemas	Extract entities and relationships from infoboxes
Research automation	Search + Scrape	Find and summarize articles on a topic
RAG knowledge base	Crawl as markdown	Feed article content into vector databases
Multilingual content	Scrape different language wikis	en.wikipedia.org, fr.wikipedia.org, etc.
Entity extraction	JSON mode with entity schemas	Pull structured data from any article type
Educational resources	Batch Scrape + LLM summarization	Create study guides from article sets

TIP

Wikipedia articles are publicly available and generally easy to scrape. For best results with infobox extraction, tailor your Zod schema to the specific article type (person, company, place, language, etc.) since infobox fields vary by topic.

Resources

• Firecrawl Scrape Docs
• Firecrawl JSON Mode
• Firecrawl Batch Scrape