Firecrawl MCP Server

Big thanks to @vrknetha, @cawstudios for the initial implementation!

You can also play around with our MCP Server on MCP.so's playground or on Klavis AI. Thanks to MCP.so and Klavis AI for hosting and @gstarwd and @xiangkaiz for integrating our server.

Features

• Scrape, crawl, search, extract, deep research and batch scrape support
• Web scraping with JS rendering
• URL discovery and crawling
• Web search with content extraction
• Automatic retries with exponential backoff
  • Efficient batch processing with built-in rate limiting
• Credit usage monitoring for cloud API
• Comprehensive logging system
• Support for cloud and self-hosted Firecrawl instances
• Mobile/Desktop viewport support
• Smart content filtering with tag inclusion/exclusion

Installation

Running with npx

env FIRECRAWL_API_KEY=fc-YOUR_API_KEY npx -y firecrawl-mcp

Manual Installation

npm install -g firecrawl-mcp

Running on Cursor

1. Open Cursor Settings
2. Go to Features > MCP Servers
3. Click "+ Add New MCP Server"
4. Enter the following:
   • Name: "firecrawl-mcp" (or your preferred name)
   • Type: "command"
   • Command:

     env FIRECRAWL_API_KEY=your-api-key npx -y firecrawl-mcp

1. Open Cursor Settings
2. Go to Features > MCP Servers
3. Click "+ Add new global MCP server"
4. Enter the following code:

   {
     "mcpServers": {
       "firecrawl-mcp": {
         "command": "npx",
         "args": ["-y", "firecrawl-mcp"],
         "env": {
           "FIRECRAWL_API_KEY": "YOUR-API-KEY"
         }
       }
     }
   }
   

If you are using Windows and are running into issues, try

cmd /c "set FIRECRAWL_API_KEY=your-api-key && npx -y firecrawl-mcp"

your-api-key

Running on Windsurf

./codeium/windsurf/model_config.json

{
  "mcpServers": {
    "mcp-server-firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Running with SSE Local Mode

env SSE_LOCAL=true FIRECRAWL_API_KEY=fc-YOUR_API_KEY npx -y firecrawl-mcp

Installing via Smithery (Legacy)

npx -y @smithery/cli install @mendableai/mcp-server-firecrawl --client claude

Configuration

Environment Variables

Required for Cloud API

• FIRECRAWL_API_KEY

  : Your Firecrawl API key
  • Required when using cloud API (default)
  • Optional when using self-hosted instance with

    FIRECRAWL_API_URL

• FIRECRAWL_API_URL

  (Optional): Custom API endpoint for self-hosted instances
  • Example:

    https://firecrawl.your-domain.com

  • If not provided, the cloud API will be used (requires API key)

Optional Configuration

Retry Configuration

• FIRECRAWL_RETRY_MAX_ATTEMPTS

  : Maximum number of retry attempts (default: 3)

• FIRECRAWL_RETRY_INITIAL_DELAY

  : Initial delay in milliseconds before first retry (default: 1000)

• FIRECRAWL_RETRY_MAX_DELAY

  : Maximum delay in milliseconds between retries (default: 10000)

• FIRECRAWL_RETRY_BACKOFF_FACTOR

  : Exponential backoff multiplier (default: 2)

Credit Usage Monitoring

• FIRECRAWL_CREDIT_WARNING_THRESHOLD

  : Credit usage warning threshold (default: 1000)

• FIRECRAWL_CREDIT_CRITICAL_THRESHOLD

  : Credit usage critical threshold (default: 100)

Configuration Examples

# Required for cloud API
export FIRECRAWL_API_KEY=your-api-key

# Optional retry configuration
export FIRECRAWL_RETRY_MAX_ATTEMPTS=5        # Increase max retry attempts
export FIRECRAWL_RETRY_INITIAL_DELAY=2000    # Start with 2s delay
export FIRECRAWL_RETRY_MAX_DELAY=30000       # Maximum 30s delay
export FIRECRAWL_RETRY_BACKOFF_FACTOR=3      # More aggressive backoff

# Optional credit monitoring
export FIRECRAWL_CREDIT_WARNING_THRESHOLD=2000    # Warning at 2000 credits
export FIRECRAWL_CREDIT_CRITICAL_THRESHOLD=500    # Critical at 500 credits

# Required for self-hosted
export FIRECRAWL_API_URL=https://firecrawl.your-domain.com

# Optional authentication for self-hosted
export FIRECRAWL_API_KEY=your-api-key  # If your instance requires auth

# Custom retry configuration
export FIRECRAWL_RETRY_MAX_ATTEMPTS=10
export FIRECRAWL_RETRY_INITIAL_DELAY=500     # Start with faster retries

Usage with Claude Desktop

claude_desktop_config.json

{
  "mcpServers": {
    "mcp-server-firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "YOUR_API_KEY_HERE",

        "FIRECRAWL_RETRY_MAX_ATTEMPTS": "5",
        "FIRECRAWL_RETRY_INITIAL_DELAY": "2000",
        "FIRECRAWL_RETRY_MAX_DELAY": "30000",
        "FIRECRAWL_RETRY_BACKOFF_FACTOR": "3",

        "FIRECRAWL_CREDIT_WARNING_THRESHOLD": "2000",
        "FIRECRAWL_CREDIT_CRITICAL_THRESHOLD": "500"
      }
    }
  }
}

System Configuration

const CONFIG = {
  retry: {
    maxAttempts: 3, // Number of retry attempts for rate-limited requests
    initialDelay: 1000, // Initial delay before first retry (in milliseconds)
    maxDelay: 10000, // Maximum delay between retries (in milliseconds)
    backoffFactor: 2, // Multiplier for exponential backoff
  },
  credit: {
    warningThreshold: 1000, // Warn when credit usage reaches this level
    criticalThreshold: 100, // Critical alert when credit usage reaches this level
  },
};

1. Retry Behavior
   • Automatically retries failed requests due to rate limits
   • Uses exponential backoff to avoid overwhelming the API
   • Example: With default settings, retries will be attempted at:
     • 1st retry: 1 second delay
     • 2nd retry: 2 seconds delay
     • 3rd retry: 4 seconds delay (capped at maxDelay)
2. Credit Usage Monitoring
   • Tracks API credit consumption for cloud API usage
   • Provides warnings at specified thresholds
   • Helps prevent unexpected service interruption
   • Example: With default settings:
     • Warning at 1000 credits remaining
     • Critical alert at 100 credits remaining

Rate Limiting and Batch Processing

• Automatic rate limit handling with exponential backoff
• Efficient parallel processing for batch operations
• Smart request queuing and throttling
• Automatic retries for transient errors

Available Tools

1. Scrape Tool (

firecrawl_scrape

)

{
  "name": "firecrawl_scrape",
  "arguments": {
    "url": "https://example.com",
    "formats": ["markdown"],
    "onlyMainContent": true,
    "waitFor": 1000,
    "timeout": 30000,
    "mobile": false,
    "includeTags": ["article", "main"],
    "excludeTags": ["nav", "footer"],
    "skipTlsVerification": false
  }
}

2. Batch Scrape Tool (

firecrawl_batch_scrape

)

{
  "name": "firecrawl_batch_scrape",
  "arguments": {
    "urls": ["https://example1.com", "https://example2.com"],
    "options": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }
}

{
  "content": [
    {
      "type": "text",
      "text": "Batch operation queued with ID: batch_1. Use firecrawl_check_batch_status to check progress."
    }
  ],
  "isError": false
}

3. Check Batch Status (

firecrawl_check_batch_status

)

{
  "name": "firecrawl_check_batch_status",
  "arguments": {
    "id": "batch_1"
  }
}

4. Search Tool (

firecrawl_search

)

{
  "name": "firecrawl_search",
  "arguments": {
    "query": "your search query",
    "limit": 5,
    "lang": "en",
    "country": "us",
    "scrapeOptions": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }
}

5. Crawl Tool (

firecrawl_crawl

)

{
  "name": "firecrawl_crawl",
  "arguments": {
    "url": "https://example.com",
    "maxDepth": 2,
    "limit": 100,
    "allowExternalLinks": false,
    "deduplicateSimilarURLs": true
  }
}

6. Extract Tool (

firecrawl_extract

)

{
  "name": "firecrawl_extract",
  "arguments": {
    "urls": ["https://example.com/page1", "https://example.com/page2"],
    "prompt": "Extract product information including name, price, and description",
    "systemPrompt": "You are a helpful assistant that extracts product information",
    "schema": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "description": { "type": "string" }
      },
      "required": ["name", "price"]
    },
    "allowExternalLinks": false,
    "enableWebSearch": false,
    "includeSubdomains": false
  }
}

{
  "content": [
    {
      "type": "text",
      "text": {
        "name": "Example Product",
        "price": 99.99,
        "description": "This is an example product description"
      }
    }
  ],
  "isError": false
}

Extract Tool Options:

• urls

  : Array of URLs to extract information from

• prompt

  : Custom prompt for the LLM extraction

• systemPrompt

  : System prompt to guide the LLM

• schema

  : JSON schema for structured data extraction

• allowExternalLinks

  : Allow extraction from external links

• enableWebSearch

  : Enable web search for additional context

• includeSubdomains

  : Include subdomains in extraction

7. Deep Research Tool (firecrawl_deep_research)

{
  "name": "firecrawl_deep_research",
  "arguments": {
    "query": "how does carbon capture technology work?",
    "maxDepth": 3,
    "timeLimit": 120,
    "maxUrls": 50
  }
}

• query (string, required): The research question or topic to explore.
• maxDepth (number, optional): Maximum recursive depth for crawling/search (default: 3).
• timeLimit (number, optional): Time limit in seconds for the research session (default: 120).
• maxUrls (number, optional): Maximum number of URLs to analyze (default: 50).

• Final analysis generated by an LLM based on research. (data.finalAnalysis)
• May also include structured activities and sources used in the research process.

8. Generate LLMs.txt Tool (firecrawl_generate_llmstxt)

{
  "name": "firecrawl_generate_llmstxt",
  "arguments": {
    "url": "https://example.com",
    "maxUrls": 20,
    "showFullText": true
  }
}

• url (string, required): The base URL of the website to analyze.
• maxUrls (number, optional): Max number of URLs to include (default: 10).
• showFullText (boolean, optional): Whether to include llms-full.txt contents in the response.

• Generated llms.txt file contents and optionally the llms-full.txt (data.llmstxt and/or data.llmsfulltxt)

Logging System

• Operation status and progress
• Performance metrics
• Credit usage monitoring
• Rate limit tracking
• Error conditions

[INFO] Firecrawl MCP Server initialized successfully
[INFO] Starting scrape for URL: https://example.com
[INFO] Batch operation queued with ID: batch_1
[WARNING] Credit usage has reached warning threshold
[ERROR] Rate limit exceeded, retrying in 2s...

Error Handling

• Automatic retries for transient errors
• Rate limit handling with backoff
• Detailed error messages
• Credit usage warnings
• Network resilience

{
  "content": [
    {
      "type": "text",
      "text": "Error: Rate limit exceeded. Retrying in 2 seconds..."
    }
  ],
  "isError": true
}

Development

# Install dependencies
npm install

# Build
npm run build

# Run tests
npm test

Contributing

1. Fork the repository
2. Create your feature branch
3. Run tests:

   npm test

4. Submit a pull request

License