Welcome to v1 - A more reliable and developer friendly API

Firecrawl V1 is here! With that we introduce a more reliable and developer friendly API.

August 29th, 2024

Here is what’s new:

• Output Formats for /scrape. Choose what formats you want your output in.
• New /map endpoint for getting most of the URLs of a webpage.
• Developer friendly API for /crawl/{id} status.
• 2x Rate Limits for all plans.
• Go SDK and Rust SDK
• Teams support
• API Key Management in the dashboard.
• onlyMainContent is now default to true.
• /crawl webhooks and websocket support.

Learn more about it here

Start using v1 right away at https://firecrawl.dev

What's Changed (including v0 + v1)

• Delete .DS_Store by @szepeviktor in #8
• [Bugfix] added normalized apikey to craw/status route by @rafaelsideguide in #12
• [Feat] improving reative paths by @rafaelsideguide in #4
• Fix typos by @szepeviktor in #9
• [Feat] Added html to markdown table parser by @rafaelsideguide in #11
• Option to extract only the main content, excluding headers, navs, footers etc. by @nickscamara in #14
• [Feat] Adding pdf parser by @rafaelsideguide in #17
• adding ci-cd workflow by @rafaelsideguide in #20
• adding workflow by @rafaelsideguide in #21
• adding env secrets by @rafaelsideguide in #22
• [Feat] Added TSDocs and types for js-sdk by @rafaelsideguide in #28
• Added option to replace all relative paths with absolute paths by @rafaelsideguide in #25
• [Bugfix] Fixed scrape preview test by @rafaelsideguide in #30
• Caleb: fixing some documentation and rebuilding the server by @calebpeffer in #32
• Rate limit fixes for crawl status by @nickscamara in #36
• Better logging by @nickscamara in #35
• [Feat] Added type declarations by @rafaelsideguide in #31
• Refactor api routes by @nickscamara in #37
• Logging by @nickscamara in #38
• Cjp/making db auth optional <> Running project locally by @calebpeffer in #40
• chore: add context.close by @mattzcarey in #46
• Fixes table parsing for websites such as news.ycombinator.com (HN) by @nickscamara in #52
• [Feat] Server health check + slack message by @rafaelsideguide in #53
• [Feat] Added blocklist for social media urls by @rafaelsideguide in #55
• [Feat:mvp] Search Endpoint => serp api + firecrawl => 🔥 🔍 by @nickscamara in #56
• [Feat] Added anthropic vision api by @rafaelsideguide in #5
• [Bugfix] Trim and Lowercase all urls by @rafaelsideguide in #13
• Implements the ability for the crawler to output all the links it found, without scraping by @nickscamara in #34
• Serper params by @nickscamara in #62
• Support for tbs, filter, lang, country and location with Serper search. by @rogerserper in #61
• [Feat] Added allowed urls by @rafaelsideguide in #64
• /search support in node sdk by @nickscamara in #72
• Free credits increase by @nickscamara in #75
• [Bugfix] JS-SDK: Remove dotenv and add tests by @mdp in #68
• [Feat] Coupon system by @rafaelsideguide in #66
• Specific website params support by @nickscamara in #83
• Greenpay fixes by @nickscamara in #84
• [Feat] Implemented retry attempts to handle 502 errors by @rafaelsideguide in #67
• feat: LLM Extraction (mvp) by @nickscamara in #90
• Update README.md by @bllchmbrs in #110
• Add Posthog Logging by @ericciarla in #109
• Refactor of main web scraper + Partial data streaming by @nickscamara in #120
• [Feat] Added includeHTML option by @rafaelsideguide in #126
• Cancel Job Route by @nickscamara in #129
• [Feat] Added max depth option by @rafaelsideguide in #130
• Add keyAuth endpoint by @ericciarla in #131
• [Test] Added integration tests suite by @rafaelsideguide in #118
• Adds Zod Integration for LLM Extraction in the Firecrawl JS SDK by @nickscamara in #135
• [Docs] Updated examples by @rafaelsideguide in #137
• Switching to AGPL - We Need Your Consent! by @calebpeffer in #134
• Nsc/refactor scraping order by @nickscamara in #139
• Update models.ts by @ericciarla in #144
• Timeout on /scrape by @nickscamara in #145
• [Doc] Added default value for crawlOptions.limit by @rafaelsideguide in #142
• feat: 4x-5x faster crawler (fast mode) by @nickscamara in #149
• Add Docker Compose for easy self hosting by @chand1012 in #119
• refactor: fix typo in WebScraper/index.ts by @eltociear in #27
• [Tests] Added crawl test suite -> crawl improvements by @rafaelsideguide in #153
• feat: Docx Support by @nickscamara in #158
• Fixes pdfs not found if .pdf is not present by @nickscamara in #29
• Update README.md: Typo fix by @elimisteve in #160
• [Feat] Added rate limits by @rafaelsideguide in #151
• Allow override of API URL by @mattjoyce in #166
• feat: HyperDX Integration by @nickscamara in #167
• beta: Fire-Engine fallback by @nickscamara in #174
• Add additional file extensions to crawler.ts by @tractorjuice in #77
• [Bug] Fixing /crawl limit by @rafaelsideguide in #143
• Update issue templates by @rafaelsideguide in #180
• [Feat] Added proxy and media blocking support for Playwright by @JakobStadlhuber in #181
• update: wait until body attached in playwright-service by @qyou in #170
• feat: Allow privacy/legal/ other pages in social media websites by @nickscamara in #168
• [Bug] Added data check for python SDK by @rafaelsideguide in #176
• Fix FIRECRAWL_API_URL bug, also various PyLint fixes by @mattjoyce in #178
• [Feat] Added idempotency key to crawl route by @rafaelsideguide in #132
• Feat: Provide more details for 429 error msg by @simonha9 in #190
• Limit on /search is not deterministic by @Keredu in #186
• Various PyPi Metadata by @mattjoyce in #191
• [Test] Added sdk e2e tests by @rafaelsideguide in #183
• Allow users to manually set the waitFor param on /scrape by @nickscamara in #200
• [Feat] Added custom scraping conditions for readme docs by @rafaelsideguide in #204
• Feat/screenshot support by @ericciarla in #207
• feat: New pricing/limits changes by @nickscamara in #216
• [sdk] Fixes waiting status not being present on check status by @nickscamara in #218
• Fixed fire-engine content bug by @rafaelsideguide in #228
• Use @ instead of # for default BULL_AUTH_KEY. Hash mark is reserved for URI fragments. by @rombru in #229
• feat: Ability to forward headers to reliable providers for auth etc... by @nickscamara in #221
• Improvements to the blocklist regex by @nickscamara in #185
• Playwright service bugs #222 #179 #197 by @mattjoyce in #224
• [Fix] Changed timeout parameter name on js sdk by @rafaelsideguide in #211
• [Bug] Improved js response and test for getting partial_data by @rafaelsideguide in #212
• Better fallbacks for initial crawl start by @nickscamara in #148
• [Feat] Added custom scraping for google-drive pdf usecase by @rafaelsideguide in #235
• Script to check local vs published versions of sdk packages by @mattjoyce in #213
• Update README.md by @calebpeffer in #140
• Refactor custom scraping and added security center vanta by @nickscamara in #237
• [Feat] Added scroll xpaths on fire-engine for handling readme docs by @rafaelsideguide in #239
• Add Kubernetes configuration for Firecrawl deployment by @JakobStadlhuber in #236
• [Feat] Improved the scrape for gdrive pdfs by @rafaelsideguide in #238
• Add support for Self-Hosted Webhook URL Usage and added project_id into the webhook payload by @JakobStadlhuber in #234
• Partial Data Sliding window of 50 by @nickscamara in #242
• feat: Transactional emails for rate limits and credit limits by @nickscamara in #244
• Python-SDK transitional build setup for pyproject.toml by @mattjoyce in #196
• [Feat] CI/CD for publishing js and python SDKs by @rafaelsideguide in #246
• 194 sdk ci pipeline for publishing pythonnode sdk by @rafaelsideguide in #248
• fixing minor problems on workflow by @rafaelsideguide in #250
• Fix 208 py sdk interval poll name by @mattjoyce in #252
• Moving from fetch to axios and preventing deadlocks by setting timeouts on fallbacks by @nickscamara in #260
• ignoreSitemap feature, pageOptions now respected in the initial crawl as well by @nickscamara in #263
• Fixed bugs associated with absolute path replacements by @nickscamara in #268
• Only fetch webhook from db if self host webhook not set and using db auth by @nickscamara in #262
• [Feat] Added route to clean completed jobs and a github action cron that triggers every 24h by @rafaelsideguide in #265
• [Feat] Added allowBackwardCrawling option by @rafaelsideguide in #269
• [Feat] Added jobId to webhook data by @rafaelsideguide in #272
• Clusters support to scale our API to # of CPUs running by @nickscamara in #274
• Added pageOptions.removeTags by @rafaelsideguide in #275
• Added logging to python sdk FIRECRAWL_LOGGING_LEVEL by @mattjoyce in #256
• Python sdk improve response handling by @mattjoyce in #253
• [Feat] Added parsePDF option to pageOptions by @rafaelsideguide in #271
• [Feat] Added metadata.pageStatusCode and metadata.pageError properties by @rafaelsideguide in #276
• Make maxDepth relative to the entered url by @ericciarla in #277
• Test/load testing by @rafaelsideguide in #175
• Fixes crawler getting confused with base paths that contain www. by @nickscamara in #283
• Feat/max depth relative by @ericciarla in #285
• [Feat] Improvements on response document types by @rafaelsideguide in #298
• test: Rate Limit Unit Tests by @nickscamara in #282
• Added local host support for the javascript SDK by @NeevJewalkar in #296
• [Test] Transcribed from e2e to unit tests for many cases by @rafaelsideguide in #294
• [Feat] Added support for RegEx in removeTags by @AndyMik90 in #297
• [Bug] Fixed includeHTML to use cleanedHtml as response by @rafaelsideguide in #301
• Cjp/email to posthog logging by @calebpeffer in #302
• Fix Broken Link by @Lakr233 in #303
• docs: Fix pydanti to pydantic by @100gle in #308
• [Bug] Fixed axios bug that were making jobs stuck on active queue by @rafaelsideguide in #312
• fix multi-word search term issue: /search (w/o Serp) by @snippet in #313
• [Bug] Fixed clean jobs by @rafaelsideguide in #317
• [Bug] Fixed the regex test for google drive pdf files by @rafaelsideguide in #318
• add some types by @snippet in #316
• [Bug] Added default values and fixed pdf bug by @rafaelsideguide in #321
• [Test] Added E2E tests for checking metadata values by @rafaelsideguide in #322
• pageOptions.onlyIncludeTags param by @nickscamara in #328
• Update CONTRIBUTING.md by @george-zakharov in #329
• Fix headers | pageOptions not being passed correctly to fire engine by @nickscamara in #331
• [Tests] Added crawl limit unit test by @rafaelsideguide in #323
• Adds pageOptions.includeRawHtml and new extraction mode "llm-extraction-from-raw-html" by @ericciarla in #332
• [Proposal] new feature allowExternalContentLinks by @snippet in #336
• [PROPOSAL] (deps): making sure all deps are always up to date by @Sanix-Darker in #338
• [PROPOSAL] (docker-compose) regroup envs vars between services by @Sanix-Darker in #337
• new playwright service by @snippet in #325
• apps/playwright-service(deps): bump the prod-deps group in /apps/playwright-service with 3 updates by @dependabot in #346
• Logging for all scraper methods by @nickscamara in #363
• setting up docker to ts playwright service by @snippet in #361
• (Docs) Self Host added new ts playwright service instructions by @snippet in #362
• apps/test-suite(deps): bump the prod-deps group in /apps/test-suite with 6 updates by @dependabot in #347
• apps/test-suite(deps-dev): bump typescript from 5.4.5 to 5.5.3 in /apps/test-suite in the dev-deps group by @dependabot in #348
• apps/api(deps): bump the prod-deps group in /apps/api with 28 updates by @dependabot in #349
• [Feat] Added veeva to special case params by @rafaelsideguide in #371
• Only check Supabase if configured to when cancelling job by @StefanTerdell in #374
• dependabot for security checks, fixed crawl test by @rafaelsideguide in #370
• [Feat] Added implementation for saving docs on supabase by @rafaelsideguide in #326
• Self-hosting quality of life fixes by @StefanTerdell in #375
• [Feat] Added fire-engine fallback for getting sitemaps by @rafaelsideguide in #386
• fix(bull): requeue jobs after restart by @mogery in #393
• Fix USE_DB_AUTHENTICATION checks for self-host by @kun432 in #395
• [BUG] Fixed docker self hosting issue by @rafaelsideguide in #396
• Slack Alerts for when queue is over capacity by @nickscamara in #397
• CI for alerts instead by @nickscamara in #398
• Log Fire-engine page errors by @nickscamara in #394
• Sitemap fallback fixes w/ fire-engine by @nickscamara in #399
• Separate Rate Limiter from Main Redis by @nickscamara in #417
• [Feat] Pass along current, total, current_step, and current_url in js sdk by @jhoseph88 in #391
• Redis Health Checks by @nickscamara in #424
• Small fix for empty pageOptions by @rafaelsideguide in #420
• Caleb: Return a list of links on a page by default by @calebpeffer in #423
• [Docs] Updating docs by @rafaelsideguide in #427
• Fix queue stuck bug via lock settings changes by @mogery in #429
• fix(js-sdk): transform tests with ts-jest and configure node by @mogery in #433
• fix(WebScraper): infinite regex leading to fly.io instance hangs by @mogery in #436
• Support chrome-cdp and restructure sitemap fire-engine support. by @tomkosm in #410
• /scrape should now be 600ms-900ms faster by @nickscamara in #447
• Notion Website Fixes by @nickscamara in #453
• Firecrawl UI template by @ericciarla in #458
• Added regex for links in sitemap by @rafaelsideguide in #449
• Added logger by @rafaelsideguide in #450
• Add scrape monitoring by @mogery in #455
• Readiness liveness probes by @JakobStadlhuber in #457
• Client side error handling by @nickscamara in #461
• Admin router + Improve redis notifications by @nickscamara in #460
• support custom models by @NiuBlibing in #414
• DNS Cacheable Lookup to avoid DNS blocking ops by @nickscamara in #478
• Fire-engine to be the default for our cloud-service by @nickscamara in #479
• [Bug] Rate limiter bug on switch for some cases by @rafaelsideguide in #482
• [Feat] Better self hosting guide by @rafaelsideguide in #483
• Fix parallelization issues with caching by @nickscamara in #484
• [Bug] Issue with crawl going beyond Limit by @rafaelsideguide in #485
• [Bug] pdfs and logging pdf events, also added trycatchs for docx by @rafaelsideguide in #475
• [Bug] Nested sitemaps by @rafaelsideguide in #491
• Fix AMZN | Removal of redis alerts by @nickscamara in #492
• Custom engine params fix by @nickscamara in #494
• [Bug] Fixed the empty excludes.filter is undefined bug by @rafaelsideguide in #503
• [Feat] Added fullpagescreenshot capabilities by @rafaelsideguide in #504
• fix(js-sdk): build both CommonJS and ESM versions by @mogery in #432
• fix(js-sdk): build both CJS and EJS versions by @mogery in #508
• Improve logs by @tak-s in #496
• Digikey support by @nickscamara in #512
• [Feat] Add Go SDK implementation by @KentHsu in #497
• Redlock cache in auth by @nickscamara in #529
• Update redis urls in example .env by @wahpiangle in #515
• Self-host fix: Moving comments of .env.example values from end-of-line to above-line. by @kevinswiber in #531
• fix: go-sdk module name by @KentHsu in #534
• Fixed e2e tests by @rafaelsideguide in #536
• Removed obsoleted declaration by @matsubo in #532
• [Feat] Added go-sdk as submodule by @rafaelsideguide in #537
• feat: Move scraper to queue by @mogery in #459
• Reduce metrics ingestion w/ HyperDX v0.8.1 by @nickscamara in #480
• Check team credits based on the crawl limit by @nickscamara in #554
• added variables to beta customers by @rafaelsideguide in #555
• Set the crawl limit to the remaining credits by @nickscamara in #559
• [Feat] Added check job and cancel to fire-engine requests by @rafaelsideguide in #560
• [Bug] Added a way for dealing with DNS without IP resolution by @rafaelsideguide in #561
• Added Sentry Monitoring by @nickscamara in #562
• Internal Concurrency Limits <> Job Priorities by @nickscamara in #566
• [V1] Release by @rafaelsideguide in #527
• [Feat]: Add RUST SDK client for firecrawl API by @Sanix-Darker in #373
• Update README.md by @ericciarla in #590
• [Bug] Moved delete rawHtml to end of controller by @rafaelsideguide in #582
• Fixed RPC cloudflare issues with job ids by @nickscamara in #593
• [v1] Websockets SDKs by @rafaelsideguide in #591
• [v1] LLM Extract by @nickscamara in #586
• [v1] Webhooks by @rafaelsideguide in #594
• fix(v1): maxDepth by @rafaelsideguide in #607
• Ensuring USE_DB_AUTHENTICATION is true in single URL scraper. by @kevinswiber in #516
• Bill team async by @nickscamara in #619
• Optimize check credits func w/ caching by @nickscamara in #624
• Feat: Added go html to md parser by @rafaelsideguide in #608
• Feat: parser singleton by @rafaelsideguide in #629

New Contributors

• @szepeviktor made their first contribution in #8
• @rafaelsideguide made their first contribution in #12
• @nickscamara made their first contribution in #14
• @calebpeffer made their first contribution in #32
• @mattzcarey made their first contribution in #46
• @rogerserper made their first contribution in #61
• @mdp made their first contribution in #68
• @bllchmbrs made their first contribution in #110
• @ericciarla made their first contribution in #109
• @chand1012 made their first contribution in #119
• @eltociear made their first contribution in #27
• @elimisteve made their first contribution in #160
• @mattjoyce made their first contribution in #166
• @tractorjuice made their first contribution in #77
• @JakobStadlhuber made their first contribution in #181
• @qyou made their first contribution in #170
• @simonha9 made their first contribution in #190
• @Keredu made their first contribution in #186
• @rombru made their first contribution in #229
• @NeevJewalkar made their first contribution in #296
• @AndyMik90 made their first contribution in #297
• @Lakr233 made their first contribution in #303
• @100gle made their first contribution in #308
• @snippet made their first contribution in #313
• @george-zakharov made their first contribution in #329
• @Sanix-Darker made their first contribution in #338
• @dependabot made their first contribution in #346
• @StefanTerdell made their first contribution in #374
• @mogery made their first contribution in #393
• @kun432 made their first contribution in #395
• @jhoseph88 made their first contribution in #391
• @tomkosm made their first contribution in #410
• @NiuBlibing made their first contribution in #414
• @tak-s made their first contribution in #496
• @KentHsu made their first contribution in #497
• @wahpiangle made their first contribution in #515
• @kevinswiber made their first contribution in #531
• @matsubo made their first contribution in #532

Full Changelog: https://github.com/mendableai/firecrawl/commits/v1.0.0