U h@sLddlmZmZmZddlmZddlmZddlm Z GdddeZ dS))IteratorLiteralOptional) BaseLoader)Document) get_from_envc@sTeZdZdZdddddeeeeeedeedddZe e d d d Z dS) FireCrawlLoadera" FireCrawlLoader document loader integration Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``. .. code-block:: bash pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key" Instantiate: .. code-block:: python from langchain_community.document_loaders import FireCrawlLoader loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... ) Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} Ncrawl)api_keyapi_urlmodeparamsr scrape)urlr r r r cCsxzddlm}Wntk r,tdYnX|dkrFtd|d|pRtdd}|||d |_||_||_||_d S) aInitialize with API key and url. Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url) and "crawl" (all accessible sub pages). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py r) FirecrawlAppzD`firecrawl` package not found, please run `pip install firecrawl-py`rUnrecognized mode '%'. Expected one of 'crawl', 'scrape'.r ZFIRECRAWL_API_KEY)r r N) firecrawlr ImportError ValueErrorrrr r )selfrr r r r rrR/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/document_loaders/firecrawl.py__init__Cs  zFireCrawlLoader.__init__)returnccs|jdkr"|jj|j|jdg}n2|jdkrB|jj|j|jd}ntd|jd|D]V}|di}|jdk r|jdidd kr|d |d <t|d d |d VqXdS)Nr)r r rrmetadataZextractorOptionsr zllm-extractionZllm_extractionZmarkdown)Z page_contentr) r rZ scrape_urlrr Z crawl_urlrgetr)rZfirecrawl_docsdocrrrr lazy_loadls&    zFireCrawlLoader.lazy_load) __name__ __module__ __qualname____doc__strrrdictrrrr rrrrrs> )rN) typingrrrZlangchain_core.document_loadersrZlangchain_core.documentsrZlangchain_core.utilsrrrrrrs