U h@sfddlZddlmZmZmZmZmZddlmZddl m Z ddl m Z e eZGddde ZdS)N)AnyIteratorListOptionalSequence)Document) BaseLoader) NewsURLLoaderc@speZdZdZdeeeeeeeeddddZ e e dd d Z e eedd d Zee dd dZdS) RSSFeedLoaderaQLoad news articles from `RSS` feeds using `Unstructured`. Args: urls: URLs for RSS feeds to load. Each articles in the feed is loaded into its own document. opml: OPML file to load feed urls from. Only one of urls or opml should be provided. The value can be a URL string, or OPML markup contents as byte or string. continue_on_failure: If True, continue loading documents even if loading fails for a particular URL. show_progress_bar: If True, use tqdm to show a loading progress bar. Requires tqdm to be installed, ``pip install tqdm``. **newsloader_kwargs: Any additional named arguments to pass to NewsURLLoader. Example: .. code-block:: python from langchain_community.document_loaders import RSSFeedLoader loader = RSSFeedLoader( urls=["", ""], ) docs = loader.load() The loader uses feedparser to parse RSS feeds. The feedparser library is not installed by default so you should install it if using this loader: https://pythonhosted.org/feedparser/ If you use OPML, you should also install listparser: https://pythonhosted.org/listparser/ Finally, newspaper is used to process each article: https://newspaper.readthedocs.io/en/latest/ NTF)urlsopmlcontinue_on_failureshow_progress_barnewsloader_kwargsreturncKs:|dk|dkkrtd||_||_||_||_||_dS)zInitialize with urls or OPML.Nz;Provide either the urls or the opml argument, but not both.) ValueErrorr r r rr)selfr r r rrrL/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/document_loaders/rss.py__init__/s zRSSFeedLoader.__init__)rc Cs\|}|jrTzddlm}Wn,tk rJ}ztd|W5d}~XYnX||}t|S)Nr)tqdmzPackage tqdm must be installed if show_progress_bar=True. Please install with 'pip install tqdm' or set show_progress_bar=False.) lazy_loadrr ImportErrorlist)riterrerrrloadDszRSSFeedLoader.loadc Csb|jr |jSz ddl}Wn,tk rD}ztd|W5d}~XYnX||j}dd|jDS)NrzPackage listparser must be installed if the opml arg is used. Please install with 'pip install listparser' or use the urls arg instead.cSsg|] }|jqSr)url).0feedrrr _sz+RSSFeedLoader._get_urls..)r listparserrparser Zfeeds)rr!rZrssrrr _get_urlsRs  zRSSFeedLoader._get_urlsc csRz ddl}Wntk r(tdYnX|jD]}z0||}t|ddrdtd|d|jWnLtk r}z.|jrt d|d|WY q0n|W5d}~XYnXzD|j D]8}t fd|j gi|j}|d}||jd<|VqWq0tk rJ}z2|jr6t d |j d|WY q0n|W5d}~XYq0Xq0dS) NrzMfeedparser package not found, please install it with `pip install feedparser`ZbozoFzError fetching z , exception: r rzError processing entry ) feedparserrr#r"getattrrZbozo_exception Exceptionr loggererrorentriesr linkrrmetadata)rr$rrrentryloaderarticlerrrrasB           zRSSFeedLoader.lazy_load)NNTF)__name__ __module__ __qualname____doc__rrstrboolrrrrrpropertyr#rrrrrrr s"$  r )loggingtypingrrrrrZlangchain_core.documentsrZ)langchain_community.document_loaders.baserZ)langchain_community.document_loaders.newsr getLoggerr/r'r rrrrs