U h\H@s"dZddlmZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZeeZd gZ eGdddZ!ddhZ"ddddddhZ#dddddZ$GdddeZ%Gdd d eZ&eGd!d"d"eZ'dS)#zLoads YouTube transcript.) annotationsN)Enum)Path)AnyDict GeneratorListOptionalSequenceUnion)parse_qsurlparse) ParseError)Document)root_validator) dataclass) BaseLoaderz0https://www.googleapis.com/auth/youtube.readonlyc@seZdZUdZeddZded<eddZded<eddZ ded<d d d d Z e d ddddddZ dd ddZ dS)GoogleApiClientaGeneric Google API Client. To use, you should have the ``google_auth_oauthlib,youtube_transcript_api,google`` python package installed. As the google api expects credentials you need to set up a google account and register your Service. "https://developers.google.com/docs/api/quickstart/python" *Security Note*: Note that parsing of the transcripts relies on the standard xml library but the input is viewed as trusted in this case. Example: .. code-block:: python from langchain_community.document_loaders import GoogleApiClient google_api_client = GoogleApiClient( service_account_path=Path("path_to_your_sec_file.json") ) z .credentialszcredentials.jsonrcredentials_pathservice_account_pathz token.json token_pathNonereturncCs||_dSN)_load_credentialscredsselfrP/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/document_loaders/youtube.py __post_init__2szGoogleApiClient.__post_init__TpreDict[str, Any]valuesrcCs |ds|dstd|S)DValidate that either folder_id or document_ids is set, but not both.rr-Must specify either channel_name or video_idsget ValueErrorclsr&rrr #validate_channel_or_videoIds_is_set5s z3GoogleApiClient.validate_channel_or_videoIds_is_setrc Cs z@ddlm}ddlm}ddlm}ddlm}ddlm }Wnt k r\t dYnXd}|j r~|j t|j S|j r|t|jt}|r|js|r|jr|jr||n|t|jt}|jdd }t|jd }||W5QRX|S) zLoad credentials.r)Request)service_account) Credentials)InstalledAppFlowYouTubeTranscriptApiYou must run`pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib youtube-transcript-api` to use the Google Drive loaderN)portw)Zgoogle.auth.transport.requestsr/Z google.oauth2r0Zgoogle.oauth2.credentialsr1Zgoogle_auth_oauthlib.flowr2youtube_transcript_apir4 ImportErrorrexistsZfrom_service_account_filestrrZfrom_authorized_user_fileSCOPESZvalidZexpiredZ refresh_tokenZrefreshZfrom_client_secrets_filerZrun_local_serveropenwriteZto_json) rr/r0r1r2r4rZflowtokenrrr rAs8         z!GoogleApiClient._load_credentialsN)__name__ __module__ __qualname____doc__rhomer__annotations__rrr!rr.rrrrr rs  rhttphttpszyoutu.bez m.youtube.comz youtube.comzwww.youtube.comzwww.youtube-nocookie.comzvid.plusr; Optional[str])urlrcCst|}|jtkrdS|jtkr$dS|j}|drp|j}t|}d|krj|d}t |t r`|n|d}qdSn|j d}| dd}t |dkrdS|S)zEParse a YouTube URL and return the video ID if valid, otherwise None.Nz/watchvr/ )r schemeALLOWED_SCHEMESnetlocALLOWED_NETLOCSpathendswithqueryr isinstancer;lstripsplitlen)rI parsed_urlrRrTZ parsed_queryidsvideo_idrrr _parse_video_idus$     r\c@seZdZdZdZdZdZdS)TranscriptFormatz3Output formats of transcripts from `YoutubeLoader`.textlineschunksN)r@rArBrCTEXTLINESCHUNKSrrrr r]sr]c @seZdZdZdddejddfdddd d dd d d dZedddddZe ddddddZ dd ddddZ dddddZ dd d!d"Z d#d d$d%ZdS)& YoutubeLoaderz!Load `YouTube` video transcripts.FenNxr;boolzUnion[str, Sequence[str]]rHr]int)r[add_video_infolanguage translationtranscript_formatcontinue_on_failurechunk_size_secondscCsR||_d|i|_||_||_t|tr0|g|_n||_||_||_||_||_ dS)z!Initialize with YouTube video ID.sourceN) r[ _metadatarirjrUr;rkrlrmrn)rr[rirjrkrlrmrnrrr __init__s    zYoutubeLoader.__init__) youtube_urlrcCs t|}|std|d|S)z*Extract video ID from common YouTube URLs.z.Could not determine the video ID for the URL "z".)r\r+)rrr[rrr extract_video_ids  zYoutubeLoader.extract_video_idr)rrkwargsrcKs||}||f|S)z|Given a YouTube URL, construct a loader. See `YoutubeLoader()` constructor for a list of keyword arguments. )rs)r-rrrtr[rrr from_youtube_urls zYoutubeLoader.from_youtube_urlz List[Dict]r) chunk_pieceschunk_start_secondsrc Cslt|d\}}t|d\}}tdtdd||j||dd|dd|dd|jd|d d d S) z0Create Document from chunk of transcript pieces.< cSs|ddSNr^rystrip)Z chunk_piecerrr z4YoutubeLoader._make_chunk_document..Z02d: https://www.youtube.com/watch?v=z&t=s)Z start_secondsZstart_timestampro page_contentmetadata)divmodrjoinmaprpr[)rrvrwmrhrrr _make_chunk_documents z"YoutubeLoader._make_chunk_documentzGenerator[Document, None, None])transcript_piecesrccs|g}d}|j}|D]J}|d|d}||krR|r@|||Vg}|}||j7}||qt|dkrx|||VdS)Nrstartduration)rnrappendrX)rrrvrwZchunk_time_limittranscript_pieceZ piece_endrrr _get_transcript_chunkss   z$YoutubeLoader._get_transcript_chunksList[Document]rcCsDzddlm}m}m}Wntk r4tdYnX|jrP|}|j|z| |j }Wn|k rxgYSXz| |j }Wn |k r| dg}YnX|j dk r||j }|}|jtjkrdtdd|}t||jd gS|jtjkrttd d|S|jtjkr8t||Std dS) z1Load YouTube transcripts into `Document` objects.r)NoTranscriptFoundTranscriptsDisabledr4zvCould not import "youtube_transcript_api" Python package. Please install it with `pip install youtube-transcript-api`.reNrycSs|ddSrzr{rrrr r}r~z$YoutubeLoader.load..rcSs(t|ddttdd|dS)Nr^rycSs |ddkS)Nrr^r)itemrrr r}#r~z6YoutubeLoader.load....r)rr|dictfilteritemsrrrr r}s zUnknown transcript format.)r8rrr4r9ri_get_video_inforpupdatelist_transcriptsr[find_transcriptrjrk translatefetchrlr]rarrrrblistrcrr+)rrrr4 video_infotranscript_list transcriptrrrr loadsL      zYoutubeLoader.loadrcCszddlm}Wntk r,tdYnX|d|j}|jpFd|jpNd|jpVd|jp^d|jrp|j dnd|j pzd|j pdd}|S)zGet important video information. Components include: - title - description - thumbnail URL, - publish_date - channel author - and more. r)YouTubezVCould not import "pytube" Python package. Please install it with `pip install pytube`.rUnknownz%Y-%m-%d %H:%M:%S)title descriptionZ view_count thumbnail_url publish_datelengthauthor) Zpytuberr9r[rrZviewsrrstrftimerr)rrZytrrrr r0s$   zYoutubeLoader._get_video_info)r@rArBrCr]rarq staticmethodrs classmethodrurrrrrrrr rds  ?rdc@seZdZUdZded<dZded<dZded<d Zd ed <d Zd ed<dZ d ed<ddddZ dddddZ e d ddddddZ d d ddd Zd dd!d"d#d$Zd d d%d&d'Zd d d(d)d*Zd dd+d,d-d.Zd+dd/d0ZdS)1GoogleApiYoutubeLoaderaLoad all Videos from a `YouTube` Channel. To use, you should have the ``googleapiclient,youtube_transcript_api`` python package installed. As the service needs a google_api_client, you first have to initialize the GoogleApiClient. Additionally you have to either provide a channel name or a list of videoids "https://developers.google.com/docs/api/quickstart/python" Example: .. code-block:: python from langchain_community.document_loaders import GoogleApiClient from langchain_community.document_loaders import GoogleApiYoutubeLoader google_api_client = GoogleApiClient( service_account_path=Path("path_to_your_sec_file.json") ) loader = GoogleApiYoutubeLoader( google_api_client=google_api_client, channel_name = "CodeAesthetic" ) load.load() rgoogle_api_clientNrH channel_namezOptional[List[str]] video_idsTrgrirer;captions_languageFrmrrcCs||jj|_dSr)_build_youtube_clientrryoutube_clientrrrr r!wsz$GoogleApiYoutubeLoader.__post_init__r)rrcCsHzddlm}ddlm}Wntk r8tdYnX|dd|dS)Nr)buildr3r5ZyoutubeZv3) credentials)Zgoogleapiclient.discoveryrr8r4r9)rrrr4rrr rzs  z,GoogleApiYoutubeLoader._build_youtube_clientr"r$r%cCs |ds|dstd|S)r'rrr(r)r,rrr r.sz:GoogleApiYoutubeLoader.validate_channel_or_videoIds_is_set)r[rcCsvddlm}m}||}z||jg}Wn,|k rX|D]}||j}q@q@YnX|}ddd|DS)Nr)rr4rycSsg|]}|ddqS)r^ryr{).0trrr szGGoogleApiYoutubeLoader._get_transcripe_for_video_id..) r8rr4rrrrrr)rr[rr4rrZavailable_transcriptrrrr _get_transcripe_for_video_ids   z3GoogleApiYoutubeLoader._get_transcripe_for_video_idr)r[rtrcKs8||}|jjd|d}t||dddS)N id,snippetpartidrrr)rrZvideosrexecuterr*)rr[rtZcaptionsZvideo_responserrr _get_document_for_video_ids   z1GoogleApiYoutubeLoader._get_document_for_video_id)rrcCs8|jjd|ddd}|}|dddd}|S)Nrchannel)rqtype maxResultsrrZ channelId)rsearchrr)rrrequestresponse channel_idrrr _get_channel_ids z&GoogleApiYoutubeLoader._get_channel_id)rrcCs4|jjd|d}|}|dddddS)NZcontentDetailsrrrZrelatedPlaylistsZuploads)rZchannelsrr)rrrrrrr _get_uploads_playlist_ids  z/GoogleApiYoutubeLoader._get_uploads_playlist_idr)rrtrc KsNzddlm}m}Wntk r0tdYnX||}||}|jjd|dd}g}|dk rJ| } | dD]} | dd d } d | i} |j r| d d | | dz | | } |t| | d Wqz||tfk r2}z4|jrtd d| dd d|n|W5d}~XYqzXqz|j|| }q`|S)Nr)rrzTYou must run`pip install --upgrade youtube-transcript-api` to use the youtube loaderr2)rZ playlistIdrrZsnippetZ resourceIdZvideoIdZ thumbnailsrzError fetching transscript ryrz , exception: )r8rrr9rrrZ playlistItemsrrripoprrrrrrmloggererrorrZ list_next)rrrtrrrZuploads_playlist_idrrrrr[Z meta_datarerrr _get_document_for_channelsR       z0GoogleApiYoutubeLoader._get_document_for_channelcsLg}jr|jn*jr@|fddjDntd|S)zLoad documents.csg|]}|qSr)r)rr[rrr rsz/GoogleApiYoutubeLoader.load..r()rextendrrr+)rZ document_listrrr rs zGoogleApiYoutubeLoader.load)r@rArBrCrErrrirrmr!rrr.rrrrrrrrrr rRs"       5r)(rC __future__rloggingenumrpathlibrtypingrrrrr r r urllib.parser r xml.etree.ElementTreerZlangchain_core.documentsrZlangchain_core.pydantic_v1rZ&langchain_core.pydantic_v1.dataclassesrZ)langchain_community.document_loaders.baser getLoggerr@rr<rrOrQr\r]rdrrrrr s:   $      R 8