o Q^i @s0dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlZ ddl mZmZdZdZdaedZedZedd d Zd d Zd ddZd ddZd!ddZdddejd d d dd f ddZd!ddZd!ddZd!ddZ Gddde!Z"Gddde!Z#dS)"a Descriptor archives are available from `CollecTor `_. If you need Tor's topology at a prior point in time this is the place to go! With CollecTor you can either read descriptors directly... .. literalinclude:: /_static/example/collector_reading.py :language: python ... or download the descriptors to disk and read them later. .. literalinclude:: /_static/example/collector_caching.py :language: python :: get_instance - Provides a singleton CollecTor used for... |- get_server_descriptors - published server descriptors |- get_extrainfo_descriptors - published extrainfo descriptors |- get_microdescriptors - published microdescriptors |- get_consensus - published router status entries | |- get_key_certificates - authority key certificates |- get_bandwidth_files - bandwidth authority heuristics +- get_exit_lists - TorDNSEL exit list File - Individual file residing within CollecTor |- read - provides descriptors from this file +- download - download this file to disk CollecTor - Downloader for descriptors from CollecTor |- get_server_descriptors - published server descriptors |- get_extrainfo_descriptors - published extrainfo descriptors |- get_microdescriptors - published microdescriptors |- get_consensus - published router status entries | |- get_key_certificates - authority key certificates |- get_bandwidth_files - bandwidth authority heuristics |- get_exit_lists - TorDNSEL exit list | |- index - metadata for content available from CollecTor +- files - files available from CollecTor .. versionadded:: 1.8.0 N) CompressionDocumentHandlerz!https://collector.torproject.org/z-(\d{4})-(\d{2})\.z%(\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2})i'cCstdurtatS)z Provides the singleton :class:`~stem.descriptor.collector.CollecTor` used for this module's shorthand functions. :returns: singleton :class:`~stem.descriptor.collector.CollecTor` instance N)SINGLETON_COLLECTOR CollecTorrr;/usr/lib/python3/dist-packages/stem/descriptor/collector.py get_instancePs r Fcc(t||||||D]}|Vq dS)zv Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_server_descriptors` on our singleton instance. N)r get_server_descriptorsstartendcache_tobridgetimeoutretriesdescrrr r `r ccr )zy Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_extrainfo_descriptors` on our singleton instance. N)r get_extrainfo_descriptorsrrrr rkrrcc&t|||||D]}|Vq dS)zt Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_microdescriptors` on our singleton instance. N)r get_microdescriptorsrrrrrrrrr rvrc cs.t||||||||| D]} | VqdS)zm Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_consensus` on our singleton instance. N)r get_consensus) rrrdocument_handlerversionmicrodescriptorrrrrrrr rs rccr)zt Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_key_certificates` on our singleton instance. N)r get_key_certificatesrrrr r rr ccr)zs Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_bandwidth_files` on our singleton instance. N)r get_bandwidth_filesrrrr r!rr!ccr)zn Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_exit_lists` on our singleton instance. N)r get_exit_listsrrrr r"rr"c@sTeZdZdZddZddddejddfddZdd d Ze d d Z e ddZ dS)FileaF File within CollecTor. :var str path: file path within collector :var tuple types: descriptor types contained within this file :var stem.descriptor.Compression compression: file compression, **None** if this cannot be determined :var int size: size of the file :var str sha256: file's sha256 checksum :var datetime start: first publication within the file, **None** if this cannot be determined :var datetime end: last publication within the file, **None** if this cannot be determined :var datetime last_modified: when the file was last modified cCs||_|r t|nd|_t||_||_||_tj |d|_ d|_ |r9|r9tj |d|_ tj |d|_ dSt|\|_ |_ dS)Nrz%Y-%m-%d %H:%M)pathtupletypesr#_guess_compression compressionsizesha256datetimestrptime last_modified_downloaded_torr_guess_time_range)selfr$r&r)r*first_publishedlast_publishedr-rrr __init__s z File.__init__Nr c cs.|dur+tdd|jD}|jstdt|dkr&tdd|j|jd}|dur]|jrAtj|jrAtj |j}nt } | | ||||||D]} | VqPt | dS||d ||} tjj| |d D]'} |dus{|| jrt| d d} | r|r| |krqm|r| |krqm| VqmdS) a Provides descriptors from this archive. Descriptors are downloaded or read from disk as follows... * If this file has already been downloaded through :func:`~stem.descriptor.collector.CollecTor.download' these descriptors are read from disk. * If a **directory** argument is provided and the file is already present these descriptors are read from disk. * If a **directory** argument is provided and the file is not present the file is downloaded this location then read. * If the file has neither been downloaded and no **directory** argument is provided then the file is downloaded to a temporary directory that's deleted after it is read. :param str directory: destination to download into :param str descriptor_type: `descriptor type `_, this is guessed if not provided :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param stem.descriptor.__init__.DocumentHandler document_handler: method in which to parse a :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` :param int timeout: timeout when connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose :returns: iterator for :class:`~stem.descriptor.__init__.Descriptor` instances in the file :raises: * **ValueError** if unable to determine the descirptor type * **TypeError** if we cannot parse this descriptor type * :class:`~stem.DownloadFailed` if the download fails NcSsg|] }|ddqS) r)split).0trrr szFile.read..z/Unable to determine this file's descriptor typerz;Unable to disambiguate file's descriptor type from among %sz, rT)r published)setr& ValueErrorlenjoinr.osr$existsdirnametempfileZmkdtempreadshutilrmtreedownloadstemZ descriptorZ parse_file startswithZtype_annotationnamegetattr) r0 directorydescriptor_typerrrrrZ base_typesZ tmp_directoryrr$r9rrr rBs8(      z File.readTFc CsN|jdd}|jtjkr|r|ddd}tj|}tj||}tj |s0t |tj |rtt |2}t t|j} t|} | | krZ|WdS|setd|| | fWdn1sowYtjjt|j||} |r|j| } t |d } | | Wdn1swY||_|S) a Downloads this file to the given location. If a file already exists this is a no-op. :param str directory: destination to download into :param bool decompress: decompress written file :param int timeout: timeout when connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose :param bool overwrite: if this file exists but mismatches CollecTor's checksum then overwrites if **True**, otherwise rases an exception :returns: **str** with the path we downloaded to :raises: * :class:`~stem.DownloadFailed` if the download fails * **IOError** if a mismatching file exists and **overwrite** is **False** /.rrNzP%s already exists but mismatches CollecTor's checksum (expected: %s, actual: %s)wb)r$r5r(r PLAINTEXTrsplitr> expanduserr=r?makedirsopenbinasciiZhexlifybase64Z b64decoder*hashlibrBZ hexdigestIOErrorrFutil connectionrE COLLECTOR_URL decompresswriter.) r0rJr\rrZ overwritefilenamer$Z prior_fileZ expected_hashZ actual_hashresponseZ output_filerrr rE+s4         z File.downloadcCs0tjtjtjfD] }||jr|SqtjS)z> Determine file comprssion from CollecTor's filename. )rLZMABZ2GZIPendswith extensionrP)r$r(rrr r'bs  zFile._guess_compressioncCst|}|r1tt|\}}t||d}|dkr&|t||ddfS|t|dddfSt|}|rLtj|dd}||tj ddfSdS)z Attemt to determine the (start, end) time range from CollecTor's filename. This provides (None, None) if this cannot be determined. r z%Y-%m-%d-%H-%M-%Sr)Zseconds)NN) YEAR_DATEsearchmapintgroupsr+SEC_DATEr,groupZ timedelta)r$Z year_matchZyearZmonthrZ sec_matchrrr r/ns  zFile._guess_time_range)TNr F) __name__ __module__ __qualname____doc__r3rENTRIESrBrE staticmethodr'r/rrrr r#s Z7  r#c @seZdZdZdddZddd Zdd d Zdd d Zdddej dddddf ddZ dddZ dddZ dddZ d ddZd!ddZeddZdS)"raS Downloader for descriptors from CollecTor. The contents of CollecTor are provided in `an index `_ that's fetched as required. :var int retries: number of times to attempt the request if downloading it fails :var float timeout: duration before we'll time out our request NcCs"||_||_d|_d|_d|_dS)Nr)rr _cached_index _cached_files_cached_index_at)r0rrrrr r3s  zCollecTor.__init__Fr c cH|sdnd}||||D]}|j||||||dD]} | VqqdS)aL Provides server descriptors published during the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param bool bridge: standard descriptors if **False**, bridge if **True** :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.server_descriptor.ServerDescriptor` for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails zserver-descriptorzbridge-server-descriptorrrNfilesrB r0rrrrrr desc_typefrrrr r  z CollecTor.get_server_descriptorsc crw)aZ Provides extrainfo descriptors published during the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param bool bridge: standard descriptors if **False**, bridge if **True** :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.extrainfo_descriptor.RelayExtraInfoDescriptor` for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails z extra-infozbridge-extra-inforxNryr{rrr rr~z#CollecTor.get_extrainfo_descriptorsc c<|d||D]}|j|d||||dD]}|VqqdS)a Provides microdescriptors estimated to be published during the given time range, sorted oldest to newest. Unlike server/extrainfo descriptors, microdescriptors change very infrequently... :: "Microdescriptors are expected to be relatively static and only change about once per week." -dir-spec section 3.3 CollecTor archives only contain microdescriptors that *change*, so hourly tarballs often contain very few. Microdescriptors also do not contain their publication timestamp, so this is estimated. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.microdescriptor.Microdescriptor for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails rrxNryr0rrrrrr}rrrr rs zCollecTor.get_microdescriptorsc cs|dkr |s |s d} n-|dkr|r|sd} n"|dkr"|s"|s"d} n|r'd} n|r3|dkr3td|td||| ||D]} | j|| ||||| d D]} | VqNq@d S) a Provides consensus router status entries published during the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param stem.descriptor.__init__.DocumentHandler document_handler: method in which to parse a :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` :param int version: consensus variant to retrieve (versions 2 or 3) :param bool microdescriptor: provides the microdescriptor consensus if **True**, standard consensus otherwise :param bool bridge: standard descriptors if **False**, bridge if **True** :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.router_status_entry.RouterStatusEntry` for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails r znetwork-status-consensus-3z$network-status-microdesc-consensus-3rsznetwork-status-2zbridge-network-statusz7Only v3 microdescriptors are available (not version %s)zCOnly v2 and v3 router status entries are available (not version %s)rxN)r;rzrB) r0rrrrrrrrrr|r}rrrr rs"   zCollecTor.get_consensusc cr)a Directory authority key certificates for the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.networkstatus.KeyCertificate for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails zdir-key-certificate-3rxNryrrrr r $ zCollecTor.get_key_certificatesc cr)a Bandwidth authority heuristics for the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.bandwidth_file.BandwidthFile for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails zbandwidth-filerxNryrrrr r!<rzCollecTor.get_bandwidth_filesc cr)a `TorDNSEL exit lists `_ for the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.tordnsel.TorDNSEL for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails ZtordnselrxNryrrrr r"TrzCollecTor.get_exit_listsbestcCs|jr t|jtkrZ|dkr%tjtjtjtjfD] }|j r#|}nqn|dur,tj}|tjkr4|j nd}t d|}| t jj||j|j}tt jj||_t|_|jS)a Provides the archives available in CollecTor. :param descriptor.Compression compression: compression type to download from, if undefiled we'll use the best decompression available :returns: **dict** with the archive contents :raises: If unable to retrieve the index this provide... * **ValueError** if json is malformed * **IOError** if unable to decompress * :class:`~stem.DownloadFailed` if the download fails rNzindex/index.json)rttimervREFRESH_INDEX_RATErr`rarbrPZ availablerdr[r\rFrYrZrErrjsonloadsZ str_toolsZ _to_unicode)r0r(optionrdZurlr_rrr indexls   zCollecTor.indexcs|jr t|jtkrtt|gddd|_g}|jD]1}|r/|jdus.|j|kr/q |r<|j dus;|j |krsz!CollecTor.files..)keyNcsg|]}|qSr)rG)r6r|rKrr r8sz#CollecTor.files..) rurrvrsortedr_filesrrranyr&append)r0rKrrmatchesr}rrr rzs   zCollecTor.filescCst|tsgSg}|D]P\}}|dkrE|D],}d||dg}|t||d|d|d|d|d|d qq |d kr]|D]}|t |||dgqKq |S) z Recursively provies files within the index. :param dict val: index hash :param list path: path we've transversed into :returns: **list** of :class:`~stem.descriptor.collector.File` rzrLr$r&r)r*r1r2r-Z directories) isinstancedictitemsr=getrr#extendrr)valr$rzkvattrZ file_pathrrr rs @ zCollecTor._files)rsNNNNFNr NNNNr )r)NNN)rmrnrorpr3r rrrrqrr r!r"rrzrrrrrrr rs   " ,    #"rrr)$rprVrUr+rWrr>rerCrArZstem.descriptorrFZstem.util.connectionZstem.util.str_toolsrrr[rrcompilerfrkrr r rrrqrr r!r"objectr#rrrrr s@/      a