o ^9 @sdZddlZddlZddlZddlZddlZddlZddlZddl Zddl m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!ddlm"Z"m#Z#m$Z$m%Z%dZ&dZ'dZ(dZ)d Z*d Z+d Z,id d dddddddddddddddddddd d!d"d#d$d%d&d'd&d(d&d)d*d*d+d,d-d.d/dd0Z-d1Z.d2Z/d3\Z0Z1id4d5d6d7e1fd8d7e1fd9d7e1fd:d;d d&e1fdd;dd<dd=dd>dd?dd@de0dAfdd&e1fddBe1fd!dCd#dDid%d;d'd;d(d;dEd;dFdGe1fdHdIdJdKdLdMdNdOdPd;dQd&e1fd)dRdSdRdTdUdVdUdWdKdXdYdZd;d[Z2Gd\d]d]e3d]gd^Z4Gd_d`d`e3d`gdaZ5Gdbdcdce3dcgddZ6ddedeej7fdfdgZ8ddhdiZ9ddjdkZ:Gdldmdme Z;dndoZdtduZ?e`_. Documents can be obtained from a few sources... * The 'cached-consensus' file in Tor's data directory. * Archived descriptors provided by `CollecTor `_. * Directory authorities and mirrors via their DirPort. ... and contain the following sections... * document header * list of :class:`stem.descriptor.networkstatus.DirectoryAuthority` * list of :class:`stem.descriptor.router_status_entry.RouterStatusEntry` * document footer **For a great graphical overview see** `Jordan Wright's chart describing the anatomy of the consensus `_. Of these, the router status entry section can be quite large (on the order of hundreds of kilobytes). As such we provide a couple of methods for reading network status documents through :func:`~stem.descriptor.__init__.parse_file`. For more information see :func:`~stem.descriptor.__init__.DocumentHandler`... :: from stem.descriptor import parse_file, DocumentHandler with open('.tor/cached-consensus', 'rb') as consensus_file: # Processes the routers as we read them in. The routers refer to a document # with an unset 'routers' attribute. for router in parse_file(consensus_file, 'network-status-consensus-3 1.0', document_handler = DocumentHandler.ENTRIES): print router.nickname **Module Overview:** :: NetworkStatusDocument - Network status document |- NetworkStatusDocumentV2 - Version 2 network status document |- NetworkStatusDocumentV3 - Version 3 network status document +- BridgeNetworkStatusDocument - Version 3 network status document for bridges KeyCertificate - Certificate used to authenticate an authority DocumentSignature - Signature of a document by a directory authority DetachedSignature - Stand alone signature used when making the consensus DirectoryAuthority - Directory authority as defined in a v3 network status document N) PGP_BLOCK_END Descriptor DigestHashDigestEncodingTypeAnnotationDocumentHandler_descriptor_content_descriptor_components_read_until_keywords_value_values_parse_simple_line_parse_if_present_parse_timestamp_line_parse_forty_character_hex_parse_protocol_line_parse_key_block _mappings_for_random_nickname_random_fingerprint_random_ipv4_address _random_date_random_crypto_blob)RouterStatusEntryV2RouterStatusEntryBridgeV2RouterStatusEntryV3RouterStatusEntryMicroV3) )network-status-versionT) dir-sourceT fingerprintT)contactTdir-signing-keyT)client-versionsF)server-versionsF) publishedT) dir-optionsF)directory-signatureT))rTTT) vote-statusTTT)consensus-methodsTFF)consensus-methodFTF)r&TFT) valid-afterTTT) fresh-untilTTT) valid-untilTTT) voting-delayTTT)r$TTF)r%TTF)packageTTF) known-flagsTTT)flag-thresholdsTFF)shared-rand-participateTFF)shared-rand-commitTFF)shared-rand-previous-valueTTF)shared-rand-current-valueTTF)bandwidth-file-headersTFF)bandwidth-file-digestTFF)recommended-client-protocolsTTF)recommended-relay-protocolsTTF)required-client-protocolsTTF)required-relay-protocolsTTF)paramsTTF))directory-footerTTF)bandwidth-weightsFTF)r(TTTrrr>r(Z bwweightscale'Z cbtdisabledZ cbtnummodesZcbtrecentcountZcbtmaxtimeoutsZ cbtmincircsd cbtquantilePcbtclosequantile_Z cbttestfreq< cbtmintimeouticbtinitialtimeout`ZcbtlearntimeoutZcbtmaxopencircs ZUseOptimisticDataZSupport022HiddenServicesZ usecreatefastz%max-consensuses-age-to-cache-for-diffHP)!try-diff-for-consensus-newer-thanonion-key-rotation-daysonion-key-grace-period-dayshs_service_max_rdv_failurescirc_max_cell_queue_sizecircpad_max_circ_queued_cells"HiddenServiceEnableIntroDoSDefense) )dir-key-certificate-versionT) dir-addressFr)dir-identity-keyT)dir-key-publishedT)dir-key-expiresTr")dir-key-crosscertF)dir-key-certificationT))consensus-digestTF)r,TF)r-TF)r.TF)additional-digestFT)additional-signatureFT)r(FT)iiZ circwindow)rErVZCircuitPriorityHalflifeMsecZ perconnbwrateZperconnbwburstZrefuseunknownexits)rrP)rPrC)rBrV)rBrA)rPrA)rOcrii)rOrM)rZUseNTorHandshakeZFastFlagMinThresholdZNumDirectoryGuards)rrOZNumEntryGuards)rPrOZ GuardLifetime)i'iSg ZNumNTorsPerTAP)rPiZAllowNonearlyExtendZAuthDirNumSRVAgreements)ri rWrX)rPZrYrZr[)rVl)rrU)r\r]c@eZdZdZdS)PackageVersionz Latest recommended version of a package that's available. :var str name: name of the package :var str version: latest recommended version :var str url: package's url :var dict digests: mapping of digest types to their value N__name__ __module__ __qualname____doc__rtrt?/usr/lib/python3/dist-packages/stem/descriptor/networkstatus.pyrnrn)nameversionurldigestsc@rm)SharedRandomnessCommitmenta Directory authority's commitment for generating the next shared random value. :var int version: shared randomness protocol version :var str algorithm: hash algorithm used to make the commitment :var str identity: authority's sha1 identity fingerprint :var str commit: base64 encoded commitment hash to the shared random value :var str reveal: base64 encoded commitment to the shared random value, **None** of not provided Nrortrtrtrur{rvr{)rx algorithmidentitycommitrevealc@rm)DocumentDigestz Digest of a consensus document. .. versionadded:: 1.8.0 :var str flavor: consensus type this digest is for (for example, 'microdesc') :var str algorithm: hash algorithm used to make the digest :var str digest: digest value of the consensus Nrortrtrtrurrvr)flavorr|digestFc ks|durt}|tkrtt}}n-|tkr|rtnt}n"|tkr&tt}}n|tkr8|||fi|VdSt d||t j krQ|||fi|VdSt t ttf|}|rh|ddrh|dd}|}t ttf|dd|} |} td|| } |t jkr|| |fi|VdS|t jkrtjjj||f|t || || |fd |} | D]} | VqdSt d |) a Parses a network status and iterates over the RouterStatusEntry in it. The document that these instances reference have an empty 'routers' attribute to allow for limited memory usage. :param file document_file: file with network status document content :param class document_type: NetworkStatusDocument subclass :param bool validate: checks the validity of the document's contents if **True**, skips these checks otherwise :param bool is_microdescriptor: **True** if this is for a microdescriptor consensus, **False** otherwise :param stem.descriptor.__init__.DocumentHandler document_handler: method in which to parse :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` :param dict kwargs: additional arguments for the descriptor constructor :returns: :class:`stem.descriptor.networkstatus.NetworkStatusDocument` object :raises: * **ValueError** if the document_version is unrecognized or the contents is malformed and validate is **True** * **IOError** if the file can't be read NzIDocument type %i isn't recognized (only able to parse v2, v3, and bridge)rs@typerPT)skip) entry_class entry_keywordZstart_positionZ end_position extra_argsz!Unrecognized document_handler: %s)NetworkStatusDocumentV3NetworkStatusDocumentV2rrrBridgeNetworkStatusDocumentrDetachedSignatureread ValueErrorrZDOCUMENTr ROUTERS_START FOOTER_STARTV2_FOOTER_START startswithtell readlinesbytesjoinZ BARE_DOCUMENTENTRIESstem descriptorrouter_status_entry _parse_file) document_fileZ document_typevalidateis_microdescriptorZdocument_handlerkwargsZ router_typeheaderZ routers_startZ routers_endZfooterdocument_contentZ desc_iteratordescrtrtrur(sV          rccsV td|}tddd}|t||d7}|r(tjjjtd||dVndSq) a Parses a file containing one or more authority key certificates. :param file certificate_file: file with key certificates :param bool validate: checks the validity of the certificate's contents if **True**, skips these checks otherwise :returns: iterator for :class:`stem.descriptor.networkstatus.KeyCertificate` instances in the file :raises: * **ValueError** if the key certificates are invalid and validate is **True** * **IOError** if the file can't be read Trd rPrrrN) r rsplitrr networkstatusKeyCertificaterr)Zcertificate_filerZkeycert_contentZblock_end_prefixrtrtru_parse_file_key_certsws rccs: td|dd}|rtjjjtd||dVndSq)a Parses a file containing one or more detached signatures. :param file detached_signature_file: file with detached signatures :param bool validate: checks the validity of the detached signature's contents if **True**, skips these checks otherwise :returns: iterator for :class:`stem.descriptor.networkstatus.DetachedSignature` instances in the file :raises: * **ValueError** if the detached signatures are invalid and validate is **True** * **IOError** if the file can't be read Tre)Z ignore_firstrrN)r rrrrrr)Zdetached_signature_filerZdetached_sig_contentrtrtru_parse_file_detached_sigssrc@s"eZdZdZejejfddZdS)NetworkStatusDocumentz1 Common parent for network status documents. cCsT|jdd}|tjkrtjt||S|tjkr$tjt ||St d|)a Digest of this descriptor's content. These are referenced by... * **DetachedSignature** * Referer: :class:`~stem.descriptor.networkstatus.DetachedSignature` **consensus_digest** attribute * Format: **SHA1/HEX** .. versionadded:: 1.8.0 :param stem.descriptor.DigestHash hash_type: digest hashing algorithm :param stem.descriptor.DigestEncoding encoding: digest encoding :returns: **hashlib.HASH** or **str** based on our encoding argument z directory-signature )endzMNetwork status document digests are only available in sha1 and sha256, not %s) _content_rangerSHA1rrZ_encode_digesthashlibsha1ZSHA256Zsha256NotImplementedError)selfZ hash_typeencodingcontentrtrtrurs    zNetworkStatusDocument.digestN) rprqrrrsrrrZHEXrrtrtrtrursrcsfdd}|S)NcsRt|}|std|ft|t|t|kr'td|fdS)Nz)Document has a non-numeric version: %s %sz._parsert)rrrrrtrru_parse_version_lines rcCstd|}|}t|dkrtd||dstd|tjj|ds0td|dtjjj|dd d sCtd |d|d|_ |d|_ |dd krXd|_ dSt |d|_ dS) NrrBz[The 'dir-source' line of a v2 network status document must have three values: dir-source %sr2Authority's hostname can't be blank: dir-source %srP2Authority's address isn't a valid IPv4 address: %srTTZ allow_zero"Authority's DirPort is invalid: %s0) r rlenrrutil connectionis_valid_ipv4_address is_valid_porthostnameaddressrdir_portrrrZdir_source_comprtrtru_parse_dir_source_lines      (rcCsVg}td|D]}|d}t|dkrtd||t|ddq||_dS)NrfrrBziadditional-digest lines should be of the form 'additional-digest [flavor] [algname] [digest]' but was: %s)r rrrappendradditional_digests)rrrzvalcomprtrtru_parse_additional_digestss    rc Csg}|dD]7\}}}|d}t|dkrtd||r"|dkr(td||t|d|d|d ||d d d q||_dS) Nrgrrkzadditional-signature lines should be of the form 'additional-signature [flavor] [algname] [identity] [signing_key_digest]' but was: %s SIGNATUREzL'additional-signature' should be followed by a SIGNATURE block, but was a %srPrTrBrT)rr)rrrrDocumentSignatureadditional_signatures)rr signaturesr block_typeblock_contentsrrtrtru_parse_additional_signaturess     , rrrxr r!r# signing_keyRSA PUBLIC KEYr$client_versionscC |dSN,rvrtrtru r)funcr%server_versionscCrrrrrtrtrurrr&r'optionscCs|SNrrrtrtrur signaturersigning_authority)Zvalue_attributereconsensus_digestcseZdZdZdZdefdefdefdefdefdefde fge fge fde fge fdefdefd Zeeeee e e e e ed Zeddd Zdfd d Zd d ZZS)ra Version 2 network status document. These have been deprecated and are no longer generated by Tor. :var dict routers: fingerprints to :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV2` contained in the document :var int version: **\*** document version :var str hostname: **\*** hostname of the authority :var str address: **\*** authority's IP address :var int dir_port: **\*** authority's DirPort :var str fingerprint: **\*** authority's fingerprint :var str contact: **\*** authority's contact information :var str signing_key: **\*** authority's public signing key :var list client_versions: list of recommended client tor version strings :var list server_versions: list of recommended server tor version strings :var datetime published: **\*** time when the document was published :var list options: **\*** list of things that this authority decides :var str signing_authority: **\*** name of the authority signing the document :var str signature: **\*** authority's signature for the document **\*** attribute is either required when we're parsed with validation or has a default value, others are left as **None** if undefined znetwork-status-2N) rxrrrr r!rrrr&rrr) rrr r!r#r$r%r&r'r(rtFc Cs^|r td|jt||dddttffdtfddtfdtd ffd d td ffS) NSigning of %s not implemented)r2rz%s %s 80r )r!zarma at mit dot edur&r#rr(Zmoria2r)rrprrrrrclsattrexcludesignrtrtrurXs zNetworkStatusDocumentV2.contentcstt|j|| dt|}tdttt f|}t j j j ||ttt f|fd}tdd|D|_t|d||}|re|||||d|jvrad|vrYd |vsctd t|dSdS||_dS) NZ lazy_loadrrrZsection_end_keywordsrcs|]}|j|fVqdSrr .0rrtrtru |z3NetworkStatusDocumentV2.__init__.. ZVersionsr$r%zVersion 2 network status documents must have a 'client-versions' and 'server-versions' when 'Versions' is listed among its dir-options: %s)superr__init__ioBytesIOrrr rrrrrrrdictroutersr r_check_constraintsrrrstr_entries)r raw_contentrrr router_iterr __class__rtrurhs(     z NetworkStatusDocumentV2.__init__cCsddtD}|D]}||vrtd|t|fq ddtD}|D]}||vr@t||dkr@td|t||t|fq#dt|dkrStd t|dS) NcSsg|]\}}|r|qSrtrt)rfield is_mandatoryrtrtru z>NetworkStatusDocumentV2._check_constraints..z6Network status document (v2) must have a '%s' line: %scSsg|]\}}|qSrtrt)rr _rtrtrur rPzINetwork status document (v2) can only have a single '%s' line, got %i: %srrz[Network status document (v2) are expected to start with a 'network-status-version' line: %s)NETWORK_STATUS_V2_FIELDSrrrlistkeys)rrrequired_fieldsrZ single_fieldsrtrtrursz*NetworkStatusDocumentV2._check_constraintsNrtFF)rprqrrrsTYPE_ANNOTATION_NAME"_parse_network_status_version_liner_parse_fingerprint_line_parse_contact_line_parse_dir_signing_key_line_parse_client_versions_line_parse_server_versions_line_parse_published_line_parse_dir_options_line_parse_directory_signature_line ATTRIBUTESPARSER_FOR_LINE classmethodrrr __classcell__rtrtrrurs@ $rcCsxtd|}d|vr|dd\}}n|d}}|s!td|t||_||_|dk|_|jdkr:td|jdS) NrrrPnszLNetwork status document has a non-numeric version: network-status-version %sZ microdescrBzFExpected a version 3 network status document, got version '%s' instead)r rrrrrxversion_flavorr)rrrrxrrtrtru)_parse_header_network_status_version_lines      r&cCsFtd|}|dkrd\|_|_dS|dkrd\|_|_dStd|)Nr) consensus)TFvoteFTz`A network status document's vote-status line can only be 'consensus' or 'vote', got '%s' instead)r is_consensusis_voterrrtrtru_parse_header_vote_status_lines  r,cCs`|jr |jr dg|_td|g}}|dD]}|s#td||t|q||_dS)NrPr*rz\A network status document's consensus-methods must be a list of integer values, but was '%s') _lazy_loadingr+consensus_methodsr rrrrr)rrrr.entryrtrtru$_parse_header_consensus_methods_lines   r0cCs>|jr |jr d|_td|}|std|t||_dS)NrPr+zMA network status document's consensus-method must be an integer, but was '%s')r-r*consensus_methodr rrrrrtrtru#_parse_header_consensus_method_lines   r2cCsdtd|}|d}t|dkr,|dr,|dr,t|d|_t|d|_dStd|)Nr/rrTrrPz^A network status document's 'voting-delay' line must be a pair of integer values, but was '%s')r rrrr vote_delay dist_delayrrrr value_comprtrtru_parse_header_voting_delay_lines  $ r7csfdd}|S)Nc sht|g}}|dD]}z |tj|Wq ty+td||fwt||dS)NrzWNetwork status document's '%s' line had '%s', which isn't a parsable tor version: %s %s)r rrrrxZ _get_versionrr)rrrr/rrrtrurs z$_parse_versions_line.._parsert)rrrrtr8ru_parse_versions_lines r9c Cstd|i}}td|D];\}}z*|dr+td|ddddd||<nd|vr6t|||<nt|||<WqtyJtd|w||_dS) Nr2%z0.rh.rPzjNetwork status document's 'flag-thresholds' line is expected to have float values, got: flag-thresholds %s) r striprendswithfloatreplacerrflag_thresholds)rrrZ thresholdskeyrrtrtru"_parse_header_flag_thresholds_lines $    rCcCsJ|jr |jr ttni|_td|}|dkr#td|d|_|dSdS)Nr=r<T)r-_default_paramsrDEFAULT_PARAMSr=r _parse_int_mappings_check_params_constraintsrrtrtru_parse_header_parameters_lines  rHcCstd|}|r td|dS)Nr>ziA network status document's 'directory-footer' line shouldn't have any content, got 'directory-footer %s')r rrrtrtru_parse_directory_footer_line$s  rIc Csg}|dD]E\}}}|ddvrtd||r|dkr$td||ddkr6d}|dd\}}n |dd \}}}|t||||d d q||_dS) Nr(r)rPrTzAuthority signatures in a network status document are expected to be of the form 'directory-signature [METHOD] FINGERPRINT KEY_DIGEST', received: %srzK'directory-signature' should be followed by a SIGNATURE block, but was a %srPrrTTr)countrrrrr) rrrZ sig_valuerrmethodr key_digestrtrtru&_parse_footer_directory_signature_line-s    rMc Csg}|dD]B\}}}|dd}t|dkrtd||dd\}}}i} t|dkr>td|dD]\} } | | | <q5|t|||| q||_dS)Nr0rrBz<'package' must at least have a 'PackageName Version URL': %srk)rrrrrrnpackages) rrZpackage_versionsrrr6rwrxryrzrBrrtrtru_parse_package_lineBs      rOc Csg}|dD]B\}}}|}t|dkrtd||dd\}}}} t|dkr/|dnd} |s;td||tt|||| | q||_dS)Nr4rkzO'shared-rand-commit' must at least have a 'Version AlgName Identity Commit': %szBThe version on our 'shared-rand-commit' line wasn't an integer: %s)rrrrrr{rshared_randomness_commitments) rrZ commitmentsrrr6rxr|r}r~rrtrtru_parsed_shared_rand_commitWs    rRcCTtd|}|d}t|dkr$|dr$t|d|_|d|_dStd|)Nr5rrTrrPzyA network status document's 'shared-rand-previous-value' line must be a pair of values, the first an integer but was '%s')r rrrr'shared_randomness_previous_reveal_count shared_randomness_previous_valuerr5rtrtru!_parse_shared_rand_previous_valuem   rVcCrS)Nr6rrTrrPzxA network status document's 'shared-rand-current-value' line must be a pair of values, the first an integer but was '%s')r rrrr&shared_randomness_current_reveal_countshared_randomness_current_valuerr5rtrtru _parse_shared_rand_current_valuezrWrZcC4td|}i}td|D]\}}|||<q ||_dS)Nr7)r rbandwidth_file_headersrrrresultsrBrrtrtru_parse_bandwidth_file_headerss   r_cCr[)Nr8)r rbandwidth_file_digestr]rtrtru_parse_bandwidth_file_digests   rar, valid_afterr- fresh_untilr. valid_untilr1 known_flagscCsdd|dDS)NcSsg|]}|r|qSrtrt)rr/rtrtrur rz..rrrrtrtrurrr?bandwidth_weightscCs td|dS)Nr?T)rFrrtrtrurs r3 is_shared_randomness_participater9recommended_client_protocolsr:recommended_relay_protocolsr;required_client_protocolsr<required_relay_protocolscseZdZdZiddefddefddefdd efd d efd gefd defd defdde fdde fdde fdde fdde fdge fdgefdgefdgefiefiefiefiefiefiefdefdefdefdefiefiefgefiefdZidedededed ede de de de d e d!ed"ed#ed$ed%ed&ed'eeeeeeed(Zeeed)Z e!dCd+d,Z"e!dDd-d.Z#dEfd/d0 Z$d1d2Z%d3d4Z&d5d6Z'd7d8Z(fd9d:Z)d;d<Z*d=d>Z+d?d@Z,dAdBZ-Z.S)Fra Version 3 network status document. This could be either a vote or consensus. :var dict routers: fingerprint to :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` mapping for relays contained in the document :var int version: **\*** document version :var str version_flavor: **\*** flavor associated with the document (such as 'ns' or 'microdesc') :var bool is_consensus: **\*** **True** if the document is a consensus :var bool is_vote: **\*** **True** if the document is a vote :var bool is_microdescriptor: **\*** **True** if this is a microdescriptor flavored document, **False** otherwise :var datetime valid_after: **\*** time when the consensus became valid :var datetime fresh_until: **\*** time when the next consensus should be produced :var datetime valid_until: **\*** time when this consensus becomes obsolete :var int vote_delay: **\*** number of seconds allowed for collecting votes from all authorities :var int dist_delay: **\*** number of seconds allowed for collecting signatures from all authorities :var list client_versions: list of recommended client tor versions :var list server_versions: list of recommended server tor versions :var list packages: **\*** list of :data:`~stem.descriptor.networkstatus.PackageVersion` entries :var list known_flags: **\*** list of :data:`~stem.Flag` for the router's flags :var dict params: **\*** dict of parameter(**str**) => value(**int**) mappings :var list directory_authorities: **\*** list of :class:`~stem.descriptor.networkstatus.DirectoryAuthority` objects that have generated this document :var list signatures: **\*** :class:`~stem.descriptor.networkstatus.DocumentSignature` of the authorities that have signed the document **Consensus Attributes:** :var int consensus_method: method version used to generate this consensus :var dict bandwidth_weights: dict of weight(str) => value(int) mappings :var int shared_randomness_current_reveal_count: number of commitments used to generate the current shared random value :var str shared_randomness_current_value: base64 encoded current shared random value :var int shared_randomness_previous_reveal_count: number of commitments used to generate the last shared random value :var str shared_randomness_previous_value: base64 encoded last shared random value **Vote Attributes:** :var list consensus_methods: list of ints for the supported method versions :var datetime published: time when the document was published :var dict flag_thresholds: **\*** mapping of internal performance thresholds used while making the vote, values are **ints** or **floats** :var dict recommended_client_protocols: recommended protocols for clients :var dict recommended_relay_protocols: recommended protocols for relays :var dict required_client_protocols: required protocols for clients :var dict required_relay_protocols: required protocols for relays :var dict bandwidth_file_headers: headers from the bandwidth authority that generated this vote :var dict bandwidth_file_digest: hashes of the bandwidth authority file used to generate this vote, this is a mapping of hash functions to their resulting digest value **\*** attribute is either required when we're parsed with validation or has a default value, others are left as None if undefined .. versionchanged:: 1.4.0 Added the packages attribute. .. versionchanged:: 1.5.0 Added the is_shared_randomness_participate, shared_randomness_commitments, shared_randomness_previous_reveal_count, shared_randomness_previous_value, shared_randomness_current_reveal_count, and shared_randomness_current_value attributes. .. versionchanged:: 1.6.0 Added the recommended_client_protocols, recommended_relay_protocols, required_client_protocols, and required_relay_protocols attributes. .. versionchanged:: 1.6.0 The is_shared_randomness_participate and shared_randomness_commitments were misdocumented in the tor spec and as such never set. They're now an attribute of votes in the **directory_authorities**. .. versionchanged:: 1.7.0 The shared_randomness_current_reveal_count and shared_randomness_previous_reveal_count attributes were undocumented and not provided properly if retrieved before their shred_randomness_*_value counterpart. .. versionchanged:: 1.7.0 Added the bandwidth_file_headers attributbute. .. versionchanged:: 1.8.0 Added the bandwidth_file_digest attributbute. rxNr%r$r*Tr+Frr.r&r1rbrcrdr3r4rrrNre)rArhrirjrkr=rTrUrXrYr\r`rrfrr)r*r+r,r-r.r/r$r%r0r1r2r9r:r;)r<r=r5r6r7r8)r>r?r(rtcCs |r td|j|durint|}|ddk}|r#dtd}nddi}|r4|dur4tj|dg}|D]\}} |rC||vrCq8||vrK| ||<q8t||d d d d d dtfdtfdtfddddddfddddt t t dfff} |rd| vr| dd} nd| vr| d d} n |r| d!7} t | d} t jjd"d#d$|Dd"} | d| | | | d} |rd| vr| dd} nd| vr| d d} n |r| d!7} t | d} t jjd"d%d$|Dd"} | d| | | | d} | S)&Nrr)r(z1 9)r*r&r+9)r+)r3)r)r')r*N)r+N)r&Nr,r-r.)r/z300 300)r$N)r%N)r0N)r1zPAuthority BadExit Exit Fast Guard HSDir Named Running Stable Unnamed V2Dir Valid)r=N)r>r<)r?Nr(z%s %s%srsdirectory-footers directory-footerrPsdirectory-signatures directory-signaturer cSg|]}t|qSrtr)rartrtrur rz3NetworkStatusDocumentV3.content..cSrortrp)rr@rtrtrur r)rrprgetrDirectoryAuthoritycreateitemsrrrfindrrr str_toolsZ _to_bytesr)rrrr authoritiesrr+Zextra_defaultskrZ desc_contentZ footer_divZauthority_contentZrouter_contentrtrtrurSsr   " "zNetworkStatusDocumentV3.contentcCs||||||||dS)Nrr)rrrrrrxrrtrtrurtzNetworkStatusDocumentV3.createc stt|j|| dt|}d|_g|_||_|||t t j j j ||tttttf|jfd|_|rL|jrLt|jdkrLtdt|j|jft j j j |||jrWtnttttf|fd}tdd|D|_|||dS) a~ Parse a v3 network status document. :param str raw_content: raw network status document data :param bool validate: **True** if the document is to be validated, **False** otherwise :param bool default_params: includes defaults in our params dict, otherwise it just contains values from the document :raises: **ValueError** if the document is invalid rFrrPzPVotes should only have an authority entry for the one that issued it, got %i: %scsrrrrrtrtrurrz3NetworkStatusDocumentV3.__init__..N)rrrrrrgrQrD_headertuplerrrrrs AUTH_STARTrrrr+Zdirectory_authoritiesrrrrrrr_footer)rrrZdefault_paramsrrrrtrurs4       z NetworkStatusDocumentV3.__init__cCsDt|tr tdddS|jst|jsdddSdddStdddS)Nbridge-network-statusrPrznetwork-status-consensus-3znetwork-status-vote-3z$network-status-microdesc-consensus-3) isinstancerrrr+rrtrtrutype_annotations   z'NetworkStatusDocumentV3.type_annotationcC"|jtjko|jkSS)aJ Checks if the current time is between this document's **valid_after** and **valid_until** timestamps. To be valid means the information within this document reflects the current network state. .. versionadded:: 1.8.0 :returns: **True** if this consensus is presently valid and **False** otherwise )rbdatetimeutcnowrdrrtrtruis_valid" z NetworkStatusDocumentV3.is_validcCr)a& Checks if the current time is between this document's **valid_after** and **fresh_until** timestamps. To be fresh means this should be the latest consensus. .. versionadded:: 1.8.0 :returns: **True** if this consensus is presently fresh and **False** otherwise )rbrrrcrrtrtruis_freshrz NetworkStatusDocumentV3.is_freshc Cs|dd}t|}d\}}t|jd}tdd|D}|jD]}|j|vr.q&| ||j|j } |d7}| |krD|d7}q&||krRt d|||fd S) a7 Validates we're properly signed by the signing certificates. .. versionadded:: 1.6.0 :param list key_certs: :class:`~stem.descriptor.networkstatus.KeyCertificates` to validate the consensus against :raises: **ValueError** if an insufficient number of valid signatures are present. rzdirectory-signature )rrg@cSsg|]}|j|jfqSrt)r r)rZcertrtrtrur r z?NetworkStatusDocumentV3.validate_signatures..rPzJNetwork Status Document has %i valid signatures out of %i total, needed %iN) rrrZ hexdigestupperrrrr}Z_digest_for_signaturerr) rZ key_certsZdigest_contentZ local_digestZ valid_digestsZ total_digestsZrequired_digestsZ signing_keysZsigZ signed_digestrtrtruvalidate_signaturess   z+NetworkStatusDocumentV3.validate_signaturescsB|jr|j|jd|jd|j|jd|jdd|_tt|S)NFZparser_for_line) r-r_header_entries_HEADER_PARSER_FOR_LINE_footer_entries_FOOTER_PARSER_FOR_LINErrget_unrecognized_linesrrrtrurs z.NetworkStatusDocumentV3.get_unrecognized_linescs:|jdur |jkS|jdurtfdd|jDSdS)a8 Checks if we meet the given consensus-method. This works for both votes and consensuses, checking our 'consensus-method' and 'consensus-methods' entries. :param int method: consensus-method to check for :returns: **True** if we meet the given consensus-method, and **False** otherwise Ncsg|]}|kr|qSrtrt)rxrKrtrur 3r zBNetworkStatusDocumentV3.meets_consensus_method..F)r1r.bool)rrKrtrrumeets_consensus_method%s  z.NetworkStatusDocumentV3.meets_consensus_methodcCstdttttf|}t||}ddtD}|rt| D] \}}t |dkr@||vr@|dkr@|dkr@t d|t |fq |j rIt t|_|j|||jd|d scd t|vrct d t||t|jrt|jstd|_dS|jr|jsdg|_dSdSdS||_|j|dS) NrcSg|]}|dqSrrtrrrtrtrur :rz3NetworkStatusDocumentV3._header..rPr0r4ANetwork status documents can only have a single '%s' line, got %irrSr=z[A network status document's 'params' line should only appear in consensus-method 7 or later)rrr r~rrr HEADER_STATUS_DOCUMENT_FIELDSrrurrrDrrEr=rrrr(_check_for_missing_and_disallowed_fieldsr*r1r+r.rrupdate)rrrrrZ header_fieldsrvaluesrtrtrur|7s* $      zNetworkStatusDocumentV3._headercCst||}ddtD}|rmt|D]\}}t|dkr5||vr5|dkr+|js5td|t|fq|j|||j d|rk| drUt| dd krTtd nt| ddkrctd t ||tdSdS||_ |j|dS) NcSrrrtrrtrtrur [rz3NetworkStatusDocumentV3._footer..rPr(rr rr>zkNetwork status document's footer should start with a 'directory-footer' line in consensus-method 9 or laterzkNetwork status document's footer should start with a 'directory-signature' line prior to consensus-method 9)r rFOOTER_STATUS_DOCUMENT_FIELDSrrurr*rrrrrrrrr)rrrrZ footer_fieldsrrrtrtrurYs(  zNetworkStatusDocumentV3._footercCs~|jD]7\}}t|ttf\}}|dkr|jd|}n |dkr*|jd|}||ks2||krUnable to parse network status document's '%s' line (%%s): %s'z¶meters must be sorted by their key+z'%s' is a non-numeric value)rrrrr) rrrr^Z seen_keysZerror_templaterBrZ prior_keyrtrtrurFs"      rFcCsZtd|}|d}t|dkrtd|tjj|dds*td|dtjj |ds;td |d|d sEtd |tjj |d sVtd |d tjj j |dddsitd|dtjj |dsztd|d|d|_ |d|_|d |_|d |_|ddkrdnt|d|_t|d|_|j d|_dS)NrrzGAuthority entry's 'dir-source' line must have six values: dir-source %sr-legacyz#Authority's nickname is invalid: %srPz"Authority's v3ident is invalid: %srTrrBrrkTrrrPz!Authority's ORPort is invalid: %sr)r rrrrr tor_toolsZis_valid_nicknamerstripis_valid_fingerprintrrrnicknamev3identrrrror_portr> is_legacyrrtrtru_parse_dirauth_source_lines.         rlegacy-dir-keylegacy_dir_key vote-digest vote_digestc seZdZdZiddefddefddefddefddefddefd d efd defd defd defdd efdge fdde fdde fdde fdde fZ eeeeee e e dZ edddZedddZdfdd ZZS) rsaC Directory authority information obtained from a v3 network status document. Authorities can optionally use a legacy format. These are no longer found in practice, but have the following differences... * The authority's nickname ends with '-legacy'. * There's no **contact** or **vote_digest** attribute. :var str nickname: **\*** authority's nickname :var str v3ident: **\*** identity key fingerprint used to sign votes and consensus :var str hostname: **\*** hostname of the authority :var str address: **\*** authority's IP address :var int dir_port: **\*** authority's DirPort :var int or_port: **\*** authority's ORPort :var bool is_legacy: **\*** if the authority's using the legacy format :var str contact: contact information, this is included if is_legacy is **False** **Consensus Attributes:** :var str vote_digest: digest of the authority that contributed to the consensus, this is included if is_legacy is **False** **Vote Attributes:** :var str legacy_dir_key: fingerprint of and obsolete identity key :var stem.descriptor.networkstatus.KeyCertificate key_certificate: **\*** authority's key certificate :var bool is_shared_randomness_participate: **\*** **True** if this authority participates in establishing a shared random value, **False** otherwise :var list shared_randomness_commitments: **\*** list of :data:`~stem.descriptor.networkstatus.SharedRandomnessCommitment` entries :var int shared_randomness_previous_reveal_count: number of commitments used to generate the last shared random value :var str shared_randomness_previous_value: base64 encoded last shared random value :var int shared_randomness_current_reveal_count: number of commitments used to generate the current shared random value :var str shared_randomness_current_value: base64 encoded current shared random value **\*** mandatory attribute .. versionchanged:: 1.4.0 Renamed our 'fingerprint' attribute to 'v3ident' (prior attribute exists for backward compatability, but is deprecated). .. versionchanged:: 1.6.0 Added the is_shared_randomness_participate, shared_randomness_commitments, shared_randomness_previous_reveal_count, shared_randomness_previous_value, shared_randomness_current_reveal_count, and shared_randomness_current_value attributes. rNrrrrrrFr!rrrgrQrTrUrXrY)rr!rrr3r4r5r6rtcCs|r td|j|durint|}|s$d|vs$|rd|vs$t|d<t||ddtttffdf}|r@|dt7}|S)Nrrrz%s %s no.place.com %s 9030 9090)r!zMike Perry r) rrprrrrrrr)rrrrr+rrtrtrurEs zDirectoryAuthority.contentTcCs||||||||dS)N)rr+rz)rrrrrr+rtrtrurt[r{zDirectoryAuthority.createcstt|j|| dtjj|}|d}|dkr0t||dd||_ |d|d}nd|_ t ||}|rJdt | dkrJt d||rd |d}}|rc|dddd }dgg} } |sp| d g7} |r|j s{t d || d g7} n|s|j rt d||s| d g7} | dg7} | D]} | |vrt d| |fq|D]} | | vr|rdnd} t d| | |fqt |D]\} } t| dkr| dvrt d| t| |fq|||n||_|j|_dS)a Parse a directory authority entry in a v3 network status document. :param str raw_content: raw directory authority entry information :param bool validate: checks the validity of the content if True, skips these checks otherwise :param bool is_vote: True if this is for a vote, False if it's for a consensus :raises: ValueError if the descriptor data is invalid rz dir-key-certificate-versionrhrPNrrzDAuthority entries are expected to start with a 'dir-source' line: %sFrr!z/Authority votes must have a key certificate: %srz@Authority consensus entries shouldn't have a key certificate: %srz+Authority entries must have a '%s' line: %sZvoteszconsensus entriesz+Authority %s shouldn't have a '%s' line: %s)rr!rrz>Authority entries can only have a single '%s' line, got %i: %s)rrsrrrrw _to_unicodervrZkey_certificater rrrrrrr>rurrrrr )rrrr+rZkey_divrrZdir_source_entryrZexcluded_fieldsrZ type_labelrrrtrur_sV             zDirectoryAuthority.__init__)NrtFF)NrtTFF)FF)rprqrrrsrr_parse_vote_digest_line_parse_legacy_dir_key_line#_parse_shared_rand_participate_linerRrVrZr r!r"rrtrr#rtrtrrurssb7       rscCsvtd|}d|vrtd||dd\}}tjj|s$td|tjj|s1td|||_t ||_ dS)Nr_:zZKey certificate's 'dir-address' is expected to be of the form ADDRESS:PORT: dir-address %srPzDKey certificate's address isn't a valid IPv4 address: dir-address %sz4Key certificate's dirport is invalid: dir-address %s) r rrsplitrrrrrrrr)rrrrZdirportrtrtru_parse_dir_address_lines    rr^rarbexpiresr` identity_keyrc crosscertz ID SIGNATURErd certificationc seZdZdZdZdefdefdefdefdefde fde fde fde fde fd Zeeee e ee e e d Zed dd Zd fd d ZZS)ra Directory key certificate for a v3 network status document. :var int version: **\*** version of the key certificate :var str address: authority's IP address :var int dir_port: authority's DirPort :var str fingerprint: **\*** authority's fingerprint :var str identity_key: **\*** long term authority identity key :var datetime published: **\*** time when this key was generated :var datetime expires: **\*** time after which this key becomes invalid :var str signing_key: **\*** directory server's public signing key :var str crosscert: signature made using certificate's signing key :var str certification: **\*** signature of this key certificate signed with the identity key **\*** mandatory attribute zdir-key-certificate-3N) rxrrr rr&rrrr) r^r_r rarbr`r#rcrdrtFc CsX|r td|jt||ddtfdtfdtfdtdfdtdffd td ffS) Nr)r^rmr rarbr`rr#rdr)rrprrrrrrtrtrurs   zKeyCertificate.contentcstt|j|| dt||}|rddt|dkr"td|dt|dkr2td|tD]'\}}|rF||vrFtd||ft| |g}|d kr[td |||fq4| ||dS||_ dS) Nrr^rzIKey certificates must start with a 'dir-key-certificate-version' line: %srdrhzAKey certificates must end with a 'dir-key-certification' line: %sz*Key certificates must have a '%s' line: %srPz=Key certificates can only have a single '%s' line, got %i: %s) rrrr rrrKEY_CERTIFICATE_PARAMSrrrrr)rrrrrr  entry_countrrtrur s       zKeyCertificate.__init__rr)rprqrrrsr'_parse_dir_key_certificate_version_linerr_parse_identity_key_line_parse_dir_key_published_line_parse_dir_key_expires_line_parse_signing_key_line_parse_dir_key_crosscert_line!_parse_dir_key_certification_liner r!r"rrr#rtrtrrurs6 rc@sJeZdZdZdddZddZdd Zd d Zd d ZddZ ddZ dS)ra Directory signature of a v3 network status document. :var str method: algorithm used to make the signature :var str identity: fingerprint of the authority that made the signature :var str key_digest: digest of the signing key :var str signature: document signature :var str flavor: consensus type this signature is for (such as 'microdesc'), **None** if for the standard consensus :param bool validate: checks validity if **True** :raises: **ValueError** if a validity check fails NFcCsZ|rtjj|std|tjj|std|||_||_||_||_||_ dS)Nz4Malformed fingerprint (%s) in the document signaturez3Malformed key digest (%s) in the document signature) rrrrrrKr}rLrr)rrKr}rLrrrrtrtrur3s   zDocumentSignature.__init__cCsPt|tsdSdD]}t||t||kr"|t||t||Sq |ddS)NF)rKr}rLrrT)rrgetattr)rotherrKrrtrtru_compareDs  zDocumentSignature._comparecCstt|Sr)hashrr=rrtrtru__hash__NzDocumentSignature.__hash__cC||ddS)NcSs||kSrrtsortrtrurRrz*DocumentSignature.__eq__..rrrrtrtru__eq__QrzDocumentSignature.__eq__cCs ||k Srrtrrtrtru__ne__Ts zDocumentSignature.__ne__cCr)NcSs||kSrrtrrtrtrurXrz*DocumentSignature.__lt__..rrrtrtru__lt__WrzDocumentSignature.__lt__cCr)NcSs||kSrrtrrtrtrur[rz*DocumentSignature.__le__..rrrtrtru__le__ZrzDocumentSignature.__le__)NF) rprqrrrsrrrrrrrrtrtrtrur$s   rcsxeZdZdZdZdefdefdefdefge fge fge fdZ eeeee e e dZ ed dd Zd fd d ZZS)raG Stand alone signature of the consensus. These are exchanged between directory authorities when determining the next hour's consensus. Detached signatures are defined in section 3.10 of the dir-spec, and only available to be downloaded for five minutes between minute 55 until the end of the hour. .. versionadded:: 1.8.0 :var str consensus_digest: **\*** digest of the consensus being signed :var datetime valid_after: **\*** time when the consensus became valid :var datetime fresh_until: **\*** time when the next consensus should be produced :var datetime valid_until: **\*** time when this consensus becomes obsolete :var list additional_digests: **\*** :class:`~stem.descriptor.networkstatus.DocumentDigest` for additional consensus flavors :var list additional_signatures: **\*** :class:`~stem.descriptor.networkstatus.DocumentSignature` for additional consensus flavors :var list signatures: **\*** :class:`~stem.descriptor.networkstatus.DocumentSignature` of the authorities that have signed the document **\*** mandatory attribute zdetached-signature-3N)rrbrcrdrrr)rer,r-r.rfrgr(rtFcCs8|r td|jt||ddtfdtfdtffS)Nr)reZ(6D3CC0EFA408F228410A4A8145E1B0BB0670E442r,r-r.)rrprrrrtrtrurszDetachedSignature.contentcstt|j|| dt||}|rWdt|dkr"td|tD]*\}}}|r7||vr7td||ft| |g}|sN|dkrNtd|||fq$| ||dS||_ dS)NrrerzADetached signatures must start with a 'consensus-digest' line: %sz-Detached signatures must have a '%s' line: %srPz@Detached signatures can only have a single '%s' line, got %i: %s) rrrr rrrDETACHED_SIGNATURE_PARAMSrrrrr)rrrrrr Z is_multiplerrrtrurs     zDetachedSignature.__init__rr)rprqrrrsr_parse_consensus_digest_linerrrrrrMr r!r"rrr#rtrtrrur^s,   rcs&eZdZdZdZdfdd ZZS)ra< Network status document containing bridges. This is only available through the metrics site. :var dict routers: fingerprint to :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` mapping for relays contained in the document :var datetime published: time when the document was published rFcstt||d|_t|}tjj | }| drD| ddd }z tjj||_WntyC|rAtd|Ynw|rQtdtjj |tjjj||t|fd}tdd|D|_dS) Nz published rrPzEBridge network status document's 'published' time wasn't parsable: %szFBridge network status documents must start with a 'published' line: %s)rrcsrrrrrtrtrurrz7BridgeNetworkStatusDocument.__init__..)rrrr&rrrrrwrreadlinerrr=Z_parse_timestamprrrrrrr)rrrrZpublished_linerrrtrurs,    z$BridgeNetworkStatusDocument.__init__r)rprqrrrsrrr#rtrtrrurs rr){rs collectionsrrrZ#stem.descriptor.router_status_entryrZstem.util.str_toolsZstem.util.tor_toolsZ stem.versionZstem.descriptorrrrrrrrr r r r r rrrrrrrrrrrrrrrrrrr~rrrrErrrrr namedtuplernr{rrrrrrrrrrrrrrrrrrrrrr&r,r0r2r7r9rCrHrIrMrOrRrVrZr_rarrrrrrrrrrrrrrrFrrrrsrrrrrrrrrobjectrrrrtrtrtrus6d             !"#(   O                     _! B       X:T