@OOfҾZddlmZddlZddlZddlmZddlmZddlmZddlm Z ddlm Z ddl m Z dd l mZdd l mZdd l mZdd l mZdd lmZddlmZGddZ d ddZ ddZdZ ddZ d ddZe GddZGddej:Zy) ) annotationsN) Generator)Iterable)Iterator) dataclass)field)Enum) contentviewsflow) flowfilter)http)base)decodeceZdZeGddZeGddeZeGddeZeGddZeGd d ZGd d e Z e dd Z e ddZ e ddZGdde Ze ddZe ddZGddZ d ddZddZd dZy)! ProtoParserc<eZdZUdZded< dZded< dZded<y) ProtoParser.ParserRulean A parser rule lists Field definitions which are applied if the filter rule matches the flow. Matching on flow-level also means, a match applies to request AND response messages. To restrict a rule to a requests only use 'ParserRuleRequest', instead. To restrict a rule to a responses only use 'ParserRuleResponse', instead. z'list[ProtoParser.ParserFieldDefinition]field_definitionsstrnamefilterN)__name__ __module__ __qualname____doc____annotations__rr\/var/www/premiumrankchecker/venv/lib/python3.12/site-packages/mitmproxy/contentviews/grpc.py ParserRulers. CB6c8 r r"ceZdZdZy)ProtoParser.ParserRuleResponsez A parser rule lists Field definitions which are applied if the filter rule matches the flow. The rule only applies if the processed message is a server response. Nrrrrrr r!ParserRuleResponser$, r r&ceZdZdZy)ProtoParser.ParserRuleRequestz A parser rule lists Field definitions which are applied if the filter rule matches the flow. The rule only applies if the processed message is a client request. Nr%rr r!ParserRuleRequestr)4r'r r*cjeZdZUdZded< eeZded< dZded < dZ d ed < d Z d ed<y)!ProtoParser.ParserFieldDefinitiona Defines how to parse a field (or multiple fields with the same tag) in a protobuf messages. This allows to apply an intended decoding (f.e. decode uint64 as double instead) and to assign a descriptive name to a field. Field definitions are aggregated into rules, which also holds a filter to match selected HTTP messages. The most natural way to use this, is to describe known parts of a single protobuf message in a set of field descriptors, pack them into a rule and set the filter of the rule in a way, that it only applies to proper protobuf messages (f.e. to request traffic against an API endpoint matched by an URL flowfilter) rtagdefault_factoryz list[str] tag_prefixesNzProtoParser.DecodedTypes | Noneintended_decoding str | NonerFz bool | None as_packed) rrrrrrlistr0r1rr3rr r!ParserFieldDefinitionr,<sM Pi"'"= i=}=A:Aujr!& ;&Yr r5c*eZdZUdZded<dZded<y)ProtoParser.ParserOptionsFboolinclude_wiretypeexclude_message_headersN)rrrr9rr:rr r! ParserOptionsr7us"'$& ).-r r;cTeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZy)ProtoParser.DecodedTypesr N)rrrint32int64uint32uint64sint32sint64r8enumfixed32sfixed32floatfixed64sfixed64doublestringbytesmessageunknownrr r! DecodedTypesr=s^r r`cd}d}|t|kr0||}||dzd|zzz }|dz }|dkr||fS|t|kr0td)NrrDr>z&varint exceeds bounds of provided data)len ValueError)dataresoffsetos r!_read_base128lezProtoParser._read_base128lesps4y V A AH!f*- -C aKF4xs{"!s4y "ABBr c>dtjd|dddfS)NrAzdtjd|dddfS)NrEzr?r@rArBN) rrrvarintbit_64 len_delimited group_start group_endbit_32rr r! WireTypesrus    r r|c |g}d}|t|krtj||d\}}tj|dz}|dz } ||z }|tjjk(rtj||d\}} ||z }| j } | dkDrtj j} | dkDrtj j} ntj j} n|tjjk(rk(s:|tjjk(s|tjj@k(rtjj}n tCdg}d}|tjj,k(rv|tE|krtjG||d\} } || z }|jItjK|||||| |jLd|tE|kreng|tjj<k(rtE|dzdk7r td|tE|krtjO||d\} } || z }|jItjK|||||| |jLd|tE|kren|tjjk(r|tE|krtjG||d\} } || z }|||| z} | tE||dkDr td |jItjK|||||| |jLd|| z }|tE|krn|tjjPk(s|tjjRk(r td |tjj4k(rtE|d zdk7r td |tE|krqtjU||d\} } || z }|jItjK|||||| |jLd|tE|kren td d|_+|S)Nz1can not unpack field with data other than bytes: z?packed fields have to be embedded in a length delimited messagez;Wire type could not be determined from packed decoding typerT)rrrr-rrris_unpacked_childrenrEzcan not parse as packed bit64z/packed length delimited field exceeds data sizez$group tags can not be encoded packedrAzcan not parse as packed bit32r), isinstancerr]retyper-rrrrrr|rxr`rOrPrQrRrSrTr8rUrvrVrWrXr{rYrZr[rwr\r^ TypeErrorrdrjrrrrsryrzrpis_packed_parent) packed_fieldrr-rrr1packed_wire_typergrrhrrs r!read_packed_fieldszProtoParser.read_packed_fieldssl,1159CDI`I`DaCbc (22 ##-9-A-A.:.@.@6B6U6U  " "k&;&;&I&I Il55u=Q  !9!9!?!? ? K$<$<$B$BB K$<$<$C$CC K$<$<$C$CC K$<$<$C$CC K$<$<$C$CC K$<$<$A$AA K$<$<$A$AA*44;;  !9!9!A!A A K$<$<$E$EE K$<$<$B$BB*44;;  !9!9!A!A A K$<$<$E$EE K$<$<$C$CC*44;;  !9!9!@!@ @ K$<$<$B$BB K$<$<$D$DD*44BB M (* {44;; ;I&)99)CD/J v  %% '+<#"2#&%1%>%>-1&  I&!6!6!=!= =9~!Q& !@AAI&)33IcdOD v  %% '+<#"2#&%1%>%>-1&  I&!6!6!D!D DI&!,!%>-1&  v %I&(  5 5 A A A;#8#8#B#BBCD D !6!6!=!= =9~!Q& !@AAI&)33IcdOD v  %% '+<#"2#&%1%>%>-1&  I&"KL L)- % r ceZdZdZ d ddZddZdZ d ddZ d ddZddZ ddZ dd Z d Z dd Z ddd Zdd Zy)ProtoParser.Fielda Represents a single field of a protobuf message and handles the varios encodings. As mitmproxy sees the data passing by as raw protobuf message, it only knows the WireTypes. Each of the WireTypes could represent different Protobuf field types. The exact Protobuf field type can not be determined from the wire format, thus different options for decoding have to be supported. In addition the parsed WireTypes are (intermediary) stored in Python types, which adds some additional overhead type conversions. WireType represented Protobuf Types Python type (intermediary) 0: varint int32, int64, uint32, uint64, enum, int (*) sint32, sint64 (both ZigZag encoded), int bool bool float (**) 1: bit_64 fixed64, sfixed64, int (*) double float 2: len_delimited string, str message, class 'Message' bytes, bytes (*) packed_repeated_field class 'Message' (fields with same tag) 3: group_start unused (deprecated) - 4: group_end unused (deprecated) - 5: bit_32 fixed32, sfixed32, int (*) float float (*) Note 1: Conversion between WireType and intermediary python representation is handled by Kaitai protobuf decoder and always uses the python representation marked with (*). Converting to alternative representations is handled inside this class. (**) Note 2: Varint is not used to represent floating point values, but some applications store native floats in uint32 protobuf types (or native double in uint64). Thus we allow conversion of varint to floating point values for convenience (A well known APIs "hide" GPS latitude and longitude values in varint types, much easier to spot such things when rendered as float) Ref: - https://developers.google.com/protocol-buffers/docs/proto3 - https://developers.google.com/protocol-buffers/docs/encoding c ||_||_||_||_||_d|_||_||_||_d|_ g|_ |jM|jjdd|_ |jj|jjd|_ |j||jrd|_ yyy)NrF)rrrr-rrrrrr parent_tagsr try_unpack apply_rules) selfrrr-rrrrrs r!__init__zProtoParser.Field.__init__s5>DN@RD #+5DODH6=DLDI7#DO    'D,E,E"'-F'r c|j}d}d}d} |jD]}|jD]}d}t|jdk(r|j |k(rd}n'|jD]} | |j z|k(sd}n|sY|rW|j |j |_|j|j|_t|j|_ y|j r |j n|}|jr |jn|}|jsd} |r||_|r||_||_ y#t$r} tj| Yd} ~ yd} ~ wwxYw)NFrT) _gen_tag_strrrrdr0r-rr1rr8r3r Exceptionloggingwarning) ronly_first_hittag_strrdecodingr3rulefdmatchrtes r!rzProtoParser.Field.apply_rulessc'')GDHI& # JJ5D"445 %r/1479J$(E&(oo*#%;'#9,0E$)*!-#%77#602DI#%#7#7#C>@>R>RD$;26r||2D &35''rwwt(*';';%'$8$8)1!) $&<<04I955> $DI.6D+"+ #"" #s2A,EE AE&AE' E E0E++E0c|jdd}|j|jdj|Dcgc] }t |c}Scc}w)N.)rrr-joinr)rtagsr-s r!rzProtoParser.Field._gen_tag_strsC##A&D KK !886#SX67 76sAc.|jtjjk(r ||j ||fS|jtjjk(r ||j ||fS|jtjjk(r ||j ||fS|jtjj k(r ||j ||fStjj(|jfS#t $rwt |jjdkDr(tjj|jfcYStjj|jfcYSwxYw#t $r)tjj|jfcYSwxYw#t $r)tjj|jfcYSwxYw#t $rtjj"tjj$tjj&g}|D]1}||k(r|s ||j |dfccYS#t $rY/wxYwYwxYw)z Tries to decode as intended, applies failover, if not possible Returns selected decoding and decoded value rF)rrr|rv decode_asrintrrr`rRrQrwrYr{rVrxr^r\r]r_)rr1 try_as_packedlen_delimited_strategyfailover_decodings r!safe_decode_asz ProtoParser.Field.safe_decode_as"so~~!6!6!=!== P,dnn)=/;#8#8#?#??M,dnn)=/ ;#8#8#?#??M,dnn)=/ ;#8#8#F#FF!,dnn)=/,++33T__D DY!P4??+6682=*77>>OO*77>>OO P!M&33;;T__LLM!M&33;;T__LLM!!$0088#0077#0066N* .D !),0AA-$!#4dnn 157$ )! ! !!ssD%F!GHAF.&FF/G  G /HHA#J(J;J J  J J  JJc |durtj|S|jtjjk(rt |j tsJ|tjjk(r|j dzdk7S|tjjk(r_|j jdkDr tdtjdtjd|j dS|tjj k(r_|j jd kDr td tjd tjd |j dS|tjj"k(r4|j jdkDr td |j S|tjj$k(s|tjj&k(r4|j jd kDr td|j S|tjj(k(rH|j jdkDr td|j dz |j dz z S|tjj*k(rH|j jd kDr td|j dz |j dz z S|tjj,k(s|tjj.k(r|j1S|jtjj2k(r|tjj4k(r |j S|tjj6k(r7tjd tjd |j dS|tjj.k(r|j1S|jtjj8k(r|tjj:k(r |j S|tjj<k(r7tjdtjd|j dS|tjj,k(r|j1S|jtjj>k(rt |j t@sJ|tjjBk(r|jEdS|tjj@k(r|j ddS|tjjFk(r7tjI|j ||jJ|jLStd)NT)rlrrzwire value too large for int32z!i!Ir~zwire value too large for int64z!q!Qzwire value too large for uint32zwire value too largezwire value too large for sint32r>zwire value too large for sint64)escape_newline)rrrrz&intended decoding mismatches wire type)'rrrr|rvrrrr`r8rOrrrmrnpackrPrQrRrUrSrTrXr[_wire_value_as_floatrwrYrZr{rVrWrxr]r\wire_value_as_utf8r^rrr)rr1r3s r!rzProtoParser.Field.decode_asbsD "5545HH~~!6!6!=!==!$//3777$ (@(@(E(EE??-??1DD&+*B*B*H*HH113b8'(HII!==v{{4/QRSTUU&+*B*B*H*HH113b8'(HII!==v{{4/QRSTUU&+*B*B*I*II113b8'(IJJ??*%)A)A)H)HH(K,D,D,I,II113b8'(>????*&+*B*B*I*II113b8'(IJJ OOq0!+5'+*B*B*I*II113b8'(IJJ!OOq0doo6I4JJJ%)A)A)G)GG(K,D,D,K,KK 4466;#8#8#?#??$ (@(@(H(HH??*&+*B*B*K*KK!==v{{4/QRSTUU&+*B*B*I*II4466;#8#8#?#??$ (@(@(H(HH??*&+*B*B*K*KK!==v{{4/QRSTUU&+*B*B*H*HH4466;#8#8#F#FF!$//5999$ (@(@(G(GG 22$2GG&+*B*B*H*HH??1--&+*B*B*J*JJ&22"&//%) $ "jj 3DE Er ctd)NzbFuture work, needed to manipulate and re-encode protobuf message, with respect to given wire types)NotImplementedError)inputvalintended_encodings r! encode_fromzProtoParser.Field.encode_froms%t r c|j}t|dk(rtjd|dSt|dk(rtjd|dSt d)a Handles double (64bit) and float (32bit). Assumes Network Byte Order (big endian). Usable for: WireType --> Protobuf Type): ---------------------------- varint --> double/float (not intended by ProtoBuf, but used in the wild) bit_32 --> float bit_64 --> double len_delimited --> 4 bytes: float / 8 bytes: double / other sizes return NaN rAz!frrEz!dz4can not be converted to floatingpoint representation)_value_as_bytesrdrmrnr)rvs r!rz&ProtoParser.Field._wire_value_as_floats`$$&A1v{}}T1-a00Q1}}T1-a00RS Sr ct|jtr |jSt|jtr|jj dkDr t d|jj dkDr t jd|jSt jd|jSt d)Nr~z-value exceeds 64bit, violating protobuf specsrrrzcan not be converted to bytes)rrr]rrrermrrs r!rz!ProtoParser.Field._value_as_bytess$//51&DOOS1??--/"4%%TUU__//1B6!;;tT__==";;tT__==!!@AAr cPt|jjddSNr)rrsplitrs r!_wire_type_strz ProtoParser.Field._wire_type_strs!t~~&,,S1"5 5r c<t|jddSr)rr)rrs r! _decoding_strzProtoParser.Field._decoding_strsx=&&s+B/ /r ct|jtr1|jjd}|r|j ddS|St |jS)Nzutf-8 z\n)rrr]rreplacer)rrrgs r!rz$ProtoParser.Field.wire_value_as_utf8sK$//51oo,,W53As{{4/JsJt' 'r c#K|j|j|j\}}|j|j |j ||j d}t|trR|tjjk(r|js d|d<||D]}|jEd{y||d<|y7w)at Returns a generator which passes the field as a dict. In order to return the field value it gets decoded (based on a failover strategy and provided ParserRules). If the field holds a nested message, the fields contained in the message are appended. Ultimately this flattens all fields recursively. )r-wireTyperrrrN)rrrrrrrrr4rr`r^rgen_flat_decoded_field_dicts)rselected_decoding decoded_valfield_desc_dictfs r!rz.ProtoParser.Field.gen_flat_decoded_field_dictss.2-@-@''. * {((* //1 ../@A O +t,%"//778 11.0OE*))$@A ==???@*5&%%@sCCCCN)F)rrurr=r-rrProtoParser.Field | Nonerz int | bytesrr7rlist[ProtoParser.ParserRule]rr8returnNone)T)r1r=rr8rz[tuple[ProtoParser.DecodedTypes, bool | float | int | bytes | str | list[ProtoParser.Field]])r1r=r3r8rz:bool | int | float | bytes | str | list[ProtoParser.Field])rr=)rrX)rr])rr=)rrrzGenerator[dict, None, None])rrrrrrrrrrrrrrrrrr r!rzProtoParser.Fields+ l*/# (,# (!9# ( # ( 3 # ( $ # (/# (0# (#'# (# (L+ #Z 8#(> E7> E > E > EBRWP F%=P FJNP F GP Fd   T, B$ 6 0 ( &r rNc||_|tj}||_|g}||_ tj |j|jd|j|_y#t$r}td|d}~wwxYw)N)rrrrznot a valid protobuf message) rfrr;rrr root_fieldsrre)rrfrparser_optionsrs r!rzProtoParser.__init__s    !(668N% =E  D8C8O8O)) !jj 9P9D   D;{}]rz[{}]rr-r)rrr:r9formatr)r field_dictcol1col2col3col4s r! gen_str_rowszProtoParser.gen_str_rows+s;;= )J 44z*i7||,,!((J)?JAWX}}Z %;<f%De$Dz%()DdD( ( )sBB)rfr]rztuple[int, int]) rr]rrrr7rrrlist[ProtoParser.Field])rrrr)NN)rfr]rz#list[ProtoParser.ParserRule] | NonerzParserOptions | Nonerrr)rz&Generator[tuple[str, ...], None, None])rrrrr"r&r*r5r;r r` staticmethodrjrprsr|rrrrrrrr r!rrs   , Z   J  6Z6Z6Zp...t4CC,3333DAA.A+A, A ! AAFR'R RRhq&q&l 6:/3 DD3D- D  D08)r rc#8Kg}d}g}|D]}t|t|}t||kr |jdt||kr tt|D] }tt||||||<"|j|tt|D]}t |||||<|D]N}g}tt|D]/}||j ||dz} |jd| f1|Pyw)a Helper function to render tables with variable column count (move to contentview base, if needed elsewhere) Note: The function has to convert generators to a list, as all rows have to be processed twice (to determine the column widths first). rr?textN)maxrdrrangeminljust) table_rows max_col_widthrows col_count cols_widthrowcol_numilinecol_vals r! format_tabler@s5#%DIJ 3s8, *o )   a *o )SX NG"%c#g,&7G9L"MJw  N C3z? #:JqM=9 1 :!SX +G'l((G)A'B! BB!%B!' B0 A<<BB! B BBB!ct|Sr)r4)generator_funcs r!hack_generator_to_listr s  r c#jKtt|||jEd{y7w)N)rfrr)rrrr^rrs r! format_pbufrs0 u ,.s )313c#Kd}t||D]F\}}dt|zdzt|r|n|zdz}d|fgt|||Ed{Hy7w)Nr)rfrz gRPC message z (compressed )rr )rrr)rfrrr message_count compressed pb_messageheadlines r! format_grpcrs M"5 &8#  J -  !  $ C D   !""~U     sAAAAcZeZdZUeej Zded<eeZ ded<y) ViewConfigr.r7rr parser_rulesN) rrrrrr;rrr4rrr r!rrs/05#111N-27t1LL.Lr rceZdZdZdZgdZddgZgdZd dfd Z dd Z dddd  dd Z dddd  dd Z xZ S)ViewGrpcProtobufz'Human friendly view of protocol bufferszgRPC/Protocol Buffer)zapplication/x-protobufzapplication/x-protobufferzapplication/grpc-protozapplication/grpczapplication/prpc)gzipidentitydeflatezstdNcHt|| t}||_yr)superrrconfig)rr! __class__s r!rzViewGrpcProtobuf.__init__s!  >\F r c.g}|s|St|tj}|D]o}|rt|tjr |st|tj r=t j|j|s_|j|q|S)a, Checks which of the give rules applies and returns a List only containing those rules Each rule defines a flow filter in rule.filter which is usually matched against a flow. When it comes to protobuf parsing, in most cases request messages differ from response messages. Thus, it has to be possible to apply a rule to a http.Request or a http.Response, only. As the name flowfilter suggests, filters are working on a flow-level, not on message-level. This means: - the filter expression '~q' matches all flows with a request, but no response - the filter expression '~s' matches all flows with a response In result, for complete flows (with a gRPC message in the request and the response), ParserRules would either be applied to request and response at the same time ('~s') or neither would match request, nor response (~q). To distinguish between rules which should be applied to response messages, request messages or both (while being applied to the whole flow), different classes with same behavior are used to wrap rules: - ParserRule: applies to requests and responses - ParserRuleRequest: applies to requests only - ParserRuleResponse: applies to responses only r ) rrRequestrr&r*r rrr)rrr^r rg is_requestrs r!_matching_rulesz ViewGrpcProtobuf._matching_ruless<-/J6  !Dj{/M/MNJt[5R5R$S $7 4  ! r ) content_typer http_messagec |j|jj||}||jvrY |J|jd}||j vr|n|j d}t||jj||} d} n$t||jj|} d} t| } | | fS#t $r|j d}YtwxYw#t $r} | d} ~ wwxYw)N)rr r^z grpc-encodingr)rfrrrgRPCr zProtobuf (flattened)) r&r!r%_ViewGrpcProtobuf__content_types_grpcheaders'_ViewGrpcProtobuf__valid_grpc_encodingsrrrrr ) rrfr'r r(unknown_metadataapplicabble_rulesh grpc_encoding text_itertitlers r!__call__zViewGrpcProtobuf.__call__s&!00++**|1  444 4 ?#/// ((9D77744Q7$#{{99#0' I E##{{99'I +E .y9Ii9 ? $ ; ;A >  ?*  G  s)2C5 C"CC" C2+C--C2c lt|r||jvryt|r||jvryy)Nr>g?r)r8r+#_ViewGrpcProtobuf__content_types_pb)rrfr'r r(r.s r!render_priorityz ViewGrpcProtobuf.render_priorityTs5 :,$*C*CC :,$*A*AAr r)r!zViewConfig | Nonerr)rrr^http.Message | Noner flow.Flow | Nonerr) rfr]r'r2r r9r(r8rzcontentviews.TViewResult) rfr]r'r2r r9r(r8rrX) rrrrrr6r+r-rr&r4r7 __classcell__)r"s@r!rrs1 !D    +++%+ + & +b$(!%,0 < < ! <  < * <  "< D$(!%,0 !    *  r r)d)rzIterable[tuple[str, ...]]rzIterator[base.TViewLine])rz)Generator[tuple[bool, bytes], None, None])r^r]rr7rr)r)rfr]rr7rr) __future__rrrmcollections.abcrrr dataclassesrrrUr mitmproxyr r r rmitmproxy.contentviewsrmitmproxy.net.encodingrrrrr rrrViewrrr r!rCs" %$$!" ')f )f )\)D"."p    -  (     -  ( 0 MM MYtyyYr