o _cc.@sdZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZed ejZed ejZed ejZed ejejBZed ejejBZedejZdZ   d7dedeede dedef ddZ!d8dedeede fddZ"d9dededeedefddZ#ed ejZ$d8dedeedefd!d"Z%   d:ded#eedeedeedef d$d%Z& d;ded#eedeedefd&d'Z' (  dded.ededefd/d0Z*   1d?ded.eded2eede ee+eeff d3d4Z,dedefd5d6Z-dS)@z( Functions for dealing with markup text N)name2codepoint)IterableMatchAnyStrOptionalPatternTupleUnion)urljoin) to_unicode)safe_url_string) StrOrByteszI&((?P[a-z\d]+)|#(?P\d+)|#x(?P[a-f\d]+))(?P;?)z<[a-zA-Z\/!].*?>z5]*href\s*=\s*[\"\']\s*([^\"\'\s]+)\s*[\"\']z}]*http-equiv[^>]*refresh[^>]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)z]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)[^>]*?\shttp-equiv\s*=[^>]*refreshz<((?P.*?)(?P\]\]>))z Tutf-8textkeepremove_illegalencodingreturncs*dtdtffdd }t|t||S)uRemove entities from the given `text` by converting them to their corresponding unicode character. `text` can be a unicode string or a byte string encoded in the given `encoding` (which defaults to 'utf-8'). If `keep` is passed (with a list of entity names) those entities will be kept (they won't be removed). It supports both numeric entities (``&#nnnn;`` and ``&#hhhh;``) and named entities (such as `` `` or ``>``). If `remove_illegal` is ``True``, entities that can't be converted are removed. If `remove_illegal` is ``False``, entities that can't be converted are kept "as is". For more information see the tests. Always returns a unicode string (with the entities removed). >>> import w3lib.html >>> w3lib.html.replace_entities(b'Price: £100') 'Price: \xa3100' >>> print(w3lib.html.replace_entities(b'Price: £100')) Price: £100 >>> mrcs|}d}|drt|dd}n-|dr t|dd}n |dr@|d}|vr4|dSt|p?t|}|durhzd|krOdkrZnn t|fd WSt|WSt ygYnwrq|d rqd S|dS) Ndec hexnamedrcp1252 semicolon) groupdictgetintlowergrouprbytesdecodechr ValueError)rgroupsnumber entity_namerrr5/usr/local/lib/python3.10/dist-packages/w3lib/html.pyconvert_entityEs,       z(replace_entities..convert_entity)rstr_ent_resubr )rrrrr.rr,r-replace_entities$s!r2cCsttt||SN)boolr0searchr )rrrrr- has_entitiesfsr6rtokencCst|t||S)acReplace all markup tags found in the given `text` by the given token. By default `token` is an empty string so it just removes all tags. `text` can be a unicode string or a regular string encoded as `encoding` (or ``'utf-8'`` if `encoding` is not given.) Always returns a unicode string. Examples: >>> import w3lib.html >>> w3lib.html.replace_tags('This text contains some tag') 'This text contains some tag' >>> w3lib.html.replace_tags('

Je ne parle pas fran\xe7ais

', ' -- ', 'latin-1') ' -- Je ne parle pas -- fran\xe7ais -- -- ' >>> )_tag_rer1r )rr7rrrr- replace_tagsjsr9z|$)cCst||}td|S)zRemove HTML Comments. >>> import w3lib.html >>> w3lib.html.remove_comments(b"test whatever") 'test whatever' >>> r)r _REMOVECOMMENTS_REr1)rrutextrrr-remove_commentss r< which_onescsrrtdddDddDdtdtffdd d tdtffd d }d }t|tjtjB}||t ||S) a;Remove HTML Tags only. `which_ones` and `keep` are both tuples, there are four cases: ============== ============= ========================================== ``which_ones`` ``keep`` what it does ============== ============= ========================================== **not empty** empty remove all tags in ``which_ones`` empty **not empty** remove all tags except the ones in ``keep`` empty empty remove all tags **not empty** **not empty** not allowed ============== ============= ========================================== Remove all tags: >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags(doc) 'This is a link: example' >>> Keep only some tags: >>> w3lib.html.remove_tags(doc, keep=('div',)) '
This is a link: example
' >>> Remove only specific tags: >>> w3lib.html.remove_tags(doc, which_ones=('a','b')) '

This is a link: example

' >>> You can't remove some and keep some: >>> w3lib.html.remove_tags(doc, which_ones=('a',), keep=('p',)) Traceback (most recent call last): ... ValueError: Cannot use both which_ones and keep >>> z#Cannot use both which_ones and keepcSh|]}|qSrr#.0tagrrr- zremove_tags..cSr>rr?r@rrr-rCrDrBrcs|}r |vS|vSr3r?)rB)rr=rr- will_removesz remove_tags..will_removercs |d}|r dS|dS)Nrr)r$)rrB)rErr- remove_tags zremove_tags..remove_tagz/]+).*?>) r(r/r4rrecompileDOTALL IGNORECASEr1r )rr=rrrGregexretagsr)rr=rEr- remove_tagss1rNcCsFt||}|r!ddd|D}t|tjtjB}|d|}|S)aRemove tags and their content. `which_ones` is a tuple of which tags to remove including their content. If is empty, returns the string unmodified. >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags_with_content(doc, which_ones=('b',)) '' >>> |c Ss$g|]}d|d|d|dqS)|rr@rrr- s$z,remove_tags_with_content..r)r joinrHrIrJrKr1)rr=rr;tagsrMrrr-remove_tags_with_contents  rT   replace_bycCs*t||}|D] }||t||}q|S)a$Remove escape characters. `which_ones` is a tuple of which escape characters we want to remove. By default removes ``\n``, ``\t``, ``\r``. `replace_by` is the string to replace the escape characters by. It defaults to ``''``, meaning the escape characters are removed. )r replace)rr=rYrr;ecrrr-replace_escape_charss r\c Cspdtdtdttttffdd}t||}d}||tD]}t|tr.|t|||d7}q|| d7}q|S) a` This function receives markup as a text (always a unicode string or a UTF-8 encoded string) and does the following: 1. removes entities (except the ones in `keep`) from any part of it that is not inside a CDATA 2. searches for CDATAs and extracts their text (if any) without modifying it. 3. removes the found CDATAs txtpatternrcssNd}||D]}|d\}}|||V|V|}q||dVdS)NrrF)finditerspan)r]r^offsetmatchZmatch_sZmatch_errr-_get_fragmentssz&unquote_markup.._get_fragmentsrr,Zcdata_d) r/rrr rr _cdata_re isinstancer2r$)rrrrrcr;Zret_textfragmentrrr-unquote_markups"   rgbaseurlcCs<t||}t|}|rtt|t|d|dSt|S)zReturn the base url if declared in the given HTML `text`, relative to the given base url. If no base url is found, the given `baseurl` is returned. rF)r)r _baseurl_rer5r r r$)rrhrr;rrrr- get_base_url0s rjscriptnoscript ignore_tagscCszt||}Wn tyt|wt||}tt|}t|p(t|}|rFt | d}t | d d|}t ||}||fSdS)aYReturn the http-equiv parameter of the HTML meta element from the given HTML text and return a tuple ``(interval, url)`` where interval is an integer containing the delay in seconds (or zero if not present) and url is a string with the absolute url to redirect. If no meta redirect is found, ``(None, None)`` is returned. r"urlz "')NN)r UnicodeDecodeErrorprintrTr<r2_meta_refresh_rer5_meta_refresh_re2floatr$r stripr )rrhrrnr;rintervalrorrr-get_meta_refreshDs    rwcCs |tS)a Strip all leading and trailing space characters (as defined in https://www.w3.org/TR/html5/infrastructure.html#space-character). Such stripping is useful e.g. for processing HTML element attributes which contain URLs, like ``href``, ``src`` or form ``action`` - HTML5 standard defines them as "valid URL potentially surrounded by spaces" or "valid non-empty URL potentially surrounded by spaces". >>> strip_html5_whitespace(' hello\n') 'hello' )ruHTML5_WHITESPACE)rrrr-strip_html5_whitespaceds ry)rTrr3)rN)rrN)rN)rUrN)rTN)rr)rrrk).__doc__rH html.entitiesrtypingrrrrrrr urllib.parser Z w3lib.utilr Z w3lib.urlr Z w3lib._typesr rIrKr0rJr8Irirrrsrdrxr/r4r2r6r9r:r<rNrTr\rgrjrtrwryrrrr-s $       B  I   )