o
7aI
@ s d Z dZdgZddlmZ zddlmZ W n ey/ Z zG dd deZW Y dZ[ndZ[ww ddl Z ddl
Z
e j dd \Z Z
Ze d koMe
d
koMed kZe d koUe
d kZe d ko]e
d kZdd lmZmZmZmZmZ dd
lmZmZ ddlm Z m Z m Z dZ G dd deZ G dd de Z!e d kre
d
kresddl"Z"e"#dZ$e$e!_$e"#de"j%Z&e&e _&ddlm'Z'm(Z( dd Z)dd Z*e)e _)e*e _*dZdS dS dS dS )zCUse the HTMLParser library to parse HTML files that aren't too bad.ZMITHTMLParserTreeBuilder )
HTMLParser)HTMLParseErrorc @ s e Zd ZdS )r N)__name__
__module__ __qualname__ r r 9/usr/lib/python3/dist-packages/bs4/builder/_htmlparser.pyr s r N )CDataComment DeclarationDoctypeProcessingInstruction)EntitySubstitution
UnicodeDammit)HTMLHTMLTreeBuilderSTRICTz html.parserc @ s| e Zd ZdZdZdZdd Zdd Zdd Zd d d Z d d
dZ
dd Z dd Z dd Z
dd Zdd Zdd Zdd Zd S ) BeautifulSoupHTMLParserzA subclass of the Python standard library's HTMLParser class, which
listens for HTMLParser events and translates them into calls
to Beautiful Soup's tree construction API.
ignorereplacec O s2 | d| j| _tj| g|R i | g | _dS )a Constructor.
:param on_duplicate_attribute: A strategy for what to do if a
tag includes the same attribute more than once. Accepted
values are: REPLACE (replace earlier values with later
ones, the default), IGNORE (keep the earliest value
encountered), or a callable. A callable must take three
arguments: the dictionary of attributes already processed,
the name of the duplicate attribute, and the most recent value
encountered.
on_duplicate_attributeN)popREPLACEr r __init__ already_closed_empty_element)selfargskwargsr r r r A s
z BeautifulSoupHTMLParser.__init__c C s t | dS )a In Python 3, HTMLParser subclasses must implement error(), although
this requirement doesn't appear to be documented.
In Python 2, HTMLParser implements error() by raising an exception,
which we don't want to do.
In any event, this method is called only on very strange
markup and our best strategy is to pretend it didn't happen
and keep going.
N)warningswarn)r msgr r r error[ s z BeautifulSoupHTMLParser.errorc C s | j ||dd}| | dS )zHandle an incoming empty-element tag.
This is only called when the markup looks like .
:param name: Name of the tag.
:param attrs: Dictionary of the tag's attributes.
F)handle_empty_elementN)handle_starttag
handle_endtag)r nameattrstagr r r handle_startendtagh s z*BeautifulSoupHTMLParser.handle_startendtagTc C s i }|D ]0\}}|du rd}||v r.| j }|| jkrn|d| jfv r'|||< n |||| n|||< d}q| \} }
| jj|dd|| |
d} | r]| jr_|ra| j|dd | j | dS dS dS dS )a3 Handle an opening tag, e.g. ''
:param name: Name of the tag.
:param attrs: Dictionary of the tag's attributes.
:param handle_empty_element: True if this tag is known to be
an empty-element tag (i.e. there is not expected to be any
closing tag).
N z"")
sourceline sourceposF)check_already_closed)
r IGNOREr getpossoupr' Zis_empty_elementr( r append) r r) r* r& Z attr_dictkeyvalueZon_dupe attrvaluer. r/ r+ r r r r' w s,
z'BeautifulSoupHTMLParser.handle_starttagc C s. |r|| j v r| j | dS | j| dS )zHandle a closing tag, e.g. ''
:param name: A tag name.
:param check_already_closed: True if this tag is expected to
be the closing portion of an empty-element tag,
e.g. ''.
N)r remover3 r( )r r) r0 r r r r( s z%BeautifulSoupHTMLParser.handle_endtagc C s | j | dS )z4Handle some textual data that shows up between tags.N)r3 handle_datar datar r r r9 s z#BeautifulSoupHTMLParser.handle_datac C s | drt|dd}n| dr t|dd}nt|}d}|dk rN| jjdfD ]!}|s1q,z
t|g|}W q, tyM } zW Y d}~q,d}~ww |skzt|}W n t t
fyj } zW Y d}~nd}~ww |pnd}| | dS )zHandle a numeric character reference by converting it to the
corresponding Unicode character and treating it as textual
data.
:param name: Character number, possibly in hexadecimal.
x XN z windows-1252u �)
startswithintlstripr3 original_encoding bytearraydecodeUnicodeDecodeErrorchr
ValueError
OverflowErrorr9 )r r) Z real_namer; encodinger r r handle_charref s2
z&BeautifulSoupHTMLParser.handle_charrefc C s0 t j|}|dur
|}nd| }| | dS )zHandle a named entity reference by converting it to the
corresponding Unicode character(s) and treating it as textual
data.
:param name: Name of the entity reference.
Nz&%s)r ZHTML_ENTITY_TO_CHARACTERgetr9 )r r) characterr; r r r handle_entityref s
z(BeautifulSoupHTMLParser.handle_entityrefc C & | j | j | | j t dS )zOHandle an HTML comment.
:param data: The text of the comment.
N)r3 endDatar9 r r: r r r handle_comment
z&BeautifulSoupHTMLParser.handle_commentc C s6 | j |tdd }| j | | j t dS )zYHandle a DOCTYPE declaration.
:param data: The text of the declaration.
zDOCTYPE N)r3 rQ lenr9 r r: r r r handle_decl s
z#BeautifulSoupHTMLParser.handle_declc C sN | drt}|tdd }nt}| j | j| | j| dS )z{Handle a declaration of unknown type -- probably a CDATA block.
:param data: The text of the declaration.
zCDATA[N)upperr@ r
rT r r3 rQ r9 )r r; clsr r r unknown_decl s
z$BeautifulSoupHTMLParser.unknown_declc C rP )z\Handle a processing instruction.
:param data: The text of the instruction.
N)r3 rQ r9 r r: r r r handle_pi rS z!BeautifulSoupHTMLParser.handle_piN)T)r r r __doc__r1 r r r% r, r' r( r9 rL rO rR rU rX rY r r r r r 7 s
4'
r c sR e Zd ZdZdZdZeZeee gZ
dZ d fdd Z d ddZ
d d
Z ZS )
r zpA Beautiful soup `TreeBuilder` that uses the `HTMLParser` parser,
found in the Python standard library.
FTNc s t }dD ]}||v r||}|||< qtt| jdi | |p#g }|p'i }|| tr5ts5d|d< tr;d|d< ||f| _ dS )a Constructor.
:param parser_args: Positional arguments to pass into
the BeautifulSoupHTMLParser constructor, once it's
invoked.
:param parser_kwargs: Keyword arguments to pass into
the BeautifulSoupHTMLParser constructor, once it's
invoked.
:param kwargs: Keyword arguments for the superclass constructor.
)r FstrictZconvert_charrefsNr )
dictr superr r updateCONSTRUCTOR_TAKES_STRICT CONSTRUCTOR_STRICT_IS_DEPRECATED"CONSTRUCTOR_TAKES_CONVERT_CHARREFS parser_args)r rb Z
parser_kwargsr! Zextra_parser_kwargsargr6 __class__r r r 3 s
z HTMLParserTreeBuilder.__init__c c s^ t |tr|dddfV dS |g}|g}||g}t|||d|d}|j|j|j|jfV dS )a Run any preliminary steps necessary to make incoming markup
acceptable to the parser.
:param markup: Some markup -- probably a bytestring.
:param user_specified_encoding: The user asked to try this encoding.
:param document_declared_encoding: The markup itself claims to be
in this encoding.
:param exclude_encodings: The user asked _not_ to try any of
these encodings.
:yield: A series of 4-tuples:
(markup, encoding, declared encoding,
has undergone character replacement)
Each 4-tuple represents a strategy for converting the
document to Unicode and parsing it. Each strategy will be tried
in turn.
NFT)known_definite_encodingsuser_encodingsZis_htmlexclude_encodings)
isinstancestrr markuprC Zdeclared_html_encodingZ contains_replacement_characters) r rk Zuser_specified_encodingZdocument_declared_encodingrh rf rg Z
try_encodingsZdammitr r r prepare_markupO s$
z$HTMLParserTreeBuilder.prepare_markupc
C sl | j \}}t|i |}| j|_z || | W n ty0 } z ttd |d}~ww g |_ dS )z{Run some incoming markup through some parsing process,
populating the `BeautifulSoup` object in self.soup.
a* Python's built-in HTMLParser cannot parse the given document. This is not a bug in Beautiful Soup. The best solution is to install an external parser (lxml or html5lib), and use Beautiful Soup with that parser. See http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser for help.N)
rb r r3 feedcloser r" r# RuntimeWarningr )r rk r r! parserrK r r r rm s
zHTMLParserTreeBuilder.feed)NN)NNN)r r r rZ Zis_xmlZ picklable
HTMLPARSERNAMEr r ZfeaturesZTRACKS_LINE_NUMBERSr rl rm
__classcell__r r rd r r & s
1zQ\s*((?<=[\'"\s])[^\s/>][^\s/=>]*)(\s*=+\s*(\'[^\']*\'|"[^"]*"|(?![\'"])[^>\s]*))?a
<[a-zA-Z][-.a-zA-Z0-9:_]* # tag name
(?:\s+ # whitespace before attribute name
(?:[a-zA-Z_][-.:a-zA-Z0-9_]* # attribute name
(?:\s*=\s* # value indicator
(?:'[^']*' # LITA-enclosed value
|\"[^\"]*\" # LIT-enclosed value
|[^'\">\s]+ # bare value
)
)?
)
)*
\s* # trailing whitespace
)tagfindattrfindc C s* d | _ | |}|dk r|S | j}||| | _ g }t||d }|s(J d| }||d | | _}||k r| jrGt ||}nt
||}|sPnT| ddd\} }
} |
s_d } n-| d d d krq| dd ksn | d d d kr| dd krn n| dd } | r| | } |
| | f | }||k s=||| } | d vr| \}
}d
| j v r|
| j d
}
t| j | j d
}n|t| j }| jr| d ||| d d f | ||| |S | d
r| || |S | || || jv r| | |S )Nr z#unexpected call to parse_starttag()r r
'")>/>
z junk characters in start tag: %r r{ )Z__starttag_textZcheck_for_whole_start_tagrawdatart matchendlowerZlasttagr[ ru attrfind_tolerantgroupZunescaper4 stripr2 countrT rfindr% r9 endswithr, r' ZCDATA_CONTENT_ELEMENTSset_cdata_mode)r iendposr~ r* r kr+ mattrnamerestr7 r linenooffsetr r r parse_starttag sj
&
r c C s$ | | _td| j tj| _d S )Nz )r Z
cdata_elemrecompileIZ interesting)r elemr r r r s
r T)+rZ Z __license____all__Z html.parserr r ImportErrorrK Exceptionsysr" version_infomajorminorreleaser_ r` ra Z bs4.elementr
r r r r Z
bs4.dammitr r Z bs4.builderr r r rq r r r r r VERBOSEZlocatestarttagendrt ru r r r r r r sR po
7