o K&b9y@s:ddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZmZddlmZddlmZmZmZmZmZmZmZmZmZddlmZmZddl m!Z!dd l"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(d Z)d Z*d Z+dZ,e-Z.dZ/dZ0dZ1dZ2dZ3dZ4e#5Z6Gdddej7Z8Gdddej9Z:Gdddej;Z< ddee=efde=ddfddZ>d e=ddfd!d"Z?e d#d$Z@d%d&ZA 'dd(ee=efd)ee=efd e=dee=effd*d+ZBedd,de=fd-d.ZCedd,dee=e=ffd/d0ZDedd,de=deEfd1d2ZFedd,de=deEfd3d4ZGedd,dd6e=deEfd7d8ZHd e=deEfd9d:ZId;e=deEfd\d+\.\d+) (LTS )?\((?P\w+).*)z169.254.169.254Zmetadataz[fd00:ec2::254]zhttp://archive.ubuntu.comzhttps://esm.ubuntu.comzhttp://api.snapcraft.iozhttps://api.snapcraft.ioc@s0eZdZejdejdiZdejdefddZ dS) LogFormatterzERROR: %(message)szDEBUG: %(message)srecordreturncCs |j|jd}t||S)Nz %(message)s)FORMATSgetZlevelnologging Formatterformat)selfrZlog_fmtr!//usr/lib/python3/dist-packages/uaclient/util.pyr9szLogFormatter.formatN) __name__ __module__ __qualname__rZERRORDEBUGrZ LogRecordstrrr!r!r!r"r2s rcs eZdZdZfddZZS)DatetimeAwareJSONEncoderzBA json.JSONEncoder subclass that writes out isoformat'd datetimes.cs t|tjr |St|SN) isinstancedatetimeZ isoformatsuperdefault)r o __class__r!r"r-As  z DatetimeAwareJSONEncoder.default)r#r$r%__doc__r- __classcell__r!r!r/r"r(>sr(cs,eZdZdZfddZeddZZS)DatetimeAwareJSONDecodera, A JSONDecoder that parses some ISO datetime strings to datetime objects. Important note: the "some" is because we seem to only be able extend Python's json library in a way that lets us convert string values within JSON objects (e.g. '{"lastModified": "2019-07-25T14:35:51"}'). Strings outside of JSON objects (e.g. '"2019-07-25T14:35:51"') will not be passed through our decoder. (N.B. This will override any object_hook specified using arguments to it, or used in load or loads calls that specify this as the cls.) cs.d|vr |dtj|d|ji|dS)N object_hook)popr,__init__r4)r argskwargsr/r!r"r6Us z!DatetimeAwareJSONDecoder.__init__c CsL|D]\}}t|tr#zt|}Wn ty|}Ynw|||<q|Sr))itemsr*r'parse_rfc3339_date ValueError)r.keyvalue new_valuer!r!r"r4Zs   z$DatetimeAwareJSONDecoder.object_hook)r#r$r%r1r6 staticmethodr4r2r!r!r/r"r3Gs  r3 orig_accessseriesrcCstt|td|vgstd||durtdn|}|di}|di|i}|D]\}}|d|}t|trG| |q1||d|<q1dS)aApply series-specific overrides to an entitlement dict. This function mutates orig_access dict by applying any series-overrides to the top-level keys under 'entitlement'. The series-overrides are sparse and intended to supplement existing top-level dict values. So, sub-keys under the top-level directives, obligations and affordance sub-key values will be preserved if unspecified in series-overrides. To more clearly indicate that orig_access in memory has already had the overrides applied, the 'series' key is also removed from the orig_access dict. :param orig_access: Dict with original entitlement access details Z entitlementz?Expected entitlement access dict. Missing "entitlement" key: {}NrA) allr*dict RuntimeErrorrget_platform_inforr5r9update)r@rAZ series_nameZorig_entitlementZ overridesr<r=Zcurrentr!r!r"apply_series_overridesgs   rGpathc CsFzt|WdSty"}z|jtkr|WYd}~dSd}~wwr))osunlinkOSErrorerrnor)rHer!r!r"del_files rNc cs~ddtjD}|stddVdS|d}|j}|tjkr(dVdS|dz dVW||dS||w)a A context manager that disables logging to console in its body N.B. This _will not_ disable console logging if it finds the console handler is configured at DEBUG level; the assumption is that this means we want as much output as possible, even if it risks duplication. This context manager will allow us to gradually move away from using the logging framework for user-facing output, by applying it to parts of the codebase piece-wise. (Once the conversion is complete, we should have no further use for it and it can be removed.) (Note that the @contextmanager decorator also allows this function to be used as a decorator.) cSsg|] }|jdkr|qS)z ua-console)name).0Zhandlerr!r!r" s z*disable_log_to_console..z0disable_log_to_console: no console handler foundNri)rZ getLoggerhandlersdebuglevelr&ZsetLevel)Zpotential_handlersZconsole_handlerZ old_levelr!r!r"disable_log_to_consoles"   rUcsfdd}|S)aDecorator to retry on exception for retry_sleeps. @param retry_sleeps: List of sleep lengths to apply between retries. Specifying a list of [0.5, 1] tells subp to retry twice on failure; sleeping half a second before the first retry and 1 second before the second retry. @param exception: The exception class to catch and retry for the provided retry_sleeps. Any other exception types will not be caught by the decorator. cstfdd}|S)Nc sx} z|i|WSy:}z!|s|dt|}tt||t|dWYd}~nd}~wwq)NT Retrying %d more times.r)copylenrrSr'timesleepr5)r7r8ZsleepsrM retry_msg) exceptionf retry_sleepsr!r" decorators z)retry..wrapper..decorator)r)r]r_r\r^)r]r"wrappers zretry..wrapperr!)r\r^rar!r`r"retrys rb orig_dictnew_dictc Csi}|D]M\}}||t}|s|n|d|}t|tr8||vr3t||||d}|r2|||<qt||<q||krSd|dt|d} tt | |||<q|D] \}}||vrd|||<qX|S)zZkey_pathZ sub_deltalogr!r!r"rjs@    rj)maxsizecCs||jr|jdid}|r|S|d}tt|fD]}tj|r0t| d}|r0|Sqt t }t |||S)z=Get system's unique machine-id or create our own in data_dir.ZmachineTokenInfoZ machineIdz machine-id )Z machine_tokenrZ data_pathETC_MACHINE_IDDBUS_MACHINE_IDrIrHexists load_filerstripr'uuidZuuid4 write_file)ZcfgZcfg_machine_idZfallback_machine_id_filerHcontentZ machine_idr!r!r"get_machine_ids     rwcCst}|dddd}|d}d|vrdj|d}tdd |}||d <tt|}|s9td |d|| }| |d |d  dt }|j|d<tddg\}}||d<|S)z Returns a dict of platform information. N.B. This dict is sent to the contract server, which requires the distribution, type and release keys. NAMEZUNKNOWNZLinux)Z distributiontypeZVERSION, z{} ({})z\.\d LTSz LTSversionzChecks to see if this code running in a container of some sortZischrootF)zsystemd-detect-virtz--quietz --containerT)Zcontainer_typezsystemd/container) rrProcessExecutionErrorIOErrorrKrIrHjoinrq)rfilenamerHr!r!r" is_containerVs$   rcCstj|o t|tjSr))rIrHisfileaccessX_OKrgr!r!r"is_exeqsrurlcCsRzt|}Wn tyYdSw|jdvrdSz|jWdSty(YdSw)NF)httpshttpT)rr;ZschemeZport)rZ parsed_urlr!r!r"is_service_urlvs    rTrdecodecCsHtd|t|d }|}Wdn1swY|dS)z!Read filename and decode content.zReading file: %srbNutf-8)rrSopenreadr)rrstreamrvr!r!r"rrs    rr release_filecCsH|sd}i}t|D]}|dd\}}|r!|d||<q |S)Nz/etc/os-release=")rr splitlinesrr)rdataliner<r=r!r!r"r~sr~msg valid_choicescCsRd}d|ddd|D} t|td}||vr# |St|q)aGInteractive prompt message, returning a valid choice from msg. Expects a structured msg which designates choices with square brackets [] around the characters which indicate a valid choice. Uppercase and lowercase responses are allowed. Loop on invalid choices. :return: Valid response character chosen. rcz{} is not one of: {}rzcSsg|]}|qSr!)upper)rPchoicer!r!r"rQsz"prompt_choices..Tz> )rreventinfoinputr)rrr=Z error_msgr!r!r"prompt_choicess    rF assume_yescCs2|rdS|s tj}t|}|dvrdSdS)aV Display a confirmation prompt, returning a bool indicating the response :param msg: String custom prompt text to emit from input call. :param assume_yes: Boolean set True to skip confirmation input and return True. This function will only prompt a single time, and defaults to "no" (i.e. it returns False). T)yZyesF)rZ PROMPT_YES_NOrrr)rrr=r!r!r"prompt_for_confirmations r http_proxy https_proxycCsi}|r||d<|r||d<dtt}dD]}tj|}|r2dtt|dtt}qt d||tjd<|tjd<|rVt |}t |}t |dSdS) a! Configure urllib to use http and https proxies. :param http_proxy: http proxy to be used by urllib. If None, it will not be configured :param https_proxy: https proxy to be used by urllib. If None, it will not be configured rr,)no_proxyNO_PROXYzSetting no_proxy: %srrN)rsortedUA_NO_PROXY_URLSrIenvironrsetrunionrrSr ProxyHandler build_openerZinstall_opener)rrZ proxy_dictrZenv_varZ proxy_value proxy_handleropenerr!r!r"configure_web_proxys.       rrheadersmethodtimeoutc s:|r|sd}tj|||d}dfddtD}ttd|p%d|||r.|dndd }z tj ||d d }Wnt j yW}z |d }WYd}~nd}~wwt d  dj} d tjddvrvt| } dfddtjD}ttd|pd||| |r| jfS)NZPOST)rrrrzcsg|] }d||qSz '{}': '{}')rrPk)rr!r"rQszreadurl..z'URL [{}]: {}, headers: {{{}}}, data: {}ZGETrFrTbodyzapplication/jsonz Content-typerccsg|] }d|j|qSr)rrr)respr!r"rQsz0URL [{}] response: {}, headers: {{{}}}, data: {})rRequestrrrrSrkrrZurlopenrZ HTTPErrorsetattrrrr'rrjsonloads) rrrrrreqZsorted_header_strZhttp_error_foundrMrvr!)rrr"readurlsT    rr7rcscaptureenvc Csdd|D}|r|tj|durdg}td|}ztj|tjtj|d}|j|d\}} Wn%t yWzt j ||j | d| dd tyVt j |d ww|j |vrmt j ||j | d| dd |rxtd ||j | | d| dfS) aRun a command and return a tuple of decoded stdout, stderr. @param args: A list of arguments to feed to subprocess.Popen @param rcs: A list of allowed return_codes. If returncode not in rcs raise a ProcessExecutionError. @param capture: Boolean set True to log the command and response. @param timeout: Optional float indicating number of seconds to wait for subp to return. @param env: Optional dictionary of environment variable to pass to Popen. @return: Tuple of utf-8 decoded stdout, stderr @raises ProcessExecutionError on invalid command or returncode not in rcs. @raises subprocess.TimeoutError when timeout specified and the command exceeds that number of seconds. cSs$g|]}t|tr |n|dqS)r)r*bytesencode)rPxr!r!r"rQ5sz_subp..Nr )stdoutstderrrrr)cmdZ exit_coderr)rzRan cmd: %s, rc: %s stderr: %s)rFrIrrkr subprocessPopenPIPEZ communicaterKrr returncoderUnboundLocalErrorrrS) r7rrrrZ bytes_argsZ redacted_cmdprocrerrr!r!r"_subpsP      rr^c Cs|dur|nd} zt|||||d\}}W||fStjyd}z<|r>ttt|d|j |j } t t| |sAdt |} ttt|| t |dWYd}~nd}~wwq )aRun a command and return a tuple of decoded stdout, stderr. @param subp: A list of arguments to feed to subprocess.Popen @param rcs: A list of allowed return_codes. If returncode not in rcs raise a ProcessExecutionError. @param capture: Boolean set True to log the command and response. @param timeout: Optional float indicating number of seconds to wait for a subp call to return. @param retry_sleeps: Optional list of sleep lengths to apply between retries. Specifying a list of [0.5, 1] instructs subp to retry twice on failure; sleeping half a second before the first retry and 1 second before the next retry. @param env: Optional dictionary of environment variables to provide to subp. @return: Tuple of utf-8 decoded stdout, stderr @raises ProcessExecutionError on invalid command or returncode not in rcs. @raises subprocess.TimeoutError when timeout specified and the command exceeds that number of seconds. NT)rzStderr: {} Stdout: {}rVr)rWrrrrrSrkr'rrrZwarningrXrYrZr5) r7rrrr^rrrrMrr[r!r!r"r]s&  rprogramcCsrtjj|vr t|r |SddtjddtjD}dd|D}|D]}tj||}t|r6|Sq%dS)z;Find whether the provided program is executable in our PATHcSsg|]}|dqS)r)rrPpr!r!r"rQs zwhich..PATHrccSsg|]}tj|qSr!)rIrHabspathrr!r!r"rQsN) rIrHseprrrrpathsepr)rpathsZnormalized_pathsrHZ program_pathr!r!r"whichs rrvmodecCs^td|t|d}||d|Wdn1s"wYt||dS)zWrite content to the provided filename encoding it if necessary. @param filename: The full path of the file to write. @param content: The content to write to the file. @param mode: The filesystem mode to set on the file. zWriting file: %swbrN)rrSrwriterflushrIchmod)rrvrZfhr!r!r"rus   ru file_pathcCs*tj|rtd|t|dSdS)z(Contract\s+value\s+for\s+'resourceToken'\s+changed\s+to\s+).*z(\'resourceToken\': \')[^\']+z(\'contractToken\': \')[^\']+zF(https://contracts.canonical.com/v1/resources/livepatch\?token=)[^\s]+z(\"identityToken\": \")[^\"]+zT(response:\s+http://metadata/computeMetadata/v1/instance/service-accounts.*data: ).* redact_regexscCs |}|D] }t|d|}q|S)z4Redact known sensitive information from log content.z\g<1>)rr)rlrZ redacted_logZ redact_regexr!r!r"rksrkinstalled_pkgscCsZtjtsdS|durdSz tttd}Wn ty#YdSwt | |dkS)ahCheck if the system needs to be rebooted. :param installed_pkgs: If provided, verify if the any packages in the list are present on /var/run/reboot-required.pkgs. If that param is provided, we will only return true if we have the reboot-required marker file and any package in reboot-required.pkgs file. FNTrnr) rIrHrqREBOOT_FILE_CHECK_PATHrrrREBOOT_PKGS_FILE_PATHrFileNotFoundErrorrX intersection)rZreboot_required_pkgsr!r!r" should_reboots   r package_namecCs:ztdd|g\}}d||vWStjyYdSw)Nr}z-lzii {} F)rrrr)rr_r!r!r" is_installeds rmsg_opscCsH|sdS|D]}t|trt|q|\}}|di|s!dSqdS)ahEmit messages to the console for user interaction :param msg_op: A list of strings or tuples. Any string items are printed. Any tuples will contain a callable and a dict of args to pass to the callable. Callables are expected to return True on success and False upon failure. :return: True upon success, False on failure. TFNr!)r*r'rr)rZmsg_opZfunctorr7r!r!r"handle_message_operationss   rdt_strcCsDtdd|}tdd|}|dd}tdd|}tj|d S) aT Parse a datestring in rfc3339 format. Originally written for compatibility with golang's time.MarshalJSON function. Also handles output of pythons isoformat datetime method. This drops subseconds. :param dt_str: a date string in rfc3339 format :return: datetime.datetime object of time represented by dt_str z(\d{2}:\d{2}:\d{2})\.\d+z\g<1>z(\d{2}:\d{2}:\d{2})$z\g<1>ZZz+00:00z(-|\+)(\d{2}):(\d{2})$z\g<1>\g<2>\g<3>z%Y-%m-%dT%H:%M:%S%z)rrreplacer+strptime)r Zdt_str_without_subsecondsZ dt_str_with_zZdt_str_without_zZdt_str_with_pythonish_tzr!r!r"r:6s  r:protocolproxytest_urlc Cs|sdSt|s t|tj|dd}t||i}t|}z|||WStj t j fyd}z,t t |dt|}t tjj|||dWdn1sVwYt|d}~ww)NZHEAD)rreason)rrr)rrZProxyInvalidUrlrrrrrsocketrrZURLErrorrUgetattrr'rrZERROR_USING_PROXYrZProxyNotWorkingError)r rrrrrrMrr!r!r"validate_proxyms,    rmessagecCsPtjjdusdtjjvr&|dd}|tjdd}|tjdd}|S)z} Verify if the system can output unicode characters and if not, remove those characters from the message string. NzUTF-8u—-rrc)sysrencodingrr rZ OKGREEN_CHECKZFAIL_X)rr!r!r"handle_unicode_characterss  rr))rc)r)T)rcF)NFNN)NFNNN)r)br+rrrIrrrrrYrt contextlibrrLr functoolsrrZ http.clientrtypingrrr r r r r rrZurllibrrZ urllib.parserZuaclientrrrrZuaclient.typesrrrrorpobjectrirrZPROXY_VALIDATION_APT_HTTP_URLZPROXY_VALIDATION_APT_HTTPS_URLZPROXY_VALIDATION_SNAP_HTTP_URLZPROXY_VALIDATION_SNAP_HTTPS_URLZget_event_loggerrrrZ JSONEncoderr(Z JSONDecoderr3r'rGrNrUrbrjrwrEboolrrrrrrrr~rrrrrrfloatrrrrurrZREDACT_SENSITIVE_LOGSrkrrrr:rrr!r!r!r"s>   ,    !  $ )     !+ "  )  3   @    .)   7