uYf, d Z ddlZddlmZ ddlmZ ddlmZ ddlmZ ej rddlm Z dd lm
Z
dd
lm Z d Z d Z
G d
de Z G d de Z G d de Z G d de Zy)aK A quantile sketch with relative-error guarantees. This sketch computes
quantile values with an approximation error that is relative to the actual
quantile value. It works on both negative and non-negative input values.
For instance, using DDSketch with a relative accuracy guarantee set to 1%, if
the expected quantile value is 100, the computed quantile value is guaranteed to
be between 99 and 101. If the expected quantile value is 1000, the computed
quantile value is guaranteed to be between 990 and 1010.
DDSketch works by mapping floating-point input values to bins and counting the
number of values for each bin. The underlying structure that keeps track of bin
counts is store.
The memory size of the sketch depends on the range that is covered by the input
values: the larger that range, the more bins are needed to keep track of the
input values. As a rough estimate, if working on durations with a relative
accuracy of 2%, about 2kB (275 bins) are needed to cover values between 1
millisecond and 1 minute, and about 6kB (802 bins) to cover values between 1
nanosecond and 1 day.
The size of the sketch can be have a fail-safe upper-bound by using collapsing
stores. As shown in
the DDSketch paper
the likelihood of a store collapsing when using the default bound is vanishingly
small for most data.
DDSketch implementations are also available in:
Go
Python
JavaScript
N )LogarithmicMapping)CollapsingHighestDenseStore)CollapsingLowestDenseStore)
DenseStore)Optional)
KeyMapping)Storeg{Gz?i c e Zd ZdZd Zd Zed Zed Zed Z ed Z
ed Z dd Z d
Z
d Zd Zd
Zy) BaseDDSketcha The base implementation of DDSketch with neither mapping nor storage specified.
Args:
mapping (mapping.KeyMapping): map btw values and store bins
store (store.Store): storage for positive values
negative_store (store.Store): storage for negative values
zero_count (float): The count of zero values
Attributes:
relative_accuracy (float): the accuracy guarantee; referred to as alpha
in the paper. (0. < alpha < 1.)
count: the number of values seen by the sketch
min: the minimum value seen by the sketch
max: the maximum value seen by the sketch
sum: the sum of the values seen by the sketch
c * || _ || _ || _ || _ |j | _ | j j | j z | j j z | _ t d | _ t d | _
d| _ y )Nz+infz-inf ) _mapping_store_negative_store _zero_countrelative_accuracy_relative_accuracycount_countfloat_min_max_sum)selfmappingstorenegative_store
zero_counts Q/var/www/highfloat_scraper/venv/lib/python3.12/site-packages/ddsketch/ddsketch.py__init__zBaseDDSketch.__init__K sz
- %")";"; **0043C3CCdkkFWFWW &M &M c dj | j | j | j | j | j
| j | j S )NzSstore: {}, negative_store: {}, zero_count: {}, count: {}, sum: {}, min: {}, max: {})formatr r r r r r r r s r __repr__zBaseDDSketch.__repr__^ sM
( & KK KK II II II
r" c | j S Nr r% s r r zBaseDDSketch.countn s {{r" c y)zstr: name of the sketchDDSketch r% s r namezBaseDDSketch.namer s r" c | j S )z*Return the number of values in the sketch.r) r% s r
num_valueszBaseDDSketch.num_valuesx s {{r" c 4 | j | j z S )z;Return the exact average of the values added to the sketch.)r r r% s r avgzBaseDDSketch.avg~ s yy4;;&&r" c | j S )z7Return the exact sum of the values added to the sketch.)r r% s r sumzBaseDDSketch.sum s yyr" c H |dk rt d|z || j j kD r6| j j | j j | | nf|| j j k r7| j j | j j | | n| xj |z
c_ | xj |z
c_ | xj ||z z
c_ || j k r|| _
|| j kD r|| _ yy)zAdd a value to the sketch.r z'weight must be a positive float, got %rN)
ValueErrorr min_possibler addkeyr r r r r r )r valweights r r7 zBaseDDSketch.add s S= F OP P ++ + KKOODMM - -c 2F ;
DMM...
. $ $T]]%6%6t%*>*D*DDC "]] 0 0 5N r" c | j | s9t d| j j d|j j |j dk( ry| j
dk( r| j
| y| j j |j | j j |j | xj |j z
c_
| xj
|j
z
c_ | xj |j z
c_ |j | j k r|j | _ |j | j kD r|j | _
yy)zMerge the given sketch into this one. After this operation, this sketch
encodes the values that were added to both this and the input sketch.
z;Cannot merge two DDSketches with different parameters, got z and r N)
_mergeabler5 r gammar r _copyr merger r r r r r sketchs r rH zBaseDDSketch.merge s
v& ==&&(=(=?
<<1 ;;! JJv
&--( ""6#9#9: F . ..
v}}$ V[[ ;; " DI ;; " DI #r" c \ | j j |j j k( S )z:Two sketches can be merged only if their gammas are equal.)r rF )r others r rE zBaseDDSketch._mergeable s# }}""enn&:&:::r" c B | j j |j | j j |j |j | _ |j | _ |j
| _ |j | _ |j | _ y)z#Copy the input sketch into this oneN)r copyr r r r r r rI s r rG zBaseDDSketch._copy sn
' !!&"8"89!--KK KK mm KK r" N)g ?)__name__
__module__ __qualname____doc__r! r& propertyr r- r/ r1 r3 r7 rC rH rE rG r, r" r r r 8 s $&
' '
* 6 $>;
r" r c $ e Zd ZdZd fd Z xZS )r+ a^ The default implementation of BaseDDSketch, with optimized memory usage at
the cost of lower ingestion speed, using an unlimited number of bins. The
number of bins will not exceed a reasonable number unless the data is
distributed with tails heavier than any subexponential.
(cf. http://www.vldb.org/pvldb/vol12/p2195-masson.pdf)
c |t }t | }t }t }t t | |||d y )Nr r r r r )DEFAULT_REL_ACCr r superr+ r! )r r r r r __class__s r r! zDDSketch.__init__ sJ $ / $%67 #
h&) '
r" r( rO rP rQ rR r!
__classcell__rY s @r r+ r+ s
r" r+ c $ e Zd ZdZd fd Z xZS ) LogCollapsingLowestDenseDDSketcha Implementation of BaseDDSketch with optimized memory usage at the cost of
lower ingestion speed, using a limited number of bins. When the maximum
number of bins is reached, bins with lowest indices are collapsed, which
causes the relative accuracy to be lost on the lowest quantiles. For the
default bin limit, collapsing is unlikely to occur unless the data is
distributed with tails heavier than any subexponential.
(cf. http://www.vldb.org/pvldb/vol12/p2195-masson.pdf)
c |t }||dk rt }t | }t | }t | }t t
| |||d y Nr r rV )rW DEFAULT_BIN_LIMITr r rX r^ r! r r bin_limitr r r rY s r r! z)LogCollapsingLowestDenseDDSketch.__init__
sb $ / A
)I$%67*953I>
.>) ?
r" NNrZ r\ s @r r^ r^
r" r^ c $ e Zd ZdZd fd Z xZS )!LogCollapsingHighestDenseDDSketcha Implementation of BaseDDSketch with optimized memory usage at the cost of
lower ingestion speed, using a limited number of bins. When the maximum
number of bins is reached, bins with highest indices are collapsed, which
causes the relative accuracy to be lost on the highest quantiles. For the
default bin limit, collapsing is unlikely to occur unless the data is
distributed with tails heavier than any subexponential.
(cf. http://www.vldb.org/pvldb/vol12/p2195-masson.pdf)
c |t }||dk rt }t | }t | }t | }t t
| |||d y r` )rW ra r r rX rg r! rb s r r! z*LogCollapsingHighestDenseDDSketch.__init__+ sc $ / A
)I$%67+I64Y?
/?) @
r" rd rZ r\ s @r rg rg ! re r" rg )rR typingr r r r r r
TYPE_CHECKINGr r r
rW ra objectr r+ r^ rg r, r" r rl so > ' . -
# o 6 o d
|
2
|
<
r"