API Reference¶

This page is generated automatically from the docstrings in the freqprob package, so it always matches the installed code.

FreqProb: frequency-based probability estimation library.

ScoringMethod ¶

ScoringMethod(config: ScoringMethodConfig)

Bases: ABC

Abstract base class for frequency-based probability smoothing methods.

This class provides a unified interface for all probability estimation methods, supporting both regular probabilities and log-probabilities.

The general workflow is to initialize a method with a configuration, fit it to a frequency distribution, and then score individual elements. Given a frequency distribution mapping elements to their observed counts, each method estimates a probability P(w) for every element w. For unobserved elements, methods reserve probability mass to avoid returning zero.

Subclasses must implement :meth:_compute_probabilities, which populates the internal probability table and the reserved mass for unobserved elements.

Attributes:

Name	Type	Description
`config`	`ScoringMethodConfig`	Configuration parameters for the method.
`name`	`str \| None`	Human-readable name of the method, or `None` before it is set.
`logprob`	`bool \| None`	Whether this instance returns log-probabilities.

Examples:

>>> from freqprob import MLE
>>> freqdist = {"apple": 3, "banana": 2, "cherry": 1}
>>> scorer = MLE(freqdist, logprob=False)
>>> scorer("apple")  # most frequent item
0.5
>>> scorer("unknown")  # unobserved item
0.0

Initialize the scoring method.

Parameters:

Name	Type	Description	Default
`config`	`ScoringMethodConfig`	Configuration object containing the method parameters.	required

Note

This constructor should typically be called by subclass constructors, not directly by users.

fit ¶

fit(freqdist: FrequencyDistribution) -> ScoringMethod

Fit the scoring method to a frequency distribution.

This method trains the scorer on the provided frequency data, computing probability estimates for all observed elements. It resets any prior state so that re-fitting behaves like a fresh fit.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Frequency distribution mapping elements to their observed counts.	required

Returns:

Type	Description
`ScoringMethod`	This scorer, to allow method chaining.

Examples:

>>> from freqprob import MLE
>>> scorer = MLE({}, logprob=False).fit({"a": 2, "b": 1})
>>> scorer("a")
0.6666666666666666

score ¶

score(element: Element) -> Probability | LogProbability

Score a single element (scikit-learn-style alias for __call__).

Parameters:

Name	Type	Description	Default
`element`	`Element`	The element to be scored.	required

Returns:

Type	Description
`Probability \| LogProbability`	The probability (or log-probability) of the element.

Examples:

>>> from freqprob import MLE
>>> scorer = MLE({"a": 2, "b": 1}, logprob=False)
>>> scorer.score("a")
0.6666666666666666

predict ¶

predict(elements: Iterable[Element]) -> list[Probability | LogProbability]

Score many elements at once (scikit-learn-style batch alias).

Parameters:

Name	Type	Description	Default
`elements`	`Iterable[Element]`	The elements to be scored.	required

Returns:

Type	Description
`list[Probability \| LogProbability]`	One probability (or log-probability) per input element, in order.

Examples:

>>> from freqprob import MLE
>>> scorer = MLE({"a": 2, "b": 1}, logprob=False)
>>> scorer.predict(["a", "b", "c"])
[0.6666666666666666, 0.3333333333333333, 0.0]

save ¶

save(path: str | Path) -> None

Serialize this fitted scorer to a file.

The scorer can be restored later with :meth:load, avoiding the cost of re-fitting. Serialization uses :mod:pickle.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Destination file path.	required

Examples:

>>> from freqprob import MLE
>>> scorer = MLE({"a": 2, "b": 1}, logprob=False)
>>> scorer.save("model.pkl")
>>> restored = MLE.load("model.pkl")
>>> restored("a") == scorer("a")
True

load `classmethod` ¶

load(path: str | Path) -> T

Load a scorer previously written with :meth:save.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to a file created by :meth:`save`.	required

Returns:

Type	Description
`T`	The restored scorer.

Raises:

Type	Description
`TypeError`	If the file does not contain a compatible scorer.

Note

Only load files you trust: unpickling executes arbitrary code from the file, so never load a scorer from an untrusted source.

Examples:

>>> from freqprob import MLE
>>> restored = MLE.load("model.pkl")
>>> restored("a")
0.6666666666666666

ELE ¶

ELE(freqdist: FrequencyDistribution, bins: int | None = None, logprob: bool = True)

Bases: Lidstone

Expected Likelihood Estimation probability distribution.

The special case of Lidstone smoothing with gamma = 0.5, corresponding to the Jeffreys prior for multinomial distributions. Each observed element gets (c_i + 0.5) / (N + 0.5 * B) and any unobserved element gets 0.5 / (N + 0.5 * B) — half a virtual observation per element. This is a middle ground between MLE and Laplace: use it when add-one smoothing feels too aggressive but you still want unseen elements handled.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`bins`	`int \| None`	Total number of possible elements. If `None` (the default), uses the support size `\|V\|`.	`None`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

Add half a count to each element, then normalize:

>>> from freqprob import ELE
>>> freqdist = {"cat": 4, "dog": 2}
>>> ele = ELE(freqdist, logprob=False)
>>> round(ele("cat"), 3)     # (4+0.5)/(6+0.5*2) = 4.5/7
0.643
>>> round(ele("dog"), 3)     # (2+0.5)/(6+0.5*2) = 2.5/7
0.357
>>> round(ele("bird"), 3)    # 0.5/(6+0.5*2) = 0.5/7
0.071

Note

ELE applies less smoothing than Laplace (gamma = 1), so it stays closer to the observed frequencies while still avoiding zero probabilities.

Initialize Expected Likelihood Estimation.

Laplace ¶

Laplace(freqdist: FrequencyDistribution, bins: int | None = None, logprob: bool = True)

Bases: Lidstone

Laplace (add-one) smoothing probability distribution.

The special case of Lidstone smoothing with gamma = 1.0, also called "add-one smoothing." Each observed element gets (c_i + 1) / (N + B) and any unobserved element gets 1 / (N + B) — i.e. one virtual observation added to every possible element, corresponding to a uniform Dirichlet prior. It is the most common additive smoother: reach for it when you want a simple, reasonable default that never assigns zero probability.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`bins`	`int \| None`	Total number of possible elements. If `None` (the default), uses the support size `\|V\|`.	`None`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

Add one to every count, then normalize:

>>> from freqprob import Laplace
>>> freqdist = {"red": 3, "blue": 2, "green": 1}
>>> laplace = Laplace(freqdist, logprob=False)
>>> round(laplace("red"), 3)     # (3+1)/(6+3) = 4/9
0.444
>>> round(laplace("blue"), 3)    # (2+1)/(6+3) = 3/9
0.333
>>> round(laplace("yellow"), 3)  # 1/(6+3) = 1/9
0.111

Initialize Laplace smoothing.

Lidstone ¶

Lidstone(freqdist: FrequencyDistribution, gamma: float, bins: int | None = None, logprob: bool = True)

Bases: ScoringMethod

Lidstone additive smoothing probability distribution.

Additive smoothing tackles the zero-probability problem by adding a virtual count gamma to every possible element before normalizing. For counts c_i, total N, and B bins, each observed element gets (c_i + gamma) / (N + B * gamma) and any unobserved element gets gamma / (N + B * gamma). This is the posterior mean under a symmetric Dirichlet prior with concentration gamma. Use it when you need a simple, principled way to keep unseen elements from scoring zero, and tune gamma to trade off between staying close to the data (small gamma) and a more uniform distribution (large gamma).

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`gamma`	`float`	Additive smoothing parameter (`gamma >= 0`). `1.0` gives Laplace smoothing, `0.5` gives the Jeffreys prior (ELE), and `gamma -> 0` approaches MLE.	required
`bins`	`int \| None`	Total number of possible elements. If `None` (the default), uses the support size `\|V\|`. A larger `bins` reserves more mass for unseen elements.	`None`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

Add-gamma smoothing gives unseen elements non-zero probability:

>>> from freqprob import Lidstone
>>> freqdist = {"apple": 3, "banana": 1}
>>> lidstone = Lidstone(freqdist, gamma=1.0, logprob=False)
>>> round(lidstone("apple"), 3)    # (3+1)/(4+2*1) = 4/6
0.667
>>> round(lidstone("banana"), 3)   # (1+1)/(4+2*1) = 2/6
0.333
>>> round(lidstone("cherry"), 3)   # 1/(4+2*1) = 1/6
0.167

A larger gamma pulls probabilities toward uniform:

>>> smooth = Lidstone(freqdist, gamma=2.0, logprob=False)
>>> smooth("apple")      # (3+2)/(4+2*2) = 5/8
0.625
>>> smooth("cherry")     # 2/(4+2*2) = 2/8
0.25

A larger bins reserves more mass for potential unseen elements:

>>> lidstone_big = Lidstone(freqdist, gamma=1.0, bins=1000, logprob=False)
>>> round(lidstone_big("apple"), 4)    # (3+1)/(4+1000) = 4/1004
0.004
>>> round(lidstone_big("unseen"), 4)   # 1/(4+1000) = 1/1004
0.001

Note

The choice of gamma is a bias-variance tradeoff: small values stay close to MLE (low bias, high variance) while large values approach a uniform distribution (higher bias, lower variance).

Initialize Lidstone smoothing.

MLE ¶

MLE(freqdist: FrequencyDistribution, unobs_prob: Probability = 0.0, logprob: bool = True)

Bases: ScoringMethod

Maximum Likelihood Estimation.

Estimates each element's probability as its relative frequency in the observed data — the most intuitive baseline. For counts c_i with total N, each observed element gets (1 - p0) * c_i / N and any unobserved element gets the reserved mass p0 (0.0 by default, i.e. unseen elements score zero).

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`unobs_prob`	`Probability`	Reserved probability mass `p0` for unobserved elements (`0.0 <= p0 <= 1.0`). Defaults to `0.0`.	`0.0`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

Probabilities are relative frequencies; unseen elements score zero:

>>> mle = MLE({"apple": 6, "banana": 3, "cherry": 1}, logprob=False)
>>> mle("apple")
0.6
>>> mle("banana")
0.3
>>> mle("unknown")
0.0

Reserving mass for unseen elements rescales the observed ones:

>>> smoothed = MLE({"apple": 6, "banana": 3, "cherry": 1},
...                unobs_prob=0.1, logprob=False)
>>> round(smoothed("apple"), 3)
0.54
>>> smoothed("unknown")
0.1

Note

MLE underlies most smoothing methods but assigns zero probability to unseen events, which is problematic for sparse data. Prefer Laplace, Lidstone, or a more advanced method when unseen elements are likely.

Initialize MLE distribution.

Random ¶

Random(freqdist: FrequencyDistribution, unobs_prob: Probability = 0.0, logprob: bool = True, seed: int | None = None)

Bases: ScoringMethod

Random probability distribution.

Assigns pseudo-random probabilities, useful for testing and as a randomized baseline. Each element is given a random count drawn from the observed count range, and those counts are then normalized like MLE. A fixed seed makes the result reproducible.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts. Only the min/max counts are used, to bound the random counts.	required
`unobs_prob`	`Probability`	Reserved probability mass for unobserved elements. Defaults to `0.0`.	`0.0`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`
`seed`	`int \| None`	Seed for the random number generator, for reproducible results. Defaults to `None`.	`None`

Examples:

The probabilities are random but form a valid distribution, and a fixed seed is reproducible:

>>> freqdist = {"apple": 5, "banana": 2, "cherry": 8}
>>> a = Random(freqdist, seed=123, logprob=False)
>>> b = Random(freqdist, seed=123, logprob=False)
>>> a("apple") == b("apple")
True
>>> round(sum(a(e) for e in freqdist), 6)
1.0

Note

Primarily for testing, debugging, and providing a randomized baseline for comparison. It is not a smoothing method in any meaningful sense.

Initialize Random distribution.

Uniform ¶

Uniform(freqdist: FrequencyDistribution, unobs_prob: Probability = 0.0, logprob: bool = True)

Bases: ScoringMethod

Uniform probability distribution.

The simplest baseline: assign equal probability to every observed element, ignoring the counts entirely. Useful as a non-informative prior or as a reference point when comparing smoothing methods.

For a vocabulary of size V with reserved mass p0, each observed element gets (1 - p0) / V and any unobserved element gets p0.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts. The counts are ignored; only the number of distinct elements matters.	required
`unobs_prob`	`Probability`	Reserved probability mass `p0` for unobserved elements (`0.0 <= p0 <= 1.0`). Defaults to `0.0`.	`0.0`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

Counts are ignored, so both observed elements score the same:

>>> uniform = Uniform({"apple": 10, "banana": 1}, unobs_prob=0.1, logprob=False)
>>> uniform("apple")
0.45
>>> uniform("banana")
0.45
>>> uniform("cherry")  # unobserved
0.1

Note

Because it discards frequency information, Uniform is meant as a baseline. Use MLE, Lidstone, or Laplace when counts should matter.

Initialize Uniform distribution.

Bayesian ¶

Bayesian(freqdist: FrequencyDistribution, alpha: float = 1.0, logprob: bool = True)

Bases: ScoringMethod

Bayesian smoothing with a Dirichlet prior.

Estimates probabilities as the posterior mean under a symmetric Dirichlet prior, adding alpha pseudocounts to each possible outcome. Each observed element gets (c_i + alpha) / (N + V * alpha) and any unobserved element gets alpha / (N + V * alpha), where N is the total count and V the support size. This is numerically identical to Lidstone smoothing with gamma = alpha, but framed so that alpha reads as prior belief: use it when you want a theoretically principled smoother whose strength you can tune from near-MLE (small alpha) to strongly uniform (large alpha).

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`alpha`	`float`	Dirichlet concentration parameter (`alpha > 0`). `alpha -> 0` approaches MLE, `alpha = 1` gives a uniform prior (Laplace smoothing), and `alpha > 1` prefers more uniform distributions. Defaults to `1.0`.	`1.0`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

Bayesian smoothing with a uniform prior (alpha = 1):

>>> from freqprob import Bayesian
>>> freqdist = {"apple": 8, "banana": 4, "cherry": 1}
>>> bayes = Bayesian(freqdist, alpha=1.0, logprob=False)
>>> bayes("apple")     # (8+1)/(13+3*1) = 9/16
0.5625
>>> bayes("banana")    # (4+1)/(13+3*1) = 5/16
0.3125
>>> bayes("unseen")    # 1/(13+3*1) = 1/16
0.0625

A larger alpha smooths more strongly toward uniform:

>>> bayes_smooth = Bayesian(freqdist, alpha=2.0, logprob=False)
>>> round(bayes_smooth("apple"), 3)    # (8+2)/(13+3*2) = 10/19
0.526
>>> round(bayes_smooth("unseen"), 3)   # 2/(13+3*2) = 2/19
0.105

A small alpha stays closer to the observed frequencies:

>>> bayes_minimal = Bayesian(freqdist, alpha=0.1, logprob=False)
>>> round(bayes_minimal("apple"), 3)   # (8+0.1)/(13+3*0.1) = 8.1/13.3
0.609

Note

A positive alpha is required; passing alpha <= 0 raises ValueError. Equivalent to Lidstone with gamma = alpha.

Initialize Bayesian smoothing.

CertaintyDegree ¶

CertaintyDegree(freqdist: FrequencyDistribution, bins: int | None = None, unobs_prob: Probability = 0.0, logprob: bool = True)

Bases: ScoringMethod

Certainty-degree smoothing.

Reserves probability mass for unseen elements based on the degree of certainty that no unobserved samples remain, derived from the number of observations relative to the support size. As more of the possible bins are observed, the reserved mass shrinks; the remaining mass rescales the Maximum-Likelihood estimate of each observed element.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`bins`	`int \| None`	Optional total number of possible sample bins for the experiment. If `None` (the default), it defaults to the number of observed types.	`None`
`unobs_prob`	`Probability`	Reserved probability mass for unobserved states (`0.0 <= unobs_prob <= 1.0`). Defaults to `0.0`.	`0.0`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities. Counts are corrected to avoid domain errors when taking logarithms.	`True`

Examples:

Observed elements are rescaled by the computed certainty mass, and the remainder is reserved for unseen elements:

>>> from freqprob import CertaintyDegree
>>> freqdist = {"apple": 8, "banana": 4, "cherry": 2, "date": 1}
>>> cd = CertaintyDegree(freqdist, logprob=False)
>>> round(cd("apple"), 3)
0.515
>>> round(cd("date"), 3)
0.064
>>> round(cd("kiwi"), 3)  # unobserved
0.035

Note

This is an experimental distribution under development by Tiago Tresoldi; it should not be used as the sole or main distribution yet.

Initialize Certainty Degree estimation.

SimpleGoodTuring ¶

SimpleGoodTuring(freqdist: FrequencyDistribution, p_value: float = 0.05, default_p0: float | None = None, bins: int | None = None, logprob: bool = True, allow_fail: bool = True)

Bases: ScoringMethod

Simple Good-Turing smoothing.

Reestimates counts using the frequency of frequencies (how many elements were seen once, twice, and so on), fitting a log-linear model to smooth the tail so that mass can be reserved for unseen elements. This is the practical Gale-Sampson formulation of the Good-Turing method originally developed by Turing and Good, and is a strong general-purpose smoother when the data show a spread of low-frequency counts (many singletons, fewer doubletons, etc.).

This implementation follows those by "maxbane" (2011, https://github.com/maxbane/simplegoodturing/blob/master/sgt.py), Geoffrey Sampson's original C code (1995-2008, https://www.grsampson.net/Resources.html), and the NLTK project (Loper, Bird et al., 2001-2018). Because of minor differences intended to guarantee non-zero probabilities under expected underflow, the reliance on scipy, and the handling of probabilities that are not computable when SGT assumptions are not met, most results will not exactly match the Gale-Sampson "gold standard", though the differences are never expected to be significant.

Note

Since v0.4.0, unseen queries return the per-word probability p0 / (bins - V) rather than the total mass p0, making the method compatible with perplexity calculations and consistent with the other smoothers. Use the :attr:total_unseen_mass property to recover p0 (the total mass for all unseen words combined).

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`p_value`	`float`	The p-value for the confidence interval of the empirical Turing estimate, guiding the choice between the Turing estimate `x` and the log-linear smoothed value `y`. Defaults to `0.05` (Sampson's reference implementation), though the authors suggest `0.1`.	`0.05`
`default_p0`	`float \| None`	Optional probability for unobserved samples (`p0`) used when no single-count samples are observed. If unspecified, `p0` defaults to a Laplace estimate for the current distribution — an intended change from the Gale-Sampson reference.	`None`
`bins`	`int \| None`	Total support size for computing per-word unseen probabilities. If `None` (the default), uses the heuristic `bins = V_observed + N1` where `N1` is the number of singletons, assuming about as many unseen types as singletons. Pass an exact support size if known.	`None`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities. Counts are corrected to avoid domain errors when taking logarithms.	`True`
`allow_fail`	`bool`	If `True` (the default), raise `RuntimeWarning` when the essential SGT assumptions are not met (log-linear regression slope `> -1.0`, or an unobserved count reached before crossing the smoothing threshold). If `False`, estimation proceeds but may yield an unreliable distribution.	`True`

Examples:

SGT needs a spread of low-frequency counts, so use a Zipfian-ish distribution with several singletons. Observed elements are discounted and unseen elements receive a share of the reserved mass:

>>> from freqprob import SimpleGoodTuring
>>> freqdist = {"a": 10, "b": 5, "c": 4, "d": 3, "e": 2,
...             "f": 2, "g": 1, "h": 1, "i": 1, "j": 1}
>>> sgt = SimpleGoodTuring(freqdist, logprob=False)
>>> round(float(sgt("a")), 3)
0.307
>>> round(float(sgt("g")), 3)  # a singleton
0.025
>>> round(sgt("unknown"), 3)  # per-word unseen probability
0.033
>>> round(sgt.total_unseen_mass, 3)  # total mass for all unseen words
0.133

Initialize Simple Good-Turing smoothing.

total_unseen_mass `property` ¶

total_unseen_mass: float

Total probability mass reserved for ALL unseen words (p0).

This is the value that Good-Turing actually estimates. The per-word unseen probability returned by __call__() is this value divided by the estimated number of unseen types (bins - V_observed).

Returns:

Type	Description
`float`	Total probability mass for all unseen words combined.

Examples:

>>> from freqprob import SimpleGoodTuring
>>> freqdist = {"a": 10, "b": 5, "c": 1, "d": 1, "e": 1}
>>> sgt = SimpleGoodTuring(freqdist, logprob=False)
>>> round(sgt.total_unseen_mass, 3)  # total mass for all unseen words
0.167
>>> round(sgt("unknown"), 3)  # per-word probability for one unseen word
0.056

WittenBell ¶

WittenBell(freqdist: FrequencyDistribution, bins: int | None = None, logprob: bool = True)

Bases: ScoringMethod

Witten-Bell smoothing.

Reserves probability mass for unseen elements based on how many distinct types were observed, treating each new type as evidence that more unseen types exist. The reserved mass is T / (N + T), where T is the number of observed types and N the total number of observations; the remaining mass is discounted proportionally so all estimates sum to one:

p = T / (Z * (N + T)) for an unobserved element, and
p = c / (N + T) for an element observed c times.

Useful as a lightweight discounting method when a simple, parameter-free way to handle unseen elements is needed.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`bins`	`int \| None`	Optional total number of possible sample bins for the experiment. If `None` (the default), it defaults to the number of observed types, so the full reserved mass is assigned to a single unseen bin.	`None`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities. Counts are corrected to avoid domain errors when taking logarithms.	`True`

Examples:

Observed elements are scored by relative frequency over N + T, and unseen elements share the reserved mass:

>>> from freqprob import WittenBell
>>> freqdist = {"the": 100, "a": 50, "cat": 3, "dog": 2, "rare": 1}
>>> wb = WittenBell(freqdist, logprob=False)
>>> round(wb("the"), 3)
0.621
>>> round(wb("rare"), 3)
0.006
>>> round(wb("unicorn"), 3)  # unobserved
0.031

Initialize Witten-Bell smoothing.

Interpolated ¶

Interpolated(high_order_dist: FrequencyDistribution, low_order_dist: FrequencyDistribution, lambda_weight: float = 0.7, logprob: bool = True)

Bases: ScoringMethod

Linear interpolation smoothing between two frequency distributions.

Blends a higher-order distribution with a lower-order one using a weighted average, lambda_weight * P_high + (1 - lambda_weight) * P_low. This keeps the specificity of the high-order model while borrowing robustness from the smoother low-order model. Two modes are detected automatically from the keys:

N-gram interpolation, when the high-order keys are longer tuples than the low-order keys. The lower-order context is taken as the suffix of each high-order n-gram (e.g. bigram (w2, w3) from trigram (w1, w2, w3)).
Same-type interpolation, when both distributions share the same key type (strings or equal-length tuples). Keys are matched directly.

Parameters:

Name	Type	Description	Default
`high_order_dist`	`FrequencyDistribution`	Higher-order frequency distribution mapping elements to counts (e.g. trigrams). For n-gram mode its tuple keys must be at least as long as those of `low_order_dist`.	required
`low_order_dist`	`FrequencyDistribution`	Lower-order frequency distribution (e.g. bigrams).	required
`lambda_weight`	`float`	Interpolation weight `0 <= lambda_weight <= 1` for the high-order model. Higher values favor specificity, lower values favor smoothing. Defaults to `0.7`.	`0.7`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

N-gram interpolation (trigrams backed off to bigrams). The score of ('the', 'big', 'cat') is 0.7 * (3/5) + 0.3 * (5/10), and an unseen trigram backs off to 0.3 * (5/10):

>>> trigrams = {("the", "big", "cat"): 3, ("a", "big", "dog"): 2}
>>> bigrams = {("big", "cat"): 5, ("big", "dog"): 3, ("small", "cat"): 2}
>>> interp = Interpolated(trigrams, bigrams, lambda_weight=0.7, logprob=False)
>>> round(interp(("the", "big", "cat")), 3)
0.57
>>> round(interp(("unseen", "big", "cat")), 3)
0.15

Same-type interpolation between two string distributions. The score of "word1" is 0.6 * (10/15) + 0.4 * (3/10):

>>> model1 = {"word1": 10, "word2": 5}
>>> model2 = {"word1": 3, "word3": 7}
>>> interp = Interpolated(model1, model2, lambda_weight=0.6, logprob=False)
>>> round(interp("word1"), 3)
0.52
>>> round(interp("word2"), 3)
0.2
>>> round(interp("word3"), 3)
0.28

Note

The mode is inferred from the key types, so mixed tuple lengths within a single distribution raise an error. Unseen high-order n-grams back off to the low-order model, and all probabilities are floored at 1e-10 for numerical stability. The weight lambda_weight is best tuned on held-out data.

Initialize Interpolated smoothing.

KneserNey ¶

KneserNey(freqdist: FrequencyDistribution, discount: float = 0.75, logprob: bool = True)

Bases: ScoringMethod

Kneser-Ney smoothing probability distribution.

One of the most effective smoothing methods for language modeling. It combines absolute discounting with interpolation and, crucially, weights a word by the diversity of contexts it appears in (its continuation probability) rather than its raw frequency. Each observed bigram scores max(c - d, 0) / c(context) + backoff(context) * P_cont(word), and unseen bigrams fall back to the average continuation probability.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Frequency distribution mapping bigrams to their observed counts, in the form `{(context, word): count}`. Non-bigram keys are ignored.	required
`discount`	`float`	Absolute discounting parameter `0 < d < 1`. Common values are `0.5`-`0.8`. Defaults to `0.75`.	`0.75`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

Observed bigrams are discounted and interpolated with the continuation model, while unseen bigrams back off to the average continuation probability:

>>> bigram_counts = {
...     ("the", "cat"): 5, ("the", "dog"): 3, ("a", "cat"): 2,
...     ("a", "dog"): 1, ("big", "cat"): 1, ("small", "dog"): 1,
... }
>>> kn = KneserNey(bigram_counts, discount=0.75, logprob=False)
>>> round(kn(("the", "cat")), 3)
0.625
>>> round(kn(("the", "mouse")), 3)
0.5
>>> round(kn(("new_context", "cat")), 3)
0.5

Note

This implementation assumes bigram input. The discount d is typically between 0.5 and 0.8; 0.75 is a robust default. Reliable continuation probabilities require sufficient bigram data.

Initialize Kneser-Ney smoothing.

ModifiedKneserNey ¶

ModifiedKneserNey(freqdist: FrequencyDistribution, logprob: bool = True)

Bases: ScoringMethod

Modified Kneser-Ney smoothing probability distribution.

An enhanced Kneser-Ney variant that applies a different discount depending on a bigram's count: d1 for singletons, d2 for doubletons, and d3 for counts of three or more. Those discounts are estimated automatically from the data's count-of-counts statistics, which usually makes it more accurate than the fixed-discount version.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Frequency distribution mapping bigrams to their observed counts, in the form `{(context, word): count}`. Non-bigram keys are ignored.	required
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

The discount adapts to each bigram's count, and the score interpolates the discounted estimate with the continuation model:

>>> bigram_counts = {
...     ("the", "cat"): 5, ("the", "dog"): 3, ("a", "cat"): 2,
...     ("a", "dog"): 1, ("big", "cat"): 1, ("small", "dog"): 1,
... }
>>> mkn = ModifiedKneserNey(bigram_counts, logprob=False)
>>> round(mkn(("the", "cat")), 3)
0.625
>>> round(mkn(("a", "cat")), 3)
0.558

Note

Modified Kneser-Ney is a standard, state-of-the-art classical smoothing method for n-gram language models. Because it estimates its discounts from the data, it is more robust than fixed-discount methods but needs enough varied counts to estimate them reliably.

Initialize Modified Kneser-Ney smoothing.

LazyBatchScorer ¶

LazyBatchScorer(lazy_scorer: LazyScoringMethod)

Batch scorer with lazy evaluation and intelligent caching.

This scorer optimizes batch operations by:

Using lazy evaluation to avoid unnecessary computations.
Intelligently ordering computations based on access patterns.
Providing memory-efficient operations for large datasets.

Parameters:

Name	Type	Description	Default
`lazy_scorer`	`LazyScoringMethod`	Lazy scoring method to wrap.	required

Examples:

Wrap a lazy MLE scorer and score a batch of elements:

>>> from freqprob import create_lazy_mle
>>> lazy_scorer = create_lazy_mle({"a": 3, "b": 2, "c": 1}, logprob=False)
>>> batch_scorer = LazyBatchScorer(lazy_scorer)
>>> [round(float(x), 4) for x in batch_scorer.score_batch(["a", "c"])]
[0.5, 0.1667]
>>> stats = batch_scorer.get_access_statistics()
>>> stats["total_accesses"]
2
>>> stats["unique_elements"]
2

Initialize lazy batch scorer.

Parameters:

Name	Type	Description	Default
`lazy_scorer`	`LazyScoringMethod`	Lazy scoring method to wrap.	required

score_batch ¶

score_batch(elements: list[Element]) -> list[float]

Score a batch of elements with lazy evaluation.

Parameters:

Name	Type	Description	Default
`elements`	`list[Element]`	Elements to score.	required

Returns:

Type	Description
`list[float]`	Scores for the elements, in the same order as the input.

score_streaming ¶

score_streaming(element_stream: Iterable[Element]) -> Iterator[float]

Score elements from a stream with adaptive lazy evaluation.

Parameters:

Name	Type	Description	Default
`element_stream`	`Iterable[Element]`	Stream of elements to score.	required

Yields:

Type	Description
`float`	Score for each element in the stream.

get_access_statistics ¶

get_access_statistics() -> dict[str, Any]

Get statistics about element access patterns.

Returns:

Type	Description
`dict[str, Any]`	Access statistics including total accesses, unique elements, and
`dict[str, Any]`	the most frequently accessed element.

LazyScoringMethod ¶

LazyScoringMethod(lazy_computer: LazyProbabilityComputer, config: ScoringMethodConfig, name: str)

Bases: ScoringMethod

Scoring method that defers probability computation until requested.

Wraps a lazy computation strategy so that probabilities are calculated only for the elements actually scored, rather than for the whole vocabulary up front. This can significantly improve performance when a large distribution is fitted but only a small subset of elements is ever queried.

Parameters:

Name	Type	Description	Default
`lazy_computer`	`LazyProbabilityComputer`	Strategy object that computes a single element's probability on demand.	required
`config`	`ScoringMethodConfig`	Configuration controlling `logprob` and any reserved probability mass.	required
`name`	`str`	Human-readable name for the scoring method.	required

Note

Instances are normally built through the public factory functions create_lazy_mle and create_lazy_laplace rather than constructed directly, since the computer and config types are internal.

Examples:

Probabilities are computed only when an element is first scored:

>>> from freqprob import create_lazy_mle
>>> scorer = create_lazy_mle({"a": 3, "b": 2, "c": 1}, logprob=False)
>>> scorer("a")
0.5
>>> sorted(scorer.get_computed_elements())
['a']
>>> scorer("b")
0.3333333333333333
>>> sorted(scorer.get_computed_elements())
['a', 'b']
>>> scorer("unseen")  # unobserved element
0.0

Initialize lazy scoring method.

Parameters:

Name	Type	Description	Default
`lazy_computer`	`LazyProbabilityComputer`	Strategy for lazy computation.	required
`config`	`ScoringMethodConfig`	Configuration parameters.	required
`name`	`str`	Name of the scoring method.	required

precompute_batch ¶

precompute_batch(elements: set[Element]) -> None

Precompute probabilities for a batch of elements.

This can be useful when you know you'll need several elements and want to compute them all at once for efficiency.

Parameters:

Name	Type	Description	Default
`elements`	`set[Element]`	Elements to precompute.	required

get_computed_elements ¶

get_computed_elements() -> set[Element]

Get the set of elements that have been computed so far.

Returns:

Type	Description
`set[Element]`	Set of elements whose probabilities have been computed.

force_full_computation ¶

force_full_computation() -> None

Force computation of all probabilities in the distribution.

This converts the lazy scorer to a regular scorer by computing all probabilities immediately.

CompressedFrequencyDistribution ¶

CompressedFrequencyDistribution(quantization_levels: int | None = None, use_compression: bool = True, intern_strings: bool = True)

Memory-efficient compressed frequency distribution.

This class uses various compression techniques to reduce memory usage for large vocabularies while maintaining fast access patterns.

Techniques used:

Integer compression for counts.
String interning for element storage.
Sparse representation for zero counts.
Optional quantization for approximate counts.

Parameters:

Name	Type	Description	Default
`quantization_levels`	`int \| None`	Number of quantization levels for count compression, or `None` for exact counts. Defaults to `None`.	`None`
`use_compression`	`bool`	Whether to use data compression when serializing. Defaults to `True`.	`True`
`intern_strings`	`bool`	Whether to intern string elements for memory efficiency. Defaults to `True`.	`True`

Examples:

Exact counts are preserved when quantization is disabled:

>>> compressed_dist = CompressedFrequencyDistribution()
>>> compressed_dist.update({"word1": 1000, "word2": 500, "word3": 1})
>>> compressed_dist.get_count("word1")
1000
>>> compressed_dist.get_vocabulary_size()
3
>>> compressed_dist.get_total_count()
1501
>>> sorted(compressed_dist.to_dict().items())
[('word1', 1000), ('word2', 500), ('word3', 1)]

Initialize compressed frequency distribution.

update ¶

update(freqdist: FrequencyDistribution) -> None

Update with a frequency distribution.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Frequency distribution to add.	required

get_count ¶

get_count(element: Element) -> int

Get count for an element.

Parameters:

Name	Type	Description	Default
`element`	`Element`	Element to query.	required

Returns:

Type	Description
`int`	Count for the element (dequantized if quantization is used).

get_total_count ¶

get_total_count() -> int

Get total count across all elements.

get_vocabulary_size ¶

get_vocabulary_size() -> int

Get support size.

items ¶

items() -> Iterator[tuple[Element, int]]

Iterate over (element, count) pairs.

keys ¶

keys() -> Iterator[Element]

Iterate over elements with non-zero counts.

values ¶

values() -> Iterator[int]

Iterate over non-zero counts.

to_dict ¶

to_dict() -> dict[Element, int]

Convert to regular dictionary.

get_memory_usage ¶

get_memory_usage() -> dict[str, int]

Get detailed memory usage information.

Returns:

Type	Description
`dict[str, int]`	Memory usage breakdown in bytes, keyed by component, including a
`dict[str, int]`	`"total"` entry.

compress_to_bytes ¶

compress_to_bytes() -> bytes

Compress the entire distribution to bytes.

Returns:

Type	Description
`bytes`	Serialized (and optionally gzip-compressed) representation.

decompress_from_bytes `classmethod` ¶

decompress_from_bytes(compressed_data: bytes, use_compression: bool = True) -> CompressedFrequencyDistribution

Decompress distribution from bytes.

Parameters:

Name	Type	Description	Default
`compressed_data`	`bytes`	Compressed data to restore.	required
`use_compression`	`bool`	Whether the data was gzip-compressed. Defaults to `True`.	`True`

Returns:

Type	Description
`CompressedFrequencyDistribution`	The reconstructed distribution.

QuantizedProbabilityTable ¶

QuantizedProbabilityTable(num_quantization_levels: int = 65536, log_space: bool = True)

Quantized probability table for memory-efficient probability storage.

This class stores probabilities using quantization to reduce memory usage while maintaining reasonable precision for most applications.

Parameters:

Name	Type	Description	Default
`num_quantization_levels`	`int`	Number of quantization levels, which determines precision. Defaults to `65536`.	`65536`
`log_space`	`bool`	Whether to quantize in log space for better precision. Defaults to `True`.	`True`

Examples:

Stored probabilities are approximate; higher level counts are more precise:

>>> prob_table = QuantizedProbabilityTable(num_quantization_levels=1024)
>>> prob_table.set_probabilities(
...     {"word1": 0.5, "word2": 0.3, "word3": 0.2}
... )
>>> round(prob_table.get_probability("word1"), 2)
0.49
>>> sorted(prob_table.get_elements())
['word1', 'word2', 'word3']

Initialize quantized probability table.

set_probabilities ¶

set_probabilities(probabilities: dict[Element, float]) -> None

Set probabilities for multiple elements.

Parameters:

Name	Type	Description	Default
`probabilities`	`dict[Element, float]`	Mapping of elements to their probabilities.	required

set_probability ¶

set_probability(element: Element, probability: float) -> None

Set probability for a single element.

Parameters:

Name	Type	Description	Default
`element`	`Element`	Element to set probability for.	required
`probability`	`float`	Probability value.	required

get_probability ¶

get_probability(element: Element) -> float

Get probability for an element.

Parameters:

Name	Type	Description	Default
`element`	`Element`	Element to query.	required

Returns:

Type	Description
`float`	Dequantized probability for the element (or the default).

set_default_probability ¶

set_default_probability(default_prob: float) -> None

Set default probability for unobserved elements.

Parameters:

Name	Type	Description	Default
`default_prob`	`float`	Default probability for unobserved elements.	required

get_elements ¶

get_elements() -> list[Element]

Get all elements with stored probabilities.

get_memory_usage ¶

get_memory_usage() -> dict[str, int]

Get memory usage information.

Returns:

Type	Description
`dict[str, int]`	Memory usage breakdown in bytes, including a `"total"` entry.

get_quantization_error_stats ¶

get_quantization_error_stats(original_probs: dict[Element, float]) -> dict[str, float]

Analyze quantization error compared to original probabilities.

Parameters:

Name	Type	Description	Default
`original_probs`	`dict[Element, float]`	Original probabilities to compare against.	required

Returns:

Type	Description
`dict[str, float]`	Error statistics including mean/max absolute and relative errors.

SparseFrequencyDistribution ¶

SparseFrequencyDistribution(default_count: int = 0, use_sorted_storage: bool = True)

Sparse representation for frequency distributions with many zero counts.

This class is optimized for distributions where most elements have zero counts, using sparse data structures for memory efficiency.

Parameters:

Name	Type	Description	Default
`default_count`	`int`	Default count for unobserved elements. Defaults to `0`.	`0`
`use_sorted_storage`	`bool`	Whether to keep elements sorted by frequency for faster top-k access. Defaults to `True`.	`True`

Examples:

Only non-default counts are stored; missing elements return the default:

>>> sparse_dist = SparseFrequencyDistribution()
>>> sparse_dist.update({"rare_word": 1, "common_word": 1000})
>>> sparse_dist.get_count("rare_word")
1
>>> sparse_dist.get_count("unseen")
0
>>> sparse_dist.get_vocabulary_size()
2
>>> sparse_dist.get_top_k(2)
[('common_word', 1000), ('rare_word', 1)]

Initialize sparse frequency distribution.

update ¶

update(freqdist: FrequencyDistribution) -> None

Update with a frequency distribution.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Frequency distribution to add.	required

increment ¶

increment(element: Element, count: int = 1) -> None

Increment count for a single element.

Parameters:

Name	Type	Description	Default
`element`	`Element`	Element to increment.	required
`count`	`int`	Count increment. Defaults to `1`.	`1`

get_count ¶

get_count(element: Element) -> int

Get count for an element.

Parameters:

Name	Type	Description	Default
`element`	`Element`	Element to query.	required

Returns:

Type	Description
`int`	Count for the element, or the default count if not present.

get_total_count ¶

get_total_count() -> int

Get total count across all elements.

get_vocabulary_size ¶

get_vocabulary_size() -> int

Get number of elements with non-default counts.

get_top_k ¶

get_top_k(k: int) -> list[tuple[Element, int]]

Get top-k most frequent elements.

Parameters:

Name	Type	Description	Default
`k`	`int`	Number of top elements to return.	required

Returns:

Type	Description
`list[tuple[Element, int]]`	List of `(element, count)` pairs, highest count first.

get_elements_with_count_range ¶

get_elements_with_count_range(min_count: int, max_count: int) -> list[Element]

Get elements with counts in a specific range.

Parameters:

Name	Type	Description	Default
`min_count`	`int`	Minimum count (inclusive).	required
`max_count`	`int`	Maximum count (inclusive).	required

Returns:

Type	Description
`list[Element]`	Elements whose counts fall within the specified range.

get_count_histogram ¶

get_count_histogram() -> dict[int, int]

Get histogram of counts (count -> frequency of that count).

Returns:

Type	Description
`dict[int, int]`	Histogram mapping each count value to how many elements have it.

items ¶

items() -> Iterator[tuple[Element, int]]

Iterate over (element, count) pairs for non-default elements.

keys ¶

keys() -> Iterator[Element]

Iterate over elements with non-default counts.

values ¶

values() -> Iterator[int]

Iterate over non-default counts.

to_dict ¶

to_dict() -> dict[Element, int]

Convert to regular dictionary.

get_memory_usage ¶

get_memory_usage() -> dict[str, int]

Get memory usage information.

Returns:

Type	Description
`dict[str, int]`	Memory usage breakdown in bytes, including a `"total"` entry.

DistributionMemoryAnalyzer ¶

DistributionMemoryAnalyzer()

Analyzer for comparing memory usage of different distribution representations.

This class helps analyze and compare memory usage between regular dictionaries, compressed distributions, sparse distributions, and other representations.

Note

compare_representations additionally contrasts compressed and sparse encodings, returning per-representation savings.

Examples:

Measure the memory footprint of a small distribution:

>>> analyzer = DistributionMemoryAnalyzer()
>>> freqdist = {"word1": 1000, "word2": 500, "word3": 1}
>>> measurement = analyzer.measure_distribution_memory(freqdist)
>>> sorted(measurement.keys())
['bytes_per_element', 'num_elements', 'total_mb']
>>> measurement["num_elements"]
3

Initialize distribution memory analyzer.

measure_distribution_memory ¶

measure_distribution_memory(freqdist: FrequencyDistribution) -> dict[str, float]

Measure memory usage of a frequency distribution.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Frequency distribution to measure.	required

Returns:

Type	Description
`dict[str, float]`	Memory usage measurements, including total MB, element count, and
`dict[str, float]`	bytes per element.

compare_representations ¶

compare_representations(freqdist: FrequencyDistribution) -> dict[str, Any]

Compare memory usage of different distribution representations.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Original frequency distribution.	required

Returns:

Type	Description
`dict[str, Any]`	Comparison results with per-representation memory usage, savings,
`dict[str, Any]`	and profiling metrics.

benchmark_scoring_methods ¶

benchmark_scoring_methods(freqdist: FrequencyDistribution, test_elements: list[str], methods_to_test: list[str] | None = None) -> dict[str, Any]

Benchmark memory usage and performance of different scoring methods.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Frequency distribution for testing.	required
`test_elements`	`list[str]`	Elements to score for performance testing.	required
`methods_to_test`	`list[str] \| None`	Scoring methods to test, or `None` to use a default set. Defaults to `None`.	`None`

Returns:

Type	Description
`dict[str, Any]`	Benchmark results keyed by method name.

MemoryMonitor ¶

MemoryMonitor(memory_threshold_mb: float = 1000.0, monitoring_interval: float = 5.0)

Continuous memory monitor for long-running processes.

This class provides continuous monitoring of memory usage patterns and can trigger alerts when memory usage exceeds thresholds.

Parameters:

Name	Type	Description	Default
`memory_threshold_mb`	`float`	Memory threshold in MB for triggering alerts. Defaults to `1000.0`.	`1000.0`
`monitoring_interval`	`float`	Monitoring interval in seconds. Defaults to `5.0`.	`5.0`

Examples:

Take a memory reading against a high threshold (no alert triggered):

>>> monitor = MemoryMonitor(memory_threshold_mb=1e9)
>>> monitor.check_memory() is None
True
>>> report = monitor.get_monitoring_report()
>>> report["threshold_violations"]
0
>>> report["total_snapshots"]
1

Initialize memory monitor.

start_monitoring ¶

start_monitoring() -> None

Start continuous memory monitoring.

stop_monitoring ¶

stop_monitoring() -> None

Stop memory monitoring.

check_memory ¶

check_memory() -> dict[str, Any] | None

Check current memory usage and trigger alerts if needed.

Returns:

Type	Description
`dict[str, Any] \| None`	Alert information if the threshold was exceeded, otherwise
`dict[str, Any] \| None`	`None`.

get_monitoring_report ¶

get_monitoring_report() -> dict[str, Any]

Get a comprehensive monitoring report.

Returns:

Type	Description
`dict[str, Any]`	Monitoring report with memory statistics, alerts, and trend.

MemoryProfiler ¶

MemoryProfiler(enable_tracemalloc: bool = True, snapshot_interval: float = 1.0)

Memory profiler for analyzing memory usage patterns.

This class provides utilities for monitoring memory usage during various operations and analyzing memory efficiency.

Parameters:

Name	Type	Description	Default
`enable_tracemalloc`	`bool`	Whether to enable detailed Python memory tracing. Defaults to `True`.	`True`
`snapshot_interval`	`float`	Interval in seconds for automatic snapshots. Defaults to `1.0`.	`1.0`

Examples:

Profile an operation and inspect the recorded metrics:

>>> profiler = MemoryProfiler()
>>> with profiler.profile_operation("build_list"):
...     data = [i * i for i in range(10000)]
>>> metrics = profiler.get_latest_metrics()
>>> metrics.operation_name
'build_list'
>>> isinstance(metrics.execution_time, float)
True
>>> isinstance(metrics.memory_delta_mb, float)
True
>>> len(profiler.get_all_metrics())
1

Initialize memory profiler.

take_snapshot ¶

take_snapshot() -> MemorySnapshot

Take a memory usage snapshot.

Returns:

Type	Description
`MemorySnapshot`	The current memory usage snapshot.

profile_operation ¶

profile_operation(operation_name: str) -> Iterator[None]

Context manager for profiling an operation.

Uses time.perf_counter() for high-precision timing measurements, ensuring accurate execution time tracking even for fast operations.

Parameters:

Name	Type	Description	Default
`operation_name`	`str`	Name of the operation being profiled.	required

Yields:

Type	Description
`None`	`None`. Metrics are recorded when the context exits.

get_latest_metrics ¶

get_latest_metrics() -> PerformanceMetrics | None

Get the latest performance metrics.

Returns:

Type	Description
`PerformanceMetrics \| None`	The most recent metrics, or `None` if no operations have been
`PerformanceMetrics \| None`	profiled.

get_all_metrics ¶

get_all_metrics() -> list[PerformanceMetrics]

Get all performance metrics.

Returns:

Type	Description
`list[PerformanceMetrics]`	A copy of all recorded metrics.

get_snapshots ¶

get_snapshots() -> list[MemorySnapshot]

Get all memory snapshots.

Returns:

Type	Description
`list[MemorySnapshot]`	A copy of all recorded snapshots.

clear_history ¶

clear_history() -> None

Clear all recorded metrics and snapshots.

get_memory_summary ¶

get_memory_summary() -> dict[str, Any]

Get a summary of memory usage patterns.

Returns:

Type	Description
`dict[str, Any]`	Memory usage summary aggregating the recorded snapshots.

StreamingDataProcessor ¶

StreamingDataProcessor(scoring_methods: dict[str, IncrementalScoringMethod], batch_size: int = 1000, auto_save_interval: int | None = None)

High-level processor for streaming text data.

This class provides utilities for processing streaming text data and maintaining multiple frequency distributions efficiently.

Parameters:

Name	Type	Description	Default
`scoring_methods`	`dict[str, IncrementalScoringMethod]`	Mapping of names to scoring methods to maintain.	required
`batch_size`	`int`	Batch size for processing text streams. Defaults to `1000`.	`1000`
`auto_save_interval`	`int \| None`	Interval for automatic state saving, or `None` to disable. Defaults to `None`.	`None`

Examples:

Process a batch through several methods and query one of them:

>>> methods = {
...     "mle": StreamingMLE(logprob=False),
...     "laplace": StreamingLaplace(logprob=False),
... }
>>> processor = StreamingDataProcessor(methods)
>>> processor.process_batch(["word1", "word2", "word1"])
>>> round(processor.get_score("mle", "word1"), 4)
0.6667
>>> sorted(processor.get_statistics().keys())
['methods', 'processed_count']

Initialize streaming data processor.

process_element ¶

process_element(element: Element, count: int = 1) -> None

Process a single element.

Parameters:

Name	Type	Description	Default
`element`	`Element`	Element to process.	required
`count`	`int`	Count for the element. Defaults to `1`.	`1`

process_batch ¶

process_batch(elements: list[Element], counts: list[int] | None = None) -> None

Process a batch of elements.

Parameters:

Name	Type	Description	Default
`elements`	`list[Element]`	Elements to process.	required
`counts`	`list[int] \| None`	Count for each element, or `None` to use `1` for all.	`None`

process_text_stream ¶

process_text_stream(text_stream: Iterator[str]) -> None

Process a stream of text tokens.

Parameters:

Name	Type	Description	Default
`text_stream`	`Iterator[str]`	Stream of text tokens.	required

get_score ¶

get_score(method_name: str, element: Element) -> float

Get score for an element from a specific method.

Parameters:

Name	Type	Description	Default
`method_name`	`str`	Name of the scoring method.	required
`element`	`Element`	Element to score.	required

Returns:

Type	Description
`float`	Score for the element from the named method.

get_statistics ¶

get_statistics() -> dict[str, Any]

Get statistics for all scoring methods.

Returns:

Type	Description
`dict[str, Any]`	Statistics for each method, keyed by method name.

save_all_states ¶

save_all_states(base_filename: str) -> None

Save states of all scoring methods.

Parameters:

Name	Type	Description	Default
`base_filename`	`str`	Base filename for saving states.	required

clear_buffers ¶

clear_buffers() -> None

Clear all internal buffers.

StreamingFrequencyDistribution ¶

StreamingFrequencyDistribution(max_vocabulary_size: int | None = None, min_count_threshold: int = 1, decay_factor: float | None = None, compression_threshold: int = 10000)

Memory-efficient streaming frequency distribution with incremental updates.

This class supports real-time updates to frequency distributions while maintaining memory efficiency through various optimization strategies.

Parameters:

Name	Type	Description	Default
`max_vocabulary_size`	`int \| None`	Maximum support size to maintain, or `None` for unlimited. Defaults to `None`.	`None`
`min_count_threshold`	`int`	Minimum count for elements to be retained. Defaults to `1`.	`1`
`decay_factor`	`float \| None`	Exponential decay factor for aging old observations, or `None` for no decay. Defaults to `None`.	`None`
`compression_threshold`	`int`	Number of updates between compression operations. Defaults to `10000`.	`10000`

Examples:

Counts accumulate across single and batch updates:

>>> stream_dist = StreamingFrequencyDistribution(max_vocabulary_size=1000)
>>> stream_dist.update("word1")
>>> stream_dist.update("word1", count=5)
>>> stream_dist.update_batch(["word2", "word3", "word1"])
>>> stream_dist.get_count("word1")
7.0
>>> len(stream_dist)
3
>>> stream_dist.get_total_count()
9.0
>>> round(stream_dist.get_frequency("word1"), 4)
0.7778

Initialize streaming frequency distribution.

Parameters:

Name	Type	Description	Default
`max_vocabulary_size`	`int \| None`	Maximum support size, or `None` for unlimited.	`None`
`min_count_threshold`	`int`	Minimum count threshold for retention.	`1`
`decay_factor`	`float \| None`	Decay factor for time-based forgetting (`0 < decay < 1`), or `None` for no decay.	`None`
`compression_threshold`	`int`	Number of updates before triggering compression.	`10000`

update ¶

update(element: Element, count: int = 1) -> None

Update count for a single element.

Parameters:

Name	Type	Description	Default
`element`	`Element`	Element to update.	required
`count`	`int`	Count increment. Defaults to `1`.	`1`

update_batch ¶

update_batch(elements: list[Element], counts: list[int] | None = None) -> None

Update counts for multiple elements efficiently.

Parameters:

Name	Type	Description	Default
`elements`	`list[Element]`	Elements to update.	required
`counts`	`list[int] \| None`	Count for each element, or `None` to use `1` for all.	`None`

get_count ¶

get_count(element: Element) -> float

Get count for an element.

Parameters:

Name	Type	Description	Default
`element`	`Element`	Element to query.	required

Returns:

Type	Description
`float`	Count for the element.

get_frequency ¶

get_frequency(element: Element) -> float

Get relative frequency for an element.

Parameters:

Name	Type	Description	Default
`element`	`Element`	Element to query.	required

Returns:

Type	Description
`float`	Relative frequency (`count / total_count`).

get_total_count ¶

get_total_count() -> float

Get total count across all elements.

get_vocabulary_size ¶

get_vocabulary_size() -> int

Get current support size.

items ¶

items() -> Iterator[tuple[Element, float]]

Iterate over (element, count) pairs.

keys ¶

keys() -> Iterator[Element]

Iterate over elements.

values ¶

values() -> Iterator[float]

Iterate over counts.

to_dict ¶

to_dict() -> dict[Element, int]

Convert to regular dictionary with integer counts.

Returns:

Type	Description
`dict[Element, int]`	Dictionary representation with integer counts.

get_statistics ¶

get_statistics() -> dict[str, Any]

Get statistics about the streaming distribution.

Returns:

Type	Description
`dict[str, Any]`	Statistics dictionary with vocabulary size, total count, update
`dict[str, Any]`	count, and configuration parameters.

StreamingLaplace ¶

StreamingLaplace(initial_freqdist: dict[Element, int] | None = None, max_vocabulary_size: int | None = None, bins: int | None = None, logprob: bool = True)

Bases: StreamingMLE

Streaming Laplace smoothing with incremental updates.

Extends StreamingMLE to provide Laplace (add-one) smoothing in a streaming context.

Parameters:

Name	Type	Description	Default
`initial_freqdist`	`dict[Element, int] \| None`	Initial frequency distribution, or `None` to start empty. Defaults to `None`.	`None`
`max_vocabulary_size`	`int \| None`	Maximum support size to maintain, or `None` for unlimited. Defaults to `None`.	`None`
`bins`	`int \| None`	Total number of possible bins, or `None` to use the observed vocabulary size. Defaults to `None`.	`None`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

Unobserved elements receive smoothed probability mass:

>>> laplace = StreamingLaplace(
...     initial_freqdist={"a": 3, "b": 1}, logprob=False
... )
>>> round(laplace("a"), 4)
0.6667
>>> round(laplace("unseen"), 4)
0.1667

Initialize streaming Laplace scorer.

StreamingMLE ¶

StreamingMLE(initial_freqdist: dict[Element, int] | None = None, max_vocabulary_size: int | None = None, unobs_prob: float | None = None, logprob: bool = True)

Bases: ScoringMethod, IncrementalScoringMethod

Streaming Maximum Likelihood Estimation with incremental updates.

This implementation maintains a streaming frequency distribution and updates probability estimates incrementally as new data arrives.

Parameters:

Name	Type	Description	Default
`initial_freqdist`	`dict[Element, int] \| None`	Initial frequency distribution, or `None` to start empty. Defaults to `None`.	`None`
`max_vocabulary_size`	`int \| None`	Maximum support size to maintain, or `None` for unlimited. Defaults to `None`.	`None`
`unobs_prob`	`float \| None`	Reserved probability mass for unobserved elements, or `None`. Defaults to `None`.	`None`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Examples:

Probabilities are updated incrementally as observations arrive:

>>> streaming_mle = StreamingMLE(logprob=False)
>>> streaming_mle.update_single("word1", 5)
>>> streaming_mle.update_batch(["word2", "word3"])
>>> round(streaming_mle("word1"), 4)
0.7143
>>> streaming_mle.get_update_count()
3

Initialize streaming MLE scorer.

update_single ¶

update_single(element: Element, count: int = 1) -> None

Update with a single element observation.

update_batch ¶

update_batch(elements: list[Element], counts: list[int] | None = None) -> None

Update with multiple element observations.

get_update_count ¶

get_update_count() -> int

Get the number of updates performed.

get_streaming_statistics ¶

get_streaming_statistics() -> dict[str, Any]

Get statistics about the streaming distribution.

save_state ¶

save_state(filepath: str) -> None

Save the current state to disk.

Parameters:

Name	Type	Description	Default
`filepath`	`str`	Path to save the state.	required

load_state `classmethod` ¶

load_state(filepath: str) -> StreamingMLE

Load state from disk.

Parameters:

Name	Type	Description	Default
`filepath`	`str`	Path to load the state from.	required

Returns:

Type	Description
`StreamingMLE`	The loaded streaming MLE instance.

BatchScorer ¶

BatchScorer(scorers: dict[str, ScoringMethod])

Efficient batch scoring using multiple scoring methods.

This class allows scoring elements using multiple methods simultaneously, with optimized operations for processing large batches of data.

Parameters:

Name	Type	Description	Default
`scorers`	`dict[str, ScoringMethod]`	Mapping of scorer names to scoring methods.	required

Examples:

Score the same elements with several methods at once:

>>> from freqprob import MLE, Laplace
>>> scorers = {
...     "mle": MLE({"a": 3, "b": 2}, logprob=False),
...     "laplace": Laplace({"a": 3, "b": 2}, logprob=False),
... }
>>> batch_scorer = BatchScorer(scorers)
>>> results = batch_scorer.score_batch(["a", "b", "c"])
>>> [round(float(x), 4) for x in results["mle"]]
[0.6, 0.4, 0.0]
>>> [round(float(x), 4) for x in results["laplace"]]
[0.5714, 0.4286, 0.1429]

Initialize batch scorer with multiple scoring methods.

Parameters:

Name	Type	Description	Default
`scorers`	`dict[str, ScoringMethod]`	Mapping of scorer names to scoring methods.	required

score_batch ¶

score_batch(elements: Sequence[Element]) -> dict[str, np.ndarray[Any, Any]]

Score elements using all configured scoring methods.

Parameters:

Name	Type	Description	Default
`elements`	`Sequence[Element]`	Elements to score.	required

Returns:

Type	Description
`dict[str, ndarray[Any, Any]]`	Mapping of scorer names to score arrays.

score_and_compare ¶

score_and_compare(elements: Sequence[Element]) -> dict[str, Any]

Score elements and provide comparison statistics.

Parameters:

Name	Type	Description	Default
`elements`	`Sequence[Element]`	Elements to score.	required

Returns:

Type	Description
`dict[str, Any]`	Dictionary with per-method scores plus aggregate statistics
`dict[str, Any]`	(mean, std, min, max) and per-method rankings.

benchmark_methods ¶

benchmark_methods(elements: Sequence[Element], num_iterations: int = 100) -> dict[str, float]

Benchmark the performance of different scoring methods.

Parameters:

Name	Type	Description	Default
`elements`	`Sequence[Element]`	Elements to use for benchmarking.	required
`num_iterations`	`int`	Number of iterations for timing. Defaults to `100`.	`100`

Returns:

Type	Description
`dict[str, float]`	Mapping of method names to average execution time in seconds.

VectorizedScorer ¶

VectorizedScorer(scorer: ScoringMethod)

Wrapper class that adds vectorized operations to any scoring method.

This class wraps existing scoring methods to provide efficient batch operations using numpy arrays and vectorized computations.

Parameters:

Name	Type	Description	Default
`scorer`	`ScoringMethod`	The underlying scoring method to wrap.	required

Examples:

Score many elements at once, returning a numpy array:

>>> from freqprob import MLE
>>> scorer = MLE({"a": 3, "b": 2, "c": 1}, logprob=False)
>>> vectorized = VectorizedScorer(scorer)
>>> scores = vectorized.score_batch(["a", "b", "c", "d"])
>>> scores.shape
(4,)
>>> [round(float(x), 4) for x in scores]
[0.5, 0.3333, 0.1667, 0.0]

Initialize vectorized scorer wrapper.

Parameters:

Name	Type	Description	Default
`scorer`	`ScoringMethod`	Scoring method to wrap with vectorized operations.	required

score_batch ¶

score_batch(elements: Sequence[Element] | ndarray[Any, Any]) -> np.ndarray[Any, Any]

Score a batch of elements efficiently using vectorized operations.

Parameters:

Name	Type	Description	Default
`elements`	`Sequence[Element] \| ndarray[Any, Any]`	Elements to score, as a sequence or numpy array.	required

Returns:

Type	Description
`ndarray[Any, Any]`	Array of scores for the elements, in the same order as the input.

Examples:

>>> from freqprob import MLE
>>> scorer = MLE({"a": 3, "b": 2}, logprob=False)
>>> vectorized = VectorizedScorer(scorer)
>>> scores = vectorized.score_batch(["a", "b", "unknown"])
>>> [round(float(x), 4) for x in scores]
[0.6, 0.4, 0.0]

score_parallel ¶

score_parallel(element_batches: Sequence[Sequence[Element]]) -> list[np.ndarray[Any, Any]]

Score multiple batches in parallel (when available).

Parameters:

Name	Type	Description	Default
`element_batches`	`Sequence[Sequence[Element]]`	List of element batches to score.	required

Returns:

Type	Description
`list[ndarray[Any, Any]]`	List of score arrays, one per input batch.

score_matrix ¶

score_matrix(elements_2d: Sequence[Sequence[Element]]) -> np.ndarray[Any, Any]

Score a 2D array of elements, returning a matrix of scores.

Parameters:

Name	Type	Description	Default
`elements_2d`	`Sequence[Sequence[Element]]`	2D list of elements to score.	required

Returns:

Type	Description
`ndarray[Any, Any]`	2D array of scores with shape
`ndarray[Any, Any]`	`(len(elements_2d), max_row_length)`; shorter rows are padded
`ndarray[Any, Any]`	with the default probability.

top_k_elements ¶

top_k_elements(k: int) -> tuple[Sequence[Element], np.ndarray[Any, Any]]

Get the top-k highest scoring elements.

Parameters:

Name	Type	Description	Default
`k`	`int`	Number of top elements to return.	required

Returns:

Type	Description
`Sequence[Element]`	A tuple of `(elements, scores)` for the top-k elements, ordered
`ndarray[Any, Any]`	from highest to lowest score.

percentile_scores ¶

percentile_scores(elements: Sequence[Element], percentiles: list[float]) -> np.ndarray[Any, Any]

Compute percentile ranks for element scores.

Parameters:

Name	Type	Description	Default
`elements`	`Sequence[Element]`	Elements to compute percentile ranks for.	required
`percentiles`	`list[float]`	Percentiles to compute (0-100).	required

Returns:

Type	Description
`ndarray[Any, Any]`	Percentile rank for each element.

clear_all_caches ¶

clear_all_caches() -> None

Clear all global computation caches.

Empties both the Simple Good-Turing cache and the general-purpose cache used to memoize expensive smoothing computations. Useful for freeing memory or forcing recomputation.

Examples:

>>> from freqprob import clear_all_caches, get_cache_stats
>>> clear_all_caches()
>>> get_cache_stats()["sgt_cache_size"]
0

get_cache_stats ¶

get_cache_stats() -> dict[str, int]

Get statistics about global cache usage.

Reports the current number of entries held in each global cache.

Returns:

Type	Description
`dict[str, int]`	A mapping with the keys `"sgt_cache_size"` and `"general_cache_size"`,
`dict[str, int]`	each giving the number of cached entries in the corresponding cache.

Examples:

>>> from freqprob import clear_all_caches, get_cache_stats
>>> clear_all_caches()
>>> stats = get_cache_stats()
>>> "sgt_cache_size" in stats
True
>>> stats["general_cache_size"]
0

cross_entropy ¶

cross_entropy(model: ScoringMethod, test_data: Iterable[Element]) -> float

Calculate the cross-entropy of a model on test data.

Cross-entropy is the average negative log-probability the model assigns to the test data. Lower values indicate a better fit.

Parameters:

Name	Type	Description	Default
`model`	`ScoringMethod`	A fitted probability model. Must be configured with `logprob=True`.	required
`test_data`	`Iterable[Element]`	Iterable of test elements to score.	required

Returns:

Type	Description
`float`	The cross-entropy value (in nats, since natural logs are used).

Raises:

Type	Description
`ValueError`	If `model` is not configured for log-probabilities.

Examples:

>>> from freqprob import MLE, cross_entropy
>>> model = MLE({"a": 3, "b": 2, "c": 1}, logprob=True)
>>> round(cross_entropy(model, ["a", "b", "a", "c"]), 4)
1.0692

kl_divergence ¶

kl_divergence(p_model: ScoringMethod, q_model: ScoringMethod, test_data: Iterable[Element]) -> float

Calculate the Kullback-Leibler divergence between two models.

KL divergence measures how much the approximate distribution Q diverges from the reference distribution P. It is not symmetric: KL(P||Q) != KL(Q||P).

Parameters:

Name	Type	Description	Default
`p_model`	`ScoringMethod`	The reference probability model `P`. Must use `logprob=True`.	required
`q_model`	`ScoringMethod`	The approximate probability model `Q`. Must use `logprob=True`.	required
`test_data`	`Iterable[Element]`	Iterable of test elements over which to accumulate the divergence.	required

Returns:

Type	Description
`float`	The KL divergence value.

Raises:

Type	Description
`ValueError`	If either model is not configured for log-probabilities.

Examples:

>>> from freqprob import MLE, Laplace, kl_divergence
>>> p_model = MLE({"a": 3, "b": 2, "c": 1}, logprob=True)
>>> q_model = Laplace({"a": 3, "b": 2, "c": 1}, logprob=True)
>>> round(kl_divergence(p_model, q_model, ["a", "b", "a", "c"]), 4)
0.0698

model_comparison ¶

model_comparison(models: dict[str, ScoringMethod], test_data: Iterable[Element]) -> dict[str, dict[str, float]]

Compare multiple models using perplexity and cross-entropy.

Each model is evaluated on the same test data, producing a nested mapping of model name to its metric values.

Parameters:

Name	Type	Description	Default
`models`	`dict[str, ScoringMethod]`	Mapping of model names to fitted models. Every model must use `logprob=True`.	required
`test_data`	`Iterable[Element]`	Iterable of test elements. It is materialized once and reused across all models.	required

Returns:

Type	Description
`dict[str, dict[str, float]]`	A mapping of each model name to a ``{"perplexity": ..., "cross_entropy":
`dict[str, dict[str, float]]`	...}`` dictionary.

Raises:

Type	Description
`ValueError`	If any model is not configured for log-probabilities.

Examples:

>>> from freqprob import MLE, Laplace, model_comparison
>>> models = {
...     "mle": MLE({"a": 3, "b": 2, "c": 1}, logprob=True),
...     "laplace": Laplace({"a": 3, "b": 2, "c": 1}, logprob=True),
... }
>>> results = model_comparison(models, ["a", "b", "a", "c"])
>>> sorted(results)
['laplace', 'mle']
>>> round(results["mle"]["perplexity"], 4)
2.913
>>> round(results["laplace"]["cross_entropy"], 4)
1.0561

perplexity ¶

perplexity(model: ScoringMethod, test_data: Iterable[Element]) -> float

Calculate the perplexity of a model on test data.

Perplexity is exp(H(p)) where H(p) is the cross-entropy. Lower perplexity indicates a model that better predicts the test data.

Parameters:

Name	Type	Description	Default
`model`	`ScoringMethod`	A fitted probability model. Must be configured with `logprob=True`.	required
`test_data`	`Iterable[Element]`	Iterable of test elements to score.	required

Returns:

Type	Description
`float`	The perplexity value.

Raises:

Type	Description
`ValueError`	If `model` is not configured for log-probabilities.

Examples:

>>> from freqprob import MLE, perplexity
>>> model = MLE({"a": 3, "b": 2, "c": 1}, logprob=True)
>>> round(perplexity(model, ["a", "b", "a", "c"]), 4)
2.913

create_lazy_laplace ¶

create_lazy_laplace(freqdist: FrequencyDistribution, bins: int | None = None, logprob: bool = True) -> LazyScoringMethod

Create a lazy Laplace (add-one) smoothing scorer.

Builds a LazyScoringMethod that computes Laplace-smoothed probabilities on demand, fitted to the given distribution.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`bins`	`int \| None`	Total number of possible elements (vocabulary size). When `None` (the default), the number of observed elements is used.	`None`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Returns:

Type	Description
`LazyScoringMethod`	A fitted lazy Laplace scorer.

Examples:

>>> scorer = create_lazy_laplace({"a": 3, "b": 2, "c": 1}, logprob=False)
>>> scorer("a")
0.4444444444444444
>>> scorer("unseen")
0.1111111111111111

create_lazy_mle ¶

create_lazy_mle(freqdist: FrequencyDistribution, unobs_prob: float | None = None, logprob: bool = True) -> LazyScoringMethod

Create a lazy Maximum Likelihood Estimation scorer.

Builds a LazyScoringMethod that computes MLE probabilities on demand, fitted to the given distribution.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Mapping of elements to observed counts.	required
`unobs_prob`	`float \| None`	Reserved probability mass for unobserved elements, or `None` for plain MLE. Defaults to `None`.	`None`
`logprob`	`bool`	Return log-probabilities if `True` (the default), otherwise plain probabilities.	`True`

Returns:

Type	Description
`LazyScoringMethod`	A fitted lazy MLE scorer.

Examples:

>>> scorer = create_lazy_mle({"a": 3, "b": 2, "c": 1}, logprob=False)
>>> scorer("a")
0.5
>>> scorer("c")
0.16666666666666666
>>> scorer("unseen")
0.0

create_compressed_distribution ¶

create_compressed_distribution(freqdist: FrequencyDistribution, quantization_levels: int | None = None, use_compression: bool = True) -> CompressedFrequencyDistribution

Create a compressed frequency distribution from a regular one.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Original frequency distribution.	required
`quantization_levels`	`int \| None`	Number of quantization levels for compression, or `None` for exact counts. Defaults to `None`.	`None`
`use_compression`	`bool`	Whether to use data compression when serializing. Defaults to `True`.	`True`

Returns:

Type	Description
`CompressedFrequencyDistribution`	A compressed distribution populated from `freqdist`.

Examples:

>>> dist = create_compressed_distribution({"a": 10, "b": 5})
>>> dist.get_count("a")
10
>>> dist.get_vocabulary_size()
2

create_sparse_distribution ¶

create_sparse_distribution(freqdist: FrequencyDistribution, default_count: int = 0) -> SparseFrequencyDistribution

Create a sparse frequency distribution from a regular one.

Parameters:

Name	Type	Description	Default
`freqdist`	`FrequencyDistribution`	Original frequency distribution.	required
`default_count`	`int`	Default count for unobserved elements. Defaults to `0`.	`0`

Returns:

Type	Description
`SparseFrequencyDistribution`	A sparse distribution populated from `freqdist`.

Examples:

>>> dist = create_sparse_distribution({"a": 10, "b": 5})
>>> dist.get_count("a")
10
>>> dist.get_count("unseen")
0

create_vectorized_batch_scorer ¶

create_vectorized_batch_scorer(scorers: dict[str, ScoringMethod]) -> BatchScorer

Create a batch scorer handling multiple scoring methods efficiently.

Parameters:

Name	Type	Description	Default
`scorers`	`dict[str, ScoringMethod]`	Mapping of scorer names to scoring methods.	required

Returns:

Type	Description
`BatchScorer`	A `BatchScorer` wrapping the given scoring methods.

Examples:

>>> from freqprob import MLE
>>> batch_scorer = create_vectorized_batch_scorer(
...     {"mle": MLE({"a": 3, "b": 2}, logprob=False)}
... )
>>> scores = batch_scorer.score_batch(["a", "b"])
>>> [round(float(x), 4) for x in scores["mle"]]
[0.6, 0.4]

generate_ngrams ¶

generate_ngrams(text: str | list[str], n: int) -> list[tuple[str, ...]]

Generate n-grams from text.

A string input is treated as a sequence of characters, while a list input is treated as a sequence of tokens. Each n-gram is returned as a tuple.

Parameters:

Name	Type	Description	Default
`text`	`str \| list[str]`	Input text, either a string (split into characters) or a list of tokens.	required
`n`	`int`	Size of the n-grams to generate. Must be positive.	required

Returns:

Type	Description
`list[tuple[str, ...]]`	A list of n-gram tuples. Empty if the input has fewer than `n` items.

Raises:

Type	Description
`ValueError`	If `n` is not positive.

Examples:

Character n-grams from a string:

>>> from freqprob import generate_ngrams
>>> generate_ngrams("abcd", 2)
[('a', 'b'), ('b', 'c'), ('c', 'd')]

Token n-grams from a list:

>>> generate_ngrams(["hello", "world", "test"], 2)
[('hello', 'world'), ('world', 'test')]

ngram_frequency ¶

ngram_frequency(text: str | list[str], n: int, normalize: bool = False) -> dict[tuple[str, ...], int | float]

Compute n-gram frequency from text.

Generates n-grams with :func:generate_ngrams and counts them. Counts are returned by default, or relative frequencies when normalize is set.

Parameters:

Name	Type	Description	Default
`text`	`str \| list[str]`	Input text, either a string (split into characters) or a list of tokens.	required
`n`	`int`	Size of the n-grams to count. Must be positive.	required
`normalize`	`bool`	If `True`, return relative frequencies that sum to `1.0` instead of raw counts. Defaults to `False`.	`False`

Returns:

Type	Description
`dict[tuple[str, ...], int \| float]`	A mapping of each n-gram tuple to its frequency (an `int` count, or a
`dict[tuple[str, ...], int \| float]`	`float` relative frequency when `normalize` is `True`).

Examples:

>>> from freqprob import ngram_frequency
>>> freq = ngram_frequency(["a", "b", "a", "b"], 2)
>>> freq[("a", "b")]
2
>>> len(freq)
2

word_frequency ¶

word_frequency(text: str | list[str], normalize: bool = False) -> dict[str, int | float]

Compute word frequency from text.

A string input is split on whitespace into words; a list input is used as-is. Counts are returned by default, or relative frequencies when normalize is set.

Parameters:

Name	Type	Description	Default
`text`	`str \| list[str]`	Input text, either a whitespace-delimited string or a list of tokens.	required
`normalize`	`bool`	If `True`, return relative frequencies that sum to `1.0` instead of raw counts. Defaults to `False`.	`False`

Returns:

Type	Description
`dict[str, int \| float]`	A mapping of each word to its frequency (an `int` count, or a `float`
`dict[str, int \| float]`	relative frequency when `normalize` is `True`).

Examples:

Raw counts:

>>> from freqprob import word_frequency
>>> word_frequency("hello world hello")
{'hello': 2, 'world': 1}

Normalized frequencies:

>>> freq = word_frequency(["hello", "world", "hello"], normalize=True)
>>> round(freq["hello"], 4)
0.6667

API Reference¶

ScoringMethod ¶

fit ¶

score ¶

predict ¶

save ¶

load classmethod ¶

ELE ¶

Laplace ¶

Lidstone ¶

MLE ¶

Random ¶

Uniform ¶

Bayesian ¶

CertaintyDegree ¶

SimpleGoodTuring ¶

total_unseen_mass property ¶

WittenBell ¶

Interpolated ¶

KneserNey ¶

ModifiedKneserNey ¶

LazyBatchScorer ¶

score_batch ¶

score_streaming ¶

get_access_statistics ¶

LazyScoringMethod ¶

precompute_batch ¶

get_computed_elements ¶

force_full_computation ¶

CompressedFrequencyDistribution ¶

update ¶

get_count ¶

get_total_count ¶

get_vocabulary_size ¶

items ¶

keys ¶

values ¶

to_dict ¶

get_memory_usage ¶

compress_to_bytes ¶

decompress_from_bytes classmethod ¶

QuantizedProbabilityTable ¶

set_probabilities ¶

set_probability ¶

get_probability ¶

set_default_probability ¶

get_elements ¶

get_memory_usage ¶

get_quantization_error_stats ¶

SparseFrequencyDistribution ¶

update ¶

increment ¶

get_count ¶

get_total_count ¶

get_vocabulary_size ¶

get_top_k ¶

get_elements_with_count_range ¶

get_count_histogram ¶

items ¶

keys ¶

values ¶

to_dict ¶

get_memory_usage ¶

DistributionMemoryAnalyzer ¶

measure_distribution_memory ¶

compare_representations ¶

benchmark_scoring_methods ¶

MemoryMonitor ¶

start_monitoring ¶

stop_monitoring ¶

check_memory ¶

get_monitoring_report ¶

MemoryProfiler ¶

take_snapshot ¶

profile_operation ¶

get_latest_metrics ¶

get_all_metrics ¶

get_snapshots ¶

clear_history ¶

get_memory_summary ¶

load `classmethod` ¶

total_unseen_mass `property` ¶

decompress_from_bytes `classmethod` ¶

load_state `classmethod` ¶