Distribuição de probabilidade para diferentes probabilidades

Se eu quisesse obter a probabilidade de 9 sucessos em 16 tentativas, com cada tentativa tendo 0,6 de probabilidade, eu poderia usar uma distribuição binomial. O que eu poderia usar se cada uma das 16 tentativas tivesse uma probabilidade diferente de sucesso?

Esta é a soma de 16 ensaios binomiais (presumivelmente independentes). A suposição de independência nos permite multiplicar probabilidades. Portanto, após duas tentativas com probabilidades $p_1$ e $p_2$ de sucesso, a chance de sucesso em ambas as tentativas é $p_1 p_2$ , a chance de sucesso não é $(1-p_1)(1-p_2)$ e a chance de sucesso um sucesso é $p_1(1-p_2) + (1-p_1)p_2$ . Essa última expressão deve sua validade ao fato de que as duas maneiras de obter exatamente um sucesso são mutuamente exclusivas: no máximo uma delas pode realmente acontecer. Isso significa que suas probabilidadesaumentam.

Por meio dessas duas regras - as probabilidades independentes se multiplicam e as mutuamente exclusivas - você pode descobrir as respostas para, digamos, 16 tentativas com probabilidades $p_1, \ldots, p_{16}$ . Para fazer isso, você precisa explicar todas as maneiras de obter cada número determinado de sucessos (como 9). Existem $\binom{16}{9} = 11440$maneiras de alcançar 9 sucessos. Um deles, por exemplo, ocorre quando os ensaios 1, 2, 4, 5, 6, 11, 12, 14 e 15 são sucessos e os outros são fracassos. Os sucessos tiveram probabilidades$p_1, p_2, p_4, p_5, p_6, p_{11}, p_{12}, p_{14},$e$p_{15}$ e as falhas tinha probabilidades$1-p_3, 1-p_7, \ldots, 1-p_{13}, 1-p_{16}$ . A multiplicação desses 16 números dá a chancedessa sequência específica de resultados. A soma desse número, juntamente com os 11.439 restantes, fornece a resposta.

Claro que você usaria um computador.

Com muitos mais de 16 ensaios, é necessário aproximar a distribuição. Nenhum fornecido da probabilidade $p_i$ e $1-p_i$ ficar muito pequeno, uma aproximação normal tende a funcionar bem. Com esse método, você observa que a expectativa da soma de $n$ tentativas é $\mu = p_1 + p_2 + \cdots + p_n$ e (porque as tentativas são independentes) a variação é $\sigma^2 = p_1(1-p_1) + p_2(1-p_2) + \cdots + p_n(1-p_n)$ . Você então finge que a distribuição de somas é Normal com média$\mu$ e desvio padrão$\sigma$ . As respostas tendem a ser boas para calcular probabilidades correspondentes a uma proporção de sucessos que difere de$\mu$ por não mais do que alguns múltiplos de$\sigma$ . À medida que$n$ cresce, essa aproximação fica ainda mais precisa e funciona para múltiplos ainda maiores de$\sigma$ de$\mu$ .

Uma alternativa à aproximação normal do @ whuber é usar probabilidades de "mistura" ou um modelo hierárquico. Isso se aplica quando o são similares de alguma maneira, e você pode modelar este por uma distribuição de probabilidade p i ~ D i s t ( θ ) com uma função de densidade de g ( p | θ ) indexados por algum parâmetro θ . você obtém uma equação integral:$p_i$$p_i\sim Dist(\theta)$$g(p|\theta)$$\theta$

$$Pr(s=9|n=16,\theta)={16 \choose 9}\int_{0}^{1} p^{9}(1-p)^{7}g(p|\theta)dp$$

A probabilidade binomial vem da configuração , a aproximação normal vem da (acho) configuração g ( p | θ ) = g ( p | μ , σ ) = $g(p|\theta)=\delta(p-\theta)$(comμeσconforme definido na resposta do @ whuber) e, em seguida, observando que as "caudas" deste PDF caem acentuadamente em torno do pico.$g(p|\theta)=g(p|\mu,\sigma)=\frac{1}{\sigma}\phi\left(\frac{p-\mu}{\sigma}\right)$$\mu$$\sigma$

Você também pode usar uma distribuição beta, o que levaria a uma forma analítica simples e que não precisará sofrer do problema do "pequeno p" que a aproximação normal faz - já que o beta é bastante flexível. Usando uma distribuição com α , β definida pelas soluções para as seguintes equações (estas são as estimativas de "divergência mínima de KL"):$beta(\alpha,\beta)$$\alpha,\beta$

$$\psi(\alpha)-\psi(\alpha+\beta)=\frac{1}{n}\sum_{i=1}^{n}log[p_{i}]$$ $$\psi(\beta)-\psi(\alpha+\beta)=\frac{1}{n}\sum_{i=1}^{n}log[1-p_{i}]$$

Where $\psi(.)$ is the digamma function - closely related to harmonic series.

We get the "beta-binomial" compound distribution:

$${16 \choose 9}\frac{1}{B(\alpha,\beta)}\int_{0}^{1} p^{9+\alpha-1}(1-p)^{7+\beta-1}dp ={16 \choose 9}\frac{B(\alpha+9,\beta+7)}{B(\alpha,\beta)}$$

This distribution converges towards a normal distribution in the case that @whuber points out - but should give reasonable answers for small $n$ and skewed $p_i$ - but not for multimodal $p_i$, as beta distribution only has one peak. But you can easily fix this, by simply using $M$ beta distributions for the $M$ modes. You break up the integral from $0<p<1$ into $M$ pieces so that each piece has a unique mode (and enough data to estimate parameters), and fit a beta distribution within each piece. then add up the results, noting that making the change of variables $p=\frac{x-L}{U-L}$ for $L<x<U$ the beta integral transforms to:

$$B(\alpha,\beta)=\int_{L}^{U}\frac{(x-L)^{\alpha-1}(U-x)^{\beta-1}}{(U-L)^{\alpha+\beta-1}}dx$$

Let $X_i$ ~ $Bernoulli(p_i)$ with probability generating function (pgf):

$$\text{pgf} = E[t^{X_i}] = 1 - p_i (1-t)$$

Let $S = \sum_{i=1}^n X_i$ denote the sum of $n$ such independent random variables. Then, the pgf for the sum $S$ of $n=16$ such variables is:

$$\begin{align*}\displaystyle \text{pgfS} &= E[t^S] \\&= E[t^{X_1}] E[t^{X_2}] \dots E[t^{X_{16}}] \text{ (... by independence)} \\ &= \prod _{i=1}^{16} \left(1-p_i(1-t) \right)\end{align*}$$

We seek $P(S=9)$, which is:

$$\frac{1}{9!}\frac{d^9 \text{pgfS}}{dt^9}|_{t=0}$$

ALL DONE. This produces the exact symbolic solution as a function of the $p_i$. The answer is rather long to print on screen, but it is entirely tractable, and takes less than $\frac{1}{100}$th of a second to evaluate using Mathematica on my computer.

Examples

If $p_i = \frac{i}{17}, i= 1 \text{ to } 16$, then: $P(S=9) = \frac{9647941854334808184}{48661191875666868481} = 0.198268 \dots$

If $p_i = \frac{\sqrt{i}}{17}, i= 1 \text{ to } 16$, then: $P(S=9) = 0.000228613 \dots$

More than 16 trials?

With more than 16 trials, there is no need to approximate the distribution. The above exact method works just as easily for examples with say $n = 50$ or $n = 100$. For instance, when $n = 50$, it takes less than $\frac{1}{10}$th of second to evaluate the entire pmf (i.e. at every value $s = 0, 1, \dots, 50$) using the code below.

Mathematica code

Given a vector of $p_i$ values, say:

n = 16;   pvals = Table[Subscript[p, i] -> i/(n+1), {i, n}];

... here is some Mathematica code to do everything required:

pgfS = Expand[ Product[1-(1-t)Subscript[p,i], {i, n}] /. pvals];
D[pgfS, {t, 9}]/9! /. t -> 0  // N

0.198268

To derive the entire pmf:

Table[D[pgfS, {t,s}]/s! /. t -> 0 // N, {s, 0, n}]

... or use the even neater and faster (thanks to a suggestion from Ray Koopman below):

CoefficientList[pgfS, t] // N

For an example with $n = 1000$, it takes just 1 second to calculate pgfS, and then 0.002 seconds to derive the entire pmf using CoefficientList, so it is extremely efficient.

@wolfies comment, and my attempt at a response to it revealed an important problem with my other answer, which I will discuss later.

Specific Case (n=16)

There is a fairly efficient way to code up the full distribution by using the "trick" of using base 2 (binary) numbers in the calculation. It only requires 4 lines of R code to get the full distribution of $Y=\sum_{i=1}^{n} Z_i$ where $Pr(Z_i=1)=p_i$. Basically, there are a total of $2^n$ choices of the vector $z=(z_1,\dots,z_n)$ that the binary variables $Z_i$ could take. Now suppose we number each distinct choice from $1$ up to $2^n$. This on its own is nothing special, but now suppose that we represent the "choice number" using base 2 arithmetic. Now take $n=3$ so I can write down all the choices so there are $2^3=8$ choices. Then $1,2,3,4,5,6,7,8$ in "ordinary numbers" becomes $1,10,11,100,101,110,111,1000$ in "binary numbers". Now suppose we write these as four digit numbers, then we have $0001,0010,0011,0100,0101,0110,0111,1000$. Now look at the last $3$ digits of each number - $001$ can be thought of as $(Z_1=0,Z_2=0,Z_3=1)\implies Y=1$, etc. Counting in binary form provides an efficient way to organise the summation. Fortunately, there is an R function which can do this binary conversion for us, called intToBits(x) and we convert the raw binary form into a numeric via as.numeric(intToBits(x)), then we will get a vector with $32$ elements, each element being the digit of the base 2 version of our number (read from right to left, not left to right). Using this trick combined with some other R vectorisations, we can calculate the probability that $y=9$ in 4 lines of R code:

exact_calc <- function(y,p){
    n       <- length(p)
    z       <- t(matrix(as.numeric(intToBits(1:2^n)),ncol=2^n))[,1:n] #don't need columns n+1,...,32 as these are always 0
    pz      <- z%*%log(p/(1-p))+sum(log(1-p))
    ydist   <- rowsum(exp(pz),rowSums(z))
    return(ydist[y+1])
}

Plugging in the uniform case $p_i^{(1)}=\frac{i}{17}$ and the sqrt root case $p_i^{(2)}=\frac{\sqrt{i}}{17}$ gives a full distribution for y as:

$$\begin{array}{c|c}y & Pr(Y=y|p_i=\frac{i}{17}) & Pr(Y=y|p_i=\frac{\sqrt{i}}{17})\\ \hline 0 & 0.0000 & 0.0558 \\ 1 & 0.0000 & 0.1784 \\ 2 & 0.0003 & 0.2652 \\ 3 & 0.0026 & 0.2430 \\ 4 & 0.0139 & 0.1536 \\ 5 & 0.0491 & 0.0710 \\ 6 & 0.1181 & 0.0248 \\ 7 & 0.1983 & 0.0067 \\ 8 & 0.2353 & 0.0014 \\ 9 & 0.1983 & 0.0002 \\ 10 & 0.1181 & 0.0000 \\ 11 & 0.0491 & 0.0000 \\ 12 & 0.0139 & 0.0000 \\ 13 & 0.0026 & 0.0000 \\ 14 & 0.0003 & 0.0000 \\ 15 & 0.0000 & 0.0000 \\ 16 & 0.0000 & 0.0000 \\ \end{array}$$

So for the specific problem of $y$ successes in $16$ trials, the exact calculations are straight-forward. This also works for a number of probabilities up to about $n=20$ - beyond that you are likely to start to run into memory problems, and different computing tricks are needed.

Note that by applying my suggested "beta distribution" we get parameter estimates of $\alpha=\beta=1.3206$ and this gives a probability estimate that is nearly uniform in $y$, giving an approximate value of $pr(y=9)=0.06799\approx\frac{1}{17}$. This seems strange given that a density of a beta distribution with $\alpha=\beta=1.3206$ closely approximates the histogram of the $p_i$ values. What went wrong?

General Case

I will now discuss the more general case, and why my simple beta approximation failed. Basically, by writing $(y|n,p)\sim Binom(n,p)$ and then mixing over $p$ with another distribution $p\sim f(\theta)$ is actually making an important assumption - that we can approximate the actual probability with a single binomial probability - the only problem that remains is which value of $p$ to use. One way to see this is to use the mixing density which is discrete uniform over the actual $p_i$. So we replace the beta distribution $p\sim Beta(a,b)$ with a discrete density of $p\sim \sum_{i=1}^{16}w_i\delta(p-p_i)$. Then using the mixing approximation can be expressed in words as choose a $p_i$ value with probability $w_i$, and assume all bernoulli trials have this probability. Clearly, for such an approximation to work well, most of the $p_i$ values should be similar to each other. This basically means that for @wolfies uniform distribution of values, $p_i=\frac{i}{17}$ results in a woefully bad approximation when using the beta mixing distribution. This also explains why the approximation is much better for $p_i=\frac{\sqrt{i}}{17}$ - they are less spread out.

The mixing then uses the observed $p_i$ to average over all possible choices of a single $p$. Now because "mixing" is like a weighted average, it cannot possibly do any better than using the single best $p$. So if the $p_i$ are sufficiently spread out, there can be no single $p$ that could provide a good approximation to all $p_i$.

One thing I did say in my other answer was that it may be better to use a mixture of beta distributions over a restricted range - but this still won't help here because this is still mixing over a single $p$. What makes more sense is split the interval $(0,1)$ up into pieces and have a binomial within each piece. For example, we could choose $(0,0.1,0.2,\dots,0.9,1)$ as our splits and fit nine binomials within each $0.1$ range of probability. Basically, within each split, we would fit a simple approximation, such as using a binomial with probability equal to the average of the $p_i$ in that range. If we make the intervals small enough, the approximation becomes arbitrarily good. But note that all this does is leave us with having to deal with a sum of indpendent binomial trials with different probabilities, instead of Bernoulli trials. However, the previous part to this answer showed that we can do the exact calculations provided that the number of binomials is sufficiently small, say 10-15 or so.

To extend the bernoulli-based answer to a binomial-based one, we simply "re-interpret" what the $Z_i$ variables are. We simply state that $Z_i=I(X_i>0)$ - this reduces to the original bernoulli-based $Z_i$ but now says which binomials the successes are coming from. So the case $(Z_1=0,Z_2=0,Z_3=1)$ now means that all the "successes" come from the third binomial, and none from the first two.

Note that this is still "exponential" in that the number of calculations is something like $k^g$ where $g$ is the number of binomials, and $k$ is the group size - so you have $Y\approx\sum_{j=1}^{g}X_j$ where $X_j\sim Bin(k,p_j)$. But this is better than the $2^{gk}$ that you'd be dealing with by using bernoulli random variables. For example, suppose we split the $n=16$ probabilities into $g=4$ groups with $k=4$ probabilities in each group. This gives $4^4=256$ calculations, compared to $2^{16}=65536$

By choosing $g=10$ groups, and noting that the limit was about $n=20$ which is about $10^7$ cells, we can effectively use this method to increase the maximum $n$ to $n=50$.

If we make a cruder approximation, by lowering $g$, we will increase the "feasible" size for $n$. $g=5$ means that you can have an effective $n$ of about $125$. Beyond this the normal approximation should be extremely accurate.

The (in general intractable) pmf is

$$\Pr(S=k) = \sum_{\substack{A\subset\{1,\dots,n\}\\ |A|=k}} \left( \prod_{i\in A} p_i \right)\left(\prod_{j\in \{1,\dots,n\}\setminus A} (1-p_j) \right) \, .$$ R code:

p <- seq(1, 16) / 17
cat(p, "\n")
n <- length(p)
k <- 9
S <- seq(1, n)
A <- combn(S, k)
pr <- 0
for (i in 1:choose(n, k)) {
    pr <- pr + exp(sum(log(p[A[,i]])) + sum(log(1 - p[setdiff(S, A[,i])])))
}
cat("Pr(S = ", k, ") = ", pr, "\n", sep = "")

For the $p_i$'s used in wolfies answer, we have:

Pr(S = 9) = 0.1982677

When $n$ grows, use a convolution.