TransWikia.com

confidence intervals for the Poisson process ($lambda$) sampled with uncertainty

Cross Validated Asked by Gideon Kogan on November 2, 2021

Say, I have a Poisson process which was measured $N$ times, and each measurement produced $k_i$ value.
Also, $k_i$ are events that I have to detect and my detection probability is $p$. In fact, I detect $widetilde{k}_i$ which correlates with $k_i$ via $p$.

Knowing, the number of the detected events $widetilde{k}_i$, the constant probability $p$, and the constant measurement time $T$, how I find the confidence interval for estimation of Poisson’s $lambda$?

One Answer

There are at least 19 ways to estimate a confidence interval (CI) based on samples from a Poisson distribution; see this page and its links for extensive discussion. The question here is what is different when you don't observe the "true" underlying process, with rate $lambda$ per unit time, but instead have a known, fixed probability $p$ of detecting a true event. The principles below will apply whichever CI method you choose.

TL;DR: what you are sampling from is still a Poisson distribution, but now with rate $p lambda$ per unit time. You first estimate $p hatlambda$ and its CI for the Poisson-distributed events you observed. Then, as $p$ is known and fixed, you correct back to the "true" process by dividing both $p hatlambda$ and its CI by $p$.

First, recognize that what matters is the total time over which you have collected counts, as counts over different time periods are independent with a Poisson distribution. Whether you have 100 observations each lasting 1 second or 1 observation lasting 100 seconds doesn't matter. So I'll take your total time of observation to be your $T$ and your total observed counts to be $tilde k$, just adding together any counts and time periods that might in practice have been observed separately.

Now think about the derivation of the Poisson distribution from the binomial distribution:

one assumes that there exists a small enough subinterval for which the probability of an event occurring twice is "negligible". With this assumption one can derive the Poisson distribution from the Binomial one, given only the information of expected number of total events in the whole interval.

In your case the word "events" is used in two ways: the number of true events, and the number of events that you observe. The expected number of true events during total time $T$ is $lambda T$. The expected number of observed events over that time, with a known and fixed probability $p$ of detecting a true event, is $plambda T$. The observations are still a sample of rare independent events, the essential characterization of a Poisson distribution, just with a lower rate than the true rate. The critical point is that, as the observations are drawn from a Poisson distribution, all the extra variability introduced by the detection probability $p$ is captured in the expected number of observed events, $plambda T$.

Let's start with the estimate for the observed rate, $phatlambda$. That is simply $tilde k/T$. For the CI around that estimate, use as a simple example the formula with a normal approximation and the square root of the variance. With variance equal to the mean in a Poisson distribution, the CI around that estimate would be $pm 1.96 sqrt{tilde k/T}$.

With the assumption that $p$ is known and fixed, you correct both the point estimate and the CI back to the scale of the "true" distribution by dividing each of them by $p$. That is, you have:

$$hatlambda = frac{tilde k}{pT} pm 1.96 frac{1}{p} sqrt{frac{tilde k}{T}}.$$

This same result could also have been derived from the basic properties of the variance in terms of multiplication by the constant $1/p$. Note that the CI is wider by a factor of $sqrt{1/p}$ than it would have been if you detected all the true events with $p = 1$, as $tilde k/p$ approaches the true number of events $k$ in your terminology as $p$ approaches 1. For other CI estimate methods the principle is the same: calculate the CI for the observed counts, then divide by $p$.

Answered by EdM on November 2, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP