Ensemble Tests

The motivation of an ensemble test is to investigate possible biases of the binned maximum likelihood technique used for the extraction of the unknown parameters $\vec{\theta}$. The goal of this section is to illustrate the pull technique used to check the reliability of the statistical event classification. The study has three aims:

1. Look for biases in the extracted parameters $\theta_i$.
2. Compute the expected statistical uncertainty and look for bias in the quoted errors $\sigma_{\theta_i}$ on the unknown parameters.
3. Study binning effects.

The Monte Carlo method [3,4] is used to generate samples of correlated random variables $x_1,x_2$ for the three classes. The marginal distributions for the random variables $x_1$ and $x_2$ for $\ensuremath {{\cal{C}}}_1, \ensuremath {{\cal{C}}}_2,$ and $\ensuremath {{\cal {C}}}_3$ are shown in Figure 1. While the correlation coefficients $\rho_{12}(\ensuremath {{\cal{C}}}_2)=0$ and $\rho_{12}(\ensuremath {{\cal{C}}}_3)=0$, the correlation coefficient $\rho_{12}(\ensuremath {{\cal{C}}}_1)$ is varied between 0.0 and 0.9 in step of 0.1. The 2-D projections for $x_1$ versus $x_2$ for $\ensuremath {{\cal {C}}}_1$ are depicted in Figure 2 for $\rho_{12}(\ensuremath {{\cal{C}}}_1)=$ 0.0, 0.3, 0.6 and 0.9.

Each PDF template uses 1 million events. A large number of events are needed for the computation of the multi-dimensional PDF. The ensemble test generates 10,000 experiments (i.e. samples). The size of each sample is approximately 900 events in which the fraction of each classes is about 1/3. The total number of events per experiment is Poisson distributed and the fraction of events $\theta_i \equiv N_i/N_{\rm {tot}}$ for class $\ensuremath {{\cal{C}}}_i$ is randomly assigned so that $\bar{\theta}_i({\rm {MC}}) = 1/3$. The simulated fluctuation on $\theta_i$ reflects the statistical uncertainty on the total number of events for each class.

The power of an ensemble test is that the law of large numbers and the central limit theorem ensures that the distribution of the fitted fraction $\theta_i$ and the statistical error $\sigma_{\theta_i}$ on the fitted fraction for the 10,000 experiments are Gaussian distributed. The sample mean $\bar{\theta_i}$ should be 1/3 within the statistical uncertainties and the mean $\bar{\sigma}_{\theta_i}$ should be the expected statistical error on the measurement of the unknown fractions. The residual is defined as the difference between the fitted fraction and the true fraction: $\Delta = \theta_i({\rm {FIT}}) - \bar{\theta}_i({\rm {MC}})$. Here it is very important to use the mean of the Poisson distributed $\bar{\theta}_i({\rm {MC}}) = 1/3$ for the definition of the residual (using the generated value will lead to a narrow pull distribution). The pull is $\ensuremath {{\cal {P}}}= \Delta / \sigma _{\theta _i}$. According to the central limit theorem the random variables $\ensuremath {{\cal {P}}}$ must be normally distributed $N(0,1)$. Hence, a pull experiment based on a large statistics ensemble test allows for a detailed study of the fitting procedure for the unknown parameters and their statistical uncertainties.

As an example, the results of the pull experiment are depicted in Figures 3 for the multi-D method with $\rho _{12}(\ensuremath {{\cal {C}}}_1)=0.3$. As noted before, the distribution of the pull $\ensuremath {{\cal {P}}}= \Delta / \sigma _{\theta _i}$ should be normally distributed since $E(\bar{\ensuremath {{\cal{P}}}})=0$ and $E(\sigma_{\ensuremath {{\cal{P}}}})=1$. Deviation from the $N(0,1)$ behavior is an indication of a true bias of the fitting method since the ensemble test relies on a very large number of experiments.