Results

The results of the pull experiment are summarized in Tables 1, 2, and 3. It clearly shows that the standard 1D method breaks down between $\rho_{12}(\ensuremath {{\cal{C}}}_1)=0.1$ and $\rho_{12}(\ensuremath {{\cal{C}}}_1)=0.2$ (c.f. Table 1). It confirms our suspicion that the use of 1D PDFs is adequate only for very small correlations. Using the 1D method for correlated variables can lead to huge inconsistencies. The PCA method is clearly biased for all values of $\rho_{12}(\ensuremath {{\cal{C}}}_1)$ (c.f. Table 2). This was to be expected since the PCA method was designed for the calculation of likelihood ratios for signal selection over a small background. Even if the PCA method is not adequate here, it does better than the 1D method for large correlations $\rho_{12}(\ensuremath {{\cal{C}}}_1)\raisebox{-.65ex}{\rlap{$\sim$}} \raisebox{.45ex}{$>$}0.5$. Overall, the Multi-D method (c.f. Table 3) is the approach with the smallest biases $\bar{\ensuremath {{\cal{P}}}}\approx$ -0.022, -0.055, -0.012 for $\ensuremath {{\cal {C}}}_1$, $\ensuremath {{\cal {C}}}_2$, and $\ensuremath {{\cal {C}}}_3$, respectively. Even if the biases are small for the multi-D approach, they are REAL since the error on the mean is expected to be $\sigma_{\bar{\ensuremath {{\cal{P}}}}}=\sigma_{\ensuremath {{\cal{P}}}}/n=0.010$ for a sample size of $n=10,000$. In fact, it shows the biases inherent to the use of binned data. The binning effects of the PDFs on the extraction of the estimators $\vec{\theta}$ is a know limitation of the binned maximum likelihood method and the ensemble tests are a robust way to quantify the loss of information due to binned data. The binning effect were studied by changing the number of bins used in the maximum likelihood fits. As the number of bins decreases (increases), the bias on the pull increases (decreases). When the number of bins $\ensuremath {{\cal{N}}}\to {\rm {large}}$, the fit is unstable because there is a small number of events per bin for the calculation of the template joint PDFs $P_k(\vec{x})$.

Another feature which can be studied with the ensemble test is the possible bias of the statistical error returned by the maximum likelihood fitter. In the study presented here the sample standard deviation is $\sigma_\ensuremath {{\cal{P}}}\sim 1$, but in some cases $\sigma_\ensuremath {{\cal{P}}}$ is NOT consistent with unity since the error on the sample standard deviation is given by $\Delta \sigma_\ensuremath {{\cal{P}}}= \sigma_\ensuremath {{\cal{P}}}/ \sqrt{2(n-1)}=0.0071$ for $n=10,000$. From Tables 1, 2, and 3, $\sigma_\ensuremath {{\cal{P}}}$ for $\ensuremath {{\cal {C}}}_1$ and $\ensuremath {{\cal {C}}}_2$ are consistent with unity; while $\sigma_\ensuremath {{\cal{P}}}$ for $\ensuremath {{\cal {C}}}_3$ is not.

The ensemble test allow to quantify the inherent binning effects. In complicated statistical analysis, biases due to binned data must be evaluated and quoted as systematic errors if they are not negligible compare to other uncertainties.