Comparison of the accuracy of a new screening test to that of the standard one can be implemented by administering both screening tests to a group of asymptomatic subjects for which the disease status can be determined using a gold standard (GS) test. Nevertheless, the GS test may be too costly or too invasive hence unethical to administer to all the study subjects, including those who screen negative on all the screening tests. When this is the case, relative accuracy of the two screening tests can be estimated when a randomized paired screen positive (RPSP) design is used to collect the data. However, this design contains cells with missing data, thus the likelihood function is not available. The objective of this study is to demonstrate a parsimonious way of estimating relative accuracy of the screening tests when the determination of the disease status of the subjects who screened negative on the new and the standard screening tests was not conducted due to ethical concerns.

Markov Chain Monte Carlo simulation technique is used to parsimoniously address the aforementioned shortcoming of the RPSP design when subjective approach to estimation is used. Multiple data imputation using Gibbs sampler is performed. Monte Carlo point and interval estimates of the missing data, measures of accuracy and the relative rates are computed when the tests are treated to be:

1. Conditionally independent
2. Dependent

The accuracy when the tests are dependent is higher than when the tests are conditionally independent. This is apparent from the narrower credible bounds and smaller standard errors of the true and false positive rates under this setting. Secondly, the estimates obtained under the two assumptions are better than when the missing data are ignored. Whether these differences were true or just due to chance warrant statistical investigation under different assumptions.

Keywords: conditional independence, dependence, gibbs sampling, markov chain monte carlo, subjective approach, credible bounds

Before a new test is approved for public health use its accuracy should be compared to that of the existing standard tests to determine whether the new test is superior to or at least as accurate as the available tests. This is achieved through the use of standard metrics used to compare the true positive rate (TPR) and the false positive rate (FPR) of two tests. This comparison is quantified using the relative performance or the relative accuracy of the tests used for detecting the disease.^1–5 Screen positive designs administer the gold standard test to subjects with positive outcome on at least one of the screening tests. These designs are preferred if the diagnostic test is too costly or too invasive to be applied to all the study subjects.

It is unethical to administer a cancer-confirming biopsy to subjects who screen negative on both the new and the standard screening tests because the results already suggest a low probability of cancer.^2,4 Therefore the biopsy data are missing with negative screening tests. This is what we end up with in screen positive designs. In this case disease detection probabilities (DP) and false referral probabilities (FP) are the key measures of accuracy that can be derived.⁴ These estimates will be biased because some subjects might have been missed by the screening test due to error. A way to mitigate this issue is to impute the missing data. Therefore the aim of this study is to demonstrate a parsimonious way of estimating relative accuracy of the screening tests when the determination of the disease status of the subjects who screened negative on the new and the standard screening tests was not conducted due to ethical concerns. Markov Chain Monte Carlo simulation (MCMC) technique was used to address the aforementioned shortcoming when subjective approach to estimation was adopted..^6,7 Non-informative conjugate prior distributions were utilized.

Inferences about the performance of the screening tests are based on the estimates of relative true positive rate (rTPR) and relative false positive rate (rFPR). These rates are respectively obtained by taking the ratio of the point estimates of TPR and FPR of the new screening test to that of the standard screening test.^1–4 In section 2 we give an outline of the methods that were used. The randomized paired screen positive design by⁴ is discussed in section 3. In this section we compute the measures of accuracy under the assumptions that the tests are

1. Conditionally independent
2. Dependent

The other assumptions were: data were observed with the RPSP design, the missing data were assumed to be missing at random (MAR), the TPR and the FPR of the standard screening test were assumed to be 0.5 and 0.02 respectively, and the rTPR and rFPR were assumed to be 1.5 and 1.25 respectively. In section 4 we provide the results while in section 5 we discuss, conclude and give recommendations.

In RPSP design n subjects are randomized to first receive test X1 or test X2. The second screening test and the GS test are administered only if the first screening test was positive.⁴ Data from such a design are summarized in (Table 1).

The variables $n_{jkl}^p$ and $z_{jkl}^p$ , $p=1,2,j,k,l=0,1$ , where
$j=\{\begin{array}{cc}0& foranegativeresultontest{X}_1\\ & \\ 1& forapositiveresultontest{X}_1\end{array},k=\{\begin{array}{cc}0& foranon-diseasedsubject\\ & \\ 1& foradiseasedsubject\end{array}$ and

$l=\{\begin{array}{cc}0& foranegativeresultontest{X}_2\\ & \\ 1& forapositiveresultontest{X}_2\end{array}$

denote the observed and the missing data, respectively, while the cells with “ $?•$ ” denote the unknown marginal totals. The superscripts denote the test that the subjects were first randomized to receive. Test X1 is the standard test while test X2 is the new test.

Measures of accuracy when the tests are assumed to be conditionally independent

The total number of subjects with the disease and without the disease in the design is fixed. Further, the row totals are fixed and the column totals are random. Thus binomial sampling model was appropriate for each test. Assume that the $n_{•1•}$ is equal to the actual sample size¹¹ determined and $n_{•0•}$ is equal to $\left(\frac{1-\psi }{\psi }\right)n_{•1•}$ where $\psi$ is the prevalence of the disease in the population. The marginal totals of the tests can be approximated based on the assumed TPRs or FPRs of the tests. (Table 2) summarizes the data from the RPSP design under the assumption that the tests are conditionally independent. (Table 2) is similar to the unpaired screen positive (USP) design.^1–4

Where
${{\displaystyle n}}_{111}={{\displaystyle n}}_{111}^1+{{\displaystyle n}}_{111}^2+{{\displaystyle n}}_{110}^1+{{\displaystyle z}}_{110}^2,{{\displaystyle n}}_{110}={{\displaystyle n}}_{011}^2+{{\displaystyle z}}_{011}^1+{{\displaystyle z}}_{010}^2,{{\displaystyle n}}_{011}={{\displaystyle n}}_{111}^1+{{\displaystyle n}}_{111}^2+{{\displaystyle n}}_{011}^2+{{\displaystyle z}}_{011}^1$ ${{\displaystyle n}}_{010}={{\displaystyle n}}_{110}^1+{{\displaystyle z}}_{110}^2+{{\displaystyle z}}_{010}^1+{{\displaystyle z}}_{010}^2$ ,
${{\displaystyle n}}_{101}={{\displaystyle n}}_{101}^1+{{\displaystyle n}}_{101}^2+{{\displaystyle n}}_{100}^1+{{\displaystyle z}}_{100}^2$ , ${{\displaystyle n}}_{100}={{\displaystyle n}}_{001}^2+{{\displaystyle z}}_{001}^1+{{\displaystyle z}}_{000}^1+{{\displaystyle z}}_{000}^2$ , ${{\displaystyle n}}_{001}={{\displaystyle n}}_{101}^1+{{\displaystyle n}}_{101}^2+{{\displaystyle n}}_{001}^2+{{\displaystyle z}}_{001}^1$ ,
${{\displaystyle n}}_{000}={{\displaystyle n}}_{100}^1+{{\displaystyle z}}_{000}^1+{{\displaystyle z}}_{100}^2+{{\displaystyle z}}_{000}^2$ .

The $n_{jkl}^p$ variables denote the observed data, while the $z_{jkl}^p$ variables denote the missing data from the RPSP design, $j,k,l=0,1$ .

Let

$Y\in \left\{0,1\right\}$ denote the outcome of the screening test where 1 denotes a positive outcome and 0 a negative outcome. Similarly, let $X\in \left\{0,1\right\}$ denote the chosen screening test such that X=1 if test X1 is chosen and X=0 if test X2 is chosen. This study treated the outcome of the screening tests as response and the type of test as the explanatory variable. Hence to describe the data, conditional distributions of the response variable Y given the explanatory variable X were computed. This led to computation of measures of accuracy of the screening tests. This setting treated the row totals (type of test) as fixed and forming separate binomial samples.

Under this setting the relative TPR is given by $\frac{\mathrm{Pr}\left(X_1=1|D=1\right)}{\mathrm{Pr}\left(X_2=1|D=1\right)}$ while the relative FPR is given by $\frac{\mathrm{Pr}\left(X_1=1|D=0\right)}{\mathrm{Pr}\left(X_2=1|D=0\right)}$ .

Under this assumption the missing data under the new and the standard screening tests were 35 and 23 respectively.

Measures of accuracy when the tests are assumed to be dependent

The second setting treated the screening tests to be dependent of each other. In (Table 3) the marginal totals for the levels of each test were random rather than fixed. Hence the joint distributions of the two tests follow a multinomial distribution. The total number with the disease and without the disease can be obtained as was done above when the tests were assumed to be conditionally independent. (Table 3) is similar to the paired screen positive (PSP) design.^1–4

where
${{\displaystyle n}}_{111}=\left({{\displaystyle n}}_{111}^1+{{\displaystyle n}}_{111}^2\right)$ , ${{\displaystyle n}}_{110}=\left({{\displaystyle n}}_{110}^1+{{\displaystyle z}}_{110}^2\right)$ , ${{\displaystyle n}}_{011}=\left({{\displaystyle n}}_{011}^2+{{\displaystyle z}}_{011}^1\right)$ , ${{\displaystyle n}}_{010}=\left({{\displaystyle z}}_{010}^1+{{\displaystyle z}}_{010}^2\right)$ , ${{\displaystyle n}}_{101}=\left({{\displaystyle n}}_{101}^1+{{\displaystyle n}}_{101}^2\right)$ ,
${{\displaystyle n}}_{100}=\left({{\displaystyle n}}_{100}^1+{{\displaystyle z}}_{100}^2\right)$ ,
${{\displaystyle n}}_{001}=\left({{\displaystyle n}}_{001}^2+{{\displaystyle z}}_{001}^1\right)$ , ${{\displaystyle n}}_{000}=\left({{\displaystyle z}}_{000}^1+{{\displaystyle z}}_{000}^2\right)$ .

When the tests were treated to be dependent of each other the relative TPR is defined as
$\frac{\mathrm{Pr}\left(X_1=1,X_2=1|D=1\right)+\mathrm{Pr}\left(X_1=1,X_2=0|D=1\right)}{\mathrm{Pr}\left(X_1=1,X_2=1|D=1\right)+\mathrm{Pr}\left(X_1=0,X_2=1|D=1\right)}$ while the relative FPR was defined
as $\frac{\mathrm{Pr}\left(X_1=1,X_2=1|D=0\right)+\mathrm{Pr}\left(X_1=1,X_2=0|D=0\right)}{\mathrm{Pr}\left(X_1=1,X_2=1|D=0\right)+\mathrm{Pr}\left(X_1=0,X_2=1|D=0\right)}$ .

Using the data from (Table 3), the two ratios are similar to the ratios from the PSP design. The eight cell counts were the sample values from a multinomial distribution having eight categories. Thus the likelihood function¹² is of the form:
$l(\underset{¯}{\pi })=\frac{n!}{{\displaystyle \prod _{i=1}^7{n}_i!\left(n-{\displaystyle \sum _{i=1}^7{n}_i}\right)!}}{{\displaystyle \prod _{i=1}^7{\pi }_i^{n_i}\left(1-{\displaystyle \sum _{i=1}^7{\pi }_i}\right)}}^{n-{\displaystyle \sum _{i=1}^7{n}_i}};\forall i=1,2,...,8$
where
$\underset{¯}{\pi }=\left({{\displaystyle \pi }}_1,{{\displaystyle \pi }}_2,\mathrm{...},{{\displaystyle \pi }}_7\right)\text{'}$ , $n={\displaystyle \sum _{i=1}^8{{\displaystyle n}}_i}$ , ${{\displaystyle n}}_1={{\displaystyle n}}_{111}^1+{{\displaystyle n}}_{111}^2$ , ${{\displaystyle n}}_2={{\displaystyle n}}_{110}^2+{{\displaystyle z}}_{110}^1$ , ${{\displaystyle n}}_3={{\displaystyle n}}_{011}^2+{{\displaystyle z}}_{011}^1$ , ${{\displaystyle n}}_4={{\displaystyle n}}_{101}^1+{{\displaystyle n}}_{101}^2$ , ${{\displaystyle n}}_5={{\displaystyle n}}_{100}^1+{{\displaystyle z}}_{100}^2$
${{\displaystyle n}}_6={{\displaystyle n}}_{001}^2+{{\displaystyle z}}_{001}^1$ , ${{\displaystyle n}}_7={{\displaystyle n}}_{010}^1+{{\displaystyle z}}_{010}^2$ , ${{\displaystyle n}}_8={{\displaystyle n}}_{000}^1+{{\displaystyle z}}_{000}^2$ .

The Geweke convergence diagnostic9 and the correlograms for the first order autoregressive (AR(1)) processes¹⁰ confirmed that the parameter posterior means and the missing data posterior means converged to stationarity after 4000 iterations. Estimation of the point estimates of the TPR and FPR of the tests and the missing data together with their corresponding credible bounds and variances was then conducted.

(Figures 1 & 2) show the distributions of the TPRs, FPRs and the missing data under the assumption that the tests were conditionally independent while (Figure 3) shows the distributions of the relative TPR and relative FPR. The point estimates and the 95% credible bounds for the TPRs, FPRs, the missing data and the rTPR and rFPR were computed. The point and credible bounds estimates under conditional independence of the tests were as shown in (Table 4).

(Figure 4) shows the distribution of the missing data when the tests were assumed to be dependent. In (Table 4) we have denoted $z_{110}^1=z2,z_{011}^1=z3,z_{100}^2=z5,and{z}_{001}^1=z6$ for convenience. In (Figure 4) z8 represents the data when the outcome of both screening tests is negative and the diagnostic test confirms that the subjects have the disease while z8 represents the data when the outcome of both screening tests is negative and the diagnostic test confirms that the subjects do not have the disease.

The relative TPR and relative FPR of the two screening tests were computed and their posterior distributions were as shown in (Figure 5).

Figure 1 Posterior distributions of the TPRs and the missing data under conditional independence of the tests.

Figure 2 Posterior distributions of the FPRs and the missing data under conditional independence of the tests.

Figure 3 Posterior distributions of the relative TPR and relative FPR of the tests under conditional independence of the tests.

Figure 4 Distributions of the missing data.

Figure 5 Distributions of the relative TPR and relative FPR of the tests under dependence of the test.

The point estimates and the 95% credible bounds of the parameters of the Dirichlet posterior distribution, the measures of accuracy and the missing data were also computed and the results were as shown in (Table 5). The relative TPR was computed as ((pi1+pi2)/(pi1+pi3)) while the relative FPR was computed as ((pi4+pi5)/(pi4+pi6)) which are the probabilities of the appropriate cells in (Table 3).

When the tests were conditionally independent the TPR of the new screening test was estimated to be 0.73 while that of the standard test was estimated to be 0.53. Thus the relative TPR of the new screening test to that of the standard test was 1.38(95% credible bounds: 1.19, 1.59). When the missing data was ignored the disease detection probability (DP) of the new screening test was 0.58 while the DP of the standard screening test was 0.44. This gives a relative TPR of 1.34(95% credible bounds: 1.12, 1.62). The relative TPR obtained when the missing data was ignored was different from that obtained when the missing data was ignored indicating downward bias.

The FPR of the standard screening test was estimated to be 0.02 while that of new screening test was estimated to be 0.025. Thus the relative FPR of the new screening test to that of the standard screening test was 1.25(95% credible bounds: 1.09, 1.43). The false referral probability (FP) of the new screening test when the missing data was ignored was 0.016 while that of the standard screening test was 0.02 giving a relative FPR of 1.25(95% credible bounds: 1.09, 1.43). There seems to be small effect on the relative FPR when the missing data was ignored since the estimate accounting for missing data is similar to that ignoring the missing data. When the tests were treated to be dependent of each other the relative TPR and the relative FPR of the new to the standard screening tests were 1.50(95% credible bounds: 1.35, 1.67) and 1.2595% credible bounds: 1.15, 1.36 respectively. The relative TPR and the relative FPR when the missing data were ignored were 1.37(95% credible bounds: 1.27, 1.47) and 1.25(95% credible bounds: 1.14, 1.36) respectively. Apparently when the missing data were ignored, the relative TPR was downward biased while the relative FPR was not biased. Estimation ignoring the missing data leads to the estimation of the DP and FP because the number of subjects who (would have) screened negative due to one screening test and positive due to the other is not taken into account to estimate the TPRs. This means that the estimates obtained are biased. Since the assumption was that the TPR of the new screening test is 0.75 and that of the standard screening test is 0.5 the relative TPR is 1.50 which is higher than that obtained when the tests were considered to be conditionally independent from each other. This value is, however, similar to that obtained when the tests were considered to be dependent of each other which were precisely found to be 1.496. Similarly, the FPR of the standard screening test was assumed to be 0.02 and that of new screening test was estimated to be 0.025. Thus the relative FPR of the new screening test to the standard screening test was 1.25. We find that the relative FPRs in all the settings (the assumed values, under conditional independence and under dependence of the tests) are similar.

In this study the missing data have been simulated. This means that the TPRs and the FPRs are estimable under the assumptions stated. Therefore, we can no longer estimate the DP and the FP as demonstrated in this study. This is an achievement from this study. As to whether the observed differences in the estimates of relative TPR are true or just due to chance a statistical assessment needs to be conducted. It is recommended that if the tests are known a priori to be dependent or conditionally independent of each other it is good practice to estimate the measures of accuracy in that setting. This is because there is clear evidence from the study that the estimates are affected by the association of the tests. The relative TPR and the relative FPR resulting after the missing data were included in (Tables 2 & 3) can be interpreted as the ratio of two posterior averages. Although these are the “best” posterior estimates of the measures of accuracy, the ratio of these posterior estimators are not necessarily the best posterior estimates. Thus, Bayesian alternatives of obtaining the relative accuracy are warranted.