In medical and social science researches, analysis of observer or inter–rater agreement data often provides a useful means of assessing the reliability of a rating system. If raters agree well enough, then we may prefer the one that is cheaper, less invasive or easier to use. However, if they do not agree well, the best method must be selected using a statistical technique. The objective of this study is to evaluate the agreement between six treatment procedures for deep carious lesions in the posterior teeth, select the best treatment method and find the highly–related factors associated with the success of a treatment. We use Kappa statistic to check the agreement between treatment methods and Chi squared test to test the significant difference between treatment methods. Moreover, we use odds ratio and relative risk values to find the best treatment method. Besides, two model fitting methods, namely, the Lasso regression method and the Probit regression method are used to model the data set related to the best treatment method. Variance Inflation Factor (VIF) values are used to test multicollinearity among predictor variables. Residual Sum of Squares (RSS), Akaike’s information criterion (AIC) and K–fold cross validation values are used to select the best model. Further, accuracies of the two models based on Lasso method and the Probit method are compared. No good agreement between six treatment methods was observed. Therefore, Indirect pulp capping with Calcium Hydroxide (Ca(OH)₂), injectable GIC and light cured posterior composite(LCC) method was selected as the best treatment method using odds ratio and relative risk tests. A model with Exposure Time, Exposure Size and Pulp Conditions for predictor variables was selected for deep carious lesions in the posterior teeth using Lasso method and a model with Exposure Time and Pulp Conditions for predictor variables was selected using Probit method. Moreover, Lasso method has a high model fitting accuracy when compared with the Probit method. We observed that the Indirect pulp capping with Calcium Hydroxide (Ca(OH)₂), injectable Glass Ionomer Cement (GIC) and light cured posterior composite (LCC) as the best treatment method for deep carious lesions in posterior permanent teeth. Further, Exposure time, Exposure size and Pulp condition are observed as highly correlated factors associated with the result of the treatment.

Keywords: agreement, dental caries, kappa statistic, lasso regression, probit regression, relative risk, reversible pulpitis

AIC, akaike’s information criterion; LCC, light cured posterior composite; MSE, mean squared error; RSS, residual sum of squared; VIF, variance inflation factor

Evaluation of agreement between two or more raters, such as a measurement method, an instrument, a medical device, a clinical observer, an assay or a technique or technology is an interesting topic and it is routinely conducted in biomedical fields. The response being measured may be a categorical variable such as success or failure of a medical treatment or it may be a continuous variable such as blood pressure or heart rate. In either case, the interest is in evaluating the agreement in order to check whether the raters can be used interchangeably.^1–5

Dental field is a major biomedical field that requires conducting agreement evaluation for pulp therapy treatments. The goal of pulp therapy is to treat, restore, and save the affected pulp of a tooth. The dental pulp is a soft tissue structure that accompanies the coronal chamber and root canals of primary and permanent teeth and it is composed of loose connective tissues, surrounded by a continuous layer of specialized secretory cells, odontoblasts, which are unique to dental pulp and are responsible for dentine deposition. Blood vessels and nerves enter the pulp through apical foramina and through lateral or accessory canals. Pulps of young and primary permanent teeth, especially those with incomplete apices, have a rich blood supply. Since immature permanent pulp tissue has a remarkable healing capacity, it generally responds well to treatment aimed at pulp preservation.

Functions of the pulp are both formative and defensive. The pulp may be breached or nearly breached because of caries, tooth surface loss, acute traumatic injury or cavity preparation. These pulpal inflammations can be divided into two types as, reversible pulpitis and irreversible pulpitis. In reversible pulpitis, pulp tissue is inflamed and responds actively to an irritant. Transient pain or sensitivity resulting from many stimuli, notably hot, cold, sweet, water and touch are the symptoms of reversible pulpitis. In reversible pulpitis, since a carious lesion has not yet reached the pulp, removal of decay and placement of a filling will return the pulp to its normal, healthy state. However, in irreversible pulpitis, the pulp is irreversibly damaged, and the pulp cannot recover from the effect of insult and damage, such as decay that has reached the pulp of the tooth and introduces bacteria in to the pulp. At this stage, pulp is still alive, but the introduction of bacteria into the pulp will not allow it to heal and necrosis will ensue. Mild dull pain of short duration, pain provoked with hot, cold or other stimuli, presence of pain even after the removal of the stimulus, spontaneous pain particularly at night or referred pain, are the symptoms of the irreversible pulpitis. The chance for recovery of the pulp of a tooth with irreversible pulpitis is zero. Root cannel treatment is the treatment option for a tooth with irreversible pulpitis before it is restored by any type of restorations. The basic principles of root cannel treatment are debridement, sterilization and filling.

The prevalence of dental caries in Sri Lanka is 39.17%, 52.27% and 89.75% respectively for 12, 15 and 35–year–olds. As a result, 12.18% of adults in Sri Lanka have lost at least one tooth in their adolescence, young or middle age due to deep carious lesions in the tooth with the belief that the preservation of the tooth in the arch would be impossible.⁶ Further, advanced treatments adopted for teeth with reversible pulpitis need special instruments, materials and well trained staff making them expensive. Hence the affordability of the advanced restorative treatments for people, who are living in a developing country such as Sri Lanka, is not achievable. Then the exploration of best alternative treatment methods for very expensive treatments would be desirable in a country such as Sri Lanka. Therefore as the aim of this study, introducing a cost effective treatment procedure to preserve vital pulp within an environment with minimal facilities for advanced treatments on deep caries lesions with reversible pulpitis will be very purposive not only for a developing country such as Sri Lanka but also patients worldwide.

The rest of this article is organized as follows. Section 2 presents the proposed methodology to measure the agreement between treatment methods and selection of the best treatment method. Besides, two model fitting techniques are proposed to find the factors associated with the success of a treatment. Section 3 analyzes the data using proposed methodology and Section 4 concludes with a discussion. The model fitting and analysis is carried out by using R statistical software.⁷

This study is aimed at proposing the best treatment procedure for reversible pulpitis in the posterior permanent teeth of adults and finding the highly–related factors associated with the success of the treatment procedure using the following treatment procedures on deep carious lesions in the posterior permanent teeth of adults.

1. Stepwise excavation (Groups I and II): Excavation will be restored with Zinc Oxide and Eugenole (ZnO/E) after application of Calcium Hydroxide liner (Ca(OH)₂) over the deepest layer of remaining carious dentine. After 24–30 days, cavity will be restored with one of the following procedures.
1. Group I: Calcium Hydroxide (Ca(OH)₂), injectable GIC and light cured posterior composite (LCC).
2. Group II: Biodentine base and LCC.
2. Indirect pulp capping (Groups III and IV): Excavation will be excavated with large spoon excavator; the inner zone of caries will be left and excavation will be stopped when a pulp exposure is feared. Cavities will be restored with one of the following procedures.
1. Group III: Calcium Hydroxide (Ca(OH)₂), injectable GIC and light cured posterior composite (LCC).
2. Group IV: Biodentine base and LCC.
3. Direct complete excavation (Groups V and VI): Excavation will be completed during the first visit for all the cavities. Criteria for evaluating the remaining dentine will be identical to those used at the second visit in the stepwise excavation procedure. (1 above)
1. Group V: Calcium Hydroxide (Ca(OH)₂), injectable GIC and light cured posterior composite (LCC).
2. Group VI: Biodentine base and LCC.

For this survey, data were collected from selected patients using a computer generated randomization list, who attended the diagnostic clinic at the Department of Restorative Dentistry, Faculty of Dental Sciences, University of Peradeniya with restorable deep carious lesions in teeth and169 patients each for Groups I, II, III and IV, 170 patients for Group V and 171 patients for Group VI were used. Variables considered for this survey are illustrated in Table 1.

In this study, we used Kappa coefficient to measure agreement and Chi Squared test to measure significant differences, between the treatment methods. Kappa coefficient measures the percentage of data values in the main diagonal of the table and then adjusts these values for agreement that could be expected due to chance alone.^8–12 The value of Kappa coefficient can be estimated as

$k=\frac{p_0-p_e}{1-p_e} (1)$

Where, $p_0=p_{11}+p_{22},p_e=p_{.}{}_1{p}_{1.}+p_{.}{}_2{p}_2,p_{11}=$Success and success probability of Rater A and Rater B,Failure and $p_{22}=$failure probability of Rater A and Rater B, $p_{1.}=$Total of success and success probability of Rater A and Rater B and success and failure probability of Rater A and Rater B, p_.1=Total of Success and Success probability of Rater A and Rater B and Failure and Success probability of Rater A and Rater B, p_2. = Total of Failure and Success probability of Rater A and Rater B and Failure and Failure probability of Rater A and Rater B, p_.2 = Total of Success and Failure probability of Rater A and Rater B and Failure and Failure probability of Rater A and Rater B. Chi squared test is used as a test of independence,^11–13 considering the hypothesis H₀: Two variables are independent vs. H₁: Two variables are dependent. The test statistic is

$x^2={\displaystyle \sum }_{i=1}^r{\displaystyle \sum }_{j=1}^c\frac{{\left(O_{i,j}-E_{i,j}\right)}^2}{E_{i,j}}$

where $E_{i,j}=\frac{p_{i.}{p}_{.j}}{N},$$p_{i.}=$Fraction of observations of type $i,p_{.j}=$Fraction of observations of type $j,N=$Total number of observations, $O_{i,j}=$Number of observations of type i and j. If the p–value$<\alpha$, the null hypothesis can be rejected at (1−α) 100% confidence. Moreover, Odds Ratio (OR) and Relative Risk (RR) values have been used to find the best treatment method.^14,15 OR is referred as a cross product ratio, which considers the ratio of two cross products, ${\pi }_A{}_1{\pi }_B{}_2$and ${\pi }_A{}_2{\pi }_B{}_1$. The OR for two raters A and B is defined as

$$OR=\frac{\raisebox{1ex}{${\pi }_{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$A$}\right.}$}\!\left/ \!\raisebox{-1ex}{${\pi }_{\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$A$}\right.}$}\right.}{\raisebox{1ex}{${\pi }_{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$B$}\right.}$}\!\left/ \!\raisebox{-1ex}{${\pi }_{\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$B$}\right.}$}\right.}=\frac{{\pi }_{A1}{\pi }_{B2}}{{\pi }_{A2}{\pi }_{B1}} (2)$$

Where $\raisebox{1ex}{${\pi }_{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$A$}\right.}$}\!\left/ \!\raisebox{-1ex}{${\pi }_{\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$A$}\right.}$}\right.=\frac{\raisebox{1ex}{${\pi }_{A1}$}\!\left/ \!\raisebox{-1ex}{${\pi }_{A+}$}\right.}{\raisebox{1ex}{${\pi }_{A2}$}\!\left/ \!\raisebox{-1ex}{${\pi }_{A+}$}\right.}$, $\raisebox{1ex}{${\pi }_{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$A$}\right.}$}\!\left/ \!\raisebox{-1ex}{${\pi }_{\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$B$}\right.}$}\right.=\frac{\raisebox{1ex}{${\pi }_{B1}$}\!\left/ \!\raisebox{-1ex}{${\pi }_{B+}$}\right.}{\raisebox{1ex}{${\pi }_{B2}$}\!\left/ \!\raisebox{-1ex}{${\pi }_{B+}$}\right.}$π_A1= Success probability of Rater A, π_A2 = Failure probability of Rater A, π_B1= Success probability of Rater B, π_B2 = Failure probability of Rater B, π_A+ = Total probability of Rater A and π_B+= Total probability of Rater B.

RR, is the ratio of the “Success” probabilities for two raters’ A and B, and it can be defined as

$RR\left(A,B\right)=\frac{{\pi }_{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$A$}\right.}}{{\pi }_{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$B$}\right.}} (3)$

where ${\pi }_{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$A$}\right.}=\frac{{\pi }_{A1}}{{\pi }_{A+}}$and ${\pi }_{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$B$}\right.}=\frac{{\pi }_{B1}}{{\pi }_{B+}}$. Furthermore, the data related to best treatment method is modeled using the Lasso method¹⁶ and the probit regression method.^17–22 Lasso regression is the best–known model fitting method in Shrinkage methods, a method which is fitting a model that contains all the predictors. A Probit model is a type of regression where the dependent variable can only take two values. The purpose of the model is to estimate the probability that an observation with particular characteristics will fall into a specific category. If those estimated probabilities are greater than 0.5 are treated as classifying an observation into a predicted category.²² When comparing Lasso and Ridge regression, Lasso has major advantage over Ridge regression, in that Lasso produces simpler and more interpretable models that involve only a subset of the predictors.¹⁶ Further, VIF is used to detect multicollinearity among predictor variables.²³ The VIF quantifies the severity of multicollinearity in an ordinary least squares regression analysis and its value for each i^th predictor variable can be calculated using equation.⁴

$VIF=\frac{1}{1-R_i{}^2} (4)$

Where $R_i{}^2$is the coefficient of determination of the regression equation with i^th predictor variable on the left–hand side and all other predictor variables on the right hand side. Residual Sum of Squared (RSS), Akaike’s Information Criterion (AIC) and K–fold cross validation methods are used as model selection techniques. K–fold cross validation is one way to improve over the holdout method which the data is removed before training begins and then when training is done, the data that was removed can be used to test the performance of the learned model on new data. The data set is divided into k subsets, and the holdout method is repeated k times. Each time, one of the k subsets is used as the test set and the other (k − 1) subsets are put together to form a training set. Then the average error across all k trials is computed. The advantage of this method is that it matters less, how the data gets divided. Every data point gets to be in a test set exactly once, and gets to be in a training set (k −1) times. The variance of the resulting estimate is reduced when k is increased.²⁴

Measuring the agreement between treatment methods was performed using Kappa coefficient. As there is no agreement between treatment methods, best treatment method was selected using Odds ratio and Relative risk. Furthermore, model fitting for the best treatment method was done using two regression methods, Probit and Lasso methods. Moreover, model selection, model validation and comparing the results of two methods were done in this study.

Measuring the agreement between treatment methods

Agreement between treatment methods was tested using Kappa coefficient statistic. The results of Kappa coefficient test are given in Table 2. Kappa Coefficient for any pair of treatment methods did not exceed the value 0.3 and hence none of the pairs of treatment methods has good agreement. Since there is no good agreement between treatment procedures, we are particularly interested in testing the null hypothesis of independence, H₀ : Group A and Group B are independent and we used Chi–squared test for this purpose.

As the first step, Group I is compared with all other groups and p–values for each pair of groups are calculated. According to results, all p–values are greater than 0.05 and hence H₀ is not rejected at 5% significance level. As a result, all pairs of groups with respect to Group I are independent. Next, Group II is compared with other groups in the previous step. All the p–values of those comparisons are greater than 0.05. Therefore, there is no dependency between Group II and other groups at 5% significance level. As the third and fourth steps Chi squared test is applied for Group IV with other groups and Group V with other groups by following the same procedure in previous two steps. P–values for all these multiple comparisons of groups are shown in Table 3.

Since there is no agreement between treatment groups, we cannot use these six treatment groups interchangeably for dental caries in posterior permanent teeth. Therefore, odds ratio and relative risk tests were used to select the best treatment group. Table 4 shows the odds ratio and relative risk results of selecting the best treatment group with respect to Group I at 5% significance level. All the odds ratio values are less than 1. However, when considering confidence limits, OR (I, III) = 1 is not included in the confidence limits and value 1 is included in between all other confidence limits. Therefore, Group I is better than groups II, IV, V, VI and Group III is better than Group I. Hence, Group III is better than all other groups, according to odds ratio results, and we can select Group III as the best treatment group. Moreover, relative risk is used to select the best treatment group. Similar results can be obtained from the relative risk test and Group III can be selected as the best treatment method.

Finally, using both Odds ratio and Relative risk tests, we selected Group III, Indirect pulp capping with Calcium Hydroxide (Ca(OH)₂), injectable GIC and light cured posterior composite (LCC) as the best treatment method for dental caries in posterior permanent teeth. The data set related with the best treatment method is divided into two sets as the training set and test set. The training set is used for model fitting and test set is used for model validation purposes. Model fitting was done using two methods, probit regression and Lasso regression and the fitted models were validated using the test set.

Model fitting under the probit regression model

Multicollinearity between predictor variables was tested using VIF values and no multicollinearity between predictor variables was observed. Therefore, the model fitting was done using the probit regression and the resulting full model is as follows.

Result = −3.8909 + 0.0829 ∗ District+0.4113 ∗ Ethnic group+ 0.1973 ∗ Occupation+0.0163 ∗ Gender+ 1.0622 ∗ Pulp horn abscess+2.1629 ∗ Exposure time+1.3089 ∗Exposure size+1.7922 ∗ Pulp condition+0.0632 ∗ Brushing frequency+0.4843 ∗ Pain3 + 0.0178 ∗ Age

RSS values and AIC values are used in model selection. Moreover, RSS values were used to compare models with the same number of predictors and AIC values are used to compare models with different predictor variables. Table 5 illustrates the RSS values for fitted models considering different number of predictor variables. In one variable model, the model with Exposure time variable is the best one variable model which minimizes the RSS. Since Exposure time is selected as the best one variable model, all the other variables with Exposure time are compared to find the best two variable model. According to RSS results, the model with exposure time and pulp condition predictor variables is selected as the best two variable model. Moreover, 3, 4, 5, 6, 7, 8, 9 and 10 predictor variable models were fitted using the same procedure as in the previous step and the best model was selected using the minimum RSS value. The best model is chosen using AIC values of selected models from Table 6 and corresponding AIC values are illustrated in Table 6.

In the Probit regression, a model with Exposure time and Pulp condition as the predictor variables is selected as the best model which minimizes the AIC value, and the resulting model is as follows.

Result = −4.6386 + 2.5576 ∗ Exposure time+ 2.3193 ∗ Pulp condition

Model validation was done by comparing actual values with the predicted values for the best model and results are shown in Figure 1. Out of ten data points seven predicted values are the same as actual values and only three data points, i.e., 4^th, 7^th and 10^th are different than the actual values. It suggests that the model fits well with the data set related to the best treatment method.

Figure 1 Distribution of Actual values and Predicted values of the Probit regression method.

Model fitting under the Lasso regression method

Model fitting was done by using the result factor as the dependent variable and all other factors as predictor variables. Furthermore, 10–fold cross validation was used as model selection method. Different lambda values corresponding to models with different number of predictor variables are considered and Mean Squared Error (MSE) for each lambda value is calculated using 10–fold cross validation method. Corresponding lambda value for minimum MSE is obtained as 0.0248 and result of selected model is as follows.

Result = 0.0058 + 0.3909 ∗ Exposure time + (−0.0524) ∗ Exposure size + 0.0665 ∗ Pulp condition

Only 3 exposure variables, namely Exposure time, Exposure size and Pulp condition are highly correlated to respond variable with minimum MSE. Model validation results using test data set are shown in Figure 2. Out of ten data points, nine predicted values are almost the same as actual values and only one data point, (10^th point) is different from the actual value. It suggests that the model fits well with the data set related to the best treatment method. When comparing the model validation results of the two selected models, the accuracy of Lasso method is higher than that that of the Probit method.

Figure 2 Distribution of Actual values and Predicted values of the Lasso regression method.

In this study, we introduced Kappa statistic to measure agreement between treatment methods for deep carious lesions in posterior permanent teeth. Due to the poor agreement between treatment methods, we cannot use the cheaper, less invasive or easier to use treatment method as the best treatment method. As a solution, the best treatment method was selected using odds ratio and relative risk statistics. Both odds ratio and relative risk confirmed that the third treatment method, ‘Indirect pulp capping with Calcium Hydroxide (Ca(OH)₂), injectable GIC and light cured posterior composite (LCC)‘ as the best treatment method. Both Probit regression and Lasso regression models were fitted for the data related with the best treatment method. Then model validation is performed comparing 10 actual values and corresponding predictor values. Lasso method has the highest prediction accuracy compared to the Probit model. In that model, three predictor variables, Exposure time, Exposure size and Pulp condition are found to be significant.

Our approach is flexible in that it can accommodate balanced or unbalanced data designs, and it works with any measure of agreement. Therefore, this proposed methodology can be used to evaluate the agreement between any treatment procedures on deep caries lesions in the posterior permanent teeth. Besides the treatment method, ‘Indirect pulp capping with Calcium Hydroxide (Ca(OH)₂), injectable GIC and light cured posterior composite(LCC)‘ can be used as the best treatment procedure for deep caries lesions in the posterior permanent teeth with reversible pulpitis and Exposure time, Exposure size of the dental caries and Pulp condition of the tooth can be consider as highly correlated factors associated with the result of the treatment.

The authors are grateful to the Faculty of Dental Sciences, University of Peradeniya for providing the data.

Author declares that there is no conflict of interest.

1. Clegg LX, Potosky AL, Harlan LC, et al. Comparison of self–reported initial treatment with medical records: Results from the prostate cancer outcomes study. American Journal of Epidemiology. 2001;154(6):582–587.
2. Aravind KR, Nawarathna LS. A statistical method for assessing agreement between two methods of heteroscedastic clinical measurements using the copula method. Journal of Medical Statistics and Informatics. 2017;5(3).
3. Nawarathna LS, Choudhary PK. A heteroscedastic measurement error model for method comparison data with replicate measurements. Stat Med. 2015;34(7):1242–1258.
4. Dassanayake ID, Nawarathna LS. Assessing inter–rater agreement between multiple medical instruments with heteroscedastic measurements. Journal of Medical Statistics and Informatics. 2017;5(1).
5. Nawarathna LS. Heteroscedastic models for method comparison data, PhD Thesis. USA: The University of Texas at Dallas; 2014.
6. Ministry of Healthcare and Nutrition. National Oral Health Survey Report 2002–2003. Sri Lanka: Ministry of Healthcare and Nutrition, Colombo; 2009.
7. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing; 2008.
8. Viera AJ, Garrett MJ. Understanding inter observer agreement: The kappa statistic. Family Medicine. 2005;37(5):360–363.
9. Myers L, Hsueh YH. Using nonparametric bootstrapping to assess kappa statistics. Proceedings of the Annual Meeting of the American Statistical Association, 2001.
10. Grill E, Mansmann U, Cieza A, et al. Assessing observer agreement when describing and classifying functioning with the international classification of functioning, disability and health. Journal of Rehabilitation Medicine. 2007;39(1):71–76.
11. Falcao MC, Leone CR, Ramos JLA. Is glycosuria a reliable indicator of adequacy of glucose infusion rate in preterm infants?. Sao Paulo Medical Journal. 1999;117(1):19–24.
12. Kleiner DE, Brunt EM, Van Natta M, et al. Design and validation of a histological scoring system for nonalcoholic fatty liver disease. Hepatology. 2005;41(6):1313–1321.
13. Frank TM, Timothy H, Christina AC. The chi–square test: Often used and more often misinterpreted. American Journal of Evaluation.2012;33(3):448–458.
14. Simon SD. Understanding the odds ratio and relative risk. Journal of Andrology. 2001;22(4):533–536.
15. Schechtman E. Odds ratio, relative risk, absolute risk reduction, and the number needed to treat– which of these should we use? Value Health. 2002;5(5):431–436.
16. Tibshirani R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological). 1996;58(1):267–288.
17. Arendt JN, Larsen HA. Probit models with Binary endogenous regressors. Discussion Papers on Business and Economics No. 4/2006; 2006.
18. Arulampalam W, Stewart MB. Simplified implementation of the heckman estimator of the dynamic probit model and a comparison with alternative estimators. Oxford bulletin of economics and statistics. 2009;71(5):659–681.
19. Razzaghi M. The probit link functions in generalized linear models for data mining applications. Journal of Modern Applied Statistical Methods. 2013;12(1):164–169.
20. Tabatabai MA, Li H, Eby WM, et.al. Robust logistic and probit methods for binary and multinomial regression. Journal of Biometrics & Biostatistics. 2014;5(4).
21. Park HM. Regression models for binary dependent variables using stata, sas, r, limdep, and spss. Working Paper. USA: The University Information Technology Services (UITS) Center for Statistical and Mathematical Computing, Indiana University; 2009.
22. Greene WH. Econometric Analysis. New Jersey: Prentice Hall, Upper Saddle River; 2002.
23. Vatcheva KP, Lee M, McCormick JB, et al. Multicollinearity in regression analyses conducted in epidemiologic studies. Epidemiology. 2016;6(2).
24. Yadav S, Shukla S. Analysis of k–fold cross–validation over hold–out validation on colossal datasets for quality classification. IEEE Xplore. 2016;78–83.