This paper provides a detailed account of the steps involved in the development and diagnostics of an OLS regression model (hedonic) using the Irish Central Statistics Offices’ Consumer Price Index for New Cars as an example. The areas of Collinearity: effects on parameter estimates, effects on inference, effects on prediction, what to do about collinearity and Model Diagnostics: residuals’ “standardized residuals, residual plots, outliers, studentized residuals (t-residuals), influential observations, leverage, cooks’ distance and transformations are discussed in detail with examples.

CPI, consumer price indices; HICP, harmonized index of consumer price index; ECB, european central bank’s; OLS, ordinary least squares

Hedonic regression
Hedonic regression is a method used to determine the value of a good or service by breaking it down into its component parts. The value of each component is then determined separately through regression analysis. See Lancaster K,¹ Griliches Z^2,3 and Diewert^4-6 for detail discussions on the development and application of hedonic prices. In addition, Moulton⁷ provides a information on the importance and expanded use of Hedonic Methods.

The term “hedonic methods” refers to the use in economic measurement of a “hedonic function,” h ( ),
$p_i=h\left(c_i\right)$
Where p is the price of a variety (or model) iof a good and ci is a vector of characteristics associated with the variety. One of the more widely-recognised examples of hedonic regression is the Consumer Price Index, which examines changes to the value of a basket of goods over time and the hedonic function is used to adjust for differences in characteristics between varieties of the good in calculating its price index. The hedonic function is normally estimated by regression analysis.

Consumer price indices
All the goods and services that consumers purchase have a price and that price may vary over time, and these changes are a measure of inflation. Consumer Price Indices (CPI) are designed to measure such changes. A useful way to understand the nature of these indices is to imagine a very large shopping basket comprising of a set, or basket of fixed composition, quantity and as far as possible quality of goods and services bought by a typical private household.

The European CPI is titled the Harmonized Index of Consumer Price Index (HICP - Council regulation (EC) No 2494/95).⁸ The HICP is an important contributor the European Central Bank’s (ECB) monetary policy and is compiled monthly using a Laspeyres index formula, see Allen RGD.⁹ This HICP it expresses the current cost of a fixed market basket of consumer goods and services as a percentage of the cost of the same identical basket at a base (normally mid- December of the year previous to the reference date).

The HICP is defined as

Where $w_{j,b}$ is the weight assigned to item j determined by the base period consumer expenditure shares and $p_{j,t}$ refers to the price of item j in period t.

Whenever a comparison between current and base period products is not possible, or when the current period basket reflects new market developments (i.e. quality improvements). This then leads to the well-known problem of how to measure price developments when the quality of the underlying goods and services is changing over time.

It is important the appropriate methods are employed to take account of quality change in the HICP as this index need to be a credible and transparent. In this context, this paper provides a detailed account of the steps involved in the development of an OLS regression model (hedonic) and associated diagnostics using the Irish Central Statistics Office’s Consumer Price Index for New Cars as an example.

This paper is designed to be a reference document for those challenged with using regression analysis to determine the value of each characteristic of a goods or service. It is divided into six sections, each dealing with a particular aspect of model development. Section one deals with summarising relationships, section two discusses fitting curves – regression analysis, while section three presents the hedonic regression model. In section four, a test for Collinearity within the model is undertaken and a strategy for dealing with Collinearity is presented. In section five, the model diagnostics: residuals – standardized residuals, residual plots, outliers, studentized residuals (t-residuals), influential observations, leverage, cook’s distance, and transformations are undertaken and discussed in detail.

Finally, section six provides an illustrative example of how hedonic based quality adjustment can be applied in a situation when the price an individual car model was available in a January of a particular year, but was not available in the February of the same year. It is shown that without the application of the quality adjustment method the New Car Price Index would provide an incorrect measurement of the price changes of new cars over the period, which in turn would miss-inform the ECB’s monetary policy.

Statistical analysis is used to document relationships among variables. Relationships that yield dependable predications can be exploited commercially or used to eliminate waste from processes. A marketing study done to learn the impact of price changes on coffee purchases is a commercial use. A study to document the relationship between moisture content of raw material and yield of usable final product in a manufacturing plant can result from finding acceptable limits on moisture content and working with suppliers to provide raw material reliably within these litmus. Such efforts can improve the efficiency of manufacturing process.

We strive to formulate statistical problems in terms of comparisons. For example, the marketing study in the preceding paragraph was conducted by measuring coffee purchases when prices were set a several different levels over a period of time. Similarly, the raw material study was conducted by comparing the yields from batches of raw materials that exhibited moisture content.

Scatter plots display statistical relationship relationships between two metric variables (e.g. price and cc) in this section the details of scatter plotting are presented using the data in (Table 1). The data were collected for used in compiling the New Car Index in the Irish CPI Scatter plots are used to try to discover a tendency for plotted variables to be related in a simple way. Thus the more the scatter plot reminds us of a mathematical curve, the more closely related we infer the variables are. In the scatter plot a direct relationship between the two variables is inferred.

The above graph shows a relatively a linear relationship between the two metric variables (price and cc). However to investigate further the relationship between these two variables we can apply the universal method of logarithmic transformation to the cc variable. This transformation discounts larger values of cc and leaves smaller and intermediate ones intact and has the effect ofincreasing the linearity of relationship (Figure 1).

The descriptive statistic most widely used to summarise a relationship between metric variables is a measure of the degree of linearity in the relationship. It is called product-moment correlation coefficient denoted by the symbol r and it is defined by

Where $\overline{x}$  and $s_x$ are the mean and standard deviation of the x variable and  and are the mean and standard deviation of the y variable.

1. Its numerical value lies between – 1 and + 1 , inclusive
2. If r = 1, then the scatter plot shows that the data lie exactly on a straight line with a positive slope; if r = -1, then the scatter plot shows that the data lie on a straight line with a negative slope.
3. An r = 0 indicates that there is no linear component in the relationship between the two variables.

These properties emphasise the role of r as a measure of linearity. Essentially, the more the scatter plot looks like a positively sloping straight line, the closer r is to +1, and the more the scatter plot looks like a negatively sloping straight line, the closer r is to -1.

Using the equation above, r is estimated for the relationship shown in (Figure 2) to be 0.92 indicating a strong linear relationship between the price of a new car and cylinder capacity. For the relationship shown in (Figure 1), r is estimated to be 0.93, indicating that using the logarithmictransformation does indeed increase the linearity of relationship between the two metric variables. The LINEST function in EXCEL was used to estimate r in both of the above relationships.

In the sections above we showed how to summarise the relationships between metric variables using correlations. Although correlations are valuable tools, they are not powerful enough to handle many complex problems in practice (Table 2). Correlations have two major limitations:

1. They summarise only linearity in relationships.
2. They do not yield models for how one variable influences another. The tool of regression analysis overcomes these limitations by using mathematical curves to summarise relationships among several variables. A regression model consists of the mathematical curve summarising the relationship together with measures of variation from that curve. Because any type of curve can be used, relationships can be nonlinear. Regression analysis also easily accommodates transformations of variables and categorical variables, and it provides a host of diagnostic statistics that help assess the utility of variables and transformations and the impact of such features as outliers and missing data.

Figure 1 Scatter plot of price (£) versus log (cc).

Figure 2 Scatter plot of price (£) versus cylinder capacity (cc).

A model describes how a process works. For scientific purposes, the most useful models are statements of the form “if certain conditions apply, then certain consequences follow”. The simplest of such statements assert that the list of conditions result in a single consequence without fails. For example, we learn in physics that if an object falls toward earth, then it accelerates at about 981 centimetres per second.

A less simple statement is one that assesses a tendency: “Loss in competition tends to arouse anger.” While admitting the existence of exceptions, this statement is intended to be universal, that is anger is the expected to loss in competition.

To be useful in documenting the behaviour of processes, models must allow for a range of consequences or outcomes. They must also be able to describe a range of conditions, fixed levels of predictor variables (x), for it is impossible to hold conditions constant in practice. When a model describes the range of consequences corresponding to a fixed set of conditions, it describes local behaviour. A summary of the local behaviours for a range of conditions is called global behaviour. Models are most useful if they describe global behaviour over a range of conditions encountered in practice. When they do, they allow us to make predictions about the consequences corresponding to conditions that have not actually been observed. In such cases, the models help us reason about processes despite being unable to observe them in complete detail.

To illustrate this topic refer back to the sample of cars in (Table 1) and (Figure 2) (the outcome of the scatter plot of price vs. cc) above. Our eyes detect a marked linear rend in the plot. Before reading further, use a straight-edge to draw a line through the points that appear to you to be the best description of the trend. Roughly estimate the co-ordinates of two points (not necessarily points corresponding to data points) that lie on the line. From these two estimated points, estimate the slope and y intercept of the line as follows:

Let $\left(x_1,y_1\right)$ and $\left(x_2,y_2\right)$ denote two points, with $\left(x_1\ne y_1\right)$ on a line whose equation is y = a + bx. Then the slope of the line is $b=\frac{\text{Difference in }y\text{ coordinates}}{\text{Difference in }x\text{ coordinates}}=\frac{y_1-y_2}{x_1-x_2}$ And the y intercept is $a=\frac{x_1{y}_2-x_2{y}_1}{x_1-x_2}$ Next, describe the manner in which the data points deviate from your estimated line. Finally, suppose you are told that car has cylinder capacity of 1600cc, and you are asked to use your model to predict the price of the car. Give your best guess at the range of plausible market values. If you do all these things, you will have performed the essential operations of a linear regression analysis of the data.

If you followed the suggestions in the previous paragraph, you were probably pleased to find that regression analysis is really quite simple. On the other hand, you may not be pleased with the prospect of analysing many large data sets “by eye” or trying to determine a complex model that relates price cylinder capacity, horse power, weight and length simultaneously. To do any but the most rudimentary, the help of a computer is needed.

Statistical software does regression calculations quickly, reliably, and efficiently. In practice one never has to do more than enter data, manipulate data, issue command that ask for calculations and graphs, and interpret output. Consequently, computational formulas are not presented here.

The most widely available routines for regression computations use least squares methods. In this section the ideals behind ordinary least squares (OLS) are explained. Ordinary least squares fit a curve to data pairs $\left(x_1,y_1\right),\left(x_2,y_2\right),\cdots ,\left(x_n,y_n\right)$  by minimising the sum of the squared vertical distances between the y values and the curve. Ordinary least squares are fundamental building block of most other fitting methods.

Fitting a line by ordinary least squares

When a computer program (in this case the LINEST function in EXCEL) is asked to fit a straight-line model to the data in (Figure 2) using the method of ordinary least squares, the following equation is obtained

The symbol y stands for a value of Price (response variable), and the symbol ^ over the y indicates that the model gives only an estimated value. The symbol x (predictor variable) stands for a value of cylinder capacity. This result can be put into the representation

Observation = Fit and Residual

Where y is the observation, $\hat{y}$ is the fit (ted) value and $y-\hat{y}$ is the residual. Consider car No. 1 in (Table 1) that has a cylinder capacity x = 1300. The corresponding observed price is y = 13,390. The fitted value given by the ordinary least square line is   $\hat{y}$ = 307 + 9.11(1300)

= 307 + 11,843

= 12,150

The vertical distance between the actual price and the fitted price is = 13,390 – 12,150 = 1,240, which is the residual. The positive sign indicated the actual price is above the fitted line. If the sign was negative it means the actual price is below the fitted line.

 Figure 3 below shows a scatter plot of the data with ordinary least squares line fitted through the points. This plot confirms that the computer can be trained to do the job of fitting a line. The OLS line was fitted using the linear trend line option in WORD for a chart. Another output from the statistical software is a measure of variation: s = 1028. This measure of variation is the standard deviation of the vertical differences between the data points and the fitted line, that is, the standard deviation of the residuals (Figure 4).

An interesting characteristic of the method of least squares is: for any data set, the residuals from fitting a straight line by the method of OLS sum to zero (assuming the model includes a y – intercept term). Also because the mean of the OLS residuals is zero, their standard deviation is the square root of the sum of their squares divided by the degrees of freedom. When fitting a straight line by OLS, the number of degrees of freedom is two less than the number of cases, denoted by n-2 because

The residuals sum to zero

The sum of the products of the fitted values and residuals, case by case is zero.

Figure 3 Scatter plot of price (£) versus cylinder capacity and OLS line.

Figure 4 Scatter plot of residual versus fitted from new car data.

Analysis of residuals

Two fundamental tests are applied to residuals from a regression analysis: a test for normality and a scatter plot of residuals versus fitted values. The first test can be performed by checking the percentage of residuals within in one, two and three standard deviations of their mean, which is zero. The second test gives visual cues of model inadequacy.

What do we look for in a plot of residuals versus fitted values? We look for a plot that suggests random scatter. As we noted above, the residuals satisfy the constraint  

where the summation is done over all cases in the data set. The constraint, in turn, implies that the product-moment correlation coefficient between the residuals and the fitted values is zero. If the scatter plot is somehow not consistent with this fact because it exhibits a trend or other peculiar behaviour, then we have evidence that the model has not adequately captured the relationship between x and y. This is the primary purpose of residual analysis; to seek evidence of inadequacy.

The scatter plot in (Figure 3) suggests random scatter and the regression equation above is, therefore, consistent with the above constraint.

The inclusion of additional metric variables

So far the variable used to account for the variation in the price of a new car is a measure of a physical characteristic which is more or less permanent, though cylinder capacity can change with improvements or deterioration’s. This variable does not link up directly with economic factors in the market place, however. Regardless of the cylinder capacity of the car the price of a new is also related to the horse power, weight, length and number of doors.

Defining the characteristics of a new car as follows 

$x_1$ $x_2$  = cylinder capacity (cc)

 $x_2$ = number of doors (d)

$x_3$  = horse power (ps)

We propose to fit a model of the form  

            $\hat{y}=b_0+b_1{x}_1+b_2{x}_2+b_3{x}_3$            (3.1)

When applying regression models (Hedonic regression) to a car index it is usual to fit a semi logarithmic form as it has been proven to fit the data best. That is  

                    ${\log }_e\hat{y}=b_0+b_1{x}_1+b_2{x}_2+b_3{x}_3$              (3.2) 

This model relates the logarithm of the price of a new car to absolute values of the characteristics. Natural logarithms are used, because in such a model an b coefficient, if multiplied by a hundred measures, will provide an estimate of the percentage increase in price due to a one unit change in the particular characteristic or “quality”, holding the level of the other characteristics constant. 

Using the LINEST function in EXCEL (or PROC REG in SAS) the following estimates for the b coefficients are obtained when the above model is applied to the data in (Table 1).

     ${\log }_e\hat{y}=8.43+0.000436x_1+0.033094x_2+0.003605x_3$                 (3.3)

 The interpretation of the above equation is as follows. Keeping the level of other characteristics constant 

1. A one unit change in cylinder capacity gives a 0.0436% increase in price
2. A one unit change in the number of doors gives a 3.3094% increase in price
3. A one unit change in brake horse power gives a 0.3605% increase in price. 

The inclusion of categorical variables

The next step is to incorporate power steering, ABS system and air bags into the model. The variables are categorical variables: their numeric values 1, and 0 stand for the inclusion or exclusion of these features in a car.

The semi logarithmic form of the model is now.

         ${\log }_e\hat{y}=b_0+b_1{x}_1+b_2{x}_2+b_3{x}_3+b_4{x}_4+b_5{x}_5+b_6{x}_6+b_7{x}_7+b_8{x}_8$              (3.4)

Where

$x_4$  = weight        

$x_5$ = length           

$x_6$  = power steering (pst)

 $x_7$ = ABS system (abs)

 $x_8$ = air bags (ab)

Using the LINEST function in EXCEL (or PROC REG in SAS) the following estimates for the b coefficients are obtained when the above model is applied to the relevant data in (Table 1).

Equation (3.5)

${\log }_e\hat{y}=9.37+0.000089x_1+0.0197x_2+0.0023x_3+0.0015x_4-0.0054x_5+0.0649x_6+0.113x_7-0.0075x_8$

The regression coefficients obtained from Equation (3.5) are interpreted as follows. Keeping the level of other characteristics constant 

1. A one unit change in cylinder capacity gives a 0.0089% increase in price.
2. A one unit change in the number of doors gives a 1.97% increase in price.
3. A one unit change in brake horse power gives a 0.23 % increase in price.
4. A one unit change in weight (kg) gives a 0.15% increase in price.
5. A one unit change in the length (cm) gives a 0.54% decrease in price
6. The inclusion of power steering gives a 6.49% increase in price.
7. The inclusion of an ABS system gives a 11.26% increase in price.
8. The inclusion of air bags gives a 0.75% decrease in price 

In Section 4 below it is shown that there is strong Collinearity between weight and length in the above regression model and, therefore, length will be omitted from the model.

${\log }_e\hat{y}=b_0+b_1{x}_1+b_2{x}_2+b_3{x}_3+b_4{x}_4+b_5{x}_5+b_6{x}_6+b_7{x}_7$ (3.6) 

Where

 $x_4$ = weight        

$x_5$  = power steering (pst)                               

$x_6$  = ABS system (abs)

 $x_7$ = air bags (ab)

 Using the LINEST function in EXCEL (or PROC REG in SAS) the following estimates for the b coefficients are obtained when the above model is applied to the relevant data in (Table 1).    ${\log }_e\hat{y}=8.43+0.00008x_1+0.015x_2+0.002x_3+0.0005x_4+0.107x_5+0.079x_6-0.034x_7$ (3.7) 

The interpretation of the above equation is as follows. Keeping the level of other characteristics constant

1. A one unit change in cylinder capacity gives a 0.008% increase in price.
2. A one unit change in the number of doors gives a 1.5% increase in price.
3. A one unit change in brake horse power gives a 0.2% increase in price.
4. A one unit change in weight (kg) gives a 0.05% increase in price.
5. The inclusion of power steering gives a 10.7% increase in price.
6. The inclusion of an ABS system gives a 7.9% increase in price.
7. The inclusion of an airbag gives a 3.4.% decrease in price

Section 4 below shows that Collinearity in not an issue in the regression model described in Equation (3.7).

The output of the regression results for Equation (3.7) is displayed below. All the regression coefficients are significantly different from zero with t statistics (t ratios) greater than 0.8. An R-square of 96% indicates that almost all of the variation in the price of new cars is explained by the selected predictors.

As part of any statistical analysis is to stand back and criticise the regression model and its assumptions. This phase is calledmodel diagnosis. If under close scrutiny the assumptions seem to be approximately satisfied and the model can be used to predict and understand the relationship between response and the response and the predictors.

In Section 5 below the regression model, as described in Equation (3.7), is proven to be adequate for predicting and to understanding the relationship between response and predictors for the new car data described in (Table 1).

Classic definition of hedonic regression
As we can see above, the hedonic hypothesis assumes that a commodity (e.g. a new car) can be viewed as a bundle of characteristics or attributes (e.g. cc, horse power, weight, etc.) for which implicit prices can be derived from prices of different versions of the same commodity containing different levels of specific characteristics.

The ability to desegregate a commodity and price its components facilities the construction of price indices and the measurement of price change across versions of the same commodity. A number of issues arise when trying to accomplish this.

1. What are the relevant characteristics of a commodity bundle?
2. How are the implicit (implied) prices to be estimated from the available data?
3. How are the resulting estimates to be used to construct price or quality indices for a particular commodity?
4. What meaning, if any, is to be given to the resulting constructs?
5. What do such indices measure?
6. Under what conditions do they measure it unambiguously?

Much criticism of the hedonic approach has focused on the last two questions, pointing out the restrictive nature of the assumptions required to establish the “existence” and meaning of such indices. However, what the hedonic approach attempts to do is provide a tool for estimating “missing” prices, prices of particular bundles not observed in the base or later periods. It does not pretend to dispose of the question of whether various observed differences are demand or supply determined, how the observed variety of model in the market is generated, and whether the resulting indices have an unambiguous interpretation of their purpose.

Suppose that in the car data (Table 1) the car weight in pounds in addition to the car weight in kilograms is used as a predictor variable. Let  denote the weight in kilograms and let  denote the weight in pounds. Now since one kilogram is the same as 2.2046 pounds,

with $\gamma ={\beta }_1+2.2046{\beta }_2$ . Here $\gamma$  represents the “true” regression coefficient associated with the predictor weigh when measured in pounds. Regardless of the value of $\gamma$ , there are infinitely many different values for ${\beta }_1$ and ${\beta }_2$  that produce the same value for $\gamma$ . If both  and  are included in the model, then ${\beta }_1$  and ${\beta }_2$  cannot be uniquely defined and cannot be estimated from the data.

The same difficulty occurs if there is a linear relationship among any of the predictor variables. If some set of predictor variables $x_1,x_2\dots ,x_m$  and some set of constants $c_1,c_2\dots ,c_{m+1}$  not all zero

            $c_1{x}_1+c_2{x}_2+\dots +c_m{x}_m=c_{m+1}$            (4.1)

For all values of $x_1,x_2\dots ,x_m$  in the data set, then the predictors $x_1,x_2\dots ,x_m$ are said to be collinear. Exact collinearity rarely occurs with actual data, but approximate collinearity occurs when predictors are nearly linearly related. As discussed later, approximate collinearity also causes substantial difficulties in regression analysis. Variables are said to be collinear even if Equation (4.1) holds only approximately. Setting aside for the moment the assessment of the effects of collinearity, how is it detected?

The search for collinearity between predictor variables are assessed by calculating the correlation coefficients between all pairs of predictor variables and displaying them in a table.

The above table of correlations are only between pairs of predictors and cannot assess more complicated (near) linear relationships among several predictors and expressed in Equation (4.1). To do so the multiple coefficient of determination, $R_j^2$ , obtained from regressing the jth predictor variable on all the other predictor variables is calculated. That is, $x_j$  is temporarily treated as the response in this regression. The closer this $R_j^2$  is to 1 (or 100%), the more serious the collinearity problem is with respect to the jth predictor.

Effects on parameter estimates.
The effect of collinearity on the estimates of regression coefficients may be best seen from the expression giving the standard errors of those coefficients. Standard errors give a measure of expected variability for coefficients – the smaller the standard error the better the coefficient tends to be estimated. It may be shown that the standard error of the jth coefficient, $b_j$ , is given by

              $se\left(b_j\right)=s\sqrt{\frac{1}{1-R_j^2}\cdot \frac{1}{{\displaystyle \sum _{i=1}^n{\left(x_{ij}-{\overline{x}}_j\right)}^2}}}$             (4.2)

Where, as before, $R_j^2$  is the $R^2$  value obtained from regressing the jth predictor variable on all other predictors? Equation (4.2) shows that, with respect to collinearity, the standard error will be smallest when $R_j^2$ zero, that is, is the jth predictor is not linearly related to the other predictors. Conversely, if $R_j^2$  is near 1, then the standard error of $b_j$  is large and the estimate is much more likely to be far from the true value of ${\beta }_j$ .
The quantity

                   $VI{F}_j=\frac{1}{1-R_j^2}$             (4.3)

Is called the variance inflation factor (VIF). The large the value of VIF for a predictor $x_j$ , the more severe the collinearity problem. As a guideline, many authors recommend that a VIF greater than 10 suggests a collinearity difficulty worthy of further study. This is equivalent to flagging predictors with $R_j^2$  greater than 90%.

Table 3 below presents the results of the collinearity diagnostics for the regression model outlined in Equation 3.5 (using PROC REG in SAS). From Table 4 and Table 5 above it is obvious that there is a strong liner relationship between the predictor variables w and l in the regression model in Equation (3.5) and they are collinear. To overcome this collinearity problem the predictor variable l (length) will be omitted from the regression model.

Effects on inference
If collinearity affects parameter estimates and their standard error then it follows that t- ratios will also be affected.

Effects on prediction
The effect of collinearity on prediction depends on the particular values specified for the predictors. If the relationship among the predictors used in fitting the model is preserved in the predictor values used for prediction, then the predictions will be little affected by collinearity. ON the other hand, if the specified predictor values are contrary to the observed relationships among the predictors in the model, then the predictions will be poor.

What to do about collinearity
The best defence against the problems associated with collinear predictors is to keep the models as simple as possible. Variables that add little to the usefulness of a regression model should be deleted from the model. When collinearity is detected among variables, none of which can reasonably be deleted from a regression model, avoid extrapolation and beware of inference on individual regression coefficients.

Table 6 below presents the results of the collinearity diagnostics for the regression model outlined in Equation 3.7 (using PROC REG in SAS).

Note that the predictor variable w (weight)does nothave a VIP value sufficiently greater than 10to warrant exclusion from the model. Table 4 and Table 6 above indicate that the regression model as described in Equation (3.7) does not have a problem with collinearity among the variables.

All the regression theory and methods presented above rely to a certain extent on the standard regression assumptions. In particular it was assumed that the data were generated by a process that could be modelled according to

$y_i={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_k{x}_{ik}+e_i$ For i = 1,2, ….,n (5.1)

Where the error terms $e_1,e_2,\dots ,e_n$ are independent of one another and are each normally distributed with mean 0 and common standard deviation. But in any practical situation, assumptions are always in doubt and can only hold approximately at best. The second part of any statistical analysis is to stand back and criticize the model and its assumptions. This phase is frequently called model diagnosis. If under close scrutiny the assumptions seem to be approximately satisfied, then the model can be used to predict and to understand the relationship between response and predictors. Otherwise, ways to improve the model are sought, once more checking the assumptions of the new model. This process is continued until either a satisfactory model is found or it is determined that none of the models are completely satisfactory. Ideally, the adequacy of the model is assessed by checking it with a new set of data. However, that is a rare luxury; most often diagnostics based on the original set must suffice. The study of diagnostics begins with the important topic of residuals (Model 1).

Residuals – standardised residuals

Most of the regression assumptions apply to the error terms. However the error terms cannot be obtained and the assessment of the errors must be based on the residuals obtained as the actual value minus the fitted value that the model predicts with all unknown parameters estimated (Model 2) for the data. Recall that in symbols the ith residual is

To analyse residuals (or any other diagnostic statistic), their behaviour when the model assumption do hold and, if possible, when at least some of the assumptions do not hold must be understood. If the regression assumptions all hold, it may be shown that the residuals have normal distributions with 0 means. It may also be shown that the distribution of the ith residual has the standard deviation$\sigma \sqrt{1-h_{ii}}$ , where $h_{ii}$  is the ith diagonal element of the “hat matrix” determined by the values of the set of predictor variables. (See Appendix I), but the particular formula given there is not needed here. In the simple case of a single predictor model it may be shown that

$h_{ii}=\frac{1}{n}+\frac{{\left(x_i-\overline{x}\right)}^2}{{\displaystyle \sum _{j=1}^n{\left(x_j-\overline{x}\right)}^2}}$ (5.3)

Note in particular that the standard deviation of the distribution of the ith residual is not $\sigma$ , the standard deviation of the distribution of the ith error term $e_i$  . It may be shown that, in general,

$\frac{1}{n}\le h_{ii}\le 1$ (5.4)

So that

$0\le \sigma \sqrt{1-h_{ii}}\le \sigma \sqrt{1-\frac{1}{n}}\le \sigma$  is at its minimum value, 1/n, when the predictors are all equal to their mean values. On the other hand, $h_{ii}$ approaches its maximum value, 1, when the predictors are very far from their mean values. Thus residuals obtained from data points that are far from the centre of the data set will tend to be smaller than the corresponding error terms. Curves fit by least squares will usually fit better at extreme values for the predicators than in the central part of the data.

Table 5 below, displays the $h_{ii}$  values (along with many other diagnostic statistics that will be discussed) for the regression of log (price) on the seven predicators described above (cc, d, ps, w, l, pst, abs and ab). To compensate for the differences in dispersion among the distributions of the different residuals, it is usually better to consider the standardized residuals defined by
ith standardized residual =    $\frac{\hat{e}}{s\sqrt{1-h_{ii}}}$           for i = 1, 2, …, n (5.6)

Notice that the unknown $\sigma$ has been estimated by s. If n is large and if the regression assumptions are all approximately satisfied, then the standardized residuals should behave about like standard normal variables. Table 5 also lists the residuals and standardized residuals for all 28 observations.

Even if all the regression assumptions are met, the residuals (and the standardized residuals) are not independent. For example, the residuals for a model that includes an intercept term always add to zero. This alone implies they are negatively correlated. It may be shown that, in fact, the theoretical correlation coefficient between the ith and jth residuals (or standard residuals) is

                            $\frac{-h_{ij}}{\sqrt{\left(1-h_{ii}\right)\left(i-h_{jj}\right)}}$    (5.7)

Where $h_{ij}$ is the ijth element of the hat matrix? Again, the general formula for these elements is not needed here. For the simple single-predictor case it may be shown that

                 $h_{ij}=\frac{1}{n}+\frac{\left(x_i-\overline{x}\right)\left(x_i-\overline{x}\right)}{{\displaystyle \sum _{i=1}^n{\left(x_i-\overline{x}\right)}^2}}$ (5.8)

From Equations (5.3), (5.7) and (5.8) (and in general) we see that the correlations will be small except for small data sets and/or residuals associated with data points very far from the central part of the predictor values. From a practical point of view this small correlation can usually be ignored, and the assumptions on the error terms can be assessed by comparing the properties of the standardized residuals to those of independent, standard normal variables.

Residual plots
Plots of the standardized residuals against other variables are very useful in detecting departures from the standard regression assumptions. Many of the most common problems may be seen by plotting (standardized) residuals against the corresponding fitted values. In this plot, residuals associated with approximately equal-sized fitted values are visually grouped together. In this way it is relatively easy to see if mostly negative (or mostly positive) residuals are associated with the largest and smallest fitted values. Such a plot would indicate curvature that the chosen regression curve did not capture. Figure 5 displays the plot of Standardized Residuals versus fitted values for the new car data.

Another important use for the plot of residuals versus fitted values is to detect lack of common standard deviation among different error terms. Contrary to the assumption of common standard deviation, it is not uncommon for variability to increase as the values for response variables increase. This situation does not occur for the data contained in (Figure 5).

Figure 5 Scatter plot of standardized residuals versus fitted from new car data.

In regression analysis the model is assumed to be appropriate for all the observations. However, it is not unusual for one or two cases to be inconsistent with the general pattern of the data in one way or another. When a single predictor is used such cases may be easily spotted in the scatter plot data. When several predictors are employed such cases will be much more difficult to detect. The non conforming data points are usually called outliers. Sometimes it is possible to retrace the steps leading to the suspect data point and isolate the reason for the outlier. For example, it could be the result of a recording error, If this is the case the data can be corrected. At other times the outlier may be due to a response obtained when variables not measured were quite different than when the rest of the data were obtained. Regardless of the reason for the outlier, its effect on regression analysis can be substantial.

Outliers that have unusual response values are the focus here. Unusual responses should be detectable by looking for unusual residuals preferably by checking for unusually large standardized residuals. If the normality of the error terms is not in question, then a standard residual larger than 3 in magnitudes certainly is unusual and the corresponding case should be investigated for a special cause for this value.

Studentized residuals (t – residuals)

A difficulty with looking at standardized residuals is that an outlier, if present, will also affect the estimate of $\sigma$  that enters into the denominator of the standardized residual. Typically, an outlier will inflate s and thus deflate the standardized residual and mask the outlier. One way to circumvent this problem is to estimate the value of $\sigma$  use in calculating the ith standard residual using all the data except the ith case. Let $s_{\left(i\right)}$  denote such an estimate where the subscript (i) indicates that the ith case has been deleted. This leads to the Studentized residual defined by

ith Studentized residual =            $\frac{{\hat{e}}_i}{s_{\left(i\right)}\sqrt{1-h_{ii}}}$ for i = 1, 2, …, n    (5.6)

The next question to be asked is “how do these diagnostic methods work the new car data?” Table 5 lists diagnostic statistics for the regression model as applied to the new car data. Notice that there is only case (observation No. 14) where the standardized and studentized residuals have a significant difference and where the standard or studentized residuals are above 3 in magnitude. These results indicate that, in general, there are no outlier problems associated with the regression model described in (3.5) above.

Influential observations
The principle of ordinary least squares gives equal weight to each case. On the other hand, each case does not have the same effect on the fitted regression curve. For example, observations with extreme predictor values can have substantial influence on the regression analysis. A number of diagnostic statistics have been invented to quantify the amount of influence (or at least potential influence) that individual cases have in a regression analysis. The first measure of influence is provided by the diagonal elements of the hat matrix.

Leverage
When considering the influence of individual cases on regression analysis, the ith diagonal element of the hat matrix $h_{ii}$  is often called the leverage for the ith case, which means a measure of the ith data point’s influence in a regression with respect to the predicator variables. In what sense does $h_{ii}$  measure influence? It may be shown that ${\hat{y}}_i=h_{ii}{y}_i+{\displaystyle {\sum }_{j\ne i}{h}_{ij}{y}_j\text{ so that }\delta {\hat{y}}_i/\delta y_i=h_{ii}}$  , that is $h_{ii}$ is the rate of change of the ith fitted value with respect to the ith response value. If $h_{ii}$  is small, then a small change in the ith response results in a small change in the corresponding fitted value. However, if  is large, then a small change in the ith response produces a large change in the corresponding ${\hat{y}}_i$ .

Further interpretation of $h_{ii}$  as leverage is based on the discussion in Section 5.1. There is shown that the standard deviation of the sampling distribution of the ith residual is not $\sigma$  but $\sigma \sqrt{1-h_{ii}}$ . Furthermore, $h_{ii}$  is equal to its smallest value 1/n, when all the predicators are equal to their mean values. These are the values for the predictors that have the least influence on the regression curve and imply, in general, the largest residuals. On the other hand, if the predictors are far from their means, then approaches its largest value of 1 and the standard deviation of such residuals are quite small. In turn this implies a tendency for small residuals, and the regression curve is pulled toward these influential observations.

How large might a leverage value be before a case is considered to have large influence? It may be shown algebraically that the average leverage over all cases is (k+1)/n, that is,

                     $\frac{1}{n}{\displaystyle \sum _{i=1}^n{h}_{ii}}=\frac{k+1}{n}$      (5.10)

Where k is the number of predictors in the model. On the basis of this result, many authors suggest making cases as influential if their leverage exceeds two or three time (k+1)/n.

For the new car data displayed in Table 5and using the regression model as described in Equation (3.7) we estimate;

1. k = 7
2. (k+1)/n = 8/28 = 0.2857
3. 2 x (k+1)/n = 0.5714
4. 3 x (k+1)/n = 0.8571

In Table 5 only two observations (No.’s 16 and 27) are above the 2 x (k+1)/n threshold and none of observations are above the 3 x (k+1)/n threshold. This result indicates that there are no observations with extreme predictor’s value impacting on the slope of the fitted values and hence none of the observation has undue influence on the regression results.

Cook’s distance
As good as large leverage values are in detecting cases influential on the regression analysis, this criterion is not without faults. Leverage values are completely determined by the values of the predictor variables and do not involve the response values at all. A data point that possesses large leverage but also lies close to the trend of the other data will not have undue influence on the regression results.

Several statistics have been proposed to better measure the influence of individual cases. One of the most popular is called Cook’s Distance, which is a measure of a data point’s influence on regression results that considers both the predictor variables and the response variables. The basic idea is to compare the predictions of the model when the ith case is and is not included in the calculations.

In particular, Cook’s Distance, $D_i$ for the ith case is defined to be

                                              $D_i=\frac{{{\displaystyle \sum _{j=1}^n\left({\hat{y}}_j-{\hat{y}}_{j\left(i\right)}\right)}}^2}{\left(k+1\right)s^2}$ (5.11)

Where ${\hat{y}}_{j\left(i\right)}$  is the predicted or fitted value for case j using the regression curve obtained when i is omitted. Large values of $D_i$  indicate that case i has large influence on the regression results, as then ${\hat{y}}_j\text{ and }{\hat{y}}_{j\left(i\right)}$  differ substantially for many cases. The deletion of a case with a large value of $D_i$ will alter conclusions substantially. If $D_i$  is not large, regression results will not change dramatically even if the leverage for the ith case is large. In general, if the largest value of $D_i$ is substantially less than 1, then no cases are especially influential. On the other hand, cases with $D_i$ greater than 1 should certainly be investigated further to more carefully assess their influence on the regression analysis results.

In Table 5 only one observation No. 27 has the largest value of Cook’s Distance, 0.722 and leverage value, 0.6988. However, neither value is high enough to influence the regression analysis results.

What are next once influential observations have been detected? If the influential observation is due to incorrect recording of the data point, an attempt to correct that observation should be made and the regression analysis rerun. If the data point is known to be faulty but cannot be corrected, then that observation should be excluded for the data set. If it is determined that the influential data point is indeed accurate, it is likely that the proposed regression model is not appropriate for the problem at hand. Perhaps an important predictor variable has been neglected or the form of the regression curve is not adequate.

Transformations
So far a variety of methods for detecting the failure of some of the underlying assumptions of regression analysis have been discussed. Transformations of the data, either of the response and/or the predictor variables, provide a powerful method for turning marginally useful regression models into quire valuable models in which the assumptions are much more credible and hence the predictions much more reliable. Some of the most common and most useful transformations include logarithms, square roots, and reciprocals. Careful consideration of various transformations for data can clarify and simplify the structure of relationships among variables.

Sometimes transformations occur “naturally” in the ordinary reporting of data. As an example, consider a bicycle computer that displays, among other things, the current speed of the bicycle in miles per hour. What is really measured is the time it takes for each revolution of the wheel. Since the exact circumference of the tire is stored in the computer, the reported speed is calculated as a constant divided by the measured time per revolution of the wheel. The speed reported is basically a reciprocal transformation of the measured variable.

As a second example, consider petrol consumption in a car. Usually these values are reported in miles per gallon. However, they are obtained by measuring the fuel consumption on a test drive of fixed distance. Miles per gallon are then calculated by computing the reciprocal of the gallons per mile figure.

A very common transformation is the logarithm transformation. It may be shown that a logarithm transformation will tend to correct the problem of non constant standard deviation in case the standard deviation of $e_i$  is proportional to the mean of $y_i$ . If the mean of y doubles, then so does the standard deviation of e and so forth.

The unadjusted index is 105.6 (Appendix I), indicating a price increase, which, if it is used, would provide an incorrect measurement of the price changes of new cars over the period.

None.

Authors declare that there are no conflicts of interests.

1. Lancaster K. A New Approach to Consumer Theory. The Journal of Political Economy. 1966;74(2):132–157.
2. Griliches Z. Hedonic price indexes for automobiles: An econometric analysis of quality change. In The Price Statistics of the Federal Government. New York: USA: National Bureau of Economic Research; 1961. p. 173–196 .
3. Griliches Z. Hedonic Price Indexes Revisited. In: Griliches, editor. Price Indexes and Quality Change. Massachusetts: USA: Harvard University Press; 1971. p. 3–15
4. Diewert WE. Hedonic Prices: A consumer theory approach. Department of Economics. Vancouver: Canada: University of British Columbia; 2001. 1–12 p.
5. Diewert WE. Harmonized Indexes of Consumer Prices: Their conceptual foundations. Vancouver: Canada: Working European Central Bank; 2002.1–89 p.
6. Diewert WE. Hedonic Producer Price Indexes and Quality Adjustment. Department of Economics. Vancouver: Canada; University of British Columbia; 2002. 2–14 p.
7. Moulten BR. The Expanding Role of Hedonic methods in the Official Statistics of the United States, Bureau of Economic Analysis. Washington DC: USA: US Department of Commerce; 2001. 1–16 p.
8. Council Regulation (EC). No 2494/95 of 23 October 1995, concerning harmonized indices of consumer prices, Official Journal L 257. 1995:1–4.
9. Allen RGD. Index Numbers in Theory and Practice. Washington DC: USA: The Macmillan Press Ltd: 1975. 25 p.