WEBVTT Kind: captions Language: en 00:00:00.090 --> 00:00:03.060 I will now demonstrate  regression analysis using R. 00:00:03.060 --> 00:00:05.190 To demonstrate regression, 00:00:05.190 --> 00:00:07.050 we need a data set to work with 00:00:07.050 --> 00:00:09.330 and we will be using the Prestige data set 00:00:09.330 --> 00:00:11.550 that is used in one of the assignments. 00:00:11.550 --> 00:00:19.680 This data set is data of 102 occupations  from the census of Canada in the early 70s. 00:00:19.680 --> 00:00:22.050 The variables that we have are, 00:00:22.050 --> 00:00:25.590 the units that we are observing are occupations, 00:00:25.590 --> 00:00:27.840 and the variables are education, 00:00:27.840 --> 00:00:33.360 which is the average number of years that  a person who has this occupation has, 00:00:33.360 --> 00:00:36.300 what is the average income of this occupation, 00:00:36.300 --> 00:00:41.880 how many people in this occupation  are women from 0 to 100, 00:00:41.880 --> 00:00:47.310 a prestige score using some kind of  scale that we don't really know about, 00:00:47.310 --> 00:00:51.240 census code, which is just an  identity that we don't care about, 00:00:51.240 --> 00:00:52.980 and then type, 00:00:52.980 --> 00:00:58.740 which is a categorical variable of white  color, blue color and professional workers. 00:00:58.740 --> 00:01:02.460 This data set is available  in the Companion to Applied   00:01:02.460 --> 00:01:05.100 Regression Analysis or CAR package in R. 00:01:05.100 --> 00:01:08.760 And you can access the  package in R by using these, 00:01:08.760 --> 00:01:12.360 and the help file that I'm showing  here by using these R commands. 00:01:12.360 --> 00:01:17.040 So when we start analyzing a data set, 00:01:17.040 --> 00:01:18.840 the first thing that we need to do is to 00:01:18.840 --> 00:01:20.400 understand what the data are about. 00:01:20.400 --> 00:01:23.100 So what are the scales of the variables, 00:01:23.100 --> 00:01:25.110 what are the means and standard deviations, 00:01:25.110 --> 00:01:26.730 how are the variables related? 00:01:26.730 --> 00:01:30.360 So calculating means and standard  deviations is straightforward, 00:01:30.360 --> 00:01:31.320 so I will not cover that, 00:01:31.320 --> 00:01:34.620 but I'll show you one very  useful data exploration tool 00:01:34.620 --> 00:01:36.990 before I go to the actual regression analysis. 00:01:36.990 --> 00:01:40.230 My favorite regression analysis tool 00:01:40.230 --> 00:01:43.050 for the exploratory analysis is 00:01:43.050 --> 00:01:44.070 the scatter plot matrix. 00:01:44.070 --> 00:01:47.610 So a scatter plot matrix is a  collection of scatter plots. 00:01:47.610 --> 00:01:49.710 So we have for each pair of variables, 00:01:49.710 --> 00:01:51.480 we have income here on the y-axis, 00:01:51.480 --> 00:01:54.420 and we have education here on the x-axis, 00:01:54.420 --> 00:01:56.430 and then we have observations here. 00:01:56.430 --> 00:01:57.780 So we can see graphically, 00:01:57.780 --> 00:02:00.180 how education and income are related. 00:02:00.180 --> 00:02:06.480 We can see that they clearly  have a positive relationship, 00:02:06.480 --> 00:02:10.020 but the relationship is not exactly linear, 00:02:10.020 --> 00:02:14.490 whereas there's a small increase initially  in income on education increases, 00:02:14.490 --> 00:02:17.070 then at some point, it starts  to increase more rapidly, 00:02:17.070 --> 00:02:19.710 so it kind of like curves up a bit. 00:02:19.710 --> 00:02:21.780 We can see one outlier here 00:02:21.780 --> 00:02:25.350 so that's an observation that  is different from others, 00:02:25.350 --> 00:02:27.510 could be a problem for our analysis, 00:02:27.510 --> 00:02:29.910 or it could be an interesting  case that we want to study more, 00:02:29.910 --> 00:02:31.890 I have another video about outliers. 00:02:31.890 --> 00:02:36.360 So what we do with this scatter plot  matrix is that we look at the data, 00:02:36.360 --> 00:02:37.980 we look for patterns, 00:02:37.980 --> 00:02:42.360 and we look for expected patterns  and also unexpected patterns. 00:02:42.360 --> 00:02:46.830 So here we have an interesting  pattern between income and women. 00:02:46.830 --> 00:02:49.710 So when the share of women is low, 00:02:49.710 --> 00:02:53.430 then we have both high-income  and low-income professions. 00:02:53.430 --> 00:02:55.770 When the share of women goes up, 00:02:55.770 --> 00:02:59.400 we only have low-income professions. 00:02:59.400 --> 00:03:03.930 So left is low, right is more,  up is more, down is less. 00:03:04.740 --> 00:03:08.550 So being a woman basically guarantees that 00:03:08.550 --> 00:03:11.250 you don't have access to  these high paying professions, 00:03:11.250 --> 00:03:14.790 but being a man doesn't guarantee you a high pay, 00:03:14.790 --> 00:03:17.970 instead, there are many  occupations that men dominated, 00:03:17.970 --> 00:03:19.260 that don't pay that much. 00:03:19.260 --> 00:03:25.440 And this upper diagonal here is the same plots, 00:03:25.440 --> 00:03:26.970 they're just transposed. 00:03:26.970 --> 00:03:29.430 So this plot is the exact same as that plot, 00:03:29.430 --> 00:03:31.590 it's just a mirror image of the same plot. 00:03:31.590 --> 00:03:36.510 So that's useful to go through  when you analyze your own data set. 00:03:36.510 --> 00:03:37.920 I will not go through that in detail, 00:03:37.920 --> 00:03:40.770 but it's a very useful tool to use 00:03:40.770 --> 00:03:42.900 when you actually start analyzing data yourself. 00:03:42.900 --> 00:03:45.540 Let's go to the actual regression model. 00:03:45.540 --> 00:03:47.790 And to do regression analysis, 00:03:47.790 --> 00:03:48.960 we need to have a research question. 00:03:48.960 --> 00:03:51.240 And our research question is: 00:03:51.240 --> 00:03:55.530 does the prestige of an occupation  depend on education, income,   00:03:55.530 --> 00:03:57.600 and share of women in the occupation? 00:03:57.600 --> 00:04:00.270 So expressed as a regression model, 00:04:00.270 --> 00:04:05.610 we say that the prestige of an  occupation is a function of beta 0, 00:04:05.610 --> 00:04:06.570 which is some base level, 00:04:06.570 --> 00:04:10.230 when education income and the share of women is 0, 00:04:10.230 --> 00:04:14.610 of course, that doesn't exist but that's  a base level in our regression analysis. 00:04:14.610 --> 00:04:17.760 Then we have beta 1 is the effective education, 00:04:17.760 --> 00:04:19.920 beta 2 is the effect of income, 00:04:19.920 --> 00:04:22.050 beta 3 is the effect of women, 00:04:22.050 --> 00:04:26.250 and then there's the error term u that  represents some variation in the data, 00:04:26.250 --> 00:04:28.140 that the model doesn't explain. 00:04:28.140 --> 00:04:33.630 Here is a link to an explanation of the output, 00:04:33.630 --> 00:04:35.430 and I will now explain the output. 00:04:35.430 --> 00:04:37.440 So I run the regression analysis myself, 00:04:37.440 --> 00:04:41.610 and running the regression  analysis is straightforward. 00:04:41.610 --> 00:04:44.250 I just specify the model here, 00:04:44.250 --> 00:04:45.780 the lm command, 00:04:45.780 --> 00:04:52.110 the tilde defines that prestige is approximately a  weighted sum of education income and women, 00:04:52.110 --> 00:04:54.390 and we are using the prestige data, 00:04:54.390 --> 00:04:57.390 and then we print out the  summary of the regression model, 00:04:57.390 --> 00:05:01.650 which I stored as reg1 object. 00:05:01.650 --> 00:05:03.030 So we get some output, 00:05:03.030 --> 00:05:04.440 and what's the output about? 00:05:04.440 --> 00:05:07.080 The regression analysis output from your software, 00:05:07.080 --> 00:05:10.050 is always some text and a lot of numbers. 00:05:10.050 --> 00:05:12.180 And we need to understand what these numbers mean, 00:05:12.180 --> 00:05:13.500 what do the numbers tell us, 00:05:13.500 --> 00:05:15.780 and which numbers are relevant for you, 00:05:15.780 --> 00:05:16.530 and which ones are not, 00:05:16.530 --> 00:05:21.210 because some of these numbers  are not very meaningful unless, 00:05:21.210 --> 00:05:23.520 you want to do, for example, moral comparisons. 00:05:23.520 --> 00:05:26.100 So the output contains a couple of parts. 00:05:26.100 --> 00:05:29.160 First, we have the summary  of the model that we ran. 00:05:29.160 --> 00:05:33.480 So just reminder that this  is the regression analysis, 00:05:33.480 --> 00:05:37.050 for example, if we run five  or ten different regressions 00:05:37.050 --> 00:05:38.520 and we print the results at the end, 00:05:38.520 --> 00:05:39.570 it's useful to know, 00:05:39.570 --> 00:05:42.720 what is the regression model for these estimates? 00:05:42.720 --> 00:05:45.570 The second is the residual statistics, 00:05:45.570 --> 00:05:48.870 so this is the residual, the part  that the model doesn't explain. 00:05:48.870 --> 00:05:52.320 The residual is assumed to  be normally distributed, 00:05:52.320 --> 00:05:58.620 so we can check that the  median should be at the mean, 00:05:58.620 --> 00:06:02.280 and residual has a mean of zero  because that's how it's defined, 00:06:02.280 --> 00:06:06.210 and then the quartiles, first  and second, the third quartile, 00:06:06.210 --> 00:06:09.900 should be about equally far apart from the mean, 00:06:09.900 --> 00:06:14.610 and also the minimums and maximums  should be about equally far from the mean, 00:06:14.610 --> 00:06:16.410 and it's roughly the same. 00:06:16.410 --> 00:06:19.140 So the difference between is two units, 00:06:19.140 --> 00:06:21.480 and the maximum is 17 units, 00:06:21.480 --> 00:06:22.650 the minimum is 19 units. 00:06:22.650 --> 00:06:25.950 So it's about 10% difference, 00:06:25.950 --> 00:06:26.760 so not a big deal. 00:06:26.760 --> 00:06:30.210 Then we have the actual regression estimates. 00:06:30.210 --> 00:06:31.710 So these are the main results, 00:06:31.710 --> 00:06:34.590 these tell us what are the individual effects. 00:06:34.590 --> 00:06:38.010 I will be looking at explaining this  part in more detail on the next slide, 00:06:38.010 --> 00:06:42.480 but that's the model estimates or the results. 00:06:42.480 --> 00:06:45.720 Then we have some model indices, 00:06:45.720 --> 00:06:48.390 most importantly we have the R-squared, 00:06:48.390 --> 00:06:50.700 we have some other things 00:06:50.700 --> 00:06:54.900 that tell us something about the  goodness of this overall model, 00:06:54.900 --> 00:06:57.840 and some of these can also be used for comparing 00:06:57.840 --> 00:07:00.540 some models that I will address in another video. 00:07:00.540 --> 00:07:03.210 So how do we interpret this? 00:07:03.210 --> 00:07:05.220 Let's look at the regression coefficients first. 00:07:05.220 --> 00:07:07.980 So we have a couple of things here, 00:07:07.980 --> 00:07:10.020 we have the regression coefficient here, 00:07:10.020 --> 00:07:13.020 and then we have the standard error, 00:07:13.020 --> 00:07:15.960 which is estimated somehow, 00:07:15.960 --> 00:07:18.300 then we have t value, 00:07:18.300 --> 00:07:23.790 which is simply the test statistic  for testing the null hypothesis. 00:07:23.790 --> 00:07:29.280 And the t value is defined as estimate  divided by the standard error, 00:07:29.280 --> 00:07:31.800 and you can verify that it is 00:07:31.800 --> 00:07:41.040 6.79 divided by 3.23 is actually -2.098  by using a hand calculator if you wish. 00:07:41.040 --> 00:07:43.950 Then we have the p-value. 00:07:43.950 --> 00:07:50.010 So the p-value here is calculated  based on some assumptions. 00:07:50.010 --> 00:07:53.490 The assumptions are not important at this point, 00:07:53.490 --> 00:07:56.550 they are important, I'll cover them later. 00:07:56.550 --> 00:08:01.020 The null hypothesis for the p-value is that 00:08:01.020 --> 00:08:02.820 this regression coefficient is 0. 00:08:02.820 --> 00:08:04.530 So what's the likelihood, 00:08:04.530 --> 00:08:08.580 or what's the probability of  obtaining an estimate of -6.8, 00:08:09.240 --> 00:08:12.330 if there is no effect in the population. 00:08:12.330 --> 00:08:16.980 The probability is 0.03, which is less than 0.05. 00:08:16.980 --> 00:08:20.190 We conclude that this is  statistically significant, 00:08:20.190 --> 00:08:26.430 as p is less than 0.05, which is  there the conventional minimum level 00:08:26.430 --> 00:08:27.330 that we want to have. 00:08:27.330 --> 00:08:36.720 This one star here indicates that  it's below the 0.05 threshold, 00:08:36.720 --> 00:08:39.990 so this is legit for the p-values. 00:08:39.990 --> 00:08:45.780 Three stars mean that it is below or 0.001. 00:08:45.780 --> 00:08:52.890 So these were the regression coefficients, 00:08:52.890 --> 00:08:55.110 then we have some other things also here, 00:08:55.110 --> 00:09:00.000 but importantly we have this very  small regression coefficient. 00:09:00.000 --> 00:09:03.150 Why is the estimate for income so small? 00:09:03.150 --> 00:09:06.210 Does it mean that the income doesn't really matter 00:09:06.210 --> 00:09:10.320 when we consider the prestigious of occupations? 00:09:10.320 --> 00:09:17.160 The reason why this coefficient or  what does this coefficient mean, 00:09:17.160 --> 00:09:20.130 we have to consider the scales of the variables. 00:09:20.130 --> 00:09:23.760 So the income was expressed as dollars. 00:09:23.760 --> 00:09:32.250 So one dollar increase in income increases  your prestigious by 0.0013 units. 00:09:32.250 --> 00:09:35.880 So incomes are in thousands of dollars, 00:09:35.880 --> 00:09:38.970 so $1 increase doesn't really make a difference. 00:09:38.970 --> 00:09:44.370 So maybe it would make more  sense to rescale the income. 00:09:44.370 --> 00:09:48.570 So that instead of being  expressed as individual dollars, 00:09:48.570 --> 00:09:51.510 we would express it as thousands of dollars. 00:09:51.510 --> 00:09:55.140 So if we multiply this by 1000 then we get, 00:09:55.140 --> 00:10:01.500 what is the effect of increasing your income  by a thousand units or thousand Canadian dollars, 00:10:01.500 --> 00:10:05.370 and then it's more meaningful. 00:10:05.370 --> 00:10:10.230 We can also see that the effect  is small in absolute magnitude, 00:10:10.230 --> 00:10:11.670 ignoring the scale, 00:10:11.670 --> 00:10:14.610 doesn't mean that it's non-significant, 00:10:14.610 --> 00:10:18.330 because of the scaling issue,  this is actually very significant, 00:10:18.330 --> 00:10:20.520 and it's actually a pretty large effect 00:10:20.520 --> 00:10:23.040 when you think about the scale of the variable 00:10:23.040 --> 00:10:28.200 from about a few thousand Canadian  dollars to about twenty-five thousand. 00:10:28.200 --> 00:10:37.290 The obvious thing to do here would  be to recode the income to thousands, 00:10:37.290 --> 00:10:40.230 so we get estimates that are more comparable, 00:10:40.230 --> 00:10:41.820 and they're easier to interpret as well. 00:10:41.820 --> 00:10:46.770 We will next be looking at the  model quality and this is done here. 00:10:46.770 --> 00:10:51.480 So the model quality indices tell us  something about the overall model fit. 00:10:51.480 --> 00:10:55.830 The most important part is  the R-squared statistic here, 00:10:55.830 --> 00:11:00.300 and R-squared tells us how  much the model explains. 00:11:00.300 --> 00:11:06.000 So these three variables together:  income, the share of women and education, 00:11:06.000 --> 00:11:09.390 explained about 80% of the  variation of the prestige. 00:11:09.390 --> 00:11:14.700 So we can say that prestige of  occupation is mostly determined by 00:11:14.700 --> 00:11:19.290 amount of women, income and education. 00:11:19.290 --> 00:11:22.470 Of course, which one of those is  the most important determinant, 00:11:22.470 --> 00:11:27.240 we would have to look at the actual  individual regression coefficients, 00:11:27.240 --> 00:11:29.790 but all together they explain  about 80% of the data. 00:11:29.790 --> 00:11:33.390 Then we have adjusted R-squared, 00:11:33.390 --> 00:11:36.390 which is 0.79 and that is 00:11:36.390 --> 00:11:42.600 only slightly smaller than the R-squared, 00:11:42.600 --> 00:11:45.210 because we had only three explanatory variables, 00:11:45.210 --> 00:11:48.150 and we had more than 100 observation. 00:11:48.150 --> 00:11:52.500 So we have more than 30 observations  for each explanatory variable, 00:11:52.500 --> 00:11:55.890 so the adjustment by the adjusted  R-squared is pretty small, 00:11:55.890 --> 00:11:59.160 because the bias can be  expected to be small as well 00:11:59.160 --> 00:12:02.940 with that good ratio of variables to observations. 00:12:02.940 --> 00:12:06.060 So that's the adjusted R-squared. 00:12:06.060 --> 00:12:10.350 Adjusted R-squared is useful for comparing models 00:12:10.350 --> 00:12:11.790 that are non-nested, 00:12:11.790 --> 00:12:13.050 what that means I'll cover later, 00:12:13.050 --> 00:12:16.620 but it's also useful for interpretation, 00:12:16.620 --> 00:12:20.880 so whenever you are unsure whether  you should be looking at R-squared, 00:12:20.880 --> 00:12:21.900 adjusted R-squared, 00:12:21.900 --> 00:12:25.740 it's always a better idea to interpret  the model using adjusted R-squared. 00:12:25.740 --> 00:12:27.330 If your sample size is large, 00:12:27.330 --> 00:12:28.710 it doesn't make a difference. 00:12:28.710 --> 00:12:29.940 These are about the same, 00:12:29.940 --> 00:12:33.090 so there's no meaningful  difference between the two. 00:12:33.090 --> 00:12:34.560 If your sample size is small 00:12:34.560 --> 00:12:41.010 then adjusted R-squared typically  is a more relevant metric for 00:12:41.010 --> 00:12:43.740 judging how well your model expressed the data. 00:12:43.740 --> 00:12:46.800 Then we have some other statistics. 00:12:46.800 --> 00:12:49.950 So we have the residual standard error, 00:12:49.950 --> 00:12:53.580 so this is the standard deviation of the residuals 00:12:53.580 --> 00:12:56.250 and it estimates, what is the standard error, 00:12:57.240 --> 00:13:01.050 it's an estimate of the standard  deviation of the error term, 00:13:01.050 --> 00:13:04.170 and it's not typically interpreted  because it depends on the scale, 00:13:04.170 --> 00:13:06.180 but we can do some calculations, 00:13:06.180 --> 00:13:10.590 for example, R-squared is  calculated using this number. 00:13:10.590 --> 00:13:15.120 Then we have 98 degrees of freedom. 00:13:15.120 --> 00:13:19.350 That tells us, how complex the  model is related to our data, 00:13:19.350 --> 00:13:23.490 and it tells that we could add  98 more things to the model and 00:13:23.490 --> 00:13:24.660 still, be able to estimate it. 00:13:24.660 --> 00:13:27.240 The degrees of freedom is  not interpreted directly, 00:13:27.240 --> 00:13:30.270 but it's used for model comparison and 00:13:30.270 --> 00:13:33.450 it's used for calculating some statistics. 00:13:33.450 --> 00:13:36.300 For example, the F-statistic was shown here, 00:13:37.470 --> 00:13:40.110 its distribution depends  on the degrees of freedom. 00:13:40.110 --> 00:13:43.110 So the F-statistics is here, 00:13:43.110 --> 00:13:45.690 it's useful for more comparisons. 00:13:46.290 --> 00:13:48.810 It can be calculated based  on the R-squared for example, 00:13:48.810 --> 00:13:51.180 it's not interpreted directly, 00:13:51.180 --> 00:13:56.250 but it has a distribution that we  can use for testing a null hypothesis 00:13:56.250 --> 00:13:58.680 that the R-squared is exactly zero. 00:13:58.680 --> 00:14:05.550 So the t statistic, it provides a test statistic  for a regression coefficient being zero, 00:14:05.550 --> 00:14:09.150 and we compare that against the t distribution. 00:14:09.150 --> 00:14:13.560 F-statistic is a test statistic  for the R-squared being zero, 00:14:13.560 --> 00:14:17.460 and we compare it against  the correct F-distribution, 00:14:17.460 --> 00:14:19.950 here is the p-value for that comparison. 00:14:19.950 --> 00:14:22.020 So getting this kind of results, 00:14:22.020 --> 00:14:25.050 if all the independent variables were   00:14:25.050 --> 00:14:28.020 completely unrelated with the  dependent variable linearly, 00:14:28.020 --> 00:14:30.840 would be it will be very unlikely. 00:14:30.840 --> 00:14:33.420 So we reject the null  hypothesis and we conclude that 00:14:33.420 --> 00:14:38.610 these variables do explain the dependent  variable also in the population. 00:14:38.610 --> 00:14:42.510 So we get all kinds of things from the model. 00:14:42.510 --> 00:14:46.650 The most important part to  interpret in these indices is 00:14:46.650 --> 00:14:50.370 the R-squared or the adjusted R-squared, 00:14:50.370 --> 00:14:51.720 if you don't know which one to use, 00:14:51.720 --> 00:14:53.250 use the adjusted one. 00:14:53.250 --> 00:14:57.330 The other ones are used for model  comparisons calculating other things, 00:14:57.330 --> 00:14:59.760 and those will be irrelevant when you do   00:15:00.570 --> 00:15:03.060 model comparisons that I'll  explain in another video.