WEBVTT 00:00:01.720 --> 00:00:03.260 Reliability and validity 00:00:03.260 --> 00:00:05.390 are two key qualities of research. 00:00:05.390 --> 00:00:09.040 Just as reliability and validity in quantitative research, 00:00:09.040 --> 00:00:12.670 we have a fairly well-understood set of tools. 00:00:12.670 --> 00:00:14.660 For example, in reliability, 00:00:14.660 --> 00:00:17.370 we measure the same thing many times 00:00:17.370 --> 00:00:19.420 and then we compare if those measures 00:00:19.420 --> 00:00:21.550 are consistent from the same person. 00:00:21.550 --> 00:00:23.440 If they are, we conclude 00:00:23.440 --> 00:00:26.270 that there's no random measurement there in the data 00:00:26.270 --> 00:00:28.180 and therefore, the data are reliable. 00:00:28.180 --> 00:00:29.813 In quantitative analysis, 00:00:29.813 --> 00:00:31.990 that analysis is done by a computer 00:00:31.990 --> 00:00:34.760 and computer doesn't make random mistakes. 00:00:34.760 --> 00:00:37.633 Therefore, reliability is only a feature of our data. 00:00:38.550 --> 00:00:41.240 When we analyze validity in quantitative research, 00:00:41.240 --> 00:00:43.220 we have the statistical conclusion validity, 00:00:43.220 --> 00:00:47.050 which refers to whether things are calculated correctly 00:00:47.050 --> 00:00:50.790 and what we calculate in the sample applies to population. 00:00:50.790 --> 00:00:55.290 Internal validity refers to whether we have a solid research design 00:00:55.290 --> 00:01:01.970 with good controls or with other techniques that ensure that our associations 00:01:01.970 --> 00:01:06.280 are actually evidence of causality and not just spurious correlations. 00:01:06.280 --> 00:01:08.230 Construct validity and more difficult 00:01:08.230 --> 00:01:11.770 to argue but that's basically an assessment 00:01:11.770 --> 00:01:15.300 of whether the data, the numbers that we have, actually measure 00:01:15.300 --> 00:01:17.080 what they are supposed to measure. 00:01:17.080 --> 00:01:19.280 So it's the correspondence between the numbers 00:01:19.280 --> 00:01:20.480 and the constructs. 00:01:20.480 --> 00:01:23.803 And then external validity, which was generalizability. 00:01:23.803 --> 00:01:26.900 Basically we analytically infer 00:01:26.900 --> 00:01:31.840 whether there are, finding from our population applies 00:01:31.840 --> 00:01:33.050 to other populations. 00:01:33.050 --> 00:01:34.420 And how we do that, 00:01:34.420 --> 00:01:37.360 we typically just look at what are the characteristics 00:01:37.360 --> 00:01:41.400 of our population and then how would those characteristics 00:01:41.400 --> 00:01:42.320 influence our result? 00:01:42.320 --> 00:01:46.868 So that's more of a theoretical exercise than a statistical exercise. 00:01:46.868 --> 00:01:52.820 So how do we do this kind of assessment in qualitative data analysis? 00:01:52.820 --> 00:01:54.230 We have to first understand 00:01:54.230 --> 00:01:58.390 that in qualitative data analysis, starting with reliability. 00:01:58.390 --> 00:02:01.110 So reliability was the lack of random noise 00:02:01.110 --> 00:02:04.200 or lack of chance errors in our data. 00:02:04.200 --> 00:02:10.110 In qualitative data analysis, the analysis itself is non-deterministic. 00:02:10.110 --> 00:02:14.400 So whereas in quantitative research, when you give numbers to a computer, 00:02:14.400 --> 00:02:18.120 the computer will always give you the same output. 00:02:18.120 --> 00:02:20.190 So if you calculate the correlation 00:02:20.190 --> 00:02:23.720 from one sample and then you calculate the same correlation 00:02:23.720 --> 00:02:25.000 from the same sample again, 00:02:25.000 --> 00:02:27.590 the correlation will always be the same. 00:02:27.590 --> 00:02:31.020 This is not necessarily the case in qualitative research. 00:02:31.020 --> 00:02:37.370 So if you have qualitative data and a person analyzes that data, 00:02:37.370 --> 00:02:39.323 then the person makes his interpretations 00:02:39.323 --> 00:02:43.680 or her interpretations based on her prior knowledge. 00:02:43.680 --> 00:02:48.768 And it is possible that if a person in a hypothetical scenario 00:02:48.768 --> 00:02:52.690 would analyze the data again, without remembering anything 00:02:52.690 --> 00:02:57.200 of the first data analysis round, the result would be different. 00:02:57.200 --> 00:03:00.770 So we can't really do this kind of recode 00:03:00.770 --> 00:03:02.002 and then code again, 00:03:02.002 --> 00:03:05.160 at least if we are one person. 00:03:05.160 --> 00:03:06.700 So how do we do it? 00:03:06.700 --> 00:03:09.210 There are two main ways 00:03:09.210 --> 00:03:14.940 of arguing reliability and checking for the reliability of a research study. 00:03:14.940 --> 00:03:19.880 The first thing is that you need to document your data well. 00:03:19.880 --> 00:03:23.400 So if your data is well documented, 00:03:23.400 --> 00:03:27.690 then this increases the likelihood 00:03:27.690 --> 00:03:30.600 that if someone were to re-analyze the data, 00:03:30.600 --> 00:03:33.280 they would come to the same conclusion. 00:03:33.280 --> 00:03:35.930 If your data is only your own field note 00:03:35.930 --> 00:03:39.460 that are just meant for you for that particular moment, 00:03:39.460 --> 00:03:43.640 then if you were to return to those data, let's say one year later, 00:03:43.640 --> 00:03:45.100 or those field notes, 00:03:45.100 --> 00:03:48.940 then it may be difficult to reproduce the old result 00:03:48.940 --> 00:03:52.860 and then your data, your study would not be reliable. 00:03:52.860 --> 00:03:57.010 Another thing that you can apply is that if you code your data, 00:03:57.010 --> 00:04:02.370 you can have two coders, so two people simultaneously analyze the data 00:04:02.370 --> 00:04:04.130 without talking to one another. 00:04:04.130 --> 00:04:08.060 And then you check whether how those people coded the data. 00:04:08.060 --> 00:04:10.060 If they coded in the same way, 00:04:10.060 --> 00:04:12.500 if yes, that is evidence for reliability, 00:04:12.500 --> 00:04:16.190 if no, that's evidence for lack of reliability. 00:04:16.190 --> 00:04:20.494 So reliability in qualitative research is typically about, 00:04:20.494 --> 00:04:25.950 how do we interpret the data that we observed? 00:04:25.950 --> 00:04:29.910 We assume basically here that if we observe something, 00:04:29.910 --> 00:04:31.810 then that observation is reliable. 00:04:31.810 --> 00:04:35.310 So reliability is more about coding. 00:04:35.310 --> 00:04:37.790 Then we had internal validity, 00:04:37.790 --> 00:04:42.700 which is about, whether the causal claims are correct or not. 00:04:42.700 --> 00:04:45.200 And in statistical analysis, this relates to, 00:04:45.200 --> 00:04:51.230 have we calculated things correctly and do we have the right controls, 00:04:51.230 --> 00:04:53.580 do we have a valid experimental design? 00:04:53.580 --> 00:04:55.980 In qualitative data analysis, 00:04:55.980 --> 00:05:02.010 what is important is that you look at the data rigorously. 00:05:02.010 --> 00:05:05.870 So you don't just cherry pick on some findings 00:05:05.870 --> 00:05:08.280 that support whatever you want to say. 00:05:08.280 --> 00:05:13.030 Instead, you give a fair assessment to the data. 00:05:13.030 --> 00:05:15.340 Then you should also look at cases 00:05:15.340 --> 00:05:20.330 that don't support your hypothesis or your theory. 00:05:20.330 --> 00:05:23.450 For example, if you are studying 00:05:23.450 --> 00:05:27.050 whether CEO gender influences profitability 00:05:27.050 --> 00:05:32.610 and you have a initial theory that if you name a woman as a CEO, 00:05:32.610 --> 00:05:35.000 then the profitability will go up. 00:05:35.000 --> 00:05:41.040 You need to also look at those cases where a woman was named as a CEO 00:05:41.040 --> 00:05:44.470 but profitability didn't go up. 00:05:44.470 --> 00:05:48.730 Or where profitability went up after naming a man as the CEO. 00:05:48.730 --> 00:05:52.970 Then you analyze those cases and that allows you to come 00:05:52.970 --> 00:05:55.290 to more valid causal interpretations 00:05:55.290 --> 00:05:59.420 because that reduces the effect of spurious findings. 00:05:59.420 --> 00:06:00.570 So you need to understand 00:06:00.570 --> 00:06:05.980 under which scenarios does the CEO gender lead 00:06:05.980 --> 00:06:07.790 to profitability differences. 00:06:07.790 --> 00:06:11.040 Then you also need to compare your finding against prior studies. 00:06:11.040 --> 00:06:14.320 So you need to check if there are prior studies, 00:06:14.320 --> 00:06:17.319 prior theories that would challenge your finding 00:06:17.319 --> 00:06:20.440 and if there's a prior study, the challenges you're finding, 00:06:20.440 --> 00:06:24.670 you need to understand why your finding is different from the prior study 00:06:24.670 --> 00:06:26.930 and then finally, theory triangulation refers 00:06:26.930 --> 00:06:31.020 to that you need to look at the data from different theoretical perspectives. 00:06:31.020 --> 00:06:32.870 So internal validity is basically, 00:06:32.870 --> 00:06:36.360 in qualitative research, refers to the level of rigor 00:06:36.360 --> 00:06:40.190 in your analysis and the more rigorous your analysis is, 00:06:40.190 --> 00:06:44.140 the more likely your causal claims are correct. 00:06:44.140 --> 00:06:46.284 Then we have construct validity, 00:06:46.284 --> 00:06:51.280 which basically refers to do my data measure the concepts 00:06:51.280 --> 00:06:53.100 that I claim them to measure? 00:06:53.100 --> 00:06:56.300 And there are a couple of different techniques. 00:06:56.300 --> 00:06:58.820 One is that we need to triangulate. 00:06:58.820 --> 00:07:02.418 So it is possible to either misinterpret our data. 00:07:02.418 --> 00:07:05.260 If we have two different kinds of data, 00:07:05.260 --> 00:07:09.010 for example, we have videos and we have interviews 00:07:09.010 --> 00:07:12.816 or we have interviews and then we have company documents, 00:07:12.816 --> 00:07:15.920 then we code both of those 00:07:15.920 --> 00:07:21.750 and if we extract the same meaning from two different datasets, 00:07:21.750 --> 00:07:25.360 for example, if three different kinds of data indicate 00:07:25.360 --> 00:07:31.610 that the company is innovative, then that supports the claim 00:07:31.610 --> 00:07:34.240 that we're actually making a valid inference, 00:07:34.240 --> 00:07:36.670 compared to just saying that, 00:07:36.670 --> 00:07:39.453 for example, if a person in a company tells 00:07:39.453 --> 00:07:41.370 that that company's innovative 00:07:41.370 --> 00:07:44.250 and that is only source of evidence that we have, 00:07:44.250 --> 00:07:47.210 then it's not very construct valid 00:07:47.210 --> 00:07:50.790 because we don't know if the person is honest or not. 00:07:50.790 --> 00:07:52.940 The second thing in construct validity 00:07:52.940 --> 00:07:57.360 is that we have to establish a clear chain of evidence and here, 00:07:57.360 --> 00:08:00.620 using qualitative data analysis also will help you. 00:08:00.620 --> 00:08:03.880 So you look from specific observations 00:08:03.880 --> 00:08:08.980 and then how you infer a general concept from those specific observations. 00:08:08.980 --> 00:08:11.159 Then people can, in ideal scenarios, 00:08:11.159 --> 00:08:14.890 people can check if they agree with our interpretations. 00:08:14.890 --> 00:08:17.220 If two people agree on the interpretation, 00:08:17.220 --> 00:08:22.170 then we would consider that also as evidence of construct validity. 00:08:22.170 --> 00:08:27.140 Finally, we have external validity, which is the same as generalizability 00:08:27.140 --> 00:08:32.540 and here things work basically the same way as in quantitative research. 00:08:32.540 --> 00:08:36.580 So in quantitative research, we have this good set of tools 00:08:36.580 --> 00:08:42.650 that we can apply to make inferences about the population that we're studying. 00:08:42.650 --> 00:08:46.200 External validity is about whether the population 00:08:46.200 --> 00:08:50.450 that we are studying represents also some other populations. 00:08:50.450 --> 00:08:55.300 So do the findings from our focal population extend to other populations as well? 00:08:55.300 --> 00:08:58.440 And in qualitative data analysis, this is the same. 00:08:58.440 --> 00:09:00.940 We don't really have any tools for that. 00:09:00.940 --> 00:09:04.920 We just have to apply what is called analytical generalization. 00:09:04.920 --> 00:09:06.700 So you try to think that okay, 00:09:06.700 --> 00:09:10.070 we studied something in Finland that relates to culture. 00:09:10.070 --> 00:09:16.330 Maybe this effect generalizes to other cultures similar to Finland. 00:09:16.330 --> 00:09:20.130 And how do you then evaluate existing studies? 00:09:20.130 --> 00:09:26.560 So when you do a study yourself, these are good principles to hold in mind. 00:09:26.560 --> 00:09:29.390 But evaluating studies done by others, 00:09:29.390 --> 00:09:32.660 you can basically evaluate the reliability and validity 00:09:32.660 --> 00:09:36.320 by checking to what extent these good principles 00:09:36.320 --> 00:09:39.663 are present in the study that you're evaluating.