The Randomization test for two means option under the StatCrunch > Applets menu is designed to illustrate the way the randomization test works for the case of comparing two means. In this case, the null hypothesis states that the two population means are the same. To simulate this situation, the test repeatedly randomizes the sample assignments for each of the data values. The differences between the means for the randomized data are then compared to the observed mean difference for the original data in order to determine statistical significance.
The interface for constructing the applet is very similar to that of the two sample t procedure within StatCrunch as shown in Result 1 for the fictitious Yellow/Green exam data set. This data set contains mythical scores for two versions of the same exam (one yellow, one green) each given to a sample of 10 students. The goal is to determine if the mean score is significantly different between the two versions. The user selects the column containing the first sample along with an optional Where clause to specify the data rows to be included in the sample. The user then selects the column containing the second sample along with an optional Where clause to specify the data rows to be included in the sample. Note that for stacked data the first and second column will typically be the same as in this example. In this case, the user may also enter optional short labels for each sample to be used in the resulting applet.
The resulting applet is shown in Result 2. The applet offers the ability to randomize the data 1 time, 5 times or 1000 times. These ideas match pedagogically with the ideas of step, walk and run, respectively. Click the 1 time button to see a single randomization of the data. A new window will pop up which shows the actual shuffling of the sample labels with the results then displayed in the data table within the applet. The difference between the means of the shuffled samples (Sample 1 - Sample 2) for the shuffled sample labels is computed below the data table. If the mean difference for the randomized data is more extreme in absolute value than the observed difference for the original data, it will be displayed in red otherwise it is displayed in black. Click the 5 times button to see this process repeated five times at a slightly faster pace. A running tally of the results is provided in the Results panel in the right portion of the applet. A histogram of color-coded randomized mean differences is displayed. A table of the frequencies and relative frequencies for the randomized mean differences that are less than the negative absolute value of the observed mean difference or greater than the absolute value of the observed mean difference is also shown. The counts/proportions totaled across both regions is also provided in the bottom line of this table. The user can use the appropriate line from this table to estimate a P-values based on the proper alternative for a large number of randomizations.
A large number of randomizations can be quickly generated by clicking the 1000 times button. In this case, the individual randomizations are not displayed, but the Results panel is quickly updated after each randomization is computed. A user may desire to click this button several times to generate an even larger number of total randomizations. The user may then go back and inspect the results by clicking on the individual randomization numbers displayed in the applet or by clicking on the histogram in the Results panel. The far left panel in the applet contains the results of each randomization ordered numerically with those providing more extreme than observed results shown in red. Clicking on a randomization number shows the corresponding randomized data in the data table and the resulting mean difference. A small triangle is also displayed at the proper location corresponding to this mean difference within the histogram. The Graph button located below the data table can be used to construct a simple dotplot of the selected randomization. A user can select randomizations within the histogram by clicking and dragging the mouse to draw a rectangle intersecting a specific bar or bars. The total number of randomizations with mean differences in the selected bar(s) is then displayed in a pop up window along with the option for selecting individual color-coded randomizations falling within the selected range. From a pedagogical perspective, students may use this applet to understand the idea of repeated randomization and may identify and explore randomizations that are extreme.
After generating a large number of realizations in the example below, a two-sided P-value of approximately 0.26 is obtained so that in this case there is not significant evidence that the mean scores for the yellow and green exams are significantly different using a standard level of significance like 0.05.
When saving the applet from within StatCrunch, the user is asked whether or not to save the current random number seed. This seed determines the randomizations that will be generated as described above. If the seed is saved, subsequent users will generate the exact same randomizations as the current user. If it is not saved, then subsequent users will each get different randomizations.