Revised 6/13/00
Traditional, Test-Based Inference
John Tukey Suggests How Knowledge and Belief are Bought
Components of Analysis
Summary
Traditional methods of statistical inference developed in the early part of the 20th century were primarily directed toward experimental data, much of which was agricultural- or laboratory-based. An experiment was planned, data were collected, an underlying distribution to the data was assumed, and data were tested for significance. The outcome of this type of analysis was to either retain or reject a null hypothesis in hopes of identifying significant differences within or between groups. After the test was completed, a new hypothesis was formulated (or the current hypothesis was modified) and the process begun anew, with data collected from scratch.
This traditional, test-based method of statistical inference, however, may not be well-suited for all types of data analysis, nor does it serve all research questions equally well. For one, it is not meant to uncover previously unrecognized patterns, nor is it suited for confirming "after the fact" patterns as nonrandom. In addition, it does not address potential biases inherent in non-experimental data, where randomization is absent and samples are frequently convenience-based. Perhaps most troubling, it fails to address magnitude of differences, coming only to black-and-white conclusions, in which 'only the black of perfect equality and the white of inequality [are] demonstrated' (Tukey, 1991, p. 101). The traditional test-based method of inference -- the method that dominates the curriculum of many introductory statistics courses -- is, therefore, inadequate for most biostatistical and epidemiological analyses; a more broad-based method of induction is needed.
The philosophy, development, and promotion of newer methods of statistical inference owe much to the work and words of statistician John Tukey. Tukey's work and philosophy are difficult to sum up without oversimplification, but insight into his many methods can be gleaned when he reminds us, "Mathematics can sometimes be put in black-and-white terms, but our knowledge and belief about the external world never can" (1991, p. 101). He also reminds us that knowledge and belief have a price, and is explicit in raising the question "With what coin do we buy knowledge and belief?" In answering this question, he identifies at least three kinds of "payments." These are:
You will notice that the first "coin" is based on care and insight, and not data analysis as such. It is therefore beyond the modest scope of this modest document. Still, it is worth considering -- at least briefly -- before each data analysis since, ultimately, it will affect our faith in the findings.
The second cost, the effort involved in collecting enough data, is a matter of collecting enough data to ensure adequate precision of our estimates and sufficient power of our tests. Sample size and power therefore, demand our attention, especially when data are inconclusive.
The third cost, formal error-rates as bases for our conclusions -- both the alpha and beta levels we are willing to "live with" -- are somewhat arbitrary but well-established by convention (a < 0.05, b < 0.20). Why these conventions have been established, and whether or not they are indeed justifiable, are, once again, beyond the scope of our brief discussion. By accepting them as a starting point -- and by understanding their basis -- we can save the energy of obsession that would be needed for their deconstruction.
So where do we go from here, once the care in which data were collected is considered, and once data are collected, cleaned, and ready for analysis? Tukey's starting point is called exploratory data analysis, or EDA for short. Fundamentally, EDA consists of graphical and numerical summaries of the data -- the type of thing you learn during the first couple of weeks of an introductory statistics class, primarily plots (e.g., stem-and-leaf plots), frequency tables (e.g., percentages), summary statistics (e.g., means and such). Possibly, because of their simplicity, simple descriptive techniques are often overlooked in preference for the mumbo-jumbo of hypothesis testing. (A big mistake!) Tukey, however, reminds us that EDA is an essential element of any analysis.
What then follows numerical and graphical descriptions? I would suggest our next step involves clearly denoting the relevant parameters we wish to infer. Inferential methods include both estimation and hypothesis testing. Estimation includes both point estimation and interval estimation. The point estimate provides a single best approximation to the parameter being studied. Confidence intervals provide an interval of likely capture. In reviewing estimates, information about the direction of observed difference (positive, negative or no difference), the precision of such estimates, and significance of observed differences should be made clear. It is clear that confidence intervals are irreplaceable and are almost always needed.
But what then of tests of significance? Are not black-and-white conclusions useful when a decision must be made? Certainly, for "both exploratory and confirmatory data analyses deserve our attention. Both detection and adjudication play crucial roles -- in the progress of science as in the control of crime" (Tukey, 1969). Hypothesis testing remains the basis for confirming results as "nonrandom." Their use may never die.
So now that we have suggested a cohesive strategy, let us review its component parts. We start with a consideration of the the research question and study design and data used to address it. Notice that this first step requires no calculations. Second, we describe the data numerically and graphically to understand its distribution and relationships between variables in the sample. Estimation is then pursued, with both point estimates and interval estimates calculated and reported. This then is followed confirmatory tests of significance, with the hypotheses, alpha levels, test statistics, and conclusion explicitly stated. In my opinion, results should then be summarized in narrative form, using concise yet not overly formal language, and, if data are not significant, the power of the test should be considered.(Type II errors happen!). In summary, these steps are:
These 6 analytic steps will be considered throughout this Web-text.
Tukey, J. W. (1969.) Analyzing data: Sanctification or detective work? American Psychologist, 24, 83 - 91.
Tukey, J. W. (1991). The philosophy of multiple comparisons. Statistical Science, 6, 100 - 116.