Is It Significant?
Two Kinds of Significance: Statistical Significance and Clinical Significance
In order for a study to have clinical significance the differences (or lack thereof) between the two (or more) drugs being studied should be statistically significant.
STATISTICAL SIGNIFICANCE
Alpha statistic [Type I error] also known as the “p” value
If you conclude that a difference exists between “Drug A” and “Drug B” then you need to be sure that the difference is real. (That is, not due to chance). If the alpha statistic is set to p < 0.05, then there is a high ikelihood (e.g. p<0.05 means that there is a 1.00 - 0.05 = 0.95; a > 95% likelihood) that there really is a difference and that it wasn't due to chance. In other words the results are reproducible. In this example (p<0.05) the Type I error rate is limited to < 5%.
Beta statistic [Type II error]
Alternatively, if you conclude that a difference DOES NOT exist between the between “Drug A” and “Drug B” then you need to be sure that you had the ability to detect a difference if one actually existed (e.g. typically Beta<0.2 which means that there is a 1.00 - 0.2 = 0.8; a > 80% likelihood) that you had the ability to detect a difference if one actually existed (also called Power). This latter aspect is crucial as small studies can fail to detect a difference (often due to too few subjects - when this occurs the study has a high type II error, sometimes also called Beta error).
The POWER of a study using a Beta of 0.2 is 80% (i.e. 1 - Beta or in this case 1 - 0.2 = 0.8). Again this statistic gets at reproducibility and in the example of Beta of 0.2 limits the Type II error rate to 20% or less.
Type II error rates can be lower than 0.2, however, the typical is 0.2.
Other notes: (standard deviation vs. standard error)
The standard error is not the same as the standard deviation. (The standard error is smaller and is often used to trick readers into believing that there is little variance in the effect - which suggest it is a highly reproducible effect). Standard error of the mean should only be used when comparing arms with different sample sizes.
Note: Standard error of the mean is the standard deviation divided by the square root of the sample size of that arm.
CLINICAL SIGNIFICANCE
Just because something is statistically significant does not mean that it has meaning to your practice (i.e. clinical significance). Mean values are often misleading (i.e. they don't mean too much). Rather, look at the 95% CONFIDENCE INTERVAL (C.I.) around the mean. The 95% C.I. will give you a better idea of the actual RANGE of effect that your specific patient might expect to experience.
If the 95% confidence interval for Drug A overlaps with that of Drug B then the difference is generally not so dramatic as the means may suggest. Do not confuse Confidence Intervals with standard deviation (that which typically follows the mean when being reported in research papers (i.e. mean ± s.d.)).
If you don't have a 95% confidence interval then you can use the s.d. in place of the 95% confidence interval. HOWEVER, a standard deviation will only produce a confidence interval of 66% (if the data are normally distributed). In statistics there is a big difference between 66% and 95%. If the standard deviations overlap then again the difference is not usually such a big deal CLINICALLY.
Note: Look for s.d. and C.I. that do not overlap then you really have a difference.
Please note: There are exceptions to every rule and please realize that this is a very abbreviated overview.