Question

Suppose you conduct 10 significant tests and obtain the following p-values:

Test 1 2 3 4 5 6 7 8 9 10

p-value 0.001 0.030 0.002 0.006 0.040 0.003 0.010 0.100 0.020 0.004

• Which tests’ null hypotheses will you reject if you wish to control the family-wise error rate (FWER) at a significance level of 0.05?

• Which tests’ null hypotheses will you reject if you wish to control the false discovery rate (FDR) at a level of 0.05? Use the Benjamini-Hochberg method to answer this question by hand.

• Verify the results by using the related function in R

Answer 1

Which tests’ null hypotheses will you reject if you wish to control the family-wise error rate (FWER) at a significance level of 0.05?

10 tests are conducted. We have p-value for each test. We need to apply the Family-wise error rate (FWER)

Bonferroni Procedure for each test:

at level of significance alpha = 0.05,

So the results of the series of tests are as follows:

test #	p-value	FWER alpha	test result
1	0.001	0.005	reject NULL
2	0.030	0.005
3	0.002	0.005	reject NULL
4	0.006	0.005
5	0.040	0.005
6	0.003	0.005	reject NULL
7	0.010	0.005
8	0.100	0.005
9	0.020	0.005
10	0.004	0.005	reject NULL

Therefore, test # 1,3, 6, 10 have p-values that enable us to reject Null Hypothesis controlling for FWER at a significance level of 0.05.

Which tests’ null hypotheses will you reject if you wish to control the false discovery rate (FDR) at a level of 0.05? Use the Benjamini-Hochberg method to answer this question by hand.

Benjamini-Hochberg method is applied and results are given in table below for FDR level = 0.05

original test #	ordered p-value	[k]	k/m*alpha	p < k/m*alpha	Reject NULL
1	0.001	1	0.005	TRUE	Reject NULL
3	0.002	2	0.01	TRUE	Reject NULL
6	0.003	3	0.015	TRUE	Reject NULL
10	0.004	4	0.02	TRUE	Reject NULL
4	0.006	5	0.025	TRUE	Reject NULL
7	0.010	6	0.03	TRUE	Reject NULL
9	0.020	7	0.035	TRUE	Reject NULL
2	0.030	8	0.04	TRUE	Reject NULL
5	0.040	9	0.045	TRUE	Reject NULL
8	0.100	10	0.05	FALSE

Therefore for all tests expect test # 8, we reject the null hypothesis.

Verify the results by using the related function in R

> pvec <- c(0.001,0.03,0.002,0.006,0.04,0.003,0.01,0.1,0.02,0.004)
> length(pvec)
> alpha_FDR <- 0.05
> alpha_seq <- seq(1:10)/10*alpha_FDR
> alpha_seq
[1] 0.005 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050

### ordered pvalues

> pvec[order(pvec)]
[1] 0.001 0.002 0.003 0.004 0.006 0.010 0.020 0.030 0.040 0.100

> compare_pvec <- ifelse(pvec[order(pvec)] <= alpha_seq, 1,0)
> compare_pvec
[1] 1 1 1 1 1 1 1 1 1 0

### max where 1

> max(which(compare_pvec == 1))
[1] 9

### ALL Tests that are ordered from 1 to above cut-off = 9

> order(pvec)[1:9]
[1] 1 3 6 10 4 7 9 2 5

### reject null hypothesis for all these tests

### verification complete. Reject null hypothesis for all except test number 8