Here’s question 6 of our exam:
6. You are applying hierarchical logistic regression on a survey of 1500 people to estimate support for a federal jobs program. The model is fit using, as a state-level predictor, the Republican presidential vote in the state. Which of the following two statements is basically true?
(a) Adding a predictor specifically for this model (for example, state-level unemployment) could improve the estimates of state-level opinion.
(b) It would not be appropriate to add a predictor such as state-level unemployment: by adding such a predictor to the model, you would essentially be assuming what you are trying to prove.
Briefly explain your answer in one to two sentences.
And the solution to question 5:
5. You have just graded an exam with 28 questions and 15 students. You fit a logistic item-response model estimating ability, difficulty, and discrimination parameters. Which of the following statements are basically true?
(a) If a question is answered correctly by students with low ability, but is missed by students with high ability, then its discrimination parameter will be near zero.
(b) It is not possible to fit an item-response model when you have more questions than students. In order to fit the model, you either need to reduce the number of questions (for example, by discarding some questions or by putting together some questions into a combined score) or increase the number of students in the dataset.
Briefly explain your answer in one to two sentences.
(a) is false. If a question is answered correctly by students with low ability, but is missed by students with high ability, then its discrimination parameter will be negative.
(b) is false. It’s no problem at all to have more questions than students. Even in a classical regression, even without a multilevel model, this is typically no problem as long as each question is asked by a few different students.
Common mistakes
Most of the students had the impression that one of (a) or (b) had to be true, so a common response was to work through one of the two options, figure out that it was false, and then mistakenly conclude that the other one was true. I guess I should rephrase the question. Instead of “Which of the following statements are basically true?”, I could say, “For each of the following statements, say whether it is true or false.”