(This article was originally published at The Data Game - Ilan Man, and syndicated at StatsBlogs.)
Recall that in last week's installment of Coursera's Machine Learning class, we covered linear regression - the most fundamental way of making a prediction where the outcome is a number, like predicting the price of a stock or how much rainfall (in inches) we should expect tomorrow. We discussed solving the linear equation in two ways (1) using normal equations and (2) a numerical approximation method, such as gradient descent. Week 3 expands on this concept and introduces a basic classification algorithm - logistic regression. Some of you might recognize this as simply another regression tool from stats class, of the non-linear variety. You'd use logistic regression when the problem you are trying to solve is a classification problem, such as determining if an incoming email is spam or not, or if the weather tomorrow will be rain, snow, sun or cloud.
We covered some high level theory and formulas in class, and were given two classification exercises to do. I will discuss the first one below. The second exercise, which I omitted, involves feature mapping and regularization, which warrants a post on its own. If there's enough demand for it, I can post about it later. It's quite cool.
For this exercise, our goal is to determine if a student will be admitted to school based on his exam scores. Because, you know, standardized testing works effectively for everyone. Or does it... Our data set consists of Exam 1 scores, Exam 2 scores, and if the student was admitted (Yes/No). By the way, in Machine Learning-speak, this is an example of supervised learning - we are training our machine based on observed inputs (Exam scores) and outputs (admitted or not). Using this data our machine will learn how to classify students going forward. In future posts we'll cover unsupervised learning. Sounds like a kindergarten teachers' worst nightmare.
To begin, as with any data analysis, let's take a look at our data:
Right. We can clearly see the breakdown of admitted vs. non admitted students. Our task is to develop a "decision boundary" which classifies students into one of the two categories. Then, we can classify incoming students as admitted or not admitted. We develop this boundary by running a logistic regression. We will use the same gradient descent algorithm as we did in our previous discussion. The only difference in this algorithm is that our hypothesis function takes the form of a sigmoid. Other than sounding like this guy's name, this function takes the lovely form of:
If you look carefully, you'll notice that it's bounded by 0 and 1, which is great for categorizing stuff. Or maybe you won't notice that, in which case, life goes on.
By comparison, our linear regression hypothesis function takes the form:
We are simply substituting the sigmoid function for the other, generic function. The rest of the logic is the same.
First off, we calculate the cost function - this is what we are trying to minimize as much as possible. Kind of like stress, inflammation, or going to Applebees (Golden Griddle was a close second, but for their breakfast buffet...). Let's just stop it right now.
Anyway, formally this unwieldy thing looks like this:
Looks nastier than it really is. Here it is in action:
## create a cost function for logistic regression
costFun <- function(X,y,theta){
m <- length(y)
h <- sigmoid(X%*%theta)
cost <- (1/m)*((-t(y) %*% log(h)) - t(1-y)%*%(log(1-h)))
return(cost)
}
Next, we implement the same gradient descent algorithm that we did in the linear regression case. Here's a snippet of the for loop, which updates theta.
for (i in 1:num_iters){
h <- sigmoid(X%*%theta)
grad <- (t(X)%*%(h - y))/m
theta <- theta - alpha * grad
}
return(theta)
This function will spit out our theta, which is the theta that minimizes the cost function. That's how these two things are related. Once we have our theta , we can then plot our decision boundary which follows:
Cool. This clearly delineates between those admitted and those not admitted. However, since it's a straight line (because our features are linear in theta, but you don't need to worry about what that means), it's not perfect. How accurate are we? Based on our training data (that is, the data set we used to calibrate this decision boundary) we achieve 89% accuracy.
paste(mean(y==pred(theta,X))*100,"%",sep="") ## accuracy = 89%
Not bad. Hopefully it's good enough to get admitted (ha!).
Lastly, let's make a few predictions based on individual exam scores. My logic is that if a student has a greater than 50% chance of being admitted, we classify them as admitted, otherwise, not admitted. Simple, yet elegant. Or maybe just simple. Say we have a student with Exam 1 score of 45 (someone obviously got blasted the night before at his fraternity party), and Exam 2 score of 90 (this guy also happens to have immigrant parents who remind him that anything less than being a doctor means disownment). His probability of being admitted is a whopping 89%. That means our classifier will put this remorseful soul into the admitted category. What about that student who aces the first exam with an 85 (sorry my U.S friends, but anything greater than 80 is an A), gets overly confident then bombs the second exam with a 38? Unfortunately, this poor guy only has a 45% chance of being admitted, which, based on our logic, puts him in the big fat "not admitted" category. That's bad news, especially given what we learned about the importance of education in a previous post.
And scene! As always, the full code can be found on my github. I hope that the above example demonstrates that basic classification methods are not overly difficult, yet produce insightful results.
Please let me know if you have any questions, or need further clarification. I love nerding out to data. And if you found this classification analysis interesting, I'd be happy to post about the second exercise we covered in class, which involves creating a decision boundary for this crazy shape!
(Hint - it's not a straight line!)
Cheers!
Please comment on the article here: The Data Game - Ilan Man
