We demonstrate a dataset that causes many good machine learning algorithms to horribly overfit. The example is designed to imitate a common situation found in predictive analytic natural language processing. In this type of application you are often building a model using many rare text features. The rare text features are often nearly unique k-grams […] Related posts: Don’t use correlation to track prediction performance Generalized linear models for predicting…
Today, we celebrate the would-be 85th birthday of Martin Luther King, Jr., a man remembered for pioneering the civil rights movement through his courage, moral leadership, and oratory prowess. This post focuses on his most famous speech, I Have a Drea...
One day, while I was walking around Cambridge, I had a random thought -- how do the characters on the Simpsons feel about each other? It doesn't take long to figure out how Homer feels about Flanders (hint: he doesn't always like him), or how Burns feels about everyone, but how does Marge feel about Bart? How does Flanders feel about Homer? I then realized that I work with algorithms…
Update: you can find the next post in this series here. In a previous post, I looked at transcripts of Simpsons episodes and tried to figure out which character was speaking which line. This worked decently, but it wasn't great. It gave us memorable scenes like this one: Homer : D'oh! A deer! A female deer. Marge : Son, you're okay! Bart : Dad, I can't let you sell him.…
Introduction This will serve as an introduction to natural language processing. I adapted it from slides for a recent talk at Boston Python. We will go from tokenization to feature extraction to creating a model using a machine learning algorithm. The goal is to provide a reasonable baseline on top of which more complex natural language processing can be done, and provide a good introduction to the material. The examples…
I just gave a talk at Boston Python about natural language processing in general, and edX ease and discern in specific. You can find the presentation source here, and the web version of it here. There is a video of it here. Nelle Varoquaux and Michael Selik also had interesting talks in the same video above, recommend checking them out.
I’m very pleased to announce the release of qdap 0.2.2 This is the third installment of the qdap package available at CRAN. The qdap package automates many of the tasks associated with quantitative discourse analysis of transcripts containing discourse, including … Continue reading →
I’m very pleased to announce the release of qdap 0.2.2 This is the third installment of the qdap package available at CRAN. The qdap package automates many of the tasks associated with quantitative discourse analysis of transcripts containing discourse, including … Continue reading →
Introduction I recently posted about using the Wikileaks cable corpus to find word use patterns, both over time, and in secret cables vs unclassified cables. I received a lot of good suggestions for further topics to pursue with the corpus, and probabl...
Introduction I recently posted about using the Wikileaks cable corpus to find word use patterns, both over time, and in secret cables vs unclassified cables. I received a lot of good suggestions for further topics to pursue with the corpus, Continue reading →
