CS690: Machine Learning
Spring 2012

---

Time and Location: Tue, Thu 1:10pm – 3:00pm, ARC 101
Instructor: Razvan Bunescu
Office: Stocker 337
Office Hours: Mon 9:10 – 10am, Wed 11:10am – 12pm, Fri 11:10am – 12pm, or by email appointment
Email: bunescu @ ohio edu

Textbook:

Pattern Recognition and Machine Learning by Christopher Bishop. Springer, 2007.

Recommended Supplementary Texts:

Machine Learning by Tom Mitchell. McGraw Hill, 1997

Pattern Classification by Richard O. Duda, Peter E. Hart, & David G. Stork. Wiley-Interscience, 2001

The Elements of Statistical Learning: Data Mining, Inference, and Prediction by T. Hastie, R. Tibshirani, & J. H. Friedman. Springer Verlag, 2009

Course description:
Machine Learning is concerned with the design and analysis of algorithms that enable computers to automatically find patterns in the data. This introductory course will give an overview of the main concepts, techniques and algorithms that are relevant for the theory and practice of machine learning. The course will cover the fundamental topics of classification, regression and clustering, starting with simple learning models such as perceptrons, decision trees and logistic regression, and ending with more advanced models including Support Vector Machines, Conditional Random Fields and Bayesian networks. The description of the formal properties of the algorithms will be supplemented with motivating applications in a wide range of areas including natural language processing, computer vision, bioinformatics and music analysis.

Prerequisites:
The students are expected to be comfortable with programming in C/C++/Java, and to exhibit a basic level of mathematical dexterity. Relevant background material in linear algebra, probability theory and information theory will be made available during the course.

Lecture notes:

1. Syllabus & Introduction
2. Regression with Linear Models
3. Fisher Linear Discriminant
4. Perceptrons and Kernels
   • Large Margin Classification Using the Perceptron Algorithm, Yoav Freund and Robert E. Schapire, Machine Learning 1999
5. Support Vector Machines
   • Support Vector Machines [Trends and Controversies] , Marti Hearst, Susan Dumais, Edgar Osuna, John Platt, Bernhard Scholkopf, IEEE Intelligent Systems, 13(4), 1998
   • A Tutorial on Support Vector Machines for Pattern Recognition, Christopher J. C. Burges, Data Mining and Knowledge Discovery 1998
6. Nearest Neighbor Methods
7. Feature Selection
8. Decision Trees
9. Naive Bayes
   • Naive Bayes and Logistic Regression, new chapter in Tom Mitchell, Machine Learning, 2005
10. Logistic Regression
    • A Maximum Entropy Approach to Natural Language Processing, Adam Berger, Vincent Della Pietra and Stephen A. Della Pietra, Computational Linguistics, 1996
11. Hidden Markov Models
    • Markov Models, chapter 9 in Chris Manning and Hinrich Schutze, Foundations of Statistical Natural Language Processing, MIT Press. Cambridge, MA, May 1999
    • A tutorial on Hidden Markov Models and selected applications in speech recognition, Lawrence R. Rabiner, Proceedings of the IEEE 77 (2), 1989
12. Conditional Random Fields
    • An Introduction to Conditional Random Fields for Relational Learning, Charles Sutton and Andrew McCallum, 2007
13. PCA, Clustering

Homework Assignments:

• Assignment 1
  • Data
• Assignment 2
• Assignment 3
• Assignment 4

Final Project:

• Project suggestions
• Project report guidelines

Other online reading materials:

• James H. Martin's Introduction to probabilities
• Jason Eisner's equestrian Introduction to probabilities
• Inderjit Dhillon's Linear Algebra Background
• Strang's Video Lectures on Linear Algebra
• Convex Optimization, Stephen Boyd and Lieven Vandenberghe, Cambridge University Press 2004
• New ranking algorithms for parsing and tagging: Kernels over Discrete Structures, and the Voted Perceptron, Michael Collins and Nigel Duffy, ACL 2002
• Convolution Kernels for Natural Language, Michael Collins and Nigel Duffy, NIPS 2001
• Text Classification using String Kernels, Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cristianini, Chris Watkins and Bernhard Scholkopf, JMLR 2002
• Optimizing Search Engines Using Clickthrough Data, Thorsten Joachims, KDD 2002

Machine learning software:

• Weka Data Mining Software in Java
• SVM^light Implementation of SVMs in C
• LIBSVM Implementation of SVMs in C++ and Java
• MALLET Java implementations of logistic regression, HMMs, linear chain CRFs, and other ML models.
• SVM applet demonstrating SVMs, by Hakan Serce
• k-Nearest Neighbor short animated video, by Antal van den Bosch