COSC 6342 Machine Learning

Fall 2016

MW 2:30 - 4:00 PM, C (Roy G. Cullen) 108

Instructor: Arjun Mukherjee


This is a introductory machine learning (ML) course. The course in intended for developing foundations in machine learing and inference principles in applied areas such as natural language processing and data and text mining. The broader goal is to understand how machine learning tasks are carried out in the real world and how to build tools for solving them. Throughout the course, emphasis will be placed on understanding of concepts and tying specific techniques to specific real-world applications through hands-on experience.

Administrative details

Instructor office hours: MW 4:00 - 5 PM, PGH 582

TA: Dainis Boumber(
TA office hours: MW 1-2.30 PGH 301

The course requires basic background in mathematics and sufficient programming skills. If you have taken and did well in one or more of the equivalent courses/topics such as Algorithms, Artificial Intelligence, Numerical methods, or have some background in probability/statistics, it will be helpful. The course however reviews and covers required mathematical and statistical foundations. Sufficient experience for building projects in a high level programming language (e.g., Java, Python) will prove beneficial.

Reading Materials


SI: Statistical Inference, Casella and Berger. Cengage Learning; 2nd edition.
MLPP: Machien Learning - A Probabilistic Perspective. Kevin P. Murphy. MIT Press
WDM: Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data. Bing Liu; Springer, 1st Edition.
FSNLP: Foundations of Statistical Natural Language Processing, Chris Manning and Hinrich Schütze. MIT Press. Cambridge, MA: May 1999. Companion website for book.

Useful reference materials:
IML: Introduction to Machine Learning. Ethem Alpaydin. 2nd Edition. MIT Press.
Online resources (OR) per topic as appearing in the schedule below.
Lecture notes

Course materials (slides, lecture notes, etc.) (You may use 7zip to unpack)

NLP tools (POS Tagger, Chunker, Naive Bayes, etc.) and templates with linked libraries for research project

Assignment Due Dates and Grading

Component Contribution Due date
Midterm 25% 10/10
Final 35% 12/12 (2-5 pm; C108)
Project 40% 12/11

Rules and policies

Late Assignments: Late assignments will not, in general, be accepted. They will never be accepted if the student has not made special arrangements with me at least one day before the assignment is due. It also needs to be a justifiable reason owing to exacting circumstances. If a late assignment is accepted it is subject to a reduction in score as a late penalty.
Cheating: All submitted work (code, homeworks, exams, etc.) must be your own. If evidence of code sharing is found, you may receive an F grade in the course.
Statute of limitations: Grading questions or complaints, will in general not be attended to beyond one week after the item in question has been returned.

Schedule of topics

Topic(s) Resources: Readings, Slides, Lecture notes, Papers, Pointers to useful materials, etc.
Course administrivia, plan, goals, ML Resources
Key tasks in ML
Required readings:
Lecture notes/slides
Chapter 1 MLPP
Chapter 1 IML
Math foundations I:Basics
Probability theory
Conditional probability and independence
Required Readings:
Lecture notes/slides
Chapter 1 SI (Full reading recommended. Focus on topics covered in class and solved examples)
Also refer to X.Zhu's notes on mathematical background
Math foundations II: Random varibales and Distributions
Random variables, density and mass fuctions
Mean, Variance
Common families of distributions
Multiple random variables: joints and marginals
Required Readings:
Lecture notes/slides
Chapter 2 SI (Theorem 2.1.10, 2.2, 2.2.1, 2.2.2, 2.2.3, 2.2.5, 2.3.1, 2.3.2, 2.3.4, and select topics covered in class).
Chapter 3 SI (All sections + worked out examples upto 3.4, and select topics covered in class).
Chapter 4 SI (4.1, 4.1.1, 4.1.2, 4.1.3, 4.1.4, 4.1.5, 4.1.6, 4.1.10, 4.1.11, 4.1.12, 4.2.1, 4.2.2, 4.2.3, 4.2.4, 4.2.5).

Optional Recommended reading/solved examples:
OR03: Notes on Joint, marginals, worked out examples by S.Fan
OR04: Tutorial on joints and marginals by M.Osborne [Contains NLP specific examples]
Supervised Learning I
Basic concepts: Data and features
Decision tress
Naive Bayes
Classifier evaluation
Required Readings: Chapter 3 WDM (3.1, 3.2, 3.3, 3.5, 3.6, 3.7.2)
F. Keller's tutorial on Naiye Bayes + notes of A.Moore for graph view (Slide 8)

Programming resources, tools, libraries for projects and homeworks:
Mallet, LingPipe
Supervised Learning II
Linear Regression
Ridge Regression
Support Vector Machines
Feature selection
Required Readings:
Chapter 7 MLPP (Seelct topics 7.1-7.3.1, 7.5, 7.5.4 covered in class)
Chapter 8 MLPP (upto 8.3.1).
Chapter 3 WDM (3.8, 3.10) Feature selection schemes: [Forman, 2003], [Mukherjee and Liu, 2010]

Programming resources, tools, libraries for projects and homeworks:
SVM: SVMLight, Boosted decision trees/Random forests: JForests, JBoost
Unsupervised Learning
Principal Component Analysis
K-means clustering
Required Readings: Chapter 4 WDM (4.1, 4.2, 4.4, 4.5, 4.6, 4.9).
Optional Recommended Reading: Clustering Analysis (Advanced) [Fred et al.], Modern Methods and Algorithmic Analyses [Müllner et al.]
Programming resources, tools, libraries for projects and homeworks:
Mallet, LingPipe, PU-Learning
Ensemble Learning
Required Readings: Chapter 3 WDM (3.10).
Markov models
Hidden markov model (HMM)
A practical application: Part of speech tagging
Required Readings:
Chapter 9 FSNLP (upto 9.4), Chapter 10 FSNLP (upto 10.2.2)
OR10: (1) Lecture notes/slides by M. Marszalek on "A Tutorial on Hidden Markov Models by Lawrence R. Rabiner", (2) Toy problems by E.Lussier, (3) POS Tagging by Y.Choi

Programming resources, tools, libraries for projects and homeworks:
(1) HMMs and sequence taggers
JAHMM: Implementation of an HMM in Java, Mallet, SVMHMM, CRF++
(2) POS Taggers
OpenNLP, Stanford Parser, (Online version), Illinois Chunker, POS Tagger