last updated: December 15, 2018 at 5p
COSC 4335: Data Mining in Fall 2018
(Dr. Eick )
Goals of the Data Mining Course
Data mining centers on finding valid, novel, interesting, and potentially useful patterns in data.
It aims at transforming a large amount of data into a well of knowledge. Data mining
has become a very important field in industry as
well as academia. For example, almost 1000 papers are submitted each year to the IEEE International
on Data Mining (ICDM) that will be held in Singapure in
Data mining tools and
suites (for example, see KDnuggets' DM Software
Survey) are used a lot in industry and
in reseach projects.
UH's Data Analysis and Intelligent Systems Lab (UH-DAIS) conducts research in
data sciences, data mining, but also geographic information systems (GIS) and Artificial Intelligence. Finally, having
basic knowledge in data mining and data analytics is a plus when you are looking for a job in
industry and at major US research
institutions, such as the Texas Medical Center in Houston or at Federal Research Labs.
The course covers the most important
data mining techniques and provides background knowledge on how to conduct a
data mining project. It also gives a basic introduction to data analysis. After defining
what knowledge discovery and
data mining is, data mining tasks such classfication, clustering, and association analysis will be discussed in
detail. Basic data analysis techniques, centering on basic visualization techniques and statistics,
to get a better understanding of the data mining task at hand will be covered.
Moreover, techniques how to preprocess a data set for a data mining
task will be introduced.
course projects you will obtain hands on experience in conducting data mining and data analysis projects. Finally,
as R will be
used in most course projects; therefore, participants of the couse will obtain
valuable exprience in using the R statistics, data mining, and visualization
packages and will learn how to write programs in R and how to develop data mining software
on top to R.
A recent 2013 poll Rexer Analytics found that R is currently the most
popular data mining tool: 24% of the respondents use R as their primary
tool, and only 30% of the respondents do not use R at all. Although R is a domain specific language, it's
In summary, having a sound background in data analytics and data mining and knowing R
well will open a lot of job opportunities for you, which, I believe, is a strong
reason to take the course.
Comments concerning this website
If you have any comments
concerning this website, send e-mail
Basic Course Information
Christoph F. Eick
Office hours (573 PGH)
Office Hours: MO 4-4:45p WE 1-2:15p
TA: Romita Banerjee
Office Hours: MO 1-2p WE noon-1p
Office: PGH 313
2018 TA website: Romita's 4335 Website (?!)
class meets: MO/WE 2:30-4p
class room: 201 SEC
Instructor's Travel: September 26, October 8, Nov. 7
TA's Travel: October 5-13
Midterms: October 1 and November 7, 2018
Lectures by 'Guests': Septenber 26, maybe November 13
Makeup class: Mo., December 3, 2:30-4p
COSC 4335 Syllabus for Fall 2018
- P.-N. Tang, M. Steinback, and V. Kumar: Introduction to Data Mining,
- Addison Wesley, 2018.
- Link to
- Jiawei Han and Micheline Kamber, Data Mining: Concepts and
- Morgan Kaufman Publishers, Third Edition, 2011.
- Link to Data Mining Book Home
NIST/SEMATECH e-Handbook of Statistical Methods (good onlne source covering
exploratory data analysis, statistics, modelling and prediction)
News COSC 4335 (Data Mining) Fall 2018
- I enjoyed teaching the course and I already like to wish you a Happy
and Successful Year 2019.
- The letter grades for the Fall 2018 teaching of COSC 4335 have been
posted. This semester we had: A:6, A-:3, B+:4, B:5, B-:3, C+:5, C:2,
C-:3, D+:2, W:3. In general, the student grades of the Fall 2018 teaching
of the course was better than the performance of the previous 2 teachings of
the course; in particular, the performance of the top 60% students was better,
whereas the performance of the bottow 40% students was about the same.
Detailed grade reports you can found in Romita's 4335 webpage.
- This is probably the last offering of COSC 4335: Data Mining; about
65% of its content will be taught in a new Data
Science I 3000-level course, and about 20% of its content will be covered
in a new Data Science II 4000-level course.
- The final exam will not be returned to students: however, you can view your final
exam at the following two dates: Monday, December 17, 11:30a-12:30p and Thursday, January 17,
11:30-12:30p in Romita's TA office.
- Concerns Number Grade Curving: As A in Dr. Eick's number grade scales ranges from 90 to 100, the top number grade assigned to course assigments and
exams usually not higher than 95 with grades 96-100 only being assigned
to exceptional, kind of outlier performances or in cases when there is a
very large spread in the distribution of scores.
Important Dates in Fall 2018
Monday, September 3: Labor Day (no lecture)
Wednesday, September 5: Introduction to R Lab (taught by Romita; please bring your laptop with R installed!)
Tuesday, September 18, 11p: Deadline Assignment1
Monday, September 24: 45 minute review for Midterm1 exam
Wednesday, September 26: Lecture will be given by guest lecturer
Monday, October 1: Midterm1 Exam (Review List 2018 Midterm1 Exam; updated recently)
Wednesday, October 3: R-lab centering on R-programmming (bring labtop)!
Wednesday, November 7: Midterm2 Exam (Review List
Fall 2018 Midterm2 Exam,
Questions for Nov. 5, 2018
Midterm2 exam Review)
Wednesday, November 21: no lecture due to Thanksgiving next day
Monday, December 3, 2:30p: "Makeup" Lecture (last lecture of
Monday, December 10, 2p: Final Exam (Review List
for Dec. 10, 2018 Final Exam, Questions and Solution
Sketches Dec. 3 Review
for the Final Exam) in SEC 201 (our regular class room).
Fall 2018 Assignments
Assignment1: Exploratory Data Analysis for a Portuguese
School Performance Dataset (Individual Assignment)
Assignment2: Similarity Assessment, Clustering, and Using
Clustering to Create Background Knowledge for Classification Tasks (individual project;
Running PAM with Distance Functions)
Assignment3: Making Sense of Data: Learning and Comparing
Classification Models for a Dataset (Individual Project)
Assignment4: Design, Implementation and Comparison of Outlier
Detection Techniques for a Spatial Dataset (Group Project,
This semester, the student performance for Assigment2 and 3 met expectation;
than 45% of the students did not do well at all in Assigment1, and most
groups did very
well in Assignment4. Based on this assessment the Assignment Number Grade Averages
after curving for the four assigments were as follows: 75, 78-79, 78-79,
and 82 approximately.
Course Elements and Their Tentative Weights for 2018
Assignments (4): 46%
Exams (3): 52% (16+16+20%)
Class Attendance: 2%
Rexer Analytics' 2015
Data Science Survey
CrowdFlower's 2016 Data Science
Summary Rexer Analytics' 2017
Data Science Survey (the complete survey is supposed
to appear in October 2018)
COSC 4335 Data Mining: Lecture Notes
I Introduction to Data Mining (COSC 4355 Knowledge Sources (to be discussed Sept. 10, 2018), Part1, Part2,
between Clustering and Classification).
II Exploratory Data Analysis (updated on August 29, 2018; covers chapter 3 from the
the First Edition of the Tan Book (download as this material is not
in the second edition); see
also Interpreting Displays; Some R Data Analysis Functions I; Some R Data Analysis Functions II)
III R (Arko's Short Intro Into R,
Scatter Plot Code, Decision Trees in R,
Some useful code for Assignment1,
Computing Statistical Summaries In the Presense of Missing Value (NA),
and Loops in R,
Directory containing R-code for Project2 (lecture on Feb. 26); moreover,
checkout Romita's 4335 webpage for more R-code, datasets, and slides)
IV Clustering and Similarity Assessment (Introduction and Hierachical Clustering and DBSCAN;
R-scripts demonstrating: K-means/medoids, DBSCAN, More on PAM and using PAM/DBSCAN with dist-objects;
K-Means, HC, and DBSCAN)
V Classification (Introduction to Classification: Basic Concepts and Decision Trees, Overfitting,
Neural Networks, kNN-Classifiers and Support Vector Machines and Ensemble Learning)
VI Outlier Detection (significantly updated on April 7, 2018)
VII Data Science (Introduction to
Data Science, Introduction to Data Science Part2; this part of the course will
discuss the new developments in Data Science
and how Computer Sciencce Departments adapt to its increased importance. It also discusses the relationship between
Data Mining and Data Science, and finally discusses some developments in Data Science,
in particular the importance of Data Storytelling).
VIII Data Preprocessing for Data
IX Association Analysis (Part1, Part2 (not covered in 2018))
Past Exam and Review Solutions
Solution Sketches Midterm1 March 10, 2015
Solution Sketches Midterm2 April 7, 2015
Solution Sketches Final Exam December 10, 2018
Solution Sketches Review1 March 1, 2016
Solution Sketches Review1 Feb. 27, 2018
Solution Sketches Review1 September 24+26, 2018
Solution Sketches Midterm1 March 3, 2016
Solution Sketches Midterm1 March 1, 2018
Solution Sketches Review2 April 5, 2016
Solution Sketches Midterm2 April 7, 2016
Solution Sketches Midterm2 April 5, 2018
Review for Final Exam, May 3, 2016
Solution Sketches oo Review for Final Exam,
April 26, 2018
Solution Sketches Final Exam May 10, 2016
Review2 solution sketches on
November 5, 2018
Students will be responsible for material covered in the
lectures and assigned in the readings. All assignment and
project reports are due at the date specified.
No late submissions
will be accepted after
the due date. This policy will be strictly enforced.
Seveal times during the semester I will check class attendance at randomly
chosen dates, and an attendence score will be computed from how many
of the those lectures you attended.
Translation number to letter grades in 2015:
A:100-90 A-:90-86 B+:86-82 B:82-78 B-:78-74 C+:74-70
C: 70-66 C-:66-62 D+:62-58 D:58-54 D-:54-50 F: 50-0
Only machine written solutions
are accepted (the only exception to this point are figures and complex formulas) in the assignments.
Be aware of the fact that our
only source of information is what you have turned in. If we are not capable to understand your
solution, you will receive a low score.
Moreover, students should not throw away returned assignments or tests.
In contrast to the exam grades where you receive your number grades immediately, the assignment scores, near the end
of the semester, still will be curved and
your curved assignment scores will be ultimately converted into a number grade using a coversion function---the conversion function incorperates
the different weights of the four assignments---and, then this number grade will count 45-48% towards your final course grade.
In general, assignment weights were selected considering amount of work required but also difficulty was considered;
moreover, group projects carry lower weights. Moreover, when
looking at the detailed grade reports, be aware of the fact that number grades of 90 or higher are A's in Dr. Eick's curving.
Students may discuss course material and homeworks, but must take special
care to discern the difference between collaborating in order to increase
understanding of course materials and collaborating on the homework /
itself. We encourage students to help each other understand course
material to clarify the meaning of homework problems or to discuss
problem-solving strategies, but it is not permissible for one
student to help or be helped by another student in working through
assignment problems and in the course project. If, in discussing course materials and problems,
students believe that their like-mindedness from such discussions could be
construed as collaboration on their assignments, students must cite each
other, briefly explaining the extent of their collaboration. Any
assistance that is not given proper citation may be considered a violation
of the Honor Code, and might result in obtaining a grade of F
in the course, and in further prosecution.
COSC 3380 and MATH 3336. You are allowed to take COSC 3380 concurrently with COSC 4335!
Spring 2018 Assignments
Assingment1: Exploratory Data Analysis using R for the Pima Indians Diabetes
Dataset (Individual Project)
Assingment2: Clustering and Using Clustering for Prediction
(Individual Project, second draft, Slides Discussing Assignment2)
Assingment3: Creating and Comparing Different Classification
Models for a Dataset
Assingment4: Developing and Comparing Outlier Detection Techniquesfor a Spatial Dataset
(Group Project; Assignment4 Groups)
Weights for the 4 Assignments: Romita and I met, and selected the following weights for
the assignments: Assignment1: 24%, Assignment2: 36%, Assignment3: 16%, Assignment4: 24%. In
general, we decided to lower the weight of Assigment2, and decided to give a higher weight to
Assignment4, so that students have a chance to make up for some points they lost for Assignment1 and Assignment2.
Evaluation of Assignment performance in Spring 2018: About 25% of the students did a very good
job in Assignment2, and about 40% of the students did quite a poor job; the class performance
for Assignment2 was below expectation; moreover, only 17 students attended the lab that was intended as a preparation for Assignment2: why?? The performance for Assignment1 and Assignment3
met expectations; however, there was a significant spead in scores. Finally, students preformed quite well in Assignment4, and the class performance was above expectation. This fact was
considered when curving the assignment scores and the average number grades for the 4 assignments after
curving where: 77, 74, 78 and 81.
Spring 2018 Assignment Evaluation Questionnare Result Summary
The questionnaire was conducted on April 24, 2018, and Romita created the following
summary of the answers you gave (based on 17 responses):
9 students felt assignment 2 was most difficult and took the most time,
2 students felt assignment 3 was most difficult and took the most time followed by 1 student for assignment 1. Several Students thought that assignment 1 was also quite time consuming,
although not as time consuming as assignment 2; assignment 1 also had a lost of second
place finishes in other categories.
3 students felt that assignment 3 was least difficult and another 3 students felt
assignment 4 was least difficult.
Concerning interestingness, 8 students felt that assignment 3 was most
interesting, followed by 5 students for assignment2 and 2 students that that assignment 1.
All the students agreed that the assignments helped in better understanding of the concepts covered in class. They also liked the fact that the assignments gave them an opportunity to work
with real datasets. 5 students felt they learned most in assignment 3 and 3 students mentioned
they learned the most in assignment 2. Another common comment was that the assignments helped
them in learning and using R.