• Course Info
• Lectures
• Exams
• Assignments
• Grading
• Links

Announcements

1. (Jan 30) Course page is online.
2. (Feb 3) Slides for Introduction to Pattern Recognition are available.
3. (Feb 10) Slides for Bayesian Decision Theory are available.
4. (Feb 16) First part of the slides for Parametric Models is available.
5. (Feb 23) Second part of the slides for Parametric Models is available.
6. (Mar 2) Third part of the slides for Parametric Models is available.
7. (Mar 3) Homework assignment 1 is available.
8. (Mar 9) Fourth part of the slides for Parametric Models is available.
9. (Mar 15) Slides for Non-parametric Methods are available.
10. (Mar 23) Slides for Feature Reduction and Selection are available.
11. (Mar 31) Homework assignment 2 is available.
12. (Apr 13) First part of the slides for Non-Bayesian Classifiers is available.
13. (Apr 13) Second part of the slides for Non-Bayesian Classifiers is available.
14. (Apr 13) Third part of the slides for Non-Bayesian Classifiers is available.
15. (Apr 28) Slides for Unsupervised Learning and Clustering are available.
16. (Apr 30) Homework assignment 3 is available.
17. (May 4) Slides for Algorithm-Independent Learning Issues are available.
18. (May 11) Slides for Structural and Syntactic Pattern Recognition are available.

Personnel

Instructor:	Selim Aksoy
Office:	EA 423
Email:

Course Information

Schedule:	Wed 13:40-17:30 (EB 201)
Office hours:	Wed 10:40-12:30 (EA 423)
Prerequisites:	Probability theory, statistics, linear algebra

Texts

• R. O. Duda, P. E. Hart, D. G. Stork, Pattern Classification, 2nd edition, John Wiley & Sons, Inc., 2000.
• S. Theodoridis, K. Koutroumbas, Pattern Recognition, 3rd edition, Academic Press, 2006.
• C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• A. Webb, Statistical Pattern Recognition, 2nd edition, John Wiley & Sons, Inc., 2002.
• T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning, Springer, 2003.
• K. Fukunaga, Introduction to Statistical Pattern Recognition, Academic Press, 1990.
• R. Schalkoff, Pattern Recognition: Statistical, Structural and Neural Approaches, John Wiley & Sons, Inc., 1992.
• A. K. Jain, R. C. Dubes, Algorithms for Clustering Data, Prentice Hall, 1988.

Lecture Schedule

Chapters	Contents
Introduction to Pattern Recognition [ Slides ] (Feb 3)	Topics: Pattern recognition systems The design cycle An example Readings: DHS Ch 1, Appendix A.1-A.2, A.4-A.5 References: A. K. Jain, R. P. W. Duin, J. Mao, "Statistical Pattern Recognition: A Review" (local copy), IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(1):4-37, January 2000. C. M. Grinstead, J. L. Snell, Introduction to Probability (local copy), American Mathematical Society, 1997. R. C. Gonzales, R. E. Woods, "Brief Review of Matrices, Vectors, Probability and Random Variables" (local copy), 2002.
Bayesian Decision Theory [ Slides ] (Feb 10)	Topics: Modeling using continuous and discrete features Discriminant functions The Gaussian density Error estimation Readings: DHS Ch 2.1-2.9 References: Receiver Operating Characteristic Curves: An Introduction T. Fawcett, "An Introduction to ROC Analysis" (local copy), Pattern Recognition Letters, 27(8):861-874, June 2006.
Parametric Models [ Slides: Part 1 | Part 2 | Part 3 | Part 4 ] (Feb 17, 24, Mar 3, 10)	Topics: Maximum-likelihood estimation Bayesian estimation Expectation-Maximization and mixture density estimation Hidden Markov Models Bayesian Belief Networks Readings: DHS Ch 3.1-3.5, 3.9, 10.2-10.4, 3.10, 2.11 References: J. A. Bilmes, "A Gentle Tutorial on the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models" (local copy), Technical Report TR-97-021, International Computer Science Institute, University of California, Berkeley, April 1998. G. J. McLachlan, T. Krishnan, The EM Algorithm and Extensions, John Wiley & Sons, Inc., 1997. L. R. Rabiner, "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition" (local copy), Proceedings of the IEEE, 77(2):257-286, February 1989. K. P. Murphy, "A Brief Introduction to Graphical Models and Bayesian Networks" (local copy), May 2001. E. Charniak, "Bayesian Networks Without Tears" (local copy), AI Magazine, 12(4):50-63, 1991. D. Heckerman, "A Tutorial on Learning With Bayesian Networks" (local copy), Technical Report MSR-TR-95-06, Microsoft Research, March 1995. W. L. Buntine, "Operations for Learning with Graphical Models" (local copy), Journal of Artificial Intelligence Research, 2:159-225, 1994.
Non-parametric Methods [ Slides ] (Mar 17)	Topics: Density estimation Histogram-based estimation Parzen windows estimation Nearest neighbor estimation Readings: DHS Ch 4.1-4.4
Feature Reduction and Selection [ Slides ] (Mar 24)	Topics: Problems of dimensionality Component analysis Principal components analysis (PCA) Linear discriminant analysis (LDA) Feature selection Readings: DHS Ch 3.7-3.8, 10.13-10.14 References: L. K. Saul, S. T. Roweis, "Think Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifolds" (local copy), Journal of Machine Learning Research, 4:119-155, 2003. A. Hyvarinen, "Survey on Independent Component Analysis" (local copy), Neural Computing Surveys, 2:94-128, 1999. A. Hyvarinen, E. Oja, "Independent Component Analysis: Algorithms and Applications" (local copy), Neural Networks, 13(4-5):411-430, 2000. A. K. Jain, D. Zongker, "Feature Selection: Evaluation, Application, and Small Sample Performance" (local copy), IEEE Trans. on Pattern Analysis and Machine Intelligence, 19(2):153-158, February 1997. Feature selection demonstrations
Non-Bayesian Classifiers [ Slides: Part 1 | Part 2 | Part 3 ] (Apr 14, 21)	Topics: k-nearest neighbor classifier Linear discriminant functions Support vector machines Neural networks Decision trees Readings: DHS Ch 4.5-4.6, 5.1-5.3, 5.11, 6.1-6.3, 8.1-8.3 References: M. A. Hearst, "Support Vector Machines" (local copy), IEEE Intelligent Systems, 13(4):18-28, July 1998. B. Scholkopf, C. Burges, A. J. Smola, "Introduction to Support Vector Learning" (local copy), in Advances in Kernel Methods - Support Vector Learning, MIT Press, 1999. C. J. C. Burges, "A Tutorial on Support Vector Machines for Pattern Recognition" (local copy), Data Mining and Knowledge Discovery, 2(2):121-167, June 1998. A. K. Jain, J. Mao, K. M. Mohiuddin, "Artificial neural networks: a tutorial" (local copy), IEEE Computer, 29(3):31-44, March 1996. P. F. Baldi, K. Hornik, "Learning in linear neural networks: a survey" (local copy), IEEE Transactions on Neural Networks, 6(4):837-858, July 1995. G. P. Zhang, "Neural networks for classification: a survey" (local copy), IEEE Transactions on Systems, Man and Cybernetics, Part C: Applications and Reviews, 30(4):451-462, November 2000. T. M. Therneau, E. J. Atkinson, "An Introduction to Recursive Partitioning Using the RPART Routines" (local copy), Technical Report 61, Department of Health Science Research, Mayo Clinic, Rochester, Minnesota, September 1997.
Unsupervised Learning and Clustering [ Slides ] (Apr 28)	Topics: Criterion functions for clustering k-means clustering Hierarchical clustering Graph-theoretic clustering Cluster validity Readings: DHS Ch 10.1, 10.6-10.7, 10.9-10.10, 10.12 References: A. K. Jain, R. C. Dubes, "Algorithms for Clustering Data" (local copy), Prentice Hall, 1988 (334 pages, 40Mb!). A. K. Jain, M. N. Murty, P. J. Flynn, "Data Clustering: A Review" (local copy), ACM Computing Surveys, 31(3):264-323, September 1999. P. Berkhin, "Survey of Clustering Data Mining Techniques" (local copy), Technical Report, Accrue Software, San Jose, CA, 2002. S. Aksoy, R. M. Haralick, "Graph-Theoretic Clustering for Image Grouping and Retrieval" (also as technical report), in IEEE International Conference on Computer Vision and Pattern Recognition, 1:63-68, Fort Collins, CO, June 1999. A. K. Jain, A. Topchy, M. H. C. Law, J. M. Buhmann, "Landscape of Clustering Algorithms" (local copy), in IAPR International Conference on Pattern Recognition, 1:260-263, Cambridge, UK, August 2004. U. von Luxburg, "A Tutorial on Spectral Clustering" (local copy), Statistics and Computing, 17(4):395-416, December 2007. A. K. Jain, "Data Clustering: 50 Years Beyond K-means" (local copy), King-Sun Fu Prize Lecture at ICPR 2008.
Algorithm-Independent Learning Issues [ Slides ] (May 5)	Topics: No Free Lunch Theorem Resampling for classifier design Comparing classifiers Combining classifiers Readings: DHS Ch 9.1-9.2, 9.5-9.7 References: J. Rissanen, "Modeling by Shortest Data Description," Automatica, 14:465-471, 1978. J. Rissanen, "A universal prior for integers and estimation by minimum description length," The Annals of Statistics, 11(2):416-431, 1983. J. Kittler, M. Hatef, R. P. W. Duin, J. Matas, "On Combining Classifiers" (local copy), IEEE Trans. on Pattern Analysis and Machine Intelligence, 20(3):226-239, March 1998. R. P. W. Duin, D. M. J. Tax, "Experiments with Classifier Combining Rules" (local copy), in First International Workshop on Multiple Classifier Systems, Lecture Notes in Computer Science, 1857:16-29, Springer, 2000. L. Breiman, "Bagging Predictors" (local copy), Machine Learning, 24(2):123-140, August 1996. R. E. Schapire, "The Boosting Approach to Machine Learning: An Overview" (local copy), in Proceedings of MSRI Workshop on Nonlinear Estimation and Classification, 2002.
Structural and Syntactic Pattern Recognition [ Slides ] (May 12)	Topics: Recognition with strings Grammatical methods Graph-theoretic methods Readings: DHS Ch 8.5-8.6 References: K.-S. Fu, T. L. Booth, "Grammatical Inference: Introduction and Survey - Part I," IEEE Trans. on Pattern Analysis and Machine Intelligence, 8(3):343-359, 1986. K.-S. Fu, T. L. Booth, "Grammatical Inference: Introduction and Survey - Part II," IEEE Trans. on Pattern Analysis and Machine Intelligence, 8(3):360-375, 1986. L. G. Shapiro, R. M. Haralick, "A Metric for Comparing Relational Descriptions," IEEE Trans. on Pattern Analysis and Machine Intelligence, 7:90-94, 1985. S. Dance, T. Caelli, Z.-Q. Liu, Picture Interpretation: A Symbolic Approach, World Scientific, 1995. B. T. Messmer, H. Bunke, "Efficient Subgraph Isomorphism Detection: A Decomposition Approach" (local copy), IEEE Trans. on Knowledge and Data Engineering, 12(2):307-323, March/April 2000. W. J. Christmas, J. Kittler, M. Petrou, "Structural Matching in Computer Vision Using Probabilistic Relaxation" (local copy), IEEE Trans. on Pattern Analysis and Machine Intelligence, 17(8):749-764, August 1995. R. Myers, R. C. Wilson, E. R. Hancock, "Bayesian Graph Edit Distance" (local copy), IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(6):628-635, June 2000. D. Conte, P. Foggia, C. Sansone, M. Vento, "Thirty years of graph matching in pattern recognition" (local copy), International Journal of Pattern Recognition and Artificial Intelligence, 18(3):265-298, May 2004. R. P. W. Duin, "Structural Inference of Sensor-Based Measurements" (local copy), Pierre Devijver Award Lecture at S+SSPR 2006. R. P. W. Duin, "Representation Related Problems in Pattern Recognition" (local copy), Invited talk at Joint Meeting of the British Statistical Society and the Royal Classification Society, 2006. H. Bunke, "Graph Classification Based on Dissimilarity Space Embedding" (local copy), Invited talk at S+SSPR 2008.

Exams

• Midterm exam will be held at EB 201 at 13:40-15:30 (during class hours) on March 31, 2010. The exam will cover all topics from the beginning of the semester until the end of the non-parametric methods chapter.
• Final exam will be held at BZ 05 at 10:00-12:00 on May 18, 2010. The exam will cover all topics from the beginning of the semester until the end of the structural pattern recognition chapter.

Assignments

1. Homework assignment 1 (Due: March 24, 2010 as hardcopy in the class)
2. Homework assignment 2 (Due: April 21, 2010 as online submission)
3. Homework assignment 3 (Due: May 23, 2010 as online submission)

Late submission policy: Unless you make prior arrangements with me (before the due date), no late homework will be accepted.

Grading Policy

Homework:	50%
Midterm exam:	20%
Final exam:	25%
Class participation:	5%

Related Links

• Previous semesters for CS 551
  • Spring 2009
  • Spring 2008
  • Spring 2007
  • Spring 2006
  • Spring 2005
• Duda, Hart, Stork book
  • Book's website
  • Make sure you check the errata for the particular printing you have.
• Webb book
  • Book's website
• Bishop book
  • Book's website
• Theodoridis and Koutroumbas book
  • Book's website
• Hastie, Tibshirani, Friedman book
  • Book's website
• Software resources
  • PRTools by the Delft Pattern Recognition Group (in Matlab) (local copy)
  • Netlab Neural Network Software (in Matlab) (local copy of software and its documentation)
  • Weka Data Mining Software (in Java)
  • Bayes Net Toolbox (in Matlab)
  • Hidden Markov Model Toolbox (in Matlab)
  • SVMlight - SVM training package (in C)
  • Sequential Minimal Optimization algorithm for SVM training
  • LIBSVM - A Library for SVM (in C++ and Java, with interfaces for additional languages)
  • Numerical Recipes (in C)
  • Software resources from Pattern Recognition Information web site
  • Software resources from Kevin Murphy's web site
  • Software resources from Kernel Machines web site
  • Software resources from Kernel Methods web site
  • Software resources from American Association for Artificial Intelligence web site
  • StatLib
  • Mathtools.net Technical Computing (in Matlab, C/C++, Java)
• Matlab tutorials
  • Matlab Primer v2
  • Matlab Tutorial by Indiana University
  • Matlab Tutorial by the University of New Hampshire
  • Matlab Tutorial by MIT
  • MathWorks - Matlab Tutorial
  • MathWorks - Student Center
  • MathWorks - HTML Documentation
• Data resources
  • Pattern recognition related archives
  • Computer vision test images
  • UCI Machine Learning Repository
  • Labeled databases for object detection
  • Image database from the University of Washington
  • Texture database from the University of Oulu
  • Document database from the University of Oulu
  • Other databases from the University of Oulu
  • Image databases from CMU Vision and Autonomous Systems Center
  • Various other datasets from the University of Washington
  • Face databases from CMU
  • Face databases from MIT
  • Another page for face databases
  • MNIST Database of handwritten digits
  • Shape database from Brown University
  • Reuters-21578 Text Categorization Collection
  • NIST Scientific and Technical Databases
  • RISC: Repository of Information on Semi-supervised Clustering
• Others
  • Pattern Recognition Information
  • International Association for Pattern Recognition (IAPR)
  • IEEE Computer Society Technical Committee on Pattern Analysis and Machine Intelligence (PAMI)
  • IAPR Education Committee Resources (Tutorials, data sets, codes, etc.)
  • IAPR Technical Committee 1 on Statistical Techniques in Pattern Recognition
  • IAPR Technical Committee 2 on Structural and Syntactical Pattern Recognition
  • MathWorld (an online encyclopedia of mathematical resources)
  • International Society for Bayesian Analysis
  • Statistical Learning/Pattern Recognition Glossary
  • Statistical Data Mining Tutorials
  • Kernel Machines
  • Learning with Kernels
  • Engineering Statistics Handbook
  • Introductory Statistics: Concepts, Models, and Applications
  • The Probability Web

Please send questions/suggestions to .

URI for this page is http://www.cs.bilkent.edu.tr/~saksoy/courses/cs551-Spring2010/index.html.
Last updated on May 11, 2010.