ECE 520.447 Introduction to Information Theory and Coding
Mon Wed Fri 1:30-2:20 PM in Hackerman 320 (Occasionally 1:30-2:45)
Information Theory addresses some fundamental questions about systems that store or communicate data.
- Most electronic communication channels are noisy. Telephone and radio channels are more obviously so, but even coaxial cables and fiber optic links will occasionally ``flip a 0 to a 1,'' especially if we try to push data through them at ever increasing rates. How can one send data in an essentially error-free manner over such noisy communication channels? What is the ultimate transmission rate that a channel can support?
- Many electronic storage systems employ some form of data compression. Data compression is also used for transmission in order to reduce the amount of data that needs to be transmitted. What is the ultimate compression factor that can be achieved while guaranteeing perfect recovery of the compressed data? What more can be achieved if one is willing to tolerate some prespecified small distortion of the recovered data (as in images or video)?
Information Theory, due to the nature of its subject matter, also makes fundamental contributions to statistical physics (thermodynamics), computer science (string complexity), economics (optimal portfolios), probability theory (large deviations) and statistics (Fisher information, hypothesis testing). This makes Information Theory a useful tool for students of other disciplines.
MyLastName at jhu dot edu
Hackerman Hall Room 325
By appointment: W 3:00-4:00. Please e-mail Ms. Ruth Scally [rscally1] in advance.
Congyuan Yang and Dung Tran
[yangcy dot ee at gmail dot com] and [FirstName.n.Lastname at jhu dot edu] respectively
Hackerman 322 and Barton 320 respectively
Monday 3:00-4:00 PM and Wednesday 10:00-11:00 AM respectively
Text and Reference BooksThe Fall 2002 offering of the course will utilize the book by Cover and Thomas as the primary textbook, with some additional reading material from Csiszár and Körner.
- T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd Edition, John Wiley & Sons, 2006. (Available from Barnes & Noble, Amazon and other sellers)
- I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems, Akadémiai Kiadó, Budapest, 1981.
- S. Lin and D. J. Costello, Jr., Error Control Coding: Fundamentals and Applications, Prentice Hall, Englewood Cliffs, NJ, 1983.
Homeworks, Exams and Class NotesEvaluation will be based on a combination of homework assignments (20%), two midterm examinations (20% each, on March 14 and April 30), and a comprehensive final examination (40%) on May 14, 9AM -12 Noon.
Homework assignments and notes will be posted here throughout the semester. Check here frequently!
An Important Note on Academic Ethics:The strength of the university depends on academic and personal integrity. In this course, you must be honest and truthful. Ethical violations include cheating on exams, plagiarism, reuse of assignments, improper use of the Internet and electronic devices, unauthorized collaboration, alteration of graded assignments, forgery and falsification, lying, facilitating academic dishonesty, and unfair competition. Report any violations you witness to the instructor. You may consult the associate dean of students and/or the chairman of the Ethics Board beforehand. See the guide on Academic Ethics for Undergraduates and the Ethics Board web site for more information.
Old HomeworksHomework assignments from the Fall 2002 offering of the course are provided below as examples. These or similar homeworks will be assigned in Spring 2014. See above for the actual homeworks for this semester.
- Homework Assignment #1: Due Sep 16, 2002 ( pdf, ps)
- Homework Assignment #2: Due Sep 23, 2002 ( pdf, ps)
- Homework Assignment #3: Due Sep 30, 2002 ( pdf, ps)
- Homework Assignment #4: Due Oct 8, 2002 ( pdf, ps)
- Homework Assignment #5: Due Oct 15, 2002 ( pdf, ps)
- Midterm Examination #1: Given Oct 16, 2002 ( pdf, ps)
- Homework Assignment #6: Due Oct 22, 2002 ( pdf, ps)
- Homework Assignment #7: Due Oct 29, 2002 ( pdf, ps)
- Reading Assignment: Due Oct 29, 2002 ( pdf, ps)
- Homework Assignment #8: Due Nov 12, 2002 ( pdf, ps)
- Homework Assignment #9: Due Nov 20, 2002 ( pdf, ps)
- Midterm Examination #2: Given Nov 25, 2002 ( pdf, ps)
- Homework Assignment #10: Due Dec 3, 2002 ( pdf, ps)
- Reading Assignment: Due Dec 9, 2002 ( pdf, ps)
- (Optional) Homework Assignment #11: Due Dec 11, 2002 ( pdf, ps)
- Fall 2002 Midterm Exam #1 ( pdf, ps)
- Fall 2002 Midterm Exam #2 ( pdf, ps)
- Fall 2002 Final Exam ( pdf, ps)
- Fall 2001 Midterm Exam #1 ( pdf, ps)
- Fall 2001 Midterm Exam #2 ( pdf, ps)
- Fall 2001 Final Exam ( pdf, ps)
- Spring 2000 Midterm Exam #1 ( pdf, ps)
- Spring 2000 Midterm Exam #2 ( pdf, ps)
- Spring 2000 Final Exam ( pdf, ps)
Old NotesSome notes on Joint Typicality ( pdf, ps)
Some notes on Rate Distortion ( pdf, ps)
Some notes on Relative Entropy ( pdf, ps)
Some notes on the Method of Types (large pdf file, or even larger ps file)
Shannon, Claude E. "A Mathematical Theory of Communication." Bell System Technical Journal 27 (July and October 1948): 379-423 (Part I), 623-656 (Part II).
These seminal papers are available in several forms (see bibliographic notes (PDF)):
PDF version of original papers (PDF)#(With corrections but without Shannon's 1949 modifications.)
Shannon, Claude E., and Warren Weaver. The Mathematical Theory of Communication. Urbana, IL: University of Illinois Press, 1998. ISBN: 9780252725463. (First edition published in 1949, incorporating a number of modifications and corrections by Shannon.)
Shannon, Claude E. "A Mathematical Theory of Communication." 50th Anniversary Edition, printed for the 1998 IEEE International Symposium on Information Theory, MIT. Cambridge, UK: Cambridge University Press, August 16-21, 1998. (Based on 1949 book, with corrections.)
Reprinted in Key Papers in the Development of Information Theory. Edited by D. Slepian. New York, NY: IEEE Press, 1974. ISBN: 9780879420284.
Reprinted in Claude Elwood Shannon: Collected Papers. Edited by N. J. A. Sloane and A. D. Wyner. New York, NY: IEEE Press, 1993. ISBN: 9780780304345.
Bhatti, Saleem N. Channel Capacity (archived). One of several excellent succinct expositions of channel capacity for symmetric binary channels.
ISO/OSI Network Model (archived)
Hedrick, Charles L. Introduction to the Internet Protocols. October 3, 1988.
IP version 6, and related specifications
Claude E. Shannon (1916-2001); obituary, Tech Talk, February 28, 2001.
There are many excellent texts on communications, most of which assume a familiarity with mathematics beyond introductory calculus. Almost all cover Shannon's work, and some also discuss feedback error control techniques.
Cover, Thomas M., and Joy A. Thomas. Elements of Information Theory. New York, NY: John Wiley & Sons, Inc., 2006. ISBN: 9780471241959.
Aimed at university seniors and first-year graduate students. One of several excellent books of that era. Professor Cover, at Stanford University, is one of the leaders in Information Theory.
Hambley, Allan R. An Introduction to Communication Systems. New York, NY: W. H. Freeman, 1989. ISBN: 9780716781844.
Discussion of various types of error control coding, including FEC (Forward Error Correction) and ARQ (Automatic Repeat Query) techniques, pp. 427-479.
Gallager, Robert G. Information Theory and Reliable Communications. New York, NY: John Wiley & Sons, Inc., 1968. ISBN: 9780471290483.
One of the early textbooks, designed for first-year graduate students, by one of the pioneers in communications, an MIT faculty member. He was later awarded the IEEE Medal of Honor, its most prestigious award.
Haykin, Simon. Communication Systems. 4th ed. New York, NY: John Wiley & Sons, Inc., 2000. ISBN: 9780471178699.
Pierce, John R. An Introduction to Information Theory: Symbols, Signals, and Noise. 2nd ed. New York, NY: Dover Publications, Inc., 1980. ISBN: 9780486240619.
Mostly nonmathematical, by one of the nation's great scientific contributors at AT&T Bell Laboratories, who was also interested in reaching a general audience. He was later on the faculty at Caltech. One of his interesting sideline activities was writing science fiction stories under the pen name J. J. Coupling. He died April 2, 2002 at the age of 92.