50.040 Natural Language Processing

Natural Language Processing (NLP) is an important area within the general field of Artificial Intelligence (AI). Modern NLP models focus on using machine learning algorithms for solving various text processing problems. This course covers fundamental topics within the domain of NLP, including word representations, language modelling, sequence-to-sequence models, attention, and transformer architectures. Students will also explore advanced methods in the modern paradigm of large language models (LLMs), such as pre-training, fine-tuning, and efficient adaptation strategies. The course emphasizes both fundamental algorithms and state-of-the-art techniques. Students will gain hands-on experience in implementing fundamental algorithms and experimenting with advanced deep learning–based NLP techniques, culminating in projects that explore real-world applications of LLMs.

Pre-requisite/Co-requisite

1. 50.007 Machine Learning / 40.319 Statistical & Machine Learning and
2. A good foundation in: 1) programming, 2) design and analysis of algorithms, 3) mathematics including linear algebra, calculus, optimization, probability, and statistics.

Learning objectives

By the end of the course, students will be able to

1. Explain the fundamental tasks within NLP
2. Explain possible algorithms as solutions to NLP tasks
3. Implement the algorithms used for various NLP tasks
4. Design novel algorithms for solving new NLP tasks, and use existing NLP technologies for solving real problems

Measurable outcomes

1. Explain the major tasks within NLP that involve supervised structured prediction
2. Explain the major tasks within NLP that involve unsupervised learning
3. Apply the relevant models that need to be used for each task
4. Apply the major guiding principles when choosing a model for a specific task within NLP
5. Decide when to and when not to use neural network based or deep learning methods for a specific task within NLP
6. Design and implement fundamental algorithms used for various NLP tasks
7. Analyse the time complexity involved for a specific NLP algorithm
8. Evaluate the performance of an NLP model based on certain evaluation metrics on standard datasets

Topics covered

• Introduction to NLP Tasks
• Recap on Machine Learning & Neural Networks
• Word Vectors
• Language Models & RNNs
• Seq2Seq Models & Attention
• Transformer Models
• Pre-training
• Post-training (SFT/RLHF)
• Efficient Adaptation (Prompting/LoRA)
• Benchmarking & Evaluation
• LLM Agents
• Advanced Topics (e.g., multilinguality)

Textbook(s) and/or other required material

Required to read some relevant chapters from:

• Dan Jurafsky and James H. Martin, Speech and Language Processing (3rd ed. draft), Aug 24 2025 release version
• Yoav Goldberg, Neural Network Methods for Natural Language Processing, 2017

Course instructor(s)

Prof Zhang Wenxuan