Deep Learning

From the perceptron to transformers — the architectures, the mathematics that makes them trainable, and the papers that introduced them.

आचार्यात् पादमादत्ते पादं शिष्यः स्वमेधया।
पादं सब्रह्मचारिभ्यः पादं कालक्रमेण च॥

ācāryāt pādam ādatte pādaṃ śiṣyaḥ svamedhayā / pādaṃ sabrahmacāribhyaḥ pādaṃ kālakrameṇa ca

A student gains one quarter of knowledge from the teacher, one quarter through their own intellect, one quarter from fellow students, and one quarter in the course of time.

Course information

Instructor

Jitin Singla · jsingla@bt.iitr.ac.in

Lectures

Mon, Wed & Fri · 3:00–3:55 PM
GB-002

Tutorial

Wed · 12:05–1:00 PM
Venue to be announced

Office hours

To be announced

Discussion

Piazza

Objectives

An introduction to deep learning, building a solid foundation in one of the most transformative areas of machine learning.

Establish mathematical foundations. The core principles underpinning deep learning models and algorithms.
Explore key architectures. The networks that have become standard in image analysis, language modelling and beyond.
Survey recent advances. Latest breakthroughs and trends, setting the stage for research in advanced topics.

Prerequisites

A working understanding of linear algebra and probability. Revision material: Linear algebra and Probability.
Python programming
A foundational course in AI or ML, covering: loss functions (MSE, negative log likelihood, cross entropy); optimization (GD, SGD, constrained convex optimization with Lagrangians); supervised methods (linear and logistic regression, perceptron, decision trees, random forests, SVMs, naive Bayes, k-NN); unsupervised methods (k-means, GMMs, EM, PCA).
Chapters 1–5 of Deep Learning by Ian Goodfellow

Course content

Introduction and motivation
Multi-layered perceptrons, neural networks, backpropagation
Loss functions, regularization: L1–L2 norms
Optimization: SGD, RMSProp, Adam, Adagrad
Dropout, batch and layer normalization
Convolutional networks (CNNs), residual networks
Recurrent neural networks (RNNs), LSTMs
Attention: Bahdanau attention, transformers
Word2Vec embeddings
Language modelling, BERT
GPT-1: improving language understanding by generative pre-training
GPT-2: language models as unsupervised multitask learners
GPT-3: language models are few-shot learners
Other LLMs: LLaMA, Mistral, Gemini, DeepSeek
Vision language models: ViT, CLIP

Schedule

#	Topic	Essential reading	Homework
1	Course info and outline	Chapters 1–5, Deep Learning	—
2	Basics of the perceptron, feedforward network notation	Chapter 6, Deep Learning	—
3	XOR network and hidden layer transformation	Chapter 6	HW-1
4	Output units and loss functions	Chapter 6	—
5	Backpropagation	Chapter 6	—
6	Regularization	Chapter 7	HW-2

Assignments

Python is the default programming language for the course.
Submit via Moodle or GitHub, as specified in each assignment.
Honour code. Any case of copying is awarded a zero on the assignment. More severe penalties may follow.
Late submissions incur penalties, as announced with the assignment.

References and resources

Text

Deep Learning — Ian Goodfellow, Yoshua Bengio and Aaron Courville
Chapter 6 — multi-layered perceptrons, backpropagation
Chapter 7 — regularization: L1/L2 and other techniques
Chapter 8 — optimization techniques, normalization
Chapter 9 — deep learning for vision, basic models (CNNs)
Chapter 10 — deep learning for NLP, basic models (RNNs, LSTM)

Papers

Evaluation

30% Continuous assessment (CWS)Quizzes and assignments

30% Mid-term exam (MTE)

40% End-term exam (ETE)

Tentative.

All courses