Shahid Beheshti University · MSc · 14 chapters

Advanced
Mathematical Software

The whole course, compressed: from cleaning a CSV to solving differential equations with neural networks. Arrow keys to move, T for contents, R to hide formulas and quiz yourself, M for the deep dive on each slide.

Notes by Shahriar Shariati · lectures by Prof. Kurosh Parand · Shahid Beheshti University · Spring 2026
Reconstructed from the 29 recorded lectures (Persian) — transcribed and distilled with AI.
Course project: Pima Indians Diabetes classification · Final project: PINN / KAN equation solving.

∇

Part I · Chapters 1–2

Foundations

1 — The analytics landscape: descriptive → predictive → prescriptive
2 — Optimization problems & parallel computing

CH 01 · Analytics landscape · AI hierarchy & attention

One word, three meanings — that's what attention buys you

Self-attention lets every token vote on what its neighbors mean. Generative ≠ Transformer — GANs and LSTMs generate without it; attention is what made the current LLM wave possible.

self-attention — the شیر (shir) example $$ \text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V $$

شیر means lion, milk, or tap — "farm/cow" vs "cage/roar" vs "sink/drip" nearby pulls it toward the right one. Context decides, not the word alone.

positional encoding $$ PE_{(pos,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right) $$

Attention alone sees a set, not a sequence — sine/cosine encodings (cos on the odd dims) restore word order.

generative ≠ transformer

GANs and LSTMs generate content with zero self-attention. What's new isn't "generative" — it's the Transformer itself.

The containment chain, precisely

AI ⊃ ML ⊃ DL ⊃ Generative AI ⊃ LLM. Artificial Intelligence is any technique that produces intelligence-associated behavior; Machine Learning narrows that to behavior learned from data; Deep Learning narrows further to multi-layer artificial neural networks; Generative AI narrows to systems that generate new content — text, images, audio, video, or code — rather than only classify or predict a scalar; Large Language Models are the text-to-text branch of Generative AI. Every LLM is a generative-AI system, but not every generative-AI system is an LLM: an image-generating diffusion model or a music-generating network sits inside Generative AI without ever entering the LLM branch.

Why "generative" ≠ "Transformer"

Generative modeling predates attention in its deep-learning form: recurrent architectures (LSTMs, GRUs) and adversarial architectures (GANs) are generative without any self-attention mechanism. What is specific to the current GPT-/BERT-style wave is not "being generative" per se but the architectural choice of the Transformer, built around self-attention from "Attention Is All You Need." Textbooks disagree on where to draw the LLM boundary — some nest LLMs strictly inside Generative AI (the reading used here), others treat them as a parallel category. Both framings appear in the literature.

Self-attention, worked through شیر

Self-attention lets each token look at every other token, assign it a learned relevance weight, and blend information from all of them to build a context-aware representation of itself. In a sentence about a farm, words like "cow" and "glass" pull شیر (shir) toward "milk"; in a sentence about a zoo, "cage" and "roar" pull it toward "lion"; in a sentence about a leaking bathroom, "sink" and "drip" pull it toward "faucet/tap." Older fixed-embedding or shallow-recurrent pipelines had no clean mechanism for this kind of weighted, context-dependent vote.

Positional encoding

Because self-attention treats a sequence as an unordered set of tokens, Transformers add a positional encoding to each token's embedding — built from sine and cosine functions of the token's position — so the model can still distinguish "the dog bit the man" from "the man bit the dog." In the standard form, $pos$ is the token's position, $i$ indexes the embedding dimension, and $d$ is the total embedding dimensionality.

Why this matters for the term projects

One of the course's term projects benchmarks reasoning-capable LLMs on a nonlinear, non-homogeneous Fredholm integral equation of the second kind, comparing zero-shot, few-shot, and chain-of-thought prompting with no fine-tuning. Reasoning models outperformed non-reasoning ones, but accuracy under plain zero-shot prompting stayed below 50%, improving with few-shot and chain-of-thought prompting. Treating an LLM as literally a text-generation model sitting at the tip of AI ⊃ ML ⊃ DL ⊃ GenAI ⊃ LLM is exactly what makes it meaningful to ask how well it reasons about equations it was never explicitly trained to solve.

from: ch01-analytics-landscape.md · session 01

CH 01 · Analytics landscape · Data-driven orgs & analytics types

What happened isn't the interesting question — what should we do about it is

Analytics forms a staircase: descriptive and diagnostic look backward, predictive/prescriptive/advanced look forward. The course's first project sits on the predictive step.

Descriptive → Diagnostic → Predictive → Prescriptive → Advanced — each step asks a harder question of the same data.

the value pipeline $$ \text{Data} \rightarrow \text{Information} \rightarrow \text{Knowledge} \rightarrow \text{Model} \rightarrow \text{Money} $$

Skip the middle stops and a data-rich org still never sees the money — that takes a data-driven culture, not just data.

analyst vs. data scientist

The Analyst turns a CEO's business question into a data question; the Data Scientist builds the model. Framing in, communicating out.

health analytics, end to end

Alzheimer's staging from MRI/fMRI (predictive); drug discovery modeled with differential equations. One domain, five kinds of analytics.

Five types of analytics

Descriptive — what happened? Summarizing historical data: averages, totals, trends.
Diagnostic — why did it happen? Drilling into descriptive results to find causes.
Predictive — what's likely to happen? Forecasting from historical data — where the course's first hands-on project sits.
Prescriptive — what should be done about it? Recommending actions given a predicted outcome.
Advanced (also Cognitive or Autonomous) — the most sophisticated tier: AI systems that adapt and act with minimal human intervention.

Data → Information → Knowledge → Model → Money

Raw data is inert. It becomes information once organized and given context, knowledge once its patterns are understood, a model once that knowledge is formalized into something that predicts, classifies, or prescribes, and finally money once the model is deployed to create measurable business value. The instructor argued many organizations — including, in his assessment, a number of institutions in Iran — fail to complete this pipeline not from lacking technical skill but from lacking a data-driven mindset and mature data-governance practices; he offered, as an anecdotal aside rather than a documented finding, that some Iranian banks' e-banking systems lag their US/Canadian counterparts for exactly this organizational reason.

Analyst vs. Data Scientist

Business leadership poses business questions. The Data/Business Analyst is the liaison: translating a business question into a well-posed data question, then translating the Data Scientist's technical results back into visualizations and reports leadership can act on. The Data Scientist builds and runs the models. Reading ahead: the classification roundup and preprocessing pipeline covered later in this chapter are Data Scientist territory, while "what business problem does this dataset actually answer?" is Analyst territory.

Health analytics as the connective domain

Health Analytics touches all five tiers: diagnostic analytics might flag an at-risk patient from lab results, predictive analytics forecast disease progression, prescriptive analytics recommend a treatment path. Two examples anchor it in the instructor's own experience — an Alzheimer's-staging problem from a past startup (predict disease stage from a patient's MRI/fMRI data) and drug delivery/discovery, which he connects to his own specialty of differential-equation modeling, underscoring that "analytics" and "mathematical software" overlap rather than sit as separate tracks.

from: ch01-analytics-landscape.md · session 01

CH 01 · Analytics landscape · Pima dataset & preprocessing

Nine columns, one target — but cleaning comes before modeling

768 Pima Indians Diabetes records, 8 predictor features plus one binary Outcome. Predictive analytics starts with prep, not algorithms.

Pima Indians Diabetes — 768 × 9

8 features (Pregnancies … Age) → Outcome (0 = non-diabetic, 1 = diabetic). Women aged 21+, sourced from Kaggle.

feature = field = dimension = variable = attribute

Five words, one idea: a single column in the dataset. Literature uses them interchangeably — don't let vocabulary fool you.

classification roundup (theory deferred)

SVM · Decision Trees · Logistic Regression · Random Forest · Naive Bayes · LDA · Ensembles · KNN · MLP.

The Pima Indians Diabetes dataset

Sourced from Kaggle's "Diabetes Dataset for Beginners": 768 records of women aged 21 and over. Columns: Pregnancies, Glucose, BloodPressure, SkinThickness, Insulin, BMI, DiabetesPedigreeFunction, Age, and Outcome (the binary target, 0 = non-diabetic, 1 = diabetic) — 8 predictor features plus 1 target, 9 columns total. The task ahead is a classification problem: given a patient's measured attributes, predict whether that patient has diabetes. Predictive analytics itself splits into classification, regression, and time series; this project is classification.

Five-stage pre-processing taxonomy

Reused from the instructor's Data Mining course: (1) data cleaning — handling missing values, errors, inconsistencies; (2) data integration — combining multiple sources into one coherent dataset, skipped here since the Pima dataset is a single file; (3) data selection — choosing which records and/or features are relevant; (4) data reduction — shrinking size or dimensionality while preserving useful signal; (5) data transformation — reshaping or rescaling (normalization, encoding) into a form suitable for modeling. The full pass (cleaning, selection, reduction, transformation) is the assigned prep work ahead of the next session's step-by-step Python walkthrough in Google Colab.

Classification method roundup

Rather than lecture on classification theory immediately, the session ran a recall exercise: students called out every algorithm they already knew — SVM, Decision Trees, Logistic Regression, Random Forest, Naive Bayes, LDA, Ensemble methods, KNN, and MLP. This is purely an inventory at this stage; theory is deferred until the dataset is cleaned and explored. Two deserve a forward note: LDA doubles as a dimensionality-reduction technique, covered more deeply in the instructor's own recent Elsevier-published Machine Learning textbook; MLP — a small feed-forward neural network — is the bridge between these "classical" classifiers and the neural-network side of the AI hierarchy from earlier in this chapter.

Where to find datasets — and a preview ahead

Kaggle and the UCI Machine Learning Repository are the recommended free, expected sources for student work; Bloomberg (via the Bloomberg Terminal) is mentioned for completeness as a paid, enterprise-grade source. The next session previews data visualization — bar charts, pie charts, scatter plots, histograms, heatmaps — each suited to different data shapes and questions. A further research aside: classical orthogonal polynomial families (Legendre, Gegenbauer, Jacobi, Hermite, Laguerre), which the instructor has adapted into kernel functions for his own SVM/SVR work and referenced in Kolmogorov-Arnold Networks and Physics-Informed Neural Networks — a specialized tool, not a Transformer replacement, but a preview of where this course's mathematical and machine-learning threads eventually meet.

from: ch01-analytics-landscape.md · session 01

CH 02 · Optimization & parallel · Taxonomy & complexity

Every ML tool is optimization — in disguise

SVM training, PCA, and LDA all reduce to optimizing an objective under constraints — the axes you classify a problem on decide which complexity class, and which solvers, apply.

optimization problem — the general form $$\text{optimize } f(x) \quad \text{s.t.} \quad g_i(x)\ \{\le,=,\ge\}\ 0$$

Phenomenon → data → model: SVM, PCA, and LDA are this same shape underneath. Name the objective and constraints, and the solver menu follows.

class	means
P	solvable in polynomial time
NP	a solution is verifiable in polynomial time
NP-Complete	hardest core of NP — every NP problem reduces to it
NP-Hard	≥ as hard as NP; need not even be checkable fast

From phenomenon to objective

Mathematical software rarely starts with a formula — it starts with a phenomenon, which becomes data, which becomes a model (a differential equation, an integral equation, or an ML model). What is easy to miss is how much of the standard data-science toolbox is optimization wearing a disguise: SVM training, PCA, and LDA all reduce to optimizing some objective function subject to constraints. Once a problem is phrased this way, an entire toolbox of solution strategies — and their complexity guarantees — becomes available.

Three independent axes

A problem is constrained or unconstrained depending on whether a feasible region is imposed; linear or nonlinear depending on the objective (most real ML objectives are nonlinear, which rules out the closed-form, polynomial-time guarantees available to linear programs); and continuous or integer/combinatorial depending on whether variables are forced to integer values. These axes are independent — a real problem is a point in the whole space, e.g. "constrained, nonlinear, integer" — and each combination changes which solvers even apply. The integer/combinatorial axis carries the sharpest consequence: these problems (covering, matching, scheduling, routing) generally fall into NP-Hard.

Solving vs. verifying

Complexity theory separates two questions about any decision problem: can a correct answer be computed in polynomial time (this defines P), and can a proposed answer be checked in polynomial time, even if finding it might not be quick (this defines NP). Whether every efficiently verifiable problem is also efficiently solvable — whether P = NP — remains one of the most famous open problems in computer science.

NP-Complete and NP-Hard

NP-Complete problems are the hardest core of NP: every other NP problem can be reduced to one in polynomial time. NP-Hard problems are at least as hard as the hardest problems in NP, but need not even belong to NP — they need not have solutions checkable quickly at all. Combinatorial and integer optimization problems are typically NP-Hard, which is exactly why the rest of this chapter is about choosing between waiting for a guaranteed answer and accepting a faster, approximate one.

from: ch02-optimization-parallel.md · session 02

CH 02 · Optimization & parallel · Exact vs. approximate

Optimal costs time — Branch & Bound proves it

NP-Hard problems force a trade: exact algorithms guarantee the best answer; heuristics trade that guarantee for speed.

exact, not exhaustive — the right subtree is never fully explored

GreedyDynamic ProgrammingBacktrackingBranch & Bound — for ILP

Branch & Bound — exact, yet fast enough to matter $$\text{ILP: } \max\; c^Tx \quad \text{s.t.} \quad Ax\le b,\; x\in\mathbb{Z}^n$$

Branch: partition on a fractional variable. Bound: drop any subproblem that can't beat the best solution found — no exhaustive search, still optimal.

exactness has a price — even B&B can stall

A real instance from the lecture took up to 13 days to solve exactly. Past some size, even mature commercial ILP solvers stop returning useful answers — teams fall back to heuristics.

Two families for NP-Hard problems

Exact algorithms — Greedy, Dynamic Programming, Backtracking, and Branch and Bound — guarantee the optimal answer, at the cost of potentially long running time. Approximate (heuristic) algorithms sacrifice that guarantee in exchange for speed, returning a "good enough" answer in practical time. Which family to reach for depends on how much the guarantee is actually worth versus how much time is available.

Branch and Bound, mechanically

Branch and Bound is the standard exact method for Integer Linear Programming (ILP): a linear objective with all-integer constraints. ILP is NP-Hard, yet B&B is exact for it. It systematically branches — partitions the solution space into subproblems, typically by fixing a fractional variable to 0 or 1 — and bounds — discards any subproblem whose best possible objective value cannot beat the best solution found so far. The result avoids exhaustive enumeration while still guaranteeing optimality, which is why one branch in the search tree can be pruned entirely without ever being fully explored.

Thirteen days, and then nothing

Exactness is not free. The lecture cited a real-world instance that required up to 13 days to solve exactly with Branch and Bound. Beyond some problem size, even mature commercial ILP solvers stop returning answers in any practically useful time frame at all — at which point practitioners abandon the exact guarantee and fall back to custom heuristics. This is the tension that motivates the rest of the chapter: if a workload is going to take a long time regardless of the algorithm, one remaining lever is more hardware — parallel computing.

Choosing exact vs. approximate

The choice is rarely abstract: a small ILP instance is worth solving exactly, since Branch and Bound's guarantee costs little extra time. A large instance — the kind that shows up once real datasets and production constraints enter the picture — can make the exact guarantee prohibitively expensive, and an approximate algorithm that returns a good, non-optimal answer quickly becomes the only practical option.

from: ch02-optimization-parallel.md · session 02

CH 02 · Optimization & parallel · Flynn & memory models

Parallelism has two independent labels — Flynn, then memory

Flynn's taxonomy classifies instruction/data streams; shared vs. distributed memory is a second, orthogonal axis — a system can be any combination of both.

shared memory — one address space, many cores

Independent cores — own compute, control, registers — but one shared memory all of them access. This is what "multi-core system" means.

distributed memory — no shared address space

Each node owns private memory; nodes exchange data over an interconnect via message passing (MPI-style).

heterogeneous — neither model fits cleanly

GPUs: thousands of light cores built for SIMD-style matrix work. FPGAs too. At scale: Spark, Hadoop.

Two separate questions

It is important to keep Flynn's taxonomy separate from the memory-model classification — the two are orthogonal. A system can be shared-memory and still be SIMD, or shared-memory and MIMD, and so on: one classification describes instruction/data flow, the other describes memory organization. Confusing the two is a common mistake.

Flynn's four cells

Flynn's taxonomy (Flynn, c. 1972) classifies architectures by how many independent instruction streams execute and how many independent data streams they act on. SISD is classical sequential execution — no parallelism. SIMD applies one instruction across many data elements simultaneously; the canonical example is a GPU adding two entire matrices element-wise in a single cycle (older texts call SIMD processors "vector processors"). MISD — multiple instruction streams on one data stream — has no real-world system built purely on this model; Flynn included it mainly for taxonomic completeness. MIMD, multiple independent instruction streams over multiple independent data streams, is the standard model for modern multi-core systems.

Shared vs. distributed memory

In the shared-memory model, a single processor package holds multiple independent cores that all share access to one memory — this is the multi-core system. In the distributed-memory model, each processor or node owns private memory with no shared address space; exchanging data requires an interconnect (a network) and message passing, as in MPI-style communication or remote procedure calls — a model that connects naturally to networking and distributed-systems coursework.

Heterogeneous systems

Some architectures fit neither mold cleanly. GPUs are the leading example: thousands of lightweight, low-power cores optimized for matrix operations and deep-learning workloads — a natural fit for the SIMD pattern above. FPGAs form another accelerator class in this category. At the largest scale, distributed multi-node systems combine many heterogeneous processing nodes to attack a single problem; big-data platforms such as Spark and Hadoop are the standard industrial examples.

from: ch02-optimization-parallel.md · session 02

CH 02 · Optimization & parallel · Dependencies & load balance

RAW is the real bottleneck — balance decides the rest

True data dependency can't be removed by any trick; how evenly you split the remaining work decides how much speedup you actually get.

RAW — read-after-write, the true dependency

1: A = E + C
2: D = A + C
Instr 2 needs A from instr 1 — genuinely can't parallelize; no renaming trick removes this.

hazard	cause	fixable?
RAW	read needs a prior write	no — structural
WAR	write must wait for a read	yes — renaming
WAW	two writes, order matters	yes — renaming

mutexsemaphorecondition variablemonitorbarrier

Three ways to split a workload

Picture one TA grading 25 submissions alone — everything runs serially. With several TAs, two strategies emerge: data parallelism (each TA grades a different subset of the same 25 submissions) and task parallelism (the grading task itself is decomposed — one TA checks correctness, another checks style — potentially on the same submissions). A third form, pipeline parallelism, overlaps different stages of a computation on a stream of data and is most relevant to deep-learning training pipelines. These map directly onto ML workloads: ensemble learning and cross-validation are data- or task-parallel (many models or folds evaluated independently), while gradient-descent training distributes batches or parameter updates across workers.

Why RAW is different from WAR/WAW

Not every instruction pair can run in parallel — dependencies constrain the order. In 1: A = E + C then 2: D = A + C, instruction 2 needs the value instruction 1 produces: a true data dependency (RAW). Anti-dependency (WAR: a later write must wait for an earlier read) and output dependency (WAW: two writes to the same location, order determines which survives) are both artifacts of variable/location reuse — they disappear with techniques like register renaming, which gives conflicting instructions distinct storage. RAW reflects an actual computational necessity and cannot be renamed away; it is the hard, structural limit on how much parallelism a piece of code admits.

Synchronization is not free

When parallel workers do share resources — memory, files, devices — access must be coordinated to avoid race conditions and preserve dependency orderings. Standard primitives include the mutex, semaphore, condition variable, monitor, and barrier. Used sparingly and correctly, synchronization preserves correctness at a modest cost; used excessively or incorrectly, it can both slow the program down and, in the worst case, produce incorrect results.

Load balancing and unpredictable work

Even with dependencies respected and synchronization handled, speedup depends on how evenly work is spread. A problem taking 17 time units sequentially might drop to only 14 with a naive parallel split — some improvement, but several units sit idle once their chunk finishes early. A smarter partition improves further, though the lecture was careful to note it was not perfectly ideal, only better. This is hardest for online algorithms, whose decisions depend on real-time or streaming input rather than being fully known in advance — any static partition risks becoming unbalanced as execution proceeds (OS schedulers are, in this sense, an example, even though OS courses don't always frame them that way).

from: ch02-optimization-parallel.md · session 02

Part II · Chapters 3–5

Working with Data

3 — Preprocessing & cleaning: missing values, outliers, scaling
4 — Visualization: choosing the honest chart
5 — Dimensionality reduction: PCA & t-SNE

CH 03 · Preprocessing · Zeros that lie

A clean CSV is not clean data — zero can mean "never measured"

Pima's isnull() reports zero nulls everywhere. That's the trap. Garbage in, garbage out.

disguised missing — not a real 0

A living patient cannot have BloodPressure = 0. Same story for Glucose, SkinThickness, Insulin, BMI — isnull() never sees it.

the fix, in order

replace(0, np.nan) first, then fillna(median) — never fill a raw zero, or you bias every statistic computed before the fix.

Why "no nulls" is misleading

The Kaggle Pima CSV passes df.isnull().sum() with zero nulls in every column, which looks like a green light to start modeling immediately. It isn't. A 3×3 grid of histograms over all 9 columns (axes[i].hist(df[col]) per column) reveals a physically impossible spike at exactly zero in Glucose, BloodPressure, SkinThickness, Insulin, and some BMI readings. These are not measurements of "zero" — they are placeholders for a measurement that was never taken. This single observation is the seed of most of the cleaning work in the chapter.

Finding the zeros safely

(df == 0).sum() only works cleanly when the whole frame is numeric; a mixed numeric/text frame can make the whole-frame comparison misbehave. The robust habit is to restrict it to the columns you actually care about: df[['Pregnancies','Glucose']].eq(0).sum().

Which zeros actually count

Deciding which zeros are "missing" is a domain-reasoning step, not a mechanical one. Zero is definitely implausible for BloodPressure, Insulin, BMI, and SkinThickness in a living person — but only uncertain for Glucose (a very low reading is at least physiologically conceivable). Only the implausible columns should be converted and imputed.

The two-step recipe

Convert first, impute second: df[cols] = df[cols].replace(0, np.nan), then df.fillna(df.median(), inplace=True). Because a DataFrame is a collection of per-column Series, pandas automatically matches each column to its own median when the fill value is itself a Series — no manual looping required.

from: ch03-preprocessing.md · session 03

CH 03 · Preprocessing · Impute, don't drop

Dropping a row throws away seven good features to punish one bad one

The right statistic follows from data type and outliers — never from convenience.

method	use when
mean	numeric, no outliers
median	numeric, outliers present
mode	categorical (nominal/ordinal)
ffill / bfill	time-ordered data

why mean breaks — one outlier, big lie

19 students earn 10–30M Toman/mo, one earns 1B. Mean ≈ 100M — meaningless. Median ignores the extreme entirely.

conditional imputation — condition on a related feature

Missing BloodPressure for a 50-year-old? Use the mean BloodPressure of other ~50-year-olds, not the whole dataset's mean.

preserves size → 768 rows stay 768 preserves shape → median keeps distribution drop risk → shrinks an already-small set

Two options, one clearly better

Once missing (or disguised-missing) values are identified, there are exactly two moves: drop the row, or impute a substitute value. Imputation is generally preferred for four concrete reasons: it avoids discarding the row's other, perfectly good features; it improves generalization by keeping the training distribution close to the true one; it preserves dataset size (Pima has only 768 rows — dropping rows missing Insulin shrinks an already-small dataset further); and mean/median/mode imputation tends to preserve the original shape of the distribution better than deletion does.

The taxonomy decides the statistic

Which central-tendency statistic is even legitimate depends on data type (§3.2): nominal categorical data only supports the mode; ordinal supports mode or median; interval/ratio numeric data supports mean, median, or mode. Layered on top of that: for numeric features, use mean only when there are no extreme outliers, and median when there are — because the median depends solely on the middle-ranked value(s) and ignores extremes, while the mean is dragged toward them.

Worked contrast: mean vs. median

A class of 20 students, 19 earning 10–30 million Toman/month and one earning 1 billion, produces a mean around 100 million/month — wildly unrepresentative of almost everyone in the class. The median, unaffected by that single extreme, is the correct summary. The same logic transfers directly to a long-tailed column like Insulin: median imputation is the safer default there.

Conditional imputation as a refinement

Rather than one global statistic, condition the fill on a related feature: impute a missing BloodPressure for a 50-year-old using the mean BloodPressure among other ~50-year-old patients, not the whole-dataset mean. The non-medical analogue: filling a missing grade using the mean of students in a similar GPA band (e.g., GPA > 17) rather than the whole class average. This is a more advanced, more contextually faithful technique.

from: ch03-preprocessing.md · session 03

CH 03 · Preprocessing · Skew fools the Z-score

Skew fools the Z-score — median-based methods don't bend

Same masking problem, same fix, downstream: this is why RobustScaler exists.

Skew inflates mean/σ, so ordinary z-score masks the same point IQR flags.

IQR rule — the robust default $$x < Q_1 - 1.5\,IQR \ \text{ or }\ x > Q_3 + 1.5\,IQR$$

Flags dozens in SkinThickness, almost none in Glucose.

modified Z-score — median + MAD $$M_i = \frac{x_i - \tilde{x}}{MAD}$$

Median and MAD resist extremes, so skew can't mask the outlier the way $\mu,\sigma$ do.

Three ways to flag an outlier

Visual detection (boxplots: min, Q1, median, Q3, max) works for a handful of features but doesn't scale past a few columns. The IQR method flags $x$ outside $[Q_1 - 1.5\,IQR,\ Q_3 + 1.5\,IQR]$ and, on Pima, produced very different counts per column — essentially none for Glucose, dozens for SkinThickness. The ordinary Z-score flags $|z| = |(x-\mu)/\sigma| > 3$, relying on the three-sigma rule for approximately normal data — but on a heavily skewed column like Insulin it detects noticeably fewer outliers than IQR, because skew inflates both $\mu$ and $\sigma$, shrinking the computed z-score of a genuinely extreme point and masking it as ordinary.

The modified Z-score fix

Replacing mean and standard deviation with the median and median absolute deviation (MAD) — $M_i = (x_i - \tilde{x})/MAD$ — removes the masking problem, since neither the median nor MAD is dragged around by extreme values. A commonly cited scaling constant (0.6745, to make the statistic comparable to a standard normal z-score) appears in some textbook treatments but was not confirmed in this course's source material, so it's omitted here — check a reference before using it on a graded assignment.

Once flagged, four legitimate responses

Remove the rows; cap/clip/Winsorize to a chosen percentile; replace using the same techniques as missing-value imputation (e.g., median); or deliberately keep the outlier when it is meaningful signal rather than noise — the classic case being fraud detection, where an account jumping from $1,000/month to $1,000,000/month is exactly the pattern a model should learn to flag, not discard.

Why scale afterward, and which scaler

Pregnancies lives in single digits, Insulin and Glucose in the hundreds, DiabetesPedigreeFunction is a small fraction — distance-based (KNN) and gradient-based methods, plus PCA, implicitly over-weight the larger-magnitude features unless rescaled. StandardScaler ($z=(x-\mu)/\sigma$) is the general default; MinMaxScaler rescales to $[0,1]$ but is itself outlier-sensitive since $x_{min}/x_{max}$ can be extreme; RobustScaler (median/IQR) is the right choice exactly where outliers were flagged repeatedly in this chapter — Insulin, SkinThickness. The same logic underlies why national exam scores and IQ ($\mu=100,\sigma=15$) are reported as standardized scores rather than raw ones.

from: ch03-preprocessing.md · session 03

CH 04 · Visualization · Choosing the right chart

Match the chart to the question — not the data to a default

Audience, purpose, and data type decide the chart — and data type even limits which statistics are valid. A "mean gender" is meaningless.

Data type gates both the valid statistics and the valid chart family.

Data type — what statistics are even valid

type	valid stats
nominal / binary	mode only
ordinal	mode, median
interval / ratio	mean, median, mode, var, skew

Categorical → mode (± median); numerical unlocks mean, variance, skew.

Goal → chart family

comparison → bar/column composition → pie distribution → hist/KDE trend → line relationship → scatter/heatmap

Who the chart is for

A CEO, CFO, CTO, and CIO each need a different lens on the same underlying data — a CFO wants finance-oriented visuals, a CTO wants technical/system-level ones. The "right" chart is the one that serves the reader's actual decision, not the one that is easiest to produce.

Exploratory vs. confirmatory

Exploratory analysis looks at the past and present — descriptive and deterministic: what happened, what the data looks like now. Confirmatory analysis looks toward the future, tied to predictive analytics, treating data probabilistically: likelihoods, forecasts, risk scores. A related split names the same divide by goal rather than time: statistical (descriptive) visualization aims to understand distributions and shape; predictive visualization aims to uncover relationships that support a modeling task. Many charts serve both purposes depending on how they're used.

Why data type gates the statistics

The variable's measurement level must be identified before a chart is picked, because it determines which summary statistics are mathematically meaningful. Nominal/binary variables admit only the mode; ordinal adds the median; interval/ratio numerical variables unlock mean, median, mode, variance, standard deviation, and skewness. Computing a "mean gender" is meaningless; computing a mean glucose level is not. The same hierarchy governs chart choice: categorical variables call for bar, column, and pie charts, while numerical variables call for histograms, KDE plots, box plots, and scatter plots.

The fuller catalogue

Beyond the five basic goal-to-chart mappings, two richer tools round out the toolkit. A pair plot condenses many scatter and distribution plots into one grid, with each diagonal cell showing a single variable's own distribution, and each panel colorable by a categorical class label to see group separation across every pair of features at once. A violin plot combines the box plot's summary statistics with a full, mirrored KDE-style density shape — useful both descriptively and for judging, visually, whether a variable is likely to help discriminate between classes. Box and violin plots are frequently drawn "by outcome" to compare how a distribution differs between groups.

from: ch04-visualization.md · sessions 11–12

CH 04 · Visualization · Scaling, engineering & imbalance

What you plot depends on what you did first — scaling, engineering, imbalance

Visualization runs on data that's already been transformed — and is often the tool that reveals a transformation is needed. One oversized feature can drown out all the rest.

768 Pima records: 500 vs 268 — "always negative" still looks accurate.

Euclidean distance — why scale at all $$ d(x,y) = \sqrt{\sum_i (x_i - y_i)^2} $$

One oversized feature dominates the sum regardless of relevance — scale first.

Two fixes $$ z=\frac{x-\mu}{\sigma} \qquad x'=\frac{x-x_{\min}}{x_{\max}-x_{\min}} $$

Standardize to mean 0, sd 1 (≈99.7% within [-3,3]); min–max maps into [0,1].

Feature engineering — a double edge

BMI → underweight/normal/overweight/obese speeds learning — unless the threshold assumption is wrong.

Distance-based distortion, worked

Many algorithms and preprocessing formulas are distance-based, most typically the Euclidean distance above. If one feature has a much larger numeric range than the others — a price feature in the millions next to features that range from 0 to 100 — its squared term dominates the sum and distorts the distance calculation regardless of how relevant the other features actually are.

Two standard fixes

z-score standardization rescales a feature to mean 0 and standard deviation 1 (about 99.7% of standardized values fall within [-3, 3]); min–max scaling maps a feature linearly into [0, 1]. Either removes the risk of one feature's raw magnitude overwhelming a distance-based computation — and, as a side benefit, puts features on comparable axes for visual comparison too.

Feature engineering's double edge

Feature engineering injects domain knowledge to help a model learn faster — for example, converting a raw BMI value into categorical labels using established medical thresholds rather than leaving the model to discover the pattern from scratch. This can genuinely speed up learning by giving the model a useful prior. But it carries a real risk: if the underlying assumption is wrong or biased — say, assuming higher BMI always implies higher diabetes risk, when evidence shows underweight individuals can also carry elevated risk — the injected label can mislead the model, causing overfitting, confusion, or wasted training effort. Every engineered feature should be checked visually (histograms or box plots split by the new category) before it is trusted.

Imbalance and the accuracy trap

A dataset is imbalanced when its target classes are far from evenly represented. A hospital dataset of 5,000 records where only about 100 are cancer-positive is strongly imbalanced — a naive classifier that always predicts "negative" would score 98% accuracy while being useless. The Pima dataset is itself mildly imbalanced (of 768 records, roughly 500 non-diabetic and 268 diabetic), exactly the kind of skew a simple pie or column chart of the outcome variable reveals immediately — which is why checking class balance visually is one of the first steps in any classification project, well before the train/test split (commonly 80/20).

from: ch04-visualization.md · sessions 11–12

CH 04 · Visualization · Pima: skew & hidden zeros

Zero isn't a measurement — it's missing data in disguise

Histograms of Glucose, BMI, and Insulin all spike at exactly zero — a biologically impossible value hiding as a real one. ≈374 of 768 Insulin rows are zero.

Insulin: skewness ≈ 2.27 — mean ~80, median ~30, and about half the column parked at zero.

Skewness by feature

feature	skew	read
Glucose	0.17	near symmetric
BMI	0.43	mild right skew
Age	1.30	young-heavy, long tail
Insulin	2.27	most skewed

Higher skew ⇒ longer tail; Insulin's 2.27 pairs with ~374 zero-valued rows.

Split by outcome

Glucose and BMI both shift higher in diabetics — with real overlap. Age barely moves.

Reading the shape of each feature

Histograms and KDE plots of the individual features tell a consistent story. BMI has a mean around 32 and a median around 30 (skewness ≈ 0.43) — a mild right skew. Age is far more skewed (mean ≈ 33, median ≈ 29, skewness ≈ 1.30): the cohort is dominated by younger patients with a long tail toward older ages, and it is the cleanest feature in the sense that it contains no implausible zero values. Insulin is the most skewed of all (skewness ≈ 2.27), with a mean near 80 but a median near 30 — heavily concentrated at low values with a long tail of large ones. Glucose is close to symmetric (skewness ≈ 0.17), the mildest skew among the numeric features.

Zero as a data-quality signal

For Glucose, BMI, and Insulin, a value of exactly zero is biologically impossible — nobody has zero blood glucose or zero BMI and is alive. Roughly half of the Insulin column (about 374 of 768 records) is zero. Rather than true measurements, these are missing values disguised as zeros, and spotting the spike at zero in a histogram is often how this data-quality problem is first noticed, well before any formal cleaning step. The same applies to skin-fold thickness and blood pressure elsewhere in the dataset.

What changes by outcome

Splitting histograms or KDE curves by Outcome shows that Glucose and BMI both shift toward higher values in the diabetic group — with real overlap between the two groups, so neither variable fully separates them on its own, but both are clearly informative. Age shows only a weak shift, making it a much less useful discriminator by itself. Bar-chart comparisons make the pattern concrete: mean glucose runs noticeably higher for diabetic patients (figures reported around 118 vs. 143 mg/dL in one presentation, and closer to 109 vs. 141 in a KDE-based comparison) — the exact numbers vary slightly between analyses, but the direction and size of the gap are consistent.

Age-group trends

Column charts binning patients into ten-year age groups show the diabetes rate climbing from the 20s into the 40s and 50s before falling off in the sparsely-sampled oldest groups, and a parallel chart of "high-risk" patients (glucose above 140 mg/dL) shows the same rate rising essentially monotonically with age. A pie chart of glucose banded into three ranges shows a large share of the dataset sitting in the pre-diabetic middle band — a reminder that falling in the "diabetic range" by glucose alone does not equal a diagnosis.

from: ch04-visualization.md · sessions 11–12

CH 04 · Visualization · Outliers, correlation & causation

A heatmap is not a causal claim — IQR finds outliers, RCTs find causes

Box plots flag the same implausible zeros as isolated points; a Glucose–BMI scatter shows diabetics clustering upper-right. Neither chart proves causation.

IQR outlier rule $$ IQR = Q_3 - Q_1 $$ $$ [\,Q_1-1.5\,IQR,\ Q_3+1.5\,IQR\,] $$

Points outside this band are flagged — the same rule fraud detection applies to a $100–300/day spender.

Heatmap read

Glucose↔Outcome is the strongest single link; Glucose↔BMI and BMI↔skin-fold are moderate.

Diabetic patients cluster high-glucose/high-BMI — association, not proof of cause.

The IQR rule, worked

The box plot's five-number summary — minimum, Q1, median, Q3, maximum — is the standard tool for flagging outliers, using IQR = Q3 − Q1, with lower bound Q1 − 1.5×IQR and upper bound Q3 + 1.5×IQR; any value outside that range is flagged. Applied to the Pima features, box plots make the implausible zero-values visible as isolated points well below the lower bound for Glucose, BMI, and Insulin, alongside a small number of genuinely extreme (but plausible) high values. The same technique is standard in business and finance analytics — spotting fraudulent credit-card activity by flagging a transaction that falls far outside a customer's normal $100–300/day spending range.

Reading a correlation heatmap

A correlation heatmap gives a compact overview of pairwise relationships, with warmer colors marking stronger correlation magnitude. The coefficient ranges from −1 to 1: values near ±1 indicate strong linear relationships, values near 0 indicate little to none. Pearson correlation is standard for numerical variables; Spearman and Kendall are used for ordinal or categorical variables. In the Pima dataset, Glucose and BMI show a notable positive correlation, and BMI and skin-fold thickness show a moderate positive correlation, consistent with both being body-composition measures; a heatmap restricted to correlation with Outcome alone highlights Glucose as the feature most strongly associated with diabetes status.

What the scatter adds — and doesn't prove

A scatter of Glucose against BMI, colored by Outcome, shows the same relationship geometrically: diabetic patients cluster toward the upper-right (high glucose and high BMI together), non-diabetic patients toward the lower-left, and the combination is a stronger signal than either feature alone. But a scatter plot — or the correlation coefficient behind it — shows association, not causation. Classical statistics establishes causal relationships through a randomized controlled trial: two homogeneous groups are matched on relevant characteristics such as age and gender, one receives a placebo and the other the real treatment, and outcomes are compared afterward with an appropriate statistical test.

Pair plots extend the view

A pair plot extends the single Glucose/BMI comparison into a full grid across all numeric features, with diagonal cells showing each variable's own distribution as a histogram or KDE. It condenses many individual scatter and distribution plots into one view, and each panel can be colored by the class label to see how groups separate across every pair of features at once — reinforcing the same Glucose/BMI pattern alongside every other pairwise relationship.

from: ch04-visualization.md · sessions 11–12

CH 05 · Dimensionality reduction · PCA mechanics

Variance is the signal — rotate, then drop

PCA looks for the axes along which the data spreads out most, then keeps only the top few. Those axes are eigenvectors of the covariance matrix.

eigen-problem — variance-maximizing directions $$ C = \frac{1}{n-1}X^TX, \qquad Cx = \lambda x $$

$C$ is symmetric, so its eigenvectors come out mutually orthogonal — a free rotated axis system. Sort $\lambda$ descending, keep top $k$.

z-score first, always $$ z = \frac{x-\mu}{\sigma} $$

Puts every feature on comparable footing before any covariance is computed — and already mean-centers the data for free.

Why standardize before anything else

PCA is built entirely on squared distances and covariances, so a feature with a much larger raw range will dominate every entry of $C$ purely from scale — not from being more informative. The chapter's example: BMI ranging roughly 15–40 against an unscaled feature running into the thousands. Z-score standardization fixes this, and because it subtracts the mean as part of the formula, it already mean-centers the data — no separate centering step is needed afterward.

What the covariance matrix actually holds

$C=\frac{1}{n-1}X^TX$ is $N\times N$ — sized by features, not samples. Entry $C_{ij}$ measures how features $i$ and $j$ move together: a large positive value (Glucose and BMI in Pima, both tied to metabolic risk) means the two rise together; a value near zero means they vary largely independently.

Eigenvectors as a free rotation

The fact PCA leans on: maximizing the variance of data projected onto a direction is exactly the eigenvalue problem $Cx=\lambda x$. Because $C$ is symmetric, its $N$ eigenvectors are mutually orthogonal for free — a rotated coordinate system where each new axis is an explicit linear combination of the standardized features, e.g. $PC_1=a_1\cdot\text{Glucose}+a_2\cdot\text{BMI}+\dots$, with weights read straight off the eigenvector.

Keeping k and projecting

Sorting eigenvalues $\lambda_1\ge\lambda_2\ge\dots$ ranks components by variance explained; the top $k$ eigenvectors become columns of $W$, giving $Z=X_{\text{centered}}\cdot W$. $k=2$ or $3$ is standard for visualization, but $Z$ is not visualization-only — since every column is a genuine linear combination of real features, it can feed a downstream model directly, unlike the embedding t-SNE produces.

from: ch05-dimensionality-reduction.md · sessions 12–14

CH 05 · Dimensionality reduction · Explained variance ratio

Two axes, most of the story — the elbow flattens fast

Explained variance ratio tells you exactly how much you lose by dropping components. On Pima, PC1 + PC2 alone keep about 80% of the total spread.

Bars: per-component variance (blue/green = the two kept). Dashed line: running total.

EVR — explained variance ratio $$ \text{EVR}_j = \frac{\lambda_j}{\sum_i \lambda_i} $$

Fraction of total variance held by component $j$. PC1+PC2 ≈ 80% on Pima's 8 features.

9 steps — raw features to $Z$

standardize → covariance $C$ → eigen $Cx=\lambda x$ → sort → project $Z=X_cW$ → (cumulative) EVR.

by hand vs sklearn

Both run on Pima; `sklearn.decomposition.PCA` uses SVD — more stable, exposes `explained_variance_ratio_` directly.

Reading the scree plot

Each bar is one component's $\text{EVR}_j=\lambda_j/\sum_i\lambda_i$; the dashed line is the running total. On Pima's eight standardized features, a representative run puts $PC_1$ at roughly 60% and $PC_2$ at another 20%, so two axes already carry about 80% of the dataset's total spread — an 8-to-2 compression with fairly small information loss, small enough that the resulting scatter plot is a legitimate stand-in for the full feature space.

The nine-step recipe

Build $X$ → standardize (z-score) → mean-center (already done by the z-score step) → covariance $C=\frac{1}{n-1}X^TX$ → solve $Cx=\lambda x$ → sort eigenvalues descending → choose $k$ → project $Z=X_{\text{centered}}W$ → compute (cumulative) EVR. Every step is mechanical once the eigenproblem is solved; the only judgment call in the whole pipeline is picking $k$.

Doing it by hand vs. letting sklearn do it

A from-scratch version — build $C$, call a numerical eigensolver, sort, project — was implemented alongside `sklearn.decomposition.PCA` on the same Pima data, specifically to make every step above explicit rather than hidden behind a library call. `sklearn.decomposition.PCA` runs the equivalent pipeline internally via SVD rather than a naive eigensolver, which is more numerically robust — especially once features are highly collinear or $C$ drifts close to singular — and it exposes `explained_variance_ratio_` directly. The resulting PC1-vs-PC2 scatter, colored by Outcome, showed the two classes occupying visibly different, if overlapping, regions of the plane.

Why this matters beyond plotting

Because $Z$ is a real linear transform of real features, it is not just a visualization tool the way t-SNE's output is — it can be handed to a downstream classifier as a lower-cost, denoised feature set, trading a small amount of variance for a large reduction in dimensionality.

from: ch05-dimensionality-reduction.md · sessions 12–14

CH 05 · Dimensionality reduction · t-SNE mechanics

Neighbors, not axes — t-SNE fights the crowding problem

t-SNE turns distances into neighbor probabilities, then matches them in 2D with a heavy-tailed distribution. KL divergence is what the optimizer actually minimizes.

cost — match neighbor probabilities $$ C = KL(P\|Q) = \sum_{i \neq j} p_{ij}\log\frac{p_{ij}}{q_{ij}} $$

Gradient pulls points together where $P_{ij}>Q_{ij}$, apart where $Q_{ij}>P_{ij}$.

perplexity — target neighbor count $$ \text{Perp}(P_i) = 2^{H(P_i)} $$

Binary search picks $\sigma_i$ per point (e.g. 20 → 30) since local density varies across the dataset.

heavy tail fixes crowding

Cauchy-shaped $q_{ij}$ decays slower than Gaussian, letting moderately-far points spread apart instead of collapsing.

From distances to conditional probabilities

t-SNE starts from pairwise Euclidean distances in the original space and turns each into a probability $p_{j|i}$ via a Gaussian centered on point $i$: "how likely would $i$ pick $j$ as a neighbor." Because a Gaussian is light-tailed, only genuinely close points get meaningful probability mass — exactly what "neighbor" should mean in the high-dimensional space.

Perplexity: one bandwidth per point

Rather than fixing the Gaussian's bandwidth $\sigma_i$ directly, t-SNE fixes a target perplexity (5–50, commonly ~30) and binary-searches $\sigma_i$ per point until $\text{Perp}(P_i)=2^{H(P_i)}$ matches it — guess $\sigma_i$, compute the entropy-based perplexity (say 20 against a target of 30), adjust, repeat. A single global bandwidth would systematically over-smooth sparse regions and under-smooth dense ones, so each point needs its own.

Symmetrizing and the crowding problem

$p_{j|i}\ne p_{i|j}$ in general — a point in a dense cluster center splits its probability mass over many close rivals, while an edge point's few neighbors each get a bigger share — so t-SNE averages: $p_{ij}=(p_{j|i}+p_{i|j})/(2N)$. For the low-dimensional side, reusing a Gaussian would hit the crowding problem: high-D space has "more room" for many moderately-distant relationships than 2D or 3D can represent without over-compressing them. The fix is a heavy-tailed Student-t (Cauchy-shaped) kernel for $Q$, decaying slowly enough that moderately-distant points keep meaningfully higher affinity than a Gaussian would give them, letting the optimizer spread them apart instead of crowding them together.

Optimizing the embedding

The cost $C=KL(P\|Q)$ is minimized by gradient descent with momentum, $Y^{(t)}=Y^{(t-1)}+\eta\,\partial C/\partial Y+\alpha(Y^{(t-1)}-Y^{(t-2)})$, pulling pairs together where $P_{ij}>Q_{ij}$ and apart where $Q_{ij}>P_{ij}$. Early exaggeration multiplies $P$ by a factor (e.g. 12) for roughly the first 250 iterations to help true clusters separate quickly, then is divided back out; $Y$ is periodically re-centered to keep the layout near the origin. Because $Y$ starts random and the trajectory is stochastic, fixing `random_state` in `sklearn.manifold.TSNE` is what makes two runs comparable at all.

from: ch05-dimensionality-reduction.md · sessions 12–14

CH 05 · Dimensionality reduction · PCA vs t-SNE

Pick by the question — not by which method is "better"

Neither method dominates universally — on Pima, t-SNE actually showed more class overlap than PCA. Match the tool to variance vs. neighborhood questions.

aspect	PCA	t-SNE
nature	linear (eigendecomposition)	nonlinear, iterative (KL descent)
deterministic?	yes	no, unless seeded
downstream modeling	yes — reusable features	no — visualization only
typical use	feature reduction + quick viz	explore nonlinear cluster structure

opposite direction — kernel PCA

Kernel methods map data to a higher-dim space to expose nonlinear structure — the reverse of reduction, sharing the same eigen machinery.

What actually happened on Pima

Run on the same eight standardized features, PCA and t-SNE told visibly different stories: PCA's PC1-vs-PC2 plot showed the two Outcome classes in different, if overlapping, regions; t-SNE, generally considered the stronger method for visual class separation, actually showed more mixing between diabetic and non-diabetic points on this particular dataset. Neither method dominates universally — the right choice depends on the dataset's actual structure, not on which technique has the better reputation.

A rule of thumb

Reach for PCA first: it is cheap, deterministic, and — because each component is a real linear combination of the original features — its output can be reused directly as engineered features for a downstream model. Reach for t-SNE specifically when the goal is exploring whether nonlinear cluster structure exists that a linear projection might be flattening away, keeping in mind it is markedly more expensive and, per scikit-learn's own documentation, not meant to feed a model.

Kernel methods: same machinery, opposite direction

PCA's components are straight lines through the original feature space, which fails on data like a ring of one class around another — no rotation of straight axes separates that. Kernel methods implicitly map data into a different, often higher-dimensional, space where such structure becomes linear; kernel PCA is this trick applied to PCA itself. The direction of travel is the reverse of this chapter's reduction goal, yet both rely on closely related eigen-problem machinery. A research direction mentioned in lecture — using classical orthogonal polynomial families (Legendre, Chebyshev, Jacobi, Gegenbauer) as kernels, including in a fractional-order setting — is flagged only as a pointer to further reading; kernel PCA proper belongs to a companion Machine Learning course.

Quick checklist

Deterministic and reusable downstream → PCA. Comparatively fast → PCA; comparatively heavy → t-SNE. Need axes with explicit feature weights (explained variance ratio) → PCA; only need a 2D/3D picture that preserves who's near whom → t-SNE, with `random_state` fixed for reproducibility.

from: ch05-dimensionality-reduction.md · sessions 12–14

Part III · Chapters 6–10

Classical Machine Learning

6 — Sampling, splitting & validation
7 — Classification: KNN, Naive Bayes, Logistic Regression, SVM
8 — Evaluation: confusion matrix, ROC, the right metric
9 — Clustering: k-means, hierarchical, DBSCAN
10 — Ensembles: bagging, boosting, stacking

III

CH 06 · Validation · Why we split the data

Test accuracy is a statistic — not a certainty

Train and evaluate on the same rows and a model just memorizes the answer key. The test set stands in for the real world.

80/20 shown here — the ratio used throughout the course's diabetes project.

parameter vs statistic $$ \hat a_{\text{test}} \;\approx\; a_{\text{population}} $$

A parameter is the true, usually unknown fact about a population; a statistic is what you measure on a sample. Test accuracy is a statistic — an estimate, not a guarantee.

split ratio by dataset size

size	ratio	why
100k–1M+	90/10	even 10% is huge
moderate	80/20	this course's diabetes set
small	70/30	20% test starves training

Why memorized answers don't count

A model evaluated on its own training rows can reach near-100% accuracy without having learned anything generalizable — it is simply reciting the answer key it was shown, like a student who aces a test built from the exact key they memorized but fails a new one. The test set exists precisely so that number can't happen by memorization; it approximates "the real world," the unseen cases the model meets after deployment.

Population, sample, parameter, statistic

A population is the entire group of interest (every patient screened for diabetes); a sample is the subset actually measured. A parameter is a true fact about the population (the real average age of all master's students); a statistic is the corresponding quantity computed on a sample, and it varies from sample to sample. Descriptive statistics reports on the sample itself; inferential statistics uses the sample statistic to say something about the population. Nearly all ML evaluation is inferential: test-set accuracy is a statistic used to infer how the model will perform on future, unseen cases.

Picking the ratio

There's no universal split. The training portion must be large enough for the model to discover real structure; the test portion must be large enough to give a statistically trustworthy performance estimate. Both get harder to satisfy at once as the dataset shrinks — which is why small datasets are given proportionally larger test allocations (70/30) while very large ones can afford a thin 10% test slice (90/10) precisely because 10% of a million rows is still enormous in absolute terms.

from: ch06-validation.md

CH 06 · Validation · Sampling theory

Random doesn't mean representative — structure the draw instead

A sample can be drawn correctly and still mislead — how you sample matters as much as how much you sample.

Same population, two draws: SRS can miss a whole subgroup by chance; stratified sampling can't.

simple random — equal chance

Every unit equally likely. Simple, but a small sample can get unlucky — all 5 picks turning out master's students.

stratified — proportional

Split into strata first, then sample each stratum in its true proportion — no subgroup gets left out.

cluster — whole groups at once

Divide into homogeneous clusters (districts), then take entire clusters wholesale — cheaper, less representative.

convenience — whoever's on hand

Fast, low-effort, no probability guarantee — asking the nearest classmates and hoping.

Simple random sampling's bad-luck risk

SRS gives every member of the population an equal chance of selection. It's conceptually the simplest option, but nothing stops an unlucky draw: picking five classmates at random to estimate a class's average age could, purely by chance, select five master's students and skew the estimate away from the true mixed-population value.

Stratified sampling removes the bad luck

The population is divided into known strata — bachelor's, master's, PhD — and the sample draws from each stratum in proportion to its true share. If those groups are 60%/30%/10% of a department, a sample of 100 draws 60, 30, and 10 respectively, guaranteeing every subgroup is represented at its real weight.

Cluster sampling trades representativeness for cost

Rather than sampling individuals, the population is split into clusters assumed internally similar — municipal districts with comparable demographics. Whole clusters are selected (say two of several similar districts) and individuals sampled only within those chosen clusters, cutting collection cost at the price of some representativeness.

Convenience sampling: fast, ungrounded

Samples come from whoever or whatever is readily available, typically under time or budget pressure. Practical, but not grounded in probability theory and carries no formal guarantee.

When collection itself biases the conclusion

How data is gathered can silently bias every downstream conclusion. Checking a store's shelf stock at the same day and time every week can look misleadingly stable: "only one unit on the shelf" every week could mean the product barely sells, or that it sells out almost immediately and gets restocked right before each visit — opposite realities that look identical if collection timing never varies. Similarly, planning a marketing analysis for a new country requires re-checking cultural, religious, and regulatory context (labeling rules, local norms) rather than assuming home-market assumptions still hold.

from: ch06-validation.md

CH 06 · Validation · Imbalance vs. stratified split

Fixing the ratio and preserving it are different jobs

SMOTE, undersampling and oversampling change the class ratio; stratified splitting only preserves whatever ratio already exists across train and test.

undersampling

Drops majority-class rows until ratios balance. Throws away real data — like pruning a tree's roots; rarely the first choice.

oversampling

Duplicates minority rows. Risk: a duplicate leaks across train/test, and the repeated pattern gets over-weighted.

SMOTE — synthetic, not copied $$ x_{\text{syn}} = x_i + \lambda\,(x_j - x_i),\quad \lambda\in[0,1] $$

Interpolates new minority points between nearby real ones — raises the minority share without duplication or leakage.

Stratified splitting doesn't touch the ratio — it just carries 30:70 into both partitions.

Three remedies for imbalance

Real datasets rarely present classes in equal proportions — in the course's diabetes project, diabetic (minority) patients are outnumbered by non-diabetic ones. Undersampling shrinks the majority class but discards real patterns the model could have learned from. Oversampling duplicates minority rows, carrying two risks: a duplicated row split across train and test causes data leakage (the model is tested on data it already memorized), and the repeated pattern can become over-weighted in the learned representation. SMOTE is generally preferred: instead of duplicating, it generates synthetic minority examples interpolated from real minority samples that lie close together in feature space, raising the minority share without the direct duplication/leakage risk.

Worked numbers: 100 samples, 30/70

Splitting 100 samples at a 30/70 diabetic/non-diabetic ratio into an 80-sample train set and a 20-sample test set, while stratifying, gives roughly 24 diabetic / 56 non-diabetic in the larger subset and 6 diabetic / 14 non-diabetic in the smaller one — the same 30/70 ratio, just at different scales.

Stratify ≠ fix

Passing stratify= to scikit-learn's train_test_split does not fix imbalance already present in the data — it only ensures the existing class ratio is preserved proportionally in both resulting subsets. Without it, a purely random split could by chance leave the test set with almost no minority examples, making evaluation unreliable.

Using both together

Imbalance remedies and stratified splitting solve separate problems and are commonly combined: SMOTE (or another remedy) rebalances the dataset itself, while stratified splitting ensures that whatever ratio results is then carried faithfully into both the train and test partitions.

from: ch06-validation.md

CH 06 · Validation · Cross-validation & hyperparameters

One split can get lucky — cross-validation averages that away

An unusually easy or hard test split can make a model look better or worse than it is. Repeat the split, then aggregate.

Magenta = the one held-out fold that round; blue = the folds trained on.

parameters vs hyperparameters $$ \sigma\!\left(\sum_{i=1}^{n} w_i x_i + b\right) $$

$w_i, b$ are parameters — learned by training. Layers, $K$ in KNN, $C$ in SVM are hyperparameters — fixed beforehand, tuned externally.

validation split

Carve the 80% train allocation further: 60% actual-train / 20% validation. Compare hyperparameters there; the test set stays untouched until the final check.

holdout vs. K-fold / LOOCV

Holdout: one split, cheap, fine for big data. LOOCV ($K=N$) trains $N$ separate models — too costly past small datasets.

K-Fold, the standard convention

The dataset is divided into $K$ equal folds (commonly 5 folds of 20% each). The procedure repeats $K$ times, rotating which fold is held out, so every fold is used exactly once as the test set; results across all $K$ runs are averaged into one performance estimate. In each round the model trains on the $K-1$ combined folds and is evaluated on the single remaining fold — for $K=5$, training on 80% and testing on 20% each rotation.

Leave-one-out: the brute-force extreme

LOOCV is the extreme case where $K$ equals the number of data points $N$, so each point forms its own fold: the model trains on all $N-1$ remaining points and is tested on the single held-out point, aggregated over all $N$ rounds. Because this requires training $N$ separate models, LOOCV becomes extremely expensive for large datasets — 100,000 or 1,000,000 rows would mean that many training runs — making it a brute-force method reserved for small datasets.

Holdout: simplicity has a cost

Holdout validation performs a single split once — commonly 80/20 — and never touches the test portion again until final evaluation. No repeated splitting or rotation. It trades the more robust, averaged estimate of resampling methods for simplicity and lower compute cost, and suits genuinely large datasets where one held-out slice is already trustworthy on its own.

Why this matters for hyperparameters

Hyperparameters (activation function, neuron/layer counts, $K$ in KNN, $C$ in SVM) are fixed before training and not learned by the optimizer — chosen by trial and error or a grid search instead. Choosing them needs its own held-out data, separate from the final test set: a validation split carved from the training data (e.g., the 80% training allocation further divided into 60% actual-train / 20% validation). Cross-validation gives a more stable way to compare candidate hyperparameter settings on this training/validation data before the test set is ever touched, avoiding a poor choice driven by one lucky or unlucky validation split.

from: ch06-validation.md

CH 07 · Classification · KNN & distance

No model, just a neighborhood — K decides how big

KNN never trains — every prediction re-scans the whole dataset. K alone tunes the bias–variance trade-off.

Same 12 points, two boundaries: K=1 carves a tiny pocket around each lone point; K=15 smooths straight through them.

Minkowski distance family $$d(x,y)=\Big(\sum_{i=1}^m |x_i-y_i|^p\Big)^{1/p}$$

$p{=}1$ Manhattan · $p{=}2$ Euclidean · $p\to\infty$ Chebyshev. Scale features first — glucose in the hundreds would swamp a 0/1 flag.

choosing K

Small $K$: jagged boundary, low bias, high variance — one outlier flips a vote. Large $K$: smoother, high bias, low variance. No closed form — search $K$ on held-out data.

even-$K$ tie → use odd K or vote weighted by 1/distance $O(N{\cdot}M)$/query → KD-tree

Lazy learning, expensive prediction

KNN has no training phase at all — it memorizes the training set and defers every bit of "work" to prediction time. For $N$ training points with $M$ features, a single query costs $O(N\cdot M)$ just to compute distances, plus a sort, and the entire training set must stay resident in memory. A KD-tree partitions feature space into nested regions so a query only searches a relevant sub-region, but this advantage erodes as dimensionality grows. For sparse, high-dimensional data such as text, an inverted index narrows the candidate set to points sharing non-zero features with the query before exact distances are computed. Missing coordinates block distance computation entirely, so values must be imputed first — the mode of the neighbors for categorical features, the mean for continuous ones.

K as a bias–variance dial

$K=1$ produces a boundary that wraps tightly around individual points: low bias (it fits the training data closely) but high variance (a single mislabeled or outlying point flips the local prediction). Large $K$ averages the vote over a bigger, more stable neighborhood — smoother, higher bias, lower variance. There is no closed-form shortcut for the "right" $K$; it is found experimentally by evaluating accuracy across candidate values on held-out data, which is exactly what makes $K$ a hyperparameter rather than something the model learns.

Ties, and how to break them

With an even $K$ and a binary target a tied vote is possible. Four remedies were discussed: choose $K$ odd so a tie cannot arise (simplest, recommended); break ties randomly (adds arbitrary noise, not recommended); weight the tie by the relative cost of each class (e.g. a missed diabetic case may be costlier than a false alarm); or weight every neighbor's vote by the inverse of its distance, so closer neighbors count more.

Scaling is not optional

Whichever Minkowski $p$ is chosen, features must be standardized before any distance is computed — glucose readings in the hundreds would otherwise dominate a 0/1 flag in every distance calculation, regardless of that flag's actual predictive value.

from: ch07-classification.md · sessions 16–18

CH 07 · Classification · Naive Bayes

45% vs 55% — Naive Bayes never hedges

A generative model: it learns $P(X\mid C)$ per class, then inverts it with Bayes' theorem. Whichever posterior wins, even barely, takes the whole prediction.

Query = Hot & Rainy: No edges out 54.5% to 45.5% — yet the model still commits fully, no "too close to call."

Bayes' theorem → decision rule $$P(C\mid X)\propto P(C)\prod_{i=1}^n P(x_i\mid C)$$

Naive independence turns an intractable joint likelihood into a product of 1-D terms. Drop the marginal $P(X)$ — it's the same for every class, so it can't change the ranking.

Laplace smoothing $$P(w\mid C)=\frac{\text{count}(w,C)+1}{N_C+|V|}$$

One unseen word gives a raw likelihood of zero — and a zero factor collapses the entire product. Add-one smoothing keeps every word's likelihood strictly positive.

Bernoulli → binary Multinomial → counts (text) Gaussian → continuous (Pima)

From conditional probability to Bayes' theorem

Conditional probability $P(A\mid B)=P(A\cap B)/P(B)$ shrinks the sample space to $B$. Since $P(A\cap B)=P(B\cap A)$, applying the definition in both directions gives Bayes' theorem, $P(A\mid B)=P(B\mid A)P(A)/P(B)$, whose four terms have standard names: posterior, likelihood, prior, and marginal (evidence). When only ranking candidate classes for the same evidence, the marginal is identical across all of them and can be dropped — it's only needed if actual normalized probabilities, not just the argmax, are required.

Generative, not discriminative

Naive Bayes is a generative model: it estimates $P(X\mid C)$, how each class generates its observations, then inverts that via Bayes' theorem into the discriminative quantity $P(C\mid X)$. Logistic Regression and SVM, by contrast, estimate $P(Y\mid X)$ directly and never model how the data itself arose.

Three variants, one assumption

The naive assumption — features are conditionally independent given the class — is almost never exactly true (glucose and BMI are correlated), but it collapses the joint likelihood into a product of one-dimensional terms. Bernoulli NB handles binary features, Multinomial NB handles discrete counts (word counts in text), and Gaussian NB assumes each continuous feature is normally distributed per class — the variant used for the all-continuous Pima Diabetes features.

Worked example: Hot and Rainy

Of 14 recorded days, 9 played ("Yes") and 5 didn't ("No"), so $P(\text{Yes})=9/14$, $P(\text{No})=5/14$. Given $P(\text{Hot}\mid\text{Yes})=2/9$, $P(\text{Rainy}\mid\text{Yes})=3/9$, $P(\text{Hot}\mid\text{No})=2/5$, $P(\text{Rainy}\mid\text{No})=2/5$, the unnormalized scores are $\tfrac1{21}\approx0.0476$ (Yes) and $\tfrac2{35}\approx0.0571$ (No). Normalizing by their sum $\tfrac{11}{105}$ gives true posteriors of $5/11\approx45.5\%$ and $6/11\approx54.5\%$ — No wins, but only barely, and Naive Bayes has no concept of "too close to call."

from: ch07-classification.md · sessions 16–18

CH 07 · Classification · Logistic Regression

Squash the line, then threshold it — a perceptron before perceptrons

Linear regression's raw score is unbounded and outlier-sensitive. The sigmoid squashes it into a probability before any decision is made.

σ(z) = 1/(1+e⁻ᶻ): as the linear score z crosses 0, the probability crosses 0.5 and the predicted label flips.

sigmoid squash $$P=\frac{1}{1+e^{-(W^TX+B)}}$$

$\sigma(0){=}0.5$, $\sigma(z)\to1$ as $z\to+\infty$, $\sigma(z)\to0$ as $z\to-\infty$. Each feature's $W_i$ measures how strongly it drives the outcome.

fit by max likelihood $$\ell(W,B)=\sum_i\big[y_i\log p_i+(1{-}y_i)\log(1{-}p_i)\big]$$

No closed form like least-squares — an iterative optimizer maximizes $\ell$ so the observed labels are as probable as possible.

softmax: scores 2,3,5 → thin 20% normal 30% obese 50% ← argmax

Why not linear regression?

Reusing $z=W^TX+B$ directly for a binary target fails on two counts: its output range is unbounded, $z\in(-\infty,+\infty)$, while the target is strictly 0 or 1; and it is highly sensitive to outliers, which can substantially distort the fitted hyperplane and every prediction depending on it.

Sigmoid, threshold, and a neuron's echo

The sigmoid $\sigma(z)=1/(1+e^{-z})$ squashes any real score into the open interval $(0,1)$, directly interpretable as a probability. A fixed threshold — conventionally 0.5 — converts that probability into a label: predict 1 if $P\geq0.5$, else 0. This mirrors a neuron's firing threshold in the perceptron model: a signal transmits only once it crosses a boundary, foreshadowing the activation functions of later chapters. Two assumptions underlie the model: observations must be independent, and the target must be binary — beyond two classes, Softmax takes over.

Fitting by Maximum Likelihood Estimation

Unlike linear regression's closed-form least-squares solution, Logistic Regression's coefficients are fit by MLE: choose $W,B$ that make the observed training labels as probable as possible under the model. The procedure: compute $z=W^TX+B$, apply the sigmoid to get $P$, form the log-likelihood $\ell(W,B)$ over the whole training set, maximize it via an iterative numerical optimizer (no closed form exists), then apply the 0.5 threshold to produce a label.

Beyond two classes: Softmax

When the target has more than two categories, the fixed 0.5 threshold no longer applies. Softmax computes one linear score per class, sums the scores, and normalizes each by that total to get a probability distribution over classes. Example: weight classified as thin/normal/obese with linear scores 2, 3, 5 — summing to 10 gives 20%/30%/50%, and "obese" wins as the argmax.

from: ch07-classification.md · sessions 16–18

CH 07 · Classification · Support Vector Machines

The middle doesn't matter — only the closest points do

Among every hyperplane that separates the classes, SVM keeps the one with the widest breathing room. Only the points touching that margin ever determine it.

Every non-support point could vanish and the boundary wouldn't move — only the ringed points on the margin fix it.

primal problem $$\min_{W,B}\tfrac12 W^TW \ \text{ s.t. } y_i(W^TX_i{+}B)\geq1$$

Maximizing the margin $2/\|W\|$ is the same as minimizing $\|W\|$. Points on $W^TX+B=\pm1$ are the support vectors.

dual via KKT $$W=\sum_i\alpha_iy_iX_i,\quad \sum_i\alpha_iy_i=0$$

Complementary slackness: $\alpha_i{=}0$ unless point $i$ sits exactly on the margin. Only those points ($\alpha_i{>}0$) shape $W,B$.

hard margin → cleanly separable soft margin → tolerates overlap kernel trick → non-linear (outline only)

Margin as an optimization target

SVM approaches classification geometrically: among all hyperplanes $W^TX+B=0$ that separate two classes, it picks the one maximizing the margin — the distance from the boundary to the nearest point of either class. Correct classification with margin requires $y_i(W^TX_i+B)\geq1$ for every point. The perpendicular distance from a point to the hyperplane is $|W^TX+B|/\|W\|$, so each margin plane sits $1/\|W\|$ from the center, giving total margin width $2/\|W\|$. Maximizing that is equivalent to minimizing $\tfrac12\|W\|^2$ — the primal problem.

From primal to dual

Introducing one Lagrange multiplier $\alpha_i\geq0$ per constraint gives the Lagrangian $L=\tfrac12 W^TW-\sum_i\alpha_i[y_i(W^TX_i+B)-1]$, a min–max saddle-point problem. Setting $\partial L/\partial W=0$ and $\partial L/\partial B=0$ yields $W=\sum_i\alpha_iy_iX_i$ and $\sum_i\alpha_iy_i=0$; substituting back eliminates $W,B$ entirely, leaving the dual — a maximization over $\alpha$ alone, expressible as a quadratic program $\min_\alpha \tfrac12\alpha^TG\alpha-\mathbf1^T\alpha$ with $G_{ij}=y_iy_jX_i^TX_j$. Notably $G$ is $N\times N$ (over training samples), not over the 8 Pima features — each entry is just a scalar dot product.

Support vectors via KKT

Complementary slackness forces $\alpha_i=0$ for any point whose margin constraint isn't tight — safely classified points contribute nothing. Only points sitting exactly on the margin get $\alpha_i>0$: the support vectors, the sole points determining the solution. Once $\alpha$ is solved, $W=\sum_i\alpha_iy_iX_i$ and $B=y_i-W^TX_i$ for any support vector $i$; a new point is classified by the sign of $W^TX+B$. Because $G$ is at least positive semi-definite, the QP is convex — any local optimum is the global one.

Hard margin, soft margin, kernels

Hard margin applies when classes are perfectly separable — every constraint holds exactly. Soft margin relaxes the constraints to tolerate some overlap or noise between classes. Kernel methods transform the feature space itself so that data with no linear separator in the original space becomes separable — named in the course material but not derived in depth. SVM was first published by Vapnik and Cortes around 1995; SVM kernels, including ones built from classical orthogonal polynomials, remain active research, with links into PCA and t-SNE.

from: ch07-classification.md · sessions 16–18

CH 08 · Evaluation · Confusion matrix & accuracy paradox

99% accurate, 0% useful — the accuracy paradox

Four raw counts — TP, FP, FN, TN — are the only ground truth; every metric in this chapter is a ratio built from them. Accuracy alone can hide a model that never finds what it was built to find.

FN — a sick patient called healthy — is the costliest cell in medicine.

Accuracy — fraction correct $$Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$$

1000 patients, 10 sick: an "always healthy" dummy model scores ~99% accuracy yet catches zero real cases.

Beyond two classes

n×n matrix (normal / ad / spam) → one-vs-rest: pick a target class, collapse the rest into "negative," recompute TP/FP/FN/TN.

Where imbalance hides

Rare-disease screening, fraud detection, defect inspection: the majority class quietly dominates accuracy in all three.

Which axis is which

The row/column convention shown here — rows = predicted, columns = actual — is not universal. Many textbooks, and some default `scikit-learn` displays, transpose it: rows = actual, columns = predicted. Before reading TP/FP/FN/TN off any confusion matrix from an unfamiliar source, first confirm which axis is which; the matrix itself is just a transpose either way.

Counts vs. proportions

Raw cell counts are not comparable across datasets: a model scored on 10,000 patients naturally produces bigger TP and FP counts than the same model scored on 100, purely from sample size, not quality. That is exactly why the field derives proportions — accuracy, precision, recall, and the rest — instead of comparing raw counts directly.

The paradox, worked

Take 1000 patients where about 10 actually have a rare disease. A classifier that always predicts "negative," regardless of input, is correct for essentially all 990 healthy patients and wrong for all 10 sick ones — roughly 99% accuracy. The number looks excellent; the model is useless, since it never identifies a single patient the system exists to find. The same pattern recurs anywhere the positive class is rare relative to normal cases: hospital-PACS image pipelines, fraud detection, defect inspection.

One-vs-rest, step by step

For an n-class matrix, pick one class as the target (e.g. spam among normal/ad/spam email). Then: (1) TP = target instances correctly called target; (2) TN = any non-target instance correctly called some non-target class, regardless of which one; (3) FP = a non-target instance wrongly called target; (4) FN = a target instance wrongly called some non-target class. That collapses the n×n matrix back to a familiar 2×2, and every metric in this chapter applies unchanged, one target class at a time.

from: ch08-evaluation.md · session 15

CH 08 · Evaluation · Precision, recall & F-beta

Precision and recall disagree — F-beta breaks the tie

Precision asks "were the alarms real?"; recall asks "did we catch everyone?" F-beta lets you weight whichever error costs more.

Raising the threshold trades recall for precision, and vice versa.

Precision vs Recall $$Precision = \frac{TP}{TP+FP} \qquad Recall = \frac{TP}{TP+FN}$$

Precision: of everything flagged +, how much was real? Recall: of everyone truly +, how many did we catch?

F1 — harmonic mean $$F_1 = \frac{2\cdot Precision \cdot Recall}{Precision + Recall}$$

P=0.9, R=0.1 → arithmetic mean 0.5 (flattering); F1 ≈ 0.18 — missing 90% isn't "medium."

F-beta — tune the weight $$F_\beta = \frac{(1+\beta^2)\cdot Precision \cdot Recall}{\beta^2\cdot Precision + Recall}$$

Beta under 1 favors precision (spam filters); beta over 1 favors recall (diabetes screening, β=2 typical).

Sweeping the threshold

Most classifiers — logistic regression's sigmoid is the canonical case — output a probability in [0,1], and a decision threshold (0.5 by default) converts it to a hard label: predict positive at or above the threshold, negative below it. A lower threshold catches more true positives, raising recall, but lets in more false positives, lowering precision; a higher threshold does the reverse. The same idea reappears in AdaBoost, where each weak learner effectively applies its own internal threshold.

Why harmonic, not arithmetic

The harmonic mean is used deliberately because it is pulled down sharply by whichever input is small. Precision 0.9 and recall 0.1 average to an arithmetic 0.5 — deceptively respectable — but F1 lands near 0.18, correctly reflecting that missing 90% of true positives is not a "medium-quality" outcome no matter how precise the few positive calls are. A high F1 therefore requires precision and recall to both be reasonably good at once, not just one of them.

Choosing beta without a universal answer

F-beta generalizes F1 with a weighting parameter beta: below 1 it weights precision more heavily (use when false positives are the greater concern), at exactly 1 it recovers ordinary F1, and above 1 it weights recall more heavily (use when false negatives are the greater concern). In medical screening, missing a real patient is usually judged worse than a false alarm, so beta = 2 is a common choice. In spam filtering, deleting a legitimate email may be judged worse than letting spam through, favoring beta under 1. There is no universally correct beta — it encodes the relative cost of the two error types for the problem at hand.

Specificity, recall's mirror

Specificity is the negative-class counterpart of recall: TN/(TN+FP), the fraction of true negatives correctly identified. It resurfaces directly in the next slide as the complement of the False Positive Rate, 1 − specificity, which becomes the x-axis of the ROC curve.

from: ch08-evaluation.md · session 15

CH 08 · Evaluation · ROC, AUC & metric choice

One curve, every threshold — then let the stakes pick the metric

Every metric so far freezes one threshold. ROC and AUC score the classifier across all of them at once.

The curve bows toward the top-left as separation improves; the dashed diagonal is a coin flip.

AUC — area under the curve $$AUC = \textstyle\int_0^1 TPR\,d(FPR)$$

1.0 = perfect, 0.5 = random (diagonal), 0 = inverted.

Ranking candidates

No threshold fixed yet? Use AUC to compare RF, LogReg, GB, KNN, SVM, MLP, NB, DT on equal footing.

Choosing for deployment

Diabetes screening: a missed case (FN) costs more than an extra test (FP) — favor recall or F-beta above 1.

Building the curve

Sweep the decision threshold from 0 to 1, and at each value plot a point whose y-coordinate is the True Positive Rate (recall) and x-coordinate is the False Positive Rate, FP/(FP+TN) = 1 − specificity. Connecting the points as the threshold sweeps produces the ROC curve. The diagonal y = x is a classifier with no learned skill at all — equivalent to a coin flip, since at every threshold its TPR equals its FPR.

Toward the top-left corner

A useful classifier's curve bows up and away from the diagonal, toward TPR near 1 and FPR near 0. The more it bows toward that corner, the better the model separates the two classes across a wide range of thresholds — and the less its apparent quality depends on which specific threshold gets chosen later.

Integral in theory, sum in practice

AUC condenses the whole curve into one scalar: in principle the integral AUC = ∫TPR d(FPR); in practice, since real predictions produce only a finite, discrete set of threshold points, it is computed as a summation over the discrete curve, AUC ≈ ΣTPR·Δ(FPR). Because it summarizes performance across every threshold at once, AUC is well suited to ranking candidate models before any deployment threshold has been fixed — once a threshold is chosen for production, precision, recall, and F-beta at that specific operating point become the operationally relevant numbers.

The diabetes case, and the moving target

Pima-style classes are typically imbalanced, so accuracy alone is not trustworthy; a false negative (a diabetic patient told they are healthy) is generally more dangerous than a false positive (a healthy patient sent for an extra test), which argues for prioritizing recall, or F-beta with beta above 1. A classroom comparison across eight classifiers — Random Forest, Logistic Regression, Gradient Boosting, KNN, SVM, a neural network, Naive Bayes, Decision Tree — found that rankings by accuracy, precision, F1, and F-beta at different beta values did not agree with each other: the top model changed depending on which metric judged it. Treat any single decimal accuracy or precision figure repeated secondhand with caution and recompute the exact numbers from your own notebook.

from: ch08-evaluation.md · session 15

CH 09 · Clustering · Distance & scaling

Distance lies when scales don't match — standardize before you cluster

Clustering is often run before supervised modeling, to catch outliers before they contaminate a classifier — but every step reduces to distance, and distance reduces to scale.

Same points, two scalings: unscaled, glucose's 0–200 range swamps BMI's 0–60; standardized, both axes carry equal weight in the distance calculation.

z-score — standardization $$ z = \frac{x-\mu}{\sigma} $$

Rescales every feature to mean 0, std 1 — so no single feature's raw range decides "closeness."

Euclidean — the default norm $$ d(x,y)=\sqrt{\textstyle\sum_i (x_i-y_i)^2} $$

One instance of a norm (non-negativity, homogeneity, triangle inequality) — the family that makes "distance" precise.

Diagnostic before descriptive

The Pima Diabetes dataset already carries a label — Outcome — so clustering it looks redundant at first glance. It isn't: clustering is routinely run before supervised modeling to surface outliers, anomalies, and noise, a standard move in fraud detection, insurance, and healthcare analytics. It also curbs overfitting, since a supervised model trained on unflagged anomalies tends to fit them as if they were signal. In analytics-maturity terms, this is the step from descriptive (what happened) to diagnostic (why it happened) — and diagnostic outlier-hunting can be done statistically (box plots, the IQR rule) or via clustering, which this chapter focuses on.

Four families, one exception

Partitioning — K-Means, K-Means++, K-Modes, CLARA, CLARANS; performance depends on the data's shape.
Hierarchical — agglomerative (bottom-up, e.g. AGNES) and divisive (top-down, e.g. DIANA); can scale poorly on large data.
Density-based — e.g. DBSCAN; suited to irregularly shaped clusters.
Grid-based — common in GIS/spatial applications.

Fuzzy clustering cuts across all four: instead of "hard" membership in exactly one cluster, each point gets a membership degree between 0 and 1, letting it belong partially to several clusters.

What a norm actually requires

A norm $\|\cdot\|$ must satisfy non-negativity, $\|x\|=0 \iff x=0$, homogeneity ($\|cx\|=|c|\,\|x\|$), and the triangle inequality. Euclidean distance (L2) is the familiar default, generalizing $\sqrt{(x_1-x_2)^2+(y_1-y_2)^2}$ to $n$ dimensions, but it is one choice among several — L1, L∞ (Chebyshev), Frobenius — and the choice of norm changes what "similar" means.

Scale first, or lose half your features

A feature with a much larger numeric range dominates distance and drowns out smaller-range features: glucose (0–200) versus BMI (0–60) is one case in this chapter's data; age (18–60) versus income (10M–500M) is another, where clustering would otherwise be driven almost entirely by income. StandardScaler fixes this by z-scoring every feature to mean 0, std 1. It's also worth checking approximate normality — via a histogram, or skewness (≈0) and kurtosis (≈3) — because for standardized, roughly normal data the three-sigma rule applies: about 99.7% of values fall in $\mu\pm3\sigma$, i.e. $[-3,3]$ once standardized, and points outside that range are flagged as likely outliers. Note also that K-Means needs numeric features; nominal categorical data (e.g. hair color) calls for a variant like K-Modes, since means aren't defined for categories.

from: ch09-clustering.md

CH 09 · Clustering · K-Means mechanics

Converged doesn't mean correct — K-Means's local-optimum trap

K-Means minimizes WCSS by definition — the mean is exactly the point minimizing summed squared distance. But the objective isn't convex, so convergence only promises a local minimum.

WCSS always falls as K grows, but the gains flatten fast — the elbow near K=3 is where extra clusters stop paying for themselves.

WCSS — what K-Means minimizes $$ WCSS=\sum_i \|x_i-\mu_{k(i)}\|^2 $$

The mean is exactly the point minimizing summed squared distance — that's why centroid = mean, not median or mode.

local optima — not global

WCSS isn't convex in the centroids, so a bad init can get stuck. Fix: n_init reruns from several random starts, keeps the lowest WCSS.

shape assumption — the blind spot

Needs convex, round, separated, similarly sized clusters. A single outlier can drag a centroid off-center — the mean resists nothing.

The iterative loop

K-Means fixes $K$, initializes $K$ centroids (randomly, or via K-Means++), then alternates: assign each point to its nearest centroid, recompute each centroid as the mean of its current members, and repeat until centroids stop moving.

Why the mean, precisely

The centroid update uses the mean for a concrete reason: the mean is exactly the point that minimizes the sum of squared distances to a set of points. That's a mathematical fact, not a convention — it's what makes the assign/recompute loop monotonically decrease WCSS.

Two caveats hiding inside the guarantee

K-Means is guaranteed to converge, but only to a local minimum of WCSS, since the objective isn't convex — a poor initialization can produce a poor, "stuck" clustering even with the right $K$. n_init mitigates this by rerunning from several random starts and keeping the lowest-WCSS result. Separately, a centroid is just a mean position and need not correspond to any real data point: it represents symmetric, blob-like clusters well, but not elongated, curved, or irregular ones. K-Means therefore assumes compact, roughly convex/round, well-separated, similarly sized clusters, and it is sensitive to outliers, since one extreme point can pull a centroid away from the bulk of its cluster.

Choosing K without ground truth

The elbow method plots WCSS against candidate $K$ values; inertia always decreases as $K$ grows, but early increases in $K$ cut it sharply while later ones barely help — the flattening point is the "elbow." It's a heuristic, not a rule: clear on some datasets, ambiguous on others. The silhouette score is the other main tool: for each point, $a$ = mean distance to its own cluster, $b$ = mean distance to the nearest other cluster, and $s=\frac{b-a}{\max(a,b)} \in [-1,1]$ — near 1 is well-placed, near 0 is boundary, near $-1$ is likely misassigned; the overall score averages $s$ over all points, higher is better. Both metrics are strictly internal: they reflect only the data's structure, not any ground truth, since real unsupervised problems have no true labels to check against. (Silhouette-based selection and the gap statistic were mentioned as further alternatives but not covered in depth.)

from: ch09-clustering.md

CH 09 · Clustering · DBSCAN vs K-Means

Density needs no K — and it calls outliers by name

DBSCAN swaps "distance to a mean" for "density of a neighborhood" — no K to guess, arbitrary shapes allowed, and noise is a first-class citizen.

Two interleaved, non-convex clusters: a straight split (dashed, red) cuts through both; DBSCAN instead grows outward from dense ε-neighborhoods and leaves the lone point labeled noise.

core point

≥ min_samples points within its ε-neighborhood.

border point

Not core itself, but sits in a core point's ε-neighborhood — joins that cluster.

noise point

Neither core nor reachable from one — flagged an outlier, typically labeled -1.

Two parameters, three roles

eps (ε) sets the neighborhood radius; two points are neighbors if their distance is below it. min_samples sets how many neighbors a point needs to count as core. Every point becomes core, border, or noise, and chaining directly-reachable points together (density-connectivity) is what forms a cluster — no centroid involved.

Why shape stops mattering

Because clusters emerge from density structure rather than an assumed geometry, DBSCAN can represent crescents, spirals, and other arbitrarily shaped clusters that K-Means cannot. It is also naturally robust to outliers: they're set aside as noise instead of being forced into a cluster and dragging a centroid toward them.

The tuning tradeoff

Too small an eps means few points clear the density threshold and almost everything gets called noise; too large an eps merges genuinely distinct clusters into one. Good values need domain knowledge and experimentation, and DBSCAN struggles when true clusters have very different densities, since one eps/min_samples setting can't fit all of them at once.

A silhouette gotcha from the student project

If DBSCAN finds only one cluster (or calls most points noise), the silhouette score becomes meaningless. The student project's fix: compute it only when more than one cluster was actually found.

The Pima result

Working from 768 rows × 9 columns (zero-as-missing values imputed with the median, duplicates removed, outliers IQR-flagged), the project withheld Outcome, standardized Glucose and BMI with StandardScaler, chose $K$ via the elbow method, and fit DBSCAN with simplified illustrative eps/min_samples values. Comparing both models' clusters against the true labels afterward — valid only because labels happened to be available for evaluation — showed both algorithms separating a higher-glucose, higher-BMI group that reasonably tracked the real diabetic group: exactly the diagnostic value clustering offers upstream of supervised modeling.

aspect	K-Means	DBSCAN
basis	centroid / mean distance	neighborhood density
cluster count	must specify K	found automatically
outlier sensitivity	high — mean shifts	low — labeled noise
cluster shape	convex, round, similar size	arbitrary
key parameters	K (+ init)	eps, min_samples
main weakness	non-convex shapes; init-sensitive	varying density; needs tuning

from: ch09-clustering.md

CH 10 · Ensembles · Bagging vs boosting

Two failure modes, two remedies — bagging averages away variance, boosting chases down bias

Both turn many weak learners into one strong one — the difference is parallel averaging vs sequential correction.

Same input, opposite architecture: bagging fans out and averages; boosting chains stages and corrects.

bagging — parallel, independent $$\hat{f}(x)=\frac{1}{B}\sum_{b=1}^{B} f_b(x)$$

Random Forest bags trees + samples features per split — bootstrap noise averages out, bias barely moves.

boosting — sequential, corrective

AdaBoost / Gradient Boosting add one stage at a time, each weighted by how well it closes the error still left.

bagging → independent teachers, no coordination boosting → 2nd teacher reteaches only the quiz misses canonical → Random Forest · AdaBoost · Gradient Boosting

Weak learners, combined

Every model studied before this chapter — a regression, a tree, a single network — is fit once and used alone. Ensemble learning instead trains several models and combines their outputs, on the intuition that a group of moderately capable predictors whose mistakes aren't all the same mistake will often beat any single predictor. A collection of weak learners (only somewhat better than chance individually) is combined into a strong learner whose aggregate accuracy exceeds any one member's. Scikit-learn groups its tools into four families: Bagging, Boosting, Voting, and Stacking.

Bagging: parallel and variance-hungry

Bagging (bootstrap aggregation) trains many copies of the same model type, each on its own bootstrap sample — drawn with replacement, generally the same size as the original but with duplicates and omissions. Because each copy trains independently, bagging parallelizes naturally; predictions are then combined by voting (classification) or averaging (regression). Random Forest is the canonical example, adding a further layer of randomness by also sampling features at each split. Since each bootstrap sample overfits its own data slightly differently, averaging cancels most of that sample-specific noise — bagging's effect is variance reduction, largely without changing bias.

Boosting: sequential and bias-hungry

Boosting instead trains weak learners stage by stage, where each new stage is built specifically to correct the ensemble-so-far's mistakes — those samples are effectively up-weighted (or, in Gradient Boosting, represented by residuals) so the next stage concentrates on exactly that error. The final prediction is a weighted combination of stages, not a simple average. Because each model is chosen deliberately to fix what its predecessors got wrong, boosting's effect is bias reduction. The sequential dependency is also its main cost: stages cannot train in parallel.

The teacher analogy

Picture a course split across teachers who never coordinate — no one knows what the others covered or where students struggled. Each does an independent job, and the students' overall command comes from combining what they separately absorbed; no teacher corrects another's shortcomings, so the benefit is purely additive. That's bagging. Now picture one teacher delivering the whole course, a quiz revealing exactly which points students struggled with, and a second teacher brought in only to close that specific gap — a third, if needed, targeting whatever gap remains. Each subsequent teacher's value comes precisely from targeting the residual confusion left by those before. That's boosting.

from: ch10-ensembles.md · session 26

CH 10 · Ensembles · Voting vs stacking

A fixed rule only goes so far — stacking learns how much to trust each model

Voting combines models with a fixed rule; stacking trains a model to learn the combination instead.

Same base models — voting combines them by a fixed rule, stacking lets a meta-model learn the weights.

soft voting — average the confidence $$\hat{y}=\arg\max_c \frac{1}{M}\sum_{m=1}^{M} p_m(c\mid x)$$

Uses each model's probability, not just its label — typically beats hard voting's all-or-nothing count.

hard voting — majority rule $$\hat{y} = \text{mode}\{\hat{y}_1,\dots,\hat{y}_M\}$$

Each model casts one vote for a class label; most votes wins.

stacking — a learned combiner

A meta-model trains on the base models' outputs, learning how much to trust each one — not a fixed rule.

Two ways to vote

Voting is a simpler combination strategy, most common for classification, in which several different models — not necessarily bootstrap copies of one type — each predict on the same input and the final class comes from combining their answers. Hard voting has each model cast a single discrete vote and the class with the most votes wins, a plain majority rule. Soft voting instead has each model output a probability distribution over classes; those probabilities are averaged across models and the highest average wins. Soft voting typically makes better use of the models' confidence than hard voting's all-or-nothing count.

Stacking's meta-model

Stacking goes one step further: instead of a fixed combination rule, it trains a meta-model whose job is to learn the best way to combine several heterogeneous base models' outputs. The base models can be of entirely different types — say a linear model, a tree, and a neural network trained on the same data — and the meta-model is trained on their outputs to learn, from data, how much to trust each one and how to combine them.

When stacking pays off

A learned combiner can outperform any fixed rule precisely when the base models have complementary strengths — each one right about different parts of the input space. If the base models mostly agree or mostly share the same blind spots, there's little for a meta-model to learn beyond what majority vote or averaging already gives you; the payoff from stacking's extra training step comes specifically from that complementarity.

Where this sits relative to bagging and boosting

Bagging and boosting both work by combining many copies of the same model type. Voting and stacking instead assume the base models are already different from one another, and focus purely on the combination step — a fixed rule for voting, a trained one for stacking.

from: ch10-ensembles.md · session 26

CH 10 · Ensembles · Gradient boosting steps

Fit the error, not the target — gradient boosting closes the gap one stage at a time

Each stage doesn't relearn $y$ — it learns what the ensemble so far still gets wrong.

Each stage nudges the ensemble closer to the true value; by stage 3 the residual gap is nearly gone.

the update — add a small correction $$F_m(x) = F_{m-1}(x) + \nu\, h_m(x)$$

Learning rate $\nu$ (0.01–0.3) keeps one stage from overcorrecting — slower, but generalizes better.

init + residual — what each stage sees $$F_0(x)=\bar{y} \qquad r_i = y_i - F_{m-1}(x_i)$$

Start at the mean; every later $h_m$ trains on the residuals, never the raw targets.

base learner → shallow regression tree general loss → residual becomes the negative gradient stopping → fixed $M$ stages or validation error

Step 1 — starting at the mean

Gradient Boosting builds an additive model $F_M(x)$ as a sum of simple base learners, typically shallow regression trees, each fit to correct the ensemble built so far. It begins with a trivial base model $F_0(x)$ predicting the same value regardless of input — for regression, simply the mean of the targets, $F_0(x)=\bar{y}=\frac{1}{n}\sum_i y_i$. Every later stage needs something to compute an error against, so the cheapest possible baseline gets things started.

Steps 2–3 — residuals become the new target

At each subsequent stage $m$, the residual $r_i = y_i - F_{m-1}(x_i)$ measures how wrong the current model is on every training point. A new weak learner $h_m(x)$ — usually a small regression tree — is then trained not on the original targets $y_i$, but on these residuals. This stage's whole job is to predict the ensemble's current error, which is exactly the general boosting mechanism: focus the next stage on what the previous stage got wrong.

Step 4 — the learning rate's job

The new learner's predictions are added to the ensemble scaled by a small learning rate $\nu$, typically between 0.01 and 0.3: $F_m(x) = F_{m-1}(x) + \nu\, h_m(x)$. This prevents any single stage from overcorrecting, trading slower convergence for better generalization — much as a second teacher should refine understanding gradually rather than overwhelm students with one drastic intervention.

Step 5 — from residuals to the general "gradient"

Steps 2–4 repeat for $M$ stages (or until a stopping criterion like validation error is met), giving $F_M(x) = F_0(x) + \nu\sum_{m=1}^{M} h_m(x)$. In the more general treatment, the residual of Step 2 is replaced by the negative gradient of an arbitrary differentiable loss function with respect to the current predictions — the source of the name "Gradient" Boosting — but the mean-squared-error case above recovers exactly the plain residual.

from: ch10-ensembles.md · session 26

Part IV · Chapters 11–14

Deep Learning & the Frontier

11 — Neural networks & deep learning
12 — Transformers & large language models
13 — Agentic AI
14 — Scientific ML: PINNs & KANs (the final project)

CH 11 · Neural networks · Neuron to perceptron

Same shape, different substrate — biology gives the pattern, not the proof

Dendrites, soma, and axon are just relabeled inputs, sum, and output — the math doesn't need the biology to be true.

perceptron — one neuron's full computation $$ z = \sum_i w_i x_i + b, \qquad \hat{y} = \phi(z) $$

$\phi$ is the only thing standing between this and plain linear regression.

parameters vs hyperparameters

$W,b$ are learned by gradient descent. Depth, width, and even the choice of $\phi$ are set by you — nothing in the math picks them in advance.

One hidden layer of these units is an ANN; stack more than one and it's deep learning.

The three-part analogy, precisely

Dendrites correspond to inputs $x_1,\dots,x_n$; the soma corresponds to the processing step — a weighted sum followed by an activation; the axon corresponds to the output, which becomes an input (via a "synapse") to the next layer. This is the generic input → process → output pattern, general enough to describe a factory turning raw materials into goods just as well as it describes a neuron.

What the analogy is, and isn't

It's a pedagogical device that motivates the shape of $z=\sum w_ix_i+b$, $\hat y=\phi(z)$ — not a claim that artificial units faithfully reproduce biochemical neuron dynamics. One place it does carry real weight: repetition strengthens biological pathways (a basketball player becomes reliably accurate only after thousands of reps), and, loosely, this mirrors how repeated weight updates over training let a network "learn" a task at all.

Parameters vs hyperparameters

$W$ and $b$ are learned automatically via gradient descent (Section 11.5) — nothing about their final values is chosen by hand. Hyperparameters — depth, width, and the choice of activation function itself — are set externally and are never touched by the gradient-descent update rule. The course flags activation-function choice specifically as a point students often get wrong: nothing in the math says in advance how many layers, how many neurons, or which $\phi$ a given problem needs.

ANN vs deep learning, and the black-box link

Exactly one hidden layer is conventionally an ANN; more than one is deep learning — "depth" just means stacking more processing between input and output. Because hyperparameters are searched rather than derived, deep learning inherits part of its "black-box" reputation from this gap, though the course treats that characterization as contested rather than settled.

from: ch11-neural-networks.md · session 20

CH 11 · Neural networks · Why nonlinearity matters

Depth without $\phi$ is a mirage — stacked lines are still a line

A composition of linear layers is still linear — nonlinearity is the entire reason depth helps.

sigmoid — squashes to a probability-like range $$ \sigma(x) = \frac{1}{1+e^{-x}} $$

Maps $\mathbb{R} \to (0,1)$ — a natural fit for binary classification outputs.

identity $\phi(z)=z$ — the collapse case $$ \hat{y} = \sum_i w_i x_i + b $$

Exactly multiple linear regression — stacking more such layers changes nothing.

not every nonlinear function qualifies

A usable $\phi$ must be differentiable almost everywhere — e.g. $\tanh$, used in the Sec. 11.6 worked example.

Same two classes, same weights — only $\phi$ decides whether the boundary can bend.

Why nonlinearity is non-negotiable

If $\phi$ is the identity, $z$ passes through unchanged and the whole network reduces to $\hat y=\sum_i w_ix_i+b$ — for a single neuron, exactly multiple linear regression. Composing more "linear" layers changes nothing, since a composition of linear maps is still linear: depth alone buys zero extra expressive power without $\phi$. The activation function's real job is mapping the weighted sum into a nonlinear, generalized feature space where more complex decision boundaries or regression surfaces become representable at all.

Sigmoid, mechanically

$\sigma(x)=1/(1+e^{-x})$ squashes any real input into $(0,1)$, a natural fit for binary classification outputs where the result reads like a probability. $\tanh$ is another common choice, and it's the one used in the worked three-layer backpropagation example of Section 11.6, at both hidden layers.

Not every nonlinear function qualifies

Nonlinearity alone isn't sufficient to be a legitimate activation function — a usable $\phi$ must also satisfy properties such as being differentiable almost everywhere, since gradient-based training (Section 11.5) needs that derivative to exist at essentially every point where it's evaluated.

Connects forward to training

This differentiability requirement is exactly what backpropagation depends on: the chain rule needs a well-defined local derivative at every layer in order to propagate gradients backward from the loss to every weight.

from: ch11-neural-networks.md · session 20

CH 11 · Neural networks · Backprop via chain rule

One chain rule turns $O(n^2)$ into $O(n)$ — that's what makes depth affordable

Backpropagation isn't a separate algorithm — it's how gradient descent affords many layers at once.

Forward once, then walk the chain rule backward — each layer's derivative is computed once and reused, not recomputed per weight.

gradient descent — the update rule $$ W_{\text{new}} = W_{\text{old}} - \alpha\, \nabla_W L $$

$\alpha$ is the learning rate — a hyperparameter, not something descent tunes itself.

loss — matched to the target type

Regression → MAE / MSE (RMSE too). Classification → cross-entropy. Same machinery either way.

backprop — why it's cheap

Finite differences cost $O(n^2)$; reusing each layer's derivative drops it to $O(n)$.

Loss depends on what you're predicting

For regression (continuous targets): $\text{MAE}=\sum_i|y_i-\hat y_i|$ and $\text{MSE}=\sum_i(y_i-\hat y_i)^2$, with RMSE a closely related variant. For classification (discrete/categorical targets): cross-entropy is standard. Regression and classification aren't disjoint families of methods — almost any regressor (tree, forest, neural net) can be adapted to classify and vice versa; what changes is mainly the loss function and the output-layer activation, not the underlying machinery.

Gradient descent is the outer loop

Training is optimization: find $W,b$ minimizing $L(W,b)$. A minimum requires $\nabla_W L=0,\ \nabla_b L=0$, which can't be solved in closed form for a nontrivial network, so it's solved iteratively: $W\leftarrow W-\alpha\nabla_W L$, with learning rate $\alpha$ a hyperparameter controlling step size. Stochastic Gradient Descent — estimating the gradient from one sample or a mini-batch rather than the full dataset every step — is the simplest member of this family and, in practice, one of the most effective; momentum-based SGD and AdaGrad refine the same basic update.

Backprop: reuse via the chain rule

The naive approach — finite differences, perturbing each weight and re-running the forward pass — costs $O(n^2)$ for $n$ weight-like quantities. Backprop exploits the chain rule ($y=f(g(x)) \Rightarrow y'=f'(g(x))\cdot g'(x)$) and works from output back to input. In the chapter's worked three-layer example (two $\tanh$ hidden layers, $W_1,W_2,W_3$): $\partial L/\partial W_3$ is computed directly; $\partial L/\partial W_2$ reuses $\partial L/\partial W_3$'s intermediate derivative composed with layer 2's local derivative; $\partial L/\partial W_1$ reuses both. Because each layer's local derivative is cached and reused rather than recomputed per weight, total cost drops to $O(n)$.

One full training step

(1) forward pass produces $\hat y$; (2) evaluate $L(\hat y,y)$; (3) backprop yields $\nabla_W L,\nabla_b L$ for every layer; (4) apply the gradient-descent update; (5) repeat until the loss is acceptably small.

from: ch11-neural-networks.md · session 20

CH 11 · Neural networks · When depth pays off

Depth is earned, not assumed — four criteria decide when

More parameters need more data and compute — deep learning is a bet, not a default.

PINN — physics as a loss term $$ y''(x)+y(x)=0, \quad y(0)=0,\ y'(0)=1 $$

Governing equation + initial conditions get folded straight into the training loss, blended with whatever data exists.

four criteria — decide before you pick an architecture

small data → classical ML tight compute → classical ML raw pixels → CNN order ≥ 10 → ML / DL

A decade of responses: each stage answers the previous stage's limit.

Four criteria, one bet

Deep learning isn't a strict upgrade over classical ML — the choice depends on the problem's shape. Data volume: deep nets are heavily overparameterized and need large datasets; with a rare cancer subtype and only a handful of recorded cases, classical ML wins because there isn't enough data to fit a deep model's parameters meaningfully. Compute: DL training is expensive and benefits from GPUs; limited hardware favors classical ML. Feature interpretability: when features are already meaningful and hand-engineerable (glucose level, BMI — named tabular columns), classical ML works directly on them; when useful features aren't obvious to a human (raw image pixels), architectures like CNNs earn their keep, because convolutional layers learn to extract features automatically rather than requiring hand-crafting.

A fourth criterion: problem order

Extending the same logic to differential equations: approximating $u$ via a spectral/orthogonal-function expansion, $u(x)=\sum_i a_i\phi_i(x)$ for one independent variable, or $u(x,t)=\sum_i\sum_j a_{ij}\phi_i(x)\phi_j(t)$ for two — each additional independent variable nests another summation, and the coefficient count grows accordingly. The instructor calls the number of independent variables the problem's "order." A low-order PDE like $u_{tt}=u_{xx}$ (order 2) stays competitive for classical numerical methods; as order climbs — 4 with $u(x,y,z,t)$, 10 with ten independent variables — the nested-summation expansion becomes impractical, and that's where ML/DL, which needs no explicit basis expansion, becomes the scalable choice.

PINN: blending physics and data

Physics-Informed Neural Networks, part of the Physics-Informed Machine Learning movement popularized by George Karniadakis's group, sit on a spectrum: fully-known governing equation with zero data at one end, data-only with no known equation at the other. Most real problems sit in between, and PINN's contribution is a loss function that can enforce a governing ODE/PDE directly — the illustrative example is $y''(x)+y(x)=0,\ y(0)=0,\ y'(0)=1$ — alongside or instead of fitting simulated or measured data.

The instructor's own research arc

Starting around 2015: an early attempt to solve the Lane–Emden equation with a neural network, benchmarked against classical methods reaching 35–36 correct digits (later 40–50 with rational/fraction arithmetic) — the network of the time couldn't compete, and the direction was shelved for a while. PINN's emergence revived it. From 2019, the instructor's own work explored an orthogonal-function activation-function architecture built specifically for solving differential equations — usable, though not numerically superior to high-precision classical methods, an honest limitation reported directly. A further line combined reinforcement learning (DQN, then Double DQN) with equation-solving: the group used RL to obtain an approximate closed-form analytical solution for a differential equation with no previously known analytical solution at all — a first-of-its-kind, well-cited result. The group has also explored combining genetic-algorithm optimization with ML-based equation solvers as a further axis.

from: ch11-neural-networks.md · session 20

CH 12 · Transformers & LLMs · Attention mechanics

Every token asks, every token answers — attention is just weighted lookup

Attention replaced recurrence entirely — parallel to compute, no vanishing gradient over long chains. The 2017 paper's title says exactly this.

Q·K learns "it" refers to "cat" — a relation static embeddings never see.

Q, K, V — scaled dot-product attention $$\text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

Query = what I'm looking for; Key = what I offer; Value = what I contribute if picked. Softmax turns raw scores into weights summing to 1.

cost — quadratic in context length

$QK^\top$ is $N\times N$ for context length $N$. An "8K-context" model computes an 8000×8000 matrix every pass.

multi-head — parallel specialists

GPT-2 runs 12 heads per layer, each free to specialize: one tracks pronoun reference, another subject–verb structure.

Why dot products, why softmax

Query·Key measures alignment in the space the learned projections define — a large dot product means "this Key answers that Query well." Raw scores range over all reals, so softmax rescales them into a proper distribution: positive weights summing to 1. The output at each position is then a weighted average of every Value vector in the sequence, weighted by how relevant each Key was. It's the same compare-and-retrieve idea a database index uses, except here the comparison itself is learned end to end rather than hand-built.

The canonical example

"The cat sat on the mat because it was tired." Static word embeddings alone put almost no similarity between "it" and "cat" — unrelated words in raw embedding space. But the trained attention matrix assigns "it" a strong learned score toward "cat" specifically, resolving the pronoun through context rather than surface form. That contextual resolution — impossible for a fixed embedding lookup — is the entire point of attention: meaning here depends on where a token sits relative to the others, not just what the token is.

Multi-head as division of labor

A single Q/K/V projection can only capture one kind of relationship at a time. Multi-head attention runs several independent projections in parallel and concatenates their outputs, letting each head specialize — noun–noun association, pronoun reference, subject–verb structure. GPT-2 uses 12 heads per layer, stacked across many layers; more heads and layers buy richer relational structure, at proportionally greater compute cost, since each head repeats the full $N\times N$ score computation.

What replacing recurrence actually bought

RNN/LSTM sequence models suffered two compounding problems: vanishing gradients over long sequences, and recurrence forcing token-by-token sequential computation that could not parallelize on modern hardware. "Attention Is All You Need" removed recurrence entirely: with no recurrent chain there is no vanishing gradient over long spans, and because every token's attention computation is independent of every other token's, the whole sequence processes in parallel. The trade it makes instead is the quadratic $N^2$ cost in context length — the reason context-window size remains one of the sharpest practical limits on transformer models today.

from: ch12-transformers-llms.md · sessions 21, 22

CH 12 · Transformers & LLMs · Decoder architecture

GPT can't peek ahead — enforced with a wall of −∞

Masking future positions before softmax is the entire difference between GPT and bidirectional BERT. No new math — just −∞ in the right cells.

Row i (query) may only look at columns j ≤ i (key) — everything to the right is masked to −∞ before softmax.

masked — causal self-attention $$\text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V$$

$M_{ij}=0$ for $j\le i$, $-\infty$ for $j>i$ — future scores vanish to exactly 0 after exponentiating.

GPT vs BERT — decoder vs encoder

GPT: decoder-only, autoregressive, $P(x_t\mid x_{

pipeline — one forward pass

tokenizeembed+ pos-encmask+attendsample

Repeat: append the sampled token, feed the longer sequence back in.

Why mask before softmax, not after

Setting future scores to $-\infty$ before the softmax, rather than zeroing the output afterward, is what makes the trick exact: $e^{-\infty}=0$, so those positions contribute nothing to the weighted sum and the remaining weights renormalize correctly over just the allowed positions. Zeroing after softmax would leave a distribution that no longer sums to 1 over the visible tokens.

Decoder-only vs BERT's bidirectional encoder

GPT is built for exactly one job: predict the next token given everything so far, $P(x_t\mid x_{t-1},\ldots,x_1)$. That autoregressive constraint is what makes it useful for open-ended generation — it can always produce one more token. BERT drops the causal mask entirely and trains on tasks that use both left and right context at once, which makes for better sentence-level understanding but no natural way to generate text left to right.

Perplexity: the pretraining scoreboard

Pretraining fits the decoder by next-token prediction over web-scale text, minimizing cross-entropy loss $L=-\sum_t \log P(x_t\mid x_{

Sampling closes the loop

Once training is done, generation still has to turn a probability distribution into one committed token. Temperature rescales logits before softmax, $P(\text{token}_i)=e^{z_i/T}/\sum_j e^{z_j/T}$: as $T\to0$ the distribution collapses onto the single most likely token; higher $T$ flattens it toward more random, "creative" choices. Top-K keeps only the $K$ most likely tokens before sampling; top-P (nucleus) instead keeps the smallest set whose cumulative probability clears a threshold $P$. The loop — sample, append, feed the longer sequence back in — is what you're watching when an LLM "streams" its answer, and it runs until a learned stop token ends it.

from: ch12-transformers-llms.md · sessions 21, 22, 24

CH 12 · Transformers & LLMs · Tokenization

Equations pay a tokenizer tax English never sees — and embeddings can't fix it

Byte-level BPE handles any script uniformly — but LaTeX still shreds into noise. Backslashes and braces fragment into near-meaningless tokens.

Byte-level BPE runs this same merge process on raw bytes, so any script fits the one scheme.

tokenize — text to integer IDs $$\text{text} \rightarrow (t_1, t_2, \ldots, t_n), \quad t_i \in \{1,\ldots,|V|\}$$

Word-level (fragile to typos) → BPE merges frequent byte-pairs (GPT-1–3.5) → byte-level BPE today, any script uniformly.

real counts — live tiktokenizer demo, GPT-2 → GPT-4o

Persian greeting: 26 → 8 tokPython snippet: 17 → 15 tok

GPT-2's mostly-English vocabulary pays dearly for Persian; Chinese, by contrast, compresses heavily.

bare symbols — no inherent meaning

$x, y, u, v$ aren't like "king" — their meaning is entirely local, fixed only by the equation they sit in.

Three eras of tokenizer

Word-level tokenizers (one token per word) are brittle to any typo or unseen word. Byte-pair encoding (BPE) instead breaks words into small character chunks and iteratively merges the most frequent adjacent pair by corpus frequency, and carried GPT-1 through GPT-3.5. Today's state of the art goes one level lower still: byte-level tokenization converts text to raw 8-bit bytes first and merges happen at that level, so any language or script is handled uniformly without a language-specific vocabulary.

What the live demo actually showed

Loading GPT-2 directly and comparing against GPT-4o's tokenizer gave real, not illustrative, numbers: a short Persian greeting took 26 GPT-2 tokens but only 8 under GPT-4o's more modern multilingual tokenizer. A short Python snippet was closer (17 vs 15), and Chinese text was reported as unusually well "compressed" — few tokens carrying a lot of meaning. Math and LaTeX expressions were the worst case discussed: backslashes and braces are exactly the kind of low-frequency byte sequences that never earn efficient merges, so an equation fragments into many small, close-to-meaningless tokens before the model ever sees a symbol.

Why embeddings don't rescue bare math symbols

Embedding training is self-supervised: a model sees the same corpus repeatedly through a sliding window and learns a token's meaning purely from its distributional neighbors — no labels needed. That works beautifully for a word like "king," which behaves consistently across a huge corpus. It works far less well for a variable like $x$ or $u$: the same symbol means something completely different in every equation it appears in, so there's no stable distributional signal to learn the way there is for natural-language words. This is the real reason natural-language embedding techniques are considered inadequate, or at least unproven, for mathematical expressions.

An unsolved lever: tokenizing math on purpose

Three strategies were floated for fixing this at the tokenizer level rather than papering over it downstream: learning a tokenization scheme via context-free-grammar/tree representations combined with metaheuristics and reinforcement learning; prompt-engineering workarounds when the tokenizer can't be touched at all (reformatting an equation as code, adding a "you are a mathematician" persona, or switching to a SymPy-like/RPN surface format instead of raw LaTeX); and editing only the tokenizer of an open-source model — adding special math tokens or merging useful pairs — then fine-tuning (full or LoRA) so general world knowledge survives. None of the three is settled; even whether "12.13" tokenizes as "12"+"."+"13", as one token, or digit-by-digit measurably changes downstream behavior.

from: ch12-transformers-llms.md · sessions 21, 24

CH 12 · Transformers & LLMs · Prompting, RAG & calibration

Sharper reasoning, same blind spot — confidence outruns correctness

Fine-tuning bakes in a reasoning style; RAG retrieves a fact instead of memorizing one. Neither teaches a model to say "no solution."

the benchmark problem — Fredholm, second kind $$y(x) = f(x) + \lambda \int_a^b K(x,t)\, y(t)\, dt$$

~500K symbolic base equations, plus ~60 hand-designed edge cases across 14 augmentation strategies.

chain-of-thought — same model, more tokens

A "thinking budget" caps reasoning tokens; GPT-5 at medium effort clearly beat GPT-4o on the instructor's own DE benchmark.

RAG — retrieval, not memorization

Finds a similar solved problem by vector similarity and supplies it in-context — no weight update, unlike CoT fine-tuning.

GPT-5.2 aces "exact"/"family" types (100%) but never flags a true "no solution" case; GPT-4o-mini sits near 0% overall.

Chain-of-thought is not a new architecture

Narrating intermediate steps instead of answering directly measurably improves quality on step-by-step, math-like problems. Mechanically this is nothing new: no attention patch, no architecture change — fine-tuning on human-written reasoning traces teaches the same autoregressive decoder to generate a "thinking" block before committing to a final answer, effectively a deeper search within its own semantic space before it speaks. A thinking budget (e.g. 256 tokens) caps how far that search runs; accuracy generally rises with budget but not perfectly monotonically — one documented case found 1,000 tokens helped while 1,500 hurt. Vendors expose this as "low/medium/high effort," and the instructor's own differential-equation benchmark found GPT-5 at medium effort clearly beating GPT-4o (no thinking process at all) — plausible given how well an algorithmic, step-by-step CoT style matches solving a differential or integral equation by hand.

RAG changes what the model knows, not how it thinks

Fine-tuning on reasoning traces changes model weights and thinking style. RAG is a different lever entirely: given a new problem, it retrieves a similar already-solved problem via vector similarity search and supplies it as in-context learning material — no weights touched at all. The two are complementary, not competing: one shapes how the model reasons, the other supplies what it can reason with.

The Fredholm benchmark, and what broke

A student project benchmarked GPT-4o-mini, GPT-4o, GPT-5, and GPT-5.2 on second-kind Fredholm integral equations, $y(x)=f(x)+\lambda\int_a^b K(x,t)y(t)\,dt$, using a roughly 500,000-equation symbolic base plus about 60 hand-designed edge cases across 14 augmentation strategies — deliberately including unsolvable and regularization-requiring problems. Outputs were regex-cleaned, standardized through Math-Verify, and checked for symbolic equivalence via SymPy after expansion, so $x^2-2x+1$ and $(x-1)^2$ count as the same answer. GPT-4o-mini scored near 0% symbolic accuracy; across all tested combinations, overall symbolic accuracy sat around 15%.

Confidence and correctness pulled apart

GPT-5.2, the most advanced model tested, was 100% correct on "exact" and "family" solution types — but 0% correct on "no solution" cases, never once recognizing an unsolvable problem and instead always producing a plausible-looking answer. GPT-4o, less sophisticated overall, was better calibrated: it correctly flagged roughly a third of the true "no solution" cases. The most capable model was also the most prone to confidently hallucinating an answer — a clean, concrete instance of overconfidence. An agentic loop that tries, checks its own residual, and iterates was floated as a possible next step, but whether that closes the calibration gap is left open.

from: ch12-transformers-llms.md · sessions 22, 24

CH 13 · Agentic AI · Capability lineage

Suggestion becomes action — the agentic layer closes the loop

Every layer below still just ranks the next token; only the top layer decides and acts on that decision.

next-token prediction — the generative core $$ \hat{w}_{t+1} = \arg\max_{w} P(w \mid w_1, w_2, \dots, w_t) $$

This is where a plain generative model stops — ranking words, never dialing an API.

Agentic AI — definition

Autonomously search, decide, and act on that decision — not just propose an answer for a human to execute.

The five rungs, one more level

Classical AI is rule-based reasoning; Machine Learning replaces hand-written rules with patterns learned from data; Deep Learning specializes ML into multi-layer neural networks; Generative AI is the current wave built on those networks, and it does one thing at its core — predict the highest-probability next token given everything so far. Agentic AI is not a bigger model sitting at the top of this stack; it is a different kind of layer, built on top of a generative model, that closes the loop between suggestion and action.

Why the argmax is inherently passive

The formula $\hat{w}_{t+1} = \arg\max_w P(w \mid w_1,\dots,w_t)$ describes a model that ranks candidate words and outputs the best one. That is genuinely powerful, but it is a language operation — it cannot itself call an API, execute a database migration, or complete a purchase. A generative model, left alone, only ever suggests text; nothing forces it to be executed.

The flight-booking test case

The chapter's running example makes the contrast concrete. A user states a goal: "book me a flight to country X." A purely generative system would describe how to book a flight — steps, tips, maybe a script. An agentic system instead executes: it searches for flight options between the stated origin and destination, gathers and compares prices across the results, selects the best price, reserves a hotel if the trip requires one, and completes the booking end to end, with no human performing any of the intermediate steps. The economic stakes are not hypothetical — agentic automation is already cited as a driver of new job growth in finance, accounting, and supply-chain roles across US and Canadian markets.

No-code arrives after the fact

Because building agentic systems still requires programming skill, platforms such as n8n, Make.com, and Zapier let non-programmers — a finance professional, for instance — assemble agentic automations visually instead of in code. None of these products originated with LLMs; they are existing workflow-automation tools that were adapted to plug into language models once agentic AI became commercially relevant.

from: ch13-agentic-ai.md

CH 13 · Agentic AI · MCP permissioning

From free-form scripts to permissioned calls — MCP fences in what an agent may do

LangChain taught models to call tools; MCP decides which calls they're allowed to make.

LangChain — chaining tool calls

Developer wires each step to a real action: search API → compare → booking server → payment.

MCP request — tool, arguments, action

tool: "shell"arguments: {cmd}action: "run"

The server executes and enforces what's allowed — the model only asks.

Local Agent Kit — a cautionary run

A local dev run triggered a DB migration against production and took it down.

Same call-and-return shape as ping or telnet — MCP just adds a gate.

Before MCP: hand-wired chains

Early generative models could already infer the right plan for the flight example — search, then compare, then reserve, then pay — but inferring a plan is not the same as executing one. Nothing in the model itself could call the search API, hit a booking server, or trigger a payment step. LangChain closed that gap by letting developers explicitly chain a model's plan to real actions: if the current step is search, call the search API; if it is booking, call the booking server; then hand off to payment. Every tool call in this era was a bespoke integration, written and wired by hand.

The riskier variant: a script per step

An even more manual pattern discussed in the session had the model asked to write a fresh script — a multi-line program — every time it needed to use a tool, run that script, observe the output, and repeat the cycle for the next tool use. It worked, but it was repetitive, hard to audit, and gave the model considerable latitude over what code actually got executed. That latitude is exactly the problem MCP was built to remove.

What an MCP request actually contains

A request to an MCP server is a structured JSON body with, typically, three parts: the name of the tool to invoke (e.g. "shell"), a set of arguments for that tool (e.g. the exact command to run), and an action field (e.g. "run"). The server executes the specified action and returns the output — conceptually no different from a command-line tool like ping or telnet. What MCP adds on top is a permissioning layer: the server can verify and limit what the model is allowed to do, and access to a resource such as a database can be scoped down to specific collections, tables, or columns, or granted read-only while withholding write access.

The Local Agent Kit incident

One contributor, testing his own open-source project ("Local Agent Kit"), had a local development run accidentally execute a database migration against the production database, taking production down. His own assessment was that a properly configured MCP server with restricted permissions would likely have prevented it, since the migration action would simply not have been available to the model. It is reported here as a first-hand, unverified anecdote — the motivating case for MCP-style permissioning over trusting a model to always generate safe ad hoc code.

from: ch13-agentic-ai.md

CH 13 · Agentic AI · Multi-agent systems

Specialists beat one generalist — split the project, then recombine

Sub-agents converge the way a human team does: each masters one slice, then they compare notes.

Decomposition — one project, many specialists

Split the work across sub-agents; each executes its slice independently, then the outputs combine into one coherent result.

Concrete specialties

DB schemaAbstract Factory patternoptimized algorithm

Each output mirrors what a human specialist on the team would propose.

Breadth over depth

A second axis along which agentic systems scale — distinct from the MCP permissioning axis — is breadth of specialization. Rather than asking one generalist agent to do everything, a project can be split among several specialized sub-agents, each responsible for one slice of the work — for example, one sub-agent focused on database architecture, another on algorithm optimization, and a third on backend or system design. A single generalist agent has to context-switch between all of these concerns at once; a team of specialists never has to.

The five-step procedure

The chapter lays out the pattern as a concrete sequence: (1) describe the overall project to the agent team; (2) split the work among specialized sub-agents; (3) each sub-agent independently executes its assigned portion — proposing a database schema, recommending a design pattern such as Abstract Factory, or recommending an optimized algorithm for a given step; (4) sub-agents communicate their outputs to one another; (5) the combined result is a single, coherent, well-organized project. The order matters — steps 3 and 4 only produce a usable result because step 2 already drew clean boundaries between specialties; skip the split and the "specialists" are just one agent wearing different hats in sequence.

Where this already exists

This pattern maps directly onto sub-agent features already available in coding-agent tools such as Claude Code and comparable "agent-team" tools. The appeal is the same one that motivates specialization among human engineers on a team: a database-design sub-agent produces a better schema than a generalist would, and letting sub-agents exchange their outputs lets the specialized pieces converge into an integrated whole rather than a set of disconnected fragments.

What the sub-agents actually hand each other

The chapter's own examples are concrete rather than abstract: the database-focused sub-agent proposes a schema; the algorithm-focused sub-agent recommends an optimized approach for a specific computational step; a design sub-agent recommends a structural choice such as the Abstract Factory pattern for how components get instantiated. None of these are generic placeholders — each is the kind of decision a human specialist on a real engineering team would be the one to make, which is precisely the point of assigning it to a dedicated sub-agent instead of a generalist.

from: ch13-agentic-ai.md

CH 13 · Agentic AI · Skills paradigm

Prompts you can actually reuse — one skill, one job, loaded on demand

Only headlines sit in context up front; the agent reads a skill's full text only when it needs it.

Skill — one markdown file, one job

Authored, shared, and version-controlled like any other file in the repo.

Unix philosophy, applied

Small, single-purpose, composable — complex behavior emerges from combining many skills, not one giant prompt.

Only headlines sit in context up front; full instructions load on demand.

From hand-built pipelines to skills

A year earlier, agentic systems were typically built as multi-step pipelines using frameworks such as LangChain, Pydantic AI, and Agno: each stage carried its own prompt, wired together by the developer. Two problems recurred. First, multi-stage pipelines were hard to debug, since a failure could originate in any stage's prompt or in the wiring between stages. Second, and more fundamentally, the prompts were not composable — a well-crafted prompt built for one stage of one pipeline could not easily be lifted out and reused inside a different pipeline built for a different purpose.

What a skill is

A skill is a single markdown file containing the prompt or instruction set needed to perform one specific task. A skill can reference anything it needs internally, and — because it is just a file — it can be authored, shared, and version-controlled through an ordinary repository. Instead of designing a bespoke pipeline architecture, a developer hands a single agent (Claude Code, OpenAI Codex, Google Antigravity, or Cursor are all candidates for "the one agent") an entire skill set, and the agent itself decides, at run time, which skill or skills are relevant to the task at hand.

Lazy loading, step by step

The mechanism that makes this scale without overwhelming the model is a lazy-loading discipline: author each capability as its own skill file with a short headline of roughly 200–250 characters; register only the headlines with the agent up front, never the full instruction text of every skill; at each step the agent scans the available headlines to judge which skill matches the current sub-task; only when a skill is actually needed does the agent load — read in full — that skill's markdown file; the agent follows the now-loaded instructions and repeats the cycle for the next step. Loading every skill's full prompt at once would bloat the context window and measurably degrade accuracy, even for skills irrelevant to the current step.

Worked example: an explainer-video skill set

The motivating problem was generating short, under-one-minute explainer videos on an AI/ML/DL or finance/health/business concept for an audience without a CS background — plausibly needing distinct skills for narration, pre-narration planning, content understanding, and invoking a video tool such as ffmpeg or Remotion. A real repository shared during the session, "Claude Code Video Toolkit," contained individual skills for narration, script/story generation, image and asset generation, composition, rendering, and audio generation; its demo pipeline — narrate → generate script → generate assets → compose → render — was produced not by a hand-engineered pipeline but by a single agent deciding, skill by skill, what to invoke next. The intended analogy is explicit: Skills mirror the Unix/Linux philosophy of small, single-purpose, composable tools.

from: ch13-agentic-ai.md

CH 14 · Scientific ML · Inductive bias

Physics is inductive bias — trade data for structure

Every learner sits somewhere between raw data and a governing equation — PINNs lean deliberately toward the equation.

PINNs sit near the physics end but keep one foot in data — that's the hybrid position.

inductive bias — the axis $$\text{data volume} \;\;\longleftrightarrow\;\; \text{structural prior (physics)}$$

Every learner sits on this line. PINNs sit deliberately toward the physics end.

physics-informed = hybrid

Equation and data both available — trained to satisfy the equation while optionally fitting whatever data exists.

why the shift, now

Same three drivers as the wider DL boom: more data, cheaper GPU/TPU compute, maturing algorithms.

Five stages, one continuous trade

The chapter traces a five-stage progression: analytical methods (Laplace transform, operator method, characteristic equations) solve textbook problems exactly but fail on nonlinear or high-dimensional systems; classical numerical methods (Euler, Runge–Kutta, multistep) trade exactness for approximations, now GPU-accelerated; weighted-residual methods (collocation, Galerkin, least squares), active since roughly the 2000s, expand the solution in orthogonal basis functions and fit coefficients to a residual; machine-learning methods from 2011 (LS-SVR) train a kernel regressor to satisfy the equation; and finally, in 2019, Karniadakis and collaborators formalize Physics-Informed Neural Networks. Each stage trades a little more exactness for a little more generality than the one before it.

What inductive bias actually means

Inductive bias is the set of structural assumptions built into a model, as opposed to what it must infer from data volume alone. Large language models sit at one extreme — almost no structural assumptions, enormous data. A model handed the governing differential equation sits at the other — it can learn accurately from very little labeled data, or none at all. Every learning system occupies some point on this single axis.

Physics-informed = hybrid, not either extreme

The paper-review session drew an explicit contrast between two classical modeling philosophies: purely physics/equation-driven (derive and solve the governing equation, no data at all) and purely data-driven (common in experimental physics — interpolation, symbolic regression, or curve fitting recovers the relationship without ever writing the equation down). Physics-informed methods are the hybrid: both an equation and some data are available, and the network is trained so its output respects the equation through a residual loss term while optionally also fitting the data. Data alone is usually too scarce or noisy to pin down a nonlinear PDE's solution; physics alone, without learned flexibility, just collapses back into classical numerics.

Same lineage, a different clock

A complementary era-based timeline names the same story by decade: finite differences (1950s–60s), finite element methods (1960s–70s), spectral methods with orthogonal polynomial bases (from ~1977/1980s), meshless methods (~2000), and now physics-informed ML layered on top. Jacobi, Chebyshev, Legendre, and Gegenbauer bases recur throughout the rest of this chapter for exactly this reason — the same toolkit that anchored spectral numerical analysis for four decades reappears as PINN collocation grids and, later, KAN edge functions.

from: ch14-scientific-ml.md · §14.1–14.2

CH 14 · Scientific ML · PINN formulation

Autodiff turns the network into the solution

No labeled solution data anywhere — only a physics residual and an initial-condition penalty.

residual + composite loss $$r(x)=\mathcal{N}[u_\theta](x)-f(x)$$ $$L(\theta)=\lambda_1 L_{\text{res}}+\lambda_2 L_{\text{IC}}$$

No exact-solution data anywhere — only the equation and the conditions that pick one solution.

worked example — u'(x)=cos(x)

MLP $1\to32\to32\to1$, tanh, Adam; 40 points, requires_grad=True; autograd.grad(u,x,create_graph=True).

result

3000 epochs, loss ↓ to ~$10^{-5}$; tested on 200 new points → MAE ≈ 3.38×10⁻⁴ vs. exact $\sin(x)$.

The mechanical trick: autodiff as the derivative engine

Because the network is a fixed, differentiable function of its input, automatic differentiation computes exact derivatives of its output with respect to $x$, to arbitrary order, with no finite-difference approximation. This is what makes the whole approach work: the network becomes the candidate solution, and its derivatives — needed to evaluate the differential operator — fall out of the same computational graph used for backpropagation.

Reading the loss term by term

Given an operator equation $\mathcal{N}[u](x)=f(x)$, the residual is $r(x)=\mathcal{N}[u_\theta](x)-f(x)$. Training minimizes a composite loss over collocation points plus points enforcing the initial/boundary conditions: $L=L_{\text{residual}}+L_{\text{IC}}(+L_{\text{BC}})$, or a weighted form $L=\lambda_1 L_{\text{residual}}+\lambda_2 L_{\text{IC}}$. Several PINN variants, such as "Self-Adaptive PINN," tune $\lambda_1,\lambda_2$ themselves — sometimes together with the distribution of the collocation points.

Hyperparameters: still a black box

In the instructor's words, hyperparameters remain essentially a black box, tuned by trial and error rather than derived from theory: layers and neurons per layer, activation ($\tanh$ vs. sigmoid vs. ReLU), optimizer (Adam by default), learning rate, the number and spatial distribution of collocation points, training epochs, and the loss weights above. None has a fixed prescription.

Inside the $u'(x)=\cos(x)$ demo

The live-coding demonstration used domain $[0,1]$, IC $u(0)=0$, exact solution $\sin(x)$ kept only for evaluation. The PyTorch implementation: an nn.Module with architecture $1\to32\to32\to1$ and tanh activation; Adam optimizer; 40 collocation points with requires_grad=True; each step forward-passes for $u$, computes $du/dx$ via torch.autograd.grad(u,x,create_graph=True)[0], forms the residual and IC loss, sums, backpropagates, and steps. Over 3000 epochs the loss fell steadily to the order of $10^{-5}$ — 1000 epochs would likely have sufficed, but 3000 gave a clean demonstration. Evaluated on 200 new collocation points distinct from training, the final MAE was about $3.38\times10^{-4}$, with predicted and exact curves overlapping closely.

from: ch14-scientific-ml.md · final session

CH 14 · Scientific ML · Validation

Low residual loss isn't proof — trust, but verify

Van der Pol's limit cycle stress-tests a PINN; QLM supplies an independent second opinion.

Van der Pol — nonlinear stress test $$y''-\mu(1-y^2)y'+y=0$$

Self-sustained limit-cycle oscillations — a network can't just memorize a sinusoid.

chebyshev vs. uniform

2nd-kind Chebyshev collocation nodes beat uniform spacing — but the gain was described as "very marginal."

QLM cross-check

Linearize $(1-y^2)y'$ around the current iterate (Newton-type) → sequence of linear ODEs → iterate to convergence.

Why Van der Pol is a real stress test

A simple linear ODE is fine for a pedagogical demo, but validating a PINN requires a genuinely difficult nonlinear case. Van der Pol's second-order nonlinear equation produces strong, self-sustained limit-cycle oscillations — unlike a simple exponential or sinusoidal solution, its trajectory does not settle into a shape a small network can memorize trivially, making it a nontrivial stress test.

Collocation node choice: Chebyshev vs. uniform

The student project solved Van der Pol with a PINN using Chebyshev collocation points of the second kind rather than uniformly spaced points — a direct callback to the spectral-methods lineage where node choice has always mattered for conditioning and convergence. The tentative finding was that Chebyshev nodes gave better accuracy than uniform nodes, though the improvement was "very marginal" — feedback was to state the conclusion more precisely rather than leaving it qualitative.

QLM as an independent anchor

Because Van der Pol has no simple closed-form solution to check against, the Quasi-Linearization Method (QLM) was recommended as cross-validation: linearize the nonlinear $(1-y^2)y'$ term around the current iterate (a Newton-type linearization), converting the nonlinear ODE into a sequence of linear ODEs, each depending on the previous iterate's solution; solve each linear ODE (e.g. via collocation) and iterate until the sequence converges. The converged QLM solution serves as an independent check against the PINN's output.

The general lesson

This pairing generalizes: when a governing equation is nonlinear and no exact solution exists, a classical linearization scheme (QLM here; Newton's method or perturbation methods elsewhere) supplies an independent numerical anchor, rather than trusting the residual loss alone. A low training loss only shows the network satisfies the equation at the sampled collocation points — it does not rule out convergence to a spurious solution, especially for an equation with Van der Pol's rich dynamical structure.

from: ch14-scientific-ml.md · paper-review session

CH 14 · Scientific ML · KAN → PIKAN → project

Edges learn now, not nodes

Swap MLP for KAN inside an unchanged PINN loop — the edge-function choice alone can decide the winner.

Kolmogorov–Arnold vs. universal approx. $$f(x)\approx\sum_i w_i\,\sigma(w_i x+b_i)\;\;\text{(MLP)}$$

KAN keeps the sum but makes each edge's univariate function learnable instead.

PIKAN = PINN with KAN swapped in

Training loop, residual, and loss stay unchanged — only the network block swaps MLP → KAN.

allen–cahn case study

Jacobi-basis PIKAN beat both a spline-KAN and a plain PINN (64×64, tanh) on $L^2$ residual error.

Nodes vs. edges: two places to put the learning

A standard MLP places learnable weights and biases on nodes, with a fixed nonlinear activation applied at each; the Universal Approximation Theorem guarantees $f(x)\approx\sum_i w_i\,\sigma(w_i x+b_i)$. A Kolmogorov–Arnold Network inverts this: a learnable univariate function sits on each edge, and nodes simply sum their incoming edge outputs. The Kolmogorov–Arnold representation theorem — an existence theorem analogous in spirit to Weierstrass approximation — guarantees such a decomposition exists, but not a construction; KAN's contribution is turning that existence guarantee into a trainable architecture fit by gradient descent.

Which function goes on the edge

The original KAN construction uses splines. This course's own research lineage instead uses orthogonal polynomial bases — Legendre, Jacobi, Gegenbauer, Chebyshev, or Hermite — yielding a Rational KAN or Jacobi-basis KAN. This extends the same weighted-residual, orthogonal-basis toolkit from earlier in the chapter into the KAN edge-function slot.

PIKAN needs almost no new machinery

Building a Physics-Informed KAN requires no new pipeline: the training loop, residual construction, and loss function are unchanged from an ordinary PINN — only the network block swaps, MLP for KAN, with the differential equation's constraints imposed exactly as before.

Allen–Cahn: basis choice decided the ranking

The Allen–Cahn equation, $u_t=\epsilon^2\Delta u-F'(u)$, is the $L^2$ gradient flow of the Ginzburg–Landau free energy — a canonical nonlinear reaction-diffusion stress test. A student paper compared a PINN baseline (64×64, tanh) against a spline-basis KAN and a Jacobi-basis PIKAN; the Jacobi-basis network achieved the lowest $L^2$ residual error of the three. A methodological lesson from the review: an early FEM solution used as a pseudo-exact reference was dropped as unsound, since comparing against another numerical solver's output silently substitutes "distance from an approximation" for "distance from the truth" — residual-based and relative-$L^2$ metrics need no reference solution at all.

Final project: "shoot at will"

Since modern LLMs can generate implementation code as well as most students or instructors, the differentiator is no longer typing speed but the creativity of translating a problem's structure into an architecture and loss. Students may combine any technique — deep architectures, classical ML, metaheuristics, or spectral/orthogonal-polynomial methods — atop two structuring threads: the PINN/PIKAN toolkit here, and a transfer-learning experiment across a bank of ~29 source/target ODE pairs (train a source PINN, save its weights, use them to initialize a related target PINN, and compare convergence and accuracy against random initialization). Every submission is judged on MAE and relative $L^2$ error at minimum.

from: ch14-scientific-ml.md · session 6 (paper review) & §14.8–14.10

RECALL ←→ move · T contents · R recall · M more 1/1

AdvancedMathematical Software

Foundations

One word, three meanings — that's what attention buys you

What happened isn't the interesting question — what should we do about it is

Nine columns, one target — but cleaning comes before modeling

Every ML tool is optimization — in disguise

Optimal costs time — Branch & Bound proves it

Parallelism has two independent labels — Flynn, then memory

RAW is the real bottleneck — balance decides the rest

Working with Data

A clean CSV is not clean data — zero can mean "never measured"

Dropping a row throws away seven good features to punish one bad one

Skew fools the Z-score — median-based methods don't bend

Match the chart to the question — not the data to a default

What you plot depends on what you did first — scaling, engineering, imbalance

Zero isn't a measurement — it's missing data in disguise

A heatmap is not a causal claim — IQR finds outliers, RCTs find causes

Variance is the signal — rotate, then drop

Two axes, most of the story — the elbow flattens fast

Neighbors, not axes — t-SNE fights the crowding problem

Pick by the question — not by which method is "better"

Classical Machine Learning

Test accuracy is a statistic — not a certainty

Random doesn't mean representative — structure the draw instead

Fixing the ratio and preserving it are different jobs

One split can get lucky — cross-validation averages that away

No model, just a neighborhood — K decides how big

45% vs 55% — Naive Bayes never hedges

Squash the line, then threshold it — a perceptron before perceptrons

The middle doesn't matter — only the closest points do

99% accurate, 0% useful — the accuracy paradox

Precision and recall disagree — F-beta breaks the tie

One curve, every threshold — then let the stakes pick the metric

Advanced
Mathematical Software