| Formula · derivative | NumPy | PyTorch | |
|---|---|---|---|
| Sigmoid | σ(x) = 1 / (1 + e−x) σ′ = σ(1 − σ) | ||
| Tanh | (ex − e−x) / (ex + e−x) tanh′ = 1 − tanh² | ||
| ReLU | max(0, x) ReLU′ = 1 if x > 0 else 0 | ||
| Leaky ReLU | x if x > 0 else αx, α ≈ 0.01 | ||
| GELU | x·Φ(x) ≈ ½x(1 + tanh[√(2/π)(x + 0.0447x³)]) | *(x+.0447*x**3))) | |
| SiLU / Swish | x·σ(x), used in Llama | ||
| Softmax | exi / Σj exj subtract max(x) for stability | e / e.sum() |
| Formula (mean over n) | NumPy | PyTorch | |
|---|---|---|---|
| MSE (L2) | 1/n Σ (y − ŷ)² | ||
| MAE (L1) | 1/n Σ |y − ŷ|, robust to outliers | ||
| Huber | ½e² if |e| ≤ δ else δ(|e| − ½δ) | np.mean(np.where(a<=d, .5*a**2, d*(a-.5*d))) | |
| Binary CE | −[y log p + (1 − y) log(1 − p)] p = σ(z); ∂L/∂z = p − y | +(1-y)*np.log(1-p)) | entropy_with_ logits(z, y) |
| Cross-entropy | −log softmax(z)y = −zy + log Σj ezj ∂L/∂z = softmax(z) − onehot(y) | ).mean() | |
| KL div. | Σ p log(p / q), distillation | reduction= "batchmean") |
PyTorch's cross_entropy and BCE…with_logits take raw logits (no softmax / sigmoid first): they're fused and numerically stable.
| Normalize inputs | zero mean, unit variance (train stats only) |
| Check initial loss | ≈ ln K for K balanced classes; NaN → lower lr, clip grads |
| Overfit one batch | loss → 0, else there's a bug |
| train() / eval() | toggle dropout & BatchNorm |
| Train ↓ val ↑ | overfit: dropout, decay, more data, early stop |
| Chain rule | ∂L/∂x = ∂L/∂y · ∂y/∂x, layer by layer | |
| Grad check | f′(x) ≈ (f(x + h) − f(x − h)) / 2h, h ≈ 1e-5 | |
| No grad | stop tracking (eval, frozen parts) |
| Update rule (g = ∇θL, η = learning rate) | torch.optim | |
|---|---|---|
| SGD | θ ← θ − η·g | |
| Momentum | v ← μv + g; θ ← θ − η·v | |
| RMSProp | s ← ρs + (1 − ρ)g²; θ ← θ − η·g / (√s + ε) | |
| Adam | m ← β₁m + (1 − β₁)g; v ← β₂v + (1 − β₂)g² m̂ = m / (1 − β₁t), v̂ = v / (1 − β₂t); θ ← θ − η·m̂ / (√v̂ + ε) | betas=(0.9, 0.999)) |
| AdamW | Adam + decoupled decay θ ← θ − ηλθ |
def adamw_step(w, g, m, v, t, lr=1e-3, b1=0.9, b2=0.999,
eps=1e-8, wd=0.01): # NumPy, t starts at 1
m = b1 * m + (1 - b1) * g # 1st moment (mean)
v = b2 * v + (1 - b2) * g ** 2 # 2nd moment (variance)
m_hat, v_hat = m / (1 - b1 ** t), v / (1 - b2 ** t) # bias fix
w = w - lr * (m_hat / (np.sqrt(v_hat) + eps) + wd * w)
return w, m, v
| StepLR | η · γ⌊epoch / s⌋ | |
| Cosine | ηmin + ½(ηmax − ηmin)(1 + cos(πt / T)) | |
| Warmup | linear ramp ~0 → η over first steps | |
| OneCycle | warm up to max_lr, then anneal | |
| On plateau | × factor when the val metric stalls |
| Formula | NumPy | PyTorch | |
|---|---|---|---|
| Accuracy | (TP + TN) / N | .mean() | |
| Precision | TP / (TP + FP), predicted + that are right | tp = (p1 & y1).sum() fp = (p1 & ~y1).sum() fn = (~p1 & y1).sum() P, R = tp/(tp+fp), tp/(tp+fn) f1 = 2*P*R / (P + R) | F1Score(task= "multiclass", num_classes=K) |
| Recall | TP / (TP + FN), real + that are found | ||
| F1 | 2PR / (P + R), for imbalanced data | ||
| Confusion | C[i, j] = #(true i, predicted j) | np.add.at(C,(y,pred),1) | ConfusionMatrix |
| ROC-AUC | area under the TPR-vs-FPR curve | AUROC | |
| R² | 1 − Σ(y − ŷ)² / Σ(y − ȳ)² | / ((y-y.mean())**2) .sum() | R2Score |
| Perplexity | exp(mean cross-entropy) |
| He / Kaiming | W ~ N(0, 2 / nin), for ReLU nets | |
| Xavier | Var = 2 / (nin + nout), tanh / sigmoid | |
| BatchNorm | γ(x − μB) / √(σ²B + ε) + β, stats over batch | |
| LayerNorm | same, stats over features of each sample | |
| Dropout | zero with prob p, scale rest by 1 / (1 − p) | |
| Weight decay | L + λ‖w‖², or θ ← θ − ηλθ (decoupled) |
Backprop is the chain rule for a dense layer Z = XW + b: dW = Xᵀ·dZ, db = Σ dZ, dX = dZ·Wᵀ.
import numpy as np
def softmax(z):
e = np.exp(z - z.max(1, keepdims=True)) # stable
return e / e.sum(1, keepdims=True)
rng = np.random.default_rng(0) # X: [N, D], y: [N]
W1 = rng.normal(0, np.sqrt(2 / D), (D, H)); b1 = np.zeros(H)
W2 = rng.normal(0, np.sqrt(2 / H), (H, K)); b2 = np.zeros(K)
for epoch in range(200):
Z1 = X @ W1 + b1; A1 = np.maximum(0, Z1) # ReLU
P = softmax(A1 @ W2 + b2)
loss = -np.log(P[np.arange(N), y]).mean()
dZ2 = P.copy(); dZ2[np.arange(N), y] -= 1 # dL/dz
dZ2 /= N # softmax-onehot
dW2, db2 = A1.T @ dZ2, dZ2.sum(0)
dZ1 = (dZ2 @ W2.T) * (Z1 > 0) # ReLU'
dW1, db1 = X.T @ dZ1, dZ1.sum(0)
for p, g in [(W1, dW1), (b1, db1), (W2, dW2), (b2, db2)]:
p -= lr * g # SGD step
model = nn.Sequential(nn.Linear(D, H), nn.ReLU(),
nn.Dropout(0.1), nn.Linear(H, K)).to(dev)
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=1e-3, total_steps=E * len(train_dl))
for epoch in range(E):
model.train() # dropout / BN on
for xb, yb in train_dl:
xb, yb = xb.to(dev), yb.to(dev)
loss = loss_fn(model(xb), yb) # logits in
opt.zero_grad() # grads accumulate!
loss.backward() # autograd = backprop
nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step(); sched.step()
model.eval(); correct = 0
with torch.no_grad():
for xb, yb in val_dl:
pred = model(xb.to(dev)).argmax(1)
correct += (pred == yb.to(dev)).sum().item()
acc = correct / len(val_dl.dataset)