{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  },
  "lecture_migration": {
   "project_id": "c0b185e7-d7e3-48b6-a605-8f1acdfe243c",
   "notebook_id": "85eeb3d8253b47bd8b9504b9e226b843",
   "destination": "L2",
   "cloud_run": false
  },
  "title": "LAB 05 · 经典架构与视觉表示"
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "1813ef285e1f48208a21540f2411d304",
     "type": "text-cell-h1",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-1"
   },
   "source": "LAB 05 · 经典架构与视觉表示",
   "id": "1813ef285e1f48208a21540f2411d304"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "c66abc7e6884498e8202b8d4fc79099e",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-2"
   },
   "source": [
    "[返回第二讲 · 经典结构的按需实现](https://codingai-lec02.pages.dev/#/architecture-inception-branches/0) · [第二讲导读与学习材料](https://codingai-lec02.pages.dev/course/guide.html) · [问卷作业 ↗](https://tally.so/r/VLroVv)（右键在新标签页打开）。\n",
    "\n",
    "**经典架构的按需实现与论文读图参考。** 若课堂要动手，先选 [第二讲 LAB03 · 架构实验室](https://deepnote.com/workspace/ise3309jingao-5ba66b0a-4283-4f5a-ae06-3f68d3201e99/project/c0b185e7-d7e3-48b6-a605-8f1acdfe243c/notebook/7d24fdb79a14466ab304927c39d1a317?utm_source=openai&utm_medium=mcp&utm_campaign=openaimcp&utm_content=7d24fdb79a14466ab304927c39d1a317&utm_term=get_notebook) 的残差或BN短练；本册用于把某一机制继续查清，不要求全本运行。\n",
    "\n",
    "| 当前问题 | 本册定位 | 可以核对什么 |\n",
    "| --- | --- | --- |\n",
    "| 小核、通道MLP与GAP改变了什么 | VGG、NiN、GAP三节 | 逐层shape、参数、空间信息取舍 |\n",
    "| Inception四支能否合并 | “GoogLeNet：并行尺度与计算预算” | 各路空间尺寸一致，输出通道求和 |\n",
    "| BN与残差各有哪些条件 | “BatchNorm”到“加法要求形状相容” | 模式开关、缓冲统计、投影与后激活 |\n",
    "| 只做一个无依赖核对 | 文末“独立核对：先预测，再运行” | 标准库算权重、BN统计与GAP读出 |\n",
    "| 迁移与空间任务怎样承接 | 本册末段接口说明 → [第三讲导读](https://codingai-lec03.pages.dev/course/guide.html) | 复用表示与新任务需求；按导读区分现有材料和待补内容 |\n",
    "\n",
    "**可选短练：** 先运行 `import inspect,copy / torch` 与 `parameters` 的准备块，再只运行 Inception 小节。把实例参数 `(16,32)` 改成 `(32,32)`：这是5×5支路的中间瓶颈变宽、最终输出仍32通道。先预测哪一支参数增加，输出总通道是否改变，再看四条打印记录。它不训练、不比较准确率。\n",
    "\n",
    "连续推理读 [第二讲 CLS04](https://deepnote.com/workspace/ise3309jingao-5ba66b0a-4283-4f5a-ae06-3f68d3201e99/project/c0b185e7-d7e3-48b6-a605-8f1acdfe243c/notebook/52d5359cf8504f069e29cebf107c5674?utm_source=openai&utm_medium=mcp&utm_campaign=openaimcp&utm_content=52d5359cf8504f069e29cebf107c5674&utm_term=get_notebook)；现代骨干与公平比较读 [第二讲 CLS06](https://deepnote.com/workspace/ise3309jingao-5ba66b0a-4283-4f5a-ae06-3f68d3201e99/project/c0b185e7-d7e3-48b6-a605-8f1acdfe243c/notebook/bd49b94c43824786bd67374282a11aea?utm_source=openai&utm_medium=mcp&utm_campaign=openaimcp&utm_content=bd49b94c43824786bd67374282a11aea&utm_term=get_notebook)。"
   ],
   "id": "c66abc7e6884498e8202b8d4fc79099e"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "3da74ed3928a4c078d3b4d0f6e47aeaa",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-3"
   },
   "source": "## 卷积之前的问题：什么图像表示值得保留",
   "id": "3da74ed3928a4c078d3b4d0f6e47aeaa"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "41f585e230804cc2b303b1b9c7a33b8d",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-4"
   },
   "source": "**Filter bank：先把像素变成局部响应。** 原始亮度会随光照变化，边缘方向、纹理频率和局部对比有时更稳定。多方向、多尺度滤波器在各处计算响应，用一组测量描述局部区域；texton 方法还会把响应组合聚成共享的纹理词汇。\n\n这一步已经包含“局部性”和“同一算子用于不同位置”的想法。它的价值在于明确声明希望保留什么变化、抑制什么变化；限制在于滤波器种类、尺度与后续汇聚仍需要设计。局部纹理统计可以帮助识别，却不自动等于物体类别。[Leung–Malik 2001](https://people.eecs.berkeley.edu/~malik/papers/LM-3dtexton.pdf)、[Filter bank 资料](https://www.robots.ox.ac.uk/~vgg/research/texclass/filters.html)",
   "id": "41f585e230804cc2b303b1b9c7a33b8d"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "1dd028d9e96d4c2fb0ead89fe95557ca",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-5"
   },
   "source": "**SIFT 与 HOG：对变化作有针对性的选择。** SIFT 在尺度空间寻找稳定局部点、对齐方向，再描述附近的梯度分布，服务不同视图间的可靠匹配；后来的分类流水线也常密集采样 SIFT，不一定只取关键点。HOG 把局部轮廓转成方向统计，并通过块归一化处理对比变化；Dalal–Triggs 的原始验证任务是行人检测。\n\n二者不是“SVM 算法的一部分”，而是可以交给不同学习器的表示。它们把困难变换转成更稳定的坐标，也会舍弃某些精确空间信息。应问这种舍弃是否符合当前任务，而不只记住缩写。[Lowe 2004](https://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf)、[Dalal–Triggs 2005](https://lear.inrialpes.fr/people/triggs/pubs/Dalal-cvpr05.pdf)",
   "id": "1dd028d9e96d4c2fb0ead89fe95557ca"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "cc68b51d6fe249669eb2883f28ef0fca",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-6"
   },
   "source": "**从局部描述到整张图像：词袋、空间金字塔与分类器。** 把局部描述聚类为视觉词典，再统计每个词出现多少次，就能把数量不同的局部点变成固定长度向量。词袋换来了对精确排列的容忍，但也会把不同布局混在一起；spatial pyramid 在粗细不同的网格内分别统计，补回一部分布局。Fisher vector 进一步聚合描述子相对分布模型的统计偏差。\n\n之后可以在这些表示上训练线性 SVM。末端是线性的，不代表像素到结果的整个系统线性。词典、分布参数与分类权重都可能从数据学习，因而传统路线并非“全部手工、完全不学习”。[Sivic–Zisserman 2003](https://www.robots.ox.ac.uk/~vgg/publications/2003/Sivic03/sivic03.pdf)、[Lazebnik 等 2006](https://slazebni.cs.illinois.edu/publications/cvpr06b.pdf)、[ILSVRC 方法回顾](https://ar5iv.labs.arxiv.org/html/1409.0575)",
   "id": "cc68b51d6fe249669eb2883f28ef0fca"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "8dfda68f33a24312b5275f3b7022d20a",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-7"
   },
   "source": "**DPM 是处理部件形变的检测支线。** 刚性窗口难以同时覆盖姿态与形变；deformable part model 将根模板、部件外观得分和偏离锚点的形变代价组织起来，对隐含的部件位置搜索。它把空间关系建成可训练结构，但也增加了搜索与隐变量优化的复杂性。\n\nDPM 主要解决检测，不宜写成 AlexNet 前分类冠军的统一路线；不过 2012 年 Oxford 确实把 DPM 分数融合到分类系统中，也不能说它与分类完全无关。这段显式空间建模保留为历史参考，不安排为第三讲主讲。[Felzenszwalb 等 TPAMI 2010](https://cs.brown.edu/people/pfelzens/papers/lsvm-pami.pdf)、[ILSVRC 2012 官方提交说明](https://image-net.org/challenges/LSVRC/2012/results.html)",
   "id": "8dfda68f33a24312b5275f3b7022d20a"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "2b95ad8a41974defa1b3450019a3da03",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-8"
   },
   "source": "## 从设计表示走向学习表示：两条并行路线",
   "id": "2b95ad8a41974defa1b3450019a3da03"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "67095480fcff437c9c8835172e6e261e",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-9"
   },
   "source": "**瓶颈在于哪些表示选择能被最终任务调整。** 2010 年分类冠军 NEC–UIUC 使用 SIFT、LBP、非线性编码与 stochastic SVM；2011 年 XRCE 用高维签名、压缩与线性 SVM。它们已是成熟的数据驱动系统，特征工程也持续带来进展。可是当最终预测失败时，并非所有前端表示操作都能直接根据任务误差联合调整。\n\nCNN 固定局部连接、位置共享与组合结构，却让滤波器及层层表示由任务目标学习。它保留了结构先验，并不是纯粹消灭先验。教学可以按“线性模型→设计表示→学习表示→卷积”推理，历史却是并行的：卷积与早期神经网络并不等到 SIFT/HOG 之后才被发明。[2010 官方结果](https://image-net.org/challenges/LSVRC/2010/results)、[2011 官方结果](https://image-net.org/challenges/LSVRC/2011/results)、[CS231n Lecture 5](https://cs231n.stanford.edu/slides/2026/lecture_5.pdf)",
   "id": "67095480fcff437c9c8835172e6e261e"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "40cbe3afe2054d7786b6749ed94f45eb",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-10"
   },
   "source": "**2006：既然已有 BP，为什么还研究逐层预训练？** 能写出深网络并不代表从随机初始化就容易优化。当时的一条尝试，是先让输入本身提供局部表示学习信号，而不是一开始只依靠最终标签穿过全部层。Hinton、Osindero、Teh 用逐层 RBM 训练构造 deep belief network，再以 contrastive wake-sleep 微调；其顶层是无向关联模型，其余部分为有向生成结构。\n\n这一故事同时涉及优化、初始化和利用未标注输入，不应只归结为“没有大数据”。也不能把 DBN 论文的全部微调都改称普通监督 BP。课堂重点是为什么逐层建立表示能给后续学习提供起点，能量函数和采样细节留给生成模型部分继续学习。[Hinton–Osindero–Teh 2006](https://www.cs.toronto.edu/~hinton/absps/ncfast.pdf)",
   "id": "40cbe3afe2054d7786b6749ed94f45eb"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "a8973cea26d64fa7b28f3686da4da59a",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-11"
   },
   "source": "**同年 autoencoder 的路径：预训练后，再优化整体重构。** Hinton–Salakhutdinov 先逐层训练 RBM，再把这些层展开为 encoder 与 decoder，最后以重构误差对整个深 autoencoder 做反向传播微调。这里的目标是从低维编码恢复输入；它展示初始化如何让深层压缩表示变得可学。\n\n这不是 VAE：Kingma–Welling 的 Auto-Encoding Variational Bayes 首稿在 2013 年 12 月，通常对应 ICLR 2014。AlexNet 后来采用的端到端监督训练也不依赖这种逐层 RBM 预训练；比赛里某些模型先用更大 ImageNet 做监督预训练，是另一种用法。遇到“pretraining”要先问其数据、目标和迁移对象。[Autoencoder 2006](https://www.cs.toronto.edu/~hinton/absps/science.pdf)、[AEVB](https://arxiv.org/abs/1312.6114)",
   "id": "a8973cea26d64fa7b28f3686da4da59a"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "24a1b33e7c56492ca4f822c92feef6e3",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-12"
   },
   "source": "## 2012 年的转折：把多个必要条件组织成一个系统",
   "id": "24a1b33e7c56492ca4f822c92feef6e3"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "047097919c184b35a44cb49d81c31e17",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-13"
   },
   "source": "**AlexNet 为什么在那个时点产生转折？** 局部共享的结构、足够多的标签、GPU 实现、ReLU、数据增强、dropout 和 SGD/momentum 共同使深 CNN 能在大规模自然图像上训练。不能把整个突破只归因于多几层，或只归因于有了 GPU。\n\n| ILSVRC 2012 分类提交 | 官方 top-5 test error | 比较边界 |\n| --- | ---: | --- |\n| 最佳非 CNN，ISI 特征与线性分类器融合 | 26.172% | 包含多种表示和得分融合 |\n| SuperVision，仅比赛训练数据的 CNN ensemble | 16.422% | 模型集成，不是单 CNN |\n| SuperVision，使用额外 ImageNet 数据的 CNN ensemble | 15.315% | 额外数据与模型集成共同存在 |\n\nAlexNet 论文另报告单 CNN 的 2012 validation top-5 为 18.2%；这些口径不能混成“单网络 15.3%”。更值得追问的是：此前哪些约束使学习表示不够可行，而这次哪些条件一起改变了？[AlexNet 原文](https://proceedings.neurips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf)、[2012 官方结果](https://image-net.org/challenges/LSVRC/2012/results.html)\n\n![数据、算力与可训练结构共同支撑 AlexNet 的概念图](https://codingai-lec02.pages.dev/art/alexnet-scale.webp)\n\n*概念图（ImageGen）：用数据、计算与网络结构的关系辅助阅读；它不是历史硬件照片或实验测量图。*",
   "id": "047097919c184b35a44cb49d81c31e17"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "696b52c094ad43bc8d00621fcca6ce4b",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-14"
   },
   "source": "## 读论文时先问：它在解决什么限制",
   "id": "696b52c094ad43bc8d00621fcca6ce4b"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "75e3af0c1ae94d968fb482f391f7205a",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-15"
   },
   "source": "卷积给出了局部、共享的参数化，但还没有决定一个完整模型该怎样分配深度、通道、采样和训练状态。经典网络的价值在于把这些问题转化为可计算、可比较的设计。\n\n本本沿着五个问题展开：数据和计算规模扩大后怎样训练；如何规则地增加深度；空间计算与通道计算如何分工；多分支怎样分配预算；深网如何保留已有计算并继续优化。每一节都给出核心实现与它背后的取舍。VGG与GoogLeNet是同期的不同路线，不是必然的单线演进。",
   "id": "75e3af0c1ae94d968fb482f391f7205a"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "7dd1f6d8b6d24c3590313e895684e0ec",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-16"
   },
   "source": "import inspect,copy\nimport torch\nfrom torch import nn\nfrom torch.nn import functional as F\ntorch.manual_seed(20260921)\ntorch.set_num_threads(2)\ndef parameters(module):\n    return sum(p.numel() for p in module.parameters())\nprint(\"PyTorch\",torch.__version__)",
   "id": "7dd1f6d8b6d24c3590313e895684e0ec",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "46ab4c641b604f3b8d6586e8f628eeda",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-17"
   },
   "source": "## AlexNet：让大规模监督学习真正工作",
   "id": "46ab4c641b604f3b8d6586e8f628eeda"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "9b70d7897846487a9074d9c63677e236",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-18"
   },
   "source": "前面的“2012 年的转折”保留官方数据口径，这里只看实现接口：原模型包含五个卷积层与三个全连接层；ReLU、增强、dropout 和 GPU 分工分别改变优化、泛化和计算条件。\n\n下方是小通道教学结构，用于跟踪大核、下采样、卷积/分类头接口；它不复现原论文规模或 ImageNet 结果。先数出卷积层，再检查第一个全连接层接收的形状。\n\n[AlexNet 原论文](https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf) · [D2L 8.1](https://d2l.ai/chapter_convolutional-modern/alexnet.html)",
   "id": "9b70d7897846487a9074d9c63677e236"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "44c005c56f6b4daa83d2100fc4616898",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-19"
   },
   "source": "alexnet_teaching=nn.Sequential(\n    nn.Conv2d(3,16,11,stride=4,padding=2),nn.ReLU(),nn.MaxPool2d(3,2),\n    nn.Conv2d(16,32,5,padding=2),nn.ReLU(),nn.MaxPool2d(3,2),\n    nn.Conv2d(32,48,3,padding=1),nn.ReLU(),\n    nn.Conv2d(48,48,3,padding=1),nn.ReLU(),\n    nn.Conv2d(48,32,3,padding=1),nn.ReLU(),nn.MaxPool2d(3,2),\n    nn.AdaptiveAvgPool2d((2,2)),nn.Flatten(),\n    nn.Dropout(.5),nn.Linear(128,64),nn.ReLU(),\n    nn.Dropout(.5),nn.Linear(64,32),nn.ReLU(),nn.Linear(32,10))\nprint(\"teaching AlexNet output:\",alexnet_teaching(torch.randn(2,3,128,128)).shape)",
   "id": "44c005c56f6b4daa83d2100fc4616898",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "3afc612cb5b64f4789dac673ddb5c546",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-20"
   },
   "source": "## VGG：规则的小卷积堆叠",
   "id": "3afc612cb5b64f4789dac673ddb5c546"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "7072f6e5be02481dadc15b7109ebd4a4",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-21"
   },
   "source": "VGG把模型设计从散落的层配置推进到重复块：若干个3×3卷积和ReLU保持分辨率，随后2×2池化减半空间。通过每个stage的层数和通道数配置，可以系统增加深度，比较同类设计。\n\n在各层通道均为C、stride=1、无中间下采样且忽略bias时，三层3×3理论感受野为7×7，权重数 $27C^2$；单层7×7为 $49C^2$。中间非线性使两者函数不同；即使去掉非线性，有限通道的分解也不能直接称为任意大核的等价替代。\n\n小核比较不意味着整网小：高分辨率中间激活占内存，宽通道增加MAC，原VGG的4096维全连接头也有大量参数。下面按stage组织一个缩小网络，继续跟踪空间、通道和参数。[VGG原论文](https://arxiv.org/abs/1409.1556) · [D2L 8.2](https://d2l.ai/chapter_convolutional-modern/vgg.html)",
   "id": "7072f6e5be02481dadc15b7109ebd4a4"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "8209924e65d14390b0c5d5f569aa4699",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-22"
   },
   "source": "def vgg_block(cin,cout,num_convs):\n    layers=[]\n    for _ in range(num_convs):\n        layers += [nn.Conv2d(cin,cout,3,padding=1),nn.ReLU()]\n        cin=cout\n    return nn.Sequential(*layers,nn.MaxPool2d(2))\nvgg_teaching=nn.Sequential(vgg_block(3,8,1),vgg_block(8,16,2),vgg_block(16,32,2))\nv=torch.zeros(1,3,64,64)\nfor i,stage in enumerate(vgg_teaching,1):\n    v=stage(v)\n    print(\"stage\",i,\"shape\",tuple(v.shape),\"parameters\",parameters(stage),\"activation bytes\",v.numel()*v.element_size())",
   "id": "8209924e65d14390b0c5d5f569aa4699",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "e65d5beae19844b7b22d7ccea21da64b",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-23"
   },
   "source": "## NiN：每个位置都可以运行一个共享的小MLP",
   "id": "e65d5beae19844b7b22d7ccea21da64b"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "770c4b6a111e47dcb2452c127e6de61d",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-24"
   },
   "source": "单个1×1卷积在每个空间位置计算 $y_{i,j}=Wx_{i,j}+b$。它不把相邻位置混合，却能重新组合通道；所有位置共享W。多个1×1与非线性交替，就是逐位置共享MLP。\n\nNiN将一个空间卷积后接两个1×1卷积，区分“收集邻域”与“加工该位置的特征向量”。这解释了为什么1×1不是无作用的卷积，也为什么通道宽度属于表示设计。[D2L 8.3](https://d2l.ai/chapter_convolutional-modern/nin.html)",
   "id": "770c4b6a111e47dcb2452c127e6de61d"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "266c577b689846e3bd7646c23c51cedb",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-25"
   },
   "source": "def nin_block(cin,cout,k=3):\n    return nn.Sequential(nn.Conv2d(cin,cout,k,padding=k//2),nn.ReLU(),\n                         nn.Conv2d(cout,cout,1),nn.ReLU(),\n                         nn.Conv2d(cout,cout,1),nn.ReLU())\npointwise=nn.Conv2d(3,5,1).double()\npixel_map=torch.randn(2,3,4,6,dtype=torch.float64)\nby_matrix=torch.einsum(\"oc,nchw->nohw\",pointwise.weight[:,:,0,0],pixel_map)+pointwise.bias[None,:,None,None]\ntorch.testing.assert_close(pointwise(pixel_map),by_matrix)\nprint(\"shared per-position transform:\",by_matrix.shape)",
   "id": "266c577b689846e3bd7646c23c51cedb",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "eae9293e089e4b80a5ffbab04d89d902",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-26"
   },
   "source": "## GAP：省下分类头参数，也舍弃位置专属权重",
   "id": "eae9293e089e4b80a5ffbab04d89d902"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "e1e95daae32d472da8b4dfaa0549a20a",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-27"
   },
   "source": "Flatten+Linear可给不同空间位置不同分类权重，参数与HW成正比。GAP先对空间平均，让同一个通道的各位置贡献对称，再接分类器；它减少参数，但丢掉精确空间布局。因此分类上的有用不变性与检测/分割需要的位置线索存在取舍。\n\n在线性操作和相同bias约定下，$Linear(GAP(X))=GAP(Conv_{1\\times1}(X))$。这说明空间类别响应可以先保留为图，再聚合成整图分数。若中间加入ReLU，通常不能任意交换顺序。下面复制同一组参数验证等式。",
   "id": "e1e95daae32d472da8b4dfaa0549a20a"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "1e9d2997565649a381ed87ec03630533",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-28"
   },
   "source": "feature=torch.randn(2,16,7,7,dtype=torch.float64)\nhead=nn.Linear(16,10).double()\nspatial_head=nn.Conv2d(16,10,1).double()\nwith torch.no_grad():\n    spatial_head.weight.copy_(head.weight[:,:,None,None]);spatial_head.bias.copy_(head.bias)\na=head(feature.mean((2,3)))\nbb=spatial_head(feature).mean((2,3))\ntorch.testing.assert_close(a,bb)\nprint(\"Flatten+Linear parameters:\",16*7*7*10+10)\nprint(\"GAP+Linear parameters:\",parameters(head))",
   "id": "1e9d2997565649a381ed87ec03630533",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "769438820da44cf49fb806c559b23779",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-29"
   },
   "source": "## GoogLeNet：并行尺度与计算预算",
   "id": "769438820da44cf49fb806c559b23779"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "93b4e24cf880463f9fd5eee569fdcda6",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-30"
   },
   "source": "Inception让输入经过不同空间尺度的分支，再沿通道拼接。各分支同时计算，普通Inception并不是按输入动态选择一个分支。拼接要求各支batch和空间尺寸一致，输出通道数是各支通道数之和。\n\n昂贵空间卷积前先以1×1压缩通道，可降低计算。课程示例：192→32的5×5直接计算需153600个权重；先192→32的1×1，再32→32的5×5需31744个权重，不计bias。若输出空间一致，MAC也按相同比例减少。\n\nD2L的第一Inception采用`(64,(96,128),(16,32),32)`，其中5×5分支先压到16，四支输出总共256；下方用这个完整配置展示模块。两种数字对应不同瓶颈宽度，不应混用。[GoogLeNet](https://arxiv.org/abs/1409.4842) · [D2L 8.4](https://d2l.ai/chapter_convolutional-modern/googlenet.html)",
   "id": "93b4e24cf880463f9fd5eee569fdcda6"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "2ccccd2594d849a9be8aae6734e7e7fc",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-31"
   },
   "source": "class Inception(nn.Module):\n    def __init__(self,cin,c1,c2,c3,c4):\n        super().__init__()\n        self.paths=nn.ModuleList([\n            nn.Sequential(nn.Conv2d(cin,c1,1),nn.ReLU()),\n            nn.Sequential(nn.Conv2d(cin,c2[0],1),nn.ReLU(),nn.Conv2d(c2[0],c2[1],3,padding=1),nn.ReLU()),\n            nn.Sequential(nn.Conv2d(cin,c3[0],1),nn.ReLU(),nn.Conv2d(c3[0],c3[1],5,padding=2),nn.ReLU()),\n            nn.Sequential(nn.MaxPool2d(3,stride=1,padding=1),nn.Conv2d(cin,c4,1),nn.ReLU())])\n    def forward(self,x):\n        outputs=[p(x) for p in self.paths]\n        assert len({tuple(y.shape[2:]) for y in outputs})==1\n        return torch.cat(outputs,dim=1)\ninc=Inception(192,64,(96,128),(16,32),32)\nix=torch.zeros(1,192,14,14)\nfor i,path in enumerate(inc.paths,1):print(\"path\",i,tuple(path(ix).shape),\"parameters\",parameters(path))\nassert inc(ix).shape==(1,256,14,14)",
   "id": "2ccccd2594d849a9be8aae6734e7e7fc",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "c7560891169f4622b53ccdceb94d920f",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-32"
   },
   "source": "## 可选深读：瓶颈为什么有表达代价",
   "id": "c7560891169f4622b53ccdceb94d920f"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "8ad01346b861434792511d8a75d77a97",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-33"
   },
   "source": "若先投影到r维再做空间线性变换，每个空间偏移的等效通道矩阵都分解为 $B_{a,b}A$，秩不超过r。降低r限制了可表达的线性映射；中间ReLU带来分段非线性，但不能无代价恢复已经被投影删除的信息。\n\n瓶颈是一项用表示约束换计算的选择。参数更少、MAC更少和实际延迟更低并非同义；新增加的1×1、内存搬运以及硬件利用率都会影响运行时间。读论文或调模型时需要同时检查计算预算和任务表现。",
   "id": "8ad01346b861434792511d8a75d77a97"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "291f500b4c534d8ebec4e960a9564c4c",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-34"
   },
   "source": "## BatchNorm：同一层在训练与推理时不完全一样",
   "id": "291f500b4c534d8ebec4e960a9564c4c"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "4ef0f7ed1f6748ba801a372936fc5c02",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-35"
   },
   "source": "BN对每个卷积通道跨N、H、W计算统计量：\n$$\\mu_c=\\frac1m\\sum_t x_{c,t},\\quad v_c=\\frac1m\\sum_t(x_{c,t}-\\mu_c)^2,\\quad y=\\gamma_c\\frac{x-\\mu_c}{\\sqrt{v_c+\\epsilon}}+\\beta_c.$$\n训练时使用当前batch统计；典型eval设置使用运行均值/方差。gamma、beta是可训练参数，运行统计是buffer。改变同batch的其他样本可改变训练输出，这是归一化操作的实际依赖。\n\nBN常允许较稳定或较大的更新，但不应把它的全部效用归结为单一“消除内部协变量偏移”解释。小batch、空间相关性与领域变化会影响统计。N=1不等于卷积BN只有一个观测值：每通道有N×H×W个位置，不过它们未必独立。[BN原论文](https://arxiv.org/abs/1502.03167) · [D2L 8.5](https://d2l.ai/chapter_convolutional-modern/batch-norm.html)",
   "id": "4ef0f7ed1f6748ba801a372936fc5c02"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "c710618da2bb456f83d732599e8bb52b",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-36"
   },
   "source": "bx=torch.tensor([1.,3.,5.,7.],dtype=torch.float64).reshape(2,1,1,2)\nbn=nn.BatchNorm2d(1,momentum=.1).double()\nybn=bn(bx)\nmean=bx.mean((0,2,3),keepdim=True)\nvar=bx.var((0,2,3),correction=0,keepdim=True)\nmanual_bn=(bx-mean)/torch.sqrt(var+bn.eps)\ntorch.testing.assert_close(ybn,manual_bn)\nprint(\"mean\",mean.item(),\"forward variance\",var.item(),\"running variance\",bn.running_var.item())\nprint(\"parameters\",list(dict(bn.named_parameters())),\"buffers\",list(dict(bn.named_buffers())))",
   "id": "c710618da2bb456f83d732599e8bb52b",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "3d1ab76c54ff4e9aa48c1a0b7ff9da02",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-37"
   },
   "source": "## train/eval和grad/no_grad是两个正交开关",
   "id": "3d1ab76c54ff4e9aa48c1a0b7ff9da02"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "66bd836e17a84b9984698828e75448b4",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-38"
   },
   "source": "`train()/eval()`控制BN与Dropout等模块行为；`no_grad()`控制自动求导记录。train+no_grad仍会更新BN运行统计，eval+梯度开启仍可对输入求导。冻结参数同样不会自动切换BN到eval。\n\nPyTorch训练前向采用correction=0方差，更新running_var采用correction=1。例子[1,3,5,7]的均值4、两种方差分别5和20/3；初始running_var=1、momentum=.1后为1.5666667。该momentum乘新观测值，与某些D2L手写函数乘旧值的约定相反。[PyTorch BatchNorm2d](https://docs.pytorch.org/docs/2.14/generated/torch.nn.BatchNorm2d.html)",
   "id": "66bd836e17a84b9984698828e75448b4"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "67169fc214ae4b31876bfba376c00109",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-39"
   },
   "source": "for training in (True,False):\n    for grad_enabled in (True,False):\n        m=nn.BatchNorm2d(1).double();m.train(training)\n        t=bx.clone().requires_grad_(True)\n        before=m.running_mean.clone()\n        with torch.set_grad_enabled(grad_enabled):\n            out=m(t)\n        changed=not torch.equal(before,m.running_mean)\n        print(\"train\",training,\"grad\",grad_enabled,\"statistics changed\",changed,\"output requires_grad\",out.requires_grad)\n        assert changed==training\n        assert out.requires_grad==grad_enabled\n# 当前安装版本的真实Python控制逻辑，可看到self.training与running buffers。\nprint(inspect.getsource(nn.modules.batchnorm._BatchNorm.forward))",
   "id": "67169fc214ae4b31876bfba376c00109",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "71a9695145b04031bb88a78861bf008a",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-40"
   },
   "source": "本地已执行的真实源码节选（PyTorch 2.14.0+cpu）；上方 `inspect` 可以在你的运行环境重新读取对应版本。这里展示 Python 调用与模式分支，不是底层 CUDA/CPU 数值实现。\n\n```python\ndef forward(self, input: Tensor) -> Tensor:\n        self._check_input_dim(input)\n\n        # exponential_average_factor is set to self.momentum\n        # (when it is available) only so that it gets updated\n        # in ONNX graph when this node is exported to ONNX.\n        if self.momentum is None:\n            exponential_average_factor = 0.0\n        else:\n            exponential_average_factor = self.momentum\n\n        if self.training and self.track_running_stats:\n            # TODO: if statement only here to tell the jit to skip emitting this when it is None\n            if self.num_batches_tracked is not None:  # type: ignore[has-type]\n                self.num_batches_tracked.add_(1)  # type: ignore[has-type]\n                if self.momentum is None:  # use cumulative moving average\n                    exponential_average_factor = 1.0 / float(self.num_batches_tracked)\n                else:  # use exponential moving average\n                    exponential_average_factor = self.momentum\n\n        r\"\"\"\n        Decide whether the mini-batch stats should be used for normalization rather than the buffers.\n        Mini-batch stats are used in training mode, and in eval mode when buffers are None.\n        \"\"\"\n        if self.training:\n            bn_training = True\n        else:\n            bn_training = (self.running_mean is None) and (self.running_var is None)\n\n        r\"\"\"\n        Buffers are only updated if they are to be tracked and we are in training mode. Thus they only need to be\n        passed when the update should occur (i.e. in training mode when they are tracked), or when buffer stats are\n        used for normalization (i.e. in eval mode when buffers are not None).\n        \"\"\"\n        return F.batch_norm(\n            input,\n            # If buffers are not to be tracked, ensure that they won't be updated\n            (\n                self.running_mean\n                if not self.training or self.track_running_stats\n                else None\n            ),\n            self.running_var if not self.training or self.track_running_stats else None,\n            self.weight,\n            self.bias,\n            bn_training,\n            exponential_average_factor,\n            self.eps,\n        )\n```",
   "id": "71a9695145b04031bb88a78861bf008a"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "792f2649654f4b47bfb99a68b96f038f",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-41"
   },
   "source": "## ResNet：先看训练误差",
   "id": "792f2649654f4b47bfb99a68b96f038f"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "8fcddcc806a240b0bc331f7acea35ef3",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-42"
   },
   "source": "原论文Figure 1在CIFAR-10上比较plain20与plain56：更深网络不仅测试误差较高，训练误差也较高。若只有过拟合，通常会看到训练拟合更好而测试变差；这里连训练目标都更难达到，提示了优化困难。\n\n但一张曲线并不能单独确定全部成因。它提供的是特定结构与训练设置下的退化观察；残差设计随后的实验才显示改造的效果。讨论时要区分模型可表示的函数、优化器实际到达的解与未见数据上的风险。[ResNet Figure 1](https://arxiv.org/pdf/1512.03385)",
   "id": "8fcddcc806a240b0bc331f7acea35ef3"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "e944193d7bac4319a412a22d3746dd4c",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-43"
   },
   "source": "## 函数类包含：有更好的解，不等于能找到它",
   "id": "e944193d7bac4319a412a22d3746dd4c"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "b6e343772aa64b3d8e8c4a92d1b0b759",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-44"
   },
   "source": "若模型类满足 $\\mathcal F\\subseteq\\mathcal F'$，同一训练集上的最优经验损失不会因可选函数更多而升高。然而实际训练不是直接取得全局最优；参数化、初始化与优化路径决定能到哪里。测试误差也不因函数类扩大而保证下降。\n\n加深网络时，一个合理要求是新增部分至少能够保留已有计算。恒等映射提供这样的基准。残差把目标映射H改写为 $H(x)=x+F(x)$，若无需改变输入，只需使残差分支接近零。它把“保留旧函数”变成较直接的参数选择。[D2L 8.6](https://d2l.ai/chapter_convolutional-modern/resnet.html)",
   "id": "b6e343772aa64b3d8e8c4a92d1b0b759"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "9b8ec8388e164088a4098de961b897ab",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-45"
   },
   "source": "## 让一个 block 学习相对输入的修正",
   "id": "9b8ec8388e164088a4098de961b897ab"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "af5f3226b0ae495385a893b6895df23d",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-46"
   },
   "source": "对不带加法后激活的理想残差块，$y=x+F(x)$。可用 $F(x)=0$保留输入；当目标只需小修正时，优化对象是该修正。这个观点解释了残差的参数化动机，不意味着训练中所有残差都很小。\n\n原版常见块为Conv-BN-ReLU-Conv-BN-add-ReLU，实际输出 $ReLU(x+F(x))$。如果输入有负分量，令F=0也不能在全域成为恒等。当前stage若接收非负激活可以讨论其局部保留；preactivation设计则进一步改善直接路径。下方实现保留BN，明确演示identity/projection两种shortcut。",
   "id": "af5f3226b0ae495385a893b6895df23d"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "f522999031684ddcbb2a07a3f81178c5",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-47"
   },
   "source": "![恒等路径与残差修正的概念图](https://codingai-lec02.pages.dev/art/residual-path.webp)\n\n*概念图（ImageGen）：残差结构提供保留已有表示的直接路径；这不是梯度永不消失的保证，条件与反例见后面的深读。 此图用于解释结构，不是模型实测输出。*",
   "id": "f522999031684ddcbb2a07a3f81178c5"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "323db1409a2f4ae6a2f91ba6a2f4a1d3",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-48"
   },
   "source": "class Residual(nn.Module):\n    def __init__(self,cin,cout,stride=1):\n        super().__init__()\n        self.conv1=nn.Conv2d(cin,cout,3,stride=stride,padding=1,bias=False)\n        self.bn1=nn.BatchNorm2d(cout)\n        self.conv2=nn.Conv2d(cout,cout,3,padding=1,bias=False)\n        self.bn2=nn.BatchNorm2d(cout)\n        self.shortcut=nn.Identity() if cin==cout and stride==1 else nn.Sequential(nn.Conv2d(cin,cout,1,stride=stride,bias=False),nn.BatchNorm2d(cout))\n    def forward(self,x):\n        residual=self.bn2(self.conv2(F.relu(self.bn1(self.conv1(x)))))\n        skip=self.shortcut(x)\n        assert residual.shape==skip.shape\n        return F.relu(residual+skip)\nfor cin,cout,stride in [(16,16,1),(16,32,2)]:\n    block=Residual(cin,cout,stride)\n    t=torch.randn(2,cin,14,14)\n    print(tuple(t.shape),\"->\",tuple(block(t).shape),type(block.shortcut).__name__)",
   "id": "323db1409a2f4ae6a2f91ba6a2f4a1d3",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "f6bad560f22b453ea692c9f43ac48839",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-49"
   },
   "source": "## 加法要求形状相容",
   "id": "f6bad560f22b453ea692c9f43ac48839"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "c743e36fd7c04a1b949a64d3066b4488",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-50"
   },
   "source": "相加需要对应位置和通道对齐。当分辨率或通道改变时，使用projection $P(x)$后再加F(x)：$y=P(x)+F(x)$。投影不再是恒等，需把它的参数、采样与梯度计入。\n\nconcat把不同分支的通道并列保留；add把对应通道相加，假设它们可以在同一表示空间组合。前者通常增加宽度和后续计算，后者维持宽度但更强地约束表示接口。不要让PyTorch广播静默掩盖残差分支的shape错误。",
   "id": "c743e36fd7c04a1b949a64d3066b4488"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "5b17de0956a74f63ada4e9b84079ac02",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-51"
   },
   "source": "## 可选深读：I + JF提供直接路径，但不保证梯度永不消失",
   "id": "5b17de0956a74f63ada4e9b84079ac02"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "28e5efa389e740c9ba22f02b105c33ba",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-52"
   },
   "source": "对 $y=x+F(x)$，Jacobian为 $J_y=I+J_F$；带projection时变为 $J_P+J_F$；原postactivation还要左乘ReLU导数的对角门控矩阵。这里需要控制的是残差分支Jacobian的范数，而不只是F(x)的输出幅度。若 $\\|J_F\\|_2\\le\\epsilon<1$，三角不等式给出\n$ (1-\\epsilon)\\|v\\|_2\\le\\|(I+J_F)v\\|_2\\le(1+\\epsilon)\\|v\\|_2. $\n这约束单个残差块在该位置附近对向量范数的缩放。许多块的Jacobian继续相乘时，仍可能累积放大或衰减。仅F输出幅度小并不充分，例如 $F(x)=\\epsilon\\sin(x/\\epsilon^2)$ 的幅度至多epsilon，但导数的最大幅度是 $1/\\epsilon$。\n\n这不是无条件保证：若 $F(x)=-x$，两项完全抵消。下面验证线性残差的Jacobian和抵消反例。它检验微分公式，不替代深网络训练对照。[Identity Mappings](https://arxiv.org/abs/1603.05027)",
   "id": "28e5efa389e740c9ba22f02b105c33ba"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "b8b7f63b4e6e41a08e917812dba176d9",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-53"
   },
   "source": "a=torch.tensor([[.1,.2],[-.1,.05]],dtype=torch.float64)\nz=torch.tensor([.6,-.4],dtype=torch.float64,requires_grad=True)\njac=torch.autograd.functional.jacobian(lambda t:t+a@t,z)\ntorch.testing.assert_close(jac,torch.eye(2,dtype=torch.float64)+a)\ncancel=torch.autograd.functional.jacobian(lambda t:t-t,z)\nassert torch.count_nonzero(cancel)==0\nprint(\"residual Jacobian\",jac); print(\"cancellation\",cancel)",
   "id": "b8b7f63b4e6e41a08e917812dba176d9",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "4020093babcd4d51b84b27a4aeb678e5",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-54"
   },
   "source": "## 四种结构选择放在同一张表里",
   "id": "4020093babcd4d51b84b27a4aeb678e5"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "a6ecb1a03a87499ca7c61145afabb4fa",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-55"
   },
   "source": "| 设计 | 原问题 | 改变了什么 | 代价或限制 |\n| --- | --- | --- | --- |\n| AlexNet | 大规模端到端视觉学习怎样可行 | 数据、计算、非线性与正则形成系统 | 不能归因于单一模块 |\n| VGG | 如何规则地增加深度 | 重复小核块 | 高分辨率激活与分类头昂贵 |\n| NiN/GAP | 通道如何非线性加工；head为何大 | 逐位置MLP与空间平均 | 位置布局与通道瓶颈取舍 |\n| Inception | 多尺度如何在预算内组合 | 并行分支与降维 | 拼接宽度、投影信息损失 |\n| BN | 中间尺度与训练状态如何处理 | batch归一化与可学习重缩放 | batch依赖、运行统计与域变化 |\n| ResNet | 加深后如何保留和优化计算 | 残差参数化与捷径 | 仍依赖初始化、激活与训练 |\n\n**LAB03架构实验室**提供完整模块和机制实验。只用随机输入可验证shape、参数或微分；架构性能需要真实训练与共同协议。",
   "id": "a6ecb1a03a87499ca7c61145afabb4fa"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "b3c2bfd6567a490c9b37470923ed0fe8",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-56"
   },
   "source": "## 一个分类器里，哪些部分值得复用",
   "id": "b3c2bfd6567a490c9b37470923ed0fe8"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "e0ce222134a149cdab6f4219a2960fda",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-57"
   },
   "source": "把分类网络分成backbone与head：前者输出表示，后者组织成当前任务的预测。源任务训练出的表示可能包含目标任务也需要的统计结构；这提供了迁移起点，但目标域的成像、标签粒度和背景关系可能不同。\n\n三种常见选择是随机初始化从头训练、冻结预训练backbone只训head、微调部分或全部参数。比较时要固定数据划分与评估，记录预训练权重版本和预处理。较小学习率常用于已有表示，新head可用较大学习率；这是要验证的策略，不是普适常数。[DeCAF](https://proceedings.mlr.press/v32/donahue14.html) · [D2L 14.2](https://d2l.ai/chapter_computer-vision/fine-tuning.html)",
   "id": "e0ce222134a149cdab6f4219a2960fda"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "bf3918345bd240d0be3eab6b718ea6c8",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-58"
   },
   "source": "![共享视觉表示用于不同任务的概念图](https://codingai-lec02.pages.dev/art/representation-transfer.webp)\n\n*概念图（ImageGen）：分类学到的表示可成为新任务起点，但目标、空间信息需求与数据分布决定如何迁移。 此图用于解释结构，不是模型实测输出。*",
   "id": "bf3918345bd240d0be3eab6b718ea6c8"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "77980a9c5aca45a8b58d897e4c2d385a",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-59"
   },
   "source": "## 冻结的是参数；BN状态仍需单独决定",
   "id": "77980a9c5aca45a8b58d897e4c2d385a"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "8752fbf52aab4498b1e4fe93c2d8a635",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-60"
   },
   "source": [
    "`requires_grad_(False)`阻止参数梯度；它不自动把模型切到eval，也不阻止BN buffer更新。冻结特征实验通常把backbone设为eval，训练新head；微调时再明确哪些stage、哪些BN统计允许更新。\n",
    "\n",
    "这项运行练习位于[第三讲 CLS01 · 从可迁移权重到可比较的表示](https://codingai-lec03.pages.dev/course/c01-transfer.html)。下方代码只打印这一入口，不在本册进行迁移训练。若继续用真实预训练模型，须明确权重及预处理版本；随机初始化不能冒充预训练。"
   ],
   "id": "8752fbf52aab4498b1e4fe93c2d8a635"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "1fcead6cbb384a0bae748ee034a3978f",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-61"
   },
   "source": "# 本迁移机制已抽到第三讲 CLS01，附独立 setup。\n# 当前对象保留历史位置；完整旧代码与原内容见归档。\nprint('迁移梯度路由核对：https://codingai-lec03.pages.dev/course/c01-transfer.html')\n",
   "id": "1fcead6cbb384a0bae748ee034a3978f",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "c20a4e7831d04c3b9185ff105ca8a474",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-62"
   },
   "source": "实际使用torchvision预训练模型时，关键入口如下。这是选读代码，不会在本讲义自动下载；执行前在LAB环境确认torchvision和网络条件。\n\n```python\nfrom torchvision.models import resnet18, ResNet18_Weights\nweights = ResNet18_Weights.DEFAULT\nmodel = resnet18(weights=weights)  # 真正加载预训练参数\npreprocess = weights.transforms() # 与权重对应的预处理\nmodel.fc = nn.Linear(model.fc.in_features, target_classes)\nbackbone_params = [p for name,p in model.named_parameters()\n                   if not name.startswith(\"fc.\")]\noptimizer = torch.optim.SGD([\n    {\"params\": backbone_params, \"lr\": 1e-3},\n    {\"params\": model.fc.parameters(), \"lr\": 1e-2}], momentum=.9)\n```\n\n对于源域与目标域差异，先检查输入通道、归一化和标签是否保真，再比较冻结与微调。`weights.transforms()`是推理预处理起点；训练增强应围绕目标任务另行选择。",
   "id": "c20a4e7831d04c3b9185ff105ca8a474"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "e652ea6c08c248a8b84b1cf160681778",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-63"
   },
   "source": "## 为什么检测与分割不能只照搬分类输出",
   "id": "e652ea6c08c248a8b84b1cf160681778"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "976b61231b3a4da4afda95be3e740418",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-64"
   },
   "source": [
    "分类把空间证据聚合为整图标签；检测还需对象位置与数量，语义分割需像素类别，实例分割还要区分同类的不同对象。NiN/GAP部分说明类别响应可以先保留为二维图，再做聚合；保留这些位置关系，才可能服务新的空间任务。\n",
    "\n",
    "这不只是换掉分类头：下采样造成的分辨率损失、感受野与定位精度、任务目标和标注格式都要重新核对。第二讲末尾用[同一图的检测与分割引子](https://codingai-lec02.pages.dev/#/representation-beyond-classification/0)建立这个问题；第三讲主题是**可迁移表示与多模态感知**，完整安排与目前已有材料见[第三讲导读](https://codingai-lec03.pages.dev/course/guide.html)。这一本保留接口解释，不将计划中的任务写成已完成实验。"
   ],
   "id": "976b61231b3a4da4afda95be3e740418"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "5967052c30cd4d60934e7c07d196eb94",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-65"
   },
   "source": "## 读一个陌生 block：逐项检查",
   "id": "5967052c30cd4d60934e7c07d196eb94"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "489636aecb22439baa2889db48fe19d9",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-66"
   },
   "source": "给定一个新block，请完成一个连贯解释：输入输出张量是什么；空间与通道在哪些步骤混合；怎样改变理论感受野和计算量；相加或拼接隐含什么接口；参数化希望改善什么困难。然后指出哪一项可以靠算术/微分直接验证，哪一项需要训练对照。\n\n可以选择一个有判别力的实验：例如固定数据和预算，比较GAP与Flatten在平移前后的结果；或给同一残差分支切换skip，观察训练损失及梯度。记录改变项与保持项，不预先替模型宣布胜利。",
   "id": "489636aecb22439baa2889db48fe19d9"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "f4e3876edbc34663a8f34d234dd40bb8",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-67"
   },
   "source": "## 来源与回看",
   "id": "f4e3876edbc34663a8f34d234dd40bb8"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "9f5d2f1585844fb7948c3b4d017a8584",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-68"
   },
   "source": "主要改编来源：D2L第8章（AlexNet、VGG、NiN、GoogLeNet、BN、ResNet）与14.2节；结合本课程2025Fall及2026Spring模块实现，补入资源、模式和Jacobian讨论。D2L[正文CC BY-SA 4.0](https://github.com/d2l-ai/d2l-en/blob/master/LICENSE)与[代码许可](https://github.com/d2l-ai/d2l-en/blob/master/LICENSE-SAMPLECODE)分别保留；本讲改编的D2L教学说明以CC BY-SA 4.0提供。PyTorch展示的是当前安装版本源码，遵循[PyTorch许可](https://github.com/pytorch/pytorch/blob/main/LICENSE)。\n\n[CS231n CNN Architectures](https://cs231n.stanford.edu/slides/2026/lecture_6.pdf) · **LAB03** · **返回导读**",
   "id": "9f5d2f1585844fb7948c3b4d017a8584"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "983212d863844d09aa607b7f50dcaad8",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-69"
   },
   "source": "## 按问题查阅：补充推导与机制核对",
   "id": "983212d863844d09aa607b7f50dcaad8"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "4786ba1f6a6e40fc91dd1b8d1d4c9e3a",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-70"
   },
   "source": [
    "**查阅方式。** A 回看学习起点，B/C 对比小核与GAP，D 核对拼接和瓶颈，E/F 区分BN状态与残差条件。先选与你当前问题直接相关的一节，再到文末标准库核对；不需要一次读完全部历史材料。\n",
    "\n",
    "原论文证据、教学构造和框架源码分别保留身份。课堂主线与材料分工以[第二讲导读](https://codingai-lec02.pages.dev/course/guide.html)为准。"
   ],
   "id": "4786ba1f6a6e40fc91dd1b8d1d4c9e3a"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "3e0b3da3861d43579ff2090c4403ad15",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-71"
   },
   "source": "## A. 逐层起点与不同训练目标",
   "id": "3e0b3da3861d43579ff2090c4403ad15"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "0cf7d2287027476487e7e8c756958628",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-72"
   },
   "source": "连乘的局部导数可能使反馈缩小或放大；饱和区域、初始化尺度与组合结构都影响起点。深度只是其中一个变量，不能看到训练困难就全部归于“层太多”。\n\n早期逐层预训练先学习靠近输入的一层，再暂时冻结它，把产生的表示交给下一层；接入分类头后，可以让最终标签损失共同微调各层。这个过程改变训练起点，不等于为每个隐藏单元人工规定语义。\n\nRBM 的可见—隐藏往返是统计模型中的条件采样：从数据估计隐藏状态，再生成可见状态并重新估计隐藏状态。短链对比散度用数据端与短链生成端的共现差异近似调整连接；它不是精确求出的完整模型梯度，也不是普通前馈网络的一次重构误差反传。\n\nAutoencoder 则把编码和解码连成重构目标。2006 年相关尝试曾使用逐层 RBM 初始化，再展开编码器与解码器，用反向传播微调。重构数字图像要保留纸面、笔画和贴纸等细节；若分类标签只是 3 或 8，替换无关贴纸时希望预测保持。不同目标偏好不同信息，不能把所有“先训练”都称为同一种方法。\n\n**先判断**：解释一种预训练方案，除了“先预训练再微调”，还必须说明哪四项？\n\n**核对提示**：数据、训练目标、当时可更新的参数、训练过程。还要说明预训练结束后怎样接入下游目标，以及哪些参数继续变化。\n\n**来源与继续阅读**\n\n- [D2L · Forward and Backward Propagation](https://d2l.ai/chapter_multilayer-perceptrons/backprop.html)\n- [CS231n 2026 · Neural Networks and Backpropagation](https://cs231n.stanford.edu/slides/2026/lecture_4.pdf)\n- [Hinton et al. · A Fast Learning Algorithm for Deep Belief Nets (2006)](https://www.cs.toronto.edu/~hinton/absps/ncfast.pdf)\n- [Hinton, Osindero & Teh · A Fast Learning Algorithm for Deep Belief Nets (2006)](https://www.cs.toronto.edu/~hinton/absps/fastnc.pdf)\n- [Hinton & Salakhutdinov · Reducing the Dimensionality of Data with Neural Networks (2006)](https://www.cs.toronto.edu/~hinton/absps/science.pdf)\n- [D2L · Fine-Tuning](https://d2l.ai/chapter_computer-vision/fine-tuning.html)\n- [Krizhevsky et al. · ImageNet Classification with Deep CNNs (2012)](https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf)",
   "id": "0cf7d2287027476487e7e8c756958628"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "67b5b3840b804509bfbe0e806068defb",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-73"
   },
   "source": "## B. 相同理论范围，是否就是相同计算",
   "id": "67b5b3840b804509bfbe0e806068defb"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "d28272bd441047d09abc08432d7775d6",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-74"
   },
   "source": "AlexNet 不能仅由层图解释：数据、计算资源、增强、非线性与优化配置共同使大规模监督训练可行。ReLU 正半轴的局部导数不随幅值饱和，针对部分优化困难；Dropout 在训练中随机屏蔽激活，减少对特定组合的依赖；二者并不是同一种正则化或相同目的的技巧。这里不假装存在一组只改变单一因素的现成对照。\n\nVGG 以重复小核建立一致的设计语言，让深度成为更清楚的实验变量。在 stride=1、没有中间下采样、各层宽度同为 C、忽略偏置时，三个 3×3 核的理论范围为 7×7，权重共 27C²；单个 7×7 核为 49C²。中间非线性使二者不再是同一个函数族；相同感受野也不保证相同延迟。\n\n架构比较需要把“理论范围”“可表示函数”“参数/MAC”“实测运行时间”分开记录。某一项较小，不自动决定所有任务的最好方案。\n\n**先判断**：三个小核比一个大核参数少，是否就能断言任何设备上都更快？\n\n**核对提示**：不能。执行次数、中间激活、硬件内核与并行方式都会影响延迟。公式只支持给定条件下的权重数比较。\n\n**来源与继续阅读**\n\n- [D2L · AlexNet](https://d2l.ai/chapter_convolutional-modern/alexnet.html)\n- [Krizhevsky et al. · ImageNet Classification with Deep CNNs (2012)](https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf)\n- [CS231n 2026 · Neural Networks and Backpropagation](https://cs231n.stanford.edu/slides/2026/lecture_4.pdf)\n- [D2L · VGG](https://d2l.ai/chapter_convolutional-modern/vgg.html)\n- [Simonyan & Zisserman · Very Deep CNNs (2014)](https://arxiv.org/abs/1409.1556)\n- [Araujo et al. · Computing Receptive Fields (2019)](https://distill.pub/2019/computing-receptive-fields/)",
   "id": "d28272bd441047d09abc08432d7775d6"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "f8af1bacdedb4dec8c8e27a51443de72",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-75"
   },
   "source": "## C. GAP 数值例与信息损失",
   "id": "f8af1bacdedb4dec8c8e27a51443de72"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "027aafe63f944989b1e8483d97a0b01a",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-76"
   },
   "source": "空间卷积读取邻域；1×1 在每个位置用共享矩阵混合通道。配合非线性并层叠，它相当于逐位置共享的 MLP，可增加局部通道变换能力，不必扩大空间窗口。\n\nGAP 对每个通道的 H×W 响应取平均，得到 C 维向量，再交给读出层。课程原算例的第一个 3×3 图有一个 9、其余为 0，均值为 1；第二个全为 2，均值为 2。读出权重 [[2,−1],[−1,2]]、偏置 [0,−1]，得到两个分数 [0,2]。这是“先逐通道平均，再混合通道”的手工构造，没有训练结果含义。\n\n它与课堂小 CNN 的 Flatten784→32→10 是不同的头部设计。Flatten 只重新排列索引，仍可保留位置；GAP 合并空间位置，减少位置专属的分类权重，但精细布局被压缩。同均值而不同布局的两张特征图会在 GAP 后相同，这是可自行构造的反例，不是声称原算例里已经有这张比较图。\n\n**先判断**：为什么不能把 Flatten 直接称作“丢掉空间位置”？\n\n**核对提示**：展平本身是可逆索引变换，每个位置仍有固定地址。GAP、池化或之后的多对一读出才会合并某些区别，具体取舍应指明算子。\n\n**来源与继续阅读**\n\n- [D2L · Network in Network](https://d2l.ai/chapter_convolutional-modern/nin.html)\n- [Lin et al. · Network in Network (2013)](https://arxiv.org/abs/1312.4400)\n- [D2L · Input and Output Channels](https://d2l.ai/chapter_convolutional-neural-networks/channels.html)\n- [Long et al. · Fully Convolutional Networks (2015)](https://arxiv.org/abs/1411.4038)",
   "id": "027aafe63f944989b1e8483d97a0b01a"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "95d38723df8a47c0b52270ba517eceef",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-77"
   },
   "source": "## D. 通道拼接与瓶颈秩约束",
   "id": "95d38723df8a47c0b52270ba517eceef"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "bbf1a85f58c448cc97666fe858eb2fe5",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-78"
   },
   "source": "简化模块让同一输入同时进入 1×1、3×3、5×5、池化后 1×1 四条路径。若各分支都得到 8×8 空间尺寸，通道分别为 8、12、4、8，沿通道拼接后是 8×8×32。它不是先选择“最合适的一条”，也不是把对应元素相加；空间对齐由 padding 等配置实现。这组宽度是教学构造，不是论文原始层表。\n\n大核的成本与输入通道数相乘。直接用 5×5 将 192 通道变成 32，权重数 192×32×25=153600；先用 1×1 压到 16，再用 5×5，权重为 192×16+16×32×25=15872，均忽略偏置。若空间尺寸一致，每个输出位置的 MAC 也按相应两项求和。\n\n压缩并非无损。中间没有非线性时，有效空间偏移映射 K_ab=B_ab A，秩至多 m，且各偏移共享投影 A。加入 ReLU 后，整体不再是同一个固定低秩线性层，但瓶颈仍会限制可传递的信息。应检验哪个宽度足够，而不是以压得最窄为目标。\n\n**先判断**：四分支输出通道分别 8、12、4、8 时，为什么不能直接逐元素相加？\n\n**核对提示**：通道形状不同，逐元素相加不成立；设计意图也是保留全部测量，沿通道拼接为 32。即使人为配成同宽，相加和拼接仍是不同的信息组织。\n\n**推导时可用的关系式**\n\n$$K_{ab}=B_{ab}A,\\qquad\\operatorname{rank}(K_{ab})\\le m$$\n\n仅在中间无非线性时成立；各偏移共享A。\n\n**来源与继续阅读**\n\n- [D2L 8.4 · GoogLeNet](https://d2l.ai/chapter_convolutional-modern/googlenet.html)\n- [Szegedy et al. · Going Deeper with Convolutions (2015)](https://arxiv.org/abs/1409.4842)\n- [D2L 7.4 · Multiple Input and Multiple Output Channels](https://d2l.ai/chapter_convolutional-neural-networks/channels.html)\n- [D2L 8.3 · Network in Network](https://d2l.ai/chapter_convolutional-modern/nin.html)",
   "id": "bbf1a85f58c448cc97666fe858eb2fe5"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "7b6eb28e60e24261965862c11c33f624",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-79"
   },
   "source": "## E. BN 同伴、统计与源码分支",
   "id": "7b6eb28e60e24261965862c11c33f624"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "1d96fae5b30a4f589ef1557ebbc36763",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-80"
   },
   "source": "卷积 BN 对每个通道沿 batch 与空间位置统计均值、方差，标准化后再使用可训练的缩放 γ 与平移 β。它改变中间表示的参数化，并引入批次依赖，不能简化为“把数字变小”。\n\n保持目标响应为 2，γ=1,β=0,ε=10⁻⁵：放入 [0,2,4] 时均值为 2，标准化后为 0；放入 [2,4,6] 时均值为 4，标准化后约 −1.225。两组总体方差均为 8/3，但同一输入的输出变了。\n\n通常训练前向使用当前 batch 统计并更新 running buffers，eval 使用已积累的运行统计。若 track_running_stats=False 等配置使运行 buffers 不存在，eval 也可能继续使用 batch 统计；应读具体实现条件。\n\nPyTorch 的前向方差采用分母 M 的有偏估计，而更新 running_var 时采用分母 M−1 的无偏估计；其 momentum=m 表示新统计权重，v_run←(1−m)v_run+m v_unbiased。它不是优化器中动量定义的同一个量。下面保留固定版本源码节选，运行时请核对实际版本。\n\n**先判断**：两段代码都写 momentum=0.1，是否意味着 BN 与优化器记忆历史的方式相同？\n\n**核对提示**：不是。BN 此参数控制运行统计中“新统计”的权重；优化器动量系数通常用于“旧速度”的保留。应先读各 API 的定义与更新式。\n\n**推导时可用的关系式**\n\n$$v_{biased}=\\frac1M\\sum_m(x_m-\\mu)^2,\\quad v_{unbiased}=\\frac1{M-1}\\sum_m(x_m-\\mu)^2$$\n\n同一组数的两种估计，不能混用名称。\n\n$$v_{run}\\leftarrow(1-m)v_{run}+m v_{unbiased}$$\n\nPyTorch momentum=m；前向仍使用v_biased。\n\n**PyTorch _BatchNorm.forward：逐字删节的统计选择分支**\n\n身份：PyTorch2.14.0实际安装源码摘录；仅保留统计路径分支。这是保留的说明节选，所需上下文未必齐全，不能按完整程序单独运行。\n\n```python\nif self.training:\n    bn_training = True\nelse:\n    bn_training = (self.running_mean is None) and (\n        self.running_var is None\n    )\n\n# 完整函数随后把 bn_training 传给 F.batch_norm\n```\n\n[固定出处](https://github.com/pytorch/pytorch/blob/08187d9e0fba026dc8217405802ab5381dc88d90/torch/nn/modules/batchnorm.py)\n\n**来源与继续阅读**\n\n- [D2L 8.5 · Batch Normalization](https://d2l.ai/chapter_convolutional-modern/batch-norm.html)\n- [Ioffe & Szegedy · Batch Normalization (2015)](https://proceedings.mlr.press/v37/ioffe15.html)\n- [PyTorch · BatchNorm2d](https://docs.pytorch.org/docs/2.14/generated/torch.nn.BatchNorm2d.html)\n- [Ioffe & Szegedy · Batch Normalization](https://arxiv.org/abs/1502.03167)\n- [PyTorch · BatchNorm2d API](https://docs.pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n- [PyTorch · Autograd Mechanics](https://docs.pytorch.org/docs/stable/notes/autograd.html)\n- [PyTorch 2.14.0 · BatchNorm source (08187d9e)](https://github.com/pytorch/pytorch/blob/08187d9e0fba026dc8217405802ab5381dc88d90/torch/nn/modules/batchnorm.py)",
   "id": "1d96fae5b30a4f589ef1557ebbc36763"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "c14777012c434b648096ede928d9b694",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-81"
   },
   "source": "## F. 残差、后激活与预激活",
   "id": "c14777012c434b648096ede928d9b694"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "d422a78b057f4fcc9872125b6563af5b",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-82"
   },
   "source": "最简残差核心 y=x+F(x) 让已有表示沿捷径保留，分支学习修正；形状不同就需要匹配投影，不能直接相加。把 F 设为零得到 identity 的论断，必须绑定“相加后没有额外截断”的简化形式。\n\n常见原始块先 Conv→BN→ReLU→Conv→BN，必要时投影捷径，相加后再 ReLU，实际是 y=ReLU(x+F(x))。负值可能被相加后的门截断，因而不能逐字套用无门控核心的梯度结论。\n\n预激活把归一化与激活前移，使跨块的 identity 路径更直接；完整顺序与投影处理需按论文和实现核对。保存的 D2L Residual.forward 是教学代码摘录，不是 PyTorch 内部源码，也不是预激活块。结构解释仍要回到同条件下的训练与测试结果，不以“加了一条路”替代性能证据。\n\n**先判断**：F(x)=0 是否保证任何叫 ResNet 的块都输出 x？\n\n**核对提示**：不保证。还要看捷径是否投影、相加后是否有 ReLU，以及输入形状。对 ReLU(x+F(x))，负的 x 分量在零分支时仍被截断。\n\n**推导时可用的关系式**\n\n$$y=x+F(x)$$\n\n残差核心；实际块的激活与投影需结合具体结构。\n\n$$y_{post}=\\operatorname{ReLU}(x+F(x)),\\quad y_{core}=x+F(x)$$\n\n用于比较路径差异；完整preactivation结构见原论文。\n\n**D2L Residual.forward：摘录（构造函数略）**\n\n身份：D2L Residual.forward源码摘录；非PyTorch内部源码。这是保留的说明节选，所需上下文未必齐全，不能按完整程序单独运行。\n\n```python\ndef forward(self, X):\n    Y = F.relu(self.bn1(self.conv1(X)))\n    Y = self.bn2(self.conv2(Y))\n    if self.conv3:\n        X = self.conv3(X)\n    Y += X\n    return F.relu(Y)\n```\n\n[固定出处](https://d2l.ai/chapter_convolutional-modern/resnet.html)\n\n**来源与继续阅读**\n\n- [D2L 8.6 · Residual Networks](https://d2l.ai/chapter_convolutional-modern/resnet.html)\n- [He et al. · Deep Residual Learning (2015)](https://arxiv.org/abs/1512.03385)\n- [He et al. · Identity Mappings in Deep Residual Networks (2016)](https://arxiv.org/abs/1603.05027)\n- [PyTorch · BatchNorm2d API](https://docs.pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)",
   "id": "d422a78b057f4fcc9872125b6563af5b"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "d1fe09e3c079460f9311f5ef8246cc9d",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-83"
   },
   "source": "## 从主讲继续：推导边界与研究来源",
   "id": "d1fe09e3c079460f9311f5ef8246cc9d"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "838242bed4dc426abb8cd26398671c0f",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-84"
   },
   "source": "历史资料与课堂计算需要不同证据。2012 ILSVRC 的不同赛道/额外训练数据口径不能混成同一百分比；ResNet 的结构动机也不能替代原论文的改善曲线。本册保留的 Figure 6 是论文深读来源；主讲当前使用 Table 2 的 ImageNet validation top-1、10-crop 配对证据，以及 Figure 1 的 CIFAR-10 训练退化观察。二者数据与指标各自独立；不能把这段历史图号说明当作当前主讲仍展示 Figure 6。固定论文中的对应模型条件不代表任意宽度、深度与训练配置都有效。\n\n**主讲研究来源**\n\n- [D2L · LeNet](https://d2l.ai/chapter_convolutional-neural-networks/lenet.html)\n- [D2L · AlexNet](https://d2l.ai/chapter_convolutional-modern/alexnet.html)\n- [CS231n 2026 · CNNs](https://cs231n.stanford.edu/slides/2026/lecture_5.pdf)\n- [CS231n 2026 · Neural Networks and Backpropagation](https://cs231n.stanford.edu/slides/2026/lecture_4.pdf)\n- [Hinton et al. · A Fast Learning Algorithm for Deep Belief Nets (2006)](https://www.cs.toronto.edu/~hinton/absps/ncfast.pdf)\n- [Hinton & Salakhutdinov · Reducing the Dimensionality of Data with Neural Networks (2006)](https://www.cs.toronto.edu/~hinton/absps/science.pdf)\n- [Krizhevsky et al. · ImageNet Classification with Deep CNNs (2012)](https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf)\n- [ILSVRC 2012 · Official Results](https://image-net.org/challenges/LSVRC/2012/results.html)\n- [He et al. · Deep Residual Learning for Image Recognition (2016)](https://arxiv.org/abs/1512.03385)\n- [D2L · ResNet](https://d2l.ai/chapter_convolutional-modern/resnet.html)\n- [He et al. · Identity Mappings in Deep Residual Networks (2016)](https://arxiv.org/abs/1603.05027)\n- [He et al. · Deep Residual Learning for Image Recognition · CVPR 2016, §4.2 & Fig. 6](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf)",
   "id": "838242bed4dc426abb8cd26398671c0f"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "38bdf60220354974bfb3538b197bf47f",
     "type": "text-cell-h2",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-85"
   },
   "source": "## 独立核对：先预测，再运行",
   "id": "38bdf60220354974bfb3538b197bf47f"
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "6236740f9b8c4208950f6f8705a74bc8",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-86"
   },
   "source": "下面是完整、独立的小规模 Python 核对，只使用标准库；无需下载、安装、训练或访问外部服务。它保留“教学构造”的身份。先写预测，运行后解释哪一项支持判断，再只改一个条件复查。下方没有预填远端执行输出。",
   "id": "6236740f9b8c4208950f6f8705a74bc8"
  },
  {
   "cell_type": "code",
   "metadata": {
    "deepnote": {
     "block_id": "289d7482107e418c905ef7d9894ca728",
     "type": "code",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-87"
   },
   "source": "# 自包含预算与统计核对；不训练、安装或下载任何模型。\nimport math\ndirect=192*32*25\nbottleneck=192*16+16*32*25\nprint('直接 / 瓶颈 权重数：',direct,bottleneck)\nassert (direct,bottleneck)==(153600,15872)\nassert sum([8,12,4,8])==32\nfor batch in ([0.,2.,4.],[2.,4.,6.]):\n    mean=sum(batch)/len(batch)\n    biased=sum((a-mean)**2 for a in batch)/len(batch)\n    unbiased=sum((a-mean)**2 for a in batch)/(len(batch)-1)\n    normalized=(2.-mean)/math.sqrt(biased+1e-5)\n    print('batch',batch,'目标2的输出',normalized,'方差',biased,unbiased)\nmeans=[1.,2.]\nscores=[2*means[0]-means[1],-means[0]+2*means[1]-1]\nassert scores==[0.,2.]\nprint('GAP 后读出：',scores)\n",
   "id": "289d7482107e418c905ef7d9894ca728",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deepnote": {
     "block_id": "0553f81e3b9043509acfc756afabc65b",
     "type": "markdown",
     "source_block_id": null,
     "logical_key": null
    },
    "html_anchor": "cell-88"
   },
   "source": "**继续一步**：修改一个数值或条件，先写变化方向，再运行核对。不要删除断言来让代码“通过”；若主动改变了断言的前提，应同时写出新预期。机制小算例通过，只说明这一小段计算与预期一致，不等于完整图像实验或新架构训练成功。",
   "id": "0553f81e3b9043509acfc756afabc65b"
  }
 ]
}