From 80f634849bafe8865079af530d8c33572aadd31e Mon Sep 17 00:00:00 2001 From: Jue Guo Date: Wed, 3 Sep 2025 17:28:51 -0400 Subject: [PATCH] update --- .idea/.gitignore | 8 + .../inspectionProfiles/profiles_settings.xml | 6 + .idea/misc.xml | 4 + .idea/ml-meta.iml | 8 + .idea/modules.xml | 8 + .idea/vcs.xml | 6 + .vscode/settings.json | 6 +- assets/css/base.css | 20 +- .../neural-network-basic-structure.svg | 184 +++++ .../neural-network-forward-pass.svg | 194 +++++ .../neural-network-with-weights.svg | 183 +++++ papers/basics/deep-learning-crash-course.html | 692 ++++-------------- 12 files changed, 770 insertions(+), 549 deletions(-) create mode 100644 .idea/.gitignore create mode 100644 .idea/inspectionProfiles/profiles_settings.xml create mode 100644 .idea/misc.xml create mode 100644 .idea/ml-meta.iml create mode 100644 .idea/modules.xml create mode 100644 .idea/vcs.xml create mode 100644 assets/svg/papers/basics/deep-learning-crash-course/neural-network-basic-structure.svg create mode 100644 assets/svg/papers/basics/deep-learning-crash-course/neural-network-forward-pass.svg create mode 100644 assets/svg/papers/basics/deep-learning-crash-course/neural-network-with-weights.svg diff --git a/.idea/.gitignore b/.idea/.gitignore new file mode 100644 index 0000000..13566b8 --- /dev/null +++ b/.idea/.gitignore @@ -0,0 +1,8 @@ +# Default ignored files +/shelf/ +/workspace.xml +# Editor-based HTTP Client requests +/httpRequests/ +# Datasource local storage ignored files +/dataSources/ +/dataSources.local.xml diff --git a/.idea/inspectionProfiles/profiles_settings.xml b/.idea/inspectionProfiles/profiles_settings.xml new file mode 100644 index 0000000..105ce2d --- /dev/null +++ b/.idea/inspectionProfiles/profiles_settings.xml @@ -0,0 +1,6 @@ + + + + \ No newline at end of file diff --git a/.idea/misc.xml b/.idea/misc.xml new file mode 100644 index 0000000..d56657a --- /dev/null +++ b/.idea/misc.xml @@ -0,0 +1,4 @@ + + + + \ No newline at end of file diff --git a/.idea/ml-meta.iml b/.idea/ml-meta.iml new file mode 100644 index 0000000..d0876a7 --- /dev/null +++ b/.idea/ml-meta.iml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/.idea/modules.xml b/.idea/modules.xml new file mode 100644 index 0000000..df8bb69 --- /dev/null +++ b/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/.idea/vcs.xml b/.idea/vcs.xml new file mode 100644 index 0000000..35eb1dd --- /dev/null +++ b/.idea/vcs.xml @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file diff --git a/.vscode/settings.json b/.vscode/settings.json index db37523..999dd68 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -1,7 +1,7 @@ { "workbench.colorCustomizations": { - "activityBar.background": "#1A3315", - "titleBar.activeBackground": "#24481E", - "titleBar.activeForeground": "#F6FBF5" + "activityBar.background": "#5F0717", + "titleBar.activeBackground": "#850A20", + "titleBar.activeForeground": "#FFFBFC" } } \ No newline at end of file diff --git a/assets/css/base.css b/assets/css/base.css index d080722..a4ba6f5 100644 --- a/assets/css/base.css +++ b/assets/css/base.css @@ -905,13 +905,13 @@ html.theme-dark .toc-chapters a { /* --- Book/Chapter dropdown button restyle --- */ .toc-books { - margin-bottom: 2rem; - margin-left: 1rem; - margin-right: 1rem; + margin-bottom: 1rem; + margin-left: 0.5rem; + margin-right: 0.5rem; display: block; } .toc-book { - margin-bottom: 2rem; + margin-bottom: 1rem; background: transparent; border: none; transition: all 0.3s ease; @@ -978,7 +978,7 @@ html.theme-dark .toc-chapters a { .toc-chapters.dropdown-content { background: transparent; margin-top: 0; - padding: 1rem 0 1.5rem 1.5rem; + padding: 0.5rem 0 0.75rem 1rem; overflow: hidden; transition: max-height 0.4s ease, opacity 0.4s ease, padding 0.4s ease; } @@ -997,7 +997,7 @@ html.theme-dark .toc-chapters a { .toc-chapters.dropdown-content.show { max-height: 1000px !important; opacity: 1 !important; - padding: 1rem 0 1.5rem 1.5rem !important; + padding: 0.5rem 0 0.75rem 1rem !important; display: block !important; overflow: visible !important; } @@ -1013,8 +1013,8 @@ html.theme-dark .toc-chapters.dropdown-content { display: block; font-size: 0.95em; font-weight: 500; - padding: 0.5rem 0; - margin: 0.25rem 0; + padding: 0.25rem 0; + margin: 0.15rem 0; color: #6b7280; text-decoration: none; transition: all 0.3s ease; @@ -1031,8 +1031,8 @@ html.theme-dark .toc-chapters.dropdown-content { /* Elegant separation between main nav and contributors */ #mainNav { - margin-bottom: 5rem; - padding-bottom: 3rem; + margin-bottom: 2.5rem; + padding-bottom: 1.5rem; position: relative; } diff --git a/assets/svg/papers/basics/deep-learning-crash-course/neural-network-basic-structure.svg b/assets/svg/papers/basics/deep-learning-crash-course/neural-network-basic-structure.svg new file mode 100644 index 0000000..4b79778 --- /dev/null +++ b/assets/svg/papers/basics/deep-learning-crash-course/neural-network-basic-structure.svg @@ -0,0 +1,184 @@ + + + + + + + + + + + + + + + + Input layer + Hidden layer + Output layer + + + + + + + + + + + + + + + + + + + + w1 + + + + w2 + + + + w3 + + + + w4 + + + + + w5 + + + + w6 + + + + + + i₁ + + + + i₂ + + + + h₁ + + + + h₂ + + + + out + + + + prediction + + + + \ No newline at end of file diff --git a/assets/svg/papers/basics/deep-learning-crash-course/neural-network-forward-pass.svg b/assets/svg/papers/basics/deep-learning-crash-course/neural-network-forward-pass.svg new file mode 100644 index 0000000..82ce41d --- /dev/null +++ b/assets/svg/papers/basics/deep-learning-crash-course/neural-network-forward-pass.svg @@ -0,0 +1,194 @@ + + + + + + + + + + + + + + + Input layer + Hidden layer + Output layer + + + + + + + + + + + + + + + + .11 + + + + .21 + + + + .12 + + + + .08 + + + + .14 + + + + .15 + + + + + + 2 + + + + 3 + + + + .85 + + + + .48 + + + + .191 + + + + prediction + + + + + + + + + + + + Forward pass + + \ No newline at end of file diff --git a/assets/svg/papers/basics/deep-learning-crash-course/neural-network-with-weights.svg b/assets/svg/papers/basics/deep-learning-crash-course/neural-network-with-weights.svg new file mode 100644 index 0000000..60c0fb3 --- /dev/null +++ b/assets/svg/papers/basics/deep-learning-crash-course/neural-network-with-weights.svg @@ -0,0 +1,183 @@ + + + + + + + + + + + + + + + + Input layer + Hidden layer + Output layer + + + + + + + + + + + + + + + + + + + + 0.11 + + + + 0.21 + + + + 0.12 + + + + 0.08 + + + + 0.14 + + + + 0.15 + + + + + + i₁ + + + + i₂ + + + + h₁ + + + + h₂ + + + + out + + + + prediction + + + + \ No newline at end of file diff --git a/papers/basics/deep-learning-crash-course.html b/papers/basics/deep-learning-crash-course.html index c6a729b..3010b69 100644 --- a/papers/basics/deep-learning-crash-course.html +++ b/papers/basics/deep-learning-crash-course.html @@ -278,189 +278,11 @@

2.1. Backpropagation Explained

- - - - - - - - - - - - - - - - Input layer - Hidden layer - Output layer - - - - - - - - - - - - - - - - - - - - w1 - - - - w2 - - - - w3 - - - - w4 - - - - - w5 - - - - w6 - - - - - - i₁ - - - - i₂ - - - - h₁ - - - - h₂ - - - - out - - - - prediction - - - - + />

@@ -489,188 +311,11 @@

2.1. Backpropagation Explained

- - - - - - - - - - - - - - - - Input layer - Hidden layer - Output layer - - - - - - - - - - - - - - - - - - - - 0.11 - - - - 0.21 - - - - 0.12 - - - - 0.08 - - - - 0.14 - - - - 0.15 - - - - - - i₁ - - - - i₂ - - - - h₁ - - - - h₂ - - - - out - - - - prediction - - - - + />

@@ -731,6 +376,154 @@

2.1. Backpropagation Explained

Forward Pass + +
+ Forward pass of two-input, two-hidden-neuron, one-output network with numeric activations and weights +
+ +

+ During the forward pass, the network makes predictions based on the + current weights. The output is compared to the actual target, and the + difference (loss) is computed. How is it calculated? $$ + \left[\begin{array}{ll}2 & 3\end{array}\right] + \cdot\left[\begin{array}{ll}0.11 & 0.12 \\ 0.21 & + 0.08\end{array}\right]=\left[\begin{array}{ll}0.85 & + 0.48\end{array}\right] \cdot\left[\begin{array}{l}0.14 \\ + 0.15\end{array}\right]=[0.191] $$ For those of you who are not + familiar with matrix multiplication, here's a quick breakdown: $$ + \begin{array}{ll}2 \times .11+3 \times .21=.85 & .85 \times .14+.48 + \times .15=.191 \\ 2 \times .12+3 \times .08=.48 & \end{array} $$ +

+ +

+ Calculate Loss The first prediction is not ideal, it + is actually very far off. So what can we do about it? We need to + calculate the loss. The loss function quantifies how far off the + predictions are from the actual targets. A common choice for + regression tasks is the Mean Squared Error (MSE) loss, defined as: $$ + \text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 $$ where + $y_i$ is the actual target and $\hat{y}_i$ is the predicted value. + Now, $$ \text { Error }=\frac{1}{2}(0.191-1.0)^{2}=0.327 $$ +

+ +

+ Reducing Error Our goal is to reduce this error + (Since the actual output is constant, the only way to reduce the + error is to change the prediction value). One way to do this is by + adjusting + the weights in the network. This is where the magic of backpropagation + comes in. By computing the gradient of the loss with respect to each + weight, we can determine how to change the weights to minimize the + loss. This process is repeated over many iterations (epochs) until the + network learns to make accurate predictions. $$ \require{color} + \begin{aligned} \mathbf{prediction} &= + \textcolor[RGB]{102,187,106}{\mathbf{out}} \\[4pt] \mathbf{prediction} + &= + \big(\textcolor[RGB]{3,155,229}{\mathbf{h_1}}\big)\,\textcolor[RGB]{138,138,138}{\mathbf{w_5}} + + + \big(\textcolor[RGB]{3,155,229}{\mathbf{h_2}}\big)\,\textcolor[RGB]{138,138,138}{\mathbf{w_6}} + \\[4pt] \mathbf{prediction} &= + \big(i_{1}\,\textcolor[RGB]{60,100,177}{\mathbf{w_1}} + + i_{2}\,\textcolor[RGB]{60,100,177}{\mathbf{w_2}}\big)\,\textcolor[RGB]{138,138,138}{\mathbf{w_5}} + + \big(i_{1}\,\textcolor[RGB]{240,123,63}{\mathbf{w_3}} + + i_{2}\,\textcolor[RGB]{240,123,63}{\mathbf{w_4}}\big)\,\textcolor[RGB]{138,138,138}{\mathbf{w_6}} + \end{aligned} $$ +

+ +

+ Backpropagation The vanilla version of it is gradient + descent.Gradient descent is an iterative optimization algorithm for + finding the minimum of a function; in our case we want to minimize th + error function. To find a local minimum of a function using gradient + descent, one takes steps proportional to the negative of the gradient + of the function at the current point. +

+ + +
+ + +
+ + $$ \require{ams}\require{color} \mathbf{W_x^{*}} \;=\; \mathbf{W_x} + \;-\; \textcolor[RGB]{255,180,0}{\boldsymbol{a}} \left( + \frac{\partial \mathrm{Error}}{\partial W_x} \right) $$ + + +
+ New weight +
+
Old weight
+
+ Learning rate +
+
+ Derivative of Error
with respect to weight +
+
+
@@ -769,179 +562,6 @@

3. Activation Functions

-
-

4. Popular Network Architectures

- -
- Different types of neural networks excel at different tasks. - Convolutional Neural Networks (CNNs) dominate computer vision, - Recurrent Neural Networks (RNNs) excel at sequential data, and - Transformers have revolutionized natural language processing. -
- -

Convolutional Neural Networks (CNNs):

-

- CNNs use convolution operations to detect features like edges, - textures, and patterns in images. They're designed to be - translation-invariant, meaning they can recognize a cat whether it - appears in the top-left or bottom-right of an image. -

- -

Recurrent Neural Networks (RNNs):

-

- RNNs maintain internal memory to process sequential data like text or - time series. Modern variants like LSTMs and GRUs solve the vanishing - gradient problem that limited earlier RNN architectures. -

- -

Transformer Networks:

-

- Transformers use attention mechanisms to process sequences in parallel - rather than sequentially. This architecture powers modern language - models like GPT and has achieved state-of-the-art results across many - domains. -

-
- -
-

5. Common Challenges and Solutions

- -
- Deep learning practitioners face several recurring challenges: - overfitting (memorizing training data), vanishing gradients (weights - in early layers barely update), and finding the right hyperparameters. -
- -

Overfitting:

- - -

Vanishing Gradients:

- - -

Hyperparameter Tuning:

- -
- -
-

6. Real-World Applications

- -
- Deep learning has achieved remarkable success across diverse fields: - image recognition surpassing human performance, natural language - processing enabling sophisticated AI assistants, and game-playing AI - mastering complex strategic games. -
- -

Computer Vision:

- - -

Natural Language Processing:

- - -

Other Domains:

- -
- -
-

7. Getting Started

- -
- The best way to learn deep learning is through hands-on practice. - Start with established frameworks like TensorFlow or PyTorch, work - through tutorials with real datasets, and gradually tackle more - complex projects. -
- -

Learning Path:

-
    -
  1. - Mathematics Foundation: Linear algebra, calculus, - probability -
    2. -
  2. Programming Skills: Python, NumPy, pandas
    3. -
  3. - Deep Learning Frameworks: TensorFlow/Keras or - PyTorch -
    4. -
  4. - Practice Projects: Image classification, sentiment - analysis -
    5. -
  5. - Advanced Topics: Specialized architectures, - optimization -
    6. -
- -

Practical Tips:

- -
- -
-

8. Future Directions

- -
- Deep learning continues to evolve rapidly. Current research focuses on - making models more efficient, interpretable, and capable of learning - from fewer examples. Emerging areas include few-shot learning, neural - architecture search, and integration with symbolic reasoning. -
- -

- The field is moving toward more efficient and generalizable AI - systems. Key trends include developing smaller models that perform as - well as larger ones, creating systems that can explain their - decisions, and building AI that can learn new tasks quickly with - minimal data - much like humans do. -

-

- As we advance, the integration of deep learning with other AI - approaches promises even more powerful and versatile systems, bringing - us closer to artificial general intelligence while addressing current - limitations in robustness, interpretability, and efficiency. -

-
-