[{"content":"Introduction I’ve always been eager to dive into the fundamentals of robotics, and I finally have the opportunity through my \u0026ldquo;Autonomous Mobile Robot\u0026rdquo; course. I decided to write this blog post not only to solidify what I’ve learned but also to create a practical reference for my future work.\nThis post focuses on the essential keywords of autonomous systems: Perception, Navigation, Localization, Planning, and SLAM.\nAt its core, an \u0026ldquo;autonomous robot\u0026rdquo; is defined by its ability to:\nMap the World: Build or maintain a model of the environment. See and Understand: Perceive and analyze its surroundings. Know Its Place: Identify its own location and state within the map. Act Strategically: Plan and execute movements to navigate effectively. Spatial Descriptions and Transformations Figure 1. coordinate system. The hat symbol denotes a unit vector. In Figure 1, {A} represents the room and {B} represents the drone. (As you can see, the drone is slightly tilted.) Why is it essential to define both a Local Coordinate System {B} and a Global Coordinate System {A} for a robot?\n1. Why do we need a Local Frame {B}?\nIt simplifies the description of the robot\u0026rsquo;s internal structure. By using {B}, the positions of the robot\u0026rsquo;s parts (e.g., propellers, sensors, or limbs) remain constant, avoiding the need to re-calculate every minor movement relative to the entire room.\n2. Why must we then express {B} in terms of {A}?\nIt provides the spatial context for Global Localization. A robot cannot navigate to a specific target (e.g., $x=5, y=3$) unless it knows its own position and orientation (heading) relative to the fixed world frame {A}.\nSLAM (Simultaneous Localization and Mapping) is the process of a robot autonomously discovering the transformation between these two frames, allowing it to understand its body’s movement ({B}) within the context of the wider world ({A}).\nPosition To know the drone\u0026rsquo;s position, we should know $^A P$. $$^A P= \\begin{bmatrix} P_x \\\\ P_y \\\\ P_z \\end{bmatrix}$$\nSo, $^A P$ is simply a position of drone relative to {A}. Then how can we calculate $^A P$? maybe by GPS, external camera system, or Odometry.\nOrientation To define the drone’s orientation (heading), we use the Rotation Matrix $^A_B R$. This matrix represents the orientation of the local frame $\\{B\\}$ relative to the global frame $\\{A\\}$: $$^A_B R = \\begin{bmatrix} ^A \\hat{X}_B, ^A \\hat{Y}_B, ^A \\hat{Z}_B \\end{bmatrix}$$\n$^A \\hat{X}_B$ is the unit vector representing the x-axis of $\\{B\\}$ expressed in the coordinate system of $\\{A\\}$. $^A \\hat{Y}_B$ and $^A \\hat{Z}_B$ similarly represent the y and z axes of the drone relative to the room. Then how can we estimate $^A_B R$ in practice? We need to implement methods to compute the rotation matrix using sensor data, such as IMU readings or visual odometry from camera feeds.\nFor example, let $\\hat{X}$ be the direction the robot is facing, $\\hat{Y}$ the direction of its left arm, and $\\hat{Z}$ the direction of its head. If $^A_B R$ is given as: $$^A_B R= \\begin{bmatrix} 1\u0026amp;0\u0026amp;0 \\\\ 0\u0026amp;0\u0026amp;-1 \\\\ 0\u0026amp;1\u0026amp;0 \\end{bmatrix}$$\nThis implies that:\nThe robot\u0026rsquo;s face ($\\hat{X}_B$) is still aligned with the world\u0026rsquo;s $+X_A$ direction $(1, 0, 0)$. The robot\u0026rsquo;s left arm ($\\hat{Y}_B$) is now pointing towards the world\u0026rsquo;s $+Z_A$ direction $(0, 0, 1)$. The robot\u0026rsquo;s head ($\\hat{Z}_B$) is pointing towards the world\u0026rsquo;s $-Y_A$ direction $(0, -1, 0)$. Suppose we have detected that the robot has rotated $30^\\circ$ counter-clockwise (CCW) using its onboard sensors. Now, our goal is to build a mathematical \u0026ldquo;Translator (Rotation Matrix)\u0026rdquo; that maps everything from the robot\u0026rsquo;s perspective $\\{B\\}$ to the room\u0026rsquo;s global frame $\\{A\\}$.\nThe most fundamental way to build this matrix in 3D is through the Dot Product. By definition, the dot product of two unit vectors represents the projection of one axis onto another. It tells us: \u0026ldquo;How much of the robot\u0026rsquo;s axis aligns with the room\u0026rsquo;s axis?\u0026rdquo;\nThe Rotation Matrix is composed of nine entries, where each column represents one of the robot\u0026rsquo;s axes ($\\hat{X}_B, \\hat{Y}_B, \\hat{Z}_B$) projected onto the room’s global axes ($\\hat{X}_A, \\hat{Y}_A, \\hat{Z}_A$):\n$$^A_B R = \\begin{bmatrix} \\hat{X}_A \\cdot \\hat{X}_B \u0026amp; \\hat{X}_A \\cdot \\hat{Y}_B \u0026amp; \\hat{X}_A \\cdot \\hat{Z}_B \\\\ \\hat{Y}_A \\cdot \\hat{X}_B \u0026amp; \\hat{Y}_A \\cdot \\hat{Y}_B \u0026amp; \\hat{Y}_A \\cdot \\hat{Z}_B \\\\ \\hat{Z}_A \\cdot \\hat{X}_B \u0026amp; \\hat{Z}_A \\cdot \\hat{Y}_B \u0026amp; \\hat{Z}_A \\cdot \\hat{Z}_B \\end{bmatrix}$$\nColumn 1 ($^A \\hat{X}_B$): Represents the robot\u0026rsquo;s \u0026ldquo;Forward\u0026rdquo; direction in the room. Column 2 ($^A \\hat{Y}_B$): Represents the robot\u0026rsquo;s \u0026ldquo;Left\u0026rdquo; direction in the room. Column 3 ($^A \\hat{Z}_B$): Represents the robot\u0026rsquo;s \u0026ldquo;Up\u0026rdquo; direction in the room. 3.The true power of this matrix lies in its ability to transform raw sensor data into global coordinates. Even if the drone is tilted or rotating, we can instantly calculate the global position of any detected object.\nIf the drone’s sensors detect a wall at a local coordinate of $^B P_{wall} = [2, 0, 1]^T$ (2m ahead and 1m above the drone), its absolute position in the room ${A}$ is:\n$$\\text{Wall Position in } A = \\ ^A P_{drone} + \\ ^A_B R \\cdot \\begin{bmatrix} 2 \\\\ 0 \\\\ 1 \\end{bmatrix}$$\nBy using this Projection-based Matrix, the robot no longer sees just a wall in front of me. it understands exactly where that wall exists in the world.\nFirst, let\u0026rsquo;s say the robot didn\u0026rsquo;t move and didn\u0026rsquo;t rotate at all. then, $$^A P_{drone} = \\begin{bmatrix} 0 \\\\ 0 \\\\ 0 \\end{bmatrix}, ^A_B R = \\begin{bmatrix} 1 \u0026amp; 0 \u0026amp; 0 \\\\ 0 \u0026amp; 1 \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; 1 \\end{bmatrix}$$\n$$\\text{Wall Position in } A = \\begin{bmatrix} 0 \\\\ 0 \\\\ 0 \\end{bmatrix} + \\begin{bmatrix} 1 \u0026amp; 0 \u0026amp; 0 \\\\ 0 \u0026amp; 1 \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; 1 \\end{bmatrix} \\begin{bmatrix} 2 \\\\ 0 \\\\ 1 \\end{bmatrix}$$ $$= \\begin{bmatrix} 2 \\\\ 0 \\\\ 1 \\end{bmatrix}$$\nNow, let\u0026rsquo;s say the robot didn\u0026rsquo;t move, but rotated $30^\\circ$ counter-clockwise (CCW) around the global Z-axis.\nEven if the robot stays at the origin ($^A P_{drone} = [0,0,0]^T$), its local axes ($\\hat{X}_B, \\hat{Y}_B, \\hat{Z}_B$) are no longer aligned with the room\u0026rsquo;s axes. To build the new Rotation Matrix, we calculate the Dot Product for each entry to see how the robot\u0026rsquo;s new directions project onto the room.\n(Following the right-hand rule, a counter-clockwise (CCW) rotation is defined as a positive angle. Therefore, we substitute $\\theta = +30^\\circ$ into our trigonometric functions to calculate the entries of the rotation matrix.)\nFor the robot\u0026rsquo;s Forward direction ($\\hat{X}_B$):\n$\\hat{X}_A \\cdot \\hat{X}_B = \\cos(30^\\circ) \\approx 0.866$ $\\hat{Y}_A \\cdot \\hat{X}_B = \\sin(30^\\circ) = 0.5$ $\\hat{Z}_A \\cdot \\hat{X}_B = 0$ (No vertical tilt) For the robot\u0026rsquo;s Left direction ($\\hat{Y}_B$):\n$\\hat{X}_A \\cdot \\hat{Y}_B = \\cos(90^\\circ+30^\\circ) = -\\sin(30^\\circ) = -0.5$ $\\hat{Y}_A \\cdot \\hat{Y}_B = \\cos(30^\\circ) \\approx 0.866$ $\\hat{Z}_A \\cdot \\hat{Y}_B = 0$ For the robot\u0026rsquo;s Up direction ($\\hat{Z}_B$):\nSince it rotated around the Z-axis, $\\hat{Z}_B$ is still $[0, 0, 1]^T$ relative to $\\{A\\}$. $$^A_B R = \\begin{bmatrix} 0.866 \u0026amp; -0.5 \u0026amp; 0 \\\\ 0.5 \u0026amp; 0.866 \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; 1 \\end{bmatrix}$$\nNow, we apply this to the same wall detection ($^B P_{wall} = [2, 0, 1]^T$):\n$$\\text{Wall Position in } A = \\begin{bmatrix} 0 \\\\ 0 \\\\ 0 \\end{bmatrix} + \\begin{bmatrix} 0.866 \u0026amp; -0.5 \u0026amp; 0 \\\\ 0.5 \u0026amp; 0.866 \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; 1 \\end{bmatrix} \\begin{bmatrix} 2 \\\\ 0 \\\\ 1 \\end{bmatrix}$$\n$$= \\begin{bmatrix} (0.866 \\times 2) + (-0.5 \\times 0) + (0 \\times 1) \\\\ (0.5 \\times 2) + (0.866 \\times 0) + (0 \\times 1) \\\\ (0 \\times 2) + (0 \\times 0) + (1 \\times 1) \\end{bmatrix} = \\begin{bmatrix} 1.732 \\\\ 1.0 \\\\ 1.0 \\end{bmatrix}$$\nAlso, as you can see, every column of $^A_B R$ is orthogonal, which means $^A_B R$ is orthogonal matrix. So, $^A_B R^{-1}=$ $^B_A R =$ $^A_B R^T$\nFrame What is Frame? : A set of four vectors giving position and orientation information\nFrame {B} = $^A \\hat{X}_B$ $^A \\hat{Y}_B$ $^A \\hat{Z}_B$ $^A P_{BORG}$\nHomogeneous Transform As discussed in the previous section, the position of an object (e.g., a wall) relative to the global frame ${A}$ can be calculated by combining translation and rotation: $$^A P_{wall} = \\ ^A P_{drone} + \\ ^A_B R \\ ^B P_{wall}$$\nWhile this equation is intuitive, handling translation and rotation separately can become computationally tedious, especially when dealing with multiple coordinate transformations. To simplify this, we can encapsulate both operations into a single matrix called the Homogeneous Transformation Matrix ($T$).\nBy augmenting our vectors with an extra dimension (using homogeneous coordinates), we can perform the same transformation with a single matrix multiplication:\n$$\\begin{bmatrix} ^A P_{wall} \\\\ 1 \\end{bmatrix} = \\begin{bmatrix} ^A_B R \u0026amp; ^A P_{drone} \\\\ 0 \\ 0 \\ 0 \u0026amp; 1 \\end{bmatrix} \\begin{bmatrix} ^B P_{wall} \\\\ 1 \\end{bmatrix}$$\n$$\\underset{(4 \\times 1)}{^A \\bar{P}_{wall}} = \\underset{(4 \\times 4)}{^A_B T} \\cdot \\underset{(4 \\times 1)}{^B \\bar{P}_{wall}}$$ This approach allows us to treat a complex sequence of movements as a chain of simple matrix multiplications, which is the standard way to handle kinematics in robotics.\nFigure 2. Example situation of a drone in a room $$^A_B T = \\begin{bmatrix} 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; -1 \u0026amp; 3 \\\\ 0 \u0026amp; 1 \u0026amp; 0 \u0026amp; 1 \\\\ 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 1 \\end{bmatrix}, \\ ^B P=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\ 1 \\end{bmatrix}$$\n$$ \\ ^A P = \\ ^A_B T \\ ^B P \\Rightarrow \\ ^A P = \\begin{bmatrix} 0 \\\\ 2 \\\\ 2 \\\\ 1 \\end{bmatrix}$$\nInverse Transform Rotation matrices are orthogonal, meaning $R^{-1} = R^T$. However, for the Homogeneous Transformation matrix, $T^{-1} \\neq T^T$.\nTo find the inverse transform $^B_A T$ (which maps a point from frame ${A}$ back to ${B}$), we use the following structure:\nIf we have:$$^A_B T = \\begin{bmatrix} ^A_B R \u0026amp; ^A P_{BORG} \\\\ 0 \\ 0 \\ 0 \u0026amp; 1 \\end{bmatrix}$$Then its inverse $^B_A T$ is calculated as:$$^B_A T = (^A_B T)^{-1} = \\begin{bmatrix} (^A_B R)^T \u0026amp; - (^A_B R)^T \\ ^A P_{BORG} \\\\ 0 \\ 0 \\ 0 \u0026amp; 1 \\end{bmatrix}$$Since $(^A_B R)^T = \\ ^B_A R$, we can also write it as:$$^B_A T = \\begin{bmatrix} ^B_A R \u0026amp; - ^B_A R \\ ^A P_{BORG} \\\\ 0 \\ 0 \\ 0 \u0026amp; 1 \\end{bmatrix}$$\n*$- ^B_A R \\ ^A P_{BORG} = \\ ^B P_{AORG}$\n$$ ^A_B T \\ ^B_C T \\ ^C_D T \\ ^D_A T = I $$ $$ ^B_A T = \\ ^B_C T \\ ^C_D T \\ ^D_A T $$\nFilter Why filter is necessary? When a robot tries to figure out where it is (Localization), it usually relies on two main methods. However, both have critical flaws.\nOdometry (Wheel Encoders): \u0026ldquo;I turned the wheels 10 times, so I must have moved 1 meter.\u0026rdquo; This is a guess based on movement. If the wheels slip or the ground is uneven, errors accumulate over time, and the robot eventually gets lost. External Sensors (GPS, LiDAR, Camera): These tell the robot its current position directly, but they are full of noise (uncertainty). GPS might have an error of several meters, and LiDAR can give erratic readings when hitting reflective surfaces like glass. Since both \u0026ldquo;Internal Prediction\u0026rdquo; and \u0026ldquo;External Observation\u0026rdquo; are never 100% accurate, we need a mathematical way to blend them to calculate the most probable state. This is the essence of a filter.\nKalman Filter The most standard approach, which assumes that all data errors follow a Normal Distribution.\nIt calculates the position as \u0026ldquo;The robot is likely at this point (Mean), and the margin of error is the much (Variance).\u0026rdquo; Kalman filter is mathematically efficient and requires very little computational power, making it ideal for real-time systems. However, Accuracy drops if the robot\u0026rsquo;s movement is highly complex (Non-linear) or if the noise doesn\u0026rsquo;t follow a normal distribution. This is why variants like EKF(Extended Kalman Filter) or UKF(Unscented Kalman Filter) are often used.\nKalman filter has two steps: Prediction and Update.\nPrediction (Prior) Using the odometry data, the robot estimates its own position and velocity.\n1. State Prediction $$ \\hat{x}^-_k = A \\hat{x}_{k-1} + B u_{k} $$\nPredicts the currents state($\\hat{x}_k$) based on the previous state($\\hat{x}_{k-1}$) and the control input($u_k$).\nA is the state transition matrix, which represents how the state changes from one time step to the next without any control input. It captures the natural physics of the system. For a robot moving at a constant velocity, $A$ would update the position based on that velocity. In our 1D example, $A=1$ means \u0026ldquo;if I don\u0026rsquo;t move, I stay where I am.\u0026rdquo;\nB is the control input matrix, which defines how the control input ($u_k$), like pushing the gas pedal or turning a motor, changes the state It translates human/AI commands into physical displacement. If $u_k$ is the motor\u0026rsquo;s power, $B$ would be the coefficient that converts that power into meters moved.\n2. Error Covariance Prediction $$ P^-_k = A P_{k-1} A^T + Q $$\nQ (Process Noise) accounts for unmodeled factors like wind or uneven terrain. Because the world isn\u0026rsquo;t perfect, our uncertainty ($P$) always grows during this phase.\nLet\u0026rsquo;s imagine a robot on a 1D linear track. It believes it is at 10m ($\\hat{x}_{k-1}=10$) with 2m of uncertainty ($P_{k-1}=2$).\nPrediction : Using our physics model ($A=1$, $B=1$), the robot predicts it is now at 15m ($\\hat{x}^-_k=10+5=15$).\nUncertainty Growth: Due to motor noise ($Q=1$), the uncertainty grows to 3 ($P_{k}^-=3$).\nUpdate (Posterior) Using the external sensor data, the filter corrects the position estimate.\n1. Kalman Gain $$K_k = \\frac{P_k^- H^T}{HP_k^- H^T + R}$$\nThe Kalman Gain ($K$) acts as a relative weight that balances the prediction error covariance ($P_k^-$) against the measurement noise covariance ($R$).\nH is the observation matrix, which maps the true space into the observed sensor space. Sometimes sensors don\u0026rsquo;t measure what we want directly. For example, if our state is \u0026ldquo;Position\u0026rdquo; but the sensor measures \u0026ldquo;Voltage,\u0026rdquo; $H$ acts as the bridge/converter between them.\nWhen $P$ is large (High Prediction Error): $K$ approaches 1. The filter \u0026ldquo;distrusts\u0026rdquo; the internal model and relies almost entirely on the sensor ($z_k$). When $R$ is large (High Sensor Noise): $K$ approaches 0. The filter \u0026ldquo;ignores\u0026rdquo; the noisy sensor and sticks to its internal physics-based prediction. 2. State Update $$\\hat{x}_k = \\hat{x}_k^- + K_k(z_k - H\\hat{x}_k^-)$$\nWe take our guess and add a correction based on the Innovation (the difference between the sensor reading $z_k$ and our prediction).\n3. Error Covariance Update $$P_k = (I - K_k H)P_k^-$$\nSince we combined two sources of information, our final uncertainty ($P$) shrinks, making the estimate more precise than either source alone.\nA GPS sensor (error variance $R=2$) reports the robot is at 18m ($z_k=18$).\nKalman Gain: $$ K_k = \\frac{P_k^- H^T}{HP_k^- H^T + R} = \\frac{3 \\times 1}{1 \\times 3 + 2} = \\frac{3}{5} = 0.6 $$\nSince our internal uncertainty (3) is higher than the sensor noise (2), we trust the sensor more ($60%$).\nState Update: $$\\hat{x}_k = \\hat{x}_k^- + K_k(z_k - H\\hat{x}_k^-) = 15 + 0.6(18 - 15) = 16.8$$\nThe result is a weighted average that leans toward the sensor.\nError Covariance Update: $$P_k = (I - K_k H)P_k^- = (1 - 0.6 \\times 1) \\times 3 = 1.2$$\nParticle Filter While the Kalman Filter relies on Gaussian assumptions, the Particle Filter (also known as Sequential Monte Carlo) uses a simulation-based approach to estimate the state of a robot. It represents the probability distribution of the robot\u0026rsquo;s position as a set of discrete samples, or \u0026ldquo;Particles.\u0026rdquo;\nEach particle $i$ consists of a state hypothesis $x_k^{(i)}$ and an associated weight $w_k^{(i)}$, which represents the probability (confidence) of that hypothesis.\n1. Prediction (Sampling) We propagate each particle forward in time using the robot\u0026rsquo;s motion model and control input $u_k$. $$x_k^{(i)} \\sim p(x_k | x_{k-1}^{(i)}, u_k)$$\n2. Update (Importance Weighting) When a sensor measurement $z_k$ is received, we evaluate how \u0026ldquo;likely\u0026rdquo; each particle is by calculating its Importance Weight. $$w_k^{(i)} = w_{k-1}^{(i)} \\cdot p(z_k | x_k^{(i)})$$\nLikelihood ($p(z_k | x_k^{(i)})$): We compare the sensor data to what the particle should see at its current location. Particles that match the sensor data closely receive high weights; those that don\u0026rsquo;t are penalized with low weights.\n3. Resampling To prevent particles with low weights from dominating the estimate, we \u0026ldquo;resample\u0026rdquo; the particles based on their weights. High-weight particles are duplicated, while low-weight particles are discarded.\nLet\u0026rsquo;s imagine that the robot is provided with a pre-installed map ($m$), but its initial pose ($x_0$) is completely unknown. To solve this, the robot uses a Particle Filter to recursively estimate its position.\nFigure 3. L-shape room Since the robot has no prior knowledge of its coordinates, it initializes $N$ particles ${x_0^{(i)}, w_0^{(i)}} \\ _{i=1}^N$ typically distributed according to a uniform prior across the map\u0026rsquo;s free space. At this stage, each particle represents a possible state of the system with an equal weight of $N^{-1}$.\nFigure 4. Particles initialized randomly in the room The robot\u0026rsquo;s camera, pointed at the right-side wall, captures a purple door. This triggers the Measurement Update phase.\nWeight Update (Importance Weighting): For every particle, we calculate the Likelihood. \u0026ldquo;Given this particle\u0026rsquo;s hypothesized location, what is the probability of seeing a purple door?\u0026rdquo; Numerical Calculation: We use a Gaussian Probability Density Function (PDF) to score the particles: $$w_k^{(i)} = \\eta \\cdot p(z_k | x_k^{(i)}, m)$$ where $z_k$ is the actual measurement and $p(z_k | x_k^{(i)}, m)$ is the likelihood based on the map. Particles located in front of any of the three purple doors match the map. Their weights stay high, while particles in front of plain walls see a massive drop in weight.\nFigure 5. Particles after the measurement update After the update, most particles have negligible weights, leading to degeneracy. To fix this, the robot performs Resampling. Particles with higher weights are more likely to be selected and multiplied, while low-weight particles are eliminated.\nFigure 6. Particles after resampling The particles now cluster into three distinct groups, one in front of each purple door on the map. This is a multi-modal distribution, representing three high-probability hypotheses.\nThe robot now drives forward ($u_k$). This triggers the Time Update (Prediction) phase. In a particle filter, each particle $i$ possesses its own state $x_k^{(i)} = [x, y, \\theta]^T$, where $\\theta$ represents its orientation. When the \u0026ldquo;move forward\u0026rdquo; command is issued, every particle projects its position based on its own heading ($\\theta$):\nWe add process noise ($Q$) to account for uncertainties like wheel slip or mechanical errors.\nFigure 7. Particles after the time update As the robot moves, the camera captures a second purple door. This is the deciding moment for our hypotheses.\nHypothesis A : These particles were correctly facing downward at the first door. Upon receiving the \u0026ldquo;forward\u0026rdquo; command, they moved down along the corridor to the second door. Their predicted measurement ($\\hat{z}$) from the map perfectly matches the actual measurement ($z_k$) of the purple door. Consequently, their likelihood $p(z_k | x_k^{(i)}, m)$ is maximized, and they will flourish during resampling.\nHypothesis B : These particles also faced downward but started at the second door. After moving forward, they are now in the open space at the bottom of the L-shape. According to the map, their right-side camera should see a distant wall or empty space. Since the sensor sees a door, the mismatch causes their likelihood to drop to near zero.\nHypothesis C : These particles were located at the third door but were facing rightward to see the door initially. When the robot commanded a \u0026ldquo;forward\u0026rdquo; move, these particles traveled toward the right wall of the map. Furthermore, because they are facing right, their right-side camera is now looking at the upper wall of the corridor. This total contradiction between their predicted view and the actual purple door seen by the robot results in their immediate elimination during Resampling.\nFigure 8. Particles after the resampling After the resampling, all 1,000 particles collapse into a single, tight cluster in front of the second purple door. The robot has successfully localized itself by mathematically proving that only one starting position is consistent with the sequence of observations.\nGraph In the previous chapter, we used Filters (Kalman Filter and Particle Filter) to answer the question: \u0026ldquo;Where am I?\u0026rdquo; But localization is just the beginning. To be truly autonomous, a robot must decide: \u0026ldquo;How do I get to my destination efficiently?\u0026rdquo; This is where Graph Theory comes in.\nThe physical world is continuous and infinitely complex, but a robot\u0026rsquo;s brain needs structure to make decisions. By abstracting the environment into a Graph $G=(V,E)$, a set of vertices (states) and edges (paths), we simplify the infinite world into a manageable mathematical map.\nWhile filters are the robot\u0026rsquo;s eyes, Graph Theory is its intelligence. Even the most advanced autonomous systems today rely on this hierarchical approach: using Graphs for Global Planning and continuous math for local movement.\nDijkstra\u0026rsquo;s Algorithm Dijkstra\u0026rsquo;s Algorithm is a graph search algorithm that finds the shortest path between two nodes in a graph. It is a greedy algorithm that works by iteratively selecting the node with the smallest distance from the start node.\nSingle Source Shortest Path Problem Dijkstra’s algorithm is designed to solve a specific optimization problem: Given a graph with non-negative edge weights, find the shortest path from a starting node (Source) to every other node in the network.\nObjective: Minimize the total cost(distance, time, energy, etc.) of travel. Constraint: Edge weights must be non-negative.($w \\geq 0$) Relaxation Relaxation is the key operation in Dijkstra’s algorithm. It updates the distance to a node if a shorter path is found.\nFigure 9. Relaxation In (1), the shortest distance to node $b$ discovered so far is 9. However, we have found a new path to $b$ via node $a$ with a lower total cost of 5. Consequently, we update the shortest distance for node $b$ to 5 and set node $a$ as its new parent in the shortest path.\nIn (2), the shortest distance to node $b$ is already 6. Although we found an alternative path to $b$ through node $a$, its total cost is 7. Since the existing path is more efficient, we maintain the shortest distance for node $b$ at 6.\nAs shown in these examples, the \u0026ldquo;relaxation\u0026rdquo; step acts as a core optimization operation that constantly refines the robot\u0026rsquo;s knowledge.\ndef relax(a, b, G): if b.dist() \u0026gt; a.dist() + G.weight(a,b): # G is a graph object b.set_dist(a.dist() + G.weight(a,b)) # set the distance of node b to a.dist() + G.weight(a,b) b.set_parent(a) # set the parent of node b to a Pseudo-code for Dijkstra\u0026rsquo;s Algorithm def Dijkstra(G, s): # Initialization: Set up initial distances and parents for a in G.nodes(): # For each node a in the graph\u0026#39;s node set a.set_dist(float(\u0026#39;inf\u0026#39;)) # Initialize distance to infinity a.set_parent(None) # Initialize parent as None # s is also in the G.nodes() s.set_dist(0) # Set distance of the source node to 0 S = set() # Set of nodes whose shortest-path weights are already determined Q = PriorityQueue(G.nodes()) # Priority Queue of nodes while not Q.is_empty(): a = Q.pop_min() S.add(a) # Add a to the set of nodes whose shortest-path weights are already determined for b in a.adjacent_nodes(): if b.dist \u0026gt; a.dist + G.weight(a, b): relax(a, b, G) # Relax the edge (a, b) Q.modify_key(b) # Reorganize the priority queue as b\u0026#39;s distance has decreased return G # Return the graph with updated shortest path information Figure 10. Dijkstra's Algorithm Example In Figure 10, $S$ is empty and $Q= \\{s(0), x_1(\\infty), x_2(\\infty), x_3(\\infty), x_4(\\infty)\\}$. Since Q is not empty, the while loop is executed.\nFirst while loop. Because of Q.pop_min(), $a=s(0)$ is extracted from $Q$ and added to $S$. $S=\\{s(0)\\}$. And the for loop is executed.\nFigure 11. Dijkstra's Algorithm Example After the for loop ran one time, $Q= \\{x_3(5), x_1(10), x_2(\\infty), x_4(\\infty)\\}$. And the while loop is executed again.\nSecond while loop. At this time, $a=x_3(5)$ is extracted from $Q$ and added to $S$. $S=\\{s(0), x_3(5)\\}$. And the for loop is executed. Because this is directed graph, adjacent nodes of $x_3(5)$ are $x_1(10)$, $x_2(\\infty)$, and $x_4(\\infty)$.\nFigure 12. Dijkstra's Algorithm Example $Q= \\{x_4(7), x_1(8), x_2(14)\\}$. Third while loop. Next time, $x_4(7)$ is extracted from $Q$ and added to $S$. $S=\\{s(0), x_3(5), x_4(7)\\}$. $a$ is $x_4(7)$ and the adjacent nodes are $s$ and $x_2(14)$.\nFigure 13. Dijkstra's Algorithm Example $x_2(14)$ is turned into $x_2(13)$. $Q= \\{x_1(8), x_2(13)\\}$. Fourth while loop. a is $x_1(8)$ and the adjacent nodes are $x_3(5)$ and $x_2(13)$. $S=\\{s(0), x_3(5), x_4(7), x_1(8)\\}$.\nFigure 14. Dijkstra's Algorithm Example Now, $Q= \\{x_2(9)\\}$. Fifth while loop. a is $x_2(9)$ and the adjacent node is only $x_4(7)$. $S=\\{s(0), x_3(5), x_4(7), x_1(8), x_2(9)\\}$. This time, nothing happens because $x_4$\u0026rsquo;s distance(7) is already less than 9+4=13.\nUnderstanding Time Complexity $O(n^2)$ Dijkstra’s algorithm operates in $O(n^2)$ time (where $n$ is the number of vertices) due to its nested search structure.\n1. Outer Loop ($n$ iterations): The algorithm must extract each of the $n$ nodes from the Priority Queue exactly once to determine its final shortest path. 2. Inner Loop (Up to $n-1$ iterations): For each extracted node, the robot scans all its adjacent neighbors to perform Relaxation. In the worst-case scenario (a complete graph where every node is connected to every other node), each node will have $n-1$ neighbors to check. $$n \\text{ (Nodes)} \\times (n-1) \\text{ (Max Neighbors per Node)} = n^2 - n \\approx O(n^2)$$\nKinematics Through Filtering (Kalman/Particle Filters), the robot can now estimate its current pose despite sensor noise. Through Graph-based Path Planning (Dijkstra’s Algorithm), it can determine the most efficient route to its destination.\nHowever, knowing where to go and where you are is not enough. A crucial question remains. How does the robot actually move its body to follow that path?\nThis is where Kinematics comes in. While our high-level planners output abstract commands like \u0026ldquo;move to $(x, y)$,\u0026rdquo; our hardware only understands low-level signals like \u0026ldquo;rotate the left wheel at $100\\text{ RPM}$.\u0026rdquo; Kinematics is the mathematical language that translates these high-level strategies into physical motion.\nKinematics is a subfield of mechanics that deals with the motion of bodies without considering the forces that cause the motion.\nForward Kinematics Forward kinematics is the process of calculating the robot’s overall position and orientation (often called the pose) based on known actuator positions or speeds.\nQ : \u0026ldquo;If my left wheel spins at $X$ RPM and my right wheel spins at $Y$ RPM, where will my robot be after 5 seconds?\u0026rdquo; This is essential for Odometry, where the robot tracks its own movement over time by integrating wheel encoder data.\nInverse Kinematics Inverse kinematics is the opposite. It starts with a desired destination or target pose and calculates the specific actuator commands needed to get there.\nQ : \u0026ldquo;I want my robot to move to coordinate $(x, y)$ with an orientation of $\\theta$. How fast should each wheel spin to achieve this?\u0026rdquo;. This is the foundation of Motion Control. When a path planner tells the robot to follow a specific trajectory, inverse kinematics translates that path into actual motor signals.\nMobile Robot kinematics Let\u0026rsquo;s compare robot arm and mobile robot.\nFigure 15. Robot Arm and Mobile Robot Feature Robot Arm Mobile Robot Input (Joint Space) $q_1, q_2, q_3, q_4$ (Joint Angles) $v_L, v_R$ (Left/Right Wheel Velocities) End-Effector (Pose) $x, y, \\theta$ (Gripper Position/Orientation) $x_R, y_R, \\theta_R$ (Robot Center Pose) Physical Parameter Link Lengths ($l_1, l_2, \\ldots$) Wheel Radius ($r$), Track Width ($b$) Forward Kinematics Calculating Tip Pose from Joint Angles Calculating Robot Trajectory from Wheel RPM Inverse Kinematics Finding Joint Angles to reach a Target Point Finding Wheel Velocities to reach a Target Pose Constraints Limited by the Reach (Workspace) Limited by Direction (Non-holonomic) Mobile robots present a unique set of challenges compared to fixed robotic arms.\nNo Unique Mapping: For robotic arms, joint angles map directly to a specific endpoint. In mobile robots, wheel encoder values do not map to a unique pose because factors like wheel slip can occur. Unbound Mobility: Unlike arms fixed to a base, mobile robots move freely and \u0026ldquo;unbound\u0026rdquo; throughout their entire environment. No Direct Measurement: There is no instantaneous way to measure a mobile robot\u0026rsquo;s exact position (unlike an arm\u0026rsquo;s end-effector). Integration Errors: A robot’s position must be calculated by integrating its motion over time. Because this depends on the specific path taken, small measurement errors accumulate, leading to inaccuracies in the final position estimate. Kinematic Constraints: Understanding mobile motion starts with the physical constraints of the wheels—such as the fact that standard wheels can roll forward but cannot slide sideways. Differential Drive Mobile Robot The differential drive system is the most common and simplest drive mechanism for mobile robots. It consists of two wheels mounted on a common axis, each controlled by an independent motor.\nFigure 16. Mobile Robot $r$: Radius of the wheels. $b$: Distance between the two wheels (track width). $P$: The center point of the robot (midpoint of the wheel axis). $v_l(t), v_r(t)$: Linear velocities of the left and right wheels. $\\omega_l(t), \\omega_r(t)$: Angular velocities of the left and right wheels. $R$: Instantaneous curvature radius of the robot\u0026rsquo;s trajectory. $IC$ (or $ICC$): Instantaneous Center of Rotation. The linear velocity of each wheel is directly proportional to its angular velocity and radius: $$v_l(t) = r \\cdot \\omega_l(t), \\quad v_r(t) = r \\cdot \\omega_r(t)$$\nFrom these, we can derive the linear velocity ($v$) and angular velocity ($\\omega$) of the entire robot body:\nLinear Velocity ($v$): The average speed of the two wheels.\n$$v(t) = \\frac{v_r(t) + v_l(t)}{2}$$\nAngular Velocity ($\\omega$): The difference in wheel speeds divided by the track width. $$\\omega(t) = \\frac{v_r(t) - v_l(t)}{b}$$\nSpecial Cases of Motion The behavior of the robot changes significantly based on the relative velocities of the wheels:\nStraight-line Motion ($v_l = v_r$): When both wheels spin at the same speed, $\\omega = 0$ and the robot moves in a straight line. Pure Rotation ($v_l = -v_r$): When wheels spin at the same speed in opposite directions, $v = 0$ and the robot rotates in place around its center ($P$). Curved Path ($v_l \\neq v_r$): The robot follows a circular trajectory around the $IC$. This motion is highly sensitive to velocity errors; small deviations in wheel speed lead to different trajectories. Kinematic Model in the Global Frame To track the robot’s pose $[x, y, \\theta]^T$ in the initial (global) frame, we integrate the velocities over time. The differential equations for the robot\u0026rsquo;s pose are: $$\\dot{x}(t) = v(t) \\cdot \\cos\\theta(t)$$ $$\\dot{y}(t) = v(t) \\cdot \\sin\\theta(t)$$ $$\\dot{\\theta}(t) = \\omega(t)$$\nThe final position at time $t$ is calculated by integrating these equations:\n$$\\begin{cases} x(t) = \\int_{0}^{t} v(\\tau) \\cos\\theta(\\tau) d\\tau \\\\ y(t) = \\int_{0}^{t} v(\\tau) \\sin\\theta(\\tau) d\\tau \\\\ \\theta(t) = \\int_{0}^{t} \\omega(\\tau) d\\tau \\end{cases}$$\n","permalink":"https://yeongseung.github.io/posts/slam/post/","summary":"\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eI’ve always been eager to dive into the fundamentals of robotics, and I finally have the opportunity through my \u0026ldquo;Autonomous Mobile Robot\u0026rdquo; course. I decided to write this blog post not only to solidify what I’ve learned but also to create a practical reference for my future work.\u003c/p\u003e\n\u003cp\u003eThis post focuses on the essential keywords of autonomous systems: \u003cstrong\u003ePerception, Navigation, Localization, Planning, and SLAM.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAt its core, an \u0026ldquo;autonomous robot\u0026rdquo; is defined by its ability to:\u003c/p\u003e","title":"Zero to SLAM"},{"content":"Introduction This blog reflects my journey through Machine Learning, Reinforcement Learning, and Robotics. While my current work spans these areas, my academic roots are in Statistics, a foundation that continues to shape how I think about problems. From a theoretical standpoint, I believe Statistics remains the core foundation of modern machine learning.\nOver time, while studying statistics, I often encountered concepts that felt confusing, subtle, or easy to misunderstand. Some of them became clearer after revisiting the ideas multiple times or thinking about them from different perspectives, while others are still things I occasionally find myself reflecting on.\nIn this series of notes, I plan to write down those questions, confusions, and small clarifications, both as a personal record and as a way to organize my own understanding of statistical concepts.\nUncategorized (yet) What exactly is the random variable? First, I want to clarify this since notations related to it are sometimes confusing. Random variable is a function from a sample space to the real numbers. It is a function that maps each outcome of a random experiment to a real number. $$ X : \\Omega \\rightarrow \\mathbb{R} $$\nAlthough it is called a variable, it is important to remember that a random variable is actually a function, not a variable in the usual sense.\nFor example, if we flip two coins, the sample space is {HH, HT, TH, TT}. And the random variable X is the number of heads. $$ X(HH) = 2, X(HT) = 1, X(TH) = 1, X(TT) = 0 $$ $$ X : \\{HH, HT, TH, TT\\} \\rightarrow \\{0, 1, 2\\} $$ So, random variable is a function, not a variable.\n$$ P(X=x) $$ At first glance, this can be confusing. What exactly is $x$? Is $x$ an input of the function $X$? No. The input of $X$ is an outcome of the random experiment. So, $x$ is the output of the function $X$. More precisely, $$ P(X=x) = P(\\{ \\omega \\in \\Omega | X(\\omega) = x \\}) $$ For example, $P(X=1) = P(\\{ \\omega \\in \\Omega | X(\\omega) = 1 \\}) = P(\\{ HT, TH \\}) = 1/2$.\nIntuitively, Why Does Integrating $-F(x)$ and $1-F(x)$ Give $E(X)$? Typically, the definition of expected value is given as: $$ E(X) = \\int_{-\\infty}^{\\infty} x f(x) dx \\tag{1} $$ But there\u0026rsquo;s another way to get $E(X)$ using the cumulative distribution function $F(x)$. $$ E(X) = -\\int_{-\\infty}^{0} F(x) dx + \\int_{0}^{\\infty} (1-F(x)) dx \\tag{2} $$ Intuitively, (1) is quite straightforward. It is the weighted average of all possible values of $X$ by their probabilities. If you think it in visual way, it is like this:\nFigure 1. $E(X) = \\int_{-\\infty}^{\\infty} x f(x) dx$ -1.5 * 0.12 + -1.4 * 0.13 + \u0026hellip; Intuitively, this sure be $E(X)$.\nWhay about (2)? It is like this:\nFigure 2. $E(X) = -\\int_{-\\infty}^{0} F(x) dx + \\int_{0}^{\\infty} (1-F(x)) dx$ In the positive side ($\\int_{0}^{\\infty} 1-F(x) dx$), imagine a crowd moving forward. At each step $dx$, the height $1-F(x)$ represents the proportion of people who have traveled at least distance $x$. Summing these vertical slices (height $\\times$ width) naturally gives us the average distance traveled (if the $y$-axis were measured in kilometers instead of proportions, this integral would simply represent the total distance traveled by all people moving in the positive direction).\nOn the negative side ($\\int_{-\\infty}^{0} F(x) dx$), the height $F(x)$ represents those who moved backward beyond point $x$. Since the integration of a positive height over a width always yields a positive area, we manually apply a negative sign to account for the direction. Summing these two parts gives us the true center of mass: $E(X)$.\nThen What about $E(X^2)$?\nFigure 3. Example distribution Think of the figure above as representing people running. Because the y-axis represents the number of people, the total shaded area does not represent $E(X)$, but rather the total distance run by all people.\nSuppose that after the race is over, we want to know the total distance everyone ran. One way to compute this is the following. We shout to the runners: \u0026ldquo;Raise your hand if you ran more than 1 meter!\u0026rdquo;, \u0026ldquo;More than 2 meters!\u0026rdquo;, \u0026ldquo;More than 3 meters!\u0026rdquo;, \u0026hellip;, \u0026ldquo;More than 150 meters!\u0026rdquo;, \u0026hellip;, \u0026ldquo;More than 200 meters!\u0026rdquo;. Each time we count how many people raise their hands. If we add up all these counts, we obtain the total distance run by everyone.\nTo see why this works, consider a particular runner who ran 150 meters. He will raise his hand for every question from 1 meter up to 150 meters. Therefore he will be counted 150 times: $$ 1 + 1 + \u0026hellip; + 1 = 150 $$ So each runner contributes exactly his running distance.\nNow suppose we want to compute something different: the sum of squared distances. For the same runner who ran 150 meters, how could we obtain $150^2$ using a similar counting idea? Consider summing even numbers from 2 to 300: $$ 2(1+2+3+\u0026hellip;+150).$$ $$ 2* \\frac{150(150+1)}{2} = 150*151 $$ This is slightly larger than $150^2$. The discrepancy comes from the fact that we were counting in discrete steps (1 meter, 2meters, 3meters, \u0026hellip;). When we move to a continuous perspective, these small step errors disappear. In the continuous case, the square can be represented exactly as $$x^2 = 2 \\int_{0}^{x} t dt$$ This leads to the identify $$ E(X^2) = 2 \\int_{0}^{\\infty} x(1-F(x)) dx $$ As generalization, we can get the following identity: $$ E(X^2) = 2 \\int_{0}^{\\infty} xP(|X| \u0026gt; x) dx $$ $$ E(X^2) = 2 \\int_{0}^{\\infty} x(1-F(x) + F(-x)) dx $$\n","permalink":"https://yeongseung.github.io/posts/statistics/post/","summary":"\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eThis blog reflects my journey through Machine Learning, Reinforcement Learning, and Robotics. While my current work spans these areas, my academic roots are in Statistics, a foundation that continues to shape how I think about problems. From a theoretical standpoint, I believe Statistics remains the core foundation of modern machine learning.\u003c/p\u003e\n\u003cp\u003eOver time, while studying statistics, I often encountered concepts that felt confusing, subtle, or easy to misunderstand. Some of them became clearer after revisiting the ideas multiple times or thinking about them from different perspectives, while others are still things I occasionally find myself reflecting on.\u003c/p\u003e","title":"Statistical Notes: Questions, Clarifications, and Intuition"},{"content":"Introduction The transformer architecture has become the foundation of modern deep learning, powering everything from language models to vision models. Since I\u0026rsquo;ll be referencing transformers frequently in future posts, I wanted to create my own comprehensive analysis of the architecture. In this post, I\u0026rsquo;ll walk through building a transformer-based translation model step by step, following the original paper by Vaswani et al. (2017)1. We\u0026rsquo;ll dive into the code, understanding each component as we implement it.\nAccording to Vaswani et al. (2017), the transformer architecture was introduced as a translation model. It can be inferred from the paper that at the time of its introduction, RNN-based encoder-decoder architectures were commonly used for translation tasks. However, as you might know, RNN-based models compute sequentially, which limits parallelization.\nFigure 1. RNN example Figure 1 illustrates how an RNN processes a sequence. Take the input \u0026ldquo;I\u0026rdquo; from the phrase \u0026ldquo;I love you\u0026rdquo;: it is tokenized (e.g., to the id 15) and then passed through an embedding layer to obtain $x_1 \\in \\mathbb{R}^{512}$ (assuming embedding size 512). At this first time step there is no previous hidden state, so the model computes $h_1$ as $W_x x_1$, where $W_x \\in \\mathbb{R}^{1024 \\times 512}$. The result is the hidden state $h_1 \\in \\mathbb{R}^{1024}$. For the next word \u0026ldquo;love\u0026rdquo;, the token (e.g., 42) is embedded to $x_2 \\in \\mathbb{R}^{512}$. The recurrence appears here: the new hidden state is $h_2 = W_h h_1 + W_x x_2$, where $W_h \\in \\mathbb{R}^{1024 \\times 1024}$ is the recurrent weight. So at each step $t$, $h_t = W_h h_{t-1} + W_x x_t$ (with $h_0$ treated as zero or omitted at $t=1$). Bias terms are omitted in the figure and in the equations for simplicity.\nSince RNNs were dominant at the time, attention mechanisms were mainly integrated into RNN-based architectures, which still suffered from sequential computation and limited scalability.\nPeople also considered CNNs for language modeling, but CNNs can only see locally based on the kernel size, so it is difficult to learn dependencies between far-away words.\nThe Transformer architecture was introduced to overcome these constraints by eliminating recurrence and convolution, relying solely on attention mechanisms. This idea is reflected in the paper’s title, Attention Is All You Need.\nThe implementation in this post draws on the code at hkproj/pytorch-transformer2.\nBig Picture Figure 2. The Transformer model architecture (Source: Vaswani et al., 2017) Left is the encoder and right is the decoder. Consider the architecture as an English–Korean translator (because Korean is relatively easy for me). The encoder receives the English sentence \u0026ldquo;I love you.\u0026rdquo; The decoder does not take this sentence directly as input. Instead, it generates the Korean sentence autoregressively. Its inputs are the previously generated tokens, for example: [\u0026quot;\u0026quot;], [\u0026quot;\u0026quot;, \u0026ldquo;나는\u0026rdquo;], [\u0026quot;\u0026quot;, \u0026ldquo;나는\u0026rdquo;, \u0026ldquo;너를\u0026rdquo;], [\u0026quot;\u0026quot;, \u0026ldquo;나는\u0026rdquo;, \u0026ldquo;너를\u0026rdquo;, \u0026ldquo;사랑해\u0026rdquo;]\nAt each step, the decoder predicts the next token while attending to the encoder’s representations through cross-attention. No matter how long the input is, the encoder runs only once, and its output is fed into the decoder’s multi-head attention.\nEncoder Figure 3. Encoder (Source: Vaswani et al., 2017) Input Embedding Building Input Embedding is quite simple since Pytorch already made pre-built one.\nclass InputEmbedding(nn.Module): def __init__(self, d_model: int, vocab_size: int): super().__init__() self.d_model = d_model # dimension of the model (embedding size) self.vocab_size = vocab_size self.embedding = nn.Embedding(vocab_size, d_model) def forward(self, x): return self.embedding(x) * math.sqrt(self.d_model) # output: (batch_size, seq_len, d_model) # The output is then passed to the positional encoding layer. nn.Embedding\nThe embedding layer must be trainable because the model needs to capture semantic relationships between words. Similar words should be positioned close to each other in the vector space, while dissimilar words should be placed farther apart.\nFor this reason, nn.Embedding is implemented as a parameter matrix (probably). Conceptually, it can be written in code as follows:\nclass Embedding(nn.Module): def __init__(self, vocab_size=10000, d_model=512): super().__init__() self.vocab_size = vocab_size self.d_model = d_model # (vocab_size, d_model) learnable parameter matrix self.W = nn.Parameter(torch.randn(vocab_size, d_model)) def forward(self, x): # x: (batch_size, seq_len) containing token indices # return: (batch_size, seq_len, d_model) return self.W[x] Anyway, for the practical use, Let\u0026rsquo;s stick to nn.Embedding, which must be implemented in more efficient way.\nAlso, In the paper, It says \u0026ldquo;In the embedding layers, we multiply those weights by $\\sqrt{d_{model}}$.\u0026rdquo; But why? This is just my theory, but I think they probably initialized the embeddings with a normal distribution like $N(0,(1/\\sqrt{d_{model}})^2)$. By multiplying the embeddings by $\\sqrt{d_{model}}$, the resulting values would have a variance of 1, roughly matching the range of the positional encodings, which go from -1 to 1. This scaling helps balance the contributions of the embeddings and positional encodings in the model.\nSince PyTorch\u0026rsquo;s nn.Embedding is already initialized as $N(0,1)$, I probably don\u0026rsquo;t need to multiply by $\\sqrt{d_{model}}$ in this implementation. That scaling was mainly for the original initialization in the paper, which probably used a smaller variance. But let\u0026rsquo;s keep this setting for now and see how it works.\nPositional Encoding RNN has ordering feature, but this architecture is without RNN. So, Intuitively, It needs one feature for recognizing position of each token. It totally makes sense. Let\u0026rsquo;s see the way that the paper suggested. $$ PE_{pos,2i}=sin(\\frac{pos}{10000^{\\frac{2i}{d_{model}}}}), when\\text{ } i:even$$ $$ PE_{pos,2i+1}=cos(\\frac{pos}{10000^{\\frac{2i}{d_{model}}}}), when\\text{ } i:odd$$\nSo, it\u0026rsquo;s telling that first position\u0026rsquo;s positional vector would be like this : $$[sin(\\frac{0}{10000^{\\frac{2*0}{512}}}),cos(\\frac{0}{10000^{\\frac{2*0+1}{512}}}),\u0026hellip;,sin(\\frac{0}{10000^{\\frac{2*255}{512}}}),cos(\\frac{0}{10000^{\\frac{2*255+1}{512}}})]$$ $$[0,1,\u0026hellip;,0,1]$$ and second positional vector would be like: $$[sin(\\frac{1}{10000^{\\frac{2*0}{512}}}),cos(\\frac{1}{10000^{\\frac{2*0+1}{512}}}),\u0026hellip;,sin(\\frac{1}{10000^{\\frac{2*255}{512}}}),cos(\\frac{1}{10000^{\\frac{2*255+1}{512}}})]$$\nFigure 4. Positional encoding Figure 5. Positional encoding matrix Based on the figures, it makes sense that every position can get a unique positional vector. The author presents a few benefits of it.\nFirst, If we use a pair of sine and cosine (instead of using only one of them), we can represent the relationship between any two positions as a linear transformation (Rotation). This means that a function calculating the distance between position 2 and position 5 is linear and simple, making it easier for the model to find the function. Plus, the distance between position 2 and position 5 is the same as the distance between position 8 and position 11.\nSecond, This formulation enables the model to generalize to sequence lengths beyond those encountered during training. With learned positional embeddings (not this one), a model trained up to length 100 may struggle when encountering position 101, since it has never learned that index. In contrast, the sinusoidal formulation can generate a positional vector for any new position simply by plugging it into the formula.\nAlso, the values remain between -1 and 1, which is a great advantage when scaling.\nAbout adding this positional encoding and input embedding, I thought positional encoding could harm the meaning of the words (Input Embeddings) by adding some noise. For example, I imagined \u0026ldquo;chimpanzee\u0026rdquo; at position 5 could be the same as \u0026ldquo;cat\u0026rdquo; at position 12. However, this post3 suggests an interesting possibility. As we saw in Figure 5, there is not that much difference after about dimension 200. So, the model might relatively avoid dimensions 0 to 200 and use dimesion atfer 200 when it forms input embeddings during training.\nclass PositionalEncoding(nn.Module): def __init__(self, d_model: int, max_seq_len: int, dropout: float): super().__init__() self.d_model = d_model self.max_seq_len = max_seq_len self.dropout = nn.Dropout(dropout) # (max_seq_len, d_model) pe = torch.zeros(max_seq_len, d_model) # (max_seq_len, 1) position = torch.arange(0, max_seq_len, dtype=torch.float32).unsqueeze(1) div_term = torch.exp(torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model)) pe[:, 0::2] = torch.sin(position * div_term) pe[:, 1::2] = torch.cos(position * div_term) # (max_seq_len, d_model) -\u0026gt; (1, max_seq_len, d_model) pe = pe.unsqueeze(0) # Buffer: not a trainable parameter, but part of the model state # it will be saved and loaded with the model self.register_buffer(\u0026#39;pe\u0026#39;, pe) # x: (batch_size, seq_len, d_model) # x is the output of the input embedding layer. def forward(self, x): x = x + self.pe[:, :x.size(1), :] return self.dropout(x) # output: (batch_size, seq_len, d_model) Multi-Head Attention Figure 6. Encoder (Source: Vaswani et al., 2017) Okay, so far, our output from positional encoding has meaning of words and position information. Now, we need to add another feature for the model to understand the relationship between words. Attention will do the job.\nFigure 7. Attention (Source: Vaswani et al., 2017) Figure 7, which was originally from the paper, is a good visualization of attention, but let me explain it in a bit more detail.\nFigure 8. Multi-head attention The objective of attention is to calculate the similarity between the query and the key, and then use the similarity to weight the value.\n(1) First, we generate three matrices: $W_Q$, $W_K$, $W_V$, and multiply them with the input. The outputs are query $Q$, key $K$, and value $V$. The space of Q, K, V is [max_seq_len, d_model], which is same as the input embedding, positional encoding, and the sum of them. If we set the number of head to 4, for example, the space of Q, K, V would be [max_seq_len, 4, d_model/4]. d_model/4 is d_k and d_v(I didn’t use $d_v$ in the notation because $d_k$ and $d_v$ are the same in this case). Next, we transpose the head dimension and the sequence length dimension to obtain the shape [4, max_seq_len, d_model/4]. This allows each head to process the entire sequence independently and enables efficient batch computation across heads. So far, Q, K, and V are [4, max_seq_len, d_k].\nAt this point, although $W_Q$, $W_K$, and $W_V$ are different, Q, K, and V share the same architecture. There is no mechanism that differentiates their roles yet.\nFigure 9. Multi-head attention second step (2) Next, we dot product Q and K. The result is [4, max_seq_len, max_seq_len]. [max_seq_len, max_seq_len] is called attention score matrix. So, we have 4 attention score matrices. Then, we divide the attention score matrix by the square root of d_k before applying softmax. This is to prevent the dot product from becoming too large.\nBut why square root of d_k? The attention score matrix is computed by dot product of $Q$ and $K^T$, which is [4, max_seq_len, d_k] and [4, d_k, max_seq_len]. We can assume that the elements of Q and K are approximately independent and follow a normal distribution N(0,1) or at least constant variance.\nSo, the dot product of Q and K follows normal distribution N(0,d_k). So, the square root of d_k is to normalize the dot product, which leads to each sample of attention score matrix to be a random variable from normal distribution N(0,1).\nWhy do we want the dot product to follow a distribution close to N(0,1)? The reason is stability before applying softmax. If the variance of the dot product is large (i.e., proportional to d_k), the attention scores can contain very large positive and negative values. Since softmax involves exponentiation, large positive values will dominate, while large negative values will be pushed close to zero. As a result, the output distribution becomes extremely sharp, almost like a one-hot vector.\nWhen softmax becomes too sharp, the gradients can become very small for most positions, making learning unstable or slow. By scaling the dot product by $\\sqrt{d_k}$, we normalize its variance to 1. This keeps the softmax input values in a reasonable range and prevents the attention distribution from becoming overly peaked.\nFinally, we apply a row-wise softmax to the attention score matrix. The resulting shape remains [4, max_seq_len, max_seq_len], but each row now represents a valid probability distribution. This step is crucial as it formally assigns distinct roles to Q and K: Q functions as the \u0026lsquo;query\u0026rsquo; (the seeker), while K acts as the \u0026lsquo;key\u0026rsquo; (the target).\nFigure 10. Multi-head attention third step (3) This normalized attention score matrix [4, max_seq_len, max_seq_len] is then multiplied by V [4,max_seq_len, d_k] to get the final output. The shape remains [4, max_seq_len, d_k].\nWhy do we conceptually separate $V$ and multiply it at the end? The attention score matrix defines how strongly each query is related to every key. However, these scores only represent similarity—they do not contain the information itself. The matrix V provides the vectors that will be combined according to these similarity weights.\nI think, this late-stage multiplication by $V$ serves a purpose similar to a residual connection within the attention mechanism itself. By keeping the \u0026ldquo;content\u0026rdquo; ($V$) separate from the \u0026ldquo;addressing logic\u0026rdquo; ($Q, K$), we ensure that the original information is not distorted or lost during the complex similarity computations. Instead of being completely overwritten, the values in $V$ are selectively combined based on the scores, which helps the model maintain a stable flow of information across layers. While it is possible for some information to \u0026ldquo;leak\u0026rdquo; into the scores, the designated role of $V$ is to provide a reliable, high-fidelity base of information that the model can \u0026ldquo;extract\u0026rdquo; and \u0026ldquo;mix\u0026rdquo; according to the guidance provided by $Q$ and $K$.\nFigure 11. Multi-head attention fourth step (4) Finally, we concatenate the outputs of the four heads and multiply by $W_O$ to get the final output. The shape remains [max_seq_len, d_model].\nIf we were to only concatenate the outputs, we would simply be laying out the information from the four independent heads side-by-side. At this stage, the insights captured by each head—hopefully different insights such as grammatical structures, semantic relationships, or distant dependencies—exist in isolation within their own subspaces.\nBy multiplying by $W_O$, we can combine the information from the four heads into a single matrix.\nclass MultiHeadAttentionBlock(nn.Module): def __init__(self, d_model: int, h: int, dropout: float): super().__init__() # dimension of the model self.d_model = d_model # number of heads self.h = h assert d_model % h == 0, \u0026#34;d_model must be divisible by h\u0026#34; # d_model = h * d_k self.d_k = d_model // h # weights for the query, key, and value matrices self.w_q = nn.Linear(d_model, d_model) # Wq self.w_k = nn.Linear(d_model, d_model) # Wk self.w_v = nn.Linear(d_model, d_model) # Wv # weights for the output matrix self.w_o = nn.Linear(d_model, d_model) # Wo self.dropout = nn.Dropout(dropout) # query: (batch_size, h, max_seq_len, d_k) # key: (batch_size, h, max_seq_len, d_k) # value: (batch_size, h, max_seq_len, d_k) # mask: (batch_size, h, max_seq_len, max_seq_len) # dropout: dropout rate @staticmethod # static method: not bound to an instance, but to the class def attention(query, key, value, mask=None, dropout=None): d_k = query.shape[-1] # (batch_size, h, max_seq_len, d_k) @ (batch_size, h, d_k, max_seq_len) -\u0026gt; (batch_size, h, max_seq_len, max_seq_len) attention_scores = (query @ key.transpose(-2, -1)) / math.sqrt(d_k) if mask is not None: attention_scores = attention_scores.masked_fill(mask == 0, -1e9) # dim=-1: softmax over the last dimension attention_scores = attention_scores.softmax(dim=-1) # (batch_size, h, max_seq_len, max_seq_len) if dropout is not None: attention_scores = dropout(attention_scores) # (batch_size, h, max_seq_len, max_seq_len) @ (batch_size, h, max_seq_len, d_k) -\u0026gt; (batch_size, h, max_seq_len, d_k) return (attention_scores @ value), attention_scores # the latter is only for visualization def forward(self, q, k, v, mask=None): query = self.w_q(q) # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, d_model) key = self.w_k(k) # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, d_model) value = self.w_v(v) # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, d_model) # split the query, key, and value into h heads # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, h, d_k) -\u0026gt; (batch_size, h, max_seq_len, d_k) query = query.view(query.shape[0], query.shape[1], self.h, self.d_k).transpose(1, 2) key = key.view(key.shape[0], key.shape[1], self.h, self.d_k).transpose(1, 2) value = value.view(value.shape[0], value.shape[1], self.h, self.d_k).transpose(1, 2) x, self.attention_scores = MultiHeadAttentionBlock.attention(query, key, value, mask, self.dropout) # output x: (batch_size, h, max_seq_len, d_k) # (batch_size, h, max_seq_len, d_k) -\u0026gt; (batch_size, max_seq_len, h, d_k) --\u0026gt; (batch_size, max_seq_len, d_model) batch_size, _, seq_len, _ = x.size() x = x.transpose(1, 2).contiguous().reshape(batch_size, seq_len, self.d_model) # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, d_model) return self.w_o(x) Let\u0026rsquo;s see the implementation of the multi-head attention block. In the forward method, we take the query, key, and value as input.\nBut, isn\u0026rsquo;t it weird that we take the query, key, and value seperatly? In self-attention, the query, key, and value are all derived from the same input.So why not simply pass a single tensor instead? The reason is flexibility. While self-attention uses the same input to generate Q, K, and V, other forms of attention—such as cross-attention in the decoder—use different sources. In cross-attention, the query comes from the decoder, whereas the key and value come from the encoder output.\nBy accepting query, key, and value separately, this implementation remains general and can be reused for both self-attention and cross-attention.\nquery = self.w_q(q) # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, d_model) key = self.w_k(k) # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, d_model) value = self.w_v(v) # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, d_model) After above operations, Q, K, and V matrices are generated.\n# (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, h, d_k) -\u0026gt; (batch_size, h, max_seq_len, d_k) query = query.view(query.shape[0], query.shape[1], self.h, self.d_k).transpose(1, 2) key = key.view(key.shape[0], key.shape[1], self.h, self.d_k).transpose(1, 2) value = value.view(value.shape[0], value.shape[1], self.h, self.d_k).transpose(1, 2) query.shape[0] is batch_size, query.shape[1] is max_seq_len, self.h is number of heads, self.d_k is dimension of the key (and value). Of course, self.h * self.d_k is d_model. If our d_model is 512 and we set the number of heads to 8, self.d_k is 64. Because first two dimensions are batch_size and max_seq_len, which is same as the before, the view operation is just to split the d_model dimension into h heads. [batch_size, max_seq_len, h, d_k].\nNext, we transpose the head dimension and the sequence length dimension to obtain the shape [h, batch_size, max_seq_len, d_k]. This allows each head to process the entire sequence independently and enables efficient batch computation across heads.\n# query: (batch_size, h, max_seq_len, d_k) # key: (batch_size, h, max_seq_len, d_k) # value: (batch_size, h, max_seq_len, d_k) # mask: (batch_size, h, max_seq_len, max_seq_len) # dropout: dropout rate @staticmethod # static method: not bound to an instance, but to the class def attention(query, key, value, mask=None, dropout=None): d_k = query.shape[-1] # (batch_size, h, max_seq_len, d_k) @ (batch_size, h, d_k, max_seq_len) -\u0026gt; (batch_size, h, max_seq_len, max_seq_len) attention_scores = (query @ key.transpose(-2, -1)) / math.sqrt(d_k) if mask is not None: attention_scores = attention_scores.masked_fill(mask == 0, -1e9) # dim=-1: softmax over the last dimension attention_scores = attention_scores.softmax(dim=-1) # (batch_size, h, max_seq_len, max_seq_len) if dropout is not None: attention_scores = dropout(attention_scores) # (batch_size, h, max_seq_len, max_seq_len) @ (batch_size, h, max_seq_len, d_k) -\u0026gt; (batch_size, h, max_seq_len, d_k) return (attention_scores @ value), attention_scores # the latter is only for visualization x, self.attention_scores = MultiHeadAttentionBlock.attention(query, key, value, mask, self.dropout) # output x: (batch_size, h, max_seq_len, d_k) # self.attention_scores: (batch_size, h, max_seq_len, max_seq_len) By calling the attention function, we get the output x and the attention scores. x is the outout of (3) in Figure 10. [batch_size, h, max_seq_len, d_k].\nMask for Encoder is called padding mask. If the token is padding, the attention score is set to -1e9. If max_seq_len is 5, and \u0026lsquo;I\u0026rsquo;, \u0026rsquo;love\u0026rsquo;, \u0026lsquo;you\u0026rsquo; are the tokens, then the mask is : $$ \\begin{bmatrix} 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\end{bmatrix} $$ However, at first glance, I thought that the mask should be like this: $$ \\begin{bmatrix} 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\end{bmatrix} $$ Because [4,1] and other elements are also related to padding. But it was not a case. The reason we use the first mask instead of the second lies in computational efficiency and the independent nature of the Transformer layers.\nFirst, using a uniform mask across all rows allows for Broadcasting. Instead of creating a complex, row-specific mask, the hardware can simply broadcast a single $[1 \\times 5]$ vector across the entire matrix. This significantly speeds up the operation on modern GPUs.\nSecond, you might worry that these \u0026ldquo;garbage\u0026rdquo; padding rows (rows 4 and 5) will eventually pollute the actual information in rows 1 to 3. However, this doesn\u0026rsquo;t happen because the subsequent layers—$W_O$ and the Feed Forward Network (FFN)—process each row independently.\nSince these layers do not mix information across different rows, the noise in the padding rows stays trapped within those rows. As long as we have masked the Keys (columns) to prevent real tokens from looking at padding, the real tokens remain pure. Therefore, we can simply ignore these padding rows at the very end of the model or during loss calculation, making a dedicated row-wise mask unnecessary during the encoding process.\n# (batch_size, h, max_seq_len, d_k) -\u0026gt; (batch_size, max_seq_len, h, d_k) --\u0026gt; (batch_size, max_seq_len, d_model) x = x.transpose(1,2).contiguous().view(x.shape[0], x.shape[1], self.h * self.d_k) By transposing the head dimension and the sequence length dimension, we get the shape [batch_size, max_seq_len, h, d_k]. Then, we concatenate the heads and get the shape [batch_size, max_seq_len, d_model] again.\nreturn self.w_o(x) Finally, we multiply by $W_O$ to get the final output, which is the output of (4) in Figure 11, which is Multi-Head Attention Output. So, the shape is [batch_size, max_seq_len, d_model].\nResidual Connection and Layer Normalization Figure 12. Encoder (Source: Vaswani et al., 2017) Now, it\u0026rsquo;s time to make the Add \u0026amp; Norm part, which is a residual connection and layer normalization. First, Let\u0026rsquo;s see the layer normalization4. $$LN(x)=\\alpha\\frac{x-\\mu}{\\sqrt{\\sigma^2+\\epsilon}}+\\beta$$\nclass LayerNormalization(nn.Module): def __init__(self, d_model: int, eps: float = 1e-6): super().__init__() self.eps = eps # learnable parameters self.alpha = nn.Parameter(torch.ones(d_model)) # scale self.bias = nn.Parameter(torch.zeros(d_model)) # shift # x: (batch_size, max_seq_len, d_model) def forward(self, x): # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, 1) mean = x.mean(dim=-1, keepdim=True) # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, 1) variance = x.var(dim=-1, keepdim=True) # (batch_size, max_seq_len, 1) -\u0026gt; (batch_size, max_seq_len, d_model) return self.alpha * (x - mean) / torch.sqrt(variance + self.eps) + self.bias Whatever the input is, its shape is typically [batch_size, max_seq_len, d_model]. From the code, we can see that the mean and variance are computed along the last dimension (dim=-1). In other words, normalization is performed token-wise. Each token vector is normalized independently from other tokens in the same sequence. This means that :\nEach token\u0026rsquo;s representation is centered and scaled on its own. There is no statistical interaction across tokens during normalization. The normalization process does not introduce cross-token dependency. If we computed the mean and variance along the sequence dimension instead (i.e., column-wise normalization), the normalization statistics would depend on other tokens in the sequence. That would make the representation of one token partially dependent on the values of other tokens through the normalization step itself. Such cross-token coupling is undesirable because:\nIt entangles token representations. It may interfere with autoregressive decoding. It makes the representation less locally stable. Transformer models are designed so that interactions between tokens occur explicitly through attention, not implicitly through normalization.\nAlso, The decisive reason LayerNorm is needed lies in the residual structure: $$x+Sublayer(x)$$\nThis residual connection continuously adds new signals to the original representation. If this process is repeated across many layers, the variance of activations can grow uncontrollably. Whitout normalization:\nThe magnitude of activations may explode. Gradient flow can become unstable. Training deep stacks becomes difficult. Layer normalization stabilizes this process by controlling the scale of the combined signal. $$LayerNorm(x+Sublayer(x))$$\nThus, LayerNorm acts as a variance regulator that keeps the representation well-conditioned throughout deep residual stacking.\nIn addition, LayerNorm includes two learnable parameters, $\\alpha$ (scale) and $\\beta$ (shift). After normalization forces the representation to have zero mean and unit variance, these parameters allow the model to re-scale and re-shift the normalized output.\nWithout $\\alpha$ and $\\beta$, normalization would strictly constrain every token representation to the same standardized distribution. However, such rigid normalization could limit the expressive power of the network. By introducing learnable scale and shift parameters, the model can recover any necessary distribution if needed, while still benefiting from stabilized training.\nIn other words, LayerNorm first standardizes the representation for numerical stability, and then $\\alpha$ and $\\beta$ give the model the flexibility to learn the most useful scaling for downstream computation. After the affine transformation, the output has mean approximately $\\beta$ and variance approximately $\\alpha^2$ (per token), but it is not necessarily Gaussian.\nNow, Let\u0026rsquo;s see the complete Add \u0026amp; Norm part.\nclass ResidualConnection(nn.Module): def __init__(self, d_model: int, dropout: float): super().__init__() self.dropout = nn.Dropout(dropout) self.norm = LayerNormalization(d_model) def forward(self, x, sublayer): return self.norm(x + self.dropout(sublayer(x))) Here, sublayer is a function and can represent any layer, such as a multi-head attention block or a feed-forward network.\nSince layer normalization is applied after adding the residual connection, this structure is called Post-LN. Although this follows the original formulation in the paper, more recent implementations often use the Pre-LN variant.\n$$x+Sublayer(LayerNorm(x))$$\nclass ResidualConnection(nn.Module): def __init__(self, d_model: int, dropout: float): super().__init__() self.dropout = nn.Dropout(dropout) self.norm = LayerNormalization(d_model) def forward(self, x, sublayer): return x + self.dropout(sublayer(self.norm(x))) In the Pre-LN structure, layer normalization is applied before the sublayer. Apprently, this design leads to more stable gradient flow and tends to work better in deeper models.\nFeed-Forward Figure 13. Encoder (Source: Vaswani et al., 2017) The last part to implement is the feed-forward network.\nclass FeedForwardBlock(nn.Module): def __init__(self, d_model: int, d_ff: int, dropout: float): super().__init__() self.linear_1 = nn.Linear(d_model, d_ff) # W1 and b1 self.linear_2 = nn.Linear(d_ff, d_model) # W2 and b2 self.dropout = nn.Dropout(dropout) self.d_model = d_model def forward(self, x): # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, d_ff) -\u0026gt; (batch_size, max_seq_len, d_model) return self.linear_2(self.dropout(torch.relu(self.linear_1(x)))) Just simply two linear layers and a dropout. But why we use Feed-Forward block? isn\u0026rsquo;t attention blocks enough?\nThe key difference is that attention mixes information across tokens, while the feed-forward network transforms each token representation independently. Attention allows each position to gather relevant information from the sequence, but it is mostly a weighted linear combination of value vectors. Without an additional non-linear transformation, the model’s expressive power would be limited.\nThe feed-forward block provides this non-linearity and increases the representational capacity of the model (There were no non-linearities at all before feed-forward!). After attention aggregates contextual information, the feed-forward network processes and refines that information at each position. In this way, attention handles interaction between tokens, and the feed-forward block performs deeper feature transformation.\nEncoder Block Figure 14. Encoder (Source: Vaswani et al., 2017) Now, let\u0026rsquo;s see the complete encoder block. It\u0026rsquo;s just a combination of all the parts we\u0026rsquo;ve seen so far.\nclass EncoderBlock(nn.Module): def __init__(self, d_model: int, self_attention_block: MultiHeadAttentionBlock, feed_forward_block: FeedForwardBlock, dropout: float): super().__init__() self.self_attention_block = self_attention_block self.feed_forward_block = feed_forward_block # two residual connections: one for the self-attention block and one for the feed-forward block self.residual_connections = nn.ModuleList([ResidualConnection(d_model, dropout) for _ in range(2)]) # source_mask is needed because the encoder needs to know what positions are valid and which are not. # \u0026#39;not valid\u0026#39; means that the position is padding def forward(self, x, source_mask=None): x = self.residual_connections[0](x, lambda x: self.self_attention_block(x, x, x, source_mask)) x = self.residual_connections[1](x, self.feed_forward_block) return x The code is quite straightforward. The first line of the forward method is exactly here :\nFigure 15. Part of the encoder block (Source: Vaswani et al., 2017) And the second line is also exactly here :\nFigure 16. Part of the encoder block (Source: Vaswani et al., 2017) Finally, In the paper, we need to repeat the encoder block N times.\nclass Encoder(nn.Module): def __init__(self, d_model: int, layers: nn.ModuleList): super().__init__() self.layers = layers self.norm = LayerNormalization(d_model) def forward(self, x, mask): for layer in self.layers: x = layer(x, mask) return self.norm(x) layers is a list of encoder blocks. As you can see, we apply layer normalization after the last encoder block.\nDecoder Figure 17. Decoder (Source: Vaswani et al., 2017) We built most of classes for decoder block during building encoder, so we can immediately go to DecoderBlock.\nDecoder Block class DecoderBlock(nn.Module): def __init__(self, d_model: int, self_attention_block: MultiHeadAttentionBlock, cross_attention_block: MultiHeadAttentionBlock, feed_forward_block: FeedForwardBlock, dropout: float): super().__init__() self.self_attention_block = self_attention_block self.cross_attention_block = cross_attention_block self.feed_forward_block = feed_forward_block self.residual_connections = nn.ModuleList([ResidualConnection(d_model, dropout) for _ in range(3)]) def forward(self, x, encoder_output, source_mask, target_mask): x = self.residual_connections[0](x, lambda x: self.self_attention_block(x, x, x, target_mask)) x = self.residual_connections[1](x, lambda x: self.cross_attention_block(x, encoder_output, encoder_output, source_mask)) x = self.residual_connections[2](x, self.feed_forward_block) return x Let\u0026rsquo;s see the first line of the forward method :\nx = self.residual_connections[0](x, lambda x: self.self_attention_block(x, x, x, target_mask)) Figure 18. Part of the decoder block (Source: Vaswani et al., 2017) In this step, we use a target mask instead of a source mask. In the context of this translator, the source mask is applied to the source language (English), while the target mask is for the target language (Korean). The source mask masks padding tokens in the source sequence. The target mask, however, not only masks padding tokens but also blocks access to future tokens to preserve the autoregressive property of the decoder.\nIf the maximum sequence length is 5 and the input tokens are “I”, “love”, and “you”, the source sequence becomes: [\u0026ldquo;I\u0026rdquo;, \u0026ldquo;love\u0026rdquo;, \u0026ldquo;you\u0026rdquo;, \u0026ldquo;\u0026lt;PAD\u0026gt;\u0026rdquo;, \u0026ldquo;\u0026lt;PAD\u0026gt;\u0026rdquo;]\nSuppose the corresponding output tokens are “나는”, “너를”, and “사랑해”. During training, the decoder input would be: [\u0026quot;\u0026lt;SOS\u0026gt;\u0026quot;, \u0026ldquo;나는\u0026rdquo;, \u0026ldquo;너를\u0026rdquo;, \u0026ldquo;사랑해\u0026rdquo;, \u0026ldquo;\u0026lt;PAD\u0026gt;\u0026rdquo;]\nTo prevent the model from attending to future tokens, we first apply a causal (look-ahead) mask of the following form: $$ \\begin{bmatrix} 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \\end{bmatrix} $$\nHowever, this is not sufficient, because the last position corresponds to a padding token. Therefore, we additionally mask the padding position, resulting in the final target mask: $$ \\begin{bmatrix} 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \\end{bmatrix} $$\nIn this way, the decoder attends only to previously generated tokens and ignores padding positions.\nAnd here, second line of the forward method :\nx = self.residual_connections[1](x, lambda x: self.cross_attention_block(x, encoder_output, encoder_output, source_mask)) Figure 19. Part of the decoder block (Source: Vaswani et al., 2017) In this cross-attention step, the query comes from the decoder’s current hidden state $x$, while both the key and value come from the encoder output. This design reflects the role of cross-attention: the decoder queries the encoded source representation to retrieve relevant information for generating the next target token. Since the goal is to produce a target-side representation for each decoder position, the output length must match the target sequence length. Therefore, the decoder provides the queries, and the encoder provides the keys and values.\nAlso, the mask must match the key sequence. Since the keys and values come from the encoder output, we apply the source mask to prevent the decoder from attending to padded positions in the source sequence. The target mask is not used here because cross-attention does not involve future target tokens.\nLast, third line of the forward method is feed-forward.\nx = self.residual_connections[2](x, self.feed_forward_block) Figure 20. Part of the decoder block (Source: Vaswani et al., 2017) Now, we repeat the decoder block N times.\nclass Decoder(nn.Module): def __init__(self, d_model: int, layers: nn.ModuleList): super().__init__() self.layers = layers self.norm = LayerNormalization(d_model) def forward(self, x, encoder_output, source_mask, target_mask): for layer in self.layers: x = layer(x, encoder_output, source_mask, target_mask) return self.norm(x) Projection Layer Figure 21. Projection Layer (Source: Vaswani et al., 2017) Projection Layer is a simple linear layer that projects the decoder output to the vocabulary size.\nclass ProjectionLayer(nn.Module): def __init__(self, d_model: int, vocab_size: int): super().__init__() self.proj = nn.Linear(d_model, vocab_size) def forward(self, x): # (batch_size, max_seq_len, d_model) -\u0026gt; (batch_size, max_seq_len, vocab_size) return torch.log_softmax(self.proj(x), dim=-1) Complete Transformer Figure 22. Transformer (Source: Vaswani et al., 2017) class Transformer(nn.Module): def __init__(self, encoder: Encoder, decoder: Decoder, src_embed: InputEmbedding, tgt_embed: InputEmbedding, src_pos: PositionalEncoding, tgt_pos: PositionalEncoding, projection_layer: ProjectionLayer): super().__init__() self.encoder = encoder self.decoder = decoder self.src_embed = src_embed self.tgt_embed = tgt_embed self.src_pos = src_pos self.tgt_pos = tgt_pos self.projection_layer = projection_layer def encode(self, src_sequence, src_mask): x = self.src_embed(src_sequence) x = self.src_pos(x) return self.encoder(x, src_mask) def decode(self, tgt_sequence, encoder_output, src_mask, tgt_mask): x = self.tgt_embed(tgt_sequence) x = self.tgt_pos(x) return self.decoder(x, encoder_output, src_mask, tgt_mask) def project(self, x): return self.projection_layer(x) We could just use forward method, but in this way, we can get more flexibility.\ndef build_transformer(src_vocab_size: int, tgt_vocab_size: int, src_seq_len: int, tgt_seq_len: int, d_model: int = 512, N: int = 6, h: int = 8, dropout: float = 0.1, d_ff: int = 2048): src_embed = InputEmbedding(d_model, src_vocab_size) tgt_embed = InputEmbedding(d_model, tgt_vocab_size) src_pos = PositionalEncoding(d_model, src_seq_len, dropout) tgt_pos = PositionalEncoding(d_model, tgt_seq_len, dropout) # Create the encoder blocks encoder_blocks = [] for _ in range(N): encoder_self_attention_block = MultiHeadAttentionBlock(d_model, h, dropout) encoder_feed_forward_block = FeedForwardBlock(d_model, d_ff, dropout) encoder_blocks.append(EncoderBlock(d_model, encoder_self_attention_block, encoder_feed_forward_block, dropout)) # Create the decoder blocks decoder_blocks = [] for _ in range(N): decoder_self_attention_block = MultiHeadAttentionBlock(d_model, h, dropout) decoder_cross_attention_block = MultiHeadAttentionBlock(d_model, h, dropout) decoder_feed_forward_block = FeedForwardBlock(d_model, d_ff, dropout) decoder_blocks.append(DecoderBlock(d_model, decoder_self_attention_block, decoder_cross_attention_block, decoder_feed_forward_block, dropout)) # Create the encoder and decoder encoder = Encoder(d_model, nn.ModuleList(encoder_blocks)) decoder = Decoder(d_model, nn.ModuleList(decoder_blocks)) # Create the projection layer projection_layer = ProjectionLayer(d_model, tgt_vocab_size) # Create the transformer transformer = Transformer(encoder, decoder, src_embed, tgt_embed, src_pos, tgt_pos, projection_layer) # Initialize the transformer for p in transformer.parameters(): if p.dim() \u0026gt; 1: nn.init.xavier_uniform_(p) return transformer To use the transformer, first we need to initialize the parts of the transformer. That\u0026rsquo;s why we have build_transformer function.\nHere, why we use xavier initialization? Xavier initialization is a popular method for initializing the weights of a neural network. It is named after Xavier Glorot and Yoshua Bengio, who introduced it in their 2010 paper \u0026ldquo;Understanding the difficulty of training deep feedforward neural networks.\u0026rdquo;5\nFirst, Let\u0026rsquo;s see forward propagation. Let\u0026rsquo;s say one layer is y. $$ y = Wx $$\nx $\\in \\mathbb{R}^{n}$ W $\\in \\mathbb{R}^{m \\times n}$ So, $y \\in \\mathbb{R}^{m}$.\nHere, n is called fan_in and m is called fan_out.\nsingle neuron in y : $y_i = \\sum_{j=1}^{n} W_{i,j} x_j$.\nLet\u0026rsquo;s say that Var($x_i$) = $\\sigma^2_x$ and Var($W_{i,j}$) = $\\sigma^2_w$.\nAssume that :\nall $x_i$ are independent, all $W_{i,j}$ are independent, x_j and W_{i,j} are independent, and their means are 0. Then the variance of $y_i$ is : First, recall that for random variables $Z_j$: $$ Var(\\sum_{j=1}^{n} Z_j) = \\sum_{j=1}^{n} Var(Z_j) + 2 \\sum_{j \u0026lt; k} Cov(Z_j, Z_k) $$ If the terms are independent, all covariance terms vanish: $$Cov(Z_j, Z_k) = 0$$ Therefore, $$ Var(y_i) = Var(\\sum_{j=1}^{n} W_{i,j} x_j) = \\sum_{j=1}^{n} Var(W_{i,j} x_j) $$\nNow, consider a single term $W_{i,j} x_j$. For two independent random variables A and B with zero mean: $$ Var(AB)=E(A^2)E(B^2)-E(A)^2E(B)^2$$ Since E(A)=E(B)=0, this simplifies to: $$ Var(AB)=E(A^2)E(B^2)=Var(A)Var(B)$$ Applying this to $W_{i,j} and x_j$, we get: $$ Var(W_{i,j} x_j) = Var(W_{i,j}) Var(x_j) = \\sigma^2_w \\sigma^2_x $$ Therefore, $$ Var(y_i) = \\sum_{j=1}^{n} \\sigma^2_w \\sigma^2_x = n \\sigma^2_w \\sigma^2_x $$\nIf the number of layers is L, then the variance of the last layer is : $$ Var(y_L) \\approx (n \\sigma^2_w)^L $$\nIn this situation, if n \\sigma^2_w is larger than 1, the variance will explode. If it is smaller than 1, the variance will vanish. So, we want to set $n \\sigma^2_w$ to 1. $$ n \\sigma^2_w = 1 $$ $$ \\sigma^2_w = \\frac{1}{n} = \\frac{1}{fan_{in}} $$\nNow, What about backward propagation? We want to compute the gradient with respect to the input: $$\\frac{\\partial L}{\\partial x} \\in \\mathbb{R}^{n}$$ From forward propagation: $$y = Wx$$ By the chain rule, $$\\frac{\\partial L}{\\partial x} =(\\frac{\\partial y}{\\partial x})^{T} \\frac{\\partial L}{\\partial y} = W^{T} \\frac{\\partial L}{\\partial y}$$ Since $\\frac{\\partial y}{\\partial x} = W$, we have: $$\\frac{\\partial L}{\\partial x} = W^{T} \\frac{\\partial L}{\\partial y}$$ Let $\\delta y = \\frac{\\partial L}{\\partial y}$, then: $$\\delta x = W^{T} \\delta y$$ For a single component: $$\\delta x_i = \\sum_{j=1}^{m} W_{i,j} \\delta y_j$$ Assume:\n$Var(\\delta y_j) = \\sigma^2_{\\delta}$ $Var(W_{i,j}) = \\sigma^2_w$ Then, $$ Var(\\delta x_i) = \\sum_{j=1}^{m} Var(W_{i,j} \\delta y_j) = m \\sigma^2_w \\sigma^2_{\\delta} $$\nIf the number of layers is L, then the variance of the last layer is : $$ Var(\\delta x_L) \\approx (m \\sigma^2_w)^L \\sigma^2_{\\delta} $$ In this situation, if $m \\sigma^2_w$ is larger than 1, the variance will explode. If it is smaller than 1, the variance will vanish. So, we want to set $m \\sigma^2_w$ to 1. $$ m \\sigma^2_w = 1 $$ $$ \\sigma^2_w = \\frac{1}{m} = \\frac{1}{fan_{out}} $$\nForward stability requires $\\sigma^2_w = \\frac{1}{fan_{in}}$ and backward stability requires $\\sigma^2_w = \\frac{1}{fan_{out}}$. To balance both, Xavier initialization uses: $$ \\sigma^2_w = \\frac{2}{fan_{in} + fan_{out}} $$\nSo, if we put these sample from normal distribution, then it\u0026rsquo;s like this : $$ W_{i,j} \\sim N(0, \\frac{2}{fan_{in} + fan_{out}}) $$\nHowever, if we want to put samples from uniform distribution, we need little more thinking. Uniform is like this : $$ W_{i,j} \\sim U(-a,a) $$ The variance is : $$ Var(U) = \\frac{a^2}{3} $$ and we want this to be equal to $\\frac{2}{fan_{in} + fan_{out}}$.\n$$ \\frac{a^2}{3} = \\frac{2}{fan_{in} + fan_{out}} $$ $$ a = \\sqrt{\\frac{6}{fan_{in} + fan_{out}}} $$\nXavier initialization fundamentally aims to control the variance of the weights; the choice between a normal and a uniform distribution is secondary. A normal distribution concentrates values around zero but still allows rare large values due to its tails, whereas a uniform distribution samples weights evenly within a fixed range and strictly prevents extreme values. In other words, the uniform version eliminates the possibility of unusually large initial weights. In practice, both approaches work well, but uniform initialization is often slightly preferred for its bounded range and empirical stability, which is why xavier_uniform_ is commonly used in PyTorch.\nUsing the transformer Okay, now we will train the transformer and use it to translate English to Korean. For this, I will make dataset.py , config.py, and train.py files.\nDataset Like every other torch implementation, we need to prepare the dataset module.\nclass BilingualDataset(Dataset): def __init__(self, ds, tokenizer_src, tokenizer_tgt, src_lang, tgt_lang, max_seq_len): super().__init__() # raw dataset self.ds = ds self.max_seq_len = max_seq_len # maximum sequence length self.tokenizer_src = tokenizer_src # tokenizer for the source language self.tokenizer_tgt = tokenizer_tgt # tokenizer for the target language self.src_lang = src_lang # source language, e.g. \u0026#34;english\u0026#34; self.tgt_lang = tgt_lang # target language, e.g. \u0026#34;korean\u0026#34; # special tokens # we pre-define the special tokens, so that we can use them in the getitem method. # If we keep calling torch.tensor([tokenizer.token_to_id(\u0026#34;[SOS]\u0026#34;)]) in the getitem method, it will be slow. self.sos_token = torch.tensor([tokenizer_src.token_to_id(\u0026#34;[SOS]\u0026#34;)], dtype=torch.int64) # start of sequence token. self.eos_token = torch.tensor([tokenizer_src.token_to_id(\u0026#34;[EOS]\u0026#34;)], dtype=torch.int64) # end of sequence token self.pad_token = torch.tensor([tokenizer_src.token_to_id(\u0026#34;[PAD]\u0026#34;)], dtype=torch.int64) # padding token # shape is (1,), dtype is int64 def __len__(self): return len(self.ds) # return the length of the dataset def __getitem__(self, idx): src_target_pair = self.ds[idx] # get the source and target text from the dataset src_text = src_target_pair[self.src_lang] # get the source text from the dataset # e.g. \u0026#34;I love you\u0026#34; tgt_text = src_target_pair[self.tgt_lang] # get the target text from the dataset # e.g. \u0026#34;나는 너를 사랑해\u0026#34; # encode the source and target text enc_input_tokens = self.tokenizer_src.encode(src_text).ids # encode the source text # e.g. [15, 45, 78] dec_input_tokens = self.tokenizer_tgt.encode(tgt_text).ids # encode the target text # e.g. [3, 88, 902] # because we will add [SOS] and [EOS] tokens, we need to subtract 2 # If max_seq_len is 10, len(enc_input_tokens) is 3, so enc_num_padding_tokens is 5. enc_num_padding_tokens = self.max_seq_len - len(enc_input_tokens) - 2 # because we will add [SOS] token, we need to subtract 1 # If max_seq_len is 10, len(dec_input_tokens) is 3, so dec_num_padding_tokens is 6. dec_num_padding_tokens = self.max_seq_len - len(dec_input_tokens) - 1 if enc_num_padding_tokens \u0026lt; 0 or dec_num_padding_tokens \u0026lt; 0: raise ValueError(\u0026#34;Sequence is too long\u0026#34;) # Add SOS and EOS tokens, and pad the rest with [PAD] tokens # [SOS] + encoder_input + [EOS] + [PAD] * enc_num_padding_tokens encoder_input = torch.cat( [ self.sos_token, torch.tensor(enc_input_tokens, dtype=torch.int64), self.eos_token, torch.tensor([self.pad_token] * enc_num_padding_tokens, dtype=torch.int64) ] ) # e.g. [\u0026lt;sos\u0026gt;, 15, 45, 78, \u0026lt;eos\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;] # e.g. [1, 15, 45, 78, 3, 0, 0, 0, 0, 0] # [SOS] + decoder_input + [PAD] * dec_num_padding_tokens # we don\u0026#39;t add [EOS] token to the decoder input # because EOS is the last token of the target sequence. decoder_input = torch.cat( [ self.sos_token, torch.tensor(dec_input_tokens, dtype=torch.int64), torch.tensor([self.pad_token] * dec_num_padding_tokens, dtype=torch.int64) ] ) # e.g. [\u0026lt;sos\u0026gt;, 3, 88, 902, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;] # e.g. [1, 3, 88, 902, 0, 0, 0, 0, 0, 0] # [decoder_input + [EOS] + [PAD] * dec_num_padding_tokens] labels = torch.cat( [ torch.tensor(dec_input_tokens, dtype=torch.int64), self.eos_token, torch.tensor([self.pad_token] * dec_num_padding_tokens, dtype=torch.int64) ] ) # e.g. [3, 88, 902, \u0026lt;eos\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;, \u0026lt;pad\u0026gt;] # e.g. [3, 88, 902, 3, 0, 0, 0, 0, 0, 0] # make sure the sequence lengths are correct assert encoder_input.size(0) == self.max_seq_len assert decoder_input.size(0) == self.max_seq_len assert labels.size(0) == self.max_seq_len return { \u0026#34;encoder_input\u0026#34;: encoder_input, # (max_seq_len, ) \u0026#34;decoder_input\u0026#34;: decoder_input, # (max_seq_len, ) # masks are used for attention score matrix, whose shape is (batch_size, h, max_seq_len, max_seq_len). # So, masks should be (batch_size, 1, 1, max_seq_len) to be broadcasted to (batch_size, h, max_seq_len, max_seq_len). # Batch_size will be added later by the DataLoader. So, here, we need to unsqueeze(0) twice. \u0026#34;encoder_mask\u0026#34;: (encoder_input != self.pad_token).unsqueeze(0).unsqueeze(0).int(), # (1, 1, max_seq_len) \u0026#34;decoder_mask\u0026#34;: (decoder_input != self.pad_token).unsqueeze(0).unsqueeze(0).int() \u0026amp; causal_mask(decoder_input.size(0)), \u0026#34;labels\u0026#34;: labels, # (max_seq_len, ) \u0026#34;src_text\u0026#34;: src_text, # source text \u0026#34;tgt_text\u0026#34;: tgt_text, # target text } # return a dictionary containing the encoder input, decoder input, encoder mask, decoder mask, labels, source text, and target text def causal_mask(size): mask = torch.triu(torch.ones(size, size), diagonal=1).type(torch.int) return mask == 0 For me, the tricky part of the dataset was here :\n\u0026#34;encoder_mask\u0026#34;: (encoder_input != self.pad_token).unsqueeze(0).unsqueeze(0).int(), \u0026#34;decoder_mask\u0026#34;: (decoder_input != self.pad_token).unsqueeze(0).unsqueeze(0).int() \u0026amp; causal_mask(decoder_input.size(0)), First, if encoder input is [1, 15, 45, 78, 3, 0, 0, 0, 0, 0], then (encoder_input != self.pad_token) is : [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]. and after unsqueeze(0) twice, it becomes [[[1, 1, 1, 1, 1, 0, 0, 0, 0, 0]]]. The shape is (1, 1, 10). And then DataLoader will add batch dimension to it, so the shape becomes (4, 1, 1, 10) if batch_size is 4.\nWhat about attention score matrix? its shape is (batch_size, h, max_seq_len, max_seq_len). So, if head is 6, the shape is (4, 6, 10, 10). In attention, the calculation goes like this :\ndef attention(query, key, value, mask=None, dropout=None): d_k = query.shape[-1] # (batch_size, h, max_seq_len, d_k) @ (batch_size, h, d_k, max_seq_len) -\u0026gt; (batch_size, h, max_seq_len, max_seq_len) attention_scores = (query @ key.transpose(-2, -1)) / math.sqrt(d_k) if mask is not None: attention_scores = attention_scores.masked_fill(mask == 0, -1e9) # dim=-1: softmax over the last dimension attention_scores = attention_scores.softmax(dim=-1) # (batch_size, h, max_seq_len, max_seq_len) if dropout is not None: attention_scores = dropout(attention_scores) So, mask\u0026rsquo;s last two dimensions looks like this : $$ \\begin{bmatrix} 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\end{bmatrix} $$ And when we call masked_fill(mask == 0, -1e9), first, the mask will be broadcasted to (4, 6, 10, 10) and the last two dimensions looks like this : $$ \\begin{bmatrix} 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\end{bmatrix} $$ This is exactly what we want.\nWhat about decoder mask? It\u0026rsquo;s similar to encoder mask, but with a causal mask.\n\u0026#34;decoder_mask\u0026#34;: (decoder_input != self.pad_token).unsqueeze(0).unsqueeze(0).int() \u0026amp; causal_mask(decoder_input.size(0)), def causal_mask(size): mask = torch.triu(torch.ones(size, size), diagonal=1).type(torch.int) return mask == 0 The shape of the causal mask is (max_seq_len, max_seq_len). If max_seq_len is 10, it looks like this : $$ \\begin{bmatrix} 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \\end{bmatrix} $$ And by \u0026amp;, causal mask will be broadcasted to (1, 10, 10) and the last two dimensions looks like this : $$ \\begin{bmatrix} 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\end{bmatrix} $$\nConfig def get_config(): return { \u0026#34;batch_size\u0026#34;: 16, \u0026#34;num_epochs\u0026#34;: 20, \u0026#34;lr\u0026#34;: 10**-4, \u0026#34;seq_len\u0026#34;: 100, \u0026#34;d_model\u0026#34;: 512, \u0026#34;lang_src\u0026#34;: \u0026#34;english\u0026#34;, \u0026#34;lang_tgt\u0026#34;: \u0026#34;korean\u0026#34;, \u0026#34;model_folder\u0026#34;: \u0026#34;weights\u0026#34;, \u0026#34;model_basename\u0026#34;: \u0026#34;tmodel_\u0026#34;, #transformer model \u0026#34;preload\u0026#34;: None, # path to a pre-trained model \u0026#34;tokenizer_file\u0026#34;: \u0026#34;tokenizer_{0}.json\u0026#34;, \u0026#34;experiment_name\u0026#34;: \u0026#34;runs/tmodel\u0026#34; } def get_weights_file_path(config, epoch: str): model_folder = config[\u0026#34;model_folder\u0026#34;] model_basename = config[\u0026#34;model_basename\u0026#34;] model_filename = f\u0026#34;{model_basename}{epoch}.pt\u0026#34; return str(Path(\u0026#39;.\u0026#39;) / model_folder / model_filename) The main reason to make config file is to make easier to change the parameters. I tried batch size 8, 16, 32, and 64. When I set it 32, I got out of memory, so set it to 16. The maximum sequence length is 96, so I set it to 100.\nTrain import torch import torch.nn as nn from torch.utils.data import Dataset, DataLoader, random_split from dataset import BilingualDataset, causal_mask from model import build_transformer from config import get_weights_file_path, get_config from datasets import load_dataset from tokenizers import Tokenizer from tokenizers.models import BPE # Byte Pair Encoding from tokenizers.trainers import BpeTrainer from tokenizers import pre_tokenizers from torch.utils.tensorboard import SummaryWriter import warnings from tqdm import tqdm from pathlib import Path def greedy_decode(model, source, source_mask, tokenizer_src, tokenizer_tgt, max_seq_len, device): sos_idx = tokenizer_tgt.token_to_id(\u0026#34;[SOS]\u0026#34;) eos_idx = tokenizer_tgt.token_to_id(\u0026#34;[EOS]\u0026#34;) # Precompute the encoder output and reuse it for every token we get from the decoder encoder_output = model.encode(source, source_mask) # Initialize the decoder input with the sos token decoder_input = torch.empty(1, 1).fill_(sos_idx).type_as(source).to(device) while True: if decoder_input.size(1) \u0026gt;= max_seq_len: break # Build mask for the target (decoder input) decoder_mask = causal_mask(decoder_input.size(1)).type_as(source_mask).to(device) # we don\u0026#39;t have any padding tokens in the decoder input, so we don\u0026#39;t need. # Calculate the output of the decoder out = model.decode(decoder_input.long(), encoder_output, source_mask, decoder_mask) # Get the next token prob = model.project(out[:, -1]) _, next_word = torch.max(prob, dim=-1) # this is the greedy search. decoder_input = torch.cat([decoder_input, torch.empty(1, 1).type_as(source).fill_(next_word.item()).to(device)], dim=1) if next_word == eos_idx: break return decoder_input.squeeze(0) def run_validation(model, validation_ds, tokenizer_src, tokenizer_tgt, max_seq_len, device, print_msg, global_state, writer, num_examples=2): model.eval() count = 0 # Size of the control window (just use a default value) console_width = 80 with torch.no_grad(): for batch in validation_ds: count += 1 encoder_input = batch[\u0026#34;encoder_input\u0026#34;].to(device) encoder_mask = batch[\u0026#34;encoder_mask\u0026#34;].to(device) assert encoder_input.size(0) == 1, \u0026#34;Batch size must be 1 for validation\u0026#34; model_out = greedy_decode(model, encoder_input, encoder_mask, tokenizer_src, tokenizer_tgt, max_seq_len, device) source_text = batch[\u0026#34;src_text\u0026#34;][0] target_text = batch[\u0026#34;tgt_text\u0026#34;][0] model_out_text = tokenizer_tgt.decode(model_out.detach().cpu().numpy()) # Print to the console print_msg(\u0026#39;.\u0026#39; * console_width) print_msg(f\u0026#34;Source: {source_text}\u0026#34;) print_msg(f\u0026#34;target: {target_text}\u0026#34;) print_msg(f\u0026#34;predicted: {model_out_text}\u0026#34;) if count == num_examples: break def get_all_sentences(ds, lang): for item in ds: yield item[lang] def get_or_build_tokenizer(config, ds, lang): # e.g. config[\u0026#34;tokenizer_file\u0026#34;] = \u0026#39;../tokenizers/tokenizer_{0}.json\u0026#39; tokenizer_path = Path(config[\u0026#34;tokenizer_file\u0026#34;].format(lang)) if not Path.exists(tokenizer_path): # If there is no word in tokenizer, then use \u0026#34;[UNK]\u0026#34; as the unkown token tokenizer = Tokenizer(BPE(unk_token=\u0026#34;[UNK]\u0026#34;)) # Split by whitespace tokenizer.pre_tokenizer = pre_tokenizers.Sequence( [pre_tokenizers.Metaspace(), pre_tokenizers.Punctuation()]) # min_frequency: the minimum frequency of a word to be included in the tokenizer trainer = BpeTrainer(special_tokens=[\u0026#34;[UNK]\u0026#34;, \u0026#34;[PAD]\u0026#34;, \u0026#34;[SOS]\u0026#34;, \u0026#34;[EOS]\u0026#34;], min_frequency=2) tokenizer.train_from_iterator(get_all_sentences(ds, lang), trainer=trainer) tokenizer.save(str(tokenizer_path)) else: tokenizer = Tokenizer.from_file(str(tokenizer_path)) return tokenizer def get_ds(config): ds_raw = load_dataset(\u0026#34;bongsoo/news_talk_en_ko\u0026#34;, split=\u0026#34;train[:10%]\u0026#34;) ds_raw = ds_raw.rename_column(\u0026#34;Skinner\u0026#39;s reward is mostly eye-watering.\u0026#34;, \u0026#34;english\u0026#34;) ds_raw = ds_raw.rename_column(\u0026#34;스키너가 말한 보상은 대부분 눈으로 볼 수 있는 현물이다.\u0026#34;, \u0026#34;korean\u0026#34;) # Build tokenizers tokenizer_src = get_or_build_tokenizer(config, ds_raw, config[\u0026#34;lang_src\u0026#34;]) tokenizer_tgt = get_or_build_tokenizer(config, ds_raw, config[\u0026#34;lang_tgt\u0026#34;]) # Keep 90% for training, 10% for validation train_ds_size = int(0.9 * len(ds_raw)) val_ds_size = len(ds_raw) - train_ds_size train_ds_raw, val_ds_raw = random_split(ds_raw, [train_ds_size, val_ds_size]) train_ds = BilingualDataset(train_ds_raw, tokenizer_src, tokenizer_tgt, config[\u0026#34;lang_src\u0026#34;], config[\u0026#34;lang_tgt\u0026#34;], config[\u0026#34;seq_len\u0026#34;]) val_ds = BilingualDataset(val_ds_raw, tokenizer_src, tokenizer_tgt, config[\u0026#34;lang_src\u0026#34;], config[\u0026#34;lang_tgt\u0026#34;], config[\u0026#34;seq_len\u0026#34;]) max_len_src = 0 max_len_tgt = 0 for item in ds_raw: src_ids = tokenizer_src.encode(item[config[\u0026#34;lang_src\u0026#34;]]).ids tgt_ids = tokenizer_tgt.encode(item[config[\u0026#34;lang_tgt\u0026#34;]]).ids max_len_src = max(max_len_src, len(src_ids)) max_len_tgt = max(max_len_tgt, len(tgt_ids)) print(f\u0026#34;Maximum length of source sentences: {max_len_src}\u0026#34;) print(f\u0026#34;Maximum length of target sentences: {max_len_tgt}\u0026#34;) train_dataloader = DataLoader(train_ds, batch_size=config[\u0026#34;batch_size\u0026#34;], shuffle=True) val_dataloader = DataLoader(val_ds, batch_size=1, shuffle=True) return train_dataloader, val_dataloader, tokenizer_src, tokenizer_tgt def get_model(config, vocab_src_len, vocab_tgt_len): model = build_transformer(vocab_src_len, vocab_tgt_len, config[\u0026#34;seq_len\u0026#34;], config[\u0026#34;seq_len\u0026#34;], d_model=config[\u0026#34;d_model\u0026#34;]) return model def train_model(config): # Define the device device = torch.device(\u0026#34;cuda\u0026#34; if torch.cuda.is_available() else \u0026#34;cpu\u0026#34;) print(f\u0026#34;Using device: {device}\u0026#34;) Path(config[\u0026#34;model_folder\u0026#34;]).mkdir(parents=True, exist_ok=True) train_dataloader, val_dataloader, tokenizer_src, tokenizer_tgt = get_ds(config) model = get_model(config, tokenizer_src.get_vocab_size(), tokenizer_tgt.get_vocab_size()).to(device) # Tensorboard writer = SummaryWriter(config[\u0026#34;experiment_name\u0026#34;]) optimizer = torch.optim.Adam(model.parameters(), lr=config[\u0026#34;lr\u0026#34;], eps=1e-9) initial_epoch = 0 global_step = 0 if config[\u0026#34;preload\u0026#34;]: model_filename = get_weights_file_path(config, config[\u0026#34;preload\u0026#34;]) print(f\u0026#34;Preloading model {model_filename}\u0026#34;) state = torch.load(model_filename) initial_epoch = state[\u0026#34;epoch\u0026#34;] + 1 optimizer.load_state_dict(state[\u0026#34;optimizer_state_dict\u0026#34;]) global_step = state[\u0026#34;global_step\u0026#34;] loss_fn = nn.CrossEntropyLoss(ignore_index=tokenizer_src.token_to_id(\u0026#34;[PAD]\u0026#34;), label_smoothing=0.1).to(device) for epoch in range(initial_epoch, config[\u0026#34;num_epochs\u0026#34;]): model.train() batch_iterator = tqdm(train_dataloader, desc=f\u0026#34;Processing Epoch {epoch:02d}\u0026#34;) for batch in batch_iterator: encoder_input = batch[\u0026#34;encoder_input\u0026#34;].to(device) # (batch_size, seq_len) decoder_input = batch[\u0026#34;decoder_input\u0026#34;].to(device) # (batch_size, seq_len) encoder_mask = batch[\u0026#34;encoder_mask\u0026#34;].to(device) # (B, 1, 1, seq_len) decoder_mask = batch[\u0026#34;decoder_mask\u0026#34;].to(device) # (B, 1, seq_len, seq_len) # Run the tensors through the transformer encoder_output = model.encode(encoder_input, encoder_mask) # (batch_size, seq_len, d_model) decoder_output = model.decode(decoder_input, encoder_output, encoder_mask, decoder_mask) # (batch_size, seq_len, d_model) proj_output = model.project(decoder_output) # (batch_size, seq_len, vocab_tgt_size) label = batch[\u0026#34;labels\u0026#34;].to(device) # (batch_size, seq_len) # (batch_size, seq_len, tgt_vocab_size) -\u0026gt; (batch_size * seq_len, tgt_vocab_size) loss = loss_fn(proj_output.view(-1, tokenizer_tgt.get_vocab_size()), label.view(-1)) batch_iterator.set_postfix({f\u0026#34;loss\u0026#34;: f\u0026#34;{loss.item():6.3f}\u0026#34;}) # Log the loss writer.add_scalar(\u0026#34;train_loss\u0026#34;, loss.item(), global_step) writer.flush() # Backpropagete the loss loss.backward() # Update the weights optimizer.step() optimizer.zero_grad() global_step += 1 run_validation(model, val_dataloader, tokenizer_src, tokenizer_tgt, config[\u0026#34;seq_len\u0026#34;], device, lambda msg: batch_iterator.write(msg), global_step, writer, num_examples=2) # Save the model at the end of the epoch model_filename = get_weights_file_path(config, f\u0026#34;{epoch:02d}\u0026#34;) torch.save({ \u0026#34;epoch\u0026#34;: epoch, \u0026#34;model_state_dict\u0026#34;: model.state_dict(), \u0026#34;optimizer_state_dict\u0026#34;: optimizer.state_dict(), \u0026#34;global_step\u0026#34;: global_step, }, model_filename) if __name__ == \u0026#34;__main__\u0026#34;: warnings.filterwarnings(\u0026#34;ignore\u0026#34;) config = get_config() train_model(config) Let\u0026rsquo;s see function by function, but I\u0026rsquo;ll skip the greedy_decode and run_validation, which are used for validation and code is quite simple.\ndef get_all_sentences(ds, lang): for item in ds: yield item[lang] This function follows the typical way to make a dataset iterator. we made this function because tokenizer requires a iterator to build the vocabulary.\ndef get_or_build_tokenizer(config, ds, lang): # e.g. config[\u0026#34;tokenizer_file\u0026#34;] = \u0026#39;../tokenizers/tokenizer_{0}.json\u0026#39; tokenizer_path = Path(config[\u0026#34;tokenizer_file\u0026#34;].format(lang)) if not Path.exists(tokenizer_path): # If there is no word in tokenizer, then use \u0026#34;[UNK]\u0026#34; as the unkown token tokenizer = Tokenizer(BPE(unk_token=\u0026#34;[UNK]\u0026#34;)) # Split by whitespace tokenizer.pre_tokenizer = pre_tokenizers.Sequence( [pre_tokenizers.Metaspace(), pre_tokenizers.Punctuation()]) trainer = BpeTrainer(special_tokens=[\u0026#34;[UNK]\u0026#34;, \u0026#34;[PAD]\u0026#34;, \u0026#34;[SOS]\u0026#34;, \u0026#34;[EOS]\u0026#34;], min_frequency=2) tokenizer.train_from_iterator(get_all_sentences(ds, lang), trainer=trainer) tokenizer.save(str(tokenizer_path)) else: tokenizer = Tokenizer.from_file(str(tokenizer_path)) return tokenizer This function is about tokenizer. If there is no tokenizer file, it will build a new one. If there is, it will load the existing one. In this case, we use BPE tokenizer, and unknown token will be \u0026ldquo;[UNK]\u0026rdquo;. By the way, BPE is an algorithm that repeatedly merges the most frequent pairs of characters to build a subword vocabulary6.\nAnd here\u0026rsquo;s pre-tokenization part. Pre-tokenization is applied before the main tokenization process to split the raw text into initial units. I use Metaspace to preserve whitespace information by replacing spaces with a special visible character, which helps the model treat spaces consistently. So, if whitespace is important in the languages you are dealing with, using Metaspace is a good choice.\nPunctuation is used to split punctuation marks(, !, ? etc.) into separate tokens so they can be learned independently. The parameter min_frequency=2 means that a token pair must appear at least twice in the training data to be considered for merging into the vocabulary. Finally, the trained tokenizer is saved to a file so it can be reused later without retraining.\ndef get_ds(config): ds_raw = load_dataset(\u0026#34;bongsoo/news_talk_en_ko\u0026#34;, split=\u0026#34;train[:30%]\u0026#34;) ds_raw = ds_raw.rename_column(\u0026#34;Skinner\u0026#39;s reward is mostly eye-watering.\u0026#34;, \u0026#34;english\u0026#34;) ds_raw = ds_raw.rename_column(\u0026#34;스키너가 말한 보상은 대부분 눈으로 볼 수 있는 현물이다.\u0026#34;, \u0026#34;korean\u0026#34;) # Build tokenizers tokenizer_src = get_or_build_tokenizer(config, ds_raw, config[\u0026#34;lang_src\u0026#34;]) tokenizer_tgt = get_or_build_tokenizer(config, ds_raw, config[\u0026#34;lang_tgt\u0026#34;]) # Keep 90% for training, 10% for validation train_ds_size = int(0.9 * len(ds_raw)) val_ds_size = len(ds_raw) - train_ds_size train_ds_raw, val_ds_raw = random_split(ds_raw, [train_ds_size, val_ds_size]) train_ds = BilingualDataset(train_ds_raw, tokenizer_src, tokenizer_tgt, config[\u0026#34;lang_src\u0026#34;], config[\u0026#34;lang_tgt\u0026#34;], config[\u0026#34;seq_len\u0026#34;]) val_ds = BilingualDataset(val_ds_raw, tokenizer_src, tokenizer_tgt, config[\u0026#34;lang_src\u0026#34;], config[\u0026#34;lang_tgt\u0026#34;], config[\u0026#34;seq_len\u0026#34;]) max_len_src = 0 max_len_tgt = 0 for item in ds_raw: src_ids = tokenizer_src.encode(item[config[\u0026#34;lang_src\u0026#34;]]).ids tgt_ids = tokenizer_tgt.encode(item[config[\u0026#34;lang_tgt\u0026#34;]]).ids max_len_src = max(max_len_src, len(src_ids)) max_len_tgt = max(max_len_tgt, len(tgt_ids)) print(f\u0026#34;Maximum length of source sentences: {max_len_src}\u0026#34;) print(f\u0026#34;Maximum length of target sentences: {max_len_tgt}\u0026#34;) train_dataloader = DataLoader(train_ds, batch_size=config[\u0026#34;batch_size\u0026#34;], shuffle=True) val_dataloader = DataLoader(val_ds, batch_size=1, shuffle=True) return train_dataloader, val_dataloader, tokenizer_src, tokenizer_tgt I used this dataset for training. I set 30% here, so approximately, I used 400K rows for training. The column names were one of the sample sentences from the dataset, so I renamed them to \u0026ldquo;english\u0026rdquo; and \u0026ldquo;korean\u0026rdquo;.\nAnd I took 10% of the dataset for validation. The for loop is to find the maximum length of the source and target sentences. This part could take some time, and based on the result of this (It was 171), I set the maximum sequence length to 180.\ndef get_model(config, vocab_src_len, vocab_tgt_len): model = build_transformer(vocab_src_len, vocab_tgt_len, config[\u0026#34;seq_len\u0026#34;], config[\u0026#34;seq_len\u0026#34;], d_model=config[\u0026#34;d_model\u0026#34;]) return model This function is for convenience later. Later, to get a model, we only need to put config, vocab_src_len, and vocab_tgt_len.\ndef train_model(config): # Define the device device = torch.device(\u0026#34;cuda\u0026#34; if torch.cuda.is_available() else \u0026#34;cpu\u0026#34;) print(f\u0026#34;Using device: {device}\u0026#34;) Path(config[\u0026#34;model_folder\u0026#34;]).mkdir(parents=True, exist_ok=True) train_dataloader, val_dataloader, tokenizer_src, tokenizer_tgt = get_ds(config) model = get_model(config, tokenizer_src.get_vocab_size(), tokenizer_tgt.get_vocab_size()).to(device) # Tensorboard writer = SummaryWriter(config[\u0026#34;experiment_name\u0026#34;]) optimizer = torch.optim.Adam(model.parameters(), lr=config[\u0026#34;lr\u0026#34;], eps=1e-9) initial_epoch = 0 global_step = 0 if config[\u0026#34;preload\u0026#34;]: model_filename = get_weights_file_path(config, config[\u0026#34;preload\u0026#34;]) print(f\u0026#34;Preloading model {model_filename}\u0026#34;) state = torch.load(model_filename) initial_epoch = state[\u0026#34;epoch\u0026#34;] + 1 optimizer.load_state_dict(state[\u0026#34;optimizer_state_dict\u0026#34;]) global_step = state[\u0026#34;global_step\u0026#34;] loss_fn = nn.CrossEntropyLoss(ignore_index=tokenizer_src.token_to_id(\u0026#34;[PAD]\u0026#34;), label_smoothing=0.1).to(device) for epoch in range(initial_epoch, config[\u0026#34;num_epochs\u0026#34;]): model.train() batch_iterator = tqdm(train_dataloader, desc=f\u0026#34;Processing Epoch {epoch:02d}\u0026#34;) for batch in batch_iterator: encoder_input = batch[\u0026#34;encoder_input\u0026#34;].to(device) # (batch_size, seq_len) decoder_input = batch[\u0026#34;decoder_input\u0026#34;].to(device) # (batch_size, seq_len) encoder_mask = batch[\u0026#34;encoder_mask\u0026#34;].to(device) # (B, 1, 1, seq_len) decoder_mask = batch[\u0026#34;decoder_mask\u0026#34;].to(device) # (B, 1, seq_len, seq_len) # Run the tensors through the transformer encoder_output = model.encode(encoder_input, encoder_mask) # (batch_size, seq_len, d_model) decoder_output = model.decode(decoder_input, encoder_output, encoder_mask, decoder_mask) # (batch_size, seq_len, d_model) proj_output = model.project(decoder_output) # (batch_size, seq_len, vocab_tgt_size) label = batch[\u0026#34;labels\u0026#34;].to(device) # (batch_size, seq_len) # (batch_size, seq_len, tgt_vocab_size) -\u0026gt; (batch_size * seq_len, tgt_vocab_size) loss = loss_fn(proj_output.view(-1, tokenizer_tgt.get_vocab_size()), label.view(-1)) batch_iterator.set_postfix({f\u0026#34;loss\u0026#34;: f\u0026#34;{loss.item():6.3f}\u0026#34;}) # Log the loss writer.add_scalar(\u0026#34;train_loss\u0026#34;, loss.item(), global_step) writer.flush() # Backpropagete the loss loss.backward() # Update the weights optimizer.step() optimizer.zero_grad() global_step += 1 run_validation(model, val_dataloader, tokenizer_src, tokenizer_tgt, config[\u0026#34;seq_len\u0026#34;], device, lambda msg: batch_iterator.write(msg), global_step, writer, num_examples=2) # Save the model at the end of the epoch model_filename = get_weights_file_path(config, f\u0026#34;{epoch:02d}\u0026#34;) torch.save({ \u0026#34;epoch\u0026#34;: epoch, \u0026#34;model_state_dict\u0026#34;: model.state_dict(), \u0026#34;optimizer_state_dict\u0026#34;: optimizer.state_dict(), \u0026#34;global_step\u0026#34;: global_step, }, model_filename) I\u0026rsquo;d like to talk about the label smoothing used in here. Label smoothing is a technique to prevent the model from overfitting by reducing the confidence of the model7. It works by replacing the one-hot encoded labels with a smoothed version of the labels.\nIf vocab size is 5 and the label index is 2, then CrossEntropyLoss set the answer to [0,0,1,0,0]. This is called one-hot label. But, if we use label smoothing 0.1, then the answer index 2 will be 0.9 and the other indexes will be 0.1/(vocab_size-1) = 0.1/4 = 0.025. [0.025, 0.025, 0.9, 0.025, 0.025].\nWhy do this? If we use one-hot label, the model will think \u0026ldquo;The answer index has to be probability 1!\u0026rdquo; If so, it could lead to overfitting, generalization performance could be degraded. On the other hand, label smoothing says \u0026ldquo;It is an answer, but don\u0026rsquo;t be too sure about it!\u0026rdquo;. This is helpful for generalization.\nResult Okay, Let\u0026rsquo;s see the result. I trained the model for 20 epochs. Below are three validation examples (source, target, and model prediction).\nExample 1\nSource Block deals are usually bad news for stock prices because large-scale supplies are released to the market. Target 통상 블록딜은 시장에 대규모의 물량이 풀리는 것이어서 주가에는 악재다. Predicted ▁블록 딜 은 ▁일반적으로 ▁대규모 ▁공급이 ▁시중에 ▁풀 어서 ▁주가에 ▁대한 ▁악재가 ▁더해 지고 ▁있다 . Example 2\nSource U.S. computer company Dell collaborated with actor Nikki Reed, to release the fashion brand By You with Love that recycles scrap metal from the fashion. Target 미국의 컴퓨터회사 델이 영화배우 니키 리드와 협업해 올 초 출시한 패션 브랜드 바이유위드러브는 폐금속을 재활용한다. Predicted ▁미국 ▁컴퓨터 ▁기업 ▁델 이 ▁배우 ▁니키 ▁러 프와 ▁협업 하며 ▁패션 ▁브랜드 ▁쪽 과 ▁금속 을 ▁제거 한 ▁사랑 으로 ▁보는 ▁패션 ▁브랜드를 ▁출시한다 . Example 3\nSource In addition, the opponent team won the game by quickly noticing from \u0026lsquo;To My Boyfriend\u0026rsquo; by Fin.K.L to \u0026lsquo;Cheap Coffee\u0026rsquo; by Jang Gi-ha and the Faces. Target 그뿐 아니라 상대팀이 내는 핑클의 \u0026lsquo;내 남자 친구에게\u0026rsquo;부터 장기하와 얼굴들 \u0026lsquo;싸구려 커피\u0026rsquo;까지 빠르게 눈치채며 팀을 우승으로 이끌었다. Predicted ▁나아가 ▁상대 ▁팀이 ▁ \u0026rsquo; 나의 ▁친구 \u0026rsquo; 부터 ▁ \u0026rsquo; 내 녀 에게 ▁꼭 ▁먹어 요 \u0026rsquo; , ▁장기 하와 ▁함께 ▁하는 ▁ \u0026rsquo; 힘 들 티 \u0026rsquo; 까지 ▁빠르게 ▁풀 며 ▁경기를 ▁이겼다 . The ▁ in the predictions is the special character used by the BPE tokenizer to mark a space or word boundary. When the tokenizer decodes the model output back to text, it leaves this symbol in place instead of converting it to an actual space, so the raw decoded string looks like that. In a production pipeline you would typically post-process the decoded text to replace ▁ with a normal space.\nEven though the translations are far from perfect, the results are still quite impressive. The model was trained on only about 130K sentence pairs for 20 epochs, which took roughly 9 hours on a single GPU.\nFigure 23. Tensorboard Vaswani, A., et al. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30). https://arxiv.org/abs/1706.03762\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nhkproj/pytorch-transformer. (n.d.). Attention is all you need implementation. GitHub. https://github.com/hkproj/pytorch-transformer\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nKazemnejad, A. (n.d.). Transformer architecture: The positional encoding. https://kazemnejad.com/blog/transformer_architecture_positional_encoding/\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBa, J. L., Kiros, J. R., \u0026amp; Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nGlorot, X., \u0026amp; Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (pp. 249–256).\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSennrich, R., Haddow, B., \u0026amp; Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the ACL (pp. 1715–1725).\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSzegedy, C., et al. (2016). Rethinking the Inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2818–2826).\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://yeongseung.github.io/posts/transformer/post/","summary":"\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eThe transformer architecture has become the foundation of modern deep learning, powering everything from language models to vision models. Since I\u0026rsquo;ll be referencing transformers frequently in future posts, I wanted to create my own comprehensive analysis of the architecture. In this post, I\u0026rsquo;ll walk through building a transformer-based translation model step by step, following the original paper by Vaswani et al. (2017)\u003csup id=\"fnref:1\"\u003e\u003ca href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\"\u003e1\u003c/a\u003e\u003c/sup\u003e. We\u0026rsquo;ll dive into the code, understanding each component as we implement it.\u003c/p\u003e","title":"Understanding and Building the Transformer"},{"content":"Introduction If you are interested in reinforcement learning, you have probably heard of CartPole [6], often considered the \u0026ldquo;Hello World\u0026rdquo; environment of the field. This is partly because the task is simple to understand, and partly because each episode ends quickly, allowing you to observe learning progress within a short time.\nFigure 1. CartPole environment visualization Looking back, I have run many experiments in CartPole before, but not in a structured or systematic way, which caused me to forget most of the details over time. In this post, I aim to consolidate those trials into a coherent narrative, focusing in particular on why Q-learning can become unstable and how techniques such as experience replay, Double DQN, and failure-focused sampling can improve robustness.\nFull implementation code on GitHub: stabilizing-q-learning-cartpole\nExperiments 1. From TD Learning to Q-Learning Intuitively, Q-learning can be viewed as a member of the Temporal-Difference (TD) family of methods [1]. TD learning focuses on updating the value function directly, rather than optimizing the policy explicitly. We can consider $V(S_t)$ as the prediction $\\hat{y}$ and $R_{t+1}+\\gamma V(S_{t+1})$ as the target $y$.\n$$ \\begin{aligned} V(S_t) \\leftarrow V(S_t) + \\alpha (R_{t+1} + \\gamma V(S_{t+1}) - V(S_t)) \\tag{1} \\end{aligned} $$ This analogy to supervised learning makes the TD update in Equation (1) straightforward to implement, especially when using a neural network to approximate the value function.\n#torch env = gym.make(\u0026#34;CartPole-v1\u0026#34;, render_mode=\u0026#34;none\u0026#34;) optimizer = torch.optim.Adam(model.parameters(), lr=0.001) criterion = nn.MSELoss() current_value = model(state) # V(S_t) # (select a action to take somehow) next_state, reward, ... = env.step(action) next_value = model(next_state) target_value = reward + (gamma* next_value) criterion = nn.MSELoss() loss = criterion(current_value, target_value) However, the (select a action to take somehow) part is a major challenge. $V(s)$ itself doesn\u0026rsquo;t contain information about what action should be taken. If the environment were simple and deterministic, such as Chess or Go, then we could simulate all actions from $S_t$ and calculate all $V(S_{t+1})$ without actually taking those actions. With all possible $V(S_{t+1})$ values in hand, we could finally select an action by choosing the one leading to the state with the best value.\n2. SARSA That\u0026rsquo;s where the action-value function Q is effective. $V(S_t) = \\max_{a}Q(S_t,a)$. Below is the SARSA algorithm [1], which leverages the action-value function Q. As you will see, we can now choose the next action without actually taking it or simulating it.\nCode: 1.SARSA_cartpole.py\nClick to see the SARSA torch code model = qnetwork(4,2) # state consists of four scalars. env = gym.make(\u0026#34;CartPole-v1\u0026#34;, render_mode=\u0026#34;none\u0026#34;) optimizer = torch.optim.Adam(model.parameters(), lr=0.001) criterion = nn.MSELoss() for episode in range(3000): # 3000 episodes obs, info = env.reset() # Example output: [ 0.01234567 -0.00987654 0.02345678 0.01456789] # [cart_position, cart_velocity, pole_angle, pole_angular_velocity] game_over = False total_reward = 0 action_space = [0,1] # 0: left, 1: right # First action of a episode should be selected here. state_tensor = torch.tensor(obs, dtype=torch.float32) q_values = model(state_tensor) if np.random.random() \u0026lt; 0.1: # epsilon = 0.1 act = np.random.choice(action_space) else: act = torch.argmax(q_values).item() while not game_over: state_tensor = torch.tensor(obs, dtype=torch.float32) q_values = model(state_tensor) # SARSA: act is already selected. (This is the key point of SARSA.) # current_q is where the backpropagation starts. current_q = q_values[act] # step the environment next_obs, reward, terminated, truncated, info = env.step(act) total_reward += reward # SARSA: calculate the target using next_act with torch.no_grad(): # to block the gradient next_state_tensor = torch.tensor(next_obs, dtype=torch.float32) if np.random.random() \u0026lt; 0.1: next_act = np.random.choice(action_space) else: next_act = torch.argmax(model(next_state_tensor)).item() next_q = model(next_state_tensor)[next_act] target_q = reward + (0.99 * next_q * (1 - terminated)) # no next reward when terminated # But, truncated is okay. because we can still get the next reward. # calculate the loss and update the model loss = criterion(current_q, target_q) #target_q here is like the label for the current_q. optimizer.zero_grad() loss.backward() optimizer.step() # SARSA: update act not only obs. obs = next_obs act = next_act # This is the key of SARSA. In my opinion, the main point of SARSA is to select the next action and target q-value using epsilon-greedy. (see the with torch.no_grad(): part) Let\u0026rsquo;s check the SARSA\u0026rsquo;s performance.\nFigure 2. Q-values tracked on fixed states across episodes Figure 3. Episode total reward (left) and episode TD error (right) SARSA reached the maximum reward of 500, and the moving average showed an increasing trend up to a point. However, it fails to stay at the peak. What could be the reason for these periodic drops in total reward?\nFIXED EPSILON? I fixed epsilon at 0.1, meaning that approximately every 10 steps, an action is selected randomly, and CartPole is a very sensitive environment. Even so, I believe that if the model were trained well, it should have been robust enough to handle 10% randomness.\nNN MODEL? class qnetwork(torch.nn.Module): def __init__(self, state_size:4, action_size:2): super().__init__() self.fc1 = nn.Linear(state_size, 128) self.fc2 = nn.Linear(128, action_size) self.relu = nn.ReLU() def forward(self, x): x = self.relu(self.fc1(x)) x = self.fc2(x) return x Was my model not large enough or not structured well to memorize all the past information? That could be the case. Even though the training data is correlated, if the model were large enough and had many state-of-the-art features, it could memorize every past experience.\nFORGETTING? If the problem is mainly caused by forgetting previous experiences, there could be several ways to mitigate this. Mnih et al. [2] demonstrated that leveraging Experience Replay is effective for reducing data correlation and smoothing the training distribution. Intuitively, it\u0026rsquo;s easy to see why Experience Replay can reduce data correlation—it allows us to randomly sample from several saved episodes rather than processing them consecutively. Then what does it mean that Experience Replay \u0026ldquo;smooths the training distribution?\u0026rdquo;\nFigure 4. Example training distributions: (1) overfitted to one side, (2) narrow distribution around equilibrium, (3) overfitting to one side In CartPole there are four scalars in the state, but let\u0026rsquo;s think about only one scalar, Pole Angle. If training data is dominated by specific scenarios, such as left-leaning angles (Curve 1), the agent overfits and fails to handle opposite situations like right tilts. As the agent masters the task, the data distribution squeezes into a narrow curve around zero (Curve 2), causing the agent to forget recovery strategies for large deviations. (This is similar to a problem in imitation learning, which learns from expert behaviors.)\nTo build a robust model, the agent must learn from a comprehensive distribution that covers the entire state space. Experience Replay addresses this by randomly sampling from a diverse history of episodes. If the experience buffer successfully saves diverse scenarios, it can reconstruct a broad distribution that encompasses all scenarios.\n3. Q-learning Okay, I decided to use Experience Replay, but SARSA is incompatible with it because SARSA is an on-policy algorithm. So, it\u0026rsquo;s time to move on to Q-learning. Q-learning is an off-policy algorithm, which means we can leverage Experience Replay, and it\u0026rsquo;s similar to SARSA.\nHowever, first, let\u0026rsquo;s look at Q-learning without Experience Replay. The main difference between Q-learning and SARSA is that when calculating the target Q, we don\u0026rsquo;t consider random actions (like in epsilon-greedy), even if the actual next action might be random—we just do exploitation. Intuitively, it makes sense to define the value of the current state based on the best possible result achievable by ideal actions from that state.\nCode: 2.Qlearning_cartpole.py\n# calculate the target with torch.no_grad(): next_state_tensor = torch.tensor(next_obs, dtype=torch.float32) next_q_max = torch.max(model(next_state_tensor)) target_q = reward + (0.99 * next_q_max * (1 - terminated)) loss = criterion(current_q, target_q) optimizer.zero_grad() loss.backward() optimizer.step() episode_losses.append(loss.item()) obs = next_obs Let\u0026rsquo;s check the results.\nFigure 5. Q-values on fixed states using Q-learning without experience replay Figure 6. Episode total reward and TD error using Q-learning without experience replay The results seem not that different from those of SARSA.\n4. Q-learning with Experience Replay Okay, I set the capacity of the replay buffer to 10,000, the batch size to 32, and the update (training) frequency to every four steps. Let\u0026rsquo;s see the results.\nCode: 3.Qlearning_cartpole_replay.py\nFigure 7. Q-values on fixed states with experience replay Figure 8. Episode total reward and TD error with experience replay Surprisingly, many things changed. First, the episode-total reward plot became more stable (compared to before), even though periodic drops still occurred. This makes sense because gathering experiences from various episodes and using a batch of 32 leads to longer periods of stability (staying at the peak). However, this prolonged stability causes the replay buffer to be filled with only stable data, which narrows the buffer\u0026rsquo;s distribution. This is likely why we still see periodic drops.\nSecond, Q-values also stabilized relatively well, but there was a sudden spike. Let\u0026rsquo;s focus on episodes between 1200 and 1400. Total rewards dropped during that period, while the Q-values of the five fixed states skyrocketed. Why? Most likely, staying at the peak caused the buffer\u0026rsquo;s distribution to become too narrow, which in turn narrowed the Q-network\u0026rsquo;s coverage. At that point, when the Q-network encountered an unfamiliar state, such as leaning right, edging left, or leaning left, it incorrectly output a very high Q-value. Consequently, the Q-network learned that leaning right or left is a desirable state. Ultimately, this caused the total reward drop. (That\u0026rsquo;s my theory.)\nSo, the problem is still forgetting, which, I believe, cannot be solved by target networks, Double DQN, or Dueling DQN alone. Intuitively, even when the episodes become full of successful experiences, the buffer still has to keep unsuccessful experiences and how to recover from them.\n5. Dual Buffer The dual buffer idea is to separate successful experiences from unsuccessful experiences, and sample from them by a fixed ratio like 7:3. I think this could be a direct remedy for periodic drops. I set the capacities of the successful buffer and unsuccessful buffer to 5000 each, with the same batch size of 32. Let\u0026rsquo;s see the results.\nCode: 4.Qlearning_cartpole_replay_dual.py\nFigure 9. Q-values on fixed states using dual buffer (successful + unsuccessful) Figure 10. Episode total reward and TD error using dual buffer Logically, I believe adding a dual buffer was the right approach, but it hasn\u0026rsquo;t solved the periodic drop problem so far. In the meantime, the Q-values became too high and the TD error skyrocketed. I think we should solve this problem first.\n6. Double DQN The idea of separating the main Q-network and the target Q-network is a well-known solution for Q-value overestimation [3, 4]. The reason I didn\u0026rsquo;t use this idea earlier was that I thought the dual buffer could solve the periodic drop problem even without separating the Q-networks. Anyway, let\u0026rsquo;s see the results.\nCode: 5.Qlearning_cartpole_replay_dual_double.py\nFigure 11. Q-values on fixed states using Double DQN with dual buffer Figure 12. Episode total reward and TD error using Double DQN with dual buffer The results improved incredibly. Q-values stabilized, which caused the TD error to stabilize as well. Periodic drops are still there, but they have been somewhat mitigated.\nSo, why did it improve? By separating the main Q-network and the target Q-network, and updating the target Q-network slowly toward the main Q-network, we can mitigate the sudden overestimation of the main Q-network.\nFor example, even when the buffer is filled with 500-reward experiences, if the main Q-network encounters a forgotten situation, like leaning right, it might overestimate the Q-value. However, because the target Q-network is updated slowly, this overestimation can be mitigated. Specifically, with a soft update factor $\\tau=0.01$, the target network only incorporates 1% of the main network\u0026rsquo;s weights at each step, ensuring the target values remain stable.\nClick to see Double DQN without Dual buffer case Code: 5_1.Qlearning_cartpole_replay_double.py\nFigure 13. Q-values using Double DQN without dual buffer Figure 14. Episode total reward and TD error using Double DQN without dual buffer 7. Refining Buffer Strategy Code: 7.Qlearning_cartpole_replay_normal_failure.py\nBack to the buffer Okay, let\u0026rsquo;s go back to buffer modification. I noticed that the Q-values tended to drift upward as the steps progressed, which was especially obvious before implementing Double DQN. My theory here is that because I defined the \u0026ldquo;unsuccessful buffer\u0026rdquo; as simply the storage storing every state of episodes where the total reward was less than 500, the buffer eventually became saturated with \u0026ldquo;long\u0026rdquo; unsuccessful experiences.\nWhy could this be a problem? \u0026ldquo;Long\u0026rdquo; unsuccessful experiences (e.g., an episode lasting 400 steps) look almost identical to successful ones for the first 399 steps. Since the majority of transitions in these episodes still yield a reward of 1, the unsuccessful buffer begins to mirror the successful buffer, diluting the \u0026ldquo;failure signal\u0026rdquo; the agent needs to learn from.\nIf we look closely at the CartPole reward system, the environment outputs a reward of 1 for every step the cart remains upright and a reward of 0 only when it falls or moves out of bounds. This means 1 is the dominant reward, while 0 is extremely rare. So, the moment the environment outputs 0 is the only time a truly meaningful signal is transmitted to the network. It marks the exact transition that should be avoided. Therefore, when \u0026ldquo;long\u0026rdquo; unsuccessful experiences dominate the buffer, these crucial reward-0 transitions become even scarcer.\nFailure Buffer To fix this, I renamed the unsuccessful buffer to the Failure Buffer. Instead of saving entire \u0026ldquo;bad\u0026rdquo; episodes, I modified it to store only the specific terminal states where the reward is 0. By specifically sampling these \u0026ldquo;failure points,\u0026rdquo; we ensure the model constantly remembers exactly what \u0026ldquo;losing\u0026rdquo; looks like, preventing the distribution from narrowing too much even when the agent becomes highly skilled.\nThis approach is conceptually related to prioritized experience replay [5], which prioritizes transitions with high TD error. Here, we prioritize failure states (reward = 0) since they carry the most critical learning signal in CartPole\u0026rsquo;s sparse reward structure. Let\u0026rsquo;s check the results.\nFigure 15. Q-values on fixed states using failure buffer (only terminal states with reward=0) Figure 16. Episode total reward and TD error using failure buffer The total rewards plot looks perfect. The TD error is also decreasing smoothly like a loss plot in supervised learning. Q-values of action right and left are also visibly separated. And at the stable state, it merges to 100, which is intuitively and ideally correct, because I set gamma (discount factor) to 0.99. $1/(1-0.99) = 100$\nHowever, it turned out that it was a lucky case. I did two more runs, and below are the results.\nFigure 17. Q-values - run 2 with failure buffer Figure 18. Episode total reward - run 2 with failure buffer Figure 19. Q-values - run 3 with failure buffer Figure 20. Episode total reward - run 3 with failure buffer As shown in the upper figures, reaching a stable score of 500 could take a considerable amount of time. Also, the lower figures indicate that the performance remains somewhat unstable. Why? The first issue likely occurs because the Q-network learns primarily from two extremes: the successful and failure buffers. Consequently, the network struggles to bridge the gap between failure and success.\nTo avoid slow learning and to be more robust, I believe the Q-network should learn incrementally rather than attempting to jump directly from failures to \u0026rsquo;lucky\u0026rsquo; successful cases. To address this, I am considering removing the successful buffer and instead maintaining a \u0026ldquo;normal\u0026rdquo; buffer, which stores all states except for failures, alongside the failure buffer. This normal buffer could effectively facilitate the incremental connection between failure and success.\nNormal Buffer and Failure Buffer Let\u0026rsquo;s check the results.\nFigure 21. Q-values using normal buffer + failure buffer (7:3 sampling ratio) Figure 22. Episode total reward and TD error using normal + failure buffer The plot looks prettier than before. I think this change is beneficial for two reasons. First, it prevents too-slow learning by incrementally connecting failed to successful experiences as I said. Second, it could also help recovery from significant weight changes, which I haven\u0026rsquo;t encountered yet.\nNow, I think changing the sampling ratio from the two buffers, which was 7:3, might improve the stability.\nAdjusting Sampling Ratio Let\u0026rsquo;s try 6:4.\nFigure 23. Q-values using 6:4 sampling ratio (normal:failed) Figure 24. Episode total reward and TD error using 6:4 sampling ratio The results could be ambivalent. Let\u0026rsquo;s go back to 7:3. However, these results present critical insights. If we look at the episodes between approximately 2700 to 2800, there are suddenly many drops. Why? Below is the log of that range.\nFigure 25. Failure mode distribution from episodes 2700-2800 Category Count Percentage Edge Left 33 2.4% Edge Right 679 49.9% Leaning Left 566 41.6% Leaning Right 83 6.1% This drop coincided with the agent encountering Edge Left and Leaning Right scenarios—classes that are significantly underrepresented in our current failure buffer (2.4% and 6.1%, respectively). This suggests that the model\u0026rsquo;s previous success was partly due to favorable sampling. Once exploration triggered these sparse failure modes, the agent lacked sufficient data in the buffer to recover quickly. Consequently, balancing the class distribution within the failure buffer is critical for robust learning.\nHowever, I am concerned that further manual adjustment of the buffer might lead to excessive \u0026lsquo;hand-crafting.\u0026rsquo; Well, I believe this is a good time to wrap up, and here are the takeaways.\nSummary Achieving the maximum reward of 500 is relatively easy, and even SARSA or a naive DQN can reach this level. Experience Replay is highly effective for stabilizing training; however, without Double DQN (i.e., separating the online and target networks), it causes Q-values to grow uncontrollably. Double DQN appears to be essential when using Experience Replay, since standard Q-learning tends to overestimate Q-values beyond their theoretically correct range. Maintaining an additional failure buffer that stores only terminal transitions (state–action pairs yielding reward 0) proved effective for long-term stability, since CartPole\u0026rsquo;s reward signal is extremely sparse and failures provide the most informative learning signal. In one run, the episode–reward curve became nearly perfectly stable. However, this was likely due to overfitting to only one or two dominant failure modes, while the buffer at that time contained mostly \u0026ldquo;successful\u0026rdquo; trajectories. To become truly robust, the agent must learn to handle all four major failure modes (edge right, edge left, leaning right, leaning left). One promising direction would be to balance the failure buffer so that each failure type is represented evenly (e.g., 25% each), although this was not explored here. Additionally, 3,000 episodes may still be insufficient to fully achieve such robustness. References [1] Sutton, R. S., \u0026amp; Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.\nhttp://incompleteideas.net/book/the-book-2nd.html\nThe foundational textbook covering TD learning, SARSA, and Q-learning algorithms.\n[2] Mnih, V., et al. (2013). Playing Atari with Deep Reinforcement Learning. arXiv preprint arXiv:1312.5602.\nhttps://arxiv.org/abs/1312.5602\nIntroduced Deep Q-Networks (DQN) with experience replay, demonstrating how neural networks can be stabilized for Q-learning.\n[3] Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533.\nhttps://www.nature.com/articles/nature14236\nExtended DQN work published in Nature, providing comprehensive analysis of experience replay and target networks.\n[4] Van Hasselt, H., Guez, A., \u0026amp; Silver, D. (2016). Deep Reinforcement Learning with Double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1).\nhttps://arxiv.org/abs/1509.06461\nIntroduced Double DQN to address Q-value overestimation by decoupling action selection and evaluation.\n[5] Schaul, T., et al. (2016). Prioritized Experience Replay. International Conference on Learning Representations (ICLR).\nhttps://arxiv.org/abs/1511.05952\nProposed prioritizing transitions based on TD error, related to the failure-focused sampling approach used in this post.\n[6] Gymnasium Documentation (formerly OpenAI Gym). CartPole-v1 Environment.\nhttps://gymnasium.farama.org/environments/classic_control/cart_pole/\nOfficial documentation for the CartPole environment used in these experiments.\n","permalink":"https://yeongseung.github.io/posts/mastering_cartpole/post/","summary":"\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eIf you are interested in reinforcement learning, you have probably heard of \u003ca href=\"https://gymnasium.farama.org/environments/classic_control/cart_pole/\"\u003eCartPole\u003c/a\u003e \u003ca href=\"#ref-6\"\u003e[6]\u003c/a\u003e, often considered the \u0026ldquo;Hello World\u0026rdquo; environment of the field. This is partly because the task is simple to understand, and partly because each episode ends quickly, allowing you to observe learning progress within a short time.\u003c/p\u003e\n\u003cfigure class=\"figure-center\"\u003e\n  \u003cimg src=\"/posts/Mastering_CartPole/cartpole.png\" width=\"300\"\u003e\n  \u003cfigcaption\u003eFigure 1. CartPole environment visualization\u003c/figcaption\u003e\n\u003c/figure\u003e\n\u003cp\u003eLooking back, I have run many experiments in CartPole before, but not in a structured or systematic way, which caused me to forget most of the details over time. In this post, I aim to consolidate those trials into a coherent narrative, focusing in particular on why Q-learning can become unstable and how techniques such as experience replay, Double DQN, and failure-focused sampling can improve robustness.\u003c/p\u003e","title":"Stabilizing Q-Learning in CartPole"},{"content":"Codes\nIntro For vision tasks in ML, we naturally need images and corresponding labels to train a model. Normally, we consider two options. searching and finding datasets we can leverage or creating datasets by taking a photos, labeling manually. The latter is rarely feasible, especially for a student or a small team, given the immense amount of time required for labeling.\nFigure 1. Isaac Sim GUI Here\u0026rsquo;s a third option, which is to utilize Isaac Sim(simulation program from NVIDIA) to generate synthetic datasets. Initially, I was kind of skeptical about whether a vision model trained on simulated images would actually perform well in real environments. However, This post changed my mind. I recommend you to read it.\nSetup In short, my goal was to develop a weed detection model and demonstrate to korean agricultural experts that synthetic data can effectively address data scarcity in the field.\nFigure 2. Smart Farm Optimization Pipeline This image made by NotebookLM roughly shows the process.\n0. Asset Preparation We curated high-quality assets from online sources, treating them as direct representations of the crops and weeds in our target agricultural site.\nFigure 3. Bog Marshcress, potato plants, and greenhouse. Weed, crop and Greenhouse. With these assets in hand, the direction was clear. I had to build a virtual potato farm being invaded by Bog Marshcress(Weed). I imagined things like this :\nFigure 4. Dreamed farm depicted by Gemini 1. Virtual Environment Creation 1.1. Ridges, Furrows, Potatoes In order to make ridges and furrows, I used sine graph since they look similar.\nFigure 5. Ridges, Furrows and Potatoes 1.2. Weeds Using Isaac Sim\u0026rsquo;s Replicator feature, I automated the placement of Bog Marshcress(weed) across the field. Replicator is one of the main feature, which allows me to shuffle the environment. Instead of manually placing every weed, I can script them to pop up randomly on the slopes of the ridges, down in the furrows, or right next to the potato plants. I also randomized their density, size, and rotation.\nFigure 6. weeds placed In Greenhouse, In outside. 2. Synthetic Data Generation 2.1. Domain Randomization In addition to randomizing the weed placement, it is best practice to randomize as many variables as possible to ensure model robustness, while I kept the scope manageable. the sun’s position, light color (temperature), and the structural difference between open fields and greenhouses.\nFigure 7. Domain Randomization example 1 and 2. 2.2. Drone-Perspective Data I decided to collect images from a drone\u0026rsquo;s perspective rather than moving the camera randomly. Maintaining a consistent top-down viewpoint makes it significantly easier for the vision model to learn and recognize patterns within the field.\n3. Automated Annotation Since this part is automated, there was surprisingly little for me to do. The real advantage of using a simulator is that labels for tons of drone-captured images are generated without bordering me. Isaac sim recognizes the unique ID of every object, whether it\u0026rsquo;s potato plant or a weed, to automatically produce labels.\nFigure 8. Bounding box and segmentation 4. Vision Model Training I simply used YOLOv11s model for this detecting task. Isaac Sim supports custom writers, so even though YOLO requires a specific format for inputs, we can handle this by writing a writer. Of course, writing a writer is time-cunsuming work, but thanks to this repository, I didn\u0026rsquo;t have to make one from scratch.\nClass Precision Recall mAP50 mAP50-95 All 0.94 0.833 0.919 0.724 Potato 0.931 0.801 0.902 0.705 Weed 0.948 0.865 0.935 0.742 Figure 9. Inference result(16 images) 5.Real World Validation To test how the model might perform in a real field, I conducted a validation using the virtual farm scenarios I built in Isaac Sim. I kept the core layout—the lighting, soil textures, and the placement of crops and weeds—but used Generative AI to transform these scenes into photorealistic images that mimic actual drone footage. This allowed me to analyze the model\u0026rsquo;s inference performance on images that look like the real world.\nFigure 10. Inference result(16 images) The images above (Figures 10) show the model\u0026rsquo;s inference results. When compared with manual visual inspection, the model successfully distinguishes between potato plants and Bog Marshcress.\nHowever, there are too many bounding boxes for a single object. For example, in the middle of Figure 10, there are at most five potato plants (based on the flowers), but there are 10 boxes shown. I think this issue is caused by the fact that I placed the potatoes much more densely in the virtual environment than they would be in reality.\n","permalink":"https://yeongseung.github.io/posts/generating_synthetic_datasets_using_isaac_sim/post/","summary":"\u003cp\u003e\u003ca href=\"https://github.com/Yeongseung/isaacsim-synthetic-data\"\u003eCodes\u003c/a\u003e\u003c/p\u003e\n\u003ch1 id=\"intro\"\u003eIntro\u003c/h1\u003e\n\u003cp\u003eFor vision tasks in ML, we naturally need images and corresponding labels to train a model. Normally, we consider two options. searching and finding datasets we can leverage or creating datasets by taking a photos, labeling manually. The latter is rarely feasible, especially for a student or a small team, given the immense amount of time required for labeling.\u003c/p\u003e\n\u003cfigure class=\"figure-center\"\u003e\n  \u003cimg src=\"/posts/Generating_Synthetic_Datasets_using_Isaac_Sim/GUI_setup.png\" width=\"500\"\u003e\n  \u003cfigcaption\u003eFigure 1. Isaac Sim GUI\u003c/figcaption\u003e\n\u003c/figure\u003e\n\u003cp\u003eHere\u0026rsquo;s a third option, which is to utilize \u003cstrong\u003eIsaac Sim\u003c/strong\u003e(simulation program from NVIDIA) to generate synthetic datasets. Initially, I was kind of skeptical about whether a vision model trained on simulated images would actually perform well in real environments. However, \u003ca href=\"https://developer.nvidia.com/blog/how-to-train-autonomous-mobile-robots-to-detect-warehouse-pallet-jacks-using-synthetic-data/\"\u003eThis post\u003c/a\u003e changed my mind. I recommend you to read it.\u003c/p\u003e","title":"Generating Synthetic Datasets using Isaac Sim"}]