LSTM Architecture | Part 2 | The How? | CampusX

LSTM’s architecture is built to decide, at every time step, what information to keep, what to overwrite, and what to discard—using a three-part “gating” system that prevents long sequences from collapsing under vanishing gradients. Instead of treating the hidden state as the only memory, the design maintains a separate long-term memory (the cell state) and a short-term working memory (the hidden state), then forces controlled interaction between them.

The architecture looks more complex than a basic RNN because it carries two parallel states: the cell state (long-term memory) and the hidden state (short-term memory). At each time step, the model receives three key inputs: the current input at time t, the previous hidden state from time t−1, and the previous cell state from time t−1. The outputs include the current hidden state and an updated cell state. The cell state update happens in two stages: first, it updates the cell state by removing irrelevant information and adding new candidate information; second, it computes the new hidden state based on the filtered cell state.

The core mechanism is the gates concept—three sigmoid-based gates that regulate information flow element-by-element. The **forget gate** decides what portion of the previous cell state should be erased. It takes the previous hidden state and the current input, passes them through a neural layer with a sigmoid activation, and produces a vector of values between 0 and 1. A value near 0 means “remove this part,” while a value near 1 means “keep it.” This is described as a controlled “removal” step: the previous cell state is multiplied by the forget gate output, effectively zeroing out unneeded components.

Next comes the **input gate**, which determines what new information should be written into the cell state. It computes a set of candidate values (via a tanh-like transformation) from the current input and previous hidden state, then uses the input gate (sigmoid) to filter those candidates—allowing only the parts deemed useful to pass through. The updated cell state is then formed by combining the retained old information (after forget) with the filtered new candidate information (after input).

Finally, the **output gate** decides what part of the cell state becomes the hidden state that will be exposed to the next time step. The cell state is transformed (described as producing h_t from the cell state), and the output gate again acts as an element-wise filter using a sigmoid vector. This gating structure is presented as the reason LSTMs can preserve relevant information over long sequences: instead of relying on repeated transformations of a single hidden state (a weakness in vanilla RNNs), the cell state can carry forward useful signals while gates block irrelevant changes.

The transcript also grounds the math in a sentiment-analysis example, where words are converted into numeric vectors (e.g., one-hot style representations), then fed into the LSTM to predict whether sentiment is 0 or 1. The overall takeaway is that LSTM’s “keep/forget/write/read” logic—implemented through vectorized gates and element-wise operations—turns sequence modeling into a controlled memory management problem rather than an uncontrolled recurrence.