FORWARD HIDDEN STATE
h_fHas read: C4
Each bar is one coordinate. The real vector has 2048.00 / ORIENTATION
MusicVAE encoder, unfolded
Two readers.
One memory of the whole.
A bidirectional LSTM is not one reader that can reverse time. It is two independent LSTMs reading the same tape from opposite ends, then joining what they learned.
past ->
current token
<- future
01 / WATCH IT READ
The tape experiment
Move both readers one token at a time.
Blue accumulates the past. Red accumulates the future. They never talk to each other while reading.
F starts here reads right
reads left starts here B
CURRENT INPUT
C4 / HOLD
x_tFirst, the token is represented numerically. Then each reader combines it with its own previous memory.
BACKWARD HIDDEN STATE
h_bHas read: HOLD
Same dimensions, separate weights, opposite history.
FINAL F MEMORY
2048-D
concatenate
FINAL B MEMORY
2048-D
=
SEQUENCE SUMMARY
4096-D
02 / FOLLOW THE SHAPES
Exact MusicVAE dimensions
The data changes shape, not identity.
Click a stage. The batch dimension B is omitted below so one melody stays easy to picture.
STAGE 01
256 time steps, one choice at each step.
A 16-bar melody has 256 sixteenth-note positions. Each position is one categorical choice from 130 tokens, represented as a one-hot row.
Mental picture: a 256-page flipbook. Every page has exactly one marked square.
TIME256
one slot per sixteenth note
VOCAB130
128 pitches + note-off + rest
MEMORY / DIRECTION2048
features learned by each reader
LATENT512
stochastic code sent to decoder
03 / OPEN ONE CELL
The memory valve
An LSTM cell is a controlled edit.
The intimidating equations reduce to four decisions: erase, write, propose, reveal.
KEEP
0.45
forget x old
+
WRITE
-0.17
input x candidate
=
NEW CELL MEMORY
0.29
c_t = f_t * c_(t-1) + i_t * g_t
0.29 = 0.75 * 0.60 + 0.55 * -0.30
h_t = o_t * tanh(c_t)
0.23 = 0.80 * tanh(0.29)
Forgetthe eraser
How much old memory survives?
Writethe pen pressure
How strongly should the new idea enter?
Candidatethe draft
What new information is being proposed?
Outputthe window
How much memory becomes visible as h?
This lab shows one scalar. A 2048-unit LSTM performs the same controlled edit across 2048 coordinates in parallel.
04 / WHY TWO DIRECTIONS?
Context resolves ambiguity
The future can explain the present.
A unidirectional encoder knows only the prefix. A bidirectional encoder is legal here because the full input already exists.
Useful, but still ambiguous.
Now bank means river edge.
ENCODERMay look both ways
The complete melody is already on the table.
DECODERMust look backward only
Future notes do not exist while generating.
05 / INTO THE LATENT
The VAE heads
The encoder predicts a cloud, not one point.
The 4096-D summary branches into a center and uncertainty. Sampling gives z.
CONCATENATED FINAL STATES4096-D
512-D512-Dz = μ + σ * εwhere ε comes from N(0, 1)512-DDISPLAYED MU
DISPLAYED EPSILON
DISPLAYED Z
MUSICVAE ENCODER RECEIPT
InputB x 256 x 130
BiLSTM layer 12 directions x 2048 units
BiLSTM layer 22 directions x 2048 units + residuals
Final state concatB x 4096
Distribution headsmu: B x 512, sigma: B x 512
Samplez: B x 512
06 / MAKE IT STICK
Three retrieval checks
Do not reread. Try to recall.
Retrieving an idea strengthens memory more than seeing it one more time.
When the forward reader is at token 5, what can it know?
Why can the encoder look into the future?
What does MusicVAE concatenate?
THE 20-SECOND RECALL
1Same tape
Two independent readers receive identical tokens.
2Opposite context
F carries the past; B carries the future.
3Join memories
2048 + 2048 coordinates become a 4096-D summary.
4Predict a cloud
Dense heads produce mu and sigma; sampling produces z.