Hi. Hmm… for now, I have a feeling that the output, loss, and derivative might be getting mixed together:
Also make the reduction explicit. For a 256-sample window:
are both valid conventions, but the gradients differ by a factor of 256. Divide once, in one known place. The MSELoss documentation is a convenient statement of the sum versus mean distinction; I am only using PyTorch here as documentation for the numerical contract, not suggesting that you replace the C implementation.
That last factor is required. tanh also saturates when z becomes large, so I would not introduce it until the linear-output derivative has passed a small check.
I would initially use that only to protect playback or WAV conversion, not as a hidden part of the training graph. A hard clamp can produce no useful gradient once the raw prediction is beyond the boundary.
The cheapest useful gradient test is probably one output bias. Temporarily disable clipping, regularisation, and the optimizer update. Compute its analytic gradient, then evaluate the same fixed window once with the bias slightly increased and once slightly decreased. The centred finite-difference result should agree with the analytic value over some useful epsilon range. This is the same basic mechanism described in gradcheck mechanics, but it is easy to do directly in C.
Why I would separate these cases, and what I would test next
1. “NLL,” absolute error, and squared error are different contracts
You wrote that the displayed error is “nll on abs(predict - target).”
It would help to pin down the exact scalar that is accumulated, without worrying too much about its name.
For example:
e = prediction - target;
loss += fabsf(e);
is L1 or absolute error. Away from exactly zero, its derivative with respect to a linear prediction is approximately:
dL_dz = (e > 0.0f) ? 1.0f : -1.0f;
possibly divided by the window length if you use a mean.
By contrast:
loss += 0.5f * e * e;
dL_dz = e;
is half squared error.
A categorical negative log-likelihood is something else: it normally consumes log-probabilities over classes, as in the NLLLoss contract.
It is possible to interpret L1 as a negative log-likelihood under a fixed-scale Laplace distribution, or squared error under a fixed-variance Gaussian distribution, but then the assumed distribution, scale, constants and reduction are part of the definition. Therefore a reported value such as 0.5 cannot really be compared with a SampleRNN or WaveRNN likelihood unless the equations and denominators match.
For debugging, I would probably print all three separately if useful:
mean absolute error
mean half-squared error
the actual scalar used for backpropagation
That avoids changing the audible progress display while changing the training objective.
2. Why I would not blame the activation yet
The sequence you observed—
loss falls to roughly 0.5-0.6
learning rate is changed
loss later rises above 2
—is compatible with several different mechanisms:
- an incorrect derivative;
- a missing or duplicated reduction factor;
- a large or non-finite gradient;
- a correct gradient applied to the wrong parameter element;
- an optimizer state problem;
- a change of data or recurrent-state boundary;
- or an ordinary unstable update.
The timing alone does not show that the activation caused it. The bad update may have happened before the learning-rate change, and Adam or momentum-like state can continue to affect later updates.
Before changing another condition, a compact one-update trace would be more informative than a long training run:
loss sum
number of contributing samples
loss mean
global gradient norm before clipping
maximum absolute gradient
first NaN or Inf, if any
parameter norm before update
update norm
parameter norm after update
raw output minimum and maximum
played/clamped output minimum and maximum
hidden-state and cell-state norms
whether clipping was activated
You do not need to print every tensor every iteration. One failing window and one ordinary window would already separate many branches.
3. Gradient clipping, value clipping, and normalization
These terms can describe different programs:
| Operation |
What it changes |
| Audio/output clamp |
Limits generated sample values |
| Gradient value clipping |
Limits each gradient component separately |
| Global gradient-norm clipping |
Scales the whole gradient vector only when its norm exceeds a threshold |
| Gradient normalization |
Often forces every update direction to a chosen norm, even when the original gradient is small |
| Optimizer update scaling |
Changes gradients through momentum, moments or adaptive denominators |
A conventional global-norm clip behaves approximately like:
norm = sqrt(sum_of_all_parameter_gradient_squares);
if (norm > max_norm)
{
scale = max_norm / norm;
multiply_every_gradient_by(scale);
}
It should not enlarge a small gradient. The official clip_grad_norm_ description describes the parameter gradients as one combined vector.
That distinction may be relevant to your observation that clipping gives recognisable text while “normalisation” can reach a displayed zero error but produce English-shaped nonsense. If normalization forces nearly every update to a similar magnitude, it can keep moving even when the true gradient becomes tiny. Another possibility is that the clipping, normalization and Adam paths do not traverse exactly the same parameter elements.
A useful comparison would therefore be one update from the same saved state:
plain SGD
SGD plus global-norm clipping
Adam
with the same gradients frozen before any of the three update paths. Then compare the first parameter element where the resulting updates differ unexpectedly.
4. The non-square paths remain especially useful probes
Your public repository README already notes remaining i/j ordering trouble, while also noting that the audio build produces recognisable signal. I would not assume that the current audio program contains the same error, but it gives a very useful audit strategy.
The recurrent matrices are square, so a transpose can sometimes remain in bounds and look plausible. In the audio build, the informative shapes should be approximately:
input -> hidden: [64][1] or [destination][source]
hidden -> output: [1][64]
Those cannot silently masquerade as their transpose as easily as [64][64].
I would follow each non-square parameter through the entire lifecycle:
declaration
initialisation
forward
backward
gradient accumulation
gradient norm calculation
gradient clipping or normalization
SGD update
Adam first moment
Adam second moment
save
load
The text header already contains distinct non-square weight, gradient and Adam-state arrays in lstm.h. The important question is not merely whether each declaration has the intended shape, but whether every loop uses the same [destination][source] convention.
A very small diagnostic value pattern can make this easier than random values. For example, initialise each test weight from both indices so that a transpose is immediately visible in a dump.
5. Save two kinds of checkpoint
There are two useful meanings of “checkpoint” here.
A weights-only sound checkpoint
Enough to hear or generate from a particular set of parameters.
A resumable training checkpoint
Enough to reproduce the next update. For Adam this normally includes:
weights and biases
first moments
second moments
Adam step counter
learning rate and other optimizer settings
recurrent/data position if the run depends on them
The comment in the current header says that Adam state is not yet stored with the model. That is fine for a weights-only listening checkpoint, but it means reloading and continuing Adam is not the same training trajectory. The general distinction is also described in the PyTorch checkpoint guide: resuming training requires optimizer state in addition to model parameters.
For experiments like this, I would save immediately before changing any one of:
activation
loss
learning rate
optimizer
gradient clipping
window/state policy
Then one change can be compared without losing the earlier branch.
6. Teacher-forced prediction and free-running generation are separate tests
The training preview can sound similar very quickly even when free-running generation is poor.
If training uses the true previous sample:
input[t] = target[t-1]
but generation uses:
input[t] = prediction[t-1]
then a small output error becomes part of the next input. The recurrent state can gradually move into a region that was rarely visited during training.
This is a known distinction between teacher-forced and free-running recurrent models. Professor Forcing studies the difference directly and includes raw-waveform vocal synthesis, although its adversarial training method would be much more machinery than I would add here.
I would only borrow the diagnostic idea. From one checkpoint and one starting sample, compare:
teacher-forced one-step error
free-running 16 samples
free-running 64 samples
free-running 256 samples
free-running 1024 samples
Record where the waveform first diverges badly, rather than judging only the final sound.
Interpretation:
Immediate failure
-> output scale, feedback index, target offset, clamp or sign
Gradual drift
-> accumulated prediction error / teacher-forcing gap
Stable local waveform but no larger structure
-> context, state, model scale, data scale or output distribution
This branch is probably separate from the rising training loss. Teacher-forcing mismatch can explain weak generation even when the one-step loss is correct and stable; it does not by itself explain a training objective suddenly rising.
7. Linear output is valid, but it answers a particular question
A linear scalar output with L1 or L2 loss is perfectly reasonable when the next value is nearly determined by the inputs and state. Neural models of deterministic audio effects often use direct continuous sample regression.
Unconditional raw-audio generation is a somewhat different problem. Given the same short waveform history, several next samples may be plausible. A deterministic squared-error model is encouraged toward a conditional average, which can be a poor closed-loop sample even when its one-step error is small.
This is why several established raw-audio generators model a distribution:
- SampleRNN is a hierarchical recurrent model that generates one audio sample at a time.
- WaveRNN uses recurrent computation with a categorical coarse/fine output.
- WaveNet is a fully probabilistic autoregressive waveform model.
These are precedents, not evidence that your linear output is wrong. They use much larger architectures and datasets, and they address different production goals.
A modest later comparison, after the derivative is verified, could be:
linear output + half squared error
tanh output + half squared error + tanh derivative
linear output + L1
8-bit or 9-bit mu-law categories + softmax + cross-entropy + sampling
Use the same audio, initial weights where possible, reduction, checkpoint points and evaluation horizons. I would not start with a Gaussian mixture or another complex density head; it would add several more derivatives and numerical boundaries before the basic path is established.
8. The 256-step window is not necessarily a 256-sample memory limit
At 11.25 kHz, 256 samples are roughly 22.8 ms. That is short for musical structure, but the truncated backpropagation length and the forward recurrent-state lifetime are not necessarily the same.
If hidden and cell state carry from one consecutive window to the next, the forward state can retain influence from earlier samples even though gradients are truncated at the boundary. Truncation changes what earlier events receive credit or blame for; it does not automatically reset the numerical state.
A very cheap pure-forward test is:
run 512 samples continuously, without updating weights
versus:
run 256 samples
carry h and c
run the next 256 samples
do not update weights between the halves
The outputs and final states should agree under the intended state contract. If they do not, the first place to inspect is window indexing, copying, reset timing or circular-buffer position—not the activation.
Also distinguish these boundaries:
consecutive windows from the same recording
new unrelated training segment
start of a file
start of an epoch
start of free-running generation
Each may need a different reset or warm-up policy. Work on truncated BPTT bias is useful background, but a boundary-equivalence test in your C program will probably tell you more immediately.
9. A few DSP-style controls may be more revealing than more training
For one fixed checkpoint, try several deliberately simple signals:
all zeros
constant small DC value
small sine wave
single impulse followed by zeros
short real audio segment
These can expose different failures:
| Input |
Useful observation |
| Silence |
Bias, drift, unstable autonomous state |
| DC |
Sign and steady-state behaviour |
| Sine |
Phase, frequency and amplitude drift |
| Impulse |
State response and decay |
| Real audio |
Actual target behaviour |
For prediction baselines, I would also compare:
next sample = previous sample
small linear AR predictor
LSTM scalar predictor
Audio is strongly locally correlated, so a training prediction that sounds immediately similar may partly reflect persistence. Beating a one-sample-copy baseline tells you that the LSTM has learned something beyond the easiest local rule.
A later DSP-oriented route would be to let a linear predictor handle the easily predictable part and ask the recurrent model to predict a residual or excitation. LPCNet is a much more developed example of that general division of labour. I would treat it as a direction on the map, not as the next rewrite.
10. A compact order of operations
If this were my test branch, I would use this order:
1. Define z, prediction, error, scalar loss and dL/dz separately.
2. Choose sum or mean over the 256 samples.
3. Use a linear output and half squared error.
4. Finite-difference one output bias.
5. Finite-difference one output weight.
6. Save a checkpoint and one-update telemetry.
7. Audit [64][1] and [1][64] paths through every optimizer stage.
8. Compare continuous 512 forward with carried 256 + 256.
9. Compare teacher-forced and free-running horizons.
10. Compare the persistence and small AR baselines.
11. Only then compare tanh, L1 or a probabilistic output.
That route should distinguish an activation choice from a derivative error, an update error, a window/state error and an autoregressive-generation limitation without requiring a framework rewrite.
If a small fragment is easier to post than the whole audio branch, the most informative part would be only:
output affine calculation
output transform or clamp
loss accumulation and reduction
initial output delta
output-weight and output-bias gradients
gradient clipping / normalization
one SGD or Adam update
That should be enough to decide whether 0.5 * y * y belongs where it currently sits, while leaving the rest of the LSTM alone.
