It reminded me of the time when I used to play around with signal processing in C and inline assembly… I never did a large amount of serious DSP programming myself, and I have not touched C in more than twenty years, but when I compare then with now, the areas that seem to have changed most are the hardware landscape and the surrounding ecosystem. For now, I tried to summarize some of that here:
AI Numerical Programming for Experienced C and DSP Developers
The reported change is a meaningful result. The current training path now produces a steadily decreasing mean negative log-likelihood instead of the earlier unstable behaviour, and the main forward and BPTT expressions are substantially more consistent with the stated [destination][source] convention.
The displayed error is accumulated from -log(p_target) in the current source. If 0.18 is the mean value over the training stream in natural-log units, it corresponds to a training-stream perplexity of approximately exp(0.18) = 1.20. That is strong evidence that the network is learning the 7,750-character document. It should still be described as training performance rather than generalisation performance unless a separate held-out stream is evaluated.
One useful clarification is that the current GRADNORM path is effectively global norm clipping, not unconditional gradient normalization:
if (norm > clip) {
scale = clip / sqrt(norm);
/* scale all gradients */
}
Gradients below the threshold are unchanged; gradients above it are rescaled to the threshold. This is broadly the same policy represented by PyTorch’s clip_grad_norm_.
A static reading of the current source at commit 994cfc6 suggests one further mechanical pass before treating the next run as a clean baseline:
keep [destination][source] as the single matrix convention
-> apply it through the complete parameter lifecycle
-> initialize a genuinely fresh model
-> run one memory-instrumented update
-> begin the next training baseline
For the reported plain-SGD and GRADNORM configuration, the highest-priority paths are:
- new-model initialization;
- global-norm accumulation and scaling;
- the plain-SGD update.
The elementwise clipping and Adam paths contain similar patterns and can be aligned before those paths are used again.
These are local indexing and state-lifecycle items rather than reasons to change the LSTM architecture or abandon the native implementation.
Non-square matrices: one complete lifecycle sweep
The two important non-square parameter families are declared as:
/* [destination][source] */
float nnh0if[hidd][idim];
float nnh0ii[hidd][idim];
float nnh0im[hidd][idim];
float nnh0io[hidd][idim];
float nno[idim][hidd];
The corresponding gradient and Adam-state arrays have the same shapes.
With this convention, the complete traversal rules are:
/* Character input -> first hidden layer. */
for (int dest = 0; dest < hidd; ++dest) {
for (int source = 0; source < idim; ++source) {
nnh0if[dest][source] = ...;
}
}
/* Second hidden layer -> character output. */
for (int dest = 0; dest < idim; ++dest) {
for (int source = 0; source < hidd; ++source) {
nno[dest][source] = ...;
}
}
The same bounds and axis meanings should apply to:
| Lifecycle stage |
Objects that must retain the same shape |
| Initialization |
weights and biases |
| Forward |
weight reads |
| Backward |
weight reads and gradient writes |
| Gradient processing |
norm, scaling and clipping |
| Optimizer |
weights, gradients, first moments and second moments |
| Persistence |
weights, moments, optimizer step and dimension metadata |
New-model initialization
The current newmodel() still traverses the first-layer input matrices using:
for (i = 0; i < idim; ++i)
for (j = 0; j < hidd; ++j)
nnh0if[i][j] = ...;
For an array declared as nnh0if[hidd][idim], the first index is the hidden destination and the second index is the character source. The matching traversal is therefore:
for (int dest = 0; dest < hidd; ++dest) {
for (int source = 0; source < idim; ++source) {
nnh0if[dest][source] = random_weight();
nnh0ii[dest][source] = random_weight();
nnh0im[dest][source] = random_weight();
nnh0io[dest][source] = random_weight();
}
}
The output matrix currently has the opposite residual pattern. For nno[idim][hidd], initialization should use:
for (int dest = 0; dest < idim; ++dest) {
for (int source = 0; source < hidd; ++source) {
nno[dest][source] = random_weight();
}
}
This matters because idim is 96 and hidd is 128.
For the first-layer input arrays, indexing the second dimension up to 127 does not match its declared extent of 96. Some accesses may remain within the larger contiguous allocation while still crossing C/C++ array-row boundaries, repeating parts of the storage and leaving other parts untouched.
For the output array, using a first index up to 127 exceeds the declared 96 destination rows. That can access storage outside the output matrix.
The exact runtime effect depends on the build and memory layout. This observation comes from the declared types and index bounds; it is not a report from an executed sanitizer run.
Gradient norm and scaling
The current GRADNORM path contains the same residual traversal for the non-square gradients.
A direct form is:
double norm2 = 0.0;
for (int dest = 0; dest < hidd; ++dest) {
for (int source = 0; source < idim; ++source) {
norm2 += (double)dnnh0if[dest][source]
* (double)dnnh0if[dest][source];
norm2 += (double)dnnh0ii[dest][source]
* (double)dnnh0ii[dest][source];
norm2 += (double)dnnh0im[dest][source]
* (double)dnnh0im[dest][source];
norm2 += (double)dnnh0io[dest][source]
* (double)dnnh0io[dest][source];
}
}
for (int dest = 0; dest < idim; ++dest) {
for (int source = 0; source < hidd; ++source) {
norm2 += (double)dnno[dest][source]
* (double)dnno[dest][source];
}
}
The recurrent matrices are square, so their two bounds are both hidd. They can use the same semantic naming even though swapping equal bounds would not produce an obvious range error:
for (int dest = 0; dest < hidd; ++dest) {
for (int source = 0; source < hidd; ++source) {
norm2 += (double)dweight[dest][source]
* (double)dweight[dest][source];
}
}
The scaling pass should use exactly the same traversals as the accumulation pass:
double norm = sqrt(norm2);
double scale = 1.0;
if (norm > max_norm) {
scale = max_norm / (norm + 1e-30);
}
scale_all_gradients(scale);
Using a double accumulator for the norm is inexpensive and reduces avoidable rounding during a large reduction, while parameters and gradients can remain float.
Plain SGD
The non-Adam path used in the reported run is in rnn-lstm.h. Its non-square loops still use the former orientation.
A direct [destination][source] version is:
for (int dest = 0; dest < idim; ++dest) {
for (int source = 0; source < hidd; ++source) {
nno[dest][source] -=
learning_rate * dnno[dest][source];
}
obias[dest] -= learning_rate * dobias[dest];
}
for (int dest = 0; dest < hidd; ++dest) {
for (int source = 0; source < idim; ++source) {
nnh0if[dest][source] -=
learning_rate * dnnh0if[dest][source];
nnh0ii[dest][source] -=
learning_rate * dnnh0ii[dest][source];
nnh0im[dest][source] -=
learning_rate * dnnh0im[dest][source];
nnh0io[dest][source] -=
learning_rate * dnnh0io[dest][source];
}
}
The same pattern applies to the first and second Adam moments:
weight[dest][source]
gradient[dest][source]
first_moment[dest][source]
second_moment[dest][source]
The current elementwise-clipping branch and Adam branch can be swept using the same rule. Their relevant sections are:
Making future drift harder
Renaming loop variables from i and j to dest and source helps, but the compiler can also carry the extents into a small helper:
template <size_t Dest, size_t Source>
void sgd_matrix(
float (&weight)[Dest][Source],
const float (&gradient)[Dest][Source],
float learning_rate)
{
for (size_t dest = 0; dest < Dest; ++dest) {
for (size_t source = 0; source < Source; ++source) {
weight[dest][source] -=
learning_rate * gradient[dest][source];
}
}
}
This preserves the existing fixed-size native representation. It does not require introducing a tensor framework or redesigning the program.
A similarly small for_each_matrix_element() helper could be reused for initialization, scaling, clipping and optimizer-state reset.
Validation branches and what each one establishes
Different checks answer different questions. A useful order is to apply the least expensive discriminating check first.
Default branch: one AddressSanitizer update
MSVC supports AddressSanitizer through:
/fsanitize=address
A minimal instrumented path is enough:
start a new process
-> create a new model
-> process one complete BPTT window
-> calculate the gradients
-> run norm clipping
-> perform one optimizer update
-> exit
This passes through the main initialization and training-update paths without requiring a long training session.
An ASan-clean run should be interpreted narrowly:
No covered memory violation was observed on this execution.
It does not establish that:
- every valid element was visited;
- each element was visited exactly once;
- destination and source were not transposed;
- the analytical gradient matches the loss;
- the recurrent state boundary is correct.
ASan is a memory-safety instrument, not a numerical oracle. In particular, an incorrect access can remain within one larger static allocation and escape detection while still violating the intended matrix contract.
Coverage branch: verify the traversal itself
For a non-square matrix, an inexpensive diagnostic is to count visits:
unsigned int input_visits[hidd][idim] = { 0 };
unsigned int output_visits[idim][hidd] = { 0 };
for (int dest = 0; dest < hidd; ++dest) {
for (int source = 0; source < idim; ++source) {
++input_visits[dest][source];
}
}
for (int dest = 0; dest < idim; ++dest) {
for (int source = 0; source < hidd; ++source) {
++output_visits[dest][source];
}
}
The acceptance condition is simple:
every expected element was visited exactly once
no unexpected element was touched
Another diagnostic is to initialize each location with a value derived from both axes:
weight[dest][source] =
1000.0f * (float)dest + (float)source;
A transpose, repeated region or omitted region then becomes visible in a small dump.
This is especially useful for square recurrent matrices, where swapped loop bounds remain numerically equal and do not expose the mistake by range alone.
Numerical branch: finite differences
The strongest small check for BPTT is a tiny deterministic unequal-shape fixture, for example:
vocabulary size = 3
hidden size = 2
layers = 2
sequence length = 3
floating-point type = double
fixed parameters and inputs
no clipping
no Adam
one scalar mean NLL
For a selected parameter:
double original = parameter[k];
parameter[k] = original + epsilon;
double loss_plus = forward_loss();
parameter[k] = original - epsilon;
double loss_minus = forward_loss();
parameter[k] = original;
double numerical_gradient =
(loss_plus - loss_minus) / (2.0 * epsilon);
Compare this value with the analytical BPTT gradient for the same parameter.
PyTorch’s gradcheck uses the same general principle: compare finite-difference estimates with analytical gradients. A PyTorch implementation can serve as an independent executable reference, but a small NumPy or hand-written scalar reference is also sufficient.
The most informative compact result is:
hand-written C/C++ BPTT
approximately equals
central finite difference
approximately equals
independent reference implementation
The comparison should include at least one value from each parameter family:
output weight
output bias
layer-2 recurrent weight
layer-2 inter-layer weight
layer-1 recurrent weight
layer-1 character-input weight
each gate bias family
This distinguishes “training loss decreases” from the narrower claim “the tested derivatives correspond to the tested scalar loss.”
Stateful-forward branch
The recurrent state contract can be checked without training.
With updates disabled, compare:
Path A:
process 2*T characters continuously
Path B:
process T characters
preserve final h and c
process the remaining T characters
The logits and final recurrent states should match within a chosen floating-point tolerance.
This separates two concepts:
- forward recurrent state may continue across chunks;
- backward history may intentionally stop at the truncated-BPTT boundary.
Generalisation branch
The reported 0.18 is calculated on the training document. A minimal held-out check does not need a sophisticated dataset:
training stream:
the current document, excluding one contiguous section
validation stream:
the excluded section, never used for updates
Record both mean NLL values after each pass or fixed number of updates.
Possible interpretations then become clearer:
| Training NLL |
Validation NLL |
Likely interpretation |
| decreases |
decreases |
useful sequence structure is being learned |
| decreases |
stays high |
training stream is being memorized |
| unstable |
unstable |
implementation, learning rate or numerical issue remains |
| stable but flat |
stable but flat |
optimization or capacity may now be the limiting factor |
For a 7,750-character source, the validation estimate will be noisy, but it still distinguishes training fit from held-out prediction.
Generation and Adam can be treated as separate side paths
These items do not need to block a fresh plain-SGD baseline, but separating them prevents their state from being confused with the core LSTM calculation.
Prompt-conditioned generation
In the current sampleoutput(), the prompt first advances the recurrent ring. The code then resets:
tests = 1;
The generation loop calculates its recurrent indices from this reset value:
twin = tests & winm;
tprev = (twin + winm) & winm;
The final prompt state is not necessarily stored in ring slot 0. Resetting the same counter used for recurrent indexing can therefore make generation continue from a different slot than the prompt’s final slot.
A clearer state split is:
uint64_t recurrent_step;
uint64_t displayed_output_step;
The recurrent counter advances for every prompt or generated character:
int twin = recurrent_step & winm;
int tprev = (twin + winm) & winm;
run_one_recurrent_step(...);
++recurrent_step;
The display counter may be reset after the prompt without changing the recurrent-state position.
An alternative simple branch is to make these two modes explicit:
Mode A: generate from zero state
Mode B: condition on a prompt and continue from its final state
Argmax initialization
The generation path currently initializes its maximum with:
float r = 0.f;
Raw logits may all be negative. In that case no class exceeds zero, so the selected character can retain an earlier value.
Argmax can start from the first class:
int best = 0;
float best_logit = netout[twin][0];
for (int i = 1; i < idim; ++i) {
if (netout[twin][i] > best_logit) {
best_logit = netout[twin][i];
best = i;
}
}
Softmax is unnecessary for argmax because softmax preserves the ordering of logits.
If stochastic generation is added later, then temperature and sampling can be applied to a stable softmax calculation as a separate policy:
temperature -> softmax -> sample
That policy should not be mixed into the correctness test for greedy generation.
Adam state lifetime
At the start of the current train() function, the step counter is reset:
adamt = 1;
The moment serialization functions store the first and second moments, but the current format does not appear to store the Adam step counter. The source comments also note that Adam state must be retained for the entire training process.
There are two clean branches.
Fresh-only Adam
For a new model:
initialize model parameters
-> zero every first moment
-> zero every second moment
-> set Adam step to zero
-> begin training
This is sufficient when resuming Adam training is not required.
The reset should be explicit in newmodel(). Static-duration arrays begin at zero when the process starts, but creating a second new model in the same process should not inherit moments from an earlier model.
Resumable Adam
A resumable checkpoint should contain one versioned training state:
format identifier and version
model dimensions
parameters and biases
first moments
second moments
Adam step counter
optimizer hyperparameters or their policy
recurrent/training cursor state, if continuation requires it
A useful checkpoint acceptance test is:
run to a defined optimizer boundary
-> save
-> perform one next update and retain the result
construct a fresh process/state
-> load
-> perform the same next update
-> compare parameters, moments and step counter
This tests the state transition rather than only whether the file can be read.
How the guide relates to this implementation
The guide linked above is not intended to replace the hand-written LSTM with a framework. Its main purpose is to connect low-level numerical work to the validation and integration practices that have developed around modern AI systems.
The central route is:
state the numerical and memory contract
-> create a tiny transparent reference
-> compare with an independent executable oracle
-> verify gradients and state updates
-> integrate through an explicit boundary when useful
-> profile before selecting a faster backend
This source/destination issue is a compact example of why a modern numerical contract includes more than the forward equation:
declaration
-> initialization
-> forward
-> backward
-> gradient processing
-> optimizer
-> checkpoint
The arithmetic can be correct in one stage while a different stage still interprets the axes differently.
Several older low-level skills transfer directly:
| C/DSP practice |
Modern numerical-system counterpart |
| Array and explicit length |
Tensor shape, stride and layout contract |
| Test waveform |
Golden tensor fixture |
| FIR/IIR delay state |
Recurrent or cache state |
| Fixed-point range analysis |
dtype, accumulation and quantization policy |
| Scalar reference loop |
Independent correctness oracle |
| Assembly inspection |
Compiler reports, profiling and backend inspection |
| Saved coefficients |
Versioned model and optimizer state |
| Scope or trace display |
Intermediate tensor and state diagnostics |
The larger changes are mostly around the arithmetic:
- automatic differentiation provides an independent derivative path;
- finite differences make gradient contracts testable;
- sanitizers detect many memory-safety failures;
- Python can be used as an orchestration and reference layer without becoming the deployment implementation;
- custom-operator systems provide explicit framework/native boundaries;
- CPU libraries, compiler vectorizers, GPUs and accelerators offer several optimization layers;
- build, test and artifact provenance have become part of numerical correctness.
For example, MSVC now provides both AddressSanitizer and auto-vectorization reporting. A practical optimization route can therefore be:
correct scalar implementation
-> retained numerical fixture
-> optimized compiler build
-> vectorization report
-> profile
-> intrinsics or inline assembly only for a measured remaining gap
Low-level knowledge remains valuable: memory layout, aliasing, alignment, accumulator precision, saturation, locality and state lifetime still determine real behaviour.
The modern addition is that those properties can be stated as contracts and checked from more than one direction.
A small Python/NumPy/PyTorch script can, for example:
load fixed C/C++ parameters
-> run the same tiny sequence
-> export named intermediate tensors
-> compare the first divergent value
The native implementation remains the program under test. Python is only the control plane and independent reference.
If the native component later needs to participate in a larger system, there are documented routes such as:
Neither is required for this project. They are examples of the integration ecosystem now available around a low-level kernel.
Using the hidden-state display as a diagnostic instrument
The hidden-state display is potentially useful beyond visual interest, provided individual units are not assigned fixed semantic meanings too quickly.
Useful per-step or per-window summaries include:
mean(h)
RMS(h)
minimum and maximum h
fraction of |h| above a chosen threshold
mean and RMS of c
gate histograms
gradient RMS by parameter family
update RMS divided by weight RMS
For example:
double sum = 0.0;
double sumsq = 0.0;
int near_edge = 0;
for (int i = 0; i < hidd; ++i) {
double value = h[t][i];
sum += value;
sumsq += value * value;
if (fabs(value) > 0.95) {
++near_edge;
}
}
double mean = sum / hidd;
double rms = sqrt(sumsq / hidd);
double near_edge_fraction =
(double)near_edge / hidd;
The hidden output h shows only part of the LSTM state. Recording the cell state and gates provides more context:
| Observation |
Possible mechanism to inspect |
h remains near zero |
output gate, cell magnitude or cancellation |
h remains near ±1 |
large ` |
| cell state persists |
forget gate near 1 |
| cell state is repeatedly cleared |
forget gate near 0 |
| little new content enters |
input gate near 0 |
| gradients disappear through time |
recurrent Jacobian and gate saturation |
| gradients spike |
recurrent products, loss scale or update scale |
These are diagnostic hypotheses rather than one-to-one semantic interpretations.
Hidden units form a learned coordinate system. Two functionally similar models can distribute information differently across units, and retraining can rotate or permute the internal representation.
The strongest use of the display is therefore comparative:
before versus after one source change
training forward versus generation forward
one continuous stream versus chunked state carry
plain SGD versus Adam
healthy training versus the first unstable window
layer 1 versus layer 2
A DSP-style spectrum or autocorrelation of selected traces may reveal periodic structure in the character stream, but it should be treated as a description of the chosen internal coordinates rather than proof that a unit implements a specific physical filter.
A practical default route from the present state is therefore:
1. align every non-square lifecycle loop with [destination][source];
2. reset all new-model optimizer state explicitly;
3. run one complete ASan-instrumented BPTT update;
4. create a genuinely fresh model;
5. repeat the plain-SGD, global-norm-clipped training run;
6. retain that result as the new baseline;
7. add a tiny unequal-shape gradient check when stronger numerical evidence is useful.
The current loss reduction is already meaningful integration evidence. The remaining observations are limited, testable and mostly mechanical. Once those paths share one orientation and one state-lifecycle policy, further work can move from implementation recovery to normal modelling choices: held-out evaluation, generation policy, hidden size, BPTT depth, optimizer comparison and the procedural applications for which the network was built.
A lot of current AI/ML libraries use Python as the glue around performance-critical C++, C, or Rust code, so performance constraints are hardly unique to this project… Anyway, now that the code is visible, I can be a little more specific:
w.s. burroughs’ “cut-up method” is likely more appreciable as practice than review. not everyone’s thing, just everything’s one.