Tiny hackable CUDA language model implementation

(github.com)

17 points | by markusheimerl 2 days ago

2 comments

yobbo 1 hour ago
Looks very nice, but I can't find numerical gradient checks, which is helpful when verifying that backward pass is correct:
https://github.com/markusheimerl/gpt/blob/main/transformer/a...