Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading
Engram extends standard token embeddings with learned multi-token lookups. Recurring local patterns retrieve vectors directly, reducing the need to reconstruct them through attention and feed-forward layers. With Engram model architecture optimization, it allows for lower HBM capacity to be needed for models at the same quality. This does not mean there won’t be an insane demand for HBM but it just means that model architecture will continue to innovate around constraints. This model architecture design is naturally codesigned for parameter offloading: each token accesses a few embedding rows whose addresses depend on token IDs, not hidden states. The runtime can prefetch those rows from host DRAM while earlier layers compute, keeping the table outside HBM without transferring entire weight matrices. Our Memory model contains our latest estimates of quarter by quarter HBM, DRAM, & NAND supply and demand. Source: SemiAnalysis Offloading frees HBM for model weights and KV cache, pote
