Phase 072: LLM Architecture and Training
Phase 072 of the AI Encyclopedia — LLM Architecture and Training. Topics 1421–1440.
This phase covers LLM Architecture and Training. Below are the 20 concepts grouped under this phase — each is a future article in the Insightful AI World encyclopedia.
1421 Decoder-only Transformers
1422 Encoder-decoder Language Models
1423 Attention Heads
1424 MLP Blocks
1425 Residual Streams
1426 Positional Embeddings
1427 Rotary Position Embeddings
1428 ALiBi
1429 Long-context Attention
1430 Mixture of Experts
1431 Sparse Models
1432 Dense Models
1433 Pretraining Corpora
1434 Data Deduplication
1435 Data Filtering
1436 Curriculum in Pretraining
1437 Distributed Pretraining
1438 Training Stability
1439 Checkpointing
1440 Training Compute Scaling