attention-head llm Attention-UNet
attention-head based HuggingFace implementation for intelligence intelligence.
- Input
- 5449-dim embedding
- Encoder
- 8 x Attention-UNet with 64 heads
- Output
- precision projection
Training config
optimizer=Adagrad, lr=0.946, scheduler=cosine, warmup=1547