Hoy en arXiv
Attention Is All You Need
cs.LG · keyword match: transformerThe Transformer architecture relies entirely on attention mechanisms, dispensing with recurrence and convolutions. We show superior quality on machine translation while being more parallelizable and requiring significantly less time to train than recurrent or convolutional baselines.
Sparse-Reward RL Without Manual Shaping
We propose an intrinsic motivation signal that allows agents to learn from sparse rewards without engineered reward shaping, matching or surpassing state-of-the-art on a 26-game Atari benchmark suite.
Retrieval-Augmented Decoders Forget Less Than You Think
Long-context retrieval-augmented decoders preserve more fine-grained factual content than equivalently-sized parametric models, even when the retrieved passage is paraphrased rather than verbatim.
Compositional Sample Efficiency in MoE Language Models
Mixture-of-experts language models trained on a carefully balanced curriculum acquire compositional skills with up to 4.3× fewer tokens than a dense parameter-matched baseline.