Siyu Chen 陈思宇

Ph.D. student, Statistics & Data Science, Yale University

Training Dynamics, Interpretability|LLM Pretraining, Scaling, Latent Reasoning|Post-training & RL, Self-Evolution

Siyu Chen
Statistics & Data Science
Yale University, New Haven, CT
siyu.chen.sc3226 [at] yale.edu

I am a Ph.D. student in the Department of Statistics and Data Science at Yale University, advised by Zhuoran Yang. Before Yale, I studied Electronic Engineering at Tsinghua University.

I work on large language models. My approach is to first understand how a model learns, and then use that understanding to train it better.

I have interned at Google Research as a student researcher, and I am now a Qingyun intern at Tencent Hunyuan, working on omni-modal pretraining.

Beyond research, I enjoy the violin and classical music. Always happy to chat about research or anything else, so feel free to reach out.

You can find my full CV here.

News

Sep 2026INFUSER and our paper on momentum in river-valley loss landscapes are accepted to NeurIPS 2026.
Aug 2026Joined Tencent Hunyuan as a Qingyun intern, working on omni-modal LLM pretraining.
Jul 2026Our momentum paper is a Spotlight at the ICML 2026 Workshop on High Dimensional Learning Dynamics.
Jul 2026Our chain-of-thought paper is published in JMLR.
Jan 2026Two papers at ICLR 2026: sparse autoencoders for feature recovery, and how transformers learn causal structure in context.
Sep 2025Franz–Parisi criterion paper accepted to NeurIPS 2025 as an Oral.
Jun 2025Started as a student researcher at Google Research, studying test-time scaling of LLMs.
Jan 2025Single-index model paper accepted to ICLR 2025 as an Oral.

Research Highlights

Foundations of Learning, Training Dynamics & Interpretability

How transformers pick up features and circuits during training, and when learning is possible at all.

  1. Taming Polysemanticity in LLMs: Theory-Grounded Feature Recovery via Sparse Autoencoders
    A new SAE training algorithm, Group-wise Bias Adaptation, with provable feature recovery. It finds cleaner, more interpretable features in real LLM activations.
    ICLR, 2026 · Invited talk, FAI Seminar 2025
    Code
  2. Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
    Shows end to end how training builds induction heads, the attention circuits behind in-context learning.
    NeurIPS, 2024 · Invited talk, FAI Seminar 2024
  3. Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
    Tracks how multi-head attention trains and proves it converges to the optimal in-context learning algorithm.
    COLT, 2024 Oral · Invited talks, RLChina 2024 and FAI Seminar 2024
  4. Can Neural Networks Achieve Optimal Computational-Statistical Tradeoff? An Analysis on Single-Index Model
    Proves that neural networks trained by gradient descent reach the best possible sample complexity for single-index models, matching statistical-query lower bounds.
    ICLR, 2025 Oral

Pretraining, Scaling & Latent Reasoning

How data, architecture and optimizer choices shape what a model learns, how that changes with compute, and how models can reason in latent space.

  1. Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
    A scaling law for looped transformers over model size, training tokens, and forward and backward loop counts. At matched compute, it shows when full backprop through fewer loops beats truncation.
    Preprint, 2026
    Project PageCode
  2. Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoning
    A looped transformer that reasons in latent space and generalizes to inputs far outside its training range. Probing shows what each loop computes.
    Preprint, 2025
    Project PageCode
  3. Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
    Explains why momentum speeds up training on the river-valley loss landscapes seen in LLM pretraining.
    NeurIPS, 2026 · ICML 2026 HiLD Workshop Spotlight

Post-training, Benchmarking & Self-Evolution

Helping models improve their own reasoning through RL and curricula they write themselves.

  1. INFUSER: Influence-Guided Self-Evolution Improves Reasoning
    A generator and a solver train together. The generator writes problems grounded in documents and learns which ones actually help the solver, using influence scores. The solver's reasoning improves as the curriculum evolves.
    NeurIPS, 2026 · Invited talk, NICETalk 2026
    Project PageCode
  2. Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
    Introduces SciAgentArena, about 200 real research tasks with stepwise checks and an interactive environment for testing AI agents. Agents do well on clear data-analysis workflows but struggle with open-ended research.
    Preprint, 2026
    Project Page

Recent Papers

2026

  1. INFUSER: Influence-Guided Self-Evolution Improves Reasoning
    Siyu Chen, Miao Lu, Beining Wu, Heejune Sheen, Fengzhuo Zhang, Shuangning Li, Zhiyuan Li, Jose Blanchet, Tianhao Wang, Zhuoran Yang
    Advances in Neural Information Processing Systems (NeurIPS), 2026
    arXivProject PageCode
  2. Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
    Miao Lu, Zeyu Bian, Kaiyue Wen, Beining Wu, Siyu Chen, Tianhao Wang, Zhiyuan Li
    Advances in Neural Information Processing Systems (NeurIPS), 2026
    arXiv
  3. Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
    Xuyuan Xiong*, Siyu Chen*, Zihan Shao, Yue Su, Weiyang Jin, Ping Luo, Zhuoran Yang, Xihui Liu
    arXiv:2610.26030, 2026
    arXivProject PageCode
  4. Taming Polysemanticity in LLMs: Theory-Grounded Feature Recovery via Sparse Autoencoders
    Siyu Chen, Heejune Sheen, Xuyuan Xiong, Tianhao Wang, Zhuoran Yang
    International Conference on Learning Representations (ICLR), 2026
    arXivCode
  5. How Transformers Learn Causal Structures In-Context: Explainable Mechanism Meets Theoretical Guarantee
    Jianzhe Wei, Siyu Chen, Jianliang He, Zhuoran Yang
    International Conference on Learning Representations (ICLR), 2026
    OpenReview
  6. Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods
    Xinyang Hu, Fengzhuo Zhang, Siyu Chen, Zhuoran Yang
    Journal of Machine Learning Research (JMLR), 27(202), 2026
    JMLRarXiv
  7. GPT-Driven Drug Optimization with Structured Policy Optimization Post-training
    Xuefeng Liu, Songhao Jiang, Siyu Chen, Zhuoran Yang, Yuxin Chen, Ian T. Foster, Rick L. Stevens
    Machine Learning for Healthcare Conference (MLHC), PMLR 340, 2026
  8. On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
    Jianliang He, Leda Wang, Siyu Chen, Zhuoran Yang
    arXiv:2602.16849, 2026
    arXiv
  9. Neural Networks Provably Learn Spectral Representations for Group Composition
    Jianliang He, Leda Wang, Fengzhuo Zhang, Siyu Chen, Zhuoran Yang
    arXiv:2606.02993, 2026
    arXiv
  10. Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
    Tianyu Liu, Allen Xin Wang, Antonia Panescu, and 30 more authors (incl. Siyu Chen)
    arXiv:2606.12736, 2026
    arXivProject Page

For the full list, see Google Scholar. * Equal contribution.

Experience

2026 –Tencent Hunyuan, Qingyun Intern, Beijing
Pretraining an omni-modal LLM; using gradient information to select and improve pretraining data.
2025Google Research, Student Researcher
Test-time scaling of LLMs; optimizing multi-step LLM workflows.
2024 – 25Yale University, Teaching Assistant
Information Theory; Optimization and Computation; Deep Learning for Scientists and Engineers.

Education

2022 – 27Yale University, Ph.D. in Statistics and Data Science
Advisor: Zhuoran Yang
2018 – 22Tsinghua University, B.S. in Electronic Engineering
China National Scholarship