LLM◆ AI-generated · Sourced
Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models
TL;DR
Function-aware Fill-in-the-Middle (FIM) mid-training boosts Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B by +2.8/+3.0 and +3.2 on SWE-Bench-Verified, and by +3.7/+4.0/+5.4 on SWE-Bench-Lite — improvements hold across R2E-Gym and SWE-Smith post-training pipelines and generalize to non-Qwen2.5 base models.
Core Motivation: Modeling the "Call-Observe-Consume" Structure
Standard left-to-right code pretraining exposes tool-use capability only unidirectionally, whereas real-world coding agents rely on an action-observation-continuation loop — semantically isomorphic to a function call site (caller binds args → callee computes & returns value elsewhere → downstream code consumes that value), a pattern abundant at internet scale in open-source code.
Method: Function-Aware Fill-in-the-Middle (FIM) Mid-Training
- Masks function bodies selected via program dependency graph analysis plus a complexity-inferability double criterion;
- Conducts self-supervised mid-training on a 2.6B-token decontaminated corpus drawn from 968 GitHub repositories;
- Applied to Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B, with existing agentic post-training pipelines (R2E-Gym, SWE-Smith) unchanged.
Empirical Results
- SWE-Bench-Verified gains: +2.8 (Qwen2.5-Coder-Instruct-7B), +3.0 (14B), +3.2 (Qwen3-8B);
- SWE-Bench-Lite gains: +3.7 (7B), +4.0 (14B), +5.4 (Qwen3-8B);
- Improvements are consistent across both R2E-Gym and SWE-Smith post-training pipelines;
- Generalization confirmed on a non-Qwen2.5 base model;
- Paper ID: arXiv:2607.12463v1.
Sources (compliance trail)
https://arxiv.org/abs/2607.12463