securereview-7b (MLX 4-bit)

A QLoRA fine-tune of Qwen2.5-Coder-7B-Instruct for function-level security code review. The model takes a single function plus a small amount of caller/callee context and emits a structured JSON findings array. Training harness, data pipeline, and evaluation code are open source at github.com/vitorallo/securereview-7b.

In one paragraph

The model is a 16-layer LoRA adapter (rank 16, scale 2.0) fused into a 4-bit AWQ-quantised Qwen2.5-Coder-7B-Instruct base, trained for three epochs on 11,792 supervised examples. Each example is a three-message chat (system / user / assistant) where the assistant message is a valid JSON findings object. The model was trained with mask_prompt: true so the cross-entropy loss was computed only on the assistant tokens — the user prompt acts purely as conditioning context, never as an optimisation target. On 200 held-out in-distribution test examples the fine-tune reaches F1 61.1% against base 14.3% (+46.8 pp), with zero false positives across 37 clean examples and 99% JSON parse rate. Training wall time was ~8 hours on Apple Silicon, with external AI API calls to a pair of data-generation passes. Everything is Apache-2.0.

Evaluation

Evaluated against the unmodified base model (mlx-community/Qwen2.5-Coder-7B-Instruct-4bit) on a 200-example held-out slice of the in-distribution test split (200 of 1,268 records, stratified over (source, language, primary_category), zero overlap with training):

Per-CWE class:

Per-language recall:

Out-of-distribution benchmark

On a separate 23-example out-of-distribution benchmark (hand-authored DVNA and Juice Shop vulnerabilities) the fine-tune scores F1 26.1% against base 34.0%. The training distribution is CVEFixes commit-level data plus rule-steered synthetic examples; the benchmark is small (n=23) and its code does not resemble the training distribution. Treat the in-distribution number as the meaningful signal. An expanded out-of-distribution benchmark is planned for the next release.

How it was trained

Technique

Supervised fine-tuning via LoRA, implemented with mlx-lm lora. The adapter wraps the attention and feed-forward projections of the last 16 transformer layers at rank 16 and scale 2.0 — roughly 23 million trainable parameters out of the base model's 7.6 billion (0.3%). The optimiser is AdamW at a constant learning rate of 2×10⁻⁴ with no warmup and no schedule. Effective batch size is 4 (1 × 4), context window 4,096 tokens, gradient checkpointing enabled to keep peak memory under 8 GB. Three epochs over 9,256 training records comes to 6,942 iterations.

The training objective is plain cross-entropy on the next-token prediction, but with mask_prompt: true the loss is zero everywhere inside the system and user messages. Only the assistant JSON contributes gradients. This means the model is not being trained to write the prompt format — it is being trained to complete the conversation given that format. At inference time, sending a prompt that deviates from the training shape moves the query outside the conditioning region and the model reverts to its base behaviour (see the inference contract below).

Data

The training corpus is 11,792 examples across nine languages, split 80/10/10 stratified on (source, language, primary_category):

Languages: Python, JavaScript, TypeScript, Go, Java, Ruby, Rust, C++, C#. Every source is permissively licensed; the dataset inherits Apache-2.0.

Credits

• TikiTribe/claude-secure-coding-rules — the claude_rules source (537 training records) was parsed from this community-maintained corpus of 100+ security rule sets covering 12 languages, 5 backend frameworks, OWASP Top 10 2025, NIST AI RMF, MITRE ATLAS, and Google SAIF. Every rule ships with structured Do/Don't/Why/Refs sections and CWE + OWASP tags that slot directly into our training schema with no API calls. If you build your own security-review model, this is a very good place to start.
• CVEFixes v1.0.8 — extracted only 5,000+ (to not unbalance the training) real CVE commits and the before/after function pairs that underpin the bulk of the training corpus. Method and citation at arxiv.org/abs/2107.08760.
• Qwen2.5-Coder-7B-Instruct — the base model. Licensed under the Qwen Research License.

Memorisation defence

All user-visible finding fields (description, recommendation, rules_text) are generated by the pipeline rather than copied from the source CVE metadata. Descriptions are built from 12 short phrasings rotated by body hash and interpolated with the vulnerable line number and code snippet. Recommendations combine a per-record prefix (function name plus line number) with a category-specific advice pool whose entries are all ≤25 characters long. The constraint is that no fixed 30-character window of assistant-message text may repeat in more than a per-field threshold of records — description × 20, recommendation × 20, code × 10. A pre-flight probe (pipeline/memorisation_probe.py) walks the final training split and refuses to launch training if any substring crosses these limits, and a hard regex list catches any CVE identifier or CVE-description prose that could otherwise slip through.

This precaution was added after an earlier iteration that stored CVEFixes descriptions verbatim taught the model to regurgitate them on arbitrary user code. See the development story below.

Development story (M1 → M3)

The current release is the third serious training run. Each earlier attempt taught something specific that shaped the data pipeline this release trained on. Recording the iterations here so the evaluation numbers land in context rather than arriving out of thin air.

M1 — assemble the dataset

The pipeline was scaffolded from scratch: a schema matching the intended output, six data generators (CVEFixes SQLite ingest, rule-steered synthetic generation, vulnerable-app extraction, clean code negatives, etc.), and a stratified 80/10/10 assembly step with body-hash dedup and leakage checks. The M1 milestone finished with 11,327 records and a clean validator pass. No training yet.

M2 — first SFT pass (withdrawn)

The first fine-tune finished cleanly: 3 epochs, val loss 1.722 → 0.239, no errors. Then the evaluation harness scored it −15 F1 points below the base model on the held-out benchmark. Inspecting the outputs showed the model emitting phrases like:

"description": "[{'lang': 'en', 'value': 'Multiple SQL injection vulnerabilities in register.php in GeniXCMS before 1.0.0 allow remote attackers to execute arbitrary SQL commands via unspecified vectors.'}]"

on a simple DVNA function. The problem was traced to the CVEFixes ingest: the v1.0.8 Zenodo dump stores cve.description as a Python repr of a list of dicts, not as plain text. The M1 pipeline stored that raw string as the training label. Because CVEFixes was 78% of the training data and every record shared the same corrupted format, the model learned the dominant label shape and replayed it. The ingest was patched to call ast.literal_eval on the column and extract the English description, and the templated "Reference fix commit from {cve_id}" recommendation was replaced with a varied pool to reduce memorisation pressure.

M2.5 — data cleanup and bundled improvements

The second iteration fixed the description corruption and, in the same run, added several improvements drawn from the M2 post-mortem:

• A sink-pattern line picker (SINK_PATTERNS_BY_CWE) that points at the actual vulnerable line in the body rather than falling back to line 1 on the function signature. Dropped line == 1 records from ~78% to ~5%.
• A text-match caller/callee heuristic for CVEFixes so records carry some surrounding context, not just the isolated vulnerable function.
• Per-CVE rules_text synthesis through a concurrent DeepSeek V3.2 prefetch (10 worker threads against OpenRouter), bringing the rules_text coverage from 0% to 100% of CVEFixes records.
• A new safe_logic_gen.py source producing ~186 clean-code affirmative-pattern examples (ownership-filtered queries, role-gated routes, race-safe balance updates) so the model learns what correct logic looks like alongside what broken logic looks like.
• Loosened claude_rules_ingest.py parser recovering more logic-rule examples from the public corpus.
• Type-coercion defences in the training-record builder so LLM output with the wrong primitive type (string confidence, missing severity) no longer crashes the validator.

M2.5 retrained on the new data and scored F1 18.6% → F1 22.2% on the same benchmark. A raw-output diff against the base model revealed two new problems masking the underlying progress that were fixed looking to the next iteration.

When the stop-token fix was applied (see next paragraph) retroactively at inference time, M2.5's test-split F1 jumped from 30.0% to 60.1% — proof that the adapter had learned the right thing but the decoder was throwing half the output away.

M3 — this release

The current release rebuilds the data against these lessons:

• Per-record interpolated descriptions. 12 phrasings, each with ≤25 characters of fixed text between interpolation points. The vulnerable snippet, line number, and category are woven through the template so no long fixed window can repeat across the stratum.
• Per-record interpolated recommendations. Five prefix templates interpolate the function name and line number; ten category-specific advice pools provide ≤25-character suffixes. The full recommendation is nearly unique per record.
• Strict CVE sanitisation. Any CVE identifier in any assistant-message field is a hard-forbidden pattern. CVE prose from the source cve.description column never reaches the training labels. Code comments referencing CVE identifiers are stripped from the body before hashing so commit-message annotations in the CVEFixes corpus do not train the model to condition on CVE-shaped comments.
• Dedicated logic-intent generation pass. A new --logic-only mode on the synthetic generator asks the model for examples where the vulnerability is a missing check (ownership filter, role gate, mass-assignment allowlist, session verification) rather than a bad API call. 276 new records added to the dataset.
• Stop-token fix as a first-class artefact. generation_config.json and config.json in this published model both specify eos_token_id: [151643, 151645], so any MLX-LM loader reading the generation config gets the correct stop set without a runtime patch. The evaluation harness and the fused-model smoke test both include regression guards that fail loudly if the !<|im_end|> cycling pattern ever returns.
• Pre-flight memorisation probe. pipeline/memorisation_probe.py walks train.jsonl before every training run and refuses to proceed if any 30-character substring in an assistant-message field repeats more than the per-field threshold.

Final val loss: 0.214 (started at 1.521). In-distribution F1 61.1%, zero false positives, 99% JSON parse rate. See the evaluation section for the full breakdown.

Inference contract

The model was trained with mask_prompt: true, so the loss was computed only on the assistant JSON. Every training example paired a specific system message with a structured user prompt. Using a shorter or differently-shaped prompt moves inference outside the conditioning region the model was trained on and recovers base-model behaviour (English prose, no JSON).

System message (exact string, 18 words)

You are a JSON API that performs security code review. You only output valid JSON. Never output markdown, explanations, or text outside JSON.

User prompt structure

Analyze this function for security vulnerabilities.

Function: 
File: ->
Role: 
Called by: 
Calls: 

Security checks:                                   # optional but recommended


Code:

### Stop-token handling

Qwen2.5-Coder-Instruct uses `<|im_end|>` (token id 151645) as its
chat-template EOS. This model ships with `generation_config.json` and
`config.json` both specifying `eos_token_id: [151643, 151645]`, so
mlx-lm 0.20+ reads both values and stops cleanly. If your loader does
not read the generation config, patch the stop set at runtime:

```python
if hasattr(tokenizer, "eos_token_ids") and 151645 not in tokenizer.eos_token_ids:
    tokenizer.eos_token_ids.add(151645)

from mlx_lm import load
model, tokenizer = load("vitorallo/securereview-7b-mlx-4bit")

uvx hf download vitorallo/securereview-7b-mlx-4bit \
    --local-dir ./securereview-7b-mlx-4bit

# Then load from the local path
python -c "
from mlx_lm import load
model, tok = load('./securereview-7b-mlx-4bit')
"

@misc{securereview-7b-2026,
  author = {Vito Rallo},
  title  = {securereview-7b: a 7B fine-tune for structured security code review},
  year   = {2026},
  url    = {https://huggingface.co/vitorallo/securereview-7b-mlx-4bit}
}