Code Llama for Solidity: Fine-Tuning and Evaluation

Frequently Asked Questions

01.Can Code Llama write Solidity out of the box?

Code Llama was trained primarily on Python, C++, Java, PHP, TypeScript, C#, and Bash, and Meta's own release materials do not list Solidity as a supported language, so raw completions are inconsistent on ERC-20 patterns, modifiers, and reentrancy guards. Fine-tuning the instruction variant at 7B or 13B on a labelled Solidity corpus with PEFT and LoRA adapters closes most of that gap without retraining the full 500-billion-token base model, and teams should still gate every generated function through Slither and a human reviewer before merge.

02.How does Code Llama compare to Codex and GPT-4 for smart contract audits?

Trail of Bits tested Codex and GPT-4 through its Toucan prompting framework in March 2023 and found the models produced 12 positive findings across five contracts checked for integer overflow, with 6 of those false, including 4 outright hallucinations, concluding the models do not yet match mature algorithmic tools like Slither for high-precision audit work. Code Llama, fine-tuned specifically on Solidity, narrows false-positive rates on pattern-recognition tasks such as missing access-control modifiers, but it is not a substitute for a manual audit or a static analyzer on ownership and reentrancy logic.

03.What hardware is needed to fine-tune Code Llama on a Solidity dataset?

The 7B parameter Code Llama Instruct model fine-tunes with QLoRA 4-bit quantization on a single 24GB-class GPU using bitsandbytes and PEFT, which is the practical entry point for most engineering teams, while the 13B variant needs roughly 40-48GB of VRAM or gradient checkpointing across two GPUs. Meta's own guidance recommends bfloat16 for the fine-tuning pass because standard float16 can overflow and produce NaNs during backpropagation on longer Solidity files with nested libraries.

04.Does using an AI-assisted audit tool change compliance obligations under the EU AI Act?

The European Parliament and Council reached a provisional agreement on the AI Act on December 9, 2023, which introduces transparency obligations for general-purpose AI providers, including technical documentation and training-content summaries, with stricter evaluation and adversarial-testing duties for foundation models judged to carry systemic risk. A protocol team fine-tuning Code Llama for internal audit tooling is not itself a GPAI provider under the draft text, but any product built on top and distributed to third parties should track how the final regulation classifies foundation-model derivatives before shipping.