EngineeringCategory

Chain of Thought Helps Go and C# But Hurts Python: When Prompt Advice Flips by Language

1 min read
Johnathen Chilcher

At GoDaddy, engineers across every team are integrating AI coding tools into their daily workflows — from generating boilerplate to scaffolding entire features. But how do you know if the prompting strategies you rely on actually work across the languages you use?

Johnathen Chilcher, Senior Site Reliability Engineer at GoDaddy, ran 5,760 benchmarks across Python, Go, JavaScript, and C# to find out. The results challenge the assumption that a single prompting approach works everywhere: chain-of-thought prompting — asking the model to reason step-by-step before writing code — improved C# output by +7.7 points and Go by +5.3 points on a 100-point correctness benchmark, but actually degraded Python by -0.5 points. Other techniques like politeness showed the inverse pattern, helping Python and JavaScript while hurting Go and C#.

The practical takeaway: if you prompt the same way across languages, you are leaving quality on the table. The post breaks down which techniques work where and offers language-specific guidance so teams can tune their AI workflows to the stack they actually ship.

Read the full post: Chain of Thought Helps Go and C# But Hurts Python

Explore the open-source benchmarking tool: claude-benchmark