SupportLM

Fine-tuned Qwen2.5-3B for Hinglish/Tanglish Indian e-commerce support  |  Model   GitHub

Classification +5.1% vs GPT-4o-mini
Language match 99.4%
Order ID extraction 99.4%
Training cost Rs 115
mere order ka kya hua yaar 3 din ho gaye ORDER123
bhai payment cut ho gayi order nahi aaya ORD456 😤
enna da ithu order vandhe illa bro paisa pochi
wrong item aaya hai mujhe return karna hai ORDER789
discount code apply nahi ho raha kya scene hai

From: modal deploy modal_app.py

Stays in your browser, never sent anywhere else

SupportLM (Qwen2.5-3B)
fine-tuned
Results appear here
GPT-4o-mini
baseline
Results appear here

Benchmark on 159 held-out test examples

Metric GPT-4o-mini SupportLM SFT SupportLM DPO
Classification accuracy 74.8% 79.9% 78.0%
Order ID extraction 98.7% 99.4% 99.4%
Language match rate 96.2% 99.4% 100.0%