Customers write in dialect. Banking models read Modern Standard Arabic.
The dialect gap
Arabic banking-intent models were trained on MSA and Palestinian. A Saudi message like «للحين ما وصلتني البطاقة» is a dialect they never saw.
The Saudi gap
There is no Saudi data to train on, and the intents describe a British digital bank. The model knows nothing of the Saudi financial system or Saudi banks: home financing, SIMAH, the Citizen Account, car leasing.
The answer: Samm
Read the customer’s message in their own dialect, find the request, and send it to the right department. If unsure, hand over to a human.
It started with ArBanking77
SinaLab, Birzeit UniversityABSTRACTINSPIRATION
What the original paper did
- Data
- Arabized the English Banking77 set (13,083 queries from a British digital bank, Casanueva et al., 2020) into MSA and Palestinian, keeping the same 77 intents.
- Model
- Fine-tuned AraBERT for intent classification.
- Results
- F1 0.92 on MSA · F1 0.90 on Palestinian.
- Later
- Saudi, Moroccan and Tunisian files were added in December 2024, for testing only. No model trained on them, and no Saudi-only score was published.
What others tried on the same data
- Setup
- Train on MSA + Palestinian, test on five dialects including Saudi. Best overall F1: 0.877, one number for all dialects together.
- Winner (MA)
- Machine-translated MSA into six dialects with Meta’s NLLB, then an ensemble of BERT models. Adding the translations moved one model from 0.79 to 0.84.
- Others
- MARBERTv2 and QARiB were the strongest single models; one team found that translating dialects to MSA hurt.
- Gap
- No team reported Saudi on its own, and none tested on real customer messages.
Every question carries one of 77 intents
Split in half by intent: 1,771 to train, 1,771 held out for testing.
Department groups are my own, for routing in the app. The intents themselves come from ArBanking77. Notice what is missing: loans, SIMAH, government support.
A model that read the news, or one that read Saudis
AraBERT v0.2
American University of Beirut, 2020. Trained on MSA text from news and Wikipedia.
My role for it: the baseline.
Antoun et al., 2020 ↗SaudiBERT
Faisal Qarah, Taibah University, 2024. Trained only on Saudi text: 142M tweets and 15.2 GB of Saudi forums (26.3 GB total). 143M parameters, 12 layers.
In its paper it beat six Arabic models, MARBERTv2 included, on 11 Saudi tasks. Nobody had tried it on banking.
Qarah, 2024 ↗Also considered: MARBERTv2 (Abdul-Mageed et al., 2021), trained on 1B Arabic tweets and one of the strongest models in AraFinNLP. I chose the Saudi-only model because my customers are Saudi.
On 1,771 unseen Saudi questions, SaudiBERT wins
+2.5 F1 points
Same test set, same settings: 4 epochs, learning rate 3e-5, batch 32.
Palestinian adds nothing
0.864 with it, 0.861 without: inside run-to-run noise. I dropped it for a cleaner MSA + Saudi model.
Macro-F1. Scores move about one point between runs; SaudiBERT led in every run. All numbers on this page come from the same run.
The second paper: what real customers say
King Saud UniversityABSTRACTDATASET
Saudi Bank Sentiment Dataset
- Collected
- Every tweet mentioning four Saudi banks during September 2021: 52,254 tweets, filtered to 12,048 in Saudi dialect.
- Banks
- Al Rajhi 7,775 · SNB 2,705 · Alinma 920 · SAIB 648
- Labels
- Sentiment only (negative 8,669 · positive 2,143 · neutral 1,236), each tweet by two of four annotators. No intent labels.
- Why it matters
- Written before ChatGPT, so human-written for certain, and in customers’ real spelling and slang.
The second most common topic has no intent at all
I measured topics across all 12,048 tweets with keyword lists, then checked each against the 77 intents by hand.
| Topic in tweets | Share | In the 77? |
|---|---|---|
| Cards | 10.3% | Covered |
| Loans & financing | 8.9% | Missing |
| Transfers | 6.7% | Covered |
| ATM & cash | 6.7% | Covered |
| Fees & deductions | 5.9% | Covered |
| App & login | 5.7% | Partly |
| Customer service not answering | 4.2% | Missing |
| Home financing | 3.3% | Missing |
| ID & data update | 2.7% | Partly |
| Account freeze | 2.1% | Missing |
| Government support (Citizen Account) | 1.6% | Missing |
| Car ownership transfer | 1.2% | Missing |
| Clearance & letters | 1.1% | Missing |
| SIMAH credit record | 0.7% | Missing |
Approximate shares from keyword matching; a tweet can mention more than one topic. The reason is simple: the intents describe a British digital bank, which has no Saudi home loans, car leases, SIMAH or Citizen Account.
Nine Saudi intents, 86 in total
| Intent | Train | Test |
|---|---|---|
| Loans & financing التمويل والقروض | 156 | 51 |
| Car ownership transfer نقل ملكية السيارة | 55 | 33 |
| App & login التطبيق وتسجيل الدخول | 50 | 30 |
| Customer service not answering خدمة العملاء ما ترد | 49 | 29 |
| Clearance & letters إخلاء طرف وخطابات | 48 | 29 |
| Account freeze تجميد الحساب | 44 | 25 |
| Government support الدعم الحكومي | 42 | 26 |
| SIMAH credit record سمة | 36 | 22 |
| ID & data update تحديث البيانات والهوية | 32 | 19 |
Repeated 3× so they are not drowned out by thousands of ArBanking77 questions.
Never seen by the model, at any stage.
Before fine-tuning, SaudiBERT learned bank talk
SaudiBERT
Knows Saudi dialect from 142M tweets
Bank language
Reads 11,779 bank tweets with no labels, guessing 15% hidden words
Fine-tuning
Classify 86 intents
Learns words like إفراغ, سداد and خصم القسط in context.
Masked language modeling · 5 epochs · learning rate 5e-5. The idea follows Gururangan et al. (2020).
The 264 test tweets were removed first, so the model never saw them, even unlabeled.
The final model: 86 intents
| Training data | Examples |
|---|---|
| MSA questions (ArBanking77) | 10,728 |
| Saudi questions (ArBanking77) | 1,771 |
| My labeled tweets × 3 | 1,536 |
| Total | 14,035 |
| Setting | Value |
|---|---|
| Base | SaudiBERT + bank pretraining |
| Epochs · learning rate | 4 · 3e-5 |
| Batch · max length | 32 · 64 tokens |
| Hardware | Colab T4 GPU |
0.87 on real customer tweets
0.861 before bank-language pretraining
0.862 before: adding nine new intents did not hurt the old ones
F1 for each new Saudi intent
The weakest intents are the ones whose wording overlaps with others. Pretraining helped a little: SaudiBERT had already read 142M Saudi tweets.
Samm: from message to department
طلبك: تجميد الحساب
الحساب والخدمات الرقمية
ما فهمت طلبك بدقة، بحوّلك لموظف
Read
The customer writes in their own dialect
Classify
One of 86 intents, mapped to one of 11 departments
Explain
Confidence and the top three candidates
Hand over
Below 50% confidence, a human takes it
Out-of-scope detection is hard: in CLINC150 (Larson et al., 2019) a BERT confidence threshold caught only about half of out-of-scope queries. Samm’s threshold is a safety net, not a guarantee.
Does it understand this week’s tweets?
Four public tweets to Saudi banks from X, posted after the model was trained. Try each in the app, then type what Samm said.
السلام عليكم عندي مشكله في خصم مستحقات بطاقه سنويه اخذتوها اربع مرات بشهر 😂Expected: transaction charged twice (one of the 77) → Cards
إلى عناية تم سحب مركبتي المؤجرة ذاتياً ومباشرة بمخالفة صريحة لأنظمة البنك المركزي. ورغم مبادرتي بسداد كامل المتأخرات وحسابي دائن، يرفض المصرف تسليمي المركبة ويماطلني بين الفروع، ويطالبني برسوم إيواء باطلة وأقساط جديدة بلا منفعة! النزاع منظور أمام اللجان القضائية التمويلية.Expected: loans & financing, or car ownership transfer
مو من المعقول أن يصدر مصرف بحجم الراجحي خطاب عدم ممانعة، ثم يعيد مبلغ شراء المديونية ويترك العميل يواجه مصيره! خسرت استقراري السكني، واضطررت للاستدانة من الآخرين، وشكواي لدى ساما منذ 4 أكتوبر بلا حل. من يتحمل مسؤولية الضرر الذي وقع عليّ؟Expected: clearance & letters, or loans & financing
حسابي موقوف من يوم الاثنين الماضي بشكل مفاجئ راجعت الفرع أكثر من مرة ورفعوا شكوى وتصعيد وكلمت الهاتف المصرفي عدة مرات لكن بدون نتيجة الحساب متوقف بالكامل وهذا سبب لي ضرر وعطل التزاماتي حلوا الموضوع بشكل عاجل وفك التجميد عن الحساب، وإلا راح أضطر أقدم شكوى لدى البنك المركزي السعوديExpected: account freeze
“Expected” is my own labeling. Results you type are saved in this browser only.
What is still missing
Limitations
One annotator labeled all tweets, with no agreement check.
The 77 old intents were tested on arabized questions, not real tweets.
Scores move about one point between runs.
Merging loans and home financing made the task easier.
Next steps
Several annotators and an agreement score.
Label real tweets for the 77 intents too.
An “upset customer” flag from the dataset’s sentiment labels.
A permanent host instead of Colab.
Everything this project stands on
- Jarrar et al. (2023). ArBanking77. ArabicNLP.
- Casanueva et al. (2020). Banking77: Efficient Intent Detection with Dual Sentence Encoders.
- Malaysha et al. (2024). AraFinNLP 2024 Shared Task.
- MA team (2024). BERT-based Ensemble for Cross-dialectal Arabic Intent Detection.
- Qarah (2024). SaudiBERT.
- Antoun, Baly & Hajj (2020). AraBERT. OSACT.
- Abdul-Mageed et al. (2021). ARBERT & MARBERT. ACL.
- Alqahtani et al. (2022). Saudi Bank Sentiment Dataset. ICNLSP.
- NLLB Team (2022). No Language Left Behind.
- Larson et al. (2019). CLINC150: Intent Classification and Out-of-Scope Prediction. EMNLP.
- Gururangan et al. (2020). Don’t Stop Pretraining. ACL.
- Skiredj et al. (2024). DarijaBanking. Moroccan banking intents, related work.