Build

How to Do Roman Urdu Sentiment Analysis in Python (Tested on 2 Million Reviews)

A TF-IDF and logistic regression pipeline for Roman Urdu and code-mixed text, with the splits, per-language checks and label fixes that keep the score honest.

Published
Read time11 min read
A laptop glowing orange in a dark room next to a phone showing a food delivery order, with the headline Teach Python to Read Roman Urdu

To do Roman Urdu sentiment analysis in Python, turn each review into TF-IDF features from word n-grams (1 to 2 words) and character n-grams (2 to 5 letters), then train a logistic regression with balanced class weights. Split the data by group (restaurant, user or product) instead of shuffling rows, and report the score separately for each language. On about 2 million Pakistani food delivery reviews, this setup scored 0.934 macro-F1 on positive vs negative and 0.709 on positive, neutral and negative, and each model trained in about 5 seconds on a laptop with no GPU.

TL;DR

  • Character n-grams handle Roman Urdu's unstable spelling better than a word-only model.
  • A random row split leaks restaurant names into the test set. Split by restaurant.
  • The overall score hid a 9-point gap: 0.721 macro-F1 on English vs 0.633 on Roman Urdu (3 classes).
  • A hand-built spelling normaliser added almost nothing once character n-grams were in.
  • Removing likely mislabelled reviews from the training set raised test macro-F1 from 0.696 to 0.725.

Most sentiment tutorials use clean English movie reviews. Pakistani users write "khana bht acha tha but delivery late thi": half English, half Urdu, typed in English letters, with spelling that changes from person to person. "Bahut" shows up as "bht", "bohat", "boht", "bhot", "bohot" and more. That is Roman Urdu, and it is where English-first pipelines fall apart.

This guide is the full pipeline I built for a freelance data project on a public Kaggle dataset of Pakistani food delivery reviews. Every number below comes from that run, including the parts that didn't help.

Eight-step pipeline in two rows: clean, detect language, group split, word and character TF-IDF, logistic regression, score per language (highlighted), optional normaliser, check labels
The pipeline in this guide, from raw CSV files to a per-language score.

Prerequisites#

  • Python 3 with pandas, numpy, scipy and scikit-learn.
  • A labelled review dataset. Here the label is a 1 to 5 star rating.
  • A column that groups related rows, such as a restaurant ID. You need it for the split in Step 3.
  • A laptop is enough. Every model in this guide trained on a CPU, with no GPU.

What is Roman Urdu, and why is it hard for NLP?#

Roman Urdu is Urdu written in the Latin alphabet. Most Pakistanis use it for texting, social media and reviews, often mixed with English in the same sentence. That mix is called code-mixed text.

Why Roman Urdu is harder than English for standard NLP tools.
ProblemExampleWhy it breaks models
No standard spellingbahut = bht, bohat, boht, bhot, bohot, buhatA word-level model treats each spelling as a separate word
Shares letters and words with English"to", "me", "main", "hi" exist in bothLanguage detection becomes guesswork
Mixed in one sentence"taste acha tha but cold"English-only tools miss half the meaning

The dataset#

The review corpus used in this guide, after de-duplication.
ItemValue
Reviews after cleaning1,987,179
Restaurants22,744
Cities27
Date rangeMarch 2023 to January 2026
Label1 to 5 star rating
Raw size14 CSV files, about 464 MB

Two facts shape everything that follows. The ratings are extreme: 42% of reviews are 5-star and 31% are 1-star, so 2, 3 and 4 stars together are only about a quarter of the data. The reviews are also short: the median is 43 characters, and 15.8% of reviews are 2 words or fewer ("good", "very nice").

Step 1: Clean the data before you train anything#

None of these problems throw an error. The code runs, the charts render and the numbers are quietly wrong. In this dataset:

  • Two source files overlapped by months, so the same review appeared twice. De-duplicate on the review ID, not on the text, because "Good" legitimately appears thousands of times.
  • A restaurant rating of 0 meant "not rated yet", not "terrible".
  • The cuisine column contained values like "25 min", a delivery time that slipped into the wrong column during scraping.
  • Timestamps were in UTC, while Pakistan is UTC+5.

Read every column as a string first, convert types yourself with errors="coerce", and look at the top values of every categorical column by eye.

Step 2: Detect the language of each review#

You need a language label to score the model per language later. Without it, a model that is strong on English and weak on Roman Urdu still produces one comfortable average.

Off-the-shelf detectors are unreliable here because Roman Urdu uses the same alphabet as English. A simple rule-based detector works well enough:

  1. If the text contains Urdu script characters, label it Urdu script.
  2. Otherwise, count known Roman Urdu marker words (hai, nahi, bht, acha, khana, tha) and known English marker words (the, good, taste, delivery).
  3. A Roman Urdu share of 75% or more is Roman Urdu, 25% or less is English, and anything in between is code-mixed.
Python
import re
import pandas as pd

# Shortened examples. The real lists hold a few hundred common words each.
ROMAN_URDU_MARKERS = {"hai", "nahi", "nhi", "bht", "bohat", "acha", "khana", "tha", "thi", "bilkul"}
ENGLISH_MARKERS = {"the", "good", "taste", "delivery", "very", "was", "late", "cold", "nice"}

URDU_RE = re.compile(r"[؀-ۿݐ-ݿﭐ-﷿ﹰ-]")
LATIN_RE = re.compile(r"[A-Za-z]")
TOKEN_RE = re.compile(r"[a-z]+")

def identify_language(text):
    if not isinstance(text, str) or not text.strip():
        return "No letters"
    n_urdu, n_latin = len(URDU_RE.findall(text)), len(LATIN_RE.findall(text))
    if n_urdu and n_urdu >= n_latin:
        return "Urdu script"
    if n_latin == 0:
        return "No letters"
    toks = TOKEN_RE.findall(text.lower())
    ru = sum(t in ROMAN_URDU_MARKERS for t in toks)
    en = sum(t in ENGLISH_MARKERS for t in toks)
    if ru + en == 0:
        return "Undetermined"
    share = ru / (ru + en)
    if share >= 0.75:
        return "Roman Urdu"
    if share <= 0.25:
        return "English"
    return "Code-mixed"

The most important detail is what you leave out. Words such as to, me, main, hi, is and so exist in both languages, so they belong in neither list. Add them and the detector turns into a coin toss.

Reviews repeat a lot, so run the function on unique texts only and map the result back:

Python
def identify_language_series(texts):
    uniq = pd.Series(texts.dropna().unique())
    mapping = dict(zip(uniq, uniq.map(identify_language)))
    return texts.map(mapping).fillna("No letters")
Language mix and star ratings across 1,987,179 reviews.
LanguageShare of reviewsAverage stars1-star share
English77.0%3.3727.8%
Roman Urdu9.3%2.6245.3%
Code-mixed9.1%2.7442.3%
Urdu script0.6%3.0537.8%
Undetermined or no letters4.1%not scorednot scored
Horizontal bar chart of 1-star review share by language: English 27.8%, Roman Urdu 45.3% (highlighted), code-mixed 42.3%, Urdu script 37.8%
Share of 1-star reviews by language. Roman Urdu is 9.3% of reviews but far angrier than English. Source: author's analysis of 1,987,179 reviews.

Roman Urdu reviews are much angrier than English ones: 45.3% are 1-star, against 27.8% for English. The data suggests people switch to their own language when they are upset. If your model is weak on Roman Urdu, it is weak exactly where the complaints are.

This detector is a heuristic. Check it against a few hundred hand-labelled reviews before you publish exact language percentages.

Step 3: Split by group, not by row#

This is the most common mistake in sentiment tutorials, and it makes the score look better than it is.

The usual approach shuffles every row and takes 80% for training and 20% for testing. But reviews of the same restaurant share words: its name, its dishes, its usual complaints. "Ali's Biryani was cold" lands in training, "Ali's Biryani cold again" lands in testing, and the model learns that Ali's Biryani gets complaints instead of learning what a complaint looks like. It is like handing out the exam questions as homework the night before: great results, nothing learned.

Split by restaurant instead. Every review of a restaurant goes entirely to training or entirely to testing, so the model is always scored on restaurants it has never seen:

Python
import numpy as np

def store_split(df, test_frac=0.2, seed=42):
    stores = df["StoreId"].unique()
    rng = np.random.default_rng(seed)
    test_stores = set(rng.choice(stores, int(len(stores) * test_frac), replace=False))
    return df["StoreId"].isin(test_stores).values   # True = test row

scikit-learn's GroupShuffleSplit does the same job if you prefer a built-in.

Two panels. In a row split, restaurants A to D appear in both train and test sets, marked as leaks. In a group split, restaurants A to C are in training and restaurant D only in testing
Row split vs group split (conceptual). The group split tests the model on restaurants it has never seen.

I also ran a second split by time: train on reviews before July 2025, test on reviews after. That checks whether the model holds up on future data. The rule carries over to any project: if rows share a user, session, product or anything else, split by that group.

Step 4: Combine word and character n-grams#

This step matters most for Roman Urdu.

A word-level model treats "bohat", "boht" and "bht" as three unrelated words, so it must see every spelling often enough to learn it. Character n-grams break each word into small chunks. Close spellings share some of those chunks, so the model can connect them without a dictionary. With scikit-learn's char_wb analyzer at 2 to 5 characters, "bohat" and "boht" share 7 chunks, and "boht" and "bht" share 4. It works the way you read messy handwriting: you recognise enough of the shape.

Character 2 to 5-gram chunks for bohat, boht and bht. Bohat and boht share 7 chunks; boht and bht share 4. Shared chunks are highlighted
Character 2 to 5-grams (char_wb) of three spellings of bahut, computed with scikit-learn. Shared chunks let the model link close spellings.
Python
from scipy import sparse
from sklearn.feature_extraction.text import TfidfVectorizer

word = TfidfVectorizer(ngram_range=(1, 2), min_df=3, max_features=300_000,
                       sublinear_tf=True, token_pattern=r"(?u)\b\w+\b",
                       dtype=np.float32)
char = TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5), min_df=5,
                       max_features=300_000, sublinear_tf=True,
                       dtype=np.float32)

X_train = sparse.hstack([word.fit_transform(train_text),
                         char.fit_transform(train_text)]).tocsr()
X_test = sparse.hstack([word.transform(test_text),
                        char.transform(test_text)]).tocsr()

Three settings make a difference:

  • token_pattern=r"(?u)\b\w+\b" keeps one-letter tokens. The default drops them, and short tokens carry meaning in Roman Urdu.
  • char_wb builds character chunks inside word boundaries, so you don't get junk chunks that span two words.
  • dtype=np.float32 halves the memory. With 300,000 training rows and about 232,000 features after min_df filtering, that matters on a laptop.

Step 5: Train simple models first#

Three classic models, trained on 300,000 reviews and tested on 60,000 reviews from unseen restaurants (the binary task used 270,481 and 54,287, because 3-star reviews are dropped):

Python
from sklearn.linear_model import SGDClassifier
from sklearn.naive_bayes import ComplementNB
from sklearn.svm import LinearSVC

models = {
    "Complement NB": ComplementNB(alpha=0.3),
    "Logistic regression": SGDClassifier(loss="log_loss", alpha=2e-6, max_iter=30,
                                         class_weight="balanced", n_jobs=-1),
    "Linear SVM": LinearSVC(C=0.3, class_weight="balanced"),
}

class_weight="balanced" is there because of the extreme ratings. Without it, the model learns to say "positive" or "negative" and almost never "neutral".

Best model per task on the restaurant-grouped split. Training time is the model fit only, on a laptop CPU.
TaskBest modelMacro-F1AccuracyTraining time
Binary (1 to 2 vs 4 to 5 stars)Logistic regression0.9340.935about 5 s
3-class (negative, neutral, positive)Logistic regression0.7090.831about 6 s
5-class (exact stars)Logistic regression0.5210.705about 7 s
Column chart of logistic regression macro-F1 by task: binary 0.934 (highlighted), 3-class 0.709, 5-class 0.521, on a 0 to 1 axis
Macro-F1 falls as the labels get finer. Logistic regression, restaurant-grouped test set.

The linear SVM was almost identical on 3 classes (0.706) but took about 114 seconds to train against about 6 for logistic regression. Naive Bayes was a few points behind. On the time-based split, 3-class macro-F1 was 0.707, so the model held up on later reviews.

Macro-F1 averages the F1 score across classes with every class weighted equally. Use it instead of accuracy when classes are uneven: a model that never predicts "neutral" can still post high accuracy, but its macro-F1 will expose it.

The drop as you add classes is not only the model's fault. A 3-star and a 4-star review often say the same thing ("ok", "good but cold"). People don't agree with each other on stars, so a model can't either. If your real question is "happy or unhappy?", use the binary setup and stop there.

Step 6: Score each language separately#

3-class results by detected language, restaurant-grouped test set.
LanguageMacro-F1 (3-class)Accuracy (3-class)
English0.7210.842
Code-mixed0.6720.804
Roman Urdu0.6330.772
All0.7090.831
Horizontal bar chart of 3-class macro-F1 by language on a 0 to 1 axis: English 0.721, code-mixed 0.672, Roman Urdu 0.633 (highlighted), with a dashed line at the overall 0.709
The overall 0.709 hides a 9-point gap between English and Roman Urdu. Restaurant-grouped test set of 60,000 reviews.

Roman Urdu trails English by about 9 points. On the binary task the gap is smaller (0.885 vs 0.940) but still there. The overall 0.709 hides it completely, because 77% of the test set is English. Ship a sentiment feature for Pakistani users on the overall score alone and it will look fine while working worst on the angriest customers.

The same breakdown by review length had a surprise: the shortest reviews were the easiest. Reviews of 1 to 2 words scored 0.768 macro-F1, while 6 to 10 words scored 0.687. "Very bad" is easy. "Taste was fine but the fries were cold and the rider was late" is not.

Step 7: Test a spelling normaliser, and report what you find#

The obvious next idea is to normalise spellings before training, so "bht", "bohat" and "bhot" become one word. There is no standard dictionary for this, so I built one from the data using a consonant skeleton: collapse repeated letters, then drop the vowels after the first letter.

Python
import re

def skeleton(tok):
    t = re.sub(r"(.)\1+", r"\1", tok)                          # baaahut -> bahut
    return t[0] + re.sub(r"[aeiouy]", "", t[1:]) if t else t   # bahut  -> bht

skeleton("bohat"), skeleton("bhot"), skeleton("buhat")   # all "bht"

Words were grouped by skeleton, and a word joined a group only when:

  • it was similar to the group's most common spelling (difflib.SequenceMatcher ratio of 0.55 or more), and
  • it ended the same way, with a vowel or a consonant, so "kuch" and "kacha" were not merged.

Very short skeletons mix up different words (ka, ki, ko), so for those only repeated letters were merged, such as haii to hai. The result was 214 spelling groups covering 704 spellings, built in about 2 seconds. The largest group was "bht" with 21 spellings, "acha" had 12 and "khana" had 9.

Tested on Roman Urdu and code-mixed reviews only:

Effect of the spelling normaliser on Roman Urdu and code-mixed reviews (3-class).
TextFeaturesMacro-F1
RawWords only0.629
NormalisedWords only0.637
RawWords + characters0.625
NormalisedWords + characters0.629

The normaliser helps a word-only model by almost a point. With character n-grams the gain is within noise, because the character chunks already do much of the same job. If you use character n-grams, you probably don't need a normaliser for classification. Keep it for search, keyword counts and word clouds, where you want one spelling per word.

Step 8: Find the labels that are wrong#

Star ratings are labels written by tired, hungry people, and some don't match the text at all. Two real examples from this dataset:

  • Rated 4 to 5 stars: "taste is very bad money waste"
  • Rated 1 to 2 stars: "Excellent Outstanding.Hughly recommended"

To find these at scale I used confident learning (Northcutt, Jiang and Chuang, the method behind the cleanlab library):

  1. Get out-of-fold probabilities for every review with cross_val_predict(..., method="predict_proba").
  2. For each class, compute the model's average confidence on reviews that carry that label. That is the class threshold.
  3. Flag a review when the model is above threshold for a different class than its label.

On a sample of 150,000 reviews, 15.1% were flagged. About 80% of the flags were extreme stars with mild text, like a 1-star review that says "the wrap was nice but the burgers and fries seemed old". That is a real person, polite in words and angry in stars, and hard to call an error. The clear mistakes were a much smaller group.

Removing the flagged reviews from the training set still helped:

Effect of removing flagged reviews from training data. Test labels unchanged.
Training dataRowsTest macro-F1
All rows120,7740.696
Flagged rows removed102,5500.725
Column chart of test macro-F1: 0.696 when training on all 120,774 rows, 0.725 (highlighted) after removing flagged rows, leaving 102,550
Almost 3 points better with 15% less training data. The test set was not cleaned.

Clean the training set, never the test set. Remove the "hard" rows from your test set and the score goes up while meaning less.

Expected result#

If you follow the same steps on similar data, expect a model that trains in seconds on a CPU and scores in the low 0.9s on positive vs negative. Your exact numbers will depend on the language mix, review length and how noisy your labels are. The two checks that matter are the per-language table and a group split: if either is missing, the headline number can't be trusted.

Which words predict a 1-star or 5-star review?#

As a side result, comparing how often each word appears in 5-star vs 1-star reviews (weighted log-odds with an informative Dirichlet prior, after Monroe, Colaresi and Quinn) gives a usable Roman Urdu sentiment lexicon:

Strongest Roman Urdu signals for 5-star and 1-star reviews.
5-star words1-star words
zabardastnahi / nhi / ni
behtreenbilkul
mazakharab
mashallahfazool
bht achabura
acha thaitna

"Bilkul" means "absolutely" and sounds neutral, but it appeared about 10 times more often in 1-star reviews than in 5-star ones. Think "bilkul bakwas" or "bilkul thanda".

Common pitfalls#

  • Random row split. Restaurant names leak into the test set and inflate the score. Split by group.
  • One overall score. A 77% English test set hides weak Roman Urdu performance. Report per language.
  • Adding shared words to the language markers. to, me, main and hi break the detector.
  • Cleaning the test set. It raises the score and removes exactly the cases you need to measure.
  • Default token_pattern. It drops one-letter tokens that carry meaning in Roman Urdu.

Do you need a transformer for Roman Urdu sentiment analysis?#

Maybe, if you need every last point. A fine-tuned multilingual model such as XLM-RoBERTa will probably beat 0.709 on 3 classes, but this project did not test one, and it needs a GPU, more time and more effort to run in production.

TF-IDF with logistic regression gave 0.934 on happy vs unhappy, trained in seconds on a laptop, is cheap to serve, and is easy to explain to whoever uses it. For many real products that is enough. Start there, measure per language, and move to a bigger model only when the simple one can't do the job.

FAQ#

Is there a Roman Urdu sentiment analysis dataset?#

Public Pakistani food delivery review datasets on Kaggle contain millions of rated reviews in English, Roman Urdu and code-mixed text. The star rating works as a sentiment label once you map stars to classes.

How do I detect Roman Urdu vs English text in Python?#

Count known Roman Urdu marker words and known English marker words in each text, leaving out words that exist in both, such as "to", "me" and "main". Label the text by the Roman Urdu share: 75% or more is Roman Urdu, 25% or less is English, and anything between is code-mixed.

What is a good macro-F1 score for sentiment analysis?#

It depends on the number of classes. On this data, about 0.93 for binary, 0.71 for 3 classes and 0.52 for 5 exact star ratings. More classes mean more overlap between neighbouring labels, so scores drop even for strong models.

Next steps#

  • Wrap the trained vectorizers and model in a small API and log predictions by detected language.
  • Hand-label a few hundred Roman Urdu reviews to check the language detector and the label-noise flags.
  • If Roman Urdu performance is the bottleneck, try a fine-tuned multilingual transformer and compare it on the same group split and per-language table.

References#

Muhammad Azlaan Zubair

Software Engineer and Researcher, writing about AI, developer tooling and automation.

Newsletter

New guides, comparisons and analysis, sent only when there is something worth your time.

  • New articles worth your time, not every link we publish.
  • Tools and resources when they launch.
  • No spam, no selling your address. Unsubscribe anytime.

Get useful signal by email

We're building the list before the first issue. You'll get an email when the newsletter starts.