NepNewsCluster

Nepali (~32 million speakers) is well-served by classification benchmarks but absent from bilingual generation evaluation. NepNewsCluster fills that gap with a single, narrowly-scoped, deeply-instrumented test: synthesise multiple news articles into one bilingual brief, measured by an LLM judge against a rubric reverse-engineered from human grading behaviour.

This is the first multi-document, bilingual, rubric-graded news consolidation benchmark for Nepali that we are aware of. Each of 107 cluster-questions packages up to fifteen real articles from different Nepali outlets — covering one news story — and asks the model to produce a Nepali-Devanagari headline, an English headline, a 3–4 sentence summary in each language, and a typed list of named entities mentioned in the English summary. Outputs were graded blind on three axes (Nepali prose, English prose, topic coverage), each scored 1–10 by Claude Opus 4.7 with extended thinking. We report results scaled to a familiar /100 axis-quality scale throughout.

The Nepali NLP gap

Nepali is the lingua franca of Nepal and a substantial diaspora language across Australia, the Gulf, and South Asia. Despite an active research community at IRIIS Nepal, Kathmandu University ILPRL, Tribhuvan University, and NAAMII, the existing benchmark suites for Nepali are concentrated on classification and reading-comprehension tasks. None of them measure the things that matter most for a production news pipeline like K cha khabar: bilingual generation, multi-document synthesis, factuality across two scripts simultaneously.

The gap NepNewsCluster targets is a triangulation: (a) multi-document consolidation, (b) bilingual parallel generation with parallel grading, and (c) rubric-based LLM-judge methodology against realistic Nepali source distribution — production RSS feeds where outlets disagree on numbers and dates, sensational framings drift across the news cycle, and Bikram Sambat ↔ Gregorian conversions catch out even competent models.

Why K cha khabar built it

K cha khabar (kchakhabar.com) is an Outback Yak product: a cross-publisher Nepali news intelligence platform that ingests RSS feeds from thirty-plus Nepali outlets, classifies and clusters articles by underlying story, then renders each cluster as a bilingual brief with ownership transparency and cross-publisher echo detection. Bilingual consolidation is the central LLM-bound step in that pipeline. The economics of running it at production scale every fifteen minutes — for a low-resource language, on multilingual models that are not Nepali-tuned — depend entirely on which model performs the task well at low cost. Without a benchmark, model choice is anecdote.

NepNewsCluster is the answer to the question we already had to answer privately. Publishing it with this much methodological transparency is our contribution back to the Nepali NLP ecosystem.

What models had to do

Given between three and fifteen short news articles from different Nepali outlets covering the same story, the model had to produce in a single completion:

• An English headline of twelve words or fewer
• A Nepali (Devanagari) headline of fifteen words or fewer — actual Nepali, not transliteration
• A 3–4 sentence English summary covering the who/what/where/when/why
• A 3–4 sentence Nepali (Devanagari) summary conveying the same facts in idiomatic Devanagari prose
• A list of named entities mentioned in the English summary, typed as person · org · place · party · event · policy
• A URL slug derived from the English headline

This is the same contract the K cha khabar production summariser fulfils on every cluster. The task is hard for three reasons that compound:

The grading rubric reflects all three pressures, scoring each one explicitly and penalising both axes when an output internally contradicts itself.

Data, models, rubric, judge

Data construction

107 cluster-questions were sampled from a snapshot of approximately 9,000 active clusters in the K cha khabar production database, ranked by COUNT(DISTINCT publisher_id) — a stronger popularity proxy than raw article count. The sample was deliberately stratified rather than pure top-N to avoid overweighting any single coverage tier:

Each cluster contributed up to its fifteen most recent articles to the question. In aggregate the corpus spans 1,310 article snippets (mean 12.2 per question, drawn from a pool of 1,782 in the unfiltered source clusters), 89% Nepali / 11% English by article count, across 23 unique publishers. The publisher list is reproduced in §10. Each article is presented to the model as publisher | language | publishedAt | EN headline | NE headline | excerpt(≤280 chars) | tags — exactly the shape the production summariser already sees. The K cha khabar production summary itself is not in the prompt, so the eval is genuinely a from-scratch consolidation for every model.

Model lineup (test-3, primary)

Thirteen models in three buckets: US frontier proprietary, Chinese frontier (open-weight, large), and locally-runnable (under 150B parameters, open-weight). Two models — DeepSeek V4 Pro and Qwen 3.6 Max — appear as both think and no-think variants, controlling for nothing but the test-time reasoning toggle and the output-token budget.

^† DeepSeek V4 Pro is on a promotional rate of $0.435 in / $0.87 out per million tokens through 31 May 2026. The post-promotion list price is $1.74 / $3.48 (4× higher). All run costs in this paper were billed at the promotional rate. Pricing for every model in this benchmark was last verified on 1 May 2026; treat figures older than ~30 days as stale.

Models we tried and dropped: Google Gemini 3.1 Pro Preview (the endpoint mandates reasoning and was incompatible with the experiment's reasoning-disabled invariant) and Moonshot AI Kimi K2.6 (returned empty content with reasoning off and timed out beyond 180 seconds with reasoning on under strict json_schema). These exclusions are statements about API surface, not capability.

Generation procedure

Each model received the same byte-identical system prompt — sourced directly from the K cha khabar production summariser code — at temperature = 0.2, max_tokens = 1536 for reasoning-OFF models and 8192 for the two reasoning-ON variants. Most models routed through OpenRouter; DeepSeek and Qwen 3.6 Max routed direct to their native APIs because OpenRouter's reasoning controls were not honoured by upstream providers in our testing. The system prompt was SHA-256 hashed and the prefix recorded in the results metadata so any drift between runs is mechanically detectable. Two independent generation runs were executed — test-2 (April 27) with eleven models and test-3 (April 30) with the full thirteen — on the same 107 questions.

Rubric

Three axes, integer 1–10 each. The rubric was reverse-engineered from the heuristics actually applied across the 107 questions in test-2, then minimally edited for test-3 to a "lenient ceiling" variant: a 9 may be awarded for "comprehensive, accurate, clean" output without requiring a non-obvious factual capture. In practice the lenient-ceiling rubric ran stricter on the 7–9 boundary than the original (mean axis 7.29 vs 7.56) — the grader simply held a higher line for 8 even when 9 was technically more reachable. Detailed anchor tables for all three axes are reproduced in Appendix B.

Throughout this paper, axis scores are presented on the more familiar /100 scale (axis × 10), and the overall quality score is the mean of the three axes — also on /100. So a paper that reports "Sonnet 4.6 at 81/100" is reporting an axis-mean of 8.13 on the original 1–10 scale.

Judge

All grading was done by Claude Opus 4.7 with extended thinking, single pass per run. The judge sees an anonymised version of each question — option letters only, no model identifiers, no transport metadata — and returns per-option scores plus per-question best-pick with rationale. A separate 20-question informal cross-judge spot-check was run with OpenAI GPT-5.5 and Perplexity Sonar Reasoning. The spot-check established that coarse claims (top model, sign of think/no-think delta, bucket aggregates) are not Anthropic-stylistic-preference artefacts; fine-grained rank claims (rank #4 vs #5) should not be load-bearing on the basis of this work.

Leaderboard

Sonnet at the top, Qwen Max think second, GPT-5.4 third. Open-weight Gemma punches above its bucket. Reasoning helps for one model and hurts for another. The middle of the table reshuffles meaningfully between runs.

Five things this benchmark says

Latency, tokens, and dollar cost

Test-3 retained full per-call performance metadata for every successful call: latency, prompt and completion tokens, separate reasoning tokens for think variants, and forensic per-call dollar cost computed from each model's published rates as of 1 May 2026. The aggregate run cost across 1,390 successful calls was ≈$15.40 at corrected rates; mean call latency was 27 seconds across the lineup, but the within-lineup spread is two orders of magnitude.

^† DeepSeek V4 Pro per-call costs reflect the promotional rate ($0.435 in / $0.87 out per million tokens) in effect through 31 May 2026. At post-promo list prices ($1.74 / $3.48), DeepSeek V4 Pro (think) per-call rises to ≈$0.0175 and (no-think) to ≈$0.0080.

A few observations worth pulling forward. The think variants spend most of their wall-clock time in reasoning rather than answer generation: DeepSeek's think trace burns 84% of its 95-second budget in the trace; Qwen 3.6 Max think burns 86% of 106 seconds. For a real-time-facing application that must finish well under a minute, neither is viable — the user would have left the page before the answer arrived.

The latency gap between Sonnet (15 s) and Haiku (8 s) is smaller than one might expect; the gap between GPT-5.4 mini (4 s) and GPT-5.4 (8 s) is what the OpenAI naming scheme advertises. GLM 5.1 at 30 seconds is unusual — its average latency is in the same range as Gemma 4 31B (28 s), but its prompt size and output size suggest a slower upstream provider rather than longer inference. Latency in this benchmark is necessarily a property of the *route*, not the *model*.

Devanagari tokenisation disparity also matters. The same Nepali summary length produces different token counts across model families — different tokenisers split Devanagari differently. The MiMo and Qwen tokenisers are notably more verbose on Devanagari than GPT-5 family or Gemma. Per-call cost comparisons are not adjusted for this; readers building cost models for a particular language should account for it directly.

What this benchmark cannot say

This is a single-author study with N=107 cluster-questions, two generation runs, and one primary judge. It is not peer reviewed. The known holes, in roughly decreasing order of severity:

1. Single primary judge. All 107 × 2 questions were graded by Claude Opus 4.7. The 20-question informal cross-judge spot-check with GPT-5.5 and Perplexity Sonar Reasoning is a sanity check on coarse claims, not a Cohen's-κ-grade calibration. A formal 3-judge × 3-run design is committed for v0.3.
2. No human gold labels. The benchmark is fully LLM-graded. Even N=1 human-expert grading on a 20-question sample would meaningfully harden it; that work has not been done.
3. N=107 is small for stable per-axis claims. Per-model averages within ±0.5 axis points (±5 on /100) of each other are not statistically distinguishable under any reasonable test. Standard errors are not reported.
4. Single sample per (model, question). Outputs were generated at temperature=0.2 once per cell. The variance attributable to sampling is unmeasured — though it can now be partially inferred from the test-2 vs test-3 deltas at the model level.
5. LLM-judge bias. The judge (Anthropic Opus 4.7) graded outputs that include 2 Anthropic models (Sonnet 4.6, Haiku 4.5). It is plausible that the judge has a stylistic preference for outputs that resemble its own family's prose. The cross-judge spot-check addresses this only at coarse granularity.
6. Rubric is descriptive, not pre-registered. It was reverse-engineered from heuristics applied during test-2 grading and minimally edited for test-3. Future versions should pre-register the rubric.
7. Source articles are RSS excerpts, not full text. The grader's "ground truth" is itself thin in the long tail of facts. Some "missing facts" complaints may be against details that weren't in the excerpts the grader saw either.
8. Same-lab clustering. Anthropic ×2, OpenAI ×2, Qwen ×4 in the test-3 lineup. Bucket averages reflect lab strategy as much as model ability.
9. API-surface exclusions. Gemini 3.1 Pro and Kimi K2.6 dropped for incompatibility with the reasoning-disabled invariant under the chosen response-format settings — capability claims are about the API surface, not the underlying models.
10. Devanagari tokeniser disparity. Per-call cost numbers are not adjusted for the fact that different tokenisers split Devanagari differently — some models emit ~30% more tokens than others for the same Nepali summary.
11. Reliability is not in the leaderboard. Empty content, schema violations, time-outs are tracked in the per-call metadata for test-3 but not aggregated into the score. A reliability-aware leaderboard is committed for v0.4.
12. Test-3 grader rubric drift ("lenient-ceiling v1"). The rubric was edited between runs to permit a 9 for "comprehensive, accurate, clean" prose without requiring a non-obvious detail. The lenient ceiling did not produce score inflation in practice (mean axis dropped 7.56 → 7.29), but the run-to-run deltas are confounded with this rubric change.

This is a list of things we know are wrong, not a list of things we think are unimportant. Most can be fixed; some need a budget the project does not yet have.

What's next

The journalists who wrote the corpus

This benchmark redistributes headlines and ≤280-character excerpts from 1,310 articles published by 23 Nepali news outlets between February and April 2026. Original copyright belongs to each publisher. Excerpts are redistributed under fair-use principles for non-commercial research only; takedown is unconditional.

Methodologically, the work also leans on three external bodies of research: the NLUE Nepali NLU benchmark (Singh et al., 2024) for the survey of what does and does not exist in Nepali NLP today; the Prometheus-2 and RubricEval rubric-based judging methodology canon; and the cautions raised in "How Reliable is Multilingual LLM-as-a-Judge?" (Findings EMNLP 2025) on per-language Kappa instability — the cross-judge spot-check is essentially that paper's recommendation, scoped down for cost.

The Nepali NLP community at IRIIS Nepal, Kathmandu University ILPRL, Tribhuvan University, NAAMII, and a long roll-call of independent contributors maintain the pretrained models, datasets, and benchmarks (NLUE especially) that make Nepali NLP a phrase that means something in 2026. This work would not exist without theirs.

Who we are. How to reach us. How to get the data.

Outback Yak

Outback Yak is an Australian-incorporated AI & engineering studio (ABN 11 672 730 773) based in Melbourne. We build AI agents, cloud-and-automation systems, and apps & platforms for clients across Australia and the wider region. K cha khabar is one of our products — a research-grade Nepali news intelligence platform, deliberately operated outside Nepal to insulate the Nepali diaspora's information environment from in-country pressures on the press.

K cha khabar

K cha khabar (kchakhabar.com) is a cross-publisher Nepali news intelligence platform: AI-clustered stories with bilingual summaries, ownership transparency across thirty-plus publishers, trending topics, breaking news. Every headline links to its source. The platform is built for the Nepali diaspora, for engaged in-country readers, and for researchers of the Nepal media ecosystem. NepNewsCluster is the formal evaluation backing the benchmark choices made in K cha khabar's production pipeline.

Data & code access

The dataset will be released under CC-BY-NC-4.0 (data) and MIT (code). Source-article excerpts are redistributed under fair-use for non-commercial research; takedown by any listed publisher is unconditional. Please cite the work as below.

Citation

@misc{nepnewscluster2026,
  author       = {Outback Yak},
  title        = {NepNewsCluster v0.2: A bilingual Nepali-English
                  news consolidation benchmark across 13 frontier
                  and open-weight LLMs},
  year         = {2026},
  howpublished = {Outback Yak Research},
  note         = {v0.2 — primary judge (Opus 4.7), N=107,
                  two generation runs, 13 models in primary lineup}
}

Self-contained methodology reference

Sampling and decoding

• Temperature = 0.2. Keeps decoding ~95% deterministic but lets the model break out of degenerate states. Lower than the popular default of 0.7 because the task is factuality-driven and does not benefit from stylistic variance; higher than 0.0 because some models (notably Haiku and DeepSeek no-think) emit subtly worse JSON at strict greedy decoding.
• max_tokens = 1536 for reasoning-OFF models, 8192 for reasoning-ON variants. The 1,536 budget covers the bilingual headline + summary + entity list + slug comfortably; the 8,192 budget covers reasoning trace plus answer.
• response_format = json_schema with strict: true for ten of thirteen models. The three native-API exceptions (DeepSeek both variants, Qwen 3.6 Max both variants, Qwen 3.6 27B) use json_object because their native APIs do not support strict schemas as of April 2026.
• Concurrency: 13 models per cluster fanned out via parallel calls; one cluster at a time. Atomic JSON write after every call → fully resumable on crash.
• Per-call retry: up to 3 attempts on transient errors (HTTP 5xx/429, timeouts, network). No retry on 4xx or empty-content failures — those are surfaced as model failures rather than retried into success.
• Request timeout: 240 seconds — DeepSeek and Qwen 3.6 Max think variants can take 60–110 seconds.

Prompt structure

The system prompt establishes the role ("bilingual news editor for a Nepali aggregator"), the rules (neutrality, headline length caps, no transliteration, content-flag handling for casualty/minor stories), and the entity-extraction contract. The user prompt enumerates each source article in a structured block:

[1] News of Nepal (en) — 2026-04-24 04:52 UTC
  Tags: news_event, politics
  EN: Speaker Devraj Ghimire calls all-party meeting...
  NE: सभामुख देवराज घिमिरेले सबै दलको बैठक बोलाए
  Excerpt: The Speaker has called all parliamentary parties...

[2] OnlineKhabar (ne) — 2026-04-24 06:13 UTC
  ...

The K cha khabar production summary is not in the prompt. Each model is asked to produce its own consolidation from the article-level inputs only.

Sampling distribution

The 107 clusters span themes including national politics, district-level governance, foreign policy, sports (cricket, football), business and economy, education policy, religious events, weather and disaster reporting, road accidents, and obituaries. The dominant theme is politics (40-odd clusters), reflecting the bias of Nepali RSS feeds in the April 2026 sample window. Future work should explicitly down-sample politics to surface non-political performance differences.

Three axes, integer 1–10 each

The full anchor table for each axis. Total = NE + EN + Topics, range 3–30, presented in the body of the paper as a /100 axis-mean.

Axis 1 — Nepali (NE) prose quality

Judges headline_ne + summary_ne together.

Axis 2 — English (EN) prose quality

Same scale as NE, applied to headline_en + summary_en. Common cross-axis failure modes:

• NE correct, EN wrong: translation that flips meaning (लेक "highland pasture" → "Lake"; सम्पत्ति शुद्धीकरण "money laundering" → "Asset Recovery"), party flip (Oli labelled "Nepali Congress President").
• NE wrong, EN correct: Devanagari date conversion mistake (Vaishakh 11 → "अप्रिल ११" instead of "अप्रिल २४"), title slip (अध्यक्ष for सभामुख).
• Both wrong, in different ways: internal NE/EN contradiction. Penalised on both axes; this is worse than a single error in either.

Specific EN deductions: −2 to −3 for translation that flips meaning; −2 for code-switching (Devanagari word inside EN prose); −1 to −2 for systematic BS-to-Gregorian date errors; −1 for plausible name transliteration drift, −2 if it changes the person.

Axis 3 — Topics (entity list) quality

Judges the entities list.

Frequently-penalised entity-type sins

• Species or animal classified as place
• Day-of-week classified as event
• Hospital or airport classified as org instead of place
• Religion or ethnicity classified as org
• Deity name classified as event

How the grader actually used the scale

Across both runs the grader (Claude Opus 4.7) used most of the 1–10 range, but the modal score is 8 in both. Test-3's lenient-ceiling rubric did not produce more 9s — it produced fewer (10.97% vs 19.66%), more 7s (34.34% vs 23.57%), and zero 10s (vs 6 in test-2). The grader interpreted "lenient ceiling" as "be more conservative on the 7→8→9 boundary" rather than "give out more 9s".

One implication for downstream consumers building tooling on top of this kind of grader: expect a Gaussian-ish spread centred on 8, not a uniform distribution across 1–10. Any score difference of less than 0.3 axis points (3 on /100) at the model level is operating well within the noise floor of how this rubric is actually applied.