← Back To Work

Research Build

RAG & LLM Benchmarking - English and Nepali

The benchmark compared recursive, paragraph-hybrid, section-aware, parent-child, and table-aware chunking approaches using a shared retrieval, generation, and evaluation pipeline on Nabil Bank annual-report documents.

Year2026
Impact

The key result is that better retrieval scores do not automatically mean better answers; the best chunking strategy is the one that packages evidence in a clean, structured, generation-friendly way.

Problem

Problem

Financial annual reports are difficult for RAG systems because the answer is often buried inside dense tables, repeated headings, similar numeric values, and long sections of formal text. A retriever may find the correct page, but the generated answer can still fail if the chunk contains too much noise, competing numbers, or poorly separated context. The main problem was to identify which chunking method helps a small local language model not only retrieve evidence, but actually use that evidence correctly...

Approach

Approach

The experiment used a controlled RAG benchmark where only the chunking strategy changed. First, the PDFs were parsed into cleaner document blocks using layout-aware extraction. Then multiple chunking methods were applied: recursive splitting as a baseline, paragraph-hybrid chunking to preserve natural paragraph boundaries, section-aware chunking to keep content within topical sections, table-aware chunking to protect financial table context, and parent-child chunking as a larger-context retrieval design. Each strategy was embedded, retrieved with top-k 5, passed into the same Qwen-based answer-generation prompt, and scored using retrieval metrics, answer F1, exact match, numeric accuracy, citation faithfulness, local judge score, latency, and chunk count.

Outcome

Outcome

The best overall result came from section_aware_512_50. It did not have the highest retrieval hit rate, but it produced the strongest final answer quality: F1 0.4721, Numeric Accuracy 0.4667, Citation Faithfulness 0.7556, Judge Score 7.6667, Judge Correct 0.6000, Judge Faithful 0.8000, and Judge Used Evidence 0.8000. Recursive chunking looked excellent on retrieval metrics, with Hit@K and Evidence Recall both at 1.0000, but its answer quality was weaker because broader chunks introduced more distraction and less precise evidence packaging. The final insight is clear: for financial-report RAG, structure-aware chunking is more useful than simply retrieving larger or more overlapping text windows.