fuzzy_duplicate_detector 🔍 #

An ultra-pro Dart/Flutter package for detecting fuzzy duplicate records — even when names are spelled differently, carry diacritics, mix Arabic and Latin scripts, or have words in different order.

The Problem #

Every real-world Arabic-language database eventually looks like this:

fuzzy_duplicate_detector solves this with a multi-layer algorithm stack that handles all of these cases simultaneously.

Features #

Installation #

dependencies:
  fuzzy_duplicate_detector: ^1.0.0

dart pub get

Quick Start #

import 'package:fuzzy_duplicate_detector/fuzzy_duplicate_detector.dart';

void main() {
  final groups = FuzzyDedup.find([
    'محمد علي',
    'محمد عَلي',    // diacritics
    'M.ALI',         // transliterated
    'علي محمد',      // word order swapped
    'أحمد',          // different person
  ]);

  for (final g in groups) {
    print('Canonical: ${g.canonical}');
    print('Members  : ${g.members}');
    print('Confidence: ${g.confidencePercent}');
  }
  // Canonical: محمد علي
  // Members  : [محمد علي, محمد عَلي, M.ALI, علي محمد]
  // Confidence: 91.4%
}

API Reference #

FuzzyDedup — Static Methods #

FuzzyDedup.find(items, {config})

Find all duplicate groups in a list of strings.

final groups = FuzzyDedup.find(names);
// Returns: List<DuplicateGroup>
// Groups sorted by averageConfidence descending.
// Items with no duplicates are NOT included.

FuzzyDedup.compare(a, b, {config})

Compare a single pair and return full details.

final r = FuzzyDedup.compare('مصطفى', 'مصطفي');
print(r.confidencePercent);    // e.g. "97.3%"
print(r.levenshteinScore);     // 0.875
print(r.soundexMatch);         // true
print(r.toDetailedString());   // full multi-line breakdown

FuzzyDedup.deduplicate(items, {config})

Return a list with duplicates replaced by their canonical form.

final clean = FuzzyDedup.deduplicate([
  'محمد علي', 'محمد عَلي', 'خالد',
]);
// → ['محمد علي', 'خالد']

FuzzyDedup.findAsMap(items, {config})

Return Map<canonical, List<non-canonical duplicates>>.

final map = FuzzyDedup.findAsMap(names);
// { 'محمد علي': ['محمد عَلي', 'M.ALI'] }

FuzzyDedup.findUniqueItems(items, {config})

Return only items that are NOT in any duplicate group.

FuzzyDedup.groupSummary(items, {config})

Return a human-readable text report of all groups.

print(FuzzyDedup.groupSummary(names));
// Found 2 duplicate group(s) in 6 item(s):
//
//   Group 1  [avg: 93.1%]
//     Canonical : "محمد علي"
//     Duplicate : "محمد عَلي"  (96.8%)
//     Duplicate : "M.ALI"  (89.4%)

Configuration #

FuzzyDedup.find(
  names,
  config: DedupConfig(
    threshold:               0.85,   // 85% minimum confidence
    removeArabicDiacritics:  true,   // محمد عَلي == محمد علي
    normalizeArabicAlef:     true,   // أحمد == احمد
    normalizeArabicTehMarbuta: true, // فاطمة == فاطمه
    normalizeArabicAlefMaqsura: true,// مصطفى == مصطفي
    enableTransliteration:   true,   // محمد ≈ Mohamed
    clusteringThreshold:     500,    // switch to blocked mode at 500 items
    weights: AlgorithmWeights(
      levenshtein: 0.35,
      jaroWinkler:  0.30,
      soundex:      0.20,
      tokenSet:     0.15,
    ),
  ),
);

Built-in Presets #

Algorithm Weight Presets #

Result Objects #

MatchResult #

final r = FuzzyDedup.compare('Abdullah', 'عبدالله');

r.a                    // 'Abdullah'
r.b                    // 'عبدالله'
r.confidence           // 0.813
r.confidencePercent    // '81.3%'
r.levenshteinScore     // 0.0 (different scripts)
r.jaroWinklerScore     // 0.0 (different scripts)
r.soundexA             // 'A430'
r.soundexB             // 'ع590'
r.soundexMatch         // false
r.soundexScore         // 0.0
r.tokenSetScore        // 0.0
r.transliterationBonus // 0.122  ← cross-script bonus!
r.isDuplicate          // true   (with threshold 0.60)

DuplicateGroup #

final g = groups.first;

g.canonical           // 'محمد علي'   (longest member)
g.members             // ['محمد علي', 'محمد عَلي', 'M.ALI']
g.size                // 3
g.averageConfidence   // 0.914
g.confidencePercent   // '91.4%'
g.maxConfidence       // 0.968
g.minConfidence       // 0.861
g.nonCanonicalMembers // ['محمد عَلي', 'M.ALI']
g.pairwiseResults     // List<MatchResult>

Algorithm Stack — How It Works #

Input: ["محمد علي", "محمد عَلي", "M.ALI", "أحمد"]
              │
              ▼
    ┌─────────────────────────┐
    │  GroupBuilder           │  ← orchestrates the pipeline
    └────────┬────────────────┘
             │  For each pair (i, j):
             ▼
    ┌──────────────────────────────────────────────┐
    │           ScoreCombiner.compare(a, b)         │
    │                                              │
    │  1. Normalize → ArabicNormalizer             │
    │                 LatinNormalizer              │
    │                                              │
    │  2. Levenshtein.similarity()  → levScore     │
    │                                              │
    │  3. JaroWinkler.similarity()  → jwScore      │
    │                                              │
    │  4. ArabicSoundex.encode()    → codeA, codeB │
    │     EnglishSoundex.encode()   (auto-detect)  │
    │     soundexScore = 1.0 / 0.75 / 0.5 / 0.0   │
    │                                              │
    │  5. Transliterator.arabicToLatin()           │
    │     (cross-script bonus, max +0.15)          │
    │                                              │
    │  6. Token-Set Ratio          → tokenScore    │
    │     (sort tokens, then JW)                   │
    │                                              │
    │  7. confidence = Σ(wᵢ × scoreᵢ) + bonus     │
    └──────────────────┬───────────────────────────┘
                       │  if confidence ≥ threshold
                       ▼
              UnionFind.union(i, j)
                       │
                       ▼
              UnionFind.getGroups()
                       │
                       ▼
    ┌──────────────────────────────────────────┐
    │  Output: List<DuplicateGroup>            │
    │  sorted by averageConfidence desc        │
    └──────────────────────────────────────────┘

Performance #

Run the bundled benchmark:

dart run benchmark/large_dataset_benchmark.dart

Running Tests #

dart test

The test suite covers:

• All 7 FuzzyDedup static methods
• All Arabic normalisation steps
• Latin normalisation (accents, punctuation, ß→ss)
• Transliterator script detection
• Levenshtein distance (classical + Arabic)
• English Soundex (Robert/Rupert, Mohamed/Muhammad)
• Arabic Soundex (مصطفى/مصطفي, diacritics invariance)
• Jaro-Winkler (MARTHA/MARHTA, prefix bonus)
• AlgorithmWeights sum validation
• UnionFind (transitive grouping, path compression)
• DedupConfig (presets, copyWith)

Use Cases #

• 🏦 Banking KYC — detect duplicate customer onboarding applications
• 🏢 CRM systems — merge duplicate contact records
• 🏛️ Government databases — national ID / voter registry deduplication
• 🛒 E-commerce — merge duplicate customer accounts
• 📊 Data warehouses — ETL pipeline deduplication
• 🏥 Healthcare — patient record deduplication (HIPAA contexts)
• 📚 Libraries — author name deduplication in bibliographic data

Dependencies #

Both are official Dart team packages — minimal footprint, no transitive deps.

Contributing #

Pull requests are welcome! Please:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/my-feature
3. Add tests for new functionality
4. Run dart test and ensure all tests pass
5. Run dart analyze and fix any warnings
6. Submit a PR against main

License #

MIT © 2026 fuzzy_duplicate_detector contributors. See LICENSE for details.