fuzzy_duplicate_detector 1.0.0
fuzzy_duplicate_detector: ^1.0.0 copied to clipboard
A lightweight Dart/Flutter package for detecting fuzzy duplicates in Arabic and English text using phonetic matching, edit distance, and similarity scoring, with support for diacritics removal, transl [...]
Changelog #
All notable changes to fuzzy_duplicate_detector will be documented here.
This project adheres to Semantic Versioning.
1.0.0 - 2026-06-02 #
Added #
- ArabicSoundex — full phonetic encoding for Arabic names with Unicode normalization (diacritics, Alef variants, Teh Marbuta, Alef Maqsura, Tatweel, zero-width characters).
- EnglishSoundex — standard Russell & Odell Soundex for Latin names with vowel-separator handling for improved accuracy.
- Levenshtein — Unicode-safe edit-distance similarity using optimised two-row dynamic programming (O(m×n) time, O(min(m,n)) space).
- JaroWinkler — prefix-weighted string similarity (standard 0.1 scaling factor, max 4-character prefix bonus).
- TokenSetRatio — word-order-independent matching by sorting tokens before comparison; handles "Ali Mohamed" vs "Mohamed Ali".
- Transliterator — bidirectional Arabic ↔ Latin phonetic transliteration for cross-script matching ("محمد" ↔ "Mohamed").
- ArabicNormalizer — full Arabic text normalisation pipeline: diacritics, Alef variants, Teh Marbuta, Alef Maqsura, Tatweel, zero-width chars.
- LatinNormalizer — Latin text normalisation: accent removal, punctuation stripping, case folding.
- ScoreCombiner — configurable weighted fusion of all algorithm scores with optional transliteration bonus.
- UnionFind — path-compressed, union-by-rank Disjoint Set Union for transitive duplicate grouping in O(α(n)) per operation.
- GroupBuilder — pairwise O(n²) mode for small datasets; Soundex-blocked O(n·L + k²·L) mode for large datasets (≥ threshold).
- FuzzyDedup — clean static-method public API:
find(),compare(),findDuplicateItems(),findAsMap(). - DedupConfig — fully configurable: threshold, algorithm weights,
preprocessing toggles, clustering threshold;
strictandlenientpresets. - AlgorithmWeights — per-algorithm weight tuning with compile-time sum-to-1.0 assertion.
- MatchResult — detailed per-pair result object with all raw scores.
- DuplicateGroup — rich group model with canonical name, members list, average confidence, and all pairwise results.
- Full unit-test suite (35+ tests across all modules).
- Benchmark suite for 50-name and 1000-name datasets.
- Complete example application with CRM use-case demonstration.