sqlite3_jieba 0.2.1 
sqlite3_jieba: ^0.2.1 copied to clipboard
Published 23 days ago • 
yooss.cn
yooss.cnA jieba-backed FTS5 tokenizer for SQLite, shipped as a native asset. Adds a `jieba` tokenizer and a `jieba_cut()` function to package:sqlite3.
0.2.1 #
- The build hook now fetches a prebuilt native library for the target from the GitHub release and
checks it against a SHA-256 shipped with the package.
rustupis needed only when it falls back to compiling the crate: an uncovered target, a lower deployment target, an unreachable download, or abuild_from_sourceuser-define. - Registers as an
fts5_tokenizer_v2when the host SQLite offersfts5_api.iVersion >= 3(3.47.0+), and as the legacyfts5_tokenizerbelow that.locale=1tables andfts5_locale()now work; jieba segments by dictionary, so the locale is accepted and ignored. - Refuses to load on SQLite older than 3.31.0, with a message naming the version it found.
- Tokenizer options:
tokenize='jieba search 0 hmm 1'. An unrecognised option now fails theCREATE VIRTUAL TABLE. jieba_cut(text, options)takes the same option string, so a query can be cut the way its table was indexed.- New
jieba_version()scalar function. - The vendored SQLite headers now record their provenance (3.53.4) in
rust/c/README.md.
0.1.0 #
- Initial release: a
jiebaFTS5 tokenizer and ajieba_cut()function forpackage:sqlite3. - Tokens carry byte offsets into the source text, so
highlight()andsnippet()return marked-up original text. - Han runs are segmented in jieba search mode with colocated sub-words; everything else uses Unicode UAX#29 word boundaries, lowercased.
Metadata
0
likes
150
points
320
downloads
Documentation
Publisher
Weekly Downloads
Metadata
A jieba-backed FTS5 tokenizer for SQLite, shipped as a native asset. Adds a `jieba` tokenizer and a `jieba_cut()` function to package:sqlite3.
Repository (GitHub)
View/report issues
Topics
#sqlite #fts5 #full-text-search #chinese #tokenizer
License
MIT (license)
Dependencies
code_assets, crypto, hooks, native_toolchain_rust, sqlite3