nip_append_only_list

Local-first Dart implementation of the Nostr append-only lists NIP (kinds 1990 Add / 1991 Remove). Built on top of ndk, broadcast_queue_shim_for_ndk and sync_engine_shim_for_ndk, with a sembast-backed cleartext projection so previously-decrypted private entries remain readable across restarts even before a signer reconnects.

What this NIP solves

NIP-51 stores lists as replaceable events: every edit rewrites the whole list, and concurrent edits from different devices silently overwrite each other. This NIP keeps the NIP-51 tag/encrypted-content format unchanged but stores each addition or removal as its own regular event:

• kind:1990 - Add one or more entries to a list
• kind:1991 - Remove one or more entries from a list

State is computed client-side as an OR-Set CRDT: an entry e is a member of list L for author P iff P has signed at least one Add for (L, e) and no Remove for (L, e) with a strictly later created_at. Adds and Removes commute, so offline / multi-device edits converge without coordination.

Local-first design

Four persistence layers are wired by the caller:

1. NDK CacheManager - raw 1990/1991 events as received from relays (encrypted content preserved). It must be persistent (e.g. SembastCacheManager).
2. SyncEngine (from sync_engine_shim_for_ndk) - fills the NDK cache from the relays and remembers, per relay, what it already covered. Each pass reaches overlapMargin (default 7 days) further back, so an edit made offline on another device and delivered late is still picked up. A relay that times out or refuses the request is not marked as covered. Its coverage must live exactly as long as the NDK cache: clear both or neither.
3. Cleartext projection (this package's ProjectionStore, sembast) - per (author, listName) OR-Set bookkeeping in cleartext. Once a private entry has been decrypted with a signer, it stays readable at every subsequent boot, with or without the signer.
4. OfflineBroadcast (from broadcast_queue_shim_for_ndk) - durable outgoing queue. Writes return as soon as the event is persisted; delivery survives restarts and retries until every targeted relay acks.

Security note: the projection stores cleartext for what was originally NIP-44 encrypted content. The caller is responsible for device-level protection (disk encryption, app sandbox).

Install

dependencies:
  nip_append_only_list: ^0.3.0
  ndk: ^0.10.0
  broadcast_queue_shim_for_ndk: ^0.6.0
  sync_engine_shim_for_ndk: ^0.7.1
  sembast: ^3.8.7

Usage

import 'package:broadcast_queue_shim_for_ndk/broadcast_queue_shim_for_ndk.dart';
import 'package:ndk/ndk.dart';
import 'package:nip_append_only_list/nip_append_only_list.dart';
import 'package:sembast/sembast_io.dart'; // or sembast_web on web
import 'package:sync_engine_shim_for_ndk/sync_engine_shim_for_ndk.dart';

final ndk = Ndk(NdkConfig(
  eventVerifier: Bip340EventVerifier(),
  cache: await SembastCacheManager.create(databasePath: 'data'),
));

final outboxDb = await databaseFactoryIo.openDatabase('outbox.db');
final syncDb = await databaseFactoryIo.openDatabase('sync.db');
final projectionDb = await databaseFactoryIo.openDatabase('projection.db');

final outbox = OfflineBroadcast.withNdk(ndk, db: outboxDb);
outbox.start();

final syncEngine = SyncEngine(ndk, db: syncDb);
syncEngine.start();

final lists = AppendOnlyLists(
  ndk: ndk,
  outbox: outbox,
  syncEngine: syncEngine,
  projection: ProjectionStore(projectionDb),
  // Optional: cap the size of consolidation events so relays don't
  // reject huge fresh-Adds or deletion bundles. Default 32 KB.
  // maxEventBytes: 32 * 1024,
);

// Log an account into NDK so write methods pick up its signer
// automatically. Alternatives: `ndk.accounts.loginExternalSigner(...)`
// for a NIP-46 bunker / Amber / NIP-07 signer, or simply pass `signer:`
// explicitly on every call.
ndk.accounts.loginPrivateKey(pubkey: myPubkey, privkey: myPrivkey);

// Add three entries to a list named "fruits".
// `relays` is optional; when omitted, the outbox looks up the author's
// NIP-65 write relays, offline included.
await lists.add(
  listName: 'fruits',
  entries: const [
    AppendOnlyListEntry(tag: 't', value: 'apple'),
    AppendOnlyListEntry(tag: 't', value: 'banana'),
    AppendOnlyListEntry(tag: 't', value: 'cherry', private: true), // encrypted
  ],
);

// Remove one.
await lists.remove(
  listName: 'fruits',
  entries: const [AppendOnlyListEntry(tag: 't', value: 'banana')],
);

// Tag elements after the value (relay hints, petnames, app data) ride
// along in `extras`. They are not part of identity: the most recent Add
// sets them, and a Remove drops the entry whatever its extras.
await lists.add(
  listName: 'fruit-opinions',
  entries: const [
    AppendOnlyListEntry(tag: 't', value: 'apple', extras: ['like']),
    AppendOnlyListEntry(tag: 't', value: 'durian', extras: ['dislike']),
  ],
);

// Read - works offline, no signer needed for entries already projected.
final state = await lists.getList(
  pubkey: myPubkey,
  listName: 'fruits',
);
print(state.entries); // {apple, cherry (private)}

// Reactive view (initial snapshot + live updates). While listened to, the
// list stays registered on the sync engine, which revisits it on its own.
final sub = lists.watchList(
  pubkey: myPubkey,
  listName: 'fruits',
).listen((s) => print(s.entries));

// Periodically compact: emits a fresh Add capturing current state and
// NIP-09-deletes the superseded events.
await lists.consolidate(listName: 'fruits');

await sub.cancel();
await lists.dispose();

API surface

The CRDT core (AppendOnlyListState.fromEvents, foldEvents) is pure and has no NDK dependency at runtime - it can be used standalone to fold any collection of parsed events into a resolved set.

Compatibility with NIP-51

This NIP is complementary to NIP-51, not a replacement. The two are intended to coexist: NIP-51 remains appropriate for human-curated, low-volume, mono-device lists; this NIP targets high-volume, automated, multi-device, and offline-first workflows. Clients may mirror state between the two for interoperability.