voice_audio #

Cross-platform audio capture, playback and processing as a single, self-contained C++20 library. One source tree builds for Android, Windows and Linux, with iOS and macOS backends written and in the tree but not yet compiled - see Platform support.

Nothing outside this directory is needed to build it: the utility layer, the logger and the WAV reader are all in-tree, and the third-party dependencies — Opus, Ogg, PortAudio, two pieces of WebRTC DSP and the Speex resampler — are vendored under third_party/.

It is also a Flutter package. The C++ tree is the native side; lib/ is the Dart API on top of it, bound through a small C shim in src/ffi/. The library itself still has no Dart dependency and builds standalone.

As a Flutter package #

dependencies:
  voice_audio:
    git: https://github.com/sevana/flutter_voice_audio

import 'package:voice_audio/voice_audio.dart';

await VoiceAudio.instance.initialize();

// record
final recorder = VoiceAudio.instance.recorder;
await recorder.start();
// recorder.progress() drives a timer and a level meter
final memo = await recorder.stop();
await db.insertMemo(id, memo.bytes);          // one row, ~120 KB a minute

// play
final player = VoiceAudio.instance.player;
await player.play(VoiceMemo(await db.readMemo(id)));
await player.pause();
await player.seek(const Duration(seconds: 5));
await player.completed;

// draw and share
final peaks = await memo.waveform(buckets: 200);
await memo.exportToFile('memo.opus');

VoiceMemo reads duration, sample rate and frame count out of the container header, so they cannot drift from the audio the way separate database columns can. recorder and player are properties rather than constructors because the native side has one capture thread and one playback stream — an API that let you make two would be lying about the second.

Dart is never in the audio path: Recorder_* runs its own thread in C++ and fvm_play_opus_start decodes on the speaker's realtime thread, so there is no ring buffer to keep fed and no underrun to report.

Microphone permission is the application's job. The package declares RECORD_AUDIO on Android so the manifest merge covers it, but it requests nothing at runtime — ask with permission_handler before recorder.start(). It stays a pure FFI binding that way. Watch RecorderProgress.level: where a platform answers a denied permission with silence rather than an error, a meter pinned at zero is the only portable way to notice. There is no background recording support, by design.

FLUTTER_API.md documents the whole surface and how it was arrived at.

Building the package #

dart run ffigen --config ffigen.yaml    # only after changing src/ffi/voice_audio_ffi.h
cd example && flutter run -d linux

example/ is a voice memo app and a test rig: a second screen runs nineteen assertions over the whole API on whatever device it is on, which is the only way to reach a phone's AAudio backend and real microphone from here. It runs headless too, so it can gate a build:

cd example
flutter build linux --debug -t lib/headless.dart
xvfb-run -a ./build/linux/x64/debug/bundle/voice_audio_example   # non-zero on any failure

See example/README.md for the Android form of the same thing.

The Dart tests run against a real build of the library rather than a mock:

cmake -S . -B build-shared -DVOICE_AUDIO_NULL_BACKEND=ON -DVOICE_AUDIO_SHARED=ON
cmake --build build-shared -j
VOICE_AUDIO_LIBRARY=$PWD/build-shared/libvoice_audio.so flutter test

VOICE_AUDIO_LIBRARY also overrides where the package loads the library from at runtime, which is how desktop development avoids an install step.

Apple platforms vendor a prebuilt framework rather than compiling the tree through CocoaPods — run tool/build_apple.sh on a Mac first. Everywhere else CMake builds from source as part of the normal Flutter build.

What it gives you #

Opus is the storage format #

PCM never crosses a module boundary and never reaches storage. Capture buffers go to the encoder as they arrive; the Player decodes on its way to the speaker. What sits in between is a blob:

Capture is 20 ms on every platform because that is exactly one Opus frame, so one fvm_source_read() buffer is one encoded frame with nothing left over.

Backends by platform #

The three declared platforms are the ones that have been built. The iOS and macOS backends are written, reviewed and still in the tree — src/audio/ios/, macos/, ios/ and tool/build_apple.sh are all intact — but neither has ever been through an Apple toolchain. Declaring them would advertise a platform nobody has compiled, so they are left out of the flutter: plugin: platforms: block until one of them builds and runs. Nothing else has to change to put them back.

Building #

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Produces libvoice_audio.a (add -DVOICE_AUDIO_SHARED=ON for a shared library) plus the bundled libwebrtc_dsp.a.

libopus and libogg #

Both are hard dependencies, like the WebRTC DSP: the Recorder encodes with Opus and the Player has no other source. CMake takes a vendored copy under third_party/ when one is there and falls back to pkg-config otherwise:

sudo apt install libopus-dev libogg-dev      # Debian / Ubuntu
brew install opus libogg                     # macOS

Neither the NDK nor the iOS SDK ships them, so the mobile targets need them vendored — and the build refuses to fall back to pkg-config when cross-compiling, because pkg-config ignores CMAKE_FIND_ROOT_PATH and would hand an Android build the host's x86_64 library. Upstream ships CMake support for both, so a subtree is enough (libogg must be v1.3.6 or newer; v1.3.5 declares a cmake_minimum_required that CMake 4 rejects):

git subtree add --prefix third_party/opus https://gitlab.xiph.org/xiph/opus.git v1.5.2 --squash
git subtree add --prefix third_party/ogg  https://gitlab.xiph.org/xiph/ogg.git  v1.3.6 --squash

Both are vendored here now, and both link statically, so a build carries no runtime dependency on either. One wrinkle if you fetch libopus any other way: it derives its version from git describe, and a shallow clone has no tags to describe. Without a package_version file at the root saying PACKAGE_VERSION="1.5.2", opus_get_version_string() reports unknown — which then goes into the vendor string of every file the exporter writes. Release tarballs ship that file; a shallow clone does not.

PortAudio #

Vendored as source and built with the rest of the tree, so it links statically and leaves nothing to ship alongside. On Linux the only libraries a build ends up needing are ALSA and the C/C++ runtime.

Only what CMake reads is kept. Upstream's autotools and SCons scaffolding — configure, ltmain.sh, aclocal.m4, config.guess, config.sub, Makefile.in, SConstruct and the C++ bindings/ — is deleted on import. This tree has never built PortAudio any way but CMake, and those files are GPL-licensed: leaving 2.7 MB of unread GPL inside an MIT package is the kind of thing a licence scanner stops a consumer's build over.

git subtree add --prefix third_party/portaudio \
    https://github.com/PortAudio/portaudio.git v19.7.0 --squash

It used to be a prebuilt archive per platform, and that could not work for a Flutter plugin: those archives were built without -fPIC, so linking them into a shared library failed at the link step, and each one was a single architecture — the macOS archive was i386 plus x86_64, so it could not produce an Apple Silicon slice at all. Building from source fixes both and removes the runtime dependency as well.

One wrinkle: PortAudio 19.7.0 declares cmake_minimum_required(VERSION 2.8), which CMake 4 refuses, and there has been no release since 2021 to move to. The build sets CMAKE_POLICY_VERSION_MINIMUM around that one add_subdirectory — CMake's own escape hatch for exactly this.

-DVOICE_AUDIO_PORTAUDIO=system uses the system library instead, and warns that doing so leaves the application with a dependency to ship.

Android #

cmake -S . -B build-android -G Ninja \
  -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-28 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build-android -j

API 28 is the floor. The AAudio backend calls AAudioStreamBuilder_setUsage, _setContentType and _setInputPreset, which the NDK only declares from API 28. The build stops with a clear message below that. To support older devices, remove USE_ANDROID_AAUDIO from include/voice_audio/fvm_config.h and the OpenSL ES backend is used instead.

iOS / macOS #

Point CMake at the usual iOS toolchain (or add the sources to an Xcode target). On macOS the PortAudio backend is used, same as Linux.

CMake options #

PortAudio on desktop #

auto builds the vendored source under third_party/portaudio and falls back to a system library when it is not there. vendored refuses to fall back; system skips the source and takes the installed library, at the cost of a runtime dependency:

sudo apt install libportaudio19-dev     # Debian / Ubuntu
brew install portaudio                  # macOS

Note that CMake checks the header really exists rather than trusting pkg-config: a stale portaudio-2.0.pc pointing at a removed /usr/local prefix is common and would otherwise configure fine and then fail to compile.

Using it #

add_subdirectory(path/to/flutter_voice_audio)
target_link_libraries(my_app PRIVATE voice_audio::voice_audio)

Recording a memo and playing it back — no PCM anywhere in it:

#include "voice_audio/fvm_api.h"

fvm_core_set_log_level(FVM_LOG_INFO);
fvm_core_set_log_callback(&MySink, nullptr);

fvm_core_init(nullptr);

// The recorder owns the microphone while it runs.
fvm_rec_start();
// ... the user talks ...
fvm_rec_stop();

unsigned size = 0;
fvm_rec_copy_data(nullptr, &size);           // a null buffer asks for the size
std::vector<char> memo(size);
fvm_rec_copy_data(memo.data(), &size);       // this is what goes in the database

int milliseconds = 0;
fvm_opus_duration(memo.data(), size, &milliseconds);

fvm_play_set_device_index(FVM_DEFAULT_DEVICE_ID_PUBLIC);
fvm_play_opus_start(memo.data(), size, 0, /*tag*/ 1);
// FVM_EVENT_PLAYER_FINISHED arrives through fvm_core_next_event_blob() when it drains.

fvm_opus_export_ogg(memo.data(), size, "memo.opus");   // to share it outside the app

// Or keep it as Ogg Opus in memory - for a host that stores recordings in a database and
// does not want a decrypted copy passing through the filesystem.
unsigned oggSize = 0;
fvm_opus_export_ogg_buffer(memo.data(), size, nullptr, &oggSize);
std::vector<unsigned char> ogg(oggSize);
fvm_opus_export_ogg_buffer(memo.data(), size, ogg.data(), &oggSize);

// And back: an imported file plays, seeks and reports its duration like any other recording.
unsigned backSize = 0;
fvm_opus_import_ogg(ogg.data(), oggSize, nullptr, &backSize);

fvm_core_shutdown();

The raw path is still there for anything that is not a memo — device enumeration, volume, and a capture loop the host drives itself:

int count = 0;
fvm_sink_get_count(&count);
for (int i = 0; i < count; ++i) {
    char name[256]; int size = 256;
    fvm_sink_get_name(i, name, &size);
}

fvm_sink_set_callback(&FeedSpeaker);   // called on the realtime speaker thread

// -1 means "the system default". Set it explicitly: a speaker opened without this
// goes to the null device on the PortAudio backends.
fvm_sink_set_device_id(FVM_DEFAULT_DEVICE_ID_PUBLIC);
fvm_source_open();
fvm_sink_open();
// ... fvm_source_read() in a loop ...
fvm_sink_close();
fvm_source_close();

Every entry point returns FVM_OK, FVM_BAD_PARAM or FVM_FAILED; on failure fvm_core_last_error_code() and fvm_core_last_error_message() describe what happened on the calling thread. The full API is documented in include/voice_audio/fvm_api.h.

Try it #

cmake -S . -B build -DVOICE_AUDIO_EXAMPLES=ON && cmake --build build -j
./build/examples/voice_audio_probe             # list configuration and devices
./build/examples/voice_audio_probe --loop 5    # echo the microphone to the speaker as PCM
./build/examples/voice_audio_probe --memo 5    # record an Opus memo and play it back
./build/examples/voice_audio_probe --memo 5 --export memo.opus

--memo prints what the recording cost against what the same audio would have cost as PCM.

And the tests, which need no audio hardware:

cmake -S . -B build-null -DVOICE_AUDIO_NULL_BACKEND=ON -DVOICE_AUDIO_TESTS=ON
cmake --build build-null -j && ctest --test-dir build-null --output-on-failure
python3 tool/check_backend_conformance.py

Threading #

• Speaker_*, Microphone_*, Player_* and Recorder_* are synchronous; drive them from one control thread.
• Recorder_* owns a thread and the microphone. It pulls capture buffers exactly the way a host would, through fvm_source_read(), so the capture filters run first. Do not drive that call yourself while a recording is running — the two would race for buffers.
• Playback decodes on the speaker's realtime thread. OpusDataReader allocates and frees nothing on that path; opus_decode() does not allocate either.
• FvmSinkProc and FvmPanicProc run on backend-owned realtime threads. Do not block, allocate or take locks held by your control thread inside them.
• The log callback can also fire from those threads.
• Error state is per-thread, so check fvm_core_last_error_message() on the thread that failed.

Layout #

include/voice_audio/     The module C surface — fvm_api.h and fvm_config.h
src/audio/               Engine core; no platform macros
  backend/               The device abstraction and the null backend
  android/ ios/ windows/ portaudio/   One backend each
src/codec/               Opus encoder, decoder, container, Ogg import and export
src/wav/                 WAV reader / writer - debug dumps only, not a storage format
src/resampler/           C++ wrapper over the vendored Speex resampler
src/util/                Utility layer - fvm::util
src/log/                 The built-in logger
src/common/              Result codes and per-thread error state
src/ffi/                 voice_audio_ffi.h - the pure C surface the Dart bindings use
lib/                     The Dart API; lib/src/bindings.g.dart is generated by ffigen
android/ ios/ macos/ linux/ windows/   Flutter plugin build wiring, one file each
third_party/webrtc/      Vendored WebRTC AGC and noise suppressor (~1.3 MB, builds as webrtc_dsp)
third_party/speex/       Vendored Speex resampler (BSD-3)
third_party/portaudio/   PortAudio header plus prebuilt fallback archives
native_example/          voice_audio_probe
native_test/             Unit tests, built against the null backend
tool/                    check_backend_conformance.py, build_apple.sh

About 26k lines across 120 files, plus the vendored third-party trees. The PCM-buffer and DTMF readers went with the player's old sources; what is left reads WAV.

Formatting #

.clang-format is the house style: 4-space indent, no tabs, Allman braces. ColumnLimit is 0, so clang-format fixes indentation and spacing but never re-wraps a line — your line breaks survive.

clang-format -i $(git ls-files 'src/*.h' 'src/*.cpp' 'src/*.c' 'src/*.mm' \
                               'include/*.h' 'native_test/*.h' 'native_test/*.cpp' 'native_example/*.cpp')

third_party/ and src/audio/agc/ are vendored and carry their own .clang-format with DisableFormat: true, so a run over the whole tree leaves them as upstream has them.

Architecture #

Every platform difference lives behind one of five interfaces in src/audio/backend/fvm_backend.h:

class Sink        { /* playback device  */ };
class Source      { /* capture device   */ };
class DeviceEnumerator { /* names, UTF-8     */ };
class VolumeControl    { /* volume           */ };
class Backend     { /* the factory      */ };

Backend& PlatformBackend();

PlatformBackend() is declared once and defined once per backend translation unit, so CMake's per-platform source lists remain the only place the choice is made — there is no #ifdef in a factory either. Linking two backends into one library is a duplicate-symbol error rather than a silent wrong pick.

fvm_core.cpp holds std::unique_ptr<Sink> and friends and contains no TARGET_* macro except around the Windows-only device hot-plug poll and the Android-only fvm_core_set_stream_type, both of which the public header also declares conditionally. It went from 62 platform conditionals to six; the whole tree went from about 310 to roughly 70, and almost all of the remainder is in src/util, which has not been cleaned up yet.

Divergent methods are base-class virtuals whose defaults are correct, not merely safe: a backend that cannot detect failure honestly reports healthy, Restart is close-then-open, and a stream-type hint is genuinely a no-op where the platform has one fixed stream.

tool/check_backend_conformance.py parses the interfaces and every class X final : public IY and reports missing or wrongly-qualified overrides. Run it after touching an interface — it is the only conformance check that covers the platforms this tree cannot build.

Where the platform code lives #

Known rough edges #

• fvm_core_set_config() takes a C++ reference. Inherited from the original header. Fine from C++, not FFI-friendly — the Flutter binding will want a thin C shim.
• FvmConfig::samplerate can be set to rates Opus cannot encode. validate() accepts any multiple of 8000 except 24000, which lets 32000 and 40000 through; Opus takes 8000, 12000, 16000, 24000 and 48000. fvm_rec_start() fails cleanly with a message rather than producing a broken stream, but the two constraints disagree and neither is wrong on its own.
• Playback resamples through the decoder, not the resampler. OpusStreamReader creates its decoder at the speaker's rate, so a recording made at another rate plays at the right pitch for free. src/resampler/ is therefore unused on this path — it is still only wired into the Android backends.
• A recording lives entirely in memory until fvm_rec_stop(). At 16 kbps that is about 120 KB a minute, so an hour is 7 MB — fine for memos, wrong for anything unbounded. Copying the growing blob periodically works (fvm_rec_copy_data is safe mid-recording and the header is kept truthful frame by frame) but there is no incremental drain that hands over only the new frames.
• The WebRTC AGC and noise suppressor are not optional at build time. fvm_agc_filter.cpp and fvm_denoise.cpp include their headers unconditionally. Both can be switched off at runtime through FvmConfig::use_agc and use_ns, but compiling them out would need those call sites stubbed.
• Noise suppression needs 16 kHz or more. The suppressor has no 8 kHz mode, so at samplerate = 8000 it stays off and says so once in the log. Gain control works at every rate the config accepts.
• There is no echo cancellation at all, by design — this module records and plays back rather than doing duplex. Nothing in the vendored subset cancels echo: aec3 and aecm were dropped along with the rest of the audio processing module. Duplex would mean vendoring a canceller again and restoring a far-end feed; AudioProcessingModule::PostProcess used to be where the speaker signal went in.
• Both device lists enumerate every PortAudio device, not just those with channels in the right direction, so a playback-only endpoint also appears as a capture device. A device's index is its identifier, so filtering would renumber everything and change what a stored device ID refers to. Left deliberately.
• PortAudioSink defaults to FVM_VIRTUAL_DEVICE_ID, so a speaker opened without a prior fvm_sink_set_device_id goes to the null device rather than the system default.
• A crash-reporter hook lingers in fvm_helper.cpp behind USE_CRASH_REPORTER, inert and Windows-only. It does not belong in an audio module and can be deleted.
• src/util has not been through the platform-macro cleanup. fvm_helper.cpp alone still has 39 conditionals, and fvm_types.h is a Win32-emulation shim that defines DWORD, SOCKET and closesocket for an audio module that needs none of them.

Next step #

FLUTTER_PLAN.md is the implementation plan for wrapping this module as a Flutter package — what the investigation found, the architecture, phases and open questions.

Verification status #

The exported .opus files were checked against opusinfo, opusdec and ffprobe: pre-skip, channel count, original rate and packet duration all read back correctly, and the playback length matches the recording minus the encoder's lookahead.

The iOS and macOS backends were written against the interfaces and reviewed, not built. tool/check_backend_conformance.py confirms every backend class implements every pure virtual, which is the failure mode that would otherwise only surface on a real toolchain — but it is not a compiler. Build them on a real host before trusting them, which is what declaring the platforms waits on. Three genuine compile errors were already found in that code by inspection alone; assume there are more.

Capture was exercised only against silence: neither the Linux box this was built on nor the Android emulator has anything speaking into its microphone. The codec itself is covered with a real signal — native_test/test_opus_codec.cpp puts a 440 Hz tone through the encoder and checks pitch, level, seek accuracy and the waveform — but record yourself talking before trusting the audio quality end to end.

The podspecs reference the MIT LICENSE at the root of this repository, which is there. They have still never been through pod lib lint, which is part of why iOS and macOS are not declared platforms yet.