Benchmarks

What is it?

How Wolbarg v0.4.0 measures startup, batch/bulk insert, recall latency, embedding-cache behavior, upsert/dedupe correctness, subscribe() delivery, and concurrency on SQLite and PostgreSQL.

Two embedding modes — do not mix them:

Interactive charts: /benchmarks.

Published 0.4 artifacts

Public suite repo: wolbarg-benchmarks.

Why it exists

Vendor graphs without methodology mislead. These docs explain what was measured so you can compare fairly and reproduce locally — especially the difference between mock stress and live API spots, and between same-process vs multi-process concurrency.

Headlines — v0.4.0 (2026-07-18)

Environment: Node v24.13.1 · win32/arm64 · 8 CPUs · mock embeddings · suite v4-stress.

*Postgres skips SQLite-only checks (schema meta table probe, SQLite EventDatabase telemetry file, file checkpoints, export/import bundles).

Embedding cache microbench

Source: embedding-cache.json · runner benchmark/embedding-cache-bench.ts.

Multi-process SQLite concurrency

Shared file, separate OS processes, BEGIN IMMEDIATE + busy retry (v0.4):

Source: multiprocess-concurrency.json · runner benchmark/multiprocess-levels.ts.

Feature coverage in the v4 suite

Beyond raw speed, the 0.4 suite asserts product correctness:

Methodology

Mock vs LIVE (read this)

Primary stress and push-to-failure concurrency use a local mock OpenAI-compatible embedding/LLM server. Live OpenAI is not used for failure ramps because API rate limits and quota errors would dominate long before SQLite or PostgreSQL contention, masking true Wolbarg/storage breaking points.

A separate LIVE spot suite (npm run benchmark:live) reports real-network latency for representative paths. It does not ramp concurrency to failure.

Failure criteria (breaking ramps)

When using breaking / brutal modes, a concurrency level fails when:

• errorRate > 1%, or
• p95 latency > 5s, or
• a hard integrity/exception failure (duplicate IDs, crash, etc.)

Reports record lastHealthyLevel and breakingLevel with reason (error_rate | p95_sla | exception | integrity | cap).

Storage matrix

cd benchmark
npm run postgres:up
npx tsx v4-stress.ts          # or your package script wrapping it
npx tsx embedding-cache-bench.ts
npx tsx multiprocess-levels.ts

Vector backend caveat: benchmarks assume pgvector/HNSW is enabled for Postgres and sqlite-vec is available for SQLite. When extensions are missing/unloadable, Wolbarg falls back to exact cosine scanning, which can change recall latency substantially.

Commands (legacy + 0.4 runners)

Metrics glossary

Hardware

Always read the environment block in the artifact you cite (Node, CPU, RAM, mode, backends, dims). The 0.4 publish used Node v24.13.1 · win32/arm64 · 8 CPUs.

Dataset

Synthetic memories with fixed templates. Mock dims typically 384; live typically 1536 (text-embedding-3-small). Labels (100, 1k, 2k) are memory counts.

Reproducibility

cd benchmark
npm install
cp .env.example .env   # DATABASE_URL for postgres; API keys for --live
npx tsx v4-stress.ts
npx tsx embedding-cache-bench.ts
npx tsx multiprocess-levels.ts

Prefer recording: date, SDK version (wolbarg@0.4.0), Node version, CPU/RAM, mode (mock/live), backends, and git SHA.

Interpretation

• Startup ms — agents open memory without multi-second cold starts
• Bulk insert vs recall — know when Postgres/pgvector or sharding strategies matter (Postgres recall p95 in this run is network/engine dominated vs SQLite in-process)
• Same-process vs multi-process — multi-process SQLite serializes; throughput plateaus while p95 climbs (expected)
• Cache reduction — measure provider calls, not only ms, when arguing cost
• Mock ≠ hosted SaaS — do not compare mock embed timings to managed GPU indexes
• Storage (mock) ≠ LIVE — never mix the two suites in one comparison cell
• Competitor accuracy evals ≠ storage ops/sec — see the policy on /benchmarks

Related pages

• Performance
• Concurrency
• Embedding cache
• Architecture
• SQLite · PostgreSQL
• Live page: /benchmarks
• What's New