OAIon

A lightweight, production-ready OAI-PMH provider built with TypeScript, Fastify, and SQLite.

Overview

OAIon is a modern TypeScript implementation of an OAI-PMH v2.0 data provider. It exposes a fully compliant OAI-PMH endpoint for metadata harvesting, a Bearer-token-protected REST API for record management, and an optional background sync worker that ingests records from a file-system source.

Designed to be simple to deploy, easy to configure, and straightforward to integrate into existing repository workflows.

Features

✅ Full OAI-PMH v2.0 compliance — Identify, ListSets, ListMetadataFormats, ListIdentifiers, ListRecords, GetRecord
📄 Resumption token pagination — configurable page sizes with automatic token expiry and cleanup
🗑️ Deleted record tracking — supports persistent, transient, and no deletion policies
🔒 Bearer-token protected REST API — create, update, delete, and query records without touching the database
🔄 Background sync worker — poll a local filesystem or Amazon S3 bucket (via FlyDrive) and upsert records automatically
⚙️ Database-backed runtime configuration — key settings overridable at runtime via the API and stored in SQLite
📋 Swagger / OpenAPI UI — interactive API documentation at /api-docs
🐳 Docker-first — multi-stage Dockerfile and Docker Compose for development and production
🧪 Vitest test suite — unit and integration tests for OAI behaviour, REST API, and sync worker

Tech Stack

Layer	Technology
Language	TypeScript 5
Runtime	Node.js 20+
HTTP Server	Fastify 5
Database	SQLite (`better-sqlite3`)
Schema Validation	Zod
API Docs	`@fastify/swagger` + Swagger UI
File Sync Driver	`@slynova/flydrive` + `@slynova/flydrive-s3`
Testing	Vitest
Container	Docker + Docker Compose

Quick Start

Prerequisites

Node.js 24.x+
npm 10+
Docker Desktop / OrbStack (for container workflows)

Local Development

# Clone and install
git clone <repo-url>
cd oaion
npm ci

# Set up environment
cp .env.example .env
# Edit .env — at minimum, set REST_API_TOKEN to something other than 'change-me'

# Start in watch mode
npm run dev

The server starts at http://localhost:3000.

Endpoint	Description
`GET /health`	Health check
`GET\|POST /oai`	OAI-PMH endpoint
`GET\|POST\|DELETE /api/records`	REST API (requires Bearer token)
`GET /api-docs`	Swagger UI

Verify it's running

# Health check
curl -sS http://localhost:3000/health

# OAI Identify
curl -sS "http://localhost:3000/oai?verb=Identify"

# List available metadata formats
curl -sS "http://localhost:3000/oai?verb=ListMetadataFormats"

Docker

Build and run

docker build -t oaion .
docker run -p 3000:3000 --env-file .env -v $(pwd)/data:/app/data oaion

Docker Compose (development)

The development compose stack enables the flydrive sync worker and uses a reduced page size for easy pagination testing:

npm run dev:run
# equivalent: docker compose -f support/development/docker-compose.yml up --build

Generate synthetic test data for local development:

./support/development/generate-test-data.sh

Ingesting Records

OAIon is designed to be populated easily from external systems. There are two primary methods for ingesting records into the provider:

1. REST API (Push)

Using the Bearer-token protected /api/records endpoints, you can push records directly into OAIon. This is ideal when integrating OAIon with an existing repository or metadata management system that can trigger HTTP webhooks or API calls when records are created, updated, or deleted.

2. Sync Worker (Pull)

OAIon includes a background sync worker mechanism that periodically polls a source and upserts records automatically.

FlyDrive Driver: The built-in flydrive driver can sync records from a local directory tree or an Amazon S3 bucket.
Extensible: The sync architecture is modular. You can easily implement custom SyncDriver classes to pull records from other APIs, databases, or object stores.

Configuration

OAIon is configured entirely via environment variables. Copy .env.example to .env and adjust as needed.

Required for production

Variable	Description
`REST_API_TOKEN`	Bearer token for `/api/` — must be changed from default*
`BASE_URL`	Publicly reachable OAI endpoint URL (e.g. `https://repo.example.org/oai`)
`ADMIN_EMAIL`	Administrator contact email (appears in OAI `Identify` response)
`DATABASE_URL`	SQLite database path (e.g. `file:/var/lib/oaion/oaion.db`)

Full reference

Variable	Default	Description
`PORT`	`3000`	HTTP listen port
`HOST`	`0.0.0.0`	HTTP listen interface
`DATABASE_URL`	`file:./data/oaion.db`	SQLite database location
`REST_API_TOKEN`	`change-me`	Bearer auth secret for REST API
`REPOSITORY_NAME`	`OAIon TypeScript Provider`	Repository name in OAI `Identify`
`REPOSITORY_IDENTIFIER`	`oai:example.org:oaion`	OAI repository identifier
`BASE_URL`	`http://localhost:3000/oai`	OAI request base URL
`ADMIN_EMAIL`	`admin@example.org`	OAI admin contact email
`EARLIEST_DATESTAMP`	`2000-01-01T00:00:00Z`	OAI earliest datestamp
`DELETED_RECORD_POLICY`	`persistent`	`no`, `transient`, or `persistent`
`OAI_PAGE_SIZE`	`100`	Default page size for OAI list verbs
`OAI_PAGE_SIZE_RECORDS`	(inherits)	Override page size for `ListRecords`
`OAI_PAGE_SIZE_IDENTIFIERS`	(inherits)	Override page size for `ListIdentifiers`
`OAI_PAGE_SIZE_SETS`	(inherits)	Override page size for `ListSets`
`TOKEN_EXPIRY_SECONDS`	`86400`	Resumption token lifetime in seconds
`SUPPORTED_FORMATS`	`oai_dc,rif`	Comma-separated supported metadata prefixes
`SETS_ENABLED`	`true`	Enable set filtering; `false` returns `noSetHierarchy`
`IDENTIFY_GRANULARITY`	`YYYY-MM-DDThh:mm:ssZ`	Datestamp granularity in `Identify` response
`FASTIFY_LOGGER`	`true`	Enable/disable Fastify request logging
`SYNC_ENABLED`	`false`	Enable background sync worker
`SYNC_DRIVER`	`flydrive`	Sync source driver (`flydrive` only currently)
`SYNC_FLYDRIVE_STORAGE_DRIVER`	`local`	Storage backend: `local` or `s3`
`SYNC_FLYDRIVE_ROOT`	(required if `local`)	Root directory for local filesystem sync
`SYNC_FLYDRIVE_S3_BUCKET`	(required if `s3`)	S3 bucket name
`SYNC_FLYDRIVE_S3_KEY`	(optional)	AWS access key — omit to use instance profile
`SYNC_FLYDRIVE_S3_SECRET`	(optional)	AWS secret key — omit to use instance profile
`SYNC_FLYDRIVE_S3_REGION`	(optional)	AWS region (e.g. `ap-southeast-2`)
`SYNC_FLYDRIVE_S3_ENDPOINT`	(optional)	Custom S3 endpoint (MinIO, DigitalOcean Spaces, etc.)
`SYNC_POLL_INTERVAL_SECONDS`	`60`	Sync polling interval
`SYNC_STARTUP_POLLING_ENABLED`	`true`	Sync on startup; blocks HTTP only when no records exist
`SYNC_MAX_CONCURRENCY`	`4`	Parallel sync upsert workers
`SYNC_BATCH_COMMIT_SIZE`	`100`	Items committed per batch
`SYNC_QUEUE_BACKPRESSURE`	`1000`	Max items staged in each processing window
`SYNC_RETRY_LIMITS`	`3`	Retry attempts for failing sync operations

Note: REPOSITORY_NAME, REPOSITORY_IDENTIFIER, BASE_URL, ADMIN_EMAIL, and EARLIEST_DATESTAMP can also be overridden at runtime via the REST API and are persisted in the database.

REST API

The REST API is protected by a Bearer token (Authorization: Bearer <REST_API_TOKEN>).

Records

# Create or update a record
curl -X POST http://localhost:3000/api/records \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "identifier": "oai:example.org:item1",
    "metadataPrefix": "oai_dc",
    "xmlContent": "<dc:title>Example Record</dc:title>",
    "setSpec": ["my-set"]
  }'

# List records
curl http://localhost:3000/api/records \
  -H "Authorization: Bearer <token>"

# Delete a record
curl -X DELETE http://localhost:3000/api/records/oai:example.org:item1 \
  -H "Authorization: Bearer <token>"

Configuration

Runtime-overridable settings can be read and updated via the API:

# Get current config
curl http://localhost:3000/api/config \
  -H "Authorization: Bearer <token>"

# Update a setting
curl -X PATCH http://localhost:3000/api/config \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{ "REPOSITORY_NAME": "My Updated Repository" }'

Full API documentation is available interactively at http://localhost:3000/api-docs.

Testing

# Run unit and integration tests
npm test

# Run Bruno-based OAI-PMH API tests (requires Docker)
npm run test:bruno:oaipmh

# Clean up Bruno test containers
npm run test:bruno:oaipmh:clean

Building for Production

npm ci
npm run build
node dist/src/index.js

systemd service example

[Unit]
Description=OAIon OAI-PMH Provider
After=network.target

[Service]
Type=simple
User=oaion
Group=oaion
WorkingDirectory=/opt/oaion
EnvironmentFile=/etc/oaion/oaion.env
ExecStart=/usr/bin/node /opt/oaion/dist/src/index.js
Restart=always
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/oaion

[Install]
WantedBy=multi-user.target

Security

Change REST_API_TOKEN before any production or internet-facing deployment. The default value change-me is intentionally insecure.
Use a 32+ byte random secret (e.g. openssl rand -hex 32).
Restrict /api/* at the network level where possible (private network, IP allowlist, or VPN).
Terminate TLS at a reverse proxy (Nginx, HAProxy, ALB) — do not expose plain HTTP publicly.
Run as a dedicated non-root service user with write access only to the data directory.
Protect your .env file: chmod 600 .env.

Project Structure

src/
├── api/          # REST API routes and auth middleware
├── oai/          # OAI-PMH verb handlers, XML builders, validation
├── sync/         # Background sync worker and driver interface
├── storage/      # File storage abstraction
├── config.ts     # Environment config loading and validation (Zod)
├── db.ts         # SQLite store factory
├── server.ts     # Fastify app builder
├── store.ts      # SQLite data access layer
└── index.ts      # Application entrypoint

support/
├── development/  # Docker Compose and test data tooling
├── testing/      # Bruno API test collections
├── schemas/      # OAI metadata schemas
└── wiki/         # Extended documentation (Development Guide, Sysadmin Guide)

tests/            # Vitest test suites

Documentation

Extended documentation lives in support/wiki/:

Ingesting Records — REST API (push) and FlyDrive Sync Worker (pull) methods
Development Guide — local setup, test workflows, common tasks
System Administration & Configuration — production deployment, tuning, backup, and troubleshooting

Contributing

Fork the repository and create a feature branch.
Install dependencies: npm ci
Make your changes with appropriate test coverage.
Before opening a PR, ensure:
- npm test passes
- npm run build succeeds
- New behaviour is covered by tests
- Documentation is updated if config or behaviour changes
- No hardcoded secrets are committed

License

See LICENSE for details.

Name		Name	Last commit message	Last commit date
.circleci		.circleci
public		public
specs/oaipmh-integration-tests/bruno		specs/oaipmh-integration-tests/bruno
src		src
support		support
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
.prettierrc		.prettierrc
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
eslint.config.mjs		eslint.config.mjs
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

Navigation Menu

Uh oh!

Repository files navigation

OAIon

Overview

Features

Tech Stack

Quick Start

Prerequisites

Local Development

Verify it's running

Docker

Build and run

Docker Compose (development)

Ingesting Records

1. REST API (Push)

2. Sync Worker (Pull)

Configuration

Required for production

Full reference

REST API

Records

Configuration

Testing

Building for Production

systemd service example

Security

Project Structure

Documentation

Contributing

License

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

OAIon

Overview

Features

Tech Stack

Quick Start

Prerequisites

Local Development

Verify it's running

Docker

Build and run

Docker Compose (development)

Ingesting Records

1. REST API (Push)

2. Sync Worker (Pull)

Configuration

Required for production

Full reference

REST API

Records

Configuration

Testing

Building for Production

systemd service example

Security

Project Structure

Documentation

Contributing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages