AppView Plans
What is the Layers AppView?
An ATProto appview is a read-only indexing service that subscribes to the protocol's firehose, extracts records it cares about, and serves them back through query endpoints. The appview never owns user data; all pub.layers.* records live in user-controlled Personal Data Servers (PDSes). If the appview's database disappears, nothing is lost because every record can be rebuilt from the firehose.
The Layers appview indexes all 26 pub.layers.* record types, maintains full-text and faceted search indexes, builds a knowledge graph from cross-references, and exposes the results through XRPC and REST APIs.
Architecture at a Glance
The appview runs as two separate processes, following Chive's two-process architecture:
- API Server (
src/index.ts): serves XRPC and REST queries, starts scheduled jobs - Firehose Indexer (
src/indexer.ts): consumes the AT Protocol firehose, writes to all storage backends
Record Type Coverage Matrix
Every pub.layers.* record type is indexed into one or more storage backends. PostgreSQL is the source of truth for all types. Elasticsearch and Neo4j are populated selectively based on query requirements.
| Record Type | PG | ES | Neo4j | Notes |
|---|---|---|---|---|
expression.expression | yes | yes | yes | Full-text search on text, graph node for cross-refs |
segmentation.segmentation | yes | — | — | Token arrays stored in PG, queried via expression |
annotation.annotationLayer | yes | yes | yes | Annotations normalized to rows in PG, nested in ES for faceted search, per-annotation nodes in Neo4j |
annotation.clusterSet | yes | — | yes | Cluster membership edges in Neo4j |
ontology.ontology | yes | yes | — | Searchable by domain and name |
ontology.typeDef | yes | yes | yes | Type hierarchy edges in Neo4j |
corpus.corpus | yes | yes | — | Searchable by name, language, license |
corpus.membership | yes | — | yes | Corpus-expression edges in Neo4j |
resource.entry | yes | yes | — | Lexical search on lemma and form |
resource.collection | yes | yes | — | Searchable by name |
resource.collectionMembership | yes | — | — | Join table |
resource.template | yes | — | — | Template storage |
resource.filling | yes | — | — | Filling storage |
resource.templateComposition | yes | — | — | Composition storage |
judgment.experimentDef | yes | yes | — | Searchable by measure/task type |
judgment.judgmentSet | yes | — | — | Linked to experiment |
judgment.agreementReport | yes | — | — | Linked to experiment |
alignment.alignment | yes | — | yes | Source-target edges in Neo4j |
graph.graphNode | yes | yes | yes | Primary Neo4j content, ES for node search |
graph.graphEdge | yes | — | yes | Primary Neo4j content |
graph.graphEdgeSet | yes | — | yes | Expanded to individual edges in Neo4j |
persona.persona | yes | yes | — | Searchable by domain and kind |
media.media | yes | yes | — | Searchable by modality |
eprint.eprint | yes | yes | — | Searchable by identifier, title |
eprint.dataLink | yes | — | yes | Eprint-corpus edges in Neo4j |
changelog.entry | yes | yes | — | Searchable by subject, collection, version |
Source Directory Layout
The src/ directory follows Chive's layered + feature-based hybrid organization:
src/
├── api/ # Hono handlers (xrpc/, rest/), middleware, openapi
├── atproto/ # AT Protocol client, repository access, errors
├── auth/ # OAuth, JWT, DID, MFA, WebAuthn, zero-trust, authorization, scopes, session
├── config/ # Configuration management
├── jobs/ # Scheduled interval-based jobs (staleness, reconciliation)
├── lexicons/ # Generated TypeScript types from lexicon JSON
├── observability/ # Logger, telemetry, tracer, metrics-exporter
├── plugins/ # core/, builtin/, sandbox/
├── services/ # Business logic (indexing/, search/, enrichment/, import/, etc.)
├── storage/ # postgresql/, elasticsearch/, neo4j/, redis/ adapters
├── types/ # models, interfaces, errors.ts, result.ts
├── utils/ # Shared utilities
├── workers/ # BullMQ worker implementations
├── index.ts # API server entry point
└── indexer.ts # Firehose consumer entry point
Design Goals
Follow Chive's Proven Architecture
The Layers appview follows Chive's production architecture closely. Chive is a running ATProto appview for scholarly eprints that has already solved the hard infrastructure problems (firehose subscription with cursor-based resumption, multi-database indexing, dual XRPC/REST APIs, BullMQ job queues, and Kubernetes deployment). Layers adopts the same technology stack, patterns, and operational practices, including the two-process split, layered directory structure, storage adapter interfaces, and dependency injection via tsyringe.
Type-Safe Error Handling
All fallible operations return Result<T, LayersError> instead of throwing exceptions, following Chive's Result<T, E> pattern. The LayersError hierarchy provides typed error classification:
ComplianceError— ATProto compliance violations (write to PDS, blob storage, non-rebuildable state)NotFoundError,ValidationError— client errorsAuthenticationError,AuthorizationError,RateLimitError— auth/access errorsDatabaseError,ServiceUnavailableError— infrastructure errorsPluginError,SandboxViolationError— plugin system errors
See Technology Stack for the full error handling architecture.
Adapt for Layers-Specific Complexity
Where Layers diverges from Chive:
- 26 record types (vs Chive's ~7) require dependency-aware ingestion ordering and more sophisticated queue topology
- Discriminated annotation model (kind/subkind/formalism) requires three-dimensional faceted search in Elasticsearch
- Dense cross-referencing (sourceUrl, sourceRef, eprintRef, graphEdge targets, knowledgeRefs) requires a dedicated cross-reference index and Neo4j edge graph
- Embedded annotation arrays (an
annotationLayercan contain hundreds of individual annotations) require careful normalization decisions per storage backend - Format import pipeline (CoNLL, BRAT, ELAN, TEI, etc.) extends the plugin system beyond Chive's harvester model
- Annotation workflow RBAC (annotator, adjudicator, corpus manager) requires more granular authorization than Chive's publish/read model
Bleeding-Edge Infrastructure
The appview targets 2026 best practices:
- OpenTelemetry 1.x stable for unified observability (traces, metrics, logs)
- Distroless container images (
gcr.io/distroless/nodejs22-debian12) for minimal attack surface - Sigstore cosign for container image signing and supply chain verification
- GitOps via ArgoCD/Flux for declarative, auditable deployments
- Zero-trust security with mTLS between services, DPoP token binding, and passkeys-first authentication
- WebTransport monitored as a future firehose transport alternative (multiplexed streams, better congestion control)
Page Directory
| Page | Content |
|---|---|
| Technology Stack | Runtime, frameworks, databases, and tooling with version pins and decision rationale |
| Database Design | PostgreSQL schema, Elasticsearch mappings, Neo4j graph model, Redis data model |
| Firehose Ingestion | Subscription, filtering, queue topology, dependency ordering, dead letter queue |
| API Design | XRPC + REST endpoints, search, composite queries, OpenAPI |
| Indexing Strategy | Per-record-type indexing, annotation normalization, cross-reference extraction |
| Query and Discovery | Use cases, query patterns, graph traversal, caching |
| Authentication | OAuth 2.0, JWT sessions, RBAC, MFA |
| Background Jobs | Workers, enrichment, format import, maintenance |
| Observability | Logging, tracing, metrics, dashboards, alerting |
| Deployment | Docker, Kubernetes, database deployment, CI/CD, backup |
| Testing Strategy | Unit, integration, compliance, E2E, performance |
| Plugin System | Sandboxed plugins, format importers, harvesters |
See Also
- Introduction for what Layers is and why it exists
- Lexicon Overview for the complete record type inventory and dependency graph
- ATProto Ecosystem Integration for how Layers records interlink with other ATProto applications