Turning Scrappy Into a Full Application Stack

After building the authentication API, the next step for Scrappy was turning the project into a real application stack.

The auth service gave the system a way to know who the user is. But identity by itself is only one part of the application. Scrappy also needs a frontend where users can interact with their documents, a backend that can coordinate uploads and chat requests, a database that can store durable application state, a cache for fast temporary data, and a retrieval layer that can support semantic search.

This part of the build was about connecting those pieces into a system that can grow.

Scrappy is not just a chatbot interface. The goal is to build a personal AI-enabled document system that can accept files, track ingestion jobs, process content in the background, store chunks and embeddings, and eventually answer questions using retrieval-augmented generation.

That requires more than one service. It requires clear boundaries between the frontend, backend, database, cache, worker system, and vector search layer.

The application shape

At a high level, Scrappy is made up of a few core parts:

  1. A frontend application for users
  2. A FastAPI backend for API requests and orchestration
  3. PostgreSQL for durable application data
  4. Redis for caching, rate limiting, and task infrastructure
  5. pgvector for storing embeddings and supporting semantic search
  6. Background workers for document processing and ingestion

The important decision was to avoid putting every responsibility into one place.

The frontend should focus on user experience. The backend should coordinate requests and enforce permissions. Postgres should store durable state. Redis should handle short-lived data and fast coordination. The vector layer should support retrieval over processed document chunks.

Each part has a job.

The frontend

The frontend is the part of Scrappy users actually touch.

Its job is to make the system feel understandable. Users should be able to sign in, upload documents, see processing status, start conversations, and understand whether their files are ready to query.

For this project, the frontend is not just a static UI. It needs to reflect backend state clearly.

A document may be uploaded but not processed yet. An ingestion job may be queued, running, failed, or complete. A chat query may need to stream a response while retrieval is happening in the background. Those states need to be represented in the interface without exposing too much of the internal complexity.

The frontend depends on the backend for the source of truth, but it is responsible for making the workflow feel simple.

The backend as the control plane

The FastAPI backend acts as Scrappy’s control plane.

It should not do every heavy task inline. Instead, it coordinates the application flow: authenticating requests, registering uploads, storing metadata, submitting jobs, enforcing permissions, and exposing read APIs for document and job status.

The backend owns the application contract.

Some of the core backend domains include:

  • Auth and session management
  • Document upload registration
  • Metadata persistence
  • Chat and query orchestration
  • Job submission
  • Permissions enforcement
  • Usage and rate limiting
  • Event publishing
  • Read APIs for job and document status

That means the backend is responsible for deciding what should happen, but not necessarily doing all of the expensive work itself.

For example, when a user uploads a document, the API should register the document, persist the metadata, store the file in object storage, and submit an ingestion job. The actual parsing, chunking, and embedding work should happen in a background worker.

That separation keeps the API responsive and makes the system easier to debug.

Core API areas

The backend is organized around the main areas of the product.

auth
users
documents
uploads
ingestion
chat
retrieval
jobs
workspaces
admin

Those modules map to the way the application is expected to grow.

The auth and users modules handle identity. The documents and uploads modules handle file registration and document lifecycle. The ingestion and jobs modules track background processing. The chat and retrieval modules support the RAG workflow. Workspaces and admin modules leave room for future multi-user or team-based functionality.

Some of the key API endpoints include:

POST /auth/login
POST /auth/logout
GET  /auth/me

POST   /documents/upload
GET    /documents
GET    /documents/{id}
DELETE /documents/{id}
POST   /documents/{id}/reprocess

GET /jobs/{id}
GET /documents/{id}/status

POST /chat/query
POST /chat/query/stream

This gives the frontend a clean set of actions: authenticate, upload, check status, and query.

Postgres as the durable source of truth

PostgreSQL is the primary database for Scrappy.

Its job is to store durable, relational application data. That includes users, documents, document chunks, jobs, conversations, messages, query logs, and retrieval logs.

The database should not be used as a dumping ground for everything. Large raw files should live in object storage, not directly in Postgres. Redis should handle temporary state and caching. But when the system needs a reliable record of what happened, Postgres should be the source of truth.

For Scrappy, that means Postgres stores records like:

users
sessions
workspace_memberships

documents
document_chunks

jobs

conversations
messages
query_logs
retrieval_logs

The document records track ownership, file metadata, storage location, checksum, status, errors, and timestamps.

The chunk records track the processed text that comes out of ingestion. Each chunk can store its document relationship, chunk index, page number, text, token count, and embedding status.

The job records track background processing. That includes job type, status, payload, attempt count, scheduling time, start time, completion time, and any error messages.

This is what makes the application observable. If ingestion fails, the system should be able to tell what failed, when it failed, and why.

Redis for fast temporary state

Redis plays a different role.

Where Postgres is durable, Redis is fast and temporary. It is useful for session caching, rate limit counters, query result caching, retrieval caching, temporary upload state, and real-time status fanout.

Redis can also support Celery as a broker or result backend, depending on how the worker system is configured.

Good Redis use cases in Scrappy include:

Caching document readiness state Caching repeated retrieval results Tracking per-user chat throttles Storing temporary signed upload state Publishing job progress events Publishing document ingestion state changes

But Redis should not be the only source of truth for workflow state.

Redis pub/sub is ephemeral. If no subscriber is listening when an event is published, that event can be lost. That is fine for live notifications, but it is not enough for durable ingestion tracking.

For Scrappy, the split is simple:

Postgres stores durable workflow state. Redis supports fast temporary state. Redis pub/sub can fan out live updates. Celery workers handle durable background execution.

That keeps the system practical without making Redis responsible for data it should not own.

Adding vector search with pgvector

Scrappy also needs a way to search document content semantically.

That is where pgvector comes in.

The current direction is to store document chunks and embeddings in Postgres using pgvector. This keeps the first version of the stack simpler because it avoids adding a separate vector database too early.

The vector layer is responsible for:

Storing embeddings for processed chunks Running semantic similarity search Filtering results by user, workspace, document, document type, or tags Supporting future hybrid search with full-text ranking

A practical first version can store embeddings alongside chunk data or in a separate chunk_embeddings table.

chunk_embeddings
   id
   chunk_id
   document_id
   embedding
   model_name
   created_at

This gives Scrappy the ability to retrieve chunks that are semantically close to a user’s question.

But vector search is only part of retrieval.

Vector-only search can miss exact keywords, IDs, names, codes, or phrases. That is why the better long-term direction is hybrid retrieval: combine vector similarity with Postgres full-text search, merge the top results, and eventually add reranking before sending the final context to the LLM.

The build order I am aiming for is:

Set up pgvector Store chunk text and embeddings in Postgres Add Postgres full-text search on chunk text Implement hybrid retrieval Add reranking later

That gives Scrappy a strong retrieval foundation without adding unnecessary infrastructure too early.

How the pieces work together

Once these pieces are connected, the document flow starts to look like a real system.

User uploads a document
   |
Frontend sends upload request
   |
Backend authenticates user
   |
Backend stores file metadata in Postgres
   |
Backend stores raw file in object storage
   |
Backend submits ingestion job
   |
Worker parses and chunks document
   |
Worker stores chunks in Postgres
   |
Worker generates embeddings
   |
Worker stores vectors with pgvector
   |
Frontend reads job/document status from backend
   |
User can query the document

The chat flow builds on top of that.

txt``` User asks a question | Frontend sends query to backend | Backend verifies user permissions | Backend embeds the query | Backend retrieves relevant chunks | Backend sends context to the LLM | Backend streams or returns the answer | Query and retrieval logs are stored for debugging


This is the point where Scrappy starts to become more than a collection of tools. The frontend, backend, database, cache, and vector store each support a different part of the same workflow.

What I learned from this part of the build

The biggest lesson from this phase was that architecture is mostly about boundaries.

It is tempting to make the backend do everything. It can accept uploads, process files, generate embeddings, answer questions, cache results, and manage state all from one place. But that becomes hard to reason about quickly.

A cleaner design gives each part of the system a narrower responsibility.

The frontend presents the workflow. The backend coordinates the workflow. Postgres records the workflow. Redis speeds up temporary operations. Workers perform expensive tasks. pgvector supports retrieval.

That structure makes Scrappy easier to extend.

The first version does not need to be perfect. It just needs the right seams. Once the seams are in place, I can improve each layer independently: a better upload experience, more reliable ingestion jobs, stronger retrieval quality, better caching, and eventually a more polished chat interface.

This is the second post in the Scrappy build series. The first post covered authentication. This one covers the application stack that authentication supports: frontend, backend, database, cache, and vector retrieval.