Moving Document Processing Into the Worker Layer

After building the core application stack for Scrappy, the next step was moving the expensive work out of the API.

The backend should be responsible for coordinating the application. It should authenticate requests, register documents, create jobs, expose status endpoints, and enforce permissions. But it should not block an HTTP request while it parses a file, splits text into chunks, calls an embedding model, and writes vector data back into the database.

That is the job of the worker layer.

For Scrappy, the worker layer is where uploaded documents become usable retrieval data. It takes a file that the user uploaded, extracts the text, normalizes the content, chunks it into smaller sections, generates embeddings, and sends the processed results back to the Scrappy backend so they can be saved in the database.

This is the point where Scrappy starts to become a real RAG application.

Why the worker layer exists

Document processing can be slow.

A small text file might process quickly, but PDFs, long documents, scanned files, and future document types can take much longer. Some files may require parsing, cleanup, chunking, embedding generation, retries, and error handling.

That work should not happen directly inside the request-response cycle.

If the API tried to do everything inline, the user would be stuck waiting for the entire ingestion pipeline to finish. The API would also become harder to scale, because every upload request could turn into a long-running processing job.

Instead, Scrappy separates the responsibilities:

  • The API accepts the request.
  • The API creates the document and job records.
  • The API kicks off a worker task.
  • The worker performs the expensive processing.
  • The worker calls Scrappy backend endpoints to save chunks and embeddings.
  • The frontend checks document and job status through the API.

That design keeps the API responsive while still allowing Scrappy to process documents in the background.

The worker as the ingestion engine

The worker layer is responsible for the parts of the system that are too expensive, too slow, or too failure-prone to run inline.

For Scrappy, that includes:

  • File text extraction
  • Document normalization
  • Chunking
  • Embedding generation
  • Embedding persistence
  • Re-indexing
  • Retry handling
  • Failure reporting

The worker is not the main application API. It is the ingestion engine behind the application.

That distinction matters. The API owns the user-facing contract. The worker owns the background processing.

The document processing flow

The current flow starts with the backend.

When a user uploads a document, the Scrappy API records the document, stores the metadata, and starts a background task. From there, the worker picks up the task and begins processing.

The flow looks roughly like this:

User uploads document
   |
Frontend sends upload request to Scrappy API
   |
API authenticates the request
   |
API creates document and job records
   |
API kicks off worker task
   |
Worker retrieves the document context
   |
Worker extracts text from the file
   |
Worker chunks the extracted text
   |
Worker generates embeddings for each chunk
   |
Worker calls Scrappy backend endpoints
   |
Backend saves chunks and embeddings in the database
   |
Document becomes available for retrieval

The important design choice is that the worker does not directly become the owner of Scrappy’s application state.

Instead, the worker uses the backend endpoints to persist processed chunks and embeddings. That keeps the backend in control of validation, permissions, schemas, and database writes.

The worker does the heavy work. The backend remains the system boundary.

Why the worker calls the backend

There are two ways to design this.

One option is to let the worker connect directly to the database and write chunks and embeddings itself. That can be efficient, but it also means the worker needs to understand more of the application’s persistence rules.

The other option is to have the worker call the backend API when it needs to save processed results.

For Scrappy, I chose the second approach.

The worker processes the file, creates chunks, generates embeddings, and then calls Scrappy backend endpoints to save the results. That gives the system a cleaner separation:

The worker owns processing logic. The backend owns application state. The database remains behind the backend contract.

This also makes the worker easier to evolve independently. If the database schema changes, the worker should not need to know every internal detail. It can rely on the backend API contract instead.

Chunking the document

Chunking is one of the most important parts of the ingestion pipeline.

An LLM cannot use an entire large document as context every time a user asks a question. The document needs to be split into smaller sections that can be embedded, searched, and retrieved later.

The goal is to create chunks that are small enough to retrieve efficiently but large enough to preserve meaning.

If the chunks are too large, retrieval becomes noisy. A single result may contain too much unrelated information. If the chunks are too small, the system may lose the surrounding context that makes an answer useful.

For Scrappy, chunking is the bridge between raw document text and retrieval-ready data.

A simplified chunking flow looks like this:

Raw document
   |
Extracted text
   |
Normalized text
   |
Chunk 1
Chunk 2
Chunk 3
Chunk 4
   |
Each chunk gets metadata
   |
Each chunk becomes eligible for embedding

Each chunk needs enough metadata to be useful later. That can include the document ID, chunk index, page number, token count, and the text itself.

That metadata matters during retrieval because the system needs to know where each result came from.

Generating embeddings

Once the worker has chunks, the next step is embedding generation.

An embedding is a vector representation of a piece of text. Instead of searching only by exact keyword matches, Scrappy can use embeddings to find chunks that are semantically related to the user’s question.

That is what makes retrieval-augmented generation possible.

The embedding flow looks like this:

Chunk text
   |
Embedding model
   |
Vector representation
   |
Backend save endpoint
   |
Postgres / pgvector

Each chunk is sent through an embedding model. The result is stored with the chunk so Scrappy can later perform similarity search.

When a user asks a question, the system can embed the query, compare it against stored chunk embeddings, retrieve the most relevant chunks, and send those chunks as context to the LLM.

Why embeddings belong in the ingestion pipeline

Embedding generation should happen during ingestion, not at query time.

If Scrappy waited until a user asked a question to embed every document, the chat experience would be too slow. The user would ask something simple, and the system would have to parse files, chunk text, generate embeddings, and search them before answering.

Instead, the ingestion pipeline prepares the document ahead of time.

By the time the user asks a question, Scrappy should already have:

Extracted document text Created chunks Generated embeddings Stored vectors Updated document readiness status

That makes the query flow much faster.

The user’s question still needs to be embedded at query time, but the document side of the work is already complete.

The worker and the LLM service boundary

Embedding generation is also where the worker layer starts to touch the model layer.

I do not want model calls scattered randomly throughout the codebase. The cleaner direction is to keep model interactions behind dedicated service modules.

That can include:

embedding_service retrieval_service generation_service prompt_service citation_service

For the worker layer, the most important one is the embedding service.

The worker should be able to say, “generate embeddings for these chunks,” without every worker task needing to know the details of the provider, model name, retry policy, timeout behavior, or cost tracking.

That abstraction becomes more important over time. If I change embedding providers or model versions later, I do not want to rewrite the ingestion pipeline.

Status tracking and visibility

A worker pipeline needs visibility.

It is not enough to kick off a task and hope it finishes. The system needs to know what state the document is in and what happened if something fails.

Useful document states include:

uploaded processing parsed chunked embedded indexed ready failed

Useful job states include:

queued running completed failed retrying

Those states should be written to durable storage so the frontend can show progress and the backend can support debugging.

For example, if a document fails during embedding, the user should not just see that the file disappeared or never became available. The system should know that the job failed, where it failed, and what error message was produced.

This is one of the reasons Postgres remains the durable source of truth. Redis or pub/sub can help with live updates, but durable status should be stored in the database.

Events and task flow

The longer-term direction for Scrappy is an event-centered ingestion pipeline.

The current worker setup starts with the API kicking off a task. That is already a good separation. As the system grows, the task flow can become more explicit through consistent event names.

A future event flow could look like this:

document.uploaded
   |
document.parse.requested
   |
document.parsed
   |
document.chunk.requested
   |
document.chunked
   |
document.embed.requested
   |
document.embedded
   |
document.indexed
   |
document.ready

This makes the pipeline easier to scale because each stage can become its own queue or worker type.

Parsing, chunking, embedding, indexing, and cleanup do not all need to run in the same worker process forever. They can start together and split apart later when the system needs more capacity or better isolation.

Queue separation

The first version of a worker system can use one queue.

That is fine for early development. But eventually, different job types should be separated.

A practical queue structure could include:

high_priority ingestion embedding cleanup

This prevents one type of work from blocking everything else.

For example, embedding jobs may be slower because they depend on model calls. Cleanup jobs may be lower priority. High-priority jobs may need to move ahead of long document-processing tasks.

Separate queues make that possible.

Retry handling

The worker layer also needs retry handling.

Document processing can fail for reasons that are not permanent. A model API call can timeout. A backend save endpoint can briefly be unavailable. A file parser can fail on one attempt and succeed later after a transient issue.

Retries are especially important for:

Model API failures Network failures Backend API timeouts Temporary storage issues Parsing failures that may be recoverable

But retries need limits. A failed job should not retry forever.

Eventually, failed jobs should be inspectable and replayable. That gives the system a path for debugging instead of silently dropping work.

How this supports retrieval

The worker layer is what makes retrieval possible.

Without the worker, Scrappy may have uploaded files, but it does not have retrieval-ready data. The documents are just stored files.

After the worker runs, the system has structured chunks, embeddings, and status records.

That changes what Scrappy can do.

Before worker processing: Document exists as a file.

After worker processing: Document exists as searchable, retrievable, LLM-ready context.

That is the transition from file storage to knowledge retrieval.

What I learned from this part of the build

The biggest lesson from this phase was that background work needs a clear contract.

It is not enough to say, “the worker processes the document.” The system needs to define what starts the task, what the worker receives, what the worker produces, where results are saved, how failures are tracked, and how the frontend knows when the work is complete.

For Scrappy, the contract is becoming clear.

The API starts the work. The worker performs the expensive processing. The worker calls backend endpoints to save chunks and embeddings. The backend persists the results. The frontend reads status from the backend.

That separation keeps the system understandable.

The API remains responsive. The worker can focus on ingestion. The database stores durable state. Redis and the queue infrastructure support coordination. The vector store makes the processed content searchable.

This is the third post in the Scrappy build series. The first post covered authentication. The second covered the frontend, backend, database, cache, and vector store. This one covers the worker layer that turns uploaded documents into chunks, embeddings, and retrieval-ready data.