Hello, my dear friend!
I have a small problem with "chat with your PDF" projects.
Not because they are bad.
They are actually a great way to learn RAG.
My problem is that most tutorials stop right when things start getting interesting.
Upload a PDF.
Split it.
Create embeddings.
Put them in a vector database.
Ask a question.
Find similar chunks.
Send them to a model.
Done.
And technically, yes.
You built RAG.
But there is one innocent-looking step in there that caused me most of the trouble:
Find the relevant chunks.
Relevant according to what?
That question became the actual project.
The Chat Was the Easy Part
The visible product is simple.
Upload documents.
Ask questions.
Get answers.
Open the source.
But before a document becomes searchable, quite a lot has to happen.
upload
↓
store
↓
extract
↓
clean
↓
chunk
↓
embed
↓
index
↓
ready
That made me realize I was not really building a chat app.
I was building a small data pipeline with a chat box at the end.
And once I looked at it that way, a lot of decisions became easier.
A 100-page PDF should not keep an HTTP request open while it gets parsed, chunked, embedded, and indexed.
So the API accepts the file, stores it, creates a job, and gets out of the way.
A worker handles the expensive part.
The user just sees:
Uploading → Reading document → Preparing content → Ready
They do not need to know that embedding batch 7 of 12 is currently fighting for its life.
They only need to know whether the document is ready.
Chunking Is Annoyingly Important
Chunking sounded boring at first.
Split every few hundred tokens.
Add overlap.
Move on.
Then I tried it.
Large chunks preserve context, but they also mix unrelated ideas together.
Small chunks are precise, but sometimes you retrieve one lonely sentence that makes no sense without the heading above it.
So the question stopped being:
What is the best chunk size?
and became:
How much context does this piece need to still make sense?
There is no magic number.
Where possible, I keep headings with their sections, respect paragraph boundaries, and preserve page information for citations.
When structure is missing, I fall back to token-based splitting.
Nothing fancy.
Just slightly less naive.
Vector Search Was Not Enough
Embeddings are great when the wording changes but the meaning stays similar.
A document might say:
Employee credentials must be revoked immediately after termination.
And the user asks:
What happens to someone's account when they leave the company?
Semantic search handles that beautifully.
Then someone asks:
What does SEC-104 say?
Now I do not want semantic creativity.
I want SEC-104.
Exactly.
The same problem shows up with product names, policy numbers, error codes, acronyms, and technical terms.
So I stopped treating vector search as the search system.
It became one search system.
The other is plain keyword search.
Semantic search asks:
What means something similar?
Keyword search asks:
Where do these exact words appear?
Then I combine both.
Sometimes the clever solution is embeddings.
Sometimes the clever solution is basically Ctrl+F.
Retrieve Broadly. Then Get Picky.
My first pipeline was:
question
↓
nearest chunks
↓
model
It worked.
Until "similar" and "useful" stopped meaning the same thing.
A passage can be about the right topic and still not contain the answer.
So I changed the job of the first retrieval step.
It does not need to find the perfect passage.
It needs to avoid throwing the perfect passage away.
That means retrieving a slightly broader candidate set first.
Then I rerank those candidates using both the question and the passage.
The pipeline became:
question
↓
semantic + keyword search
↓
merge candidates
↓
rerank
↓
select context
↓
generate answer
The mental model I ended up keeping was:
Retrieve broadly. Rank carefully. Generate last.
Reranking costs time, though.
So more candidates are not automatically better.
More context can easily become more noise.
I Kept the Vectors in Postgres
I used PostgreSQL with pgvector.
Not because dedicated vector databases are bad.
I just already needed Postgres.
The app has users, workspaces, documents, conversations, permissions, processing states, and citation metadata.
Postgres was already doing all of that.
Keeping embeddings there meant one less service to deploy, configure, secure, monitor, and eventually forget why I added.
Would I make the same choice with hundreds of millions of vectors?
Maybe not.
But I did not have hundreds of millions of vectors.
I had the project that actually existed.
That is the scale I wanted to design for.
Citations Became a Debugging Tool
Originally, citations were there for the user.
Ask something.
Get an answer.
Click the source.
Nice.
Then I realized they were even more useful when the answer was wrong.
A bad answer can mean two very different things.
Retrieval found the wrong source.
Or:
Retrieval found the right source, but generation went wrong.
Without citations, both failures look like:
The AI got it wrong.
With citations, I can inspect exactly what reached the model.
Wrong source?
Fix retrieval.
Right source?
Look at generation.
The citation is not just decoration.
It is part of the debugging system.
I Stopped Evaluating by Vibes
For a while, my evaluation process was basically:
Change something.
Ask a few questions.
Stare at the answers.
Say:
Hmm. Feels better.
Very scientific.
Eventually I made a small evaluation dataset.
Each test has a question and the passage that should answer it.
Then I run retrieval and check whether that passage appears near the top.
I mostly care about two simple things.
Recall@K: Did the right passage show up in the first few results?
MRR: If it showed up, how high was it?
And I track latency because there is always a wonderfully stupid way to improve quality by making the system ten times slower.
Now I can change chunk size, candidate count, hybrid search behavior, or reranking depth and actually compare what happened.
Not a research lab.
Just enough to stop guessing.
No, I Did Not Turn Everything Into an Agent
I could have.
One agent could inspect the question.
Another could choose the documents.
Another could rewrite the query.
Another could judge retrieval.
Another could decide whether to retrieve again.
Another could write the answer.
And suddenly six language models are having a meeting about one PDF.
I kept it simpler.
The interesting problem was retrieval, and I wanted that pipeline to stay visible.
The backend stayed modular.
The worker stayed separate because background processing genuinely has different operational needs.
Postgres stayed Postgres.
Redis handled the queue.
Object storage kept the raw files.
That was enough.
Complexity should buy something.
If it does not, it is just rent.
The Answer Is Almost the End of the Story
From the outside, the product is still simple.
Upload something.
Ask something.
Get an answer.
Check the source.
That is exactly how I want it.
But the useful part of building it was realizing how much happens before the answer appears.
Someone has to decide how documents become data.
How that data gets split.
How search works when meaning matters.
How search works when exact words matter more.
How candidates get ranked.
How much context is enough.
What happens when the answer is not there.
And how you know the whole system actually improved instead of simply becoming more complicated.
That is the part I wanted to understand.
The chat just gave me somewhere nice to put it.
With that being said, see you in the next one.
Much love,
Yassine Erradouani