You built a beautiful RAG pipeline. You used the latest embedding model (OpenAI text-embedding-3-large). You set up a vector database (Pinecone or Weaviate). You wrote a perfect system prompt.
Then you uploaded a real-world PDF from a customer. And the bot started hallucinating.
Why? Because your fancy AI is reading garbage.
The PDF Bottleneck
Most developers treat PDF parsing as a solved problem. They pip install pypdf and move on.
But pypdf (and its cousins) only extracts text layers.
If your document is a scanned contract, an invoice with a table, or a slide deck with images, you get nothing. Or worse, you get \n\n\n\n interspersed with gibberish.
The best RAG pipelines in 2026 don’t just “read text.” They perform Document Layout Analysis. They identify headers, footers, tables, and images. They use OCR (Tesseract or PaddleOCR) to read pixels, not just bytes.
This is computationally expensive. Parsing a single 50-page complex PDF with high-quality OCR takes 30-60 seconds on a CPU. Parsing 10,000 documents takes days.
The Privacy Trap
To solve the speed problem, you might reach for an API.
- OpenAI Vision API: Sends your document to OpenAI.
- Unstructured Cloud API: Sends your document to Unstructured.
- LlamaParse: Sends your document to LlamaIndex.
If you are a startup building a demo, this is fine. If you are an enterprise handling legal discovery, medical records (HIPAA), or financial audits, this is a non-starter. You cannot send PII (Personally Identifiable Information) to a third-party API just to extract text.
Data Sovereignty is the new latency. You need the power of the API, but you need it inside your own VPC.
Enter Unstructured.io (Self-Hosted)
Unstructured.io is the gold standard for open-source ETL. It packages the best OCR and layout analysis tools into a single library. Crucially, they offer a Docker image that runs the API locally.
docker run -p 8000:8000 quay.io/unstructured-io/unstructured-api:latest
This gives you the same power as their cloud API, but on your machine. Great. Now you have privacy. But you still have the speed problem. Running this on a single EC2 instance will still take days to process your backlog.
The Case for Ephemeral ETL Clusters
You don’t need a server. You need a swarm. You have a “burst” workload. You have 10,000 documents right now. Once they are indexed, you might not have another batch for a week.
Provisioning a permanent cluster of 50 GPU instances for this is a waste of money. Using Lambda is painful because the Unstructured Docker image is huge (several GBs) and has cold start times of 30s+.
This is where Ephemeral Containers shine. With PrevHQ, you can treat infrastructure as a function loop.
- Spin Up: Request 50 high-cpu containers with the Unstructured image pre-loaded.
- Shard: Split your document list into 50 chunks.
- Process: Each container processes 200 documents in parallel.
- Tear Down: As soon as the batch is done, the containers die.
You pay for 10 minutes of massive compute, not 24 hours of idle time. And your data never leaves the ephemeral environment.
How to Deploy
You can test this flow today. Instead of fighting with AWS ECS or Kubernetes just to run a batch job, use a preview environment as a worker.
- Fork the Unstructured API repo.
- Deploy it to PrevHQ (or your own private cloud).
- Send your PDFs to the endpoint.
- Destroy the environment.
The future of data engineering isn’t “always-on” pipelines. It’s “just-in-time” compute. Stop sending your data to strangers. Process it yourself, then delete the evidence.
FAQ
Q: Can I run Unstructured on AWS Lambda? A: It’s difficult. The Docker image is large (often exceeding Lambda limits) and OCR models require significant memory and CPU, leading to timeouts on large PDFs.
Q: How does this compare to LlamaParse? A: LlamaParse is excellent for proprietary reasoning over charts, but it is a managed service. Unstructured is open source and can be self-hosted for strict data privacy.
Q: Does Unstructured support GPU? A: Yes, if you use the GPU-tagged Docker images. This significantly speeds up OCR and layout analysis but requires GPU-enabled infrastructure.
Q: Is Tesseract enough for OCR? A: For simple text, yes. For complex layouts or handwriting, newer models (like PaddleOCR or vision transformers) are often superior, which Unstructured supports.
Q: How do I handle tables in PDFs?
A: Unstructured has specific strategies (hi_res strategy) that detect table structures and output them as HTML or Markdown tables, preserving the grid data.