Data & RAG
Tools for building and testing RAG pipelines in VS Code
Most retrieval-augmented generation problems start before the model: chunks that are too small or too big, duplicates, missing metadata, or retrieval nobody measured. These tools let you check each step on your own documents, locally, before you pay to embed anything.
On this page 7 sections
Prepare and chunk documents
Chunking Tester splits a document recursively (paragraphs, then lines, then sentences), by Markdown section, by sentence or by fixed size, measured in tokens or characters. It shows every chunk with its size and heading trail, and warns about chunks much smaller than the target, duplicate chunks, overlap above 30% and chunks over 8K tokens that most embedding models truncate.
It exports chunks., ready for an embedding job. Two Markdown sections give two records like these:
{"id":"docs_install.md#0","text":"# Install\n\nRun npm install. Then start the server with npm start.","metadata":{"source":"docs/install.md","chunk_index":0,"headings":"Install","tokens":15}}
{"id":"docs_install.md#1","text":"## Configure\n\nSet PORT in .env. The default is 3000.","metadata":{"source":"docs/install.md","chunk_index":1,"headings":"Install > Configure","tokens":16}}Then Near-Duplicate Chunk Finder, Chunk Metadata Validator, JSON Lines Inspector and Data Profiler check the result before it is indexed.
Plan the index
- Chunk & Index Size Calculator: from corpus size and chunk settings, the number of chunks, embedding cost, monthly cost, vector storage and HNSW index RAM.
- Embedding Model Guide: compare embedding models by dimensions, input limit and price.
- Vector Store Setup: collection settings, a local Docker service, environment variables and Python client code for Chroma, Qdrant, pgvector, Pinecone, Weaviate or OpenSearch.
Retrieve and generate
- Retrieval Configuration: pick similarity, MMR, threshold or hybrid search and reranking, and get LangChain or LlamaIndex code.
- Vector Similarity and Hybrid Search Fusion (RRF) for combining keyword and vector results.
- Context Window Budget: how many retrieved chunks fit next to the system prompt, question and answer.
- Grounded Prompt Assembler: a prompt that cites its sources.
- RAG Pipeline Generator: a runnable ingest-and-ask project for your stack, with chunking, embeddings, a vector store and grounded answers with citations.
Measure it
Retrieval Evaluation scores retrieval on your own questions and flags the ones that found nothing relevant in the top results. Answer Grounding Checker marks each sentence of an answer that is not supported by the retrieved context, and the citations that are missing.
The same section has data engineering helpers: Partition Planner, Spark Cluster & Cost, Delta Lake Log Analyzer, Schema Viewer & Diff and Data Transformer.
Get started
- Open a document from your knowledge base in the editor.
- Run DevSnip Pro: Chunking Tester; the document is picked up from the active editor.
- Try two or three strategies and sizes, and watch the warnings.
- Run DevSnip Pro: Browse Data & RAG Tools for the rest of the pipeline.
Tools in this area
Data & RAG 21 tools
- Chunking Tester
- RAG Pipeline Generator
- Vector Store Setup
- Embedding Model Guide
- Retrieval Configuration
- Context Window Budget
- Grounded Prompt Assembler
- Chunk & Index Size Calculator
- Retrieval Evaluation
- Answer Grounding Checker
- Vector Similarity
- Near-Duplicate Chunk Finder
- Hybrid Search Fusion
- Chunk Metadata Validator
- Data Transformer
- Data Profiler
- JSON Lines Inspector
- Schema Viewer & Diff
- Partition Planner
- Spark Cluster & Cost
- Delta Lake Log Analyzer
Questions
- Do these tools call an embedding API?
- No. They run locally on the text you give them. Embedding and RAG pipeline tests against a real provider run from the REST API client and use points.
- What chunk size should I start with?
- Recursive splitting at 200–500 tokens with 10–20% overlap suits most question answering. Then measure on your own questions; the RAG chunking guide shows how.