Pipeline Overview

ToolStorePy has an author-time pipeline and a build-time pipeline.

End-to-end architecture

tools.json
  |
  v
embedding pipeline
  |
  v
ChromaDB index
  |
  v
queries.json
  |
  v
semantic retrieval
  |
  v
cross-encoder reranking
  |
  v
repo cloning
  |
  v
AST security scan
  |
  v
LLM security scan (optional, --llm-scan)
  |
  v
.env merge + validation
  |
  v
AST parsing
  |
  v
MCP synthesis

Author-time stages

• define tools in a structured manifest, preferably JSON
• chunk metadata records
• embed them locally
• persist the collection for later distribution

Build-time stages

• resolve an index by name or URL
• retrieve best-matching tools for each query
• clone and inspect repos
• scan with AST scanner (always)
• scan with LLM scanner if --llm-scan is passed (autonomous, no human prompt)
• build one MCP server from approved tool code

Security scan modes

Mode	How it works	Human prompt
AST only (default)	Static analysis, flags HIGH/MEDIUM/LOW	Yes — user approves or skips HIGH repos
LLM scan (--llm-scan)	LLM reads full source, returns INCLUDE/SKIP	No — LLM decision is final
Both	AST runs, LLM reviews all repos, findings merged	No — LLM decision is final

Server configuration

The generated server runs on streamable-http transport. Host and port are baked in at build time via --host and --port (defaults: 0.0.0.0:8000).

Design principle

Index authoring is part of the product surface, not just a maintenance task. Without the index pipeline there is no semantic discovery layer, and without semantic discovery the build pipeline falls back to manual curation.