Close-the-Loop automates regeneration and evaluation cycles to iteratively improve sample quality. It links embeddings, teacher generation, judge scoring, and decision thresholds into a feedback loop.
Noted-Problems
These issues are likely due to hardware constraints, as generation is relatively slow on a single RTX 3090 GPU. While vLLM can significantly improve throughput, the evaluation (“judge”) step relies on a much larger model accessed via an API or OpenRouter. In a fully automated pipeline, the added overhead from network requests, responses, and throughput limits actually makes the process slower overall. As a result, I typically generate all samples first and then run the LLM-as-a-Judge evaluation in a single batch.
Process
Hit the button and let it autonomously create your synthetic dataset.
Key behaviors
Uses the configured teacher model for generation; judge model only scores (no retrieval).
Threshold determines if samples are accepted or regenerated with judge feedback.
Accepted samples flow into the dataset and can be embedded/synced to Qdrant for future retrieval.
Tips
Set a sensible judge threshold (e.g., 7/10) to balance acceptance vs. regeneration.
Ensure embeddings are up to date; better retrieval grounding improves generation quality.
Monitor the loop status and average scores to know when to stop or adjust parameters.
