Data Schema & Profiles

Profiles Overview

Profiles

  • Define dataset schemas (instruction/input/output/metadata) in data/datasetProfiles.ts.
  • Built-in profiles cover common instruction/response layouts; you can add custom profiles and register them.

Why profiles matter

  • Enforces consistent fields across generation, judging, and export.
  • UI adapts to the profile, ensuring labels and metadata align end-to-end.

Adding a custom profile

1) Create a new profile entry with id, name, description, and field definitions. 2) Register it so Generate/Judge/Dataset views can select it. 3) (Optional) Tailor evaluation axes per profile if you need custom scoring dimensions.

JSON structure

  • Each profile typically includes:
  • id: unique key
  • name: human-readable label
  • description: short explanation
  • fields: array of fields (e.g., instruction/input/output/metadata)
  • evaluationAxes (optional): scoring dimensions for judge UI

Example: Full-Stack Engineering Expert profile

{
  "id": "full_stack_engineering_expert",
  "name": "Full-Stack Engineering Expert",
  "summary": "Teaches end-to-end reasoning across the full web stack.",
  "description": "Generates tasks requiring reasoning about frontend, backend, database design, API behavior, deployment, state management, architectural choices, and full-stack debugging.",
  "audience": "Full-stack developers, software engineers, architects.",
  "recordType": "alpaca",
  "topics": [
    "frontend",
    "backend",
    "api",
    "databases",
    "architecture",
    "devops"
  ],
  "useCases": [
    "Train reasoning about full-stack workflows",
    "Model backend/frontend interaction",
    "Debug end-to-end system behavior",
    "Design scalable architectures",
    "Analyze data flow and API correctness"
  ],
  "recordSchema": {
    "instructionLabel": "Engineering Task",
    "inputLabel": "System Context",
    "outputLabel": "Final Answer",
    "metadataFields": [
      { "key": "reasoning", "label": "Engineering Reasoning", "description": "Step-by-step full-stack analysis." },
      { "key": "complexity", "label": "System Complexity", "description": "simple | moderate | complex" },
      { "key": "domain", "label": "Stack Domain", "description": "frontend, backend, api, db, architecture, devops" },
      { "key": "strategy", "label": "Problem Strategy", "description": "decomposition, debugging, flow analysis, design" }
    ]
  },
  "generationPrompt": "You are generating full-stack engineering tasks.\nUse only <context>.\nReturn JSON with instruction, input, output, metadata (reasoning, complexity, domain, strategy).\n- Final answer only in output.\n- All reasoning goes to metadata.reasoning.\n- No hallucinated tech.\n<context>\n{context}\n</context>",
  "groundTruthPrompt": "Produce the final full-stack solution only, without reasoning.\n\nSummary:\n{summary}\n\nScenario:\n{question}",
  "judgeGuidance": "Evaluate technical accuracy across full-stack boundaries.",
  "evaluationAxes": [
    { "key": "accuracy", "label": "Accuracy", "description": "Correct full-stack behavior." },
    { "key": "reasoning_quality", "label": "Reasoning Quality", "description": "Sound full-stack logic." },
    { "key": "completeness", "label": "Completeness", "description": "Addresses all stack layers." },
    { "key": "clarity", "label": "Clarity", "description": "Readable and structured." },
    { "key": "overall", "label": "Overall", "description": "Total engineering strength." }
  ]
}

UI walkthrough (Template Builder)

Template Builder

  • Profile Name: Full-Stack Engineering Expert
  • Profile ID: full_stack_engineering_expert
  • Description: Generates tasks requiring reasoning about frontend, backend, database design, API behavior, deployment, state management, architectural choices, and full-stack debugging.
  • Audience: Full-stack developers, software engineers, architects.
  • Record Type: alpaca
  • Use Cases: Train reasoning about full-stack workflows; Model backend/frontend interaction; Debug end-to-end system behavior; Design scalable architectures; Analyze data flow and API correctness.
  • Topics: frontend; backend; api; databases; architecture; devops.
  • Instruction Label: Engineering Task
  • Input Label: System Context
  • Output Label: Final Answer
  • Metadata Fields:
  • Engineering Reasoning (key: reasoning) — Step-by-step full-stack analysis.
  • System Complexity (key: complexity) — simple | moderate | complex.
  • Stack Domain (key: domain) — frontend, backend, api, db, architecture, devops.
  • Problem Strategy (key: strategy) — decomposition, debugging, flow analysis, design.
  • Generation Prompt: “You are generating full-stack engineering tasks... (see JSON above)”
  • Ground Truth Prompt: “Produce the final full-stack solution only... (see JSON above)”
  • Judge Guidance: Evaluate technical accuracy across full-stack boundaries.
  • Evaluation Axes: Accuracy; Reasoning Quality; Completeness; Clarity; Overall.