{"openapi":"3.1.0","info":{"title":"Not Diamond API","version":"0.1.0"},"paths":{"/v2/modelRouter/modelSelect":{"post":{"tags":["Model Router","Model Router"],"summary":"Select best LLM for your query","description":"Select the optimal LLM to handle your query based on Not Diamond's routing algorithm.\n\nThis endpoint analyzes your messages and returns the best-suited model from your specified models.\nThe router considers factors like query complexity, model capabilities, cost, and latency based on your preferences.\n\n**Key Features:**\n- Intelligent routing across multiple LLM providers\n- Support for custom routers trained on your evaluation data\n- Optional cost/latency optimization\n- Function calling support for compatible models\n\n**Usage:**\n1. Pass your messages in OpenAI format (array of objects with 'role' and 'content')\n2. Specify which LLM providers you want to route between\n3. Optionally provide a preference_id to use a custom router that you've trained\n4. Receive a recommended model and session_id\n5. Use the session_id to submit feedback and improve routing for non-hashed sessions\n\n**Related Endpoints:**\n- `POST /v2/preferences/userPreferenceCreate` - Create a preference ID for personalized routing\n- `POST /v2/pzn/trainCustomRouter` - Train a custom router on your evaluation data","operationId":"token_model_select_v2_modelRouter_modelSelect_post","security":[{"bearerAuth":[]}],"parameters":[{"name":"type","in":"query","required":false,"schema":{"anyOf":[{"type":"string"},{"type":"null"}],"description":"Optional format type. Use 'openrouter' to accept and return OpenRouter-format model identifiers","title":"Type"},"description":"Optional format type. Use 'openrouter' to accept and return OpenRouter-format model identifiers"}],"requestBody":{"required":true,"content":{"application/json":{"schema":{"$ref":"#/components/schemas/OptimizerBaseRequest"},"examples":{"basic_routing":{"summary":"Basic routing example","description":"Route a simple query between three models","value":{"messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Explain quantum computing in simple terms"}],"llm_providers":[{"provider":"openai","model":"gpt-4o"},{"provider":"anthropic","model":"claude-sonnet-4-5-20250929"},{"provider":"google","model":"gemini-2.5-flash"}]}},"with_preference":{"summary":"Routing with preference ID","description":"Use a preference ID for personalized routing","value":{"messages":[{"role":"user","content":"Write a Python function to sort a list"}],"llm_providers":[{"provider":"openai","model":"gpt-4o"},{"provider":"anthropic","model":"claude-sonnet-4-5-20250929"}],"preference_id":"550e8400-e29b-41d4-a716-446655440000"}},"cost_optimization":{"summary":"Cost-optimized routing","description":"Prioritize cost savings while maintaining quality","value":{"messages":[{"role":"user","content":"Summarize this article"}],"llm_providers":[{"provider":"openai","model":"gpt-4o"},{"provider":"openai","model":"gpt-4o-mini"}],"tradeoff":"cost"}},"function_calling":{"summary":"Routing with function calling","description":"Route queries that require function calling","value":{"messages":[{"role":"user","content":"What's the weather in San Francisco?"}],"llm_providers":[{"provider":"openai","model":"gpt-4o"},{"provider":"anthropic","model":"claude-sonnet-4-5-20250929"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get the current weather","parameters":{"type":"object","properties":{"location":{"type":"string"}}}}}]}}}}}},"responses":{"200":{"description":"Successfully selected optimal model","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ModelSelectResponse"},"example":{"providers":[{"provider":"openai","model":"gpt-4o"}],"session_id":"550e8400-e29b-41d4-a716-446655440000"}}}},"404":{"description":"Preference ID not found"},"400":{"description":"Invalid request (e.g., no providers specified, invalid preference_id)"},"401":{"description":"Authentication failed"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}},"/v2/preferences/userPreferenceCreate":{"post":{"tags":["Preferences","Preferences"],"summary":"Create a preference ID","description":"Create a new preference ID for personalized LLM routing.\n\nA preference ID enables personalized routing by tracking your feedback and learning your preferences over time.\nOnce created, you can:\n1. Use it in model_select() calls to get personalized routing decisions\n2. Train a custom router specific to your use case\n\n**Workflow:**\n1. Create a preference ID (this endpoint)\n2. Use the preference_id in POST /v2/modelRouter/modelSelect requests\n3. Optionally train a custom router via POST /v2/pzn/trainCustomRouter\n\n**Benefits:**\n- Personalized routing for your specific use case\n- Ability to train custom routers on your evaluation data\n\n**Note:** If you don't provide a preference_id in model_select() calls, the default router will be used.","operationId":"create_preference_v2_preferences_userPreferenceCreate_post","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/PreferenceCreateBase"},"examples":{"default":{"summary":"Create with auto-generated name","description":"Create a preference with an auto-generated timestamp name","value":{}},"with_name":{"summary":"Create with custom name","description":"Create a preference with a custom name for easy identification","value":{"name":"My Production Router"}}}}},"required":true},"responses":{"200":{"description":"Successfully created preference ID","content":{"application/json":{"schema":{"$ref":"#/components/schemas/PreferenceCreateResponse"},"example":{"preference_id":"550e8400-e29b-41d4-a716-446655440000"}}}},"404":{"description":"Not found"},"401":{"description":"Authentication failed"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}},"security":[{"bearerAuth":[]}]}},"/v2/preferences/userPreferenceUpdate":{"put":{"tags":["Preferences"],"summary":"Update User Preference","operationId":"update_user_preference_v2_preferences_userPreferenceUpdate_put","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/UpdatePreferenceBase"}}},"required":true},"responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{}}}},"404":{"description":"Not found"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}},"security":[{"bearerAuth":[]}]}},"/v2/preferences/userPreferenceDelete/{preference_id}":{"delete":{"tags":["Preferences"],"summary":"Delete User Preference","operationId":"delete_user_preference_v2_preferences_userPreferenceDelete__preference_id__delete","security":[{"bearerAuth":[]}],"parameters":[{"name":"preference_id","in":"path","required":true,"schema":{"type":"string","title":"Preference Id"}}],"responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{}}}},"404":{"description":"Not found"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}},"/v2/prompt/optimize":{"post":{"tags":["Prompt Optimization"],"summary":"Adapt prompts across LLMs","description":"Optimize your prompt from one LLM to work optimally across different target LLMs.\n\nThis endpoint automatically optimizes your prompt (system prompt + user message template) to improve accuracy on your use case across various models. Each model has unique characteristics, and what works well for GPT-5 might not work as well for Claude or Gemini.\n\n**How Prompt Optimization Works:**\n1. You provide your current prompt and optionally your current origin model\n2. You specify the target models you want to optimize your prompt to\n3. You provide evaluation examples (golden records) with expected answers\n4. The system runs optimization to find the best prompt for each target model\n5. You receive optimized prompts that perform well on your target models\n\n**Evaluation Metrics:**\nChoose either a standard metric or provide custom evaluation:\n- **Standard metrics**: LLMaaJ:Sem_Sim_1 (semantic similarity), JSON_Match\n- **Custom evaluation**: Provide evaluation_config with your own LLM judge, prompt, and cutoff\n\n**Dataset Requirements:**\n- Minimum 25 examples in train_goldens (more examples = better optimization)\n- **Prototype mode**: Set `prototype_mode: true` to use as few as 3 examples for prototyping\n  - Recommended when you don't have enough data yet to build a proof-of-concept\n  - Note: Performance may be degraded compared to standard mode (25+ examples)\n  - Trade-off: Faster iteration with less data vs. potentially less generalizability\n- Each example must have fields matching your template placeholders\n- Supervised evaluation requires 'answer' field in each golden record\n- Unsupervised evaluation can work without answers\n\n**Training Time:**\n- Processing is asynchronous and typically takes 10-30 minutes\n- Time depends on: number of target models, dataset size, model availability\n- Use the returned optimization_run_id to check status and retrieve results\n\n**Example Workflow:**\n```\n1. POST /v2/prompt/optimize - Submit optimization request\n2. GET /v2/prompt/optimizeStatus/{id} - Poll status until completed\n3. GET /v2/prompt/optimizeResults/{id} - Retrieve optimized prompts\n4. Use optimized prompts in production with target models\n```","operationId":"optimize_prompt_v2_prompt_optimize_post","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/PromptAdaptationRequest"},"examples":{"basic_adaptation":{"summary":"Basic prompt optimization","description":"Optimize a mathematical digit counting prompt to Claude and Gemini","value":{"fields":["question"],"system_prompt":"You are a mathematical assistant that counts digits accurately.","target_models":[{"model":"claude-sonnet-4-5-20250929","provider":"anthropic"},{"model":"gemini-2.5-flash","provider":"google"}],"template":"Question: {question}\nAnswer:","train_goldens":[{"fields":{"question":"How many digits are in (23874045494*2789392485)?"},"answer":"20"},{"fields":{"question":"How many odd digits are in (999*777*555*333*111)?"},"answer":"10"},{"fields":{"question":"How often does the number '17' appear in the digits of (287558*17)?"},"answer":"0"},{"fields":{"question":"How many even digits are in (222*444*666*888)?"},"answer":"16"},{"fields":{"question":"How many 0s are in (1234567890*1357908642)?"},"answer":"2"}],"test_goldens":[{"fields":{"question":"How many digits are in (9876543210*123456)?"},"answer":"15"},{"fields":{"question":"How many odd digits are in (135*579*246)?"},"answer":"8"},{"fields":{"question":"How often does the number '42' appear in the digits of (123456789*42)?"},"answer":"1"},{"fields":{"question":"How many even digits are in (1111*2222*3333)?"},"answer":"10"},{"fields":{"question":"How many 9s are in (999999*888888)?"},"answer":"11"}],"evaluation_metric":"LLMaaJ:Sem_Sim_1","prototype_mode":true}},"custom_judge":{"summary":"Optimization with custom LLM judge","description":"Use a custom evaluation prompt and judge model","value":{"system_prompt":"You are a SQL expert.","template":"Generate a SQL query for: {task}","fields":["task"],"train_goldens":[{"fields":{"task":"Get all users"},"answer":"SELECT * FROM users;"},{"fields":{"task":"Count orders"},"answer":"SELECT COUNT(*) FROM orders;"},{"fields":{"task":"Find active customers"},"answer":"SELECT * FROM customers WHERE status = 'active';"},{"fields":{"task":"Sum total sales"},"answer":"SELECT SUM(amount) FROM sales;"},{"fields":{"task":"Get unique products"},"answer":"SELECT DISTINCT product_id FROM orders;"}],"test_goldens":[{"fields":{"task":"List recent orders"},"answer":"SELECT * FROM orders ORDER BY created_at DESC LIMIT 10;"}],"origin_model":{"provider":"openai","model":"gpt-4o"},"target_models":[{"provider":"anthropic","model":"claude-sonnet-4-5-20250929"}],"evaluation_config":{"llm_judging_prompt":"Evaluate if this SQL query correctly solves: {question}. Query: {answer}. Rate 0-1.","llm_judge":"openai/gpt-4o","correctness_cutoff":0.7}}},"multi_field_template":{"summary":"Multi-field template optimization","description":"Optimize a prompt with multiple template fields","value":{"system_prompt":"You are an expert at answering questions using provided context.","template":"Context: {context}\n\nQuestion: {question}\n\nProvide a detailed answer based only on the context.","fields":["context","question"],"goldens":[{"fields":{"context":"The Earth orbits the Sun once per year.","question":"How long does it take Earth to orbit the Sun?"},"answer":"One year"},{"fields":{"context":"Python is a high-level programming language.","question":"What type of language is Python?"},"answer":"A high-level programming language"},{"fields":{"context":"The Pacific Ocean is the largest ocean on Earth.","question":"Which is the largest ocean?"},"answer":"The Pacific Ocean"},{"fields":{"context":"Photosynthesis is the process plants use to create food.","question":"What is photosynthesis?"},"answer":"The process plants use to create food"},{"fields":{"context":"The Eiffel Tower is located in Paris, France.","question":"Where is the Eiffel Tower?"},"answer":"Paris, France"}],"origin_model":{"provider":"openai","model":"gpt-4o"},"target_models":[{"provider":"anthropic","model":"claude-sonnet-4-5-20250929"}],"evaluation_metric":"LLMaaJ:Sem_Sim_1","prototype_mode":true}}}}},"required":true},"responses":{"200":{"description":"Successfully started prompt optimization","content":{"application/json":{"schema":{"$ref":"#/components/schemas/PromptAdaptationResponse"},"example":{"optimization_run_id":"00000000-0000-0000-0000-000000000000"}}}},"404":{"description":"Not found"},"400":{"description":"Invalid request (e.g., insufficient examples, invalid models, missing required fields)","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ErrorResponse"},"examples":{"insufficient_examples":{"summary":"Insufficient training examples","value":{"error":{"code":"INSUFFICIENT_GOLDEN_EXAMPLES","message":"train_goldens must be a non-empty list with minimum 25 examples (or 3 with prototype_mode=true)","type":"validation_error","request_id":"req_abc123"}}},"unsupported_model":{"summary":"Unsupported model","value":{"error":{"code":"UNSUPPORTED_MODEL","message":"target_model anthropic/claude-2 not supported","type":"validation_error","request_id":"req_abc123"}}},"tier_limit_exceeded":{"summary":"Target model count exceeded for tier","value":{"error":{"code":"TARGET_MODEL_COUNT_EXCEEDED","message":"target_models must be a non-empty list with maximum 1 models for free tier","type":"validation_error","request_id":"req_abc123"}}}}}}},"401":{"description":"Authentication failed or no prompt optimization access","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ErrorResponse"},"example":{"error":{"code":"UNAUTHORIZED","message":"Authentication failed","type":"authentication_error","request_id":"req_abc123"}}}}},"422":{"description":"Validation error (e.g., must provide either goldens or train_goldens/test_goldens)","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ErrorResponse"},"example":{"error":{"code":"MISSING_GOLDENS","message":"Either goldens must be provided, or both train_goldens and test_goldens must be provided","type":"validation_error","request_id":"req_abc123"}}}}},"429":{"description":"Concurrent run limit exceeded","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ErrorResponse"},"examples":{"concurrent_run":{"summary":"Concurrent run limit","value":{"error":{"code":"CONCURRENT_RUN_LIMIT","message":"A prompt optimization is already in progress (run ID: 123e4567-e89b-12d3-a456-426614174000). Please wait for it to complete before starting a new one.","type":"rate_limit_error","request_id":"req_abc123"}}}}}}}},"security":[{"bearerAuth":[]}]}},"/v2/prompt/optimizeStatus/{optimization_run_id}":{"get":{"tags":["Prompt Optimization"],"summary":"Check prompt optimization status","description":"Check the status of a prompt optimization run.\n\nUse this endpoint to poll the status of your optimization request. Processing is asynchronous,\nso you'll need to check periodically until the status indicates completion.\n\n**Status Values:**\n- `created`: Initial state, not yet processing\n- `queued`: Waiting for processing capacity (check queue_position)\n- `processing`: Currently optimizing prompts\n- `completed`: All target models have been processed successfully\n- `failed`: One or more target models failed to process\n\n**Polling Recommendations:**\n- Poll every 30-60 seconds during processing\n- Check queue_position if status is 'queued' to estimate wait time\n- Stop polling once status is 'completed' or 'failed'\n- Use GET /v2/prompt/optimizeResults to retrieve results after completion\n\n**Queue Position:**\n- Only present when status is 'queued'\n- Lower numbers mean earlier processing (position 1 is next)\n- Typical wait time: 1-5 minutes per position\n\n**Note:** This endpoint only returns status information. To get the actual optimized prompts\nand evaluation results, use GET /v2/prompt/optimizeResults once status is 'completed'.","operationId":"get_optimize_status_v2_prompt_optimizeStatus__optimization_run_id__get","security":[{"bearerAuth":[]}],"parameters":[{"name":"optimization_run_id","in":"path","required":true,"schema":{"type":"string","title":"Optimization Run Id"}}],"responses":{"200":{"description":"Successfully retrieved optimization status","content":{"application/json":{"schema":{"$ref":"#/components/schemas/PromptAdaptationStatusResponse"},"examples":{"processing":{"summary":"Optimization in progress","value":{"optimization_run_id":"00000000-0000-0000-0000-000000000000","status":"processing"}},"queued":{"summary":"Optimization queued","value":{"optimization_run_id":"00000000-0000-0000-0000-000000000000","status":"queued","queue_position":3}},"completed":{"summary":"Optimization completed","value":{"optimization_run_id":"00000000-0000-0000-0000-000000000000","status":"completed"}}}}}},"404":{"description":"Optimization run ID not found","content":{"application/json":{"example":{"error":{"code":"RUN_ID_NOT_FOUND","message":"Optimization run ID 123e4567-e89b-12d3-a456-426614174000 not found","type":"not_found_error","request_id":"req_abc123"}},"schema":{"$ref":"#/components/schemas/ErrorResponse"}}}},"401":{"description":"Authentication failed or no prompt optimization access","content":{"application/json":{"example":{"error":{"code":"UNAUTHORIZED","message":"Authentication failed","type":"authentication_error","request_id":"req_abc123"}},"schema":{"$ref":"#/components/schemas/ErrorResponse"}}}},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}},"/v2/prompt/optimizeResults/{optimization_run_id}":{"get":{"tags":["Prompt Optimization"],"summary":"Get prompt optimization results","description":"Retrieve the complete results of a prompt optimization run, including optimized prompts for all target models.\n\nThis endpoint returns the optimized prompts and evaluation metrics for each target model in your optimization request.\nCall this endpoint after the optimization status is 'completed' to get your optimized prompts.\n\n**Response Structure:**\n- **origin_model**: Baseline performance of your original prompt on the origin model\n  - Includes: system_prompt, user_message_template, score, evaluation metrics, cost\n- **target_models**: Array of results for each target model\n  - Includes: optimized system_prompt, user_message_template, template_fields\n  - pre_optimization_score: Performance before optimization\n  - post_optimization_score: Performance after optimization\n  - Evaluation metrics and cost information\n\n**Using Optimized Prompts:**\n1. Extract the `system_prompt` and `user_message_template` from each target model result\n2. Use `user_message_template_fields` to know which fields to substitute\n3. Apply the optimized prompts when calling the respective target models\n4. Compare pre/post optimization scores to see improvement\n\n**Status Handling:**\n- If optimization is still processing, target model results will have `result_status: \"processing\"`\n- Only completed target models will have system_prompt and template values\n- Failed target models will have `result_status: \"failed\"` with null values\n\n**Cost Information:**\n- Each model result includes cost in USD for the optimization process\n- Costs vary based on model pricing and number of evaluation examples\n- Typical range: $0.10 - $2.00 per target model\n\n**Best Practices:**\n1. Wait for status 'completed' before calling this endpoint\n2. Check result_status for each target model\n3. Validate that post_optimization_score > pre_optimization_score\n4. Save optimized prompts for production use\n5. A/B test optimized prompts against originals in production","operationId":"get_optimize_results_v2_prompt_optimizeResults__optimization_run_id__get","security":[{"bearerAuth":[]}],"parameters":[{"name":"optimization_run_id","in":"path","required":true,"schema":{"type":"string","title":"Optimization Run Id"}}],"responses":{"200":{"description":"Successfully retrieved optimization results","content":{"application/json":{"schema":{"$ref":"#/components/schemas/AdaptationRunResultsResponse"},"example":{"id":"00000000-0000-0000-0000-000000000000","created_at":"2024-01-15T10:30:00Z","updated_at":"2024-01-15T10:45:00Z","origin_model":{"model_name":"openai/gpt-4o","score":8.5,"evals":{"accuracy":0.85},"cost":0.05,"system_prompt":"You are a helpful assistant.","user_message_template":"Question: {question}\nAnswer:","result_status":"completed"},"target_models":[{"model_name":"anthropic/claude-sonnet-4-5-20250929","pre_optimization_score":7.2,"pre_optimization_evals":{"accuracy":0.72},"post_optimization_score":8.8,"post_optimization_evals":{"accuracy":0.88},"cost":0.75,"system_prompt":"You are an expert assistant specialized in providing accurate answers.","user_message_template":"Please answer the following question:\n{question}\n\nProvide a clear and concise answer:","user_message_template_fields":["question"],"result_status":"completed"}],"job_status":"completed","evaluation_metric":"LLMaaJ:Sem_Sim_3"}}}},"404":{"description":"Optimization run ID not found","content":{"application/json":{"example":{"error":{"code":"RUN_ID_NOT_FOUND","message":"Optimization run not found","type":"not_found_error","request_id":"req_abc123"}},"schema":{"$ref":"#/components/schemas/ErrorResponse"}}}},"401":{"description":"Authentication failed or no prompt optimization access","content":{"application/json":{"example":{"error":{"code":"UNAUTHORIZED","message":"Authentication failed","type":"authentication_error","request_id":"req_abc123"}},"schema":{"$ref":"#/components/schemas/ErrorResponse"}}}},"403":{"description":"Run ID does not belong to user","content":{"application/json":{"example":{"error":{"code":"RUN_ID_NOT_OWNED","message":"Run ID 123e4567-e89b-12d3-a456-426614174000 does not belong to user abc-123","type":"authorization_error","request_id":"req_abc123"}},"schema":{"$ref":"#/components/schemas/ErrorResponse"}}}},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}},"/v2/prompt/optimize/{optimization_run_id}/costs":{"get":{"tags":["Prompt Optimization"],"summary":"Get LLM cost for a prompt optimization run","description":"Get LLM usage costs for a specific prompt optimization run.\n\nThis endpoint returns the total cost and detailed usage records for all LLM requests\nmade during a prompt optimization run. Use this to track costs associated with\noptimizing prompts for different target models.\n\n**Cost Breakdown:**\n- Total cost across all models used in the optimization\n- Individual usage records with provider, model, tokens, and costs\n- Timestamps for each LLM request\n\n**Access Control:**\n- Only accessible by the user who created the optimization run\n- Requires prompt optimization access","operationId":"get_optimization_run_costs_v2_prompt_optimize__optimization_run_id__costs_get","security":[{"bearerAuth":[]}],"parameters":[{"name":"optimization_run_id","in":"path","required":true,"schema":{"type":"string","title":"Optimization Run Id"}}],"responses":{"200":{"description":"Successfully retrieved optimization run costs","content":{"application/json":{"schema":{"$ref":"#/components/schemas/AdaptationRunCostResponse"},"example":{"optimization_run_id":"550e8400-e29b-41d4-a716-446655440000","total_cost":0.0456,"usage_records":[{"id":"llm_usage_123","user_id":"user_abc","organization_id":"org_xyz","optimization_run_id":"550e8400-e29b-41d4-a716-446655440000","provider":"openai","model":"gpt-4","input_tokens":250,"output_tokens":100,"input_cost":0.0125,"output_cost":0.006,"total_cost":0.0185,"timestamp":1705320600.123456,"task_type":"optimization"},{"id":"llm_usage_124","user_id":"user_abc","organization_id":"org_xyz","optimization_run_id":"550e8400-e29b-41d4-a716-446655440000","provider":"anthropic","model":"claude-3-opus-20240229","input_tokens":180,"output_tokens":75,"input_cost":0.0135,"output_cost":0.01125,"total_cost":0.02475,"timestamp":1705320615.456789,"task_type":"post-optimization evaluation"}]}}}},"404":{"description":"Optimization run not found"},"401":{"description":"Authentication failed"},"403":{"description":"Access denied - optimization run does not belong to user"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}},"/v2/pzn/trainCustomRouter":{"post":{"tags":["PZN","PZN"],"summary":"Train a custom LLM router","description":"Train a custom router on your evaluation data to optimize routing for your specific use case.\n\nThis endpoint allows you to train a domain-specific router that learns which models perform best\nfor different types of queries in your application. The router analyzes your evaluation dataset,\nclusters similar queries, and learns model performance patterns.\n\n**Training Process:**\n1. Upload a CSV file with your evaluation data\n2. Specify which models to route between\n3. Define the evaluation metric (score column)\n4. The system trains asynchronously and returns a preference_id\n5. Use the preference_id in model_select() calls once training completes\n\n**Dataset Requirements:**\n- Format: CSV file\n- Minimum samples: 25 (more is better for accuracy)\n- Required columns:\n  - Prompt column (specified in prompt_column parameter)\n  - For each model: `{provider}/{model}/score` and `{provider}/{model}/response`\n\n**Example CSV structure:**\n```\nprompt,openai/gpt-4o/score,openai/gpt-4o/response,anthropic/claude-sonnet-4-5-20250929/score,anthropic/claude-sonnet-4-5-20250929/response\n\"Explain quantum computing\",0.95,\"Quantum computing uses...\",0.87,\"Quantum computers leverage...\"\n\"Write a Python function\",0.82,\"def my_function()...\",0.91,\"Here's a Python function...\"\n```\n\n**Model Selection:**\n- Specify standard models: `{\"provider\": \"openai\", \"model\": \"gpt-4o\"}`\n- Or custom models with pricing: `{\"provider\": \"custom\", \"model\": \"my-model\", \"is_custom\": true, \"input_price\": 10.0, \"output_price\": 30.0, \"context_length\": 8192, \"latency\": 1.5}`\n\n**Training Time:**\n- Training is asynchronous and typically takes 5-15 minutes\n- Larger datasets or more models take longer\n- You'll receive a preference_id immediately\n- Check training status by attempting to use the preference_id in model_select()\n\n**Best Practices:**\n1. Use diverse, representative examples from your production workload\n2. Include at least 50-100 samples for best results\n3. Ensure consistent evaluation metrics across all models\n4. Use the same models you plan to route between in production\n\n**Related Documentation:**\nSee https://docs.notdiamond.ai/docs/adapting-prompts-to-new-models for detailed guide.","operationId":"train_custom_router_v2_pzn_trainCustomRouter_post","requestBody":{"content":{"multipart/form-data":{"schema":{"$ref":"#/components/schemas/Body_train_custom_router_v2_pzn_trainCustomRouter_post"}}},"required":true},"responses":{"200":{"description":"Successfully started training custom router","content":{"application/json":{"schema":{"$ref":"#/components/schemas/TrainCustomRouterResponse"},"example":{"preference_id":"550e8400-e29b-41d4-a716-446655440000"}}}},"404":{"description":"Preference ID not found"},"400":{"description":"Invalid request (e.g., insufficient samples, invalid dataset format, missing columns)"},"401":{"description":"Authentication failed"},"408":{"description":"Request timeout - dataset file took too long to parse"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}},"security":[{"bearerAuth":[]}]}},"/v2/models":{"get":{"tags":["Models"],"summary":"List Models","description":"List all supported text generation models with optional filtering.\n\nincluding pricing, context length, latency, and OpenRouter availability.\n\n**Note:** Image generation models are excluded from this endpoint.\n\n**Examples:**\n- Get all models: `/v2/models`\n- OpenRouter only: `/v2/models?openrouter_only=true`\n- Specific provider: `/v2/models?provider=openai`\n- Multiple providers: `/v2/models?provider=openai&provider=anthropic`\n\n**Query Parameters:**\n- **provider**: Filter by provider name(s). Can specify multiple times for multiple providers\n  (e.g., `?provider=openai&provider=anthropic`)\n- **openrouter_only**: Return only models that have OpenRouter support (default: false)\n\n**Returns:**\n- **models**: List of active text generation model objects with metadata\n- **total**: Total number of active models returned\n- **deprecated_models**: List of deprecated text generation model objects with metadata\n  (respects the same filters as active models)\n\n**Caching:**\n- Response is cacheable for 1 hour (model list rarely changes)","operationId":"list_models_v2_models_get","parameters":[{"name":"provider","in":"query","required":false,"schema":{"anyOf":[{"type":"array","items":{"type":"string"}},{"type":"null"}],"description":"Filter by provider name(s). Can specify multiple providers (e.g., 'openai', 'anthropic')","title":"Provider"},"description":"Filter by provider name(s). Can specify multiple providers (e.g., 'openai', 'anthropic')","style":"form","explode":true},{"name":"openrouter_only","in":"query","required":false,"schema":{"type":"boolean","description":"Return only OpenRouter-supported models","default":false,"title":"Openrouter Only"},"description":"Return only OpenRouter-supported models"}],"responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ModelsListResponse"}}}},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}}},"components":{"schemas":{"AdaptationRunCostResponse":{"properties":{"optimization_run_id":{"type":"string","title":"Optimization Run Id","description":"Unique identifier for the adaptation run"},"total_cost":{"type":"number","title":"Total Cost","description":"Total cost in USD across all LLM requests in this adaptation run"},"usage_records":{"items":{"$ref":"#/components/schemas/LLMUsageResponse"},"type":"array","title":"Usage Records","description":"Detailed usage records for each LLM request made during the adaptation"}},"type":"object","required":["optimization_run_id","total_cost","usage_records"],"title":"AdaptationRunCostResponse","description":"Response model for GET /v2/prompt/optimize/{optimization_run_id}/costs endpoint.\n\nContains the total LLM costs and detailed usage records for a prompt adaptation run.\nUse this to track costs associated with optimizing prompts for different target models.","example":{"optimization_run_id":"550e8400-e29b-41d4-a716-446655440000","total_cost":0.0456,"usage_records":[{"id":"llm_usage_123","input_cost":0.0125,"input_tokens":250,"model":"gpt-4","optimization_run_id":"550e8400-e29b-41d4-a716-446655440000","organization_id":"org_xyz","output_cost":0.006,"output_tokens":100,"provider":"openai","task_type":"optimization","timestamp":1705320600.123456,"total_cost":0.0185,"user_id":"user_abc"}]}},"AdaptationRunResultsResponse":{"properties":{"evaluation_metric":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Evaluation Metric"},"evaluation_config":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Evaluation Config"},"id":{"type":"string","title":"Id","description":"Unique ID for this adaptation run"},"created_at":{"type":"string","format":"date-time","title":"Created At","description":"Timestamp when this adaptation run was created"},"updated_at":{"anyOf":[{"type":"string","format":"date-time"},{"type":"null"}],"title":"Updated At","description":"Timestamp of last update to this adaptation run"},"origin_model":{"anyOf":[{"$ref":"#/components/schemas/OriginModelAdaptationResultsResponse"},{"type":"null"}],"description":"Results for the origin model (baseline performance with original prompts)"},"target_models":{"items":{"$ref":"#/components/schemas/TargetModelAdaptationResultsResponse"},"type":"array","title":"Target Models","description":"Results for each target model with optimized prompts and improvement scores"},"job_status":{"$ref":"#/components/schemas/JobStatus","description":"Overall status of the adaptation run (queued, running, completed, failed)"},"llm_request_metrics":{"anyOf":[{"items":{"additionalProperties":true,"type":"object"},"type":"array"},{"type":"null"}],"title":"Llm Request Metrics","description":"Metrics for the LLM requests made during the adaptation run. List of {model: str, num_requests: int}."},"prototype_mode":{"type":"boolean","title":"Prototype Mode","description":"Whether this adaptation run was created with prototype mode (3-24 training examples allowed). Prototype mode may have degraded performance compared to standard mode (25+ examples)","default":false}},"type":"object","required":["id","created_at","updated_at","target_models","job_status"],"title":"AdaptationRunResultsResponse","description":"Response model for GET /v2/prompt/optimizeResults/{optimization_run_id} endpoint.\n\nContains the complete results of a prompt adaptation run, including optimized prompts\nand evaluation metrics for all target models. Use this to retrieve your adapted prompts\nafter the adaptation status is 'completed'.\n\nThe response includes:\n- Baseline performance of your original prompt on the origin model\n- Optimized prompts for each target model with pre/post optimization scores\n- Evaluation metrics and cost information for each model","example":{"created_at":"2024-01-15T10:00:00.123456","id":"550e8400-e29b-41d4-a716-446655440000","job_status":"completed","llm_request_metrics":{"avg_latency_ms":1250.5,"total_requests":45},"origin_model":{"cost":0.012,"evaluation_type":"llm_as_judge","model":"openai/gpt-4","score":0.85,"system_prompt":"You are a helpful assistant.","user_message_template":"Question: {question}"},"target_models":[{"cost":0.018,"evaluation_type":"llm_as_judge","model":"anthropic/claude-3-sonnet-20240229","post_optimization_score":0.89,"pre_optimization_score":0.72,"score_improvement":0.17,"system_prompt":"You are an expert assistant focused on providing clear, accurate answers.","user_message_template":"Question: {question}\n\nProvide a detailed response.","user_message_template_fields":["question"]}],"updated_at":"2024-01-15T10:15:30.789012"}},"Body_survey_response_v2_pzn_surveyResponse_post":{"properties":{"user_id":{"type":"string","title":"User Id","description":"User ID from Supabase"},"email":{"type":"string","title":"Email","description":"User email address"},"use_case_desc":{"type":"string","title":"Use Case Desc","description":"Description of the user's use case"},"llm_providers":{"type":"string","title":"Llm Providers","description":"JSON string of LLM providers array"},"constraint_priorities":{"type":"string","title":"Constraint Priorities","description":"JSON string of constraint priorities object"},"name":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Name","description":"Optional preference name"},"prompts":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Prompts","description":"Optional JSON string of prompts array"},"prompt_file":{"anyOf":[{"type":"string","contentMediaType":"application/octet-stream"},{"type":"null"}],"title":"Prompt File","description":"Optional CSV file with prompts"},"dataset_file":{"anyOf":[{"type":"string","contentMediaType":"application/octet-stream"},{"type":"null"}],"title":"Dataset File","description":"Optional CSV file with evaluation dataset"},"additional_preferences":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Additional Preferences","description":"Optional additional preferences text"}},"type":"object","required":["user_id","email","use_case_desc","llm_providers","constraint_priorities"],"title":"Body_survey_response_v2_pzn_surveyResponse_post"},"Body_train_custom_router_v2_pzn_trainCustomRouter_post":{"properties":{"language":{"type":"string","title":"Language","description":"Language of the evaluation data. Use 'english' for English-only data or 'multilingual' for multi-language support","examples":["english"]},"llm_providers":{"type":"string","title":"Llm Providers","description":"JSON string array of LLM providers to train the router on. Format: '[{\"provider\": \"openai\", \"model\": \"gpt-4o\"}, {\"provider\": \"anthropic\", \"model\": \"claude-sonnet-4-5-20250929\"}]'","examples":["[{\"provider\": \"openai\", \"model\": \"gpt-4o\"}, {\"provider\": \"anthropic\", \"model\": \"claude-sonnet-4-5-20250929\"}]"]},"prompt_column":{"type":"string","title":"Prompt Column","description":"Name of the column in the CSV file that contains the prompts","examples":["prompt"]},"dataset_file":{"type":"string","contentMediaType":"application/octet-stream","title":"Dataset File","description":"CSV file containing evaluation data with prompt column and score/response columns for each model"},"maximize":{"type":"boolean","title":"Maximize","description":"Whether higher scores are better. Set to true if higher scores indicate better performance, false otherwise"},"preference_id":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Preference Id","description":"Optional preference ID to update an existing router. If not provided, a new preference will be created"},"override":{"anyOf":[{"type":"boolean"},{"type":"null"}],"title":"Override","description":"Whether to override an existing custom router for this preference_id","default":true}},"type":"object","required":["language","llm_providers","prompt_column","dataset_file","maximize"],"title":"Body_train_custom_router_v2_pzn_trainCustomRouter_post"},"Body_train_v2_router_train_post":{"properties":{"models":{"type":"string","title":"Models","description":"JSON array of models with pricing. Format: '[{\"provider\": \"openai\", \"model\": \"gpt-4o\", \"dollars_per_million_input\": 2.5, \"dollars_per_million_output\": 10.0}]'","examples":["[{\"provider\": \"openai\", \"model\": \"gpt-4o\", \"dollars_per_million_input\": 2.5, \"dollars_per_million_output\": 10.0}, {\"provider\": \"anthropic\", \"model\": \"claude-sonnet-4-5-20250929\", \"dollars_per_million_input\": 3.0, \"dollars_per_million_output\": 15.0}]"]},"training_data_file":{"type":"string","contentMediaType":"application/octet-stream","title":"Training Data File","description":"JSONL file containing an array of training examples with prompt, scores, and responses"}},"type":"object","required":["models","training_data_file"],"title":"Body_train_v2_router_train_post"},"ErrorDetail":{"properties":{"code":{"type":"string","title":"Code","description":"Machine-readable error code (e.g., 'INVALID_MODEL', 'RATE_LIMIT_EXCEEDED')"},"message":{"type":"string","title":"Message","description":"Human-readable error message"},"type":{"type":"string","title":"Type","description":"Error type/category (e.g., 'validation_error', 'authentication_error')"},"request_id":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Request Id","description":"Unique request ID for tracking and debugging"},"details":{"anyOf":[{"additionalProperties":true,"type":"object"},{"type":"null"}],"title":"Details","description":"Additional error-specific details for debugging"}},"type":"object","required":["code","message","type"],"title":"ErrorDetail","description":"Standardized error response structure."},"ErrorResponse":{"properties":{"error":{"$ref":"#/components/schemas/ErrorDetail","description":"Error details"}},"type":"object","required":["error"],"title":"ErrorResponse","description":"Top-level error response wrapper."},"FeedbackReport":{"properties":{"session_id":{"type":"string","title":"Session Id","description":"Session ID returned from POST /v2/modelRouter/modelSelect"},"provider":{"$ref":"#/components/schemas/RequestProvider","description":"The provider that was selected by the router"},"feedback":{"additionalProperties":true,"type":"object","title":"Feedback","description":"Feedback dictionary with 'accuracy' key (0 for thumbs down, 1 for thumbs up)","examples":[{"accuracy":1}]}},"type":"object","required":["session_id","provider","feedback"],"title":"FeedbackReport","description":"Request model for submitting feedback on routing decisions."},"GoldenRecord":{"properties":{"fields":{"additionalProperties":{"type":"string"},"type":"object","title":"Fields","description":"Dictionary mapping field names to their values. Keys must match the fields specified in the template","examples":[{"context":"Basic arithmetic","question":"What is 2+2?"}]},"answer":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Answer","description":"Expected answer for supervised evaluation. Required for supervised metrics, optional for unsupervised","examples":["4"]}},"type":"object","required":["fields"],"title":"GoldenRecord","description":"A training or test example for prompt adaptation."},"HTTPValidationError":{"properties":{"detail":{"items":{"$ref":"#/components/schemas/ValidationError"},"type":"array","title":"Detail"}},"type":"object","title":"HTTPValidationError"},"JobStatus":{"type":"string","enum":["created","queued","processing","completed","failed","cancelled"],"title":"JobStatus","description":"Status enum for asynchronous jobs (prompt adaptation, custom router training, etc.).\n\nRepresents the current state of a long-running operation:\n- **created**: Job has been initialized but not yet queued\n- **queued**: Job is waiting in the queue to be processed\n- **processing**: Job is currently being executed\n- **completed**: Job finished successfully and results are available\n- **failed**: Job encountered an error and did not complete\n- **cancelled**: Job was cancelled due to a restart operation"},"LLMUsageResponse":{"properties":{"id":{"type":"string","title":"Id","description":"Unique identifier for this usage record"},"user_id":{"type":"string","title":"User Id","description":"User ID who made the request"},"organization_id":{"type":"string","title":"Organization Id","description":"Organization ID associated with the request"},"adaptation_run_id":{"type":"string","title":"Adaptation Run Id","description":"Adaptation run ID this usage is associated with"},"provider":{"type":"string","title":"Provider","description":"LLM provider (e.g., 'openai', 'anthropic', 'google')"},"model":{"type":"string","title":"Model","description":"Model name (e.g., 'gpt-4', 'claude-3-opus-20240229')"},"input_tokens":{"type":"integer","title":"Input Tokens","description":"Number of input tokens consumed"},"output_tokens":{"type":"integer","title":"Output Tokens","description":"Number of output tokens generated"},"input_cost":{"type":"number","title":"Input Cost","description":"Cost of input tokens in USD"},"output_cost":{"type":"number","title":"Output Cost","description":"Cost of output tokens in USD"},"total_cost":{"type":"number","title":"Total Cost","description":"Total cost (input + output) in USD"},"timestamp":{"type":"number","title":"Timestamp","description":"Unix timestamp when the request was made"},"task_type":{"type":"string","title":"Task Type","description":"Type of task: 'pre-optimization evaluation', 'optimization', or 'post-optimization evaluation'"}},"type":"object","required":["id","user_id","organization_id","adaptation_run_id","provider","model","input_tokens","output_tokens","input_cost","output_cost","total_cost","timestamp","task_type"],"title":"LLMUsageResponse","description":"Individual LLM usage record with token counts and cost breakdown.\n\nReturned by GET /llm-usage endpoint and included in AdaptationRunCostResponse.\nEach record represents a single LLM API call with detailed usage metrics.","example":{"adaptation_run_id":"550e8400-e29b-41d4-a716-446655440000","id":"llm_usage_abc123","input_cost":0.0125,"input_tokens":250,"model":"gpt-4","organization_id":"org_def456","output_cost":0.006,"output_tokens":100,"provider":"openai","task_type":"optimization","timestamp":1705320600.123456,"total_cost":0.0185,"user_id":"user_xyz789"}},"LMProviderResponse":{"properties":{"id":{"type":"string","title":"Id"},"model_name":{"type":"string","title":"Model Name"},"provider_name":{"type":"string","title":"Provider Name"},"dollars_per_million_input":{"type":"number","title":"Dollars Per Million Input"},"dollars_per_million_output":{"type":"number","title":"Dollars Per Million Output"}},"type":"object","required":["id","model_name","provider_name","dollars_per_million_input","dollars_per_million_output"],"title":"LMProviderResponse"},"ModelResponse":{"properties":{"provider":{"type":"string","title":"Provider","description":"Provider name (e.g., 'openai', 'anthropic', 'google')"},"model":{"type":"string","title":"Model","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229')"},"context_length":{"type":"integer","title":"Context Length","description":"Maximum context window size in tokens"},"input_price":{"type":"number","title":"Input Price","description":"Price per million input tokens in USD"},"output_price":{"type":"number","title":"Output Price","description":"Price per million output tokens in USD"},"openrouter_model":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Openrouter Model","description":"OpenRouter model identifier if available, null if not supported via OpenRouter"}},"type":"object","required":["provider","model","context_length","input_price","output_price"],"title":"ModelResponse","description":"Response model for a single LLM model from GET /v2/models endpoint.\n\nContains metadata about a supported text generation model including pricing,\ncontext limits, and availability information.","example":{"context_length":8192,"input_price":30.0,"model":"gpt-4","openrouter_model":"openai/gpt-4","output_price":60.0,"provider":"openai"}},"ModelScore":{"properties":{"model":{"$ref":"#/components/schemas/notdiamond_server__custom_routing__schemas__ProviderResponse","description":"The model that was scored"},"score":{"type":"number","title":"Score","description":"Normalized score (0-1)"},"rank":{"type":"integer","title":"Rank","description":"Rank position (1 = best)"}},"type":"object","required":["model","score","rank"],"title":"ModelScore","description":"Model ranking with score information."},"ModelSelectResponse":{"properties":{"providers":{"items":{"$ref":"#/components/schemas/notdiamond_server__api__optimizer__ProviderResponse"},"type":"array","title":"Providers","description":"List containing the selected provider"},"session_id":{"type":"string","title":"Session Id","description":"Unique session ID for this routing decision"}},"type":"object","required":["providers","session_id"],"title":"ModelSelectResponse","description":"Response from model selection endpoint.","examples":[{"providers":[{"model":"gpt-4o","provider":"openai"}],"session_id":"00000000-0000-0000-0000-000000000000"}]},"ModelsListResponse":{"properties":{"models":{"items":{"$ref":"#/components/schemas/ModelResponse"},"type":"array","title":"Models","description":"List of active/supported text generation models with their metadata"},"total":{"type":"integer","title":"Total","description":"Total count of active models in the response"},"deprecated_models":{"items":{"$ref":"#/components/schemas/ModelResponse"},"type":"array","title":"Deprecated Models","description":"List of deprecated models that are no longer recommended but may still work"}},"type":"object","required":["models","total","deprecated_models"],"title":"ModelsListResponse","description":"Response model for GET /v2/models endpoint.\n\nReturns a list of all supported text generation models with their metadata,\nseparated into active and deprecated models.","example":{"deprecated_models":[{"context_length":4096,"input_price":0.5,"model":"gpt-4o","openrouter_model":"openai/gpt-3.5-turbo","output_price":1.5,"provider":"openai"}],"models":[{"context_length":8192,"input_price":30.0,"model":"gpt-4","openrouter_model":"openai/gpt-4","output_price":60.0,"provider":"openai"},{"context_length":200000,"input_price":15.0,"model":"claude-3-opus-20240229","openrouter_model":"anthropic/claude-3-opus-20240229","output_price":75.0,"provider":"anthropic"}],"total":2}},"OpenRouterProvider":{"properties":{"is_custom":{"type":"boolean","title":"Is Custom","description":"Whether this is a custom model not in Not Diamond's supported model list","default":false},"context_length":{"anyOf":[{"type":"integer"},{"type":"null"}],"title":"Context Length","description":"Maximum context length for the model (required for custom models)"},"input_price":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Input Price","description":"Input token price per million tokens in USD (required for custom models)"},"output_price":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Output Price","description":"Output token price per million tokens in USD (required for custom models)"},"latency":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Latency","description":"Average latency in seconds (required for custom models)"},"model":{"type":"string","title":"Model","description":"OpenRouter model identifier in 'provider/model' format (e.g., 'openai/gpt-4o', 'anthropic/claude-sonnet-4-5-20250929')","examples":["openai/gpt-4o","anthropic/claude-sonnet-4-5-20250929","google/gemini-2.5-flash"]}},"type":"object","required":["model"],"title":"OpenRouterProvider","description":"Model for specifying an LLM provider using OpenRouter format.\n\nUsed in model routing requests when you want to specify providers using the\nOpenRouter naming convention (combined 'provider/model' format). This is an\nalternative to the standard RequestProvider which uses separate provider and\nmodel fields.\n\n**When to use:**\n- When working with OpenRouter-compatible systems\n- When you prefer the unified 'provider/model' format\n- For models accessed via OpenRouter proxy","example":{"model":"openai/gpt-4o"}},"OptimizerBaseRequest":{"properties":{"messages":{"anyOf":[{"items":{"additionalProperties":{"anyOf":[{"type":"string"},{"items":{},"type":"array"}]},"type":"object"},"type":"array"},{"type":"string"}],"title":"Messages","description":"Array of message objects in OpenAI format (with 'role' and 'content' keys)","examples":[[{"content":"You are a helpful assistant.","role":"system"},{"content":"Explain quantum computing","role":"user"}]]},"llm_providers":{"items":{"anyOf":[{"$ref":"#/components/schemas/RequestProvider"},{"$ref":"#/components/schemas/OpenRouterProvider"}]},"type":"array","title":"Llm Providers","description":"List of LLM providers to route between. Specify at least one provider in format {provider, model}","examples":[[{"model":"gpt-4o","provider":"openai"},{"model":"claude-sonnet-4-5-20250929","provider":"anthropic"},{"model":"gemini-2.5-flash","provider":"google"}]]},"tools":{"anyOf":[{"items":{"additionalProperties":true,"type":"object"},"type":"array"},{"type":"null"}],"title":"Tools","description":"OpenAI-format function calling tools"},"hash_content":{"type":"boolean","title":"Hash Content","description":"Whether to hash message content for privacy. Hashed content is not persisted.","default":false},"metric":{"type":"string","title":"Metric","description":"Optimization metric for model selection","default":"accuracy"},"max_model_depth":{"anyOf":[{"type":"integer"},{"type":"null"}],"title":"Max Model Depth","description":"Maximum number of models to consider for routing. If not specified, considers all provided models"},"tradeoff":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Tradeoff","description":"Optimization tradeoff strategy. Use 'cost' to prioritize cost savings or 'latency' to prioritize speed","examples":["cost","latency"]},"cost_quality_tradeoff":{"anyOf":[{"type":"integer","maximum":10.0,"minimum":0.0},{"type":"null"}],"title":"Cost Quality Tradeoff","description":"Blend cost and quality: 0 = pure quality (default), 10 = pure cost. Mutually exclusive with `tradeoff`. Not supported with custom routers."},"preference_id":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Preference Id","description":"Preference ID for personalized routing. Create one via POST /v2/preferences/userPreferenceCreate"},"previous_session":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Previous Session","description":"Previous session ID to link related requests"}},"type":"object","required":["messages","llm_providers"],"title":"OptimizerBaseRequest","description":"Request model for LLM routing via Not Diamond."},"OriginModelAdaptationResultsResponse":{"properties":{"model_name":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Model Name"},"score":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Score"},"evals":{"anyOf":[{"additionalProperties":true,"type":"object"},{"type":"null"}],"title":"Evals"},"cost":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Cost"},"system_prompt":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"System Prompt","description":"Original system prompt used for the origin model"},"user_message_template":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"User Message Template","description":"Original user message template used for the origin model"},"result_status":{"anyOf":[{"$ref":"#/components/schemas/JobStatus"},{"type":"null"}],"description":"Status of the origin model evaluation job"}},"type":"object","required":["model_name","score","evals","cost"],"title":"OriginModelAdaptationResultsResponse","description":"Baseline results for the origin model in prompt adaptation.\n\nPart of AdaptationRunResultsResponse. Contains the performance metrics and prompt\nconfiguration for your original prompt on the origin model. This serves as the\nbaseline to compare against optimized prompts for target models.\n\n**Fields include:**\n- Original system prompt and user message template\n- Baseline performance score and evaluation metrics\n- Cost of running the baseline evaluation\n- Job status for the origin model evaluation","example":{"cost":0.012,"evals":{"accuracy":0.85,"latency":1.2},"model_name":"openai/gpt-4","result_status":"completed","score":0.85,"system_prompt":"You are a helpful assistant.","user_message_template":"Question: {question}"}},"ParetoPoint":{"properties":{"id":{"type":"string","title":"Id"},"accuracy":{"type":"number","maximum":1.0,"minimum":0.0,"title":"Accuracy","description":"Accuracy value in [0, 1]"},"input_cost":{"type":"number","minimum":0.0,"title":"Input Cost"},"output_cost":{"type":"number","minimum":0.0,"title":"Output Cost"},"lm_provider_id":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Lm Provider Id"},"router_topic_id":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Router Topic Id"}},"type":"object","required":["id","accuracy","input_cost","output_cost"],"title":"ParetoPoint","description":"A point on the Pareto frontier representing a cost/accuracy tradeoff."},"PreferenceCreateBase":{"properties":{"name":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Name","description":"Optional name for the preference. If not provided, an auto-generated timestamp will be used. Use descriptive names like 'Production API' or 'Customer Support Bot' for easy identification"}},"type":"object","title":"PreferenceCreateBase","description":"Request model for POST /v2/preferences/userPreferenceCreate endpoint.\n\nCreate a new preference ID for personalized LLM routing. A preference ID enables\nthe system to learn your routing preferences over time based on feedback, allowing\nfor increasingly accurate model selection tailored to your specific use case.\n\n**Use cases:**\n- Track feedback across multiple routing decisions\n- Enable personalized routing that improves with usage\n- Train custom routers specific to your application\n- Organize routing configurations by project or use case","examples":[{"name":"My Production Router"},{}]},"PreferenceCreateResponse":{"properties":{"preference_id":{"type":"string","title":"Preference Id","description":"Unique identifier for the newly created preference. Use this in the 'preference_id' parameter of model_select() calls to enable personalized routing"}},"type":"object","required":["preference_id"],"title":"PreferenceCreateResponse","description":"Response model for POST /v2/preferences/userPreferenceCreate endpoint.\n\nReturns the newly created preference ID which can be used to enable personalized\nLLM routing. Store this ID and include it in subsequent model_select() calls to\nactivate personalized routing based on your feedback and usage patterns.\n\n**Next steps after creation:**\n1. Use the preference_id in POST /v2/modelRouter/modelSelect requests\n2. Submit feedback on routing decisions to improve accuracy\n3. Optionally train a custom router using your evaluation data","example":{"preference_id":"550e8400-e29b-41d4-a716-446655440000"}},"PromptAdaptationRequest":{"properties":{"evaluation_metric":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Evaluation Metric"},"evaluation_config":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Evaluation Config"},"system_prompt":{"type":"string","title":"System Prompt","description":"System prompt to use with the origin model. This sets the context and role for the LLM","examples":["You are a helpful assistant that answers questions about science."]},"template":{"type":"string","title":"Template","description":"User message template with placeholders for fields. Use curly braces for field substitution","examples":["Question: {question}\nContext: {context}\nAnswer:"]},"fields":{"items":{"type":"string"},"type":"array","title":"Fields","description":"List of field names that will be substituted into the template. Must match keys in golden records","examples":[["question","context"]]},"goldens":{"anyOf":[{"items":{"$ref":"#/components/schemas/GoldenRecord"},"type":"array"},{"type":"null"}],"title":"Goldens","description":"Training examples (legacy parameter). Use train_goldens and test_goldens for better control. Minimum 25 examples (or 3 with prototype_mode=true)"},"train_goldens":{"anyOf":[{"items":{"$ref":"#/components/schemas/GoldenRecord"},"type":"array"},{"type":"null"}],"title":"Train Goldens","description":"Training examples for prompt optimization. Minimum 25 examples required (or 3 with prototype_mode=true). Cannot be used with 'goldens' parameter"},"test_goldens":{"anyOf":[{"items":{"$ref":"#/components/schemas/GoldenRecord"},"type":"array"},{"type":"null"}],"title":"Test Goldens","description":"Test examples for evaluation. Required if train_goldens is provided. Used to measure final performance on held-out data"},"origin_model":{"anyOf":[{"$ref":"#/components/schemas/RequestProvider"},{"type":"null"}],"description":"The model your current prompt is optimized for (baseline)."},"target_models":{"items":{"$ref":"#/components/schemas/RequestProvider"},"type":"array","title":"Target Models","description":"List of models to optimize the prompt for. Maximum count depends on your subscription tier (Free: 1, Starter: 3, Startup: 5, Enterprise: 10)"},"origin_model_evaluation_score":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Origin Model Evaluation Score","description":"Optional baseline score for the origin model. If provided, can skip origin model evaluation"},"prototype_mode":{"type":"boolean","title":"Prototype Mode","description":"Enable prototype mode to use as few as 3 training examples (instead of 25). Note: Performance may be degraded with fewer examples. Recommended for prototyping AI applications when you don't have enough data yet","default":false}},"type":"object","required":["system_prompt","template","fields","target_models"],"title":"PromptAdaptationRequest","description":"Request model for POST /v2/prompt/optimize endpoint.\n\nSubmits a prompt adaptation job to optimize your prompt for different target LLMs.\nThe system evaluates your original prompt on the origin model, then automatically\ngenerates and tests optimized prompts for each target model to maximize performance.\n\n**Key concepts:**\n- **system_prompt + template**: Your current prompt configuration\n- **origin_model**: The model your prompt currently works well with (baseline)\n- **target_models**: The models you want to optimize for\n- **train_goldens**: Evaluation examples used to optimize the prompts\n- **test_goldens**: Held-out examples used to measure final performance\n\n**Workflow:**\n1. Submit this request to start adaptation\n2. System evaluates baseline performance on origin model\n3. Optimizes prompts for each target model\n4. Returns adaptation_run_id for tracking progress\n5. Poll /adaptStatus until complete\n6. Retrieve optimized prompts from /adaptResults\n\n**Requirements:**\n- Minimum 25 examples in train_goldens (or 3 examples with prototype_mode=True)\n- test_goldens required when using train_goldens\n- Either use (goldens) or (train_goldens + test_goldens), not both\n- For supervised metrics, all examples must include 'answer' field\n\n**Prototype Mode:**\n- Set prototype_mode=True to allow as few as 3 training examples\n- Useful for prototyping AI applications when you don't have enough data yet\n- Note: Performance may be degraded compared to standard mode (25+ examples)"},"PromptAdaptationResponse":{"properties":{"optimization_run_id":{"type":"string","title":"Optimization Run Id","description":"Unique identifier for this optimization run. Use this to poll status and retrieve optimized prompts when complete"}},"type":"object","required":["optimization_run_id"],"title":"PromptAdaptationResponse","description":"Response model for POST /v2/prompt/optimize endpoint.\n\nReturned immediately after submitting a prompt optimization request. The optimization\nprocess runs asynchronously, so use the returned optimization_run_id to track progress\nand retrieve results when complete.\n\n**Next steps:**\n1. Store the optimization_run_id\n2. Poll GET /v2/prompt/optimizeStatus/{optimization_run_id} to check progress\n3. When status is 'completed', retrieve optimized prompts from GET /v2/prompt/optimizeResults/{optimization_run_id}\n4. Use the optimized prompts with your target models","example":{"optimization_run_id":"550e8400-e29b-41d4-a716-446655440000"}},"PromptAdaptationStatusResponse":{"properties":{"optimization_run_id":{"type":"string","title":"Optimization Run Id","description":"Unique identifier for this optimization run. Use this to poll status and retrieve optimized prompts when complete"},"status":{"$ref":"#/components/schemas/JobStatus","description":"Current status of the optimization run. Poll until this is 'completed' or 'failed'"},"queue_position":{"anyOf":[{"type":"integer"},{"type":"null"}],"title":"Queue Position","description":"Position in queue when status is 'queued'. Lower numbers process sooner. Null when not queued"}},"type":"object","required":["optimization_run_id","status"],"title":"PromptAdaptationStatusResponse","description":"Response model for GET /v2/prompt/optimizeStatus/{optimization_run_id} endpoint.\n\nReturns the current status of an asynchronous prompt optimization job. Poll this\nendpoint periodically to track progress. When status is 'completed', you can\nretrieve the optimized prompts using the /optimizeResults endpoint.\n\n**Status values:**\n- **created**: Job has been initialized\n- **queued**: Waiting in queue (check queue_position for your place in line)\n- **processing**: Currently running optimization\n- **completed**: Finished successfully, results available via /optimizeResults\n- **failed**: Encountered an error during processing\n\n**Polling recommendations:**\n- Poll every 30-60 seconds while status is incomplete\n- Stop polling once status is 'completed' or 'failed'\n- Optimization typically takes 10-30 minutes total","examples":[{"optimization_run_id":"550e8400-e29b-41d4-a716-446655440000","status":"processing"}]},"RequestProvider":{"properties":{"is_custom":{"type":"boolean","title":"Is Custom","description":"Whether this is a custom model not in Not Diamond's supported model list","default":false},"context_length":{"anyOf":[{"type":"integer"},{"type":"null"}],"title":"Context Length","description":"Maximum context length for the model (required for custom models)"},"input_price":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Input Price","description":"Input token price per million tokens in USD (required for custom models)"},"output_price":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Output Price","description":"Output token price per million tokens in USD (required for custom models)"},"latency":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Latency","description":"Average latency in seconds (required for custom models)"},"provider":{"type":"string","title":"Provider","description":"Provider name (e.g., 'openai', 'anthropic', 'google')","examples":["openai","anthropic","google"]},"model":{"type":"string","title":"Model","description":"Model name (e.g., 'gpt-4o', 'claude-sonnet-4-5-20250929')","examples":["gpt-4o","claude-sonnet-4-5-20250929","gemini-2.5-flash"]}},"type":"object","required":["provider","model"],"title":"RequestProvider","description":"Model for specifying an LLM provider in API requests."},"RouterInformationResponse":{"properties":{"router_id":{"type":"string","title":"Router Id"},"created_at":{"type":"string","format":"date-time","title":"Created At","description":"Timestamp when the router was created"},"status":{"$ref":"#/components/schemas/RouterStatus","description":"Current status of the router"},"lm_providers":{"items":{"$ref":"#/components/schemas/LMProviderResponse"},"type":"array","title":"Lm Providers"},"pareto_points":{"anyOf":[{"items":{"$ref":"#/components/schemas/ParetoPoint"},"type":"array"},{"type":"null"}],"title":"Pareto Points"}},"type":"object","required":["router_id","created_at","status","lm_providers"],"title":"RouterInformationResponse","description":"Response model for GET /v2/router/{router_id} endpoint."},"RouterPredictRequest":{"properties":{"messages":{"items":{"additionalProperties":{"type":"string"},"type":"object"},"type":"array","title":"Messages","description":"Array of message objects in OpenAI format (with 'role' and 'content' keys)","examples":[[{"content":"You are a helpful assistant.","role":"system"},{"content":"Explain quantum computing","role":"user"}]]},"router_id":{"type":"string","title":"Router Id","description":"Router ID for personalized routing."},"pareto_point_id":{"type":"string","title":"Pareto Point Id","description":"Pareto point ID specifying the cost/accuracy tradeoff to use for routing."},"tools":{"anyOf":[{"items":{"additionalProperties":true,"type":"object"},"type":"array"},{"type":"null"}],"title":"Tools","description":"OpenAI-format function calling tools"}},"type":"object","required":["messages","router_id","pareto_point_id"],"title":"RouterPredictRequest","description":"Request model for POST /v2/router/predict endpoint.","examples":[{"messages":[{"content":"You are a helpful assistant.","role":"system"},{"content":"Explain quantum computing","role":"user"}],"pareto_point_id":"660e8400-e29b-41d4-a716-446655440001","router_id":"550e8400-e29b-41d4-a716-446655440000"}]},"RouterPredictResponse":{"properties":{"model":{"$ref":"#/components/schemas/notdiamond_server__custom_routing__schemas__ProviderResponse","description":"The selected model"},"session_id":{"type":"string","title":"Session Id","description":"Unique session ID for this routing decision"},"rankings":{"anyOf":[{"items":{"$ref":"#/components/schemas/ModelScore"},"type":"array"},{"type":"null"}],"title":"Rankings","description":"Optional full ranking of all models considered"}},"type":"object","required":["model","session_id"],"title":"RouterPredictResponse","description":"Response model for POST /v2/router/predict endpoint.","example":{"model":{"model":"gpt-4o","provider":"openai"},"rankings":[{"model":{"model":"gpt-4o","provider":"openai"},"rank":1,"score":0.95},{"model":{"model":"claude-sonnet-4-5-20250929","provider":"anthropic"},"rank":2,"score":0.87}],"session_id":"550e8400-e29b-41d4-a716-446655440000"}},"RouterStatus":{"type":"string","enum":["active","pending","failed"],"title":"RouterStatus","description":"Status of a custom router."},"RouterTrainResponse":{"properties":{"router_id":{"type":"string","title":"Router Id","description":"Unique identifier for the trained router. Use this in /v2/router/predict calls."}},"type":"object","required":["router_id"],"title":"RouterTrainResponse","description":"Response model for POST /v2/router/train endpoint.","example":{"router_id":"550e8400-e29b-41d4-a716-446655440000"}},"TargetModelAdaptationResultsResponse":{"properties":{"model_name":{"type":"string","title":"Model Name"},"pre_optimization_score":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Pre Optimization Score"},"pre_optimization_evals":{"anyOf":[{"additionalProperties":true,"type":"object"},{"type":"null"}],"title":"Pre Optimization Evals"},"post_optimization_score":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Post Optimization Score"},"post_optimization_evals":{"anyOf":[{"additionalProperties":true,"type":"object"},{"type":"null"}],"title":"Post Optimization Evals"},"cost":{"anyOf":[{"type":"number"},{"type":"null"}],"title":"Cost"},"task_type":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Task Type"},"system_prompt":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"System Prompt","description":"Optimized system prompt for this target model. Use this as the system message in your LLM calls"},"user_message_template":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"User Message Template","description":"Optimized user message template with placeholders. Substitute fields using your data before calling the LLM"},"user_message_template_fields":{"anyOf":[{"items":{"type":"string"},"type":"array"},{"type":"null"}],"title":"User Message Template Fields","description":"List of field names to substitute in the template (e.g., ['question', 'context']). These match the curly-brace placeholders in user_message_template"},"result_status":{"anyOf":[{"$ref":"#/components/schemas/JobStatus"},{"type":"null"}],"description":"Status of this specific target model's optimization job. Check this is 'completed' before using the optimized prompts"}},"type":"object","required":["model_name","pre_optimization_score","pre_optimization_evals","post_optimization_score","post_optimization_evals","cost","task_type"],"title":"TargetModelAdaptationResultsResponse","description":"Optimized prompt results for a single target model in prompt adaptation.\n\nPart of AdaptationRunResultsResponse. Contains the optimized system prompt and\nuser message template for a specific target model, along with performance scores\nbefore and after optimization. Use these optimized prompts with the target model\nto achieve better performance than the original prompt.\n\n**Key metrics:**\n- **pre_optimization_score**: Performance with original prompt on this target model\n- **post_optimization_score**: Performance with optimized prompt on this target model\n- **Score improvement**: post - pre shows how much optimization helped\n\n**Usage:**\n1. Extract the optimized system_prompt and user_message_template\n2. Replace placeholders in user_message_template using fields from your data\n3. Use these prompts when calling this target model\n4. Compare pre/post scores to see improvement gained","example":{"cost":0.018,"model_name":"anthropic/claude-3-opus-20240229","post_optimization_evals":{"accuracy":0.89,"latency":1.4},"post_optimization_score":0.89,"pre_optimization_evals":{"accuracy":0.72,"latency":1.5},"pre_optimization_score":0.72,"result_status":"completed","system_prompt":"You are an expert assistant focused on providing clear, accurate answers based on context.","task_type":"optimization","user_message_template":"Question: {question}\n\nContext: {context}\n\nProvide a detailed, well-structured answer:","user_message_template_fields":["question","context"]}},"TrainCustomRouterResponse":{"properties":{"preference_id":{"type":"string","title":"Preference Id","description":"Unique identifier for the custom router. Use this in model_select() calls to enable routing with your custom-trained router"}},"type":"object","required":["preference_id"],"title":"TrainCustomRouterResponse","description":"Response model for POST /v2/pzn/trainCustomRouter endpoint.\n\nReturned immediately after submitting a custom router training request. The training\nprocess runs asynchronously (typically 5-15 minutes), so use the returned preference_id\nto make routing calls once training completes.\n\n**Next steps:**\n1. Store the preference_id\n2. Wait for training to complete (typically 5-15 minutes)\n3. Use this preference_id in POST /v2/modelRouter/modelSelect requests\n4. The router will use your custom-trained model to make routing decisions\n\n**How to use the preference_id:**\n- Include it in the 'preference_id' field of model_select() calls\n- The system automatically uses your custom router once training is complete\n- No need to poll status - you can start using it immediately (will use default until ready)","example":{"preference_id":"550e8400-e29b-41d4-a716-446655440000"}},"UpdatePreference":{"properties":{"preference_id":{"type":"string","title":"Preference Id"},"name":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Name"},"user_id":{"type":"string","title":"User Id"},"preference_weights":{"anyOf":[{"additionalProperties":true,"type":"object"},{"type":"null"}],"title":"Preference Weights"},"samples":{"items":{"additionalProperties":true,"type":"object"},"type":"array","title":"Samples","default":[]}},"type":"object","required":["preference_id","user_id"],"title":"UpdatePreference"},"UpdatePreferenceBase":{"properties":{"preference_id":{"type":"string","title":"Preference Id"},"name":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Name"}},"type":"object","required":["preference_id"],"title":"UpdatePreferenceBase"},"ValidationError":{"properties":{"loc":{"items":{"anyOf":[{"type":"string"},{"type":"integer"}]},"type":"array","title":"Location"},"msg":{"type":"string","title":"Message"},"type":{"type":"string","title":"Error Type"},"input":{"title":"Input"},"ctx":{"type":"object","title":"Context"}},"type":"object","required":["loc","msg","type"],"title":"ValidationError"},"notdiamond_server__api__optimizer__ProviderResponse":{"properties":{"provider":{"type":"string","title":"Provider","description":"Provider name for the selected model (e.g., 'openai', 'anthropic', 'google')","examples":["openai"]},"model":{"type":"string","title":"Model","description":"Model identifier for the selected model (e.g., 'gpt-4o', 'claude-3-opus-20240229')","examples":["gpt-4o"]}},"type":"object","required":["provider","model"],"title":"ProviderResponse","description":"Selected LLM provider information from model selection endpoints.\n\nPart of ModelSelectResponse. Contains the provider and model that Not Diamond's\nrouting algorithm selected as optimal for your query. Use these values to make\nyour LLM API call to the recommended model.","example":{"model":"claude-3-opus-20240229","provider":"anthropic"}},"notdiamond_server__custom_routing__schemas__ProviderResponse":{"properties":{"provider":{"type":"string","title":"Provider","description":"Provider name (e.g., 'openai', 'anthropic', 'google')","examples":["openai"]},"model":{"type":"string","title":"Model","description":"Model identifier (e.g., 'gpt-4o', 'claude-3-opus-20240229')","examples":["gpt-4o"]}},"type":"object","required":["provider","model"],"title":"ProviderResponse","description":"Selected LLM provider information for routing responses."}},"securitySchemes":{"bearerAuth":{"type":"http","scheme":"bearer","description":"API key authentication using Bearer token"}}},"servers":[{"url":"https://api.notdiamond.ai","description":"API Server"}]}