waveterm/aiprompts/openai-request.md

OpenAI Request Input Field Structure (On-the-Wire Format)

This document describes the actual JSON structure sent to the OpenAI API in the input field of OpenAIRequest.

Overview

The input field is a JSON array containing one of three object types:

1. Messages (user/assistant) - OpenAIMessage objects
2. Function Calls (tool invocations) - OpenAIFunctionCallInput objects
3. Function Call Results (tool outputs) - OpenAIFunctionCallOutputInput objects

These are converted from OpenAIChatMessage internal format and cleaned before transmission (see lines 485-494).

1. Message Objects (User/Assistant)

User and assistant messages sent as OpenAIMessage:

{
  "role": "user",
  "content": [
    {
      "type": "input_text",
      "text": "Hello, analyze this image"
    },
    {
      "type": "input_image",
      "image_url": "data:image/png;base64,iVBORw0KG..."
    }
  ]
}

Key Points:

• role: Always "user" or "assistant"
• content: Always an array of content blocks (never a plain string)

Content Block Types

Text Block

{
  "type": "input_text",
  "text": "message content here"
}

Image Block

{
  "type": "input_image",
  "image_url": "data:image/png;base64,..."
}

• Can be a data URL or https:// URL
• filename field is removed during cleaning

PDF File Block

{
  "type": "input_file",
  "file_data": "JVBERi0xLjQKJeLjz9M...",
  "filename": "document.pdf"
}

• file_data: Base64-encoded PDF content

Function Call Block (in assistant messages)

{
  "type": "function_call",
  "call_id": "call_abc123",
  "name": "search_files",
  "arguments": {"query": "test"}
}

2. Function Call Objects (Tool Invocations)

Tool calls from the model sent as OpenAIFunctionCallInput:

{
  "type": "function_call",
  "call_id": "call_abc123",
  "name": "search_files",
  "arguments": "{\"query\":\"test\",\"path\":\"./src\"}"
}

Key Points:

• type: Always "function_call"
• call_id: Unique identifier generated by model
• name: Function name to execute
• arguments: JSON-encoded string of parameters
• status: Optional ("in_progress", "completed", "incomplete")
• Internal toolusedata field is removed during cleaning

3. Function Call Output Objects (Tool Results)

Tool execution results sent as OpenAIFunctionCallOutputInput:

{
  "type": "function_call_output",
  "call_id": "call_abc123",
  "output": "Found 3 files matching query"
}

Key Points:

• type: Always "function_call_output"
• call_id: Must match the original function call's call_id
• output: Can be text, image array, or error object

Output Value Types

Text Output

{
  "type": "function_call_output",
  "call_id": "call_abc123",
  "output": "Result text here"
}

Image Output

{
  "type": "function_call_output",
  "call_id": "call_abc123",
  "output": [
    {
      "type": "input_image",
      "image_url": "data:image/png;base64,..."
    }
  ]
}

Error Output

{
  "type": "function_call_output",
  "call_id": "call_abc123",
  "output": "{\"ok\":\"false\",\"error\":\"File not found\"}"
}

• Error output is a JSON-encoded string containing ok and error fields

Complete Example

{
  "model": "gpt-4o",
  "input": [
    {
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "What files are in src/?"
        }
      ]
    },
    {
      "type": "function_call",
      "call_id": "call_xyz789",
      "name": "list_files",
      "arguments": "{\"path\":\"src/\"}"
    },
    {
      "type": "function_call_output",
      "call_id": "call_xyz789",
      "output": "main.go\nutil.go\nconfig.go"
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "The src/ directory contains 3 files: main.go, util.go, and config.go"
        }
      ]
    }
  ],
  "stream": true,
  "max_output_tokens": 4096
}

Cleaning Process

Before transmission, internal fields are removed (cleanup code):

• Messages: previewurl field removed, filename removed from input_image blocks
• Function Calls: toolusedata field removed
• Function Outputs: Sent as-is (no cleaning needed)

This ensures the API receives only the fields it expects.