waveterm/aiprompts/openai-request.md

# OpenAI Request Input Field Structure (On-the-Wire Format)

This document describes the actual JSON structure sent to the OpenAI API in the `input` field of [`OpenAIRequest`](../pkg/aiusechat/openai/openai-convertmessage.go:111).

## Overview

The `input` field is a JSON array containing one of three object types:

1. **Messages** (user/assistant) - `OpenAIMessage` objects
2. **Function Calls** (tool invocations) - `OpenAIFunctionCallInput` objects
3. **Function Call Results** (tool outputs) - `OpenAIFunctionCallOutputInput` objects

These are converted from [`OpenAIChatMessage`](../pkg/aiusechat/openai/openai-backend.go:46-52) internal format and cleaned before transmission ([see lines 485-494](../pkg/aiusechat/openai/openai-backend.go:485-494)).

## 1. Message Objects (User/Assistant)

User and assistant messages sent as [`OpenAIMessage`](../pkg/aiusechat/openai/openai-backend.go:54-57):

```json
{
  "role": "user",
  "content": [
    {
      "type": "input_text",
      "text": "Hello, analyze this image"
    },
    {
      "type": "input_image",
      "image_url": "data:image/png;base64,iVBORw0KG..."
    }
  ]
}
```

**Key Points:**
- `role`: Always `"user"` or `"assistant"`
- `content`: **Always an array** of content blocks (never a plain string)

### Content Block Types

#### Text Block
```json
{
  "type": "input_text",
  "text": "message content here"
}
```

#### Image Block
```json
{
  "type": "input_image",
  "image_url": "data:image/png;base64,..."
}
```
- Can be a data URL or https:// URL
- `filename` field is **removed** during cleaning

#### PDF File Block
```json
{
  "type": "input_file",
  "file_data": "JVBERi0xLjQKJeLjz9M...",
  "filename": "document.pdf"
}
```
- `file_data`: Base64-encoded PDF content

#### Function Call Block (in assistant messages)
```json
{
  "type": "function_call",
  "call_id": "call_abc123",
  "name": "search_files",
  "arguments": {"query": "test"}
}
```

## 2. Function Call Objects (Tool Invocations)

Tool calls from the model sent as [`OpenAIFunctionCallInput`](../pkg/aiusechat/openai/openai-backend.go:59-67):

```json
{
  "type": "function_call",
  "call_id": "call_abc123",
  "name": "search_files",
  "arguments": "{\"query\":\"test\",\"path\":\"./src\"}"
}
```

**Key Points:**
- `type`: Always `"function_call"`
- `call_id`: Unique identifier generated by model
- `name`: Function name to execute
- `arguments`: JSON-encoded string of parameters
- `status`: Optional (`"in_progress"`, `"completed"`, `"incomplete"`)
- Internal `toolusedata` field is **removed** during cleaning

## 3. Function Call Output Objects (Tool Results)

Tool execution results sent as [`OpenAIFunctionCallOutputInput`](../pkg/aiusechat/openai/openai-backend.go:69-75):

```json
{
  "type": "function_call_output",
  "call_id": "call_abc123",
  "output": "Found 3 files matching query"
}
```

**Key Points:**
- `type`: Always `"function_call_output"`
- `call_id`: Must match the original function call's `call_id`
- `output`: Can be text, image array, or error object

### Output Value Types

#### Text Output
```json
{
  "type": "function_call_output",
  "call_id": "call_abc123",
  "output": "Result text here"
}
```

#### Image Output
```json
{
  "type": "function_call_output",
  "call_id": "call_abc123",
  "output": [
    {
      "type": "input_image",
      "image_url": "data:image/png;base64,..."
    }
  ]
}
```

#### Error Output
```json
{
  "type": "function_call_output",
  "call_id": "call_abc123",
  "output": "{\"ok\":\"false\",\"error\":\"File not found\"}"
}
```
- Error output is a JSON-encoded string containing `ok` and `error` fields

## Complete Example

```json
{
  "model": "gpt-4o",
  "input": [
    {
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "What files are in src/?"
        }
      ]
    },
    {
      "type": "function_call",
      "call_id": "call_xyz789",
      "name": "list_files",
      "arguments": "{\"path\":\"src/\"}"
    },
    {
      "type": "function_call_output",
      "call_id": "call_xyz789",
      "output": "main.go\nutil.go\nconfig.go"
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "The src/ directory contains 3 files: main.go, util.go, and config.go"
        }
      ]
    }
  ],
  "stream": true,
  "max_output_tokens": 4096
}
```

## Cleaning Process

Before transmission, internal fields are removed ([cleanup code](../pkg/aiusechat/openai/openai-backend.go:485-494)):

- **Messages**: `previewurl` field removed, `filename` removed from `input_image` blocks
- **Function Calls**: `toolusedata` field removed
- **Function Outputs**: Sent as-is (no cleaning needed)

This ensures the API receives only the fields it expects.