Session logging

Chatbot executions in AIP Chatbot Studio are logged as structured events that can be exported to a Foundry streaming dataset for monitoring and analysis. Each message sent to a chatbot counts as one execution, and every execution is assigned a unique trace identifier that connects all related log entries.

Prerequisites

Exporting chatbot session logs requires the log export feature. See export permissions for the required roles. Chatbot execution logs contain user identifiers that can be traced back to the user who sent the message, so access to exported log data should be carefully managed.

To set up log exporting, follow the steps in the Configure logging documentation, including the recommended Apply markings step to apply security markings to the output streaming dataset to control access to log data.

Learn how to view logs in the platform with AIP observability.

Understanding chatbot execution logs

Each log entry contains common fields provided by the log schema, as well as chatbot-specific event data. The chatbot-specific data consists of an event_name and a payload describing what occurred during the execution.

Common log fields

The following fields are included in every log entry and are useful for filtering and correlating logs. Logs from products across the execution flow, such as function executions and language model usage, also share these fields, so you can use them to trace the complete execution request:

Chatbot event data

Each chatbot event includes an event_name identifying the type of event and a payload with event-specific fields. Every event payload includes a session_rid, which identifies the chatbot session. Use this to group all events from a single conversation.

Sample log structure

The following is a simplified example of a log entry for a user_request event:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
  "time": "2024-01-15T09:30:00.000Z",
  "uid": "<USER_ID>",
  "traceId": "<TRACE_ID>",
  "content": {
    "event_name": "user_request",
    "payload": {
      "session_rid": "ri.aip-agents..session.<SESSION_ID>",
      "user_query": {
        "content": [
          {
            "text": {
              "content": "Summarize the key points from the latest press conference transcript."
            }
          }
        ]
      }
    }
  }
}

Event types

The following event types are logged during a chatbot execution:

Session metadata

Logged once at the start of each execution with identifying information about the chatbot and session.

User request

Captures the full context of a user's message, including any retrieval context and application variables that were resolved for this execution.

Chat messages

The system_chat_message, user_chat_message, and assistant_chat_message events share a similar structure.

Tool calls

The tool_call event records when the chatbot invokes a tool.

Tool call results

The tool_call_result event records the outcome of a tool invocation.

The ToolResult type is one of:

• Success: Contains an llm_value (String) with the value returned to the LLM, and a variable_updates list with the variable name and new value for any application variables that were updated.
• Failure: Contains an error_message (String) and a reason (String) describing why the tool call failed.

Final response

Contains the chatbot's final response to the user.

The FinalResponse type is one of:

• Chatbot response: Contains a list of content items, which may include text and media.
• Client tool call: Indicates the chatbot is deferring to a client-side action rather than returning a direct response.

Execution errors

Logged when an error occurs during execution.

Examples

Reconstruct a conversation

To reconstruct a chat conversation with a specific chatbot:

1. Filter logs where owning_rid matches the chatbot's RID.
2. Filter further by session_rid to isolate a single conversation.
3. For user messages, find entries where event_name is user_request and extract user_query from the payload.
4. For chatbot replies, find entries where event_name is final_response and extract response from the payload.

Evaluate tool performance

To analyze how tools are performing:

1. Find entries where event_name is tool_call_result. Each entry includes the tool_name, duration_milliseconds, and whether the call succeeded or failed.
2. To measure reliability, compare the count of success results versus failure results per tool.
3. To identify slow tools, sort by duration_milliseconds.
4. To detect retries, look for multiple tool_call and tool_call_result pairs with the same tool_name within a single traceId. A failure followed by another call to the same tool indicates the chatbot retried.