Citations

AIP Agents with document context or Ontology context will output citations that link out to the source material when selected. These citations will also appear in a Sources dropdown menu at the bottom of each message, giving users a clear understanding of the specific context used and referenced in each response.

For other types of context, such as function-backed context or tools, citations are not provided by default. However, users can prompt the agent to include them. If the LLM responds with the correct citation format, the conversation user interface will render the citation. See citation formats below for the full list of formats, and see custom prompts for citations for examples on how to use these in your agent.

Citation formats

The AIP Agent interface currently supports the following citation formats:

• Ontology object citations can be returned in either of the two formats below. Selecting the citation bubble will link to the Object Explorer view for that object.

  Copied!
  1
  <citation><key>ri.phonograph2-objects.main.object....</key></citation>

  Where the <key></key> tags encapsulate the object RID.

  Copied!
  1
  <citation><objectTypeId>...</objectTypeId><primaryKey>...</primaryKey></citation>

  Where the <objectTypeId></objectTypeId> tags encapsulate the object type ID, which can be found in Ontology Manager, and the <primaryKey></primaryKey> tags encapsulate the value of the primary key for that object.

• Document (PDF) citations should be returned in the following format:

  Copied!
  1
  <citation><mediaSetKey>ri.mio.main.media-set...</mediaSetKey><mediaItemKey>ri.mio.main.media-item...</mediaItemKey></citation>

  Selecting the citation bubble will open a dialog that displays the first page of the document. If you would like to specify a page of the document in your citation, you can provide a page tag like so:

  Copied!
  1
  <citation><mediaSetKey>ri.mio.main.media-set...</mediaSetKey><mediaItemKey>ri.mio.main.media-item...</mediaItemKey><page>12</page></citation>

  The document dialog will then display that page of the document.

• External URL citations should be returned in the following format:

  Copied!
  1
  <citation><name>My Website</name><href>www.mywebsite.com</href></citation>

  The citation bubble will display the provided name, for example My website, and selecting it will link to the provided URL, for example www.mywebsite.com.

You can set up an agent with Ontology context to cite objects in any one of the three formats above by navigating to the Citations tab under the Ontology context configuration panel.

Citation variable updates

Agents with Ontology object citations can also be configured to update application variables when an object citation is selected. This allows consumers of AIP Agents to display additional information about cited objects outside of the conversation panel.

A common example of this is the AIP Interactive Widget in Workshop, where you can use the application state to configure pop-up overlays on selection. You can find a walkthrough on how to set this up in the integrating with Workshop overlays section below.

For agents with Ontology context, you can open the Variables tab to configure a citation output variable. Whenever a citation is selected, this variable will populate with an object set of one object, which will be the object referenced in the citation.

For agents that use function-backed context, you can prompt the agent to return object citations using one of the custom prompts below. Once this is done, you can navigate to the Citation variables tab in the configuration panel to set up a variable that will get populated whenever a citation is selected.

If your function deals with multiple object types, it may return citations where some correspond to object type A, others to object type B, and so on. To handle such cases, you can create multiple citation variables for your function. When a citation is selected, the citation variable that matches the object type will be set to an object set containing the referenced object, while all other citation variables will be set to an empty object set. This approach ensures that only the most relevant citation variable is populated at any given time.

Integrate with Workshop overlays

After configuring your variable updates in AIP Agent Studio, you can connect them to the AIP Interactive widget in Workshop. For more information on the AIP Interactive widget, refer to the section on application state.

1. In Workshop, create an empty object set variable for each citation variable you configured in your agent. These variables will be populated whenever a citation from the AIP Interactive widget is selected.
2. Create an overlay in the Workshop module.
3. Set the overlay to Variable-based visibility and create a Boolean variable that checks whether the Workshop variable you configured in step one is empty. If you have multiple citation variables, you can create multiple overlays or a single overlay with visibility determined by the union of multiple Boolean variables.

Pop-up overlays that appear on citation selection often provide the best user experience for users of the AIP Interactive widget. However, you can explore other options by navigating to the Workshop documentation.

Custom prompts for citations

To ensure citations are rendered in the AIP Agent user interface, the citations must be included in the response from the underlying LLM. To accomplish this, the LLM needs to be prompted using one of the formats listed above. This prompt is provided automatically for agents with document and Ontology context. However, for other agents, a custom prompt tailored to the specific use case is necessary.

You can provide a custom prompt to the agent in a few different ways. The first is to provide instructions using the LLM Settings for your agent. You can see an example of this with the agent below, which prompts the LLM to respond with citations every time it uses the object query tool for drivers.

Below is the prompt used in the example image above to output object citations from an agent with the object query tool:

Copied!
1
2
3
4
5
6
7
8
9
Whenever the user asks a question about support tickets, use the object query tool to answer it. After using the object query tool, respond to the user query with inline citations that link to the objects returned by the object query tool.

WHENEVER you mention a support ticket, output a citation in the format below:

"<citation><objectTypeId>titan-technologies-support-ticket</objectTypeId><primaryKey>...</primaryKey></citation>"

For example, if I wanted to cite the ticket titled Office Dress Code, I would get the primary key of the ticket, which is TICKET-015 (since the Ticket Id is TICKET-015), and then output the citation like so:

"The ticket about the office dress code says ... <citation><objectTypeId>titan-technologies-support-ticket</objectTypeId><primaryKey>TICKET-015</primaryKey></citation>.""

Another option is to use function-backed context, since the output of the context retrieval function is directly pasted into the LLM system prompt. In the example below, the context retrieval function generates a prompt that supplies the LLM with the relevant objects to include in its response, along with the format it should use to provide citations.

Below is the prompt from the example image above, used to output object citations from an agent with function-backed context:

Copied!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
Incorporate a citation from the context source in your answer whenever the context content is used.

For example, if the message asks "What is X?"
A valid response would be "X is Y, according to <citation><name>Source 1</name><key>ri.phonograph2-objects.main.object.abc-123</key></citation>, ..."

Another example: if the message asks "Why is X?"
A valid response would be: "X is Y because ... For more details or further clarification, please refer to the following sources: <citation><name>Source 2</name><key>ri.phonograph2-objects.main.object.def-456</key></citation> <citation><name>Source 3</name><key>ri.phonograph2-objects.main.object.abc-123</key></citation>"

Remember, always include a citation in your answer by using the provided context sources.

Example context sources:

<citation><name>Source 1</name><key>ri.phonograph2-objects.main.object.0c94f9b2-e0c5-4e90-a054-96e570cd11dd</key></citation>
Consistency is key when teaching commands to your dog.
<citation><name>Source 1</name><key>ri.phonograph2-objects.main.object.0c94f9b2-e0c5-4e90-a054-96e570cd11dd</key></citation>

<citation><name>Source 2</name><key>ri.phonograph2-objects.main.object.8c6adc92-cfc6-4b76-b729-2744e369dac3</key></citation>
Socialization is important for a dog's development and behavior.
<citation><name>Source 2</name><key>ri.phonograph2-objects.main.object.8c6adc92-cfc6-4b76-b729-2744e369dac3</key></citation>

Example response with citation:

"Maintaining consistency is essential when training your dog to follow commands <citation><name>Source 1</name><key>ri.phonograph2-objects.main.object.0c94f9b2-e0c5-4e90-a054-96e570cd11dd</key></citation>. Socializing your dog is crucial for its behavioral development <citation><name>Source 2</name><key>ri.phonograph2-objects.main.object.8c6adc92-cfc6-4b76-b729-2744e369dac3</key></citation>."

[Context Sources Sorted by Relevancy]
<citation><key>ri.phonograph2-objects.main.object.60599f54-c745-412b-98a0-2c7f5f0fa9bf</key></citation>
.... The Titan Technologies dress code is business casual ....
<citation><key>ri.phonograph2-objects.main.object.60599f54-c745-412b-98a0-2c7f5f0fa9bf</key></citation>

<citation><key>ri.phonograph2-objects.main.object.35910c26-8a87-4eb5-8243-52afb88bba11</key></citation>
.... Titan Technologies Employee Handbook ....
<citation><key>ri.phonograph2-objects.main.object.35910c26-8a87-4eb5-8243-52afb88bba11</key></citation>

<citation><key>ri.phonograph2-objects.main.object.3f21b8a6-536b-4374-aaf5-cae641ca836c</key></citation>
.... At Titan Technologies, flexible work hours may be possible .....
<citation><key>ri.phonograph2-objects.main.object.3f21b8a6-536b-4374-aaf5-cae641ca836c</key></citation>

<citation><key>ri.phonograph2-objects.main.object.10b2d64a-4043-438d-894a-80c001ff26e2</key></citation>
.... The dress code for the office is business casual ....
<citation><key>ri.phonograph2-objects.main.object.10b2d64a-4043-438d-894a-80c001ff26e2</key></citation>
[/Context Sources Sorted by Relevancy]

REMINDER:
ALWAYS include citations in the correct format <citation><name>sourceName</name><key>sourceKey</key></citation> in all responses. ALWAYS include the source name and the source key in all citations.

Valid examples are <citation><name>Source 1</name><key>ri.phonograph2-objects.main.object.0c94f9b2-e0c5-4e90-a054-96e570cd11dd</key></citation>, <citation><name>Source 2</name><key>ri.phonograph2-objects.main.object.8c6adc92-cfc6-4b76-b729-2744e369dac3</key></citation>.