Getting started with Feedback

Capturing feedback enables you to understand your users better and flag issues before they become problems. With log10, you can define complex feedback tasks and log feedback from end users over the API.

End-user Feedback

A common use case is to capture end-user sentiment about a generation, such as a chatbot response or a summary with thumbs up or down, or wider ranges of sentiment like emojis. This can be used to improve the quality of your LLM application through prompt enginering or model fine-tuning.

First a log10 user defines a feedback task with a schema that captures the feedback they want to collect.
Within your application, the LLM output is logged and then shown to the user. The logs are tagged with a unique identifier defined by the you e.g. internal session id.
User receives the LLM output.
User provides feedback on the LLM output. This could be a simple thumbs up or down, or a more complex feedback form. This is captured using the log10 SDK or API.
Feedback can now be reviewed in the log10 dashboard or downloaded for further analysis.

End-user Feedback

Beyond simple sentiment, another common workflow is to do internal evaluations of the quality of the completions generated by your models. In that setting, more complex feedback tasks can be defined to capture multiple aspects of the completion quality, such as relevance, coherence, and fluency on pre-defined scales.

First a log10 user defines a feedback task with a schema that captures the feedback they want to collect. In addition to the feedback schema, the user can also define a tag selector to filter logs that should be matched to this feedback task. Logs will start appearing in a shared feedback inbox.
Logs are generated by the LLM and tagged with the unique identifier defined by the user.
User receives the LLM output.
A log10 reviewer reviews the completion and provides feedback on the quality of the completion. This is captured using the log10 UI, SDK or API.
Feedback can now be reviewed in the log10 dashboard or downloaded for further analysis.

We will walk you through the process of setting up a minimal examples of the first use-case by creating feedback task, logging feedback, and viewing it in the log10 dashboard in this guide. For more in-depth information, see the feedback API reference.

Install the log10 client library

$ pip install log10-io

Then add your log10 credentials to your environment. See here (opens in a new tab) for more information on how to get your credentials.

$ source ~/.bashrc

Create a task

Tasks define the structure of the feedback you want to capture using JSON schema. You can define the fields and their types, and set default values and validation rules.

$ log10 feedback task create --name emoji_feedback_task --json-schema {"type": "object", "properties": {"feedback": {"type": "string", "enum": ["😀", "😬", "😐", "🙁", "😫"]}}, "required": ["feedback"]}' --completion-tags-selector "unique-tag" --instruction "Provide feedback using emojis"

This will return a task id that you can use to log completions. Save it for later.

$ export TASK_ID=<TASK ID FROM CREATE CALL>

Log completions

We map feedback to completions using tags and tag selectors. Tag selectors are used to filter completions based on tags.

To make sure you don't accidentially apply feedback to the wrong completions, you can use a unique tag for each completion (or group of completions). For instance, tracking it with your internal session ids.

To run a test completion which applies unique-tag, save the following to example.py

from log10.load import OpenAI
 
client = OpenAI(tags=["unique-tag"])
response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Count to 200"}],
    temperature=0,
)
 
print(response)

And run it with this:

$ python example.py

Add feedback

You can now log feedback from your users using the task id you saved and attach it to the completion(s).

$ log10 feedback create --task_id $TASK_ID --values '
  {
    "feedback": "😀"
  }' --completion_tags_selector "unique-tag"

View feedback

You can view feedback in the log10 dashboard or using CLI.

Feedback screenshot

CLI

$ log10 feedback list --help
Usage: log10 feedback list [OPTIONS]
 
  List feedback based on the provided criteria. This command allows fetching
  feedback for a specific task or across all tasks, with control over the
  starting point and the number of items to retrieve.
 
Options:
  --offset INTEGER  The starting index from which to begin the feedback fetch.
                    Defaults to 0.
  --limit INTEGER   The maximum number of feedback items to retrieve. Defaults
                    to 25.
  --task_id TEXT    The specific Task ID to filter feedback. If not provided,
                    feedback for all tasks will be fetched.

Download feedback

You can download feedback using CLI.

$ log10 feedback download --help
Usage: log10 feedback download [OPTIONS]
 
  Download feedback based on the provided criteria. This command allows
  fetching feedback for a specific task or across all tasks, with control over
  the starting point and the number of items to retrieve.
 
Options:
  --offset INTEGER  The starting index from which to begin the feedback fetch.
                    Leave empty to start from the beginning.
  --limit TEXT      The maximum number of feedback items to retrieve. Leave
                    empty to retrieve all.
  --task_id TEXT    The specific Task ID to filter feedback. If not provided,
                    feedback for all tasks will be fetched.
  -f, --file TEXT   Path to the file where the feedback will be saved. The
                    feedback data is saved in JSON Lines (jsonl) format. If
                    not specified, feedback will be printed to stdout.

Reports Feedback Reference