Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
AI & knowledge

collect-user-feedback

Use this to capture user feedback on LLM outputs (thumbs up/down, edits, corrections, implicit signals) and feed it back into observability and evals. Trigger on "add thumbs up/down", "collect feedback on responses", "how do I know if users like the answers", "improve from real usage", "human feedback loop". Turn real user signal into your best source of eval data.

ContextJet-ai · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add ContextJet-ai/awesome-llm-observability --skill collect-user-feedback

About this skill


name: collect-user-feedback description: Use this to capture user feedback on LLM outputs (thumbs up/down, edits, corrections, implicit signals) and feed it back into observability and evals. Trigger on "add thumbs up/down", "collect feedback on responses", "how do I know if users like the answers", "improve from real usage", "human feedback loop". Turn real user signal into your best source of eval data. license: CC0-1.0

Collect user feedback

Your users are running the best eval you have, for free, every day. Capturing their reactions and attaching them to the trace turns production into a continuous source of labeled data. Most teams either skip this or collect it and never use it.

Capture both explicit and implicit signals

Explicit (ask directly):

  • Thumbs up/down on a response (the simplest, highest-signal control).
  • A short reason on thumbs-down (dropdown: wrong / unhelpful / unsafe / other).
  • Accept / edit / reject on a suggested output (great for copilot-style apps).

Implicit (infer from behavior, no extra UI):

  • Did the user retry, rephrase, or immediately give up? (dissatisfaction).
  • Did they copy/use the output, or continue the task? (satisfaction).
  • Did a downstream action succeed (the code ran, the ticket resolved)?

Implicit signals are noisier but you get them on 100% of traffic instead of the small % who click thumbs.

Wire it to the trace (this is the key step)

Attach every feedback signal to the trace/span it is about (by trace ID), with the user, timestamp, and the reason. Most observability platforms (Langfuse, Phoenix, Opik, LangSmith) have a feedback/scores API for exactly this. Without the trace link, feedback is a number with no context; with it, a thumbs-down is a fully debuggable example.

Use it (do not just collect it)

  1. Monitor feedback rate + thumbs-down rate over time; alert on spikes (see set-up-drift-alerts).
  2. Triage thumbs-down traces into candidate test cases (see trace-based-testing) and your eval set (build-eval-dataset).
  3. Correlate feedback with your automated eval scores to check your judge actually agrees with humans (calibration).
  4. Close the loop: the corrections/edits users make are gold-standard reference outputs, use them.

Handle it responsibly

  • PII / privacy: feedback text can contain sensitive data, redact before it hits a third-party backend (redact-pii-for-tracing).
  • Bias: thumbs are a biased sample (angry and delighted users click most). Do not treat thumbs-up rate as ground-truth quality; use it as signal, validate with evals.

Verify

  • A thumbs-down in the UI shows up attached to the right trace in your backend.
  • Thumbs-down traces are being triaged into the eval/regression set, not just counted.
  • A dashboard shows feedback trends with an alert on a drop.

Anti-patterns

  • Collecting feedback that never links to a trace (a number you cannot act on).
  • Collecting it and never feeding it back into evals or fixes.
  • Treating thumbs-up rate as objective quality (selection bias).
  • Logging raw feedback with PII to a SaaS backend without redaction.

Grounding

Human feedback as the signal for LLM quality underpins RLHF and preference modeling (Ouyang et al., InstructGPT, arXiv:2203.02155) and human-aligned evaluation (Zheng et al. 2023, arXiv:2306.05685). Here it feeds the lighter-weight eval + observability loop (trace-based-testing, build-eval-dataset).