LLM Observability: Monitoring AI Applications in Production

Frequently Asked Questions

01.What is the difference between LLM monitoring and LLM observability?

Monitoring tracks preset signals such as latency, errors and uptime. Observability adds the records needed to explain one specific answer: the trace of retrieval, tool and model calls, the token count and cost, and a quality score. A practical test is whether the team can reproduce a past bad answer from stored logs. In a study of production agent builders, reliability, meaning consistent correct behaviour over time, was the top development challenge (arXiv, December 2025).

02.Does the EU AI Act require logging for LLM applications?

Only for high-risk systems. Article 12 requires high-risk AI systems to allow the automatic recording of events over their lifetime, and Article 26(6) requires deployers to keep those logs for at least six months unless other Union or national law, including data-protection law, provides otherwise. Under the Digital Omnibus on AI, these duties apply to Annex III systems from 2 December 2027 (Regulation (EU) 2026/1744, July 2026). A general customer chatbot usually sits outside Annex III.

03.Should prompts and outputs be stored in observability logs?

Store them deliberately, not by default. The OpenTelemetry conventions for generative AI say instrumentations should not capture prompts or outputs by default, and recommend storing content externally with references on the trace when production data is sensitive (OpenTelemetry, 2026). That keeps operational dashboards apart from personal data, which GDPR says must be kept no longer than necessary for its purpose.