• Sat. Jun 15th, 2024

Datadog brings OpenAI model monitoring into the fold, launches new integration


May 10, 2023
Datadog brings OpenAI model monitoring into the fold, launches new integration


Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success. Learn More

New York-based Datadog, which provides a cloud observability platform for enterprise applications and infrastructure, today announced an integration to monitor OpenAI models such as GPT-4.

The offering, Datadog says, will help enterprise teams understand user interactions with their GPT-powered applications, ultimately enabling them to fine-tune the models for better performance and economies.

It comes as OpenAI’s large language models continue to see adoption across a variety of enterprise-specific use cases, including business-critical areas such as customer service and data querying.

How does the OpenAI integration help?

Once up and running, the Datadog-OpenAI integration automatically tracks GPT usage patterns, providing teams with actionable insights into model performance and costs via dashboards and alerts.


Transform 2023

Join us in San Francisco on July 11-12, where top executives will share how they have integrated and optimized AI investments for success and avoided common pitfalls.


Register Now

For performance, the plugin looks at OpenAI API error rates, rate limits, and response times, allowing users to identify and isolate issues within their applications. It also offers the ability to view OpenAI request volumes – along with metrics, traces, and logs containing prompts and corresponding completions – to understand how end customers are interacting with the applications as well as gauge output quality generated by their OpenAI models. 

“Customers can install the integration by instrumenting the OpenAI Python library to emit metrics, traces and logs for requests made to the completions, chat completions and embedded endpoints. Once instrumented, the metrics, traces and logs will be automatically available in the out-of-the-box dashboard provided by Datadog,” Yrieix Garnier, VP of product at Datadog, tells VentureBeat. 

These dashboards can then be customized to drill down further into performance issues and optimize the models for improved user experience, the VP added.

On the costs front, Datadog says, the integration allows users to review token allocation by model or service and analyze the associated costs of OpenAI API calls. This can then be used to manage expenses more effectively and avoid unexpected bills for using the service.

While Garnier did confirm that customers of both companies are testing the integration, he did not share specific results witnessed by them so far. The connector currently works for multiple AI models from OpenAI, including the GPT family of LLMs, Ada, Babbage, Curie and Davinci.

New Relic also offers something similar

New Relic, another player in the observability space, also offers a similar OpenAI integration that tracks API response time, average tokens per request, and the cost associated. However, Garnier claims Datadog’s offering also covers additional elements like response-time-to-prompt token ratio as well as metrics providing contextual insights into individual user queries. 

“Furthermore, for API response times, API requests, and other metrics, we allow users to break this down by model, service, and API keys. This is critical in order to understand the primary drivers of usage, token consumption and cost,” he noted.

Moving ahead, monitoring solutions like these, including those specifically tracking hallucinations, are expected to witness an increase in demand, given the meteoric rise of large language models within enterprises. Companies are either using or planning to use LLMs (most prominently those from OpenAI) to accelerate key business functions, starting from querying their data stack to optimizing customer service.

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Discover our Briefings.

Leave a Reply

Your email address will not be published. Required fields are marked *