Context

Datadog is a powerful service monitoring tool, but its costs can escalate quickly, sometimes accounting for a significant portion of a company's infrastructure budget. In this case, a client in the e-commerce industry, generating 14 million visits per month, found that Datadog made up 20% of their infrastructure costs. While the tool was valuable, this expense seemed disproportionate for a monitoring service.

‍

Problem Identification and Consequences

The primary challenge was to reduce Datadog expenses without sacrificing the benefits of its monitoring capabilities. The client was reluctant to abandon Datadog due to its excellent user interfaces for logs, metrics, and monitors. However, continuing with the current spending level would mean allocating a large portion of the infrastructure budget to a single tool, potentially limiting resources for other crucial areas.

‍

Solution Implementation

To address this issue, we implemented a systematic approach to optimize Datadog usage:

1. We analyzed the current expenses, breaking down costs by functionality (logs, metrics, API tests).
2. For custom metrics, we reviewed usage and removed unnecessary metrics and tags, reducing unique combinations.
3. We audited ingested logs, identifying and eliminating unnecessary log entries, such as repetitive logs and health checks.
4. We reassessed the need for synthetic monitoring, removing redundant tests.
5. We set up monitors to alert the team when Datadog usage exceeds expected limits.
6. We adjusted the contract strategy, avoiding overcommitment to estimated usage on yearly contracts.
7. We considered alternatives to Datadog for some services, weighing the cost-benefit of setting up open-source solutions.

‍

Business and Product Gains

Through these optimization efforts, we achieved significant cost savings:

1. Custom metrics usage was reduced by half, saving approximately 12,000 USD annually.
2. Overall Datadog budget was reduced by as much as 40%.
3. The team gained better visibility into their Datadog usage, allowing for ongoing optimization.
4. The client maintained the benefits of Datadog's monitoring capabilities while significantly reducing costs.
5. The optimization process provided insights that could be applied to future projects and other tools.

This approach demonstrated that it's possible to significantly reduce monitoring costs without compromising on the quality of service monitoring, allowing for more efficient allocation of infrastructure budget.

‍