Senior AI Engineer - APM Experiences
The opportunity Datadog’s APM Experiences team owns the core product experience for Application Performance Monitoring — including distributed tracing, service representation, and more. We’re building a new wave of AI-powered capabilities that help customers detect, resolve, and prevent performance issues faster.
What this role actually needs.
Senior AI Engineer - APM Experiences at Datadog in New York City. UpJobz keeps this listing high-signal for applicants targeting serious high-tech roles across the United States, Canada, and Mexico. The opportunity Datadog’s APM Experiences team owns the core product experience for Application Performance Monitoring — including distributed tracing, service representation, and more. We’re building a new wave of AI-powered capabilities that help customers detect, resolve, and prevent performance issues faster.
Day-to-day expectations
A clear list of the work this role is designed to cover.
- Shape AI experiences for APM. Design and ship LLM/agentic workflows that analyze traces, metrics, logs, and other telemetry to generate diagnoses, explanations, and guided fixes.
- Own the full loop. Prototype quickly, define success metrics and evals, run experiments, iterate, and ultimately productionize for scale and reliability.
- Build robust agent systems. Develop tools, retrieval and planning strategies, and guardrails; manage prompts/evals; design fallbacks and human‑in‑the‑loop paths.
- Integrate with Datadog’s platform. Leverage surfaces like Trace Explorer, Service Catalog, monitors, and workflows to deliver end‑to‑end value in the APM UI.
- Partner deeply. Collaborate with PM, Design, and partner teams to build cohesive experiences.
- Raise the bar on engineering. Write performant, maintainable backend code, own services in production, and improve reliability for high‑throughput, low‑latency data systems.
What a strong candidate brings
This keeps the job page specific, readable, and easier to match.
- 4+ years building backend or real-time ML systems; you value simplicity, correctness, and performance
- Proven experience delivering LLM/agent features to production (prompting, tooling, evals, safety/guardrails)
- Comfortable owning user journeys, iterating from prototype → alpha → GA, and measuring impact with clear product metrics
- You have demonstrated ability to use AI coding tools in day-to-day workflows and validate, critique, and refine AI-generated output
- You’re motivated to push the boundaries of how AI can improve software engineering best practices and contribute to building AI-enabled products
- Solid grasp of the ML lifecycle (task definition, dataset collection, modeling, evaluation, deployment, iteration) and statistics (experiment design, confidence intervals)
Why people would want this job
Benefits help searchers understand whether the role is a real fit before they apply.
- Get to build tools for software engineers, just like yourself. And use the tools we build to accelerate our development.
- Have a lot of influence on product direction and impact on the business.
- Work with skilled, knowledgeable, and kind teammates who are happy to teach and learn.
- Competitive global benefits.
- Continuous professional development.
Browse similar jobs
Turn this listing into an application plan.
This is the first pass at the premium UpJobz layer: a fast brief that helps serious applicants move with more clarity.
Next moves
- Tailor your resume around ai and llm instead of sending a generic application.
- Use the first two bullets of your application to connect your background directly to senior ai engineer - apm experiences is a high-signal on-site role in new york city, and it is most realistic for united states residents.
- Open the role quickly if it fits and bookmark three similar jobs before you leave the page.
Interview themes
Watchouts
- Compensation is hidden, so get range clarity in the first recruiter conversation.
- Use united states residents as part of your positioning so the recruiter does not have to infer it.
- Show concrete examples of succeeding in on-site environments.
Search intent signals for this listing
Helpful keyword hooks for serious tech searchers and future programmatic job pages.
Ready to move on this role?
This page keeps the application flow simple while giving you enough context to decide quickly and move.