New Relic: Application Performance Monitoring Explained

In this tutorial, you'll learn about New Relic: Application Performance Monitoring Explained. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

What You Will Learn

This tutorial teaches you how to set up New Relic for application performance monitoring, configure the APM agent for a web application, use distributed tracing, and create custom dashboards and alerts.

Why It Matters

Application performance directly impacts revenue and user retention. A 500ms increase in page load time can reduce conversion rates by 20%. New Relic helps you identify the exact line of code causing a slowdown before it affects users.

Real-World Use

The DodaTech engineering team uses New Relic to monitor the DodaZIP compression API. When the p99 latency spiked from 200ms to 2s after a deployment, the New Relic APM trace pointed directly to a new regular expression function that was doing catastrophic Backtracking. The team rolled back the change in 10 minutes.

New Relic is a full-stack Observability platform. It provides APM, infrastructure monitoring, log management, browser monitoring, and synthetic monitoring. The New Relic agent runs inside your application process and collects performance data with minimal overhead.

Prerequisites

• A New Relic account (free tier available with 100 GB/month data ingest)
• An application to instrument (Python Flask or Node.js Express)
• Access to deploy the New Relic agent
• Basic understanding of Datadog or APM concepts is helpful

Step-by-Step Tutorial

Step 1: Get Your New Relic License Key

1. Log in to New Relic
2. Go to API Keys
3. Copy your INGEST_LICENSE key

Step 2: Set Up Environment Variables

export NEW_RELIC_LICENSE_KEY=your_license_key_here
export NEW_RELIC_APP_NAME="My Python App"
export NEW_RELIC_LOG=stdout

Step 3: Install the New Relic Python Agent

pip install newrelic

Create newrelic.ini:

[newrelic]
license_key = ${NEW_RELIC_LICENSE_KEY}
app_name = My Python App
log_level = info
monitor_mode = true

Step 4: Instrument a Flask Application

Create app.py:

from flask import Flask
import newrelic.agent
import time
import random

app = Flask(__name__)

@app.route("/")
def home():
    return {"service": "flask-api", "status": "healthy"}

@app.route("/api/users")
def get_users():
    time.sleep(random.uniform(0.05, 0.3))
    return {"users": ["alice", "bob", "charlie"]}

@app.route("/api/process")
def process_data():
    start = time.time()
    time.sleep(random.uniform(0.1, 0.8))
    duration = time.time() - start
    return {"processed": True, "duration_ms": round(duration * 1000)}

if __name__ == "__main__":
    app.run(port=5000)

Run the application:

NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program python app.py

Step 5: View APM Data

1. In New Relic, go to APM > Applications
2. Select "My Python App"
3. View the Service Overview dashboard: throughput, error rate, and response time

Step 6: Add Custom Instrumentation

import newrelic.agent

@app.route("/api/custom")
@newrelic.agent.background_task(name="custom_task", group="Task")
def custom_work():
    time.sleep(0.5)
    return {"task": "completed"}

Step 7: Add Distributed Tracing Configuration

Set these environment variables:

export NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true
export NEW_RELIC_SPAN_EVENTS_ENABLED=true

Restart the application. Traces now show the full request path across services.

Step 8: Create Custom Dashboards

1. Go to Dashboards > Create a dashboard
2. Add a chart:
   • Select "APM: Transaction overview"
   • Query: SELECT average(duration) FROM Transaction WHERE appName = 'My Python App' TIMESERIES
3. Add a second chart:
   • Query: SELECT count(*) FROM TransactionError WHERE appName = 'My Python App' TIMESERIES
4. Add a third chart showing the top 5 slowest transactions:
   • Query: SELECT average(duration) FROM Transaction WHERE appName = 'My Python App' FACET name LIMIT 5

Step 9: Set Up Alerts

1. Go to Alerts & AI > Alert policies > New alert policy
2. Add a condition:
   • Product: APM
   • Event type: Transaction
   • Metric: Response time average
   • Threshold: > 1 second for 5 minutes
3. Set notification channel: email, Slack, or PagerDuty

Learning Path

flowchart LR
    A[New Relic Account] --> B[Install Agent]
    B --> C[Instrument App]
    C --> D[APM Dashboard]
    D --> E[Distributed Tracing]
    E --> F[Custom Dashboards]
    F --> G[Alerts]
    C -.-> H[Browser Monitoring]
    C -.-> I[Infrastructure]
    style A fill:#4a90d9,color:#fff
    style G fill:#e67e22,color:#fff

Common Errors

1. Application does not appear in New Relic -- The agent failed to connect. Check NEW_RELIC_LICENSE_KEY is correct and network access to collector.newrelic.com is allowed.

2. No Transaction data after 10 minutes -- The agent is not instrumenting the web framework. Verify Flask or Django instrumentation is loaded with newrelic-admin --version.

3. Distributed tracing shows incomplete traces -- The downstream service does not have distributed tracing enabled. Ensure all services set NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true.

4. High overhead from custom instrumentation -- Too many custom segments are being created. Use background tasks only for significant operations.

5. NRQL query returns no data -- The metric or event type name is wrong. Use the Data Explorer to browse available event types.

6. Agent log shows "license key is invalid" -- The license key is expired or copied incorrectly. Generate a new key from the New Relic API Keys page.

7. Browser monitoring data does not appear -- The browser agent snippet is not installed on the page. Copy the auto-injection HTML snippet from APM > Settings.

Practice Questions

1. What is the role of the New Relic agent? Answer: The agent runs inside the application process, collecting performance data and sending it to the New Relic backend.

2. How do you add custom instrumentation to a function? Answer: Use the @newrelic.agent.background_task() decorator to wrap the function.

3. What is NRQL and how is it used? Answer: NRQL is New Relic Query Language, similar to SQL, used to query event data for custom dashboards.

4. How does distributed tracing work across multiple services? Answer: The agent injects trace context into HTTP headers. Downstream services extract the context and continue the trace.

5. What is the difference between a Transaction and a span? Answer: A Transaction represents the entire web request. Spans are the individual segments within a Transaction (database calls, external calls, etc.).

Challenge

Set up New Relic APM for a three-tier application (Node.js frontend, Python API, and a PostgreSQL database). Instrument all three services and enable distributed tracing. Create a load test that generates at least 100 requests per minute for 10 minutes. Build a New Relic dashboard with: throughput and error rate for each service, a distributed tracing map showing service dependencies, the top 5 slowest database queries, and a heatmap of response times by hour. Set up an alert that triggers when the error rate exceeds 2% for any service. Verify the alert fires by introducing an intentional error in one service.

FAQ