APM — Java Application Performance Monitoring

In production, answering the question "why is it slow?" requires APM (Application Performance Monitoring). APM goes beyond simple logs or server metrics: it traces exactly which services an individual HTTP request passed through, how long each hop took, and where the bottleneck occurred. This chapter covers monitoring Spring Boot application performance using Elastic APM and Pinpoint.

What Is APM?

APM measures an application's performance and reliability in real time. Its three core capabilities are:

Distributed Tracing connects a single user request across multiple microservices into one unified trace. When a request flows through "API Gateway → User Service → Order Service → DB", you can immediately see that a specific hop consumed 200 ms.

Slow Query Detection automatically identifies operations—DB queries, external API calls, cache misses—whose response times exceed a defined threshold.

Error Tracking records every exception with its full stack trace along with contextual information: which user made the request, what the request looked like, and exactly when the error occurred.

User Request
    │
    ├── Span 1: Nginx (2ms)
    ├── Span 2: Spring Boot Controller (15ms)
    │       ├── Span 3: Service Layer (12ms)
    │       │       ├── Span 4: JPA Query (8ms)  ← slow query
    │       │       └── Span 5: Redis Cache (1ms)
    │       └── Span 6: External HTTP Call (3ms)
    └── Total: 33ms

Setting Up Elastic APM

Docker Compose Setup (APM Server + Elasticsearch + Kibana)

# docker-compose.yml (with APM)
version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - ELASTIC_PASSWORD=changeme123!
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - apm
    healthcheck:
      test: ["CMD-SHELL", "curl -s -u elastic:changeme123! http://localhost:9200/_cluster/health | grep -q yellow\\|green"]
      interval: 20s
      timeout: 10s
      retries: 10

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=changeme123!
    ports:
      - "5601:5601"
    networks:
      - apm
    depends_on:
      elasticsearch:
        condition: service_healthy

  apm-server:
    image: docker.elastic.co/apm/apm-server:8.12.0
    container_name: apm-server
    command: >
      apm-server -e
        -E apm-server.rum.enabled=true
        -E setup.kibana.host=kibana:5601
        -E setup.template.settings.index.number_of_replicas=0
        -E apm-server.kibana.enabled=true
        -E apm-server.kibana.host=kibana:5601
        -E output.elasticsearch.hosts=["elasticsearch:9200"]
        -E output.elasticsearch.username=elastic
        -E output.elasticsearch.password=changeme123!
    ports:
      - "8200:8200"
    networks:
      - apm
    depends_on:
      elasticsearch:
        condition: service_healthy

volumes:
  es_data:

networks:
  apm:
    driver: bridge

# Start the stack
docker-compose up -d

# Verify APM Server is ready
curl http://localhost:8200/

# Expected response:
# {"build_date":"2024-01-15T...","build_sha":"...","publish_ready":true,"version":"8.12.0"}

Attaching the Elastic APM Agent to Spring Boot

Step 1: Download the APM agent JAR

wget -O elastic-apm-agent.jar \
  https://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/1.49.0/elastic-apm-agent-1.49.0.jar

Step 2: Add Elastic APM API dependency to pom.xml (optional — for custom spans)

<!-- pom.xml -->
<dependency>
    <groupId>co.elastic.apm</groupId>
    <artifactId>apm-agent-api</artifactId>
    <version>1.49.0</version>
</dependency>

<!-- Spring Boot Actuator for metrics exposure -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Step 3: Configure application.properties

# src/main/resources/application.properties

# ─── Elastic APM ─────────────────────────────────────────────────────────
elastic.apm.service_name=my-spring-app
elastic.apm.server_url=http://localhost:8200
elastic.apm.environment=production
elastic.apm.application_packages=com.example.myapp
elastic.apm.log_level=INFO

# Trace sampling rate (1.0 = 100%, 0.1 = 10%)
elastic.apm.transaction_sample_rate=1.0

# Minimum span duration to capture
elastic.apm.span_min_duration=5ms

# Stack trace frame limit
elastic.apm.stack_trace_limit=50

# ─── Spring Boot Actuator ────────────────────────────────────────────────
management.endpoints.web.exposure.include=health,info,metrics,prometheus
management.endpoint.health.show-details=always

Step 4: Attach the agent via JVM options

# Running directly
java -javaagent:/path/to/elastic-apm-agent.jar \
     -Delastic.apm.service_name=my-spring-app \
     -Delastic.apm.server_url=http://localhost:8200 \
     -Delastic.apm.environment=production \
     -Delastic.apm.application_packages=com.example.myapp \
     -jar my-spring-app.jar

# Running in Docker — add to the service in docker-compose.yml:
#   environment:
#     - JAVA_TOOL_OPTIONS=-javaagent:/app/elastic-apm-agent.jar
#   volumes:
#     - ./elastic-apm-agent.jar:/app/elastic-apm-agent.jar

Step 5: Custom tracing for specific methods

// src/main/java/com/example/myapp/service/OrderService.java
package com.example.myapp.service;

import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Span;
import co.elastic.apm.api.Transaction;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

    private final OrderRepository orderRepository;
    private final PaymentClient paymentClient;

    public OrderService(OrderRepository orderRepository, PaymentClient paymentClient) {
        this.orderRepository = orderRepository;
        this.paymentClient = paymentClient;
    }

    public Order processOrder(Long userId, OrderRequest request) {
        // Add custom labels to the current transaction
        Transaction transaction = ElasticApm.currentTransaction();
        transaction.setLabel("user_id", userId);
        transaction.setLabel("product_count", request.getItems().size());

        // Track the payment service call as a dedicated Span
        Span paymentSpan = transaction.startSpan("external", "payment", "charge");
        try {
            paymentSpan.setName("Payment Service - charge");
            PaymentResult result = paymentClient.charge(request.getAmount());
            return orderRepository.save(new Order(userId, request, result));
        } catch (Exception e) {
            // Record the error in APM with full context
            paymentSpan.captureException(e);
            throw e;
        } finally {
            paymentSpan.end();
        }
    }
}

Kibana APM UI

Once the APM Server collects data, navigate to Kibana → APM to explore:

Service Map: Automatically visualizes dependencies between microservices. Each service node shows response time and throughput at a glance.

Transactions: Shows average response time, throughput (req/min), and error rate per HTTP endpoint. The slowest transactions list immediately reveals bottlenecks.

Traces: Displays the complete flow of an individual request as a waterfall chart, with each Span's duration and order visible at a glance.

Errors: Groups exceptions by stack trace, occurrence frequency, and number of affected users.

Pinpoint APM Overview and Setup

Pinpoint is an open-source APM developed by Naver, specialized in distributed tracing. It instruments applications automatically using bytecode instrumentation — no code changes required.

# docker-compose.pinpoint.yml
version: '3.8'

services:
  # HBase (Pinpoint data store)
  hbase:
    image: dajobe/hbase:latest
    container_name: pinpoint-hbase
    ports:
      - "16000:16000"
      - "16010:16010"
      - "16020:16020"
      - "16030:16030"
      - "2181:2181"
    networks:
      - pinpoint

  # Pinpoint Collector
  pinpoint-collector:
    image: pinpointdocker/pinpoint-collector:2.5.3
    container_name: pinpoint-collector
    environment:
      - HBASE_HOST=hbase
      - CLUSTER_ENABLE=false
    ports:
      - "9991:9991/udp"
      - "9992:9992/udp"
      - "9993:9993/udp"
      - "9994:9994"
      - "9995:9995/udp"
    networks:
      - pinpoint
    depends_on:
      - hbase

  # Pinpoint Web UI
  pinpoint-web:
    image: pinpointdocker/pinpoint-web:2.5.3
    container_name: pinpoint-web
    environment:
      - HBASE_HOST=hbase
      - CLUSTER_ENABLE=false
    ports:
      - "8080:8080"
    networks:
      - pinpoint
    depends_on:
      - hbase

networks:
  pinpoint:
    driver: bridge

Attaching the Pinpoint Java agent:

# Download the agent
wget https://github.com/pinpoint-apm/pinpoint/releases/download/v2.5.3/pinpoint-agent-2.5.3.tar.gz
tar xzf pinpoint-agent-2.5.3.tar.gz

# Edit pinpoint-root.config:
# profiler.collector.ip=localhost
# profiler.sampling.rate=1  (1/1 = 100% sampling)

# Attach the agent
java -javaagent:pinpoint-agent-2.5.3/pinpoint-bootstrap-2.5.3.jar \
     -Dpinpoint.agentId=my-spring-app-001 \
     -Dpinpoint.applicationName=MY-SPRING-APP \
     -Dpinpoint.config=pinpoint-agent-2.5.3/profiles/local/pinpoint-root.config \
     -jar my-spring-app.jar

Spring Boot Actuator + Micrometer + Prometheus

Expose metrics in Prometheus format using Actuator and Micrometer.

<!-- pom.xml -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.properties
management.endpoints.web.exposure.include=health,info,metrics,prometheus
management.endpoint.metrics.enabled=true
management.metrics.export.prometheus.enabled=true
management.metrics.web.server.request.autotime.enabled=true
management.metrics.tags.application=${spring.application.name}

// src/main/java/com/example/myapp/config/MetricsConfig.java
package com.example.myapp.config;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class MetricsConfig {

    // Business metric: total orders processed
    @Bean
    public Counter orderCounter(MeterRegistry registry) {
        return Counter.builder("orders.processed.total")
                .description("Total number of orders processed")
                .tag("status", "success")
                .register(registry);
    }

    // Payment processing duration
    @Bean
    public Timer paymentTimer(MeterRegistry registry) {
        return Timer.builder("payment.processing.duration")
                .description("Time taken to process payment")
                .publishPercentiles(0.5, 0.95, 0.99)
                .register(registry);
    }
}

// Using metrics in a service
@Service
public class OrderService {

    private final Counter orderCounter;
    private final Timer paymentTimer;

    public OrderService(Counter orderCounter, Timer paymentTimer) {
        this.orderCounter = orderCounter;
        this.paymentTimer = paymentTimer;
    }

    public Order processOrder(OrderRequest request) {
        return paymentTimer.record(() -> {
            Order order = doProcess(request);
            orderCounter.increment();
            return order;
        });
    }
}

Verify metrics:

# Confirm Prometheus metrics endpoint
curl http://localhost:8080/actuator/prometheus | grep orders_processed

# Sample output:
# orders_processed_total{application="my-spring-app",status="success"} 142.0

Slow Query Detection

# application.properties

# Hibernate slow query logging
spring.jpa.properties.hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS=100

# P6Spy integration (full SQL + execution time logging)
# Add to pom.xml:
# <dependency><groupId>p6spy</groupId><artifactId>p6spy</artifactId><version>3.9.1</version></dependency>
spring.datasource.driver-class-name=com.p6spy.engine.spy.P6SpyDriver
spring.datasource.url=jdbc:p6spy:mysql://localhost:3306/mydb

# Elastic APM slow span threshold
elastic.apm.span_min_duration=5ms

Error Tracking and Stack Trace Analysis

// src/main/java/com/example/myapp/exception/GlobalExceptionHandler.java
package com.example.myapp.exception;

import co.elastic.apm.api.ElasticApm;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;

import java.util.Map;

@RestControllerAdvice
public class GlobalExceptionHandler {

    private static final Logger log = LoggerFactory.getLogger(GlobalExceptionHandler.class);

    @ExceptionHandler(RuntimeException.class)
    public ResponseEntity<Map<String, String>> handleRuntimeException(RuntimeException e) {
        // Report exception to APM with full stack trace
        ElasticApm.currentTransaction().captureException(e);

        // Emit structured log (parseable by ELK)
        log.error("Unhandled exception: type={}, message={}, trace_id={}",
                e.getClass().getSimpleName(),
                e.getMessage(),
                ElasticApm.currentTransaction().ensureParentId(),
                e);

        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
                .body(Map.of(
                        "error", "Internal Server Error",
                        "message", e.getMessage()
                ));
    }
}

Clicking an error in Kibana APM → Errors reveals:

• Full Java stack trace
• HTTP request details at the time of the error (URL, headers, body)
• Linked trace ID for distributed tracing correlation
• Error occurrence trend chart over time

Comparison: Elastic APM vs Pinpoint

Feature	Elastic APM	Pinpoint
Language support	Java, Node.js, Python, Go, and more	Primarily Java/PHP
Setup difficulty	Easy (integrates with ELK)	Moderate (requires HBase)
Infrastructure	Requires Elasticsearch	Requires HBase + ZooKeeper
Service map	Built-in	Detailed call graph
Cost	Open source + paid cloud	Fully open source
Real-time latency	Seconds	Seconds
Auto-instrumentation coverage	Broad	Very broad (200+ plugins)

In practice, teams already running ELK typically choose Elastic APM for its seamless integration. Teams with a Naver-influenced tech stack or a preference for call-graph-level visibility often choose Pinpoint.