Spans in tracing

The word span doesn’t come from the profiling world, it comes from the distributed tracing one. Before we start playing with the async-profiler, let’s agree on what a span is there, since the profiler borrows both the name and the idea. The most popular tracing standard nowadays is OpenTelemetry, so let’s use it as an example.

Imagine a very simple e-commerce system. A user clicks “buy” and the request travels through a few services:

gateway ---> order-service ---> payment-service
                           ---> warehouse-service ---> database

From the user’s perspective this is one operation that either succeeded or not, and either was fast or was slow. From the infrastructure perspective it is a handful of HTTP calls, a few database queries, maybe a message on a queue, executed by different threads on different machines, each of them producing its own logs. Tracing is a technique of gluing all of that back together into a single picture.

Two definitions:

• Span - a single unit of work, with a name, a start timestamp and a duration. It can be POST /orders, SELECT * FROM ORDERS, or just a plain method call like calculateDiscount.
• Trace - the whole tree of spans that belongs to one end-to-end operation.

Span context

Every span carries a span context. The three fields that matter for us are:

• traceId - 16 bytes (32 hex characters), the same for every span in the trace
• spanId - 8 bytes (16 hex characters), unique for that particular span
• parent spanId - empty for the root span

The last one is what makes a trace a tree, and not just a bag of spans. For our “buy” request it may look like this:

traceId = 4bf92f3577b34da6a3ce929d0e0e4736

spanId=00f067aa0ba902b7  POST /orders                    [=================================] 850ms
spanId=a2fb4a1d1a96d312    HTTP POST /payments           [=======]                           190ms
spanId=b9c7c989f97918e1      INSERT INTO PAYMENTS        [==]                                 55ms
spanId=eee19b7ec3c1b174    HTTP POST /reservations       [==========================]        620ms
spanId=00b976fa8a4d2f1c      SELECT ... FROM WAREHOUSE   [=========================]         600ms

Every service reports its own spans, they are shipped to some collector, and the collector groups them by traceId and rebuilds the tree using the parent spanId. This is exactly what you see in Jaeger, Zipkin, Grafana Tempo or any commercial APM.

Besides the context, a span can also carry:

• attributes - key/value pairs, like http.response.status_code=200 or db.system=postgresql
• events - timestamped messages inside the span, exceptions are usually reported that way
• status - UNSET, OK or ERROR
• kind - SERVER, CLIENT, INTERNAL, PRODUCER or CONSUMER

How the spans are created

In the OpenTelemetry Java API, you create a span manually like this:

Tracer tracer = openTelemetry.getTracer("com.example.orders");

Span span = tracer.spanBuilder("processOrder")
        .setAttribute("order.id", orderId)
        .startSpan();

try (Scope scope = span.makeCurrent()) {
    processOrder(orderId);
} catch (Exception e) {
    span.recordException(e);
    span.setStatus(StatusCode.ERROR);
    throw e;
} finally {
    span.end();
}

Three things are worth noticing here, because we will see the very same pattern in the async-profiler later on:

• the span is started and ended explicitly, the difference between those two timestamps is the duration
• span.makeCurrent() puts the span into a ThreadLocal, so that any code executed below can find out “in which span am I running right now?” - this is what all the instrumentation libraries use
• the Scope has to be closed on the same thread that opened it

In practice you rarely write that code yourself. The OpenTelemetry Java agent instruments the popular frameworks and libraries for you, and in a Spring Boot 3 application you get Micrometer Tracing, which can bridge to OpenTelemetry, out of the box.

Context propagation

The traceId alone would be useless if it stopped at the service boundary. When order-service calls payment-service, the span context is serialized into an HTTP header - the W3C Trace Context standard calls it traceparent:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^  ^                                ^                ^
             |  |                                |                flags (sampled)
             |  traceId                          parent spanId
             version

The receiving service reads that header, and every span it creates inherits the same traceId. That’s the whole magic. The same mechanism works for messaging systems, where the context is put into the message headers instead.

What tracing gives you, and what it doesn’t

Let’s go back to our waterfall. It tells us a lot:

• the whole request took 850ms
• 620ms of that was spent in warehouse-service
• and almost all of it, 600ms, was a single database query

That is a great starting point for an investigation, and very often it is enough. But now let’s change the example a little bit:

traceId = 4bf92f3577b34da6a3ce929d0e0e4736

spanId=00f067aa0ba902b7  POST /orders                    [=================================] 850ms
spanId=a2fb4a1d1a96d312    HTTP POST /payments           [=======]                           190ms
spanId=eee19b7ec3c1b174    HTTP POST /reservations       [====]                              110ms

Where did the remaining 550ms go? Somewhere in the order-service code, between the calls. Tracing tells us that it happened, but it will never tell us why, because spans exist only where somebody - you, or the instrumentation library - decided to put them. Adding more and more spans to find out is not a great idea either: they are not free, they have to be created, stored and shipped, and you would need to know upfront where to put them, which is precisely the thing you don’t know.

Filling that gap is a job for a profiler. A profiler doesn’t care about your instrumentation, it just samples what the threads are doing. The problem is the opposite one: a profile gives you a beautiful aggregation over all the requests, while the thing you are usually asked about is that one slow request from the production incident.

This is where the async-profiler’s spans come in. As we will see in the next sections, an async-profiler span is a much simpler structure than the OpenTelemetry one. There is no spanId, no parent, no attributes, no events. It is just a [start, end] interval on one thread, plus a single string called a tag:

long span = Span.start();
actualWork();
Span.end(span, tag);

And that’s on purpose. The profiler doesn’t need to draw a waterfall, it needs just enough information to answer one question: which samples were gathered by this thread while that span was open?

The tag is an arbitrary label, and it is entirely up to you what you put there. It can be an endpoint name like POST /orders, a message type, a tenant, a query ID. But it can also be the traceId we’ve just been talking about - and then the profiler and your tracing system start speaking the same language. Filtering the profile by a single traceId gives us a flame graph of one distributed request, with all its threads and all its services, which is exactly the “why” that the waterfall above was missing.

Feeding the profiler from your tracing

The idea of joining the two worlds is trivial:

Whenever a telemetry span becomes active, start a profiler span. Whenever it stops being active, end it, tagged with the tracing IDs.

No new instrumentation, no new decisions about “where to put the spans” - somebody has already made all of them for you, and did it in the places that matter. You are just piggybacking on that work.

The whole trick is in the words becomes active, so let’s look at them before writing any code.

The trap: a span’s lifetime is not a thread’s work

The naive implementation hooks the span lifecycle - start the profiler span when the telemetry span starts, end it when the telemetry span ends. It works in a simple synchronous application, and it breaks everywhere else, for one reason: a telemetry span is not bound to a thread, and a profiler span is.

Span.start() and Span.end() record the interval on whatever thread calls them. But an OpenTelemetry span:

• can be started on one thread and ended on another - any asynchronous or reactive code does that
• can be open for 850ms while the thread that created it went off to serve three other requests in the meantime
• can be active on several threads at once, if the work was forked out

Hook the lifecycle and you get spans that begin on one thread and end on another (garbage), or spans that cover 850ms of a thread’s time when only 50ms of it was actually spent on that request (worse - it looks plausible).

Fortunately OpenTelemetry already has a concept that means exactly “this thread is working under this span right now”, and it is the Scope:

try (Scope scope = span.makeCurrent()) {
    // this thread, and only this thread, is now working on `span`
}

The Scope is thread-bound by contract - it must be closed on the thread that opened it - which is precisely the contract Span.start()/Span.end() needs. So the rule is:

Don’t hook the span. Hook the scope.

And notice that this gives us the right behaviour for free: if one telemetry span is made current five times, on three threads, we get five profiler spans carrying the same tag. That is not a bug, that is the honest answer to “which stretches of thread time were spent on this request”.

Plain OpenTelemetry

Every makeCurrent() goes through ContextStorage, and OpenTelemetry lets us wrap it. That is a single place through which every context activation in the whole application has to pass:

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanContext;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.ContextStorage;
import io.opentelemetry.context.Scope;

public final class ProfilerContextStorage implements ContextStorage {
    private final ContextStorage delegate;

    private ProfilerContextStorage(ContextStorage delegate) {
        this.delegate = delegate;
    }

    public static void install() {
        ContextStorage.addWrapper(ProfilerContextStorage::new);
    }

    @Override
    public Scope attach(Context toAttach) {
        Scope scope = delegate.attach(toAttach);

        SpanContext spanContext = Span.fromContext(toAttach).getSpanContext();
        if (!spanContext.isValid()) {
            return scope;
        }

        String tag = spanContext.getTraceId() + '/' + spanContext.getSpanId();
        long profilerSpan = one.profiler.Span.start();

        return () -> {
            one.profiler.Span.endIfProfiled(profilerSpan, tag);
            scope.close();
        };
    }

    @Override
    public Context current() {
        return delegate.current();
    }
}

A few notes:

• yes, there are two classes called Span here, io.opentelemetry.api.trace.Span and one.profiler.Span. Sorry about that, one of them has to be fully qualified.
• ContextStorage.addWrapper has to be called before the first makeCurrent() in the application. The javadoc suggests a static initializer in your main class, and it means it.
• the profiler span is ended before the delegate scope is closed, so that the interval we record is a subset of the interval the context was actually active.
• spanContext.isValid() skips contexts that carry no span at all - there is nothing to tag them with.

Spring Boot and Micrometer

In a Spring Boot 3 application you usually don’t touch OpenTelemetry directly, you go through Micrometer Tracing. Its ObservationHandler has scope callbacks, so we can use the same approach:

public class ProfilerObservationHandler implements ObservationHandler<Observation.Context> {
    private static final ThreadLocal<Deque<Long>> STARTS = ThreadLocal.withInitial(ArrayDeque::new);

    @Override
    public boolean supportsContext(Observation.Context context) {
        return true;
    }

    @Override
    public void onScopeOpened(Observation.Context context) {
        STARTS.get().push(Span.start());
    }

    @Override
    public void onScopeClosed(Observation.Context context) {
        Long start = STARTS.get().poll();
        if (start != null) {
            Span.endIfProfiled(start, tag(context));
        }
    }

    private static String tag(Observation.Context context) {
        TracingContext tracingContext = context.get(TracingContext.class);
        if (tracingContext == null || tracingContext.getSpan() == null) {
            return context.getContextualName();
        }
        TraceContext traceContext = tracingContext.getSpan().context();
        return traceContext.traceId() + '/' + traceContext.spanId() + '/' + context.getContextualName();
    }
}

Scopes on a single thread nest strictly, last opened is first closed, which is why a Deque is enough here - no need to match the start timestamp with any particular observation.

Registering it is the same as any other handler:

@Bean
ObservedAspect observedAspect(ObservationRegistry observationRegistry) {
    observationRegistry.observationConfig().observationHandler(new ProfilerObservationHandler());
    return new ObservedAspect(observationRegistry);
}

What to put in the tag

Now the interesting part. traceId + "/" + spanId is a good default, and here is why it works so well:

• both IDs are fixed width - traceId is 32 lowercase hex characters, spanId is 16 - and hex never contains a /, so the tag is unambiguous to split and to match
• the coarse key comes first, so a “starts with” filter on 4bf92f35.../ gives you the whole distributed request, across every thread and every service, while a full equality match narrows it down to one single tracing span
• it is a join key back to your tracing backend. You find a slow span in Jaeger or Tempo, you copy its ID, you paste it into the profiler viewer’s filter. That’s the whole workflow, and it is worth a lot when somebody is standing behind you asking why yesterday’s order took 4 seconds.

The one thing I would add on top is the operation name, as a suffix - traceId/spanId/name. Opaque hex is fine when you arrive from Jaeger with an ID in your clipboard, but it is unreadable when you are just browsing the Span stats of a JFR file and trying to work out which of these took 4 seconds. Putting the name last keeps both prefix filters (by trace, and by trace + span) working exactly as before.

Whether you can do it depends on which side you hooked:

• Micrometer hands you the name for free - Observation.Context has getName() and getContextualName(), which is why the handler above uses it.
• Plain OpenTelemetry does not. io.opentelemetry.api.trace.Span has updateName(), but there is no getName() - reading the name back means casting to the SDK’s ReadableSpan, and coupling your instrumentation to the SDK internals. Not worth it. Stay with traceId/spanId there and let the tracing backend resolve names for you.

One trade-off worth knowing: if what you mostly do is aggregate profiling - “show me the flame graph of everything that ran under POST /orders this week” - then the name is the useful prefix and the IDs are the noise, and you may prefer name/traceId/spanId instead. You can’t have both prefixes at once; pick the one that matches how you actually search. My experience is that people reach for spans when investigating a single bad request, so I keep the IDs in front.

Your application is not my application

Everything I’ve written above is a default, not a rule. The tag is a free-form string, the profiler doesn’t parse it, and nothing in async-profiler knows or cares that OpenTelemetry exists. I picked traceId/spanId because this chapter is about joining the profiler with tracing, but that is my use case, and yours may be a completely different one.

The question you should be asking is not “what does the standard say”, it is:

When somebody wakes me up at 3 a.m. with a production problem, what will I want to filter the profile by?

The answer to that is domain knowledge, and you have it - I don’t.

The best example is a multitenant system. Sooner or later you get a ticket saying “customer ACME reports the system is slow”, while every dashboard you own looks perfectly fine, because ACME is 2% of your traffic and the averages have swallowed them whole. If the customer ID is part of your tag, that investigation is one filter away - you get a flame graph of everything your application did for that one customer, and very often the answer is immediately visible: they are the only client with 40 thousand records in a collection that somebody loops over, or the only one with a feature flag that turns on an expensive code path.

Adding it is trivial. If you propagate the tenant in OpenTelemetry baggage, it is already sitting in the context you are wrapping:

String tenant = Baggage.fromContext(toAttach).getEntryValue("tenant.id");
String tag = tenant + '/' + spanContext.getTraceId() + '/' + spanContext.getSpanId();

and if you keep it in your own ThreadLocal, in an MDC, or in a security context, use that instead - it makes no difference to the profiler.

Other things I have seen work well, depending on what the application actually does:

• batch and ETL jobs - the job name plus the chunk or partition number. “Which partition is the slow one, and why” is the whole investigation.
• message-driven systems - the message or event type, sometimes the topic and consumer group. A single poison message type can dominate a consumer without showing up anywhere else.
• reporting - the report type and its parameters. Reports are the classic case where one rarely used variant is three orders of magnitude more expensive than the rest.
• rollouts and feature flags - the variant name. Then you can put two flame graphs side by side and see what the new version actually costs, on production traffic, in the same JFR file.
• cache hit/miss, hot/cold paths - a crude one, but a profile split by cache miss is sometimes all the evidence you need.

And of course you can combine them - tenant/traceId/spanId is a perfectly reasonable tag. Just remember the ordering rule from the previous section: whatever you put first is what you can prefix-filter by, so put the dimension you group by in front, and the unique identifier behind it.

Two warnings before you go wild with this:

• Mind the cardinality. Every distinct tag is a separate string in the JFR file. A traceId is unbounded too, but it earns its place - it is a join key you will actually use. Something like a full SQL statement with literals inlined, or a user agent, gives you thousands of unique strings that you will never once filter by exactly. Put the low-cardinality thing in the tag, and let the flame graph tell you the rest.
• Mind what you are writing to disk. A JFR file gets copied to a laptop, attached to a ticket, sometimes sent to a vendor. A customer ID is usually fine, a customer name, e-mail or a document number is exactly the sort of thing that shouldn’t leave production in a profiling artifact. Tag with the surrogate key, resolve it to a human-readable name later, in a system that has the right to show it to you.

Nesting, and why endIfProfiled matters

Telemetry spans nest, so profiler spans will nest too. A sample taken deep inside payment-service ends up covered by the HTTP server span, the service span and the JDBC span all at once.

That is a feature. Filtering by the outer tag gives you that whole subtree, filtering by an inner tag narrows you down to just that part of the work - the same drill-down you have in the tracing UI, only now every level has a flame graph attached to it.

It does have a cost though. Every span is a JFR event, and every tag is a distinct string in the JFR constant pool - and with traceId/spanId every single tag is unique, so nothing deduplicates. An application with fine-grained instrumentation inside a loop can produce a lot of them.

This is exactly what endIfProfiled is for:

Span.endIfProfiled(profilerSpan, tag);

It records the span only if at least one profiling sample was actually taken on this thread while the span was open, and it decides that without entering native code at all. A span that encloses no sample contributes nothing to the profile - there is nothing to filter with it - so dropping it costs you no information whatsoever. With wall-clock sampling at the default interval, that quietly removes the overwhelming majority of the short nested spans, and leaves you the ones that actually have a flame graph behind them.

Use plain end() only when you genuinely want a complete list of spans, for example when you are also using the JFR file for latency statistics rather than just for filtering samples.

A note on tracing sampling

Your tracing backend almost certainly does not store 100% of the traces - head-based sampling at 1% or 5% is the norm. Mind that an unsampled trace still gets a perfectly valid traceId, still propagates it downstream, and will still end up in your profiler spans - it just never reaches Jaeger, so the tag will point at a trace you cannot look up.

You can gate on it if that bothers you:

if (!spanContext.isSampled()) {
    return scope;
}

I would rather not. Profiler spans are far cheaper than exported traces, and the profile stands on its own - a tag is still a perfectly good grouping key even if nobody can resolve it to a waterfall. Being able to profile 100% of the requests while tracing 1% of them is a feature, not a problem.

The demo application

Every snippet above is taken from a working application, which you can find here:

https://github.com/krzysztofslusarski/async-profiler-spans-demo

It is a Maven project with two independent examples - one on the plain OpenTelemetry SDK with the ContextStorage wrapper, one on Spring Boot with the ObservationHandler - plus a fake workload that does exactly what the waterfalls in this article show.

You don’t need any infrastructure to run it. There is no Jaeger, no Tempo, no collector:

• finished traces are printed to the console as an ASCII waterfall, the same one you have been reading here
• the resulting JFR file is read back with jdk.jfr.consumer, which is a part of the JDK, and the spans are correlated with the profiling samples

./run-plain-otel.sh
./run-spring.sh

The only thing you have to provide is the async-profiler native library itself - the scripts will look for it in a few usual places, or you can point them at it with ASPROF_LIB=/path/to/libasyncProfiler.so.

The output is worth a look even if you don’t intend to write any of this yourself, because it makes the claims from this chapter checkable rather than merely plausible. The un-instrumented calculateDiscounts really is a hole in the waterfall and really is the top method once you filter the samples by tag. The root span’s tag really does show up three times, once per thread it was active on. And a 290ms span really does cover 29 samples at a 10ms wall-clock interval.

In this article, I would like to tackle the native memory leaks problem. Usually such a problem starts with some monitoring system that alerts about high memory (RAM) utilization. We start the investigation and we see that our Java process has eaten 8GB of RAM with Xmx/Xms set to 4GB. How is that possible?

Well, the heap is not the only memory needed for Java applications to run. Other pretty obvious memory areas are:

• Threads - each thread needs some native memory for its data
• Metaspace - for our class definitions
• Code cache/code heap - for JIT compiler output
• …

NMT

JVM has a dedicated feature to track where the memory is needed. It’s called “Native memory tracking” (NMT) and is disabled by default. There are two modes of NMT:

• Summary - This mode tracks the number of bytes allocated by JVM in each of the NMT category
• Detail - This one gathers the same data as the summary + keeps track of which part of the code needs that memory

There are many rumors and misconceptions about NMT overhead. In the Oracle documentation you can find the following:

Enabling NMT will result in 5-10 percent JVM performance drop and memory usage for NMT adds 2 words to all malloc memory as malloc header. NMT memory usage is also tracked by NMT.

The confusing part of this quote is the fact that it’s written in detail mode explanation, and I cannot tell if it’s a generic statement for both modes or just the latter.

The much better explanation you can find in this Stack Overflow response by Andrei Pangin:

The overhead of Native Memory Tracking obviously depends on how often the application allocates native memory. Usually, this is not something too frequent in a Java application, but cases may differ. Since you’ve already tried and didn’t notice performance difference, your application is apparently not an exception.

In the summary mode, Native Memory Tracking roughly does the following things:

• increases every malloc request in the JVM by 2 machine words (16 bytes);
• records the allocation size and flags in these 2 words;
• atomically increments (or decrements on free) the counter corresponding to the given memory type;
• besides malloc and free, it also handles changes in virtual memory reservation and allocations of new arenas, but these are even less frequent than malloc/free calls.

So, to me, the overhead is quite small; 5-10% is definitely a large overestimation (the numbers would make sense for detail mode which collects and stores stack traces, which is expensive, but summary doesn’t do that). When many threads concurrently allocate/free native memory, the update of an atomic counter could become a bottleneck, but again, that’s more like an extreme case. In short, if you measured a real application and didn’t notice any degradation, you’re likely safe to enable NMT summary in production.

The most important question is: can I run NMT in production? From my experience: most likely yes, in summary mode. We also need to mind circumstances when such a feature is needed - when we trace strange memory consumption. It’s a really hard job, so we should use all the tools we can to find the root cause of the problem.

Usage - summary

To enable NMT in summary mode you need to start JVM with -XX:NativeMemoryTracking=summary. After that whenever we want to understand why JVM allocated some native memory we can run:

jcmd <pid> VM.native_memory summary scale=MB

Sample output:

Native Memory Tracking:

(Omitting categories weighting less than 1MB)

Total: reserved=2340MB, committed=1237MB
-                 Java Heap (reserved=1024MB, committed=1024MB)
                            (mmap: reserved=1024MB, committed=1024MB) 
 
-                     Class (reserved=1025MB, committed=11MB)
                            (classes #16326)
                            (  instance classes #15180, array classes #1146)
                            (malloc=1MB #31555) 
                            (mmap: reserved=1024MB, committed=10MB) 
                            (  Metadata:   )
                            (    reserved=64MB, committed=56MB)
                            (    used=55MB)
                            (    waste=0MB =0,63%)
                            (  Class space:)
                            (    reserved=1024MB, committed=10MB)
                            (    used=10MB)
                            (    waste=0MB =2,88%)
 
-                    Thread (reserved=50MB, committed=4MB)
                            (thread #51)
                            (stack: reserved=50MB, committed=4MB)
 
-                      Code (reserved=49MB, committed=15MB)
                            (malloc=1MB #7383) 
                            (mmap: reserved=48MB, committed=14MB) 
 
-                        GC (reserved=90MB, committed=90MB)
                            (malloc=19MB #9300) 
                            (mmap: reserved=70MB, committed=70MB) 
 
-                    Symbol (reserved=17MB, committed=17MB)
                            (malloc=15MB #401954) 
                            (arena=1MB #1)
 
-    Native Memory Tracking (reserved=7MB, committed=7MB)
                            (tracking overhead=7MB)
 
-        Shared class space (reserved=12MB, committed=12MB)
                            (mmap: reserved=12MB, committed=12MB) 
 
-               Arena Chunk (reserved=1MB, committed=1MB)
                            (malloc=1MB) 
 
-                 Metaspace (reserved=64MB, committed=56MB)
                            (mmap: reserved=64MB, committed=56MB) 

Usage - diff

If we want to understand which NMT category is increasing we don’t need to compare two outputs like the one above. NMT has built-in diff operation. First, we need to set a baseline with:

jcmd <pid> VM.native_memory baseline

After some time (when we see that more memory was consumed by the Java process) we can run:

jcmd <pid> VM.native_memory summary.diff scale=MB

Sample output:

Native Memory Tracking:

(Omitting categories weighting less than 1MB)

Total: reserved=3350MB +1004MB, committed=2257MB +1013MB

-                 Java Heap (reserved=1024MB, committed=1024MB)
                            (mmap: reserved=1024MB, committed=1024MB)
 
-                     Class (reserved=1026MB, committed=12MB)
                            (classes #16935 +7)
                            (  instance classes #15732 +4, array classes #1203 +3)
                            (malloc=2MB #34724 +2040)
                            (mmap: reserved=1024MB, committed=10MB)
                           : (  Metadata)
                            (    reserved=64MB, committed=58MB)
                            (    used=57MB)
                            (    waste=0MB =0,58%)
                           : (  Class space)
                            (    reserved=1024MB, committed=10MB)
                            (    used=10MB)
                            (    waste=0MB =2,82%)
 
-                    Thread (reserved=48MB -4MB, committed=4MB)
                            (thread #0)
                            (stack: reserved=48MB -4MB, committed=4MB)
 
-                      Code (reserved=50MB, committed=21MB +6MB)
                            (malloc=1MB #10044 +2426)
                            (mmap: reserved=48MB, committed=20MB +6MB)
 
-                        GC (reserved=90MB, committed=90MB)
                            (malloc=19MB #13489 +3620)
                            (mmap: reserved=70MB, committed=70MB)
 
-                     Other (reserved=1011MB +1007MB, committed=1011MB +1007MB)
                            (malloc=1011MB +1007MB #1025 +1008)
 
-                    Symbol (reserved=17MB, committed=17MB)
                            (malloc=16MB #412735 +275)
                            (arena=1MB #1)
 
-    Native Memory Tracking (reserved=7MB, committed=7MB)
                            (tracking overhead=7MB)
 
-        Shared class space (reserved=12MB, committed=12MB)
                            (mmap: reserved=12MB, committed=12MB)
 
-                 Metaspace (reserved=64MB, committed=58MB)
                            (mmap: reserved=64MB, committed=58MB)

First, we can look at information in line starting with Total where we can see +1013MB. It means that our JVM allocated almost 1GB of native memory since we ran the baseline command. When we browse all the categories we can see this one:

-                     Other (reserved=1011MB +1007MB, committed=1011MB +1007MB)
                            (malloc=1011MB +1007MB #1025 +1008)

NMT - categories

Let’s now go through those categories and try to understand what we can do if memory consumption is increasing there.

• Java Heap - An increase in that category means that your initial or minimal heap size is different from the max one and the heap simply grows.
• Class - New classes were loaded in the runtime. Depending on the volume of consumed memory it can be the natural behavior of the application. Many libraries define/load classes in the runtime. If the increase here is abnormal you can try to profile where new classes are loaded with Async-profiler.
• Thread - In most configurations, every Java thread can use up to 1MB (-Xss). Increasing consumption in that category usually means:
  • that existing threads needed more memory - for example, it reached code with deep stack
  • new threads were created - you can find places that create new threads with Async-profiler.
• Code - This is a space for JIT, it can grow up to ~240MB (-XX:ReservedCodeCacheSize). Increasing consumption here is natural. The longer our JVM is running the more methods are compiled by JIT.
• Other - Here you can find a native memory allocated with ByteBuffer.allocateDirect. If you want to understand why this category grows you can run Async-profiler in method mode to find invocations of the allocateDirect method, or you can analyze the heap dump. You can also find here other allocations done with Unsafe.allocateMemory method.

Those are the most common scenarios I approached in my career. Mind that JVM can also have issues in its code that result in large memory consumption like JDK-8240723.

NMT vs RSS/PSS vs total memory allocated in the system

To move on I want to point out the difference between three metrics:

• NMT committed memory - how much memory JVM used, counted by JVM internals.
• RSS/PSS - how much memory the Java process used from an OS perspective.
• The total memory allocated in the system - this one is straightforward, it counts not only the Java process but everything system-wide.

Today many of the new applications written in Java are run in Docker containers so we would expect those three values to be the same. Well, the reality is a bit different.

Native libs

When you run your Java code you are allowed to invoke native code with JNI (and other mechanisms, but that’s irrelevant). Additionally, JDK can invoke native libraries via JNI (as en example you can look at the source code of java.util.zip.Inflater class, a lot of native methods). The code inside the external native library is out of JVM’s control. That means that if our native code does malloc, JVM doesn’t know about that. This basically means that NMT cannot trace such an allocation. Remember that you don’t need to invoke JNI by yourself, your dependencies can do it for you. A lot of libraries do that. If your code or your dependency code has a native memory leak done inside the JNI section RSS/PSS can grow while NMT committed memory will stay the same.

To track down native memory leaks (no matter if it is inside JVM or native libs) we need to find allocated memory that hasn’t been freed. For many years there was no tool that could do it and give you Java stacktraces. In the past, we could use Async-profiler and trace:

• page-faults
• malloc invocations
• mprotect invocations

But all we could get from it was where memory was allocated. We didn’t know which parts of the allocated memory were freed later. For all those years my approach to finding native memory leaks was as follows:

• Check JMX metrics (either in Grafana/Zabbix/… or by manual JMX connection) to see what JVM reports for classloading and direct buffer allocations.
• If there was no visible leak - turn on NMT in summary mode and pray that the answer is there.
• If there was no visible leak in NMT - profile for page-faults and filter manually every allocation that is tracked by NMT
• For every native allocation found while RSS/PSS was increasing:
  • Filter manually every allocation that is tracked by NMT
  • Run such a code in isolation.
  • Check if RSS/PSS increases.

Current “microservices” can have easily tens (and even more than a hundred) of different places where native memory is allocated. That was really time-consuming work, and from time to time there was no native memory leak after all - we will get to that later.

Thankfully those dark times are over. There is a new mode in the Async-profiler called nativemem. It keeps track of allocations and frees and can show us which memory hasn’t been freed during profiling. That mode literally can save you hours of your life with investigation. This new mode is available in the 4.0 release. Let’s see the difference between nativemem and previous approaches.

I have an application with two endpoints:

@RestController
@RequestMapping("/examples/nativemem")
@RequiredArgsConstructor
class NativeMemController {
    private final NativeMemService nativeMemService;

    @GetMapping("/leak")
    String getWithLeak() {
        nativeMemService.createNewBuffer(true);
        return "OK";
    }

    @GetMapping("/no-leak")
    String getNoLeak() {
        nativeMemService.createNewBuffer(false);
        return "OK";
    }
}

class NativeMemService {
    private final List<ByteBuffer> buffers = new ArrayList<>();

    void createNewBuffer(boolean leak) {
        ByteBuffer direct = ByteBuffer.allocateDirect(1024 * 1024);
        if (leak) {
            buffers.add(direct);
        }
    }
}

A call to /leak endpoint will create 1 MB of native memory leak, a call to /no-leak endpoint will allocate 1 MB of native memory that should be freed by GC.

Here is a full script that I did:

#Warm-up
ab -n 300 http://localhost:8081/examples/nativemem/leak
ab -n 300 http://localhost:8081/examples/nativemem/no-leak

./asprof -e nativemem -f nativemem.jfr start 11817
ab -n 300 http://localhost:8081/examples/nativemem/leak
ab -n 300 http://localhost:8081/examples/nativemem/no-leak
jcmd 11817 GC.run
./asprof stop 11817
./jfrconv --total --nativemem --leak nativemem.jfr nativemem.html

./asprof -e page-faults -f faults.jfr start 11817
ab -n 300 http://localhost:8081/examples/nativemem/leak
ab -n 300 http://localhost:8081/examples/nativemem/no-leak
jcmd 11817 GC.run
./asprof stop 11817
./jfrconv faults.jfr faults.html

./asprof -e mprotect -f mprotect.jfr start 11817
ab -n 300 http://localhost:8081/examples/nativemem/leak
ab -n 300 http://localhost:8081/examples/nativemem/no-leak
jcmd 11817 GC.run
./asprof stop 11817
./jfrconv mprotect.jfr mprotect.html

nativemem flame graph shows the correct code to blame: Since we used the --total option with jfrconv it also shows that we leaked 300 MB:

mprotect flame graph shows both allocations, leaky and non-leaky. In the real world, you can easily get one leaky and one hundred non-leaky:

page-faults is the worst in that test, it blames the wrong endpoint and doesn’t show the proper allocation path in the flame graph:

Malloc

If you’ve tried to find a native memory leak with no results … well, there are still some other strange cases to consider. In this article, I cover only the most popular that I’ve seen many times. This strange case is malloc from GLIBC causing strange memory consumption. There are many articles on that topic in the internet. Here is one of them from Ruby. Mind that malloc an is allocator used by many languages, so if you do your search on that topic don’t narrow it to Java only.

Let’s consider a program:

class ByteBufferFragmentationExample {
    public static void main(String[] args) throws InterruptedException {
        Thread.sleep(10_000);
        for (int i = 0; i < 60; i++) {
            new Thread(() -> {
                Random random = new Random();
                while (true) {
                    ByteBuffer.allocateDirect(random.nextInt(1024 * 1024));
                }
            }).start();
        }
    }
}

Let’s run it with -Xmx1G -Xms1G -XX:+AlwaysPreTouch -XX:NativeMemoryTracking=summary, so our heap is limited to 1 GB and allocated at the start of JVM. At the beginning, the RSS and PSS are:

$ smem -c "pid command rss pss" -ak -P "ByteBufferFragmentationExample"

  PID Command                                                  RSS     PSS 
15753 /home/pasq/JDK/amazon-corretto-17.0.1.12.1-linux-x64    1.1G    1.1G 

After some time:

  PID Command                                                  RSS     PSS 
15474 /home/pasq/JDK/amazon-corretto-17.0.1.12.1-linux-x64    5.1G    5.1G 

If we check the NMT output we will get:

jcmd ByteBufferFragmentationExample VM.native_memory summary scale=MB

15753:

Native Memory Tracking:

(Omitting categories weighting less than 1MB)

Total: reserved=3519MB, committed=2176MB
-                 Java Heap (reserved=1024MB, committed=1024MB)
                            (mmap: reserved=1024MB, committed=1024MB) 
 
-                     Class (reserved=1024MB, committed=1MB)
                            (classes #2311)
                            (  instance classes #2082, array classes #229)
                            (mmap: reserved=1024MB, committed=1MB) 
                            (  Metadata:   )
                            (    reserved=8MB, committed=5MB)
                            (    used=5MB)
                            (    waste=0MB =1,86%)
                            (  Class space:)
                            (    reserved=1024MB, committed=1MB)
                            (    used=1MB)
                            (    waste=0MB =13,09%)
 
-                    Thread (reserved=91MB, committed=8MB)
                            (thread #92)
                            (stack: reserved=91MB, committed=8MB)
 
-                      Code (reserved=242MB, committed=8MB)
                            (mmap: reserved=242MB, committed=7MB) 
 
-                        GC (reserved=89MB, committed=89MB)
                            (malloc=19MB #4046) 
                            (mmap: reserved=70MB, committed=70MB) 
 
-                  Compiler (reserved=1MB, committed=1MB)
 
-                     Other (reserved=1024MB, committed=1024MB)
                            (malloc=1024MB #2048) 
 
-                    Symbol (reserved=2MB, committed=2MB)
                            (malloc=1MB #22702) 
                            (arena=1MB #1)
 
-    Native Memory Tracking (reserved=1MB, committed=1MB)
                            (tracking overhead=1MB)
 
-        Shared class space (reserved=12MB, committed=12MB)
                            (mmap: reserved=12MB, committed=12MB) 
 
-               Arena Chunk (reserved=1MB, committed=1MB)
                            (malloc=1MB) 
 
-                 Metaspace (reserved=8MB, committed=5MB)
                            (mmap: reserved=8MB, committed=5MB)

Looks like Java only committed 2 GB of memory. There is no point in looking for a leak in our program, there is no leak. The result from allocateDirect method can be collected by GC the on next run.

Nowadays, I see that problem most commonly when engineers decide to use an Amazon Linux, usually by fetching an Amazon Corretto docker image. This problem is not Amazon specific, in last few years I mostly profiled applications deployed on AWS. Such applications usually used Corretto docker images. Example Dockerfile for Amazon Linux starts with:

FROM amazoncorretto:21

My approach to that problem is to change allocator to jemalloc:

FROM amazoncorretto:21
RUN amazon-linux-extras install -y epel
RUN yum install -y jemalloc-devel
ENV LD_PRELOAD="/usr/lib64/libjemalloc.so"

Mind that if you’re using Alpine images/Linux you’re using musl libc, not GLIBC. I’ve never seen that issue with musl.

This blog post contains examples of async-profiler usages that I have found helpful in my job. Some content of this post is copy-pasted from previous entries, as I just wanted to avoid unnecessary jumps between articles.

The goal of that post is to give examples. It’s not a replacement for the project README. I advise you to read it, as it tells you how to obtain async-profiler binaries and more.

All the examples that you are going to see here are synthetic reproductions of real-world problems that I solved during my career. Even if some examples look like “it’s too stupid to happen anywhere,” well, it isn’t.

That post will be maintained. Whenever I find a new use case that I think is worth sharing, I will update this post.

• Change log
• Acknowledgments
• Profiled application
• How to run an async-profiler
  • Command line
  • During JVM startup
  • From Java API
  • From JMH benchmark
  • AP-Loader
  • IntelliJ Idea
• Output formats
• Flame graphs
• Basic resources profiling
  • Wall-clock
  • Wall-clock - filtering
  • CPU - easy-peasy
  • CPU - a bit harder
  • Allocation
  • Allocation - humongous objects
  • Allocation - live objects
  • Locks
• Time to safepoint
• Methods profiling
• Native functions
  • Exceptions
  • G1GC humongous allocation
  • Thread start
  • Class loading
• Perf events
  • Cache misses
  • Page faults
  • Cycles
• Native memory leaks
• Filtering single request
  • Why aggregated results are not enough
  • Real life example - DNS
• Continuous profiling
  • Command line
  • Java
  • Spring Boot
• Contextual profiling
  • Spring Boot microservices
  • Distributed systems
• Stability
• Overhead
• Random thoughts

Change log

• 2022-12-16 - Initial version

Acknowledgments

I would like to say thank you to Andrei Pangin (Lightrun) for all the work he did to create async-profiler and for his time and remarks on that article, Johannes Bechberger (SapMachine team at SAP) for all the work on making OpenJDK more stable with profilers, the input he gave me on overhead and stability, and the copy editing of this document, Marcin Grzejszczak (VMware) for great insight on how to integrate this profiler with Spring, Krystian Zybała for the review.

Profiled application

I’ve created a Spring Boot application for this post so that you can run the following examples on your own. It’s available on my GitHub. To build the application, do the following:

git clone https://github.com/krzysztofslusarski/async-profiler-demos
cd async-profiler-demos
mvn clean package

To run the application, you need three terminals where you run the following (you need the ports 8081, 8082, and 8083 available):

java -Xms1G -Xmx1G -XX:+AlwaysPreTouch \
-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints \
-Duser.language=en-US \
-Xlog:safepoint,gc+humongous=trace \
-jar first-application/target/first-application-0.0.1-SNAPSHOT.jar 

java -Xms1G -Xmx1G -XX:+AlwaysPreTouch \
-jar second-application/target/second-application-0.0.1-SNAPSHOT.jar

java -Xms1G -Xmx1G -XX:+AlwaysPreTouch \
-jar third-application/target/third-application-0.0.1-SNAPSHOT.jar

I’m using Corretto 17.0.2:

$ java -version

openjdk version "17.0.2" 2022-01-18 LTS
OpenJDK Runtime Environment Corretto-17.0.2.8.1 (build 17.0.2+8-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.2.8.1 (build 17.0.2+8-LTS, mixed mode, sharing)

And to create simple load tests, I’m using an ancient tool:

$ ab -V

This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

I will not go through the source code to explain all the details before I jump into examples. I often profile applications with source code I haven’t seen in my work. Just consider it as a typical micro-service build with Spring Boot and Hibernate.

In all the examples, I will assume that the application is started with the java -jar command. If you are running the application from the IDE, then the name of the application is switched from first-application-0.0.1-SNAPSHOT.jar to FirstApplication.

All the JFRs generated while writing that post are available here.

How to run an Async-profiler

Command line

One of the easiest ways of running the async-profiler is using the command line. You just need to execute the following in the profiler folder:

./profiler.sh -e <event type> -d <duration in seconds> \
-f <output file name> <pid or application name>

# examples
./profiler.sh -e cpu -d 10 -f prof.jfr first-application-0.0.1-SNAPSHOT.jar
./profiler.sh -e wall -d 10 -f prof.html 1234 # where 1234 is the PID of the Java process

There are a lot of additional switches that are explained in the README.

You can also use async-profiler to output JFR files:

./profiler.sh start -e <event type> -f <output JFR file name> <pid or application name>
# do something
./profiler.sh stop -f <output JFR file name> <pid or application name>

For formats different from JFR, you need to pass the file name during stop, but for JFR it is needed during start.

The profiler.sh script was removed in the 3.0 release, now we need to use asprof executable.

WARNING: This way of attaching any profiler to a JVM is vulnerable to JDK-8212155. That issue can crash your JVM during attachment. It has been fixed in JDK 17.

If you are attaching a profiler this way, it is recommended to use -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints JVM flags (see this blog post by Jean-Philippe Bempel for more information on why these flags are essential).

During JVM startup

You can add a parameter when you are starting a java process:

java -agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=prof.jfr

The parameters passed this way differ from the switches used in the command line approach. You can find the list of parameters in the arguments.cpp file and the mapping between those in the profiler.sh file in the source code.

You can also attach a profiler without starting it using -agentpath, which is the safest way of starting your JVM if you want to profile it anytime soon.

From Java API

The Java API is published to maven central. All you need to do is to include a dependency:

<dependency>
  <groupId>tools.profiler</groupId>
  <artifactId>async-profiler</artifactId>
  <version>2.9</version>
</dependency>

That gives you an API where you can use Async-profiler from Java code. Example usage:

AsyncProfiler profiler = AsyncProfiler.getInstance(); 

That gives you an instance of AsyncProfiler object, with which you can send orders to the profiler:

profiler.execute(String.format("start,jfr,event=wall,file=%s.jfr", fileName));
// do something, like sleep
profiler.execute(String.format("stop,file=%s.jfr", fileName));

Since async-profiler 2.9, the AsyncProfiler.getInstance() extracts and loads the libasyncProfiler.so from the JAR. In the previous version, this file needed to be in one of the following directories:

• /usr/java/packages/lib
• /usr/lib64
• /lib64
• /lib
• /usr/lib

You can also point to any location of that file with API:

AsyncProfiler profiler = AsyncProfiler.getInstance("/path/to/libasyncProfiler.so");

From JMH benchmark

It’s worth mentioning that the async-profiler is supported in JMH benchmarks. If you have one, you just need to run the following:

java -jar benchmarks.jar -prof async:libPath=/path/to/libasyncProfiler.so\;output=jfr\;event=cpu

JMH will take care of every magic, and you get proper JFR files from async-profiler.

AP-Loader

There is a pretty new project called AP-Loader by Johannes Bechberger that can also be helpful to you. This project packages all native distributions into a single JAR, so It is convenient when deploying on different CPU architectures. You can also use the Java API with this loader without caring where the binary of the profiler is located and which platform you’re running on. I recommend reading the README of that project. It may be suitable for you.

IntelliJ Idea Ultimate

If you are using IntelliJ Idea Ultimate, you have a built-in async-profiler at your fingertips. You can profile any JVM running on your machine and visualize the results. Honestly, I don’t use it that much. Most of the time, I run profilers on remote machines, and I’ve got used to it, so I run it the same way on my localhost.

Output formats

Async-profiler gives you a choice of how the results should be saved:

• default - printing results to the terminal
• JFR
• Collapsed stack
• Flame graphs
• …

From that list, I choose JFR 95% of the time. It’s a binary format containing all the information gathered by the profiler. That file can be post-processed later by some external tool. I’m using my own open-sourced JVM profiling toolkit, which can read JFR files with additional filters and gives me the possibility to add/remove additional levels during the conversion to a flame graph. I will use the filter names of my viewer in the following and the names of filters from my viewer. There are other products (including the jfr2flame converter that is a part of async-profiler) that can visualize the JFR output, but you should use the tool that works for you. None worked for me, so I wrote my own, but it doesn’t mean it is the best choice for everybody.

All flame graphs in this post are generated by my tool from JFR files. The JFR file format for each sample contains the following:

• Stack trace
• Java thread name
• Thread state (is the Java thread consumes CPU)
• Timestamp
• Monitor class - for lock mode
• Waiting for lock duration - for lock mode
• Allocated object class - for alloc mode
• Allocated object size - for alloc mode
• Context ID - if you are using async-profiler with Context ID PR merged

So all the information is already there. We just need to extract what we need and present it visually.

Flame graphs

If you do sampling profiling you need to visualize the results. The results are nothing more than a set of stack traces. My favorite visualization is a flame graph. The easiest way to understand what flame graphs are is to know how they are created.

First, we draw a rectangle for each frame of each stack trace. The stack traces are drawn bottom-up and sorted alphabetically. For example, such a graph looks like this:

This corresponds to the following set of stack traces:

• 3 samples - a() -> h()
• 5 samples - b() -> d() -> e() -> f()
• 2 samples - b() -> d() -> e() -> g()
• 2 samples - b() -> d()
• 2 samples - c()

The next step is joining the rectangles with the same method name to one bar:

The flame graph usually shows you how your application utilizes a specific resource. The resource is utilized by the top methods of that graph (visualized with green bar):

So in this example, method b() does not utilize the resource. It just invokes methods that transitively use it. Flame graphs are commonly used for the CPU utilization, but the CPU is just one of the resources we can visualize this way. If you use wall-clock mode, your resource is time. If you use allocation mode, then your resource is heap. If you want to learn more about flame graphs, you can check Brendan’s Gregg video, he invented flame graphs.

Basic resources profiling

Before you start any profiler, the first thing you need to know is what your goal is. Only after that can you choose the proper mode of async-profiler. Let’s start with the basics.

Wall-clock

If your goal is to optimize time, you should run the async-profiler in wall-clock mode. This is the most common mistake made by engineers starting their journey with profilers. The majority of applications that I profiled so far were applications that were working with a distributed architecture, using some DBs, MQ, Kafka, … In such applications, the majority of time is spent on IO - waiting for other services/DB/… to respond. During such actions, Java is not using the CPU.

# warmup
ab -n 100 -c 4 http://localhost:8081/examples/wall/first
ab -n 100 -c 4 http://localhost:8081/examples/wall/second

# profiling of the first request
./profiler.sh start -e cpu -f first-cpu.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 100 -c 4 http://localhost:8081/examples/wall/first
./profiler.sh stop -f first-cpu.jfr first-application-0.0.1-SNAPSHOT.jar

./profiler.sh start -e wall -f first-wall.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 100 -c 4 http://localhost:8081/examples/wall/first
./profiler.sh stop -f first-wall.jfr first-application-0.0.1-SNAPSHOT.jar

# profiling of the second request
./profiler.sh start -e cpu -f second-cpu.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 100 -c 4 http://localhost:8081/examples/wall/second
./profiler.sh stop -f second-cpu.jfr first-application-0.0.1-SNAPSHOT.jar

./profiler.sh start -e wall -f second-wall.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 100 -c 4 http://localhost:8081/examples/wall/second
./profiler.sh stop -f second-wall.jfr first-application-0.0.1-SNAPSHOT.jar

In the ab output, we can see that the basic stats are similar for each request:

# first request
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:   521  654 217.9    528    1055
Waiting:      521  654 217.9    528    1054
Total:        521  654 217.9    528    1055

# second request
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   522  665 190.2    548    1028
Waiting:      522  665 190.1    548    1028
Total:        522  665 190.2    549    1029

To give you a taste of how the CPU profile can mislead you, here are flame graphs for those two executions in CPU mode.

First execution: (HTML)

Second execution: (HTML)

I agree that they are not identical. But they have one thing in common. They show that the most CPU-consuming method invoked by my controller is CpuConsumer.mathConsumer(). It is not a lie: It consumes CPU. But does it consume most of the time of the request? Look at the flame graphs in wall-clock mode:

First execution: (HTML)

Second execution: (HTML)

I highlighted the ```CpuConsumer.mathConsumer()`` method. Wall-clock mode shows us that this method is responsible just for ~4% of execution time.

Thing to remember: if your goal is to optimize time, and you use external systems (including DBs, queues, topics, microservices) or locks, sleeps, disk IO, It would help if you started with wall-clock mode.

In wall-clock mode, we can also see that these flame graphs differ. The first execution spends most of its time in SocketInputStream.read():

Over 95% of the time is consumed there. But the second execution:

spends just 75% on the socket. To the right of the method SocketInputStream.read() you can spot an additional bar. Let’s zoom in:

It’s the InternalExecRuntime.acquireEndpoint() method, which executes PoolingHttpClientConnectionManager$1.get() from Apache HTTP Client, which in the end executes Object.wait(). What does it do? Basically, what we are trying to do in those two executions is to invoke a remote REST service. The first execution uses an HTTP Client instance with 20 available connections, so no thread needs to wait for a connection from the pool:

// FirstApplicationConfiguration
@Bean("pool20RestTemplate")
RestTemplate pool20RestTemplate() {
    return createRestTemplate(20);
}

// WallService
void calculateAndExecuteSlow() {
    Random random = ThreadLocalRandom.current();
    CpuConsumer.mathConsumer(random.nextDouble(), CPU_MATH_ITERATIONS);

    invokeWithLogTime(() ->
        pool20RestTemplate.getForObject(SECOND_APPLICATION_URL + 
            "/examples/wall/slow", String.class)
    );
} 

The time spent on a socket is entirely spent waiting for the REST endpoint to respond. The second execution uses a different instance of RestTemplate that has just 3 connections in the pool. Since the load is generated from 4 threads by the ab, one thread must wait for a connection from the pool. You may think this is a stupid human error, that someone created a pool without enough connections. In the real world, the problem is with defaults that are quite low. In our testing application, the default settings for the thread pool are:

• maxTotal - 25 connections totally in the pool
• defaultMaxPerRoute - 5 connections to the same address

That number varies between versions. I remember one application with HTTP Client 4.x with defaults set to 2.

There are plenty of tools that log the invocation time of external services. The common problem in those tools is that the time waiting on the pool for a connection is usually included in the invocation time, which is a lie. I saw this in the past when the caller had a line in the logs that gave an execution time of X ms; the callee had a similar log that presented 1/10 * X ms. What were those teams doing to understand that? They tried to convince the network department that this was a network issue. Big waste of time.

I also saw plenty of custom logic that traced external execution time. In our application you can see such a pattern:

void calculateAndExecuteFast() {
    // ...
    invokeWithLogTime(() ->
            pool3RestTemplate.getForObject(SECOND_APPLICATION_URL + "/examples/wall/fast", String.class)
    );
}

private <T> T invokeWithLogTime(Supplier<T> toInvoke) {
    StopWatch stopWatch = new StopWatch();

    stopWatch.start();
    T ret = toInvoke.get();
    stopWatch.stop();

    log.info("External WS invoked in: {}ms", stopWatch.getTotalTimeMillis());
    return ret;
}

The logs don’t trace the time using an external service; the time also includes all the magic done by Spring, including waiting for a connection from the pool. You can easily see that for the second request, the logs look like this:

External WS invoked in: 937ms

But the second service that is invoked is:

    @GetMapping("/fast")
    String fast() throws InterruptedException {
        Thread.sleep(500);
        return "OK";
    }

Wall-clock - filtering

If you open the wall-clock flame graph for the first time, you may be confused, since it usually looks like this:

You see all the threads, even if they are sleeping or waiting in some queue for a job. Wall-clock shows you all of them. Most of the time, you want to focus on the frames where your application is doing something, not when it is waiting. All you need to do is to filter the stack traces. If you are using Spring Boot with the embedded Tomcat, you can filter stack traces that contain the SocketProcessorBase.run method. In my viewer, you can just paste it to stack trace filter, and you are done. It’s just a matter of proper filtering if you want to focus on one controller, class, method, etc.

CPU - easy-peasy

If you know that your application is CPU intensive, or you want to decrease CPU consumption, then the CPU mode is suitable.

Let’s prepare our application:

# execute it once
curl -v http://localhost:8081/examples/cpu/prepare

# little warmup
ab -n 5 -c 1 http://localhost:8081/examples/cpu/inverse

# profiling time:
./profiler.sh start -e cpu -f cpu.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 5 -c 1 http://localhost:8081/examples/cpu/inverse
./profiler.sh stop -f cpu.jfr first-application-0.0.1-SNAPSHOT.jar

You can check during the benchmark what the CPU utilization of our JVM is:

$ pidstat -p `pgrep -f first-application-0.0.1-SNAPSHOT.jar` 5
 
09:42:49      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
09:42:54     1000     49813  115,40    0,40    0,00    0,00  115,80     1  java
09:42:59     1000     49813  111,40    0,60    0,00    0,00  112,00     1  java
09:43:04     1000     49813  106,80    0,20    0,00    0,00  107,00     1  java
09:43:09     1000     49813  113,00    0,20    0,00    0,00  113,20     1  java 

We are using a bit more than one CPU core. Our load generator creates the load with a single thread so that the CPU usage is pretty high. Let’s see what our CPU is doing while executing our spring controller: (HTML)

I know that flame graph is pretty large, but hey, welcome to Spring and Hibernate. I highlighted the existsById() method. You can see that it consumes 95% of the CPU time. But why? It doesn’t look scary at all when looking at the code:

@Transactional
public void inverse(UUID uuid) {
    sampleEntityRepository.findById(uuid).ifPresent(sampleEntity -> {
        boolean allConfigPresent = true;
        for (int i = 0; i < 100; i++) {
            allConfigPresent = allConfigPresent && sampleConfigurationRepository.existsById("key-" + i);
        }
        sampleEntity.setFlag(!sampleEntity.isFlag());
    });
}

We are just executing existsById() on the Spring Data JPA repository. The answer why that method is slow is at the beginning of the method:

sampleEntityRepository.findById(uuid)

and in JPA mapping:

public class SampleEntity {
    @Id
    private UUID id;

    @Fetch(FetchMode.JOIN)
    @JoinColumn(name = "fk_entity")
    @OneToMany(cascade = CascadeType.ALL, fetch = FetchType.EAGER, orphanRemoval = true)
    private Set<SampleSubEntity> subEntities;

    private boolean flag;
}

This means that when we get one SampleEntity by id, we also extract the subEntities from the database because of fetch = FetchType.EAGER. This is not a problem yet. All JPA entities are loaded into the Hibernate session. That mechanism is pretty cool because it gives you the dirty checking functionality. The downside, however, is that the dirty entities need to be flushed by Hibernate to DB. You have different flush strategies in Hibernate. The default one is AUTO: you can read about them in the Javadocs. What you see in the flame graph is exactly Hibernate looking for dirty entities that should be flushed.

What can we do about this? Well, first of all, it should be forbidden to develop a large Hibernate application without reading Vlad Mihalcea’s book. If you are developing such an application, buy that book, it’s great. From my experience, some engineers tend to abuse Hibernate. Let’s look at the code sample I pasted before. We are loading a huge SampleEntity. What are we doing with it?

@Transactional
public void inverse(UUID uuid) {
    sampleEntityRepository.findById(uuid).ifPresent(sampleEntity -> {
        // irrelevant
        sampleEntity.setFlag(!sampleEntity.isFlag());
    });
}

So we’re basically changing one column in one row in the DB. We can do it more efficiently by using the update query, even with Spring Data JPA repository or simple JDBC. But do we really need to use Hibernate everywhere?

CPU - a bit harder

Sometimes the result of a CPU profiler is just the beginning of the fun. Let’s consider the following example:

# little warmup
ab -n 10 -c 1 http://localhost:8081/examples/cpu/matrix-slow

# profiling time
./profiler.sh start -e cpu -f matrix-slow.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 10 -c 1 http://localhost:8081/examples/cpu/matrix-slow
./profiler.sh stop -f matrix-slow.jfr first-application-0.0.1-SNAPSHOT.jar

# little warmup
ab -n 10 -c 1 http://localhost:8081/examples/cpu/matrix-fast

# profiling time
./profiler.sh start -e cpu -f matrix-fast.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 10 -c 1 http://localhost:8081/examples/cpu/matrix-fast
./profiler.sh stop -f matrix-fast.jfr first-application-0.0.1-SNAPSHOT.jar

Let’s see the run times of the matrix-slow request:

              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:  1601 1795 181.5   1785    2078
Waiting:     1600 1794 181.5   1785    2077
Total:       1601 1795 181.5   1786    2078

The profile looks like the following: (HTML)

The whole CPU is wasted in the matrixMultiplySlow method:

public static int[][] matrixMultiplySlow(int[][] a, int[][] b, int size) {
    int[][] result = new int[size][size];
    for (int i = 0; i < size; i++) {
        for (int j = 0; j < size; j++) {
            int sum = 0;
            for (int k = 0; k < size; k++) {
                sum += a[i][k] * b[k][j];
            }
            result[i][j] = sum;
        }
    }
    return result;
}

If we look at the run times of the matrix-fast request:

              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   107  114   6.5    114     128
Waiting:      106  113   6.5    113     128
Total:        107  114   6.5    114     128

That request is 18 times faster than the matrix-slow request, but if we look at the profile (HTML)

We see that the whole CPU is wasted in the method matrixMultiplyFaster:

public static int[][] matrixMultiplyFaster(int[][] a, int[][] b, int size) {
    int[][] result = new int[size][size];
    for (int i = 0; i < size; i++) {
        for (int k = 0; k < size; k++) {
            int current = a[i][k];
            for (int j = 0; j < size; j++) {
                result[i][j] += current * b[k][j];
            }
        }
    }
    return result;
}

Both methods matrixMultiplySlow and matrixMultiplyFaster have the same complexity O(n^3). So why is one faster than the other? Well, if you want to understand exactly how a CPU-intensive algorithm works, you need to understand how the CPU works, which is far away from the topic of this post. Be aware that if you want to optimize such algorithms; you will probably need at least one of the following:

• Knowledge of CPU architecture
• Top-Down performance analysis methodology
• Looking at the ASM of generated methods

Many Java programmers forget that all the execution is done on the CPU. Java needs to use ASM to run on the CPU. That’s basically what the JIT compiler does: It converts your hot methods and loops into optimized ASM. At the assembly level, you can check, for example, if the JIT used vectorized instruction for your loops. So yes, sometimes you need to get dirty with such low-level stuff. For now, async-profiler gives you a hint on which methods to focus.

We will return to this example in the Cache misses section.

Allocation

The most common use cases where the allocations are tracked are:

• decreasing GC run frequency
• finding allocations outside the TLAB, which are done in the slow path
• fighting with single/tens of milliseconds latency, where even the creation of heap objects matters

First, let’s understand how new objects on a heap are created so we have a better understanding of what the async-profiler shows us.

A portion of our Java heap is called an eden. This is a place where new objects are born. Let’s assume for simplicity that eden is a continuous slice of memory. The very efficient way of allocation in such a case is called bumping pointer. We keep a pointer to the first address of the free space:

When we do new Object(), we simply calculate its size, locate the next free address and bump the pointer by sizeof(Object):

But there is one major problem with that technique: We have to synchronize the object allocation if we have more than one thread that can create new objects in parallel, but this is quite costly. We solve this by giving each thread a portion of eden dedicated to only that thread. This portion is called TLAB - thread-local allocation buffer. With this, each thread can use bumping pointer at its TLAB safely and in parallel.

Introducing TLABs creates two more issues that the JVM needs to deal with:

• a thread can allocate an object, but there is not enough space in its TLAB - the JVM creates a new TLAB if there is still space in eden
• a thread can allocate a big object, so it’s not optimal to use the TLAB mechanism - the JVM will use the slow path of the allocation that allocates the object directly in eden or in the old generation

What is important to us is that in both these cases, the JVM emits an event that a profiler can capture. That’s basically how async-profiler samples allocations:

• if the allocation of an object needed a new TLAB - we see an aqua frame for that
• if the allocation was done outside the TLAB - we see a brown frame

In real-world systems, the frequency of GC can be monitored by systems like Grafana or Zabbix. Here we have a synthetic application, so let’s measure the allocation size differently:

# little warmup
ab -n 2 -c 1 http://localhost:8081/examples/alloc/

# measuring the heap allocations of a request
jcmd first-application-0.0.1-SNAPSHOT.jar GC.run
jcmd first-application-0.0.1-SNAPSHOT.jar GC.heap_info
ab -n 10 -c 1 http://localhost:8081/examples/alloc/
jcmd first-application-0.0.1-SNAPSHOT.jar GC.heap_info

# profiling time
./profiler.sh start -e alloc -f alloc.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 1000 -c 1 http://localhost:8081/examples/alloc/
./profiler.sh stop -f alloc.jfr first-application-0.0.1-SNAPSHOT.jar

Let’s look at the output of the GC.heap_info command:

 garbage-first heap   total 1048576K, used 41926K [0x00000000c0000000, 0x0000000100000000)
  region size 1024K, 2 young (2048K), 0 survivors (0K)
 Metaspace       used 67317K, committed 67904K, reserved 1114112K
  class space    used 9868K, committed 10176K, reserved 1048576K

 garbage-first heap   total 1048576K, used 110018K [0x00000000c0000000, 0x0000000100000000)
  region size 1024K, 8 young (8192K), 0 survivors (0K)
 Metaspace       used 67317K, committed 67904K, reserved 1114112K
  class space    used 9868K, committed 10176K, reserved 1048576K

We executed alloc requests ten times, and our heap usage has increased from 41926K to 110018K. So we are creating over 6MB of objects per request on the heap. If we look at the controller source code it’s hard to justify that:

@RestController
@RequestMapping("/examples/alloc")
@RequiredArgsConstructor
class AllocController {
    @GetMapping("/")
    String get() {
        return "OK";
    }
}

Let’s look at the allocation flame graph: (HTML)

The class AbstractRequestLoggingFilter is responsible for over 99% of recorder allocations. You can find its main creation sites using the techniques from the Methods profiling section; feel free to skip it for now. Here is the answer:

@Bean
CommonsRequestLoggingFilter requestLoggingFilter() {
    CommonsRequestLoggingFilter loggingFilter = new CommonsRequestLoggingFilter() {
        @Override
        protected boolean shouldNotFilter(HttpServletRequest request) throws ServletException {
            return !request.getRequestURI().contains("alloc");
        }
    };

    loggingFilter.setIncludeClientInfo(true);
    loggingFilter.setIncludeQueryString(true);
    loggingFilter.setIncludePayload(true);
    loggingFilter.setMaxPayloadLength(5 * 1024 * 1024);
    loggingFilter.setIncludeHeaders(true);
    return loggingFilter;
}

I have seen such code many times; this CommonsRequestLoggingFilterclass helps you log the REST endpoint communication. The setMaxPayloadLength() method sets the maximum number of bytes of the payload which are logged. You can browse over the Spring source code to see that the implementation creates byte arrays of such size in the constructor. No matter how big the payload is, we always create a 5MB array here.

The advice that I gave to users of that code was to create their own filter that would do the same job but allocate the array lazily.

Allocation - humongous objects

If you use the G1 garbage collector, JVM’s default since JDK 9, your heap is divided into regions. The region sizes vary from 1 MB to 32 MB depending on the heap size. The goal is to have no more than 2048 regions. You can check the region size for different heap sizes with the following:

java -Xms1G -Xmx1G -Xlog:gc,exit*=debug -version

The output contains a line containing the region size 1024K information.

If you are trying to allocate an object larger or equal to half of the region size, you are doing a humongous allocation. Long story short: It has been, and it is, painful. It is allocated directly in the old generation but is cleared during minor GCs. I saw situations where G1 GC needed to invoke a FullGC phase because of the humongous allocation. If you do much of this, G1 will also invoke more concurrent collections, which can waste your CPU.

While running the previous example, you could spot in first-application-0.0.1-SNAPSHOT.jar logs:

...
[70,149s][debug][gc,humongous] GC(34) Reclaimed humongous region 436 (object size 5242896 @ 0x00000000db400000)
[70,149s][debug][gc,humongous] GC(34) Reclaimed humongous region 442 (object size 5242896 @ 0x00000000dba00000)
[70,149s][debug][gc,humongous] GC(34) Reclaimed humongous region 448 (object size 5242896 @ 0x00000000dc000000)
[70,149s][debug][gc,humongous] GC(34) Reclaimed humongous region 454 (object size 5242896 @ 0x00000000dc600000)
...

These GC logs tell us that some humongous object of size 5242896 was reclaimed. The nice thing about JFR files is that they also keep the size of sampled allocations. Using this, we should be able to find out the stack trace that has created that object.

We don’t need sophisticated JFR viewers for that. We get the jfr command with any JDK distribution. Let’s use it:

$ jfr summary alloc.jfr
...
 Event Type                          Count  Size (bytes) 
=========================================================
 jdk.ObjectAllocationOutsideTLAB      1013         19220
 jdk.ObjectAllocationInNewTLAB         359          6719
...

Let’s focus on allocations outside the TLAB; it is unlikely to allocate humongous objects in the TLAB.

$ jfr print --events jdk.ObjectAllocationOutsideTLAB --stack-depth 10 alloc.jfr
...
jdk.ObjectAllocationOutsideTLAB {
  startTime = 2022-12-05T08:56:04.354766183Z
  objectClass = byte[] (classLoader = null)
  allocationSize = 5242896
  eventThread = "http-nio-8081-exec-9" (javaThreadId = 49)
  stackTrace = [
    java.io.ByteArrayOutputStream.<init>(int) line: 81
    org.springframework.web.util.ContentCachingRequestWrapper.<init>(HttpServletRequest, int) line: 90
    org.springframework.web.filter.AbstractRequestLoggingFilter.doFilterInternal(HttpServletRequest, HttpServletResponse, FilterChain) line: 281
    org.springframework.web.filter.OncePerRequestFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 116
    org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 185
    org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 158
    org.springframework.web.filter.RequestContextFilter.doFilterInternal(HttpServletRequest, HttpServletResponse, FilterChain) line: 100
    org.springframework.web.filter.OncePerRequestFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 116
    org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 185
    org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 158
    ...
  ]
}
...

We can easily match allocationSize = 5242896` with the object size from the GC logs, so we can find and eliminate humongous allocations using that technique. You can filter the allocation JFR file for objects with a size larger or equal to half of our G1 region size. All of these allocations are humongous allocations.

Allocation - live objects

Now on to memory leaks, which form the reason for tracking live object allocations:

A memory leak occurs when a Garbage Collector cannot collect Objects that are no longer needed by the Java application.

Memory leaks are one of the most common problems related to Java heaps; the other is

• not enough space on a heap - sometimes, a Java application may work fine with the heap it has, but there is the possibility to run a part of the application that needs more heap than specified via -Xmx
• a gray area between - these are cases when we allocate memory indefinitely, but our application needs these objects

How can we detect memory leaks? The GC emits the following kind of entry at the end of each GC cycle into the GC logs at info level:

GC(11536) Pause Young (Normal) (G1 Evacuation Pause) 6746M->2016M(8192M) 40.514ms

You can find three sizes in such an entry A->B(C) that are:

• A - used size of a heap before GC cycle
• B - used size of a heap after GC cycle
• C - the current size of a whole heap

If we take the B value from each collection and put it on a chart, we can generate the Heap after GC chart. We can use such a chart to then detect if we have a memory leak: If a chart looks like those (from a 7 days period):

then there is no memory leak. The garbage collector can clean up the heap to the same level every day. The chart with a memory leak looks like the following one:

These spikes to the roof are to-space exhausted situations in the G1 algorithm; those are not OutOfMemoryErrors. After each of those spikes, there was a Full GC phase that is a failover in that algorithm.

Here is an example of the not enough space on a heap problem:

This one spike is an OutOfMemoryError. One service was run with arguments that needed ~16GB on a heap to complete. Unfortunately -Xmx was set to 4GB. It is not a memory leak.

We must be careful if our application is entirely stateless and we use GC with young/old generations (like G1, parallel, serial, and CMS). We must remember that objects from a memory leak live in the old generation. In stateless applications, that part of the heap can be cleared even once a week. Here is an example recording 3 days of the stateless application:

It looks like a memory leak, the min(heap after GC) increasing every day, but if we look at the same chart with one additional day:

The GC cleared the heap to the previous level. This was done by an old-generation cleanup that didn’t happen in the previous days.

The Heap after GC chart can be generated by probing through JMX. The JVM gives that information via mBeans:

• java.lang:type=GarbageCollector,name=G1 Young Generation
• java.lang:type=GarbageCollector,name=G1 Old Generation

Both mBeans provide attributes with the name LastGcInfo from which we can extract the needed information.

Most memory leaks I discovered in recent years in enterprise applications were either in frameworks/libraries or in some kind of bridge between them. Recreating such an issue in our example application would require introducing a lot of strange dependencies, so I chose to recreate one custom-made heap memory leak I discovered a few years ago.

# preparation
curl http://localhost:8081/examples/leak/prepare
ab -n 1000 -c 4 http://localhost:8081/examples/leak/do-leak

# profiling
jcmd first-application-0.0.1-SNAPSHOT.jar GC.run 
jcmd first-application-0.0.1-SNAPSHOT.jar GC.heap_info
./profiler.sh start -e alloc --live -f live.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 1000000 -c 4 http://localhost:8081/examples/leak/do-leak
jcmd first-application-0.0.1-SNAPSHOT.jar GC.run
jcmd first-application-0.0.1-SNAPSHOT.jar GC.heap_info
./profiler.sh stop -f live.jfr first-application-0.0.1-SNAPSHOT.jar

Let’s look at the output of the GC.heap_info commands that were invoked soon after running the GC:

 garbage-first heap   total 1048576K, used 50144K [0x00000000c0000000, 0x0000000100000000)
  region size 1024K, 1 young (1024K), 0 survivors (0K)
 Metaspace       used 68750K, committed 69376K, reserved 1114112K
  class space    used 10054K, committed 10368K, reserved 1048576K

 garbage-first heap   total 1048576K, used 237806K [0x00000000c0000000, 0x0000000100000000)
  region size 1024K, 1 young (1024K), 0 survivors (0K)
 Metaspace       used 68841K, committed 69440K, reserved 1114112K
  class space    used 10063K, committed 10368K, reserved 1048576K

So invoking our do-leak request created ~183MB of objects that GC couldn’t free.

Let’s look at the allocation flame graph with the --live option enabled: (HTML)

The largest part of the leak is created in the JdbcQueryProfiler class; let’s look at the sources:

class JdbcQueryProfiler {
    private final Map<String, ProfilingData> profilingResults = new ConcurrentHashMap<>();
  
    <T> T runWithProfiler(String queryStr, Supplier<T> query) {
        StopWatch stopWatch = new StopWatch();
        stopWatch.start();
        T ret = query.get();
        stopWatch.stop();
        profilingResults.computeIfAbsent(queryStr, ProfilingData::new).nextInvocation(stopWatch.getTotalTimeMillis());
        return ret;
    }
    // ...
}

So that class calculates the execution time for each query and remembers it in some ProfilingData structure that is placed in ConcurrentHashMap. That doesn’t look scary; as long as we use parametrized queries under our control, the map should have a finite size. Let’s look at the usage of the runWithProfiler() method:

String getValueForKey(int key) {
    String sql = "select a_value from LEAKY_ENTITY where a_Key = " + key;

    return jdbcQueryProfiler.runWithProfiler(sql, () -> {
        try {
            return jdbcTemplate.queryForObject(sql, String.class);
        } catch (EmptyResultDataAccessException e) {
            return null;
        }
    });
}

So, well, so good; we are not using parameterized queries; we create new query strings for every key. This way mentioned ConcurrentHashMap is growing with every new key passed to the getValueForKey method.

If you have a heap memory leak in your application, then you have two groups of objects:

• Live set - a group of objects that are still needed by your application
• Memory leak - a group of objects that are no longer needed

Garbage collectors cannot free the second group if there is at least one strong reference from the live set to the memory leak. The biggest problems with diagnosing memory leaks are:

• the fact that the object was created is not an issue - it was created because it was needed for something
• the fact that the mentioned reference was created is not an issue - it had some purpose too
• we need to understand why that reference was not removed from our application

The last one is not trivial. All the observability/profiling tools give us a great possibility to understand why some event has happened, but with memory leaks, we need to understand why something has not yet happened. Two additional tools come to our rescue:

• heap dump - shows us the current state of a heap - we can find out what kind of objects are there but shouldn’t be
• profiler - shows us where these objects were created

In this simple example, any of those tools is enough. In more complicated ones, I needed both to find the root cause of the problem. It is nice to finally have a tool that can profile memory leaks on production systems.

It is worth mentioning that the --live option is available only since async-profiler 2.9, it needs JDK >= 11 and might still contain bugs. I didn’t have a chance to test it on any production system yet.

Locks

Async-profiler has a lock mode. This mode is useful when looking into look contention in our application. Let’s try to use it and understand the internals of ConcurrentHashMap. The get() method is obviously lock-free, but what about computeIfAbsent()? Let’s profile a code that uses it:

class LockService {
    private final Map<String, String> map = new ConcurrentHashMap<>();
  
    LockService() {
        String a = "AaAa";
        String b = "BBBB";
        log.info("Hashcode equals: {}", a.hashCode() == b.hashCode()); // true
        map.computeIfAbsent(a, s -> a);
        map.computeIfAbsent(b, s -> b);
    }
  
    void withLock(String key) {
        map.computeIfAbsent(key, s -> key);
    }
  
    void withoutLock(String key) {
        if (map.get(key) == null) {
            map.computeIfAbsent(key, s -> key);
        }
    }
}

Let’s use lock mode to profile that:

# preparation
ab -n 100 -c 1 http://localhost:8081/examples/lock/with-lock
ab -n 100 -c 1 http://localhost:8081/examples/lock/without-lock

# profiling
./profiler.sh start -e lock -f lock.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 100000 -c 100 http://localhost:8081/examples/lock/with-lock
ab -n 100000 -c 100 http://localhost:8081/examples/lock/without-lock
./profiler.sh stop -f lock.jfr first-application-0.0.1-SNAPSHOT.jar

The lock flame graph: (HTML)

I highlighted the LockController occurrence. Let’s zoom it:

We see that only the withLock() method acquires locks. You can study the internals of the computeIfAbsent() method to see that it might lock on hash collisions. The easiest way to confirm this is by creating a small program that triggers hash collisions on purpose.

public static void main(String[] args) {
    Map<String, String> map = new ConcurrentHashMap<>();
    String a = "AaAa";
    String b = "BBBB";

    map.computeIfAbsent(a, s -> a);

    // it enters the synchronized section here
    map.computeIfAbsent(b, s -> b);

    // it enters the synchronized section here, and all the following
    // execution of computeIfAbsent with "BBBB" as a key.
    map.computeIfAbsent(b, s -> b);
    map.computeIfAbsent(b, s -> b);
    map.computeIfAbsent(b, s -> b);
    map.computeIfAbsent(b, s -> b);
}

If you observe a considerable lock contention in this method, and most of the time a key is already in the map; then you may consider the approach used in the withoutLock() method.

Time to safepoint

The common misconception in the Java world is that garbage collectors need a Stop-the-world (STW) phase to clean dead objects: But not only GC needs it. Other internal mechanisms require application threads to be paused. For example, the JVM needs an STW phase to deoptimize some compilations and to revoke biased locks. Let’s get a closer look at how the STW phase works.

On our JVM, there are running some application threads:

While running those threads from time to time, JVM needs to do some work in the STW phase. So it starts this phase, with a global safepoint request, which informs every thread to go to “sleep”:

Every thread has to find out about this. Stopping at a safepoint is cooperative: Each thread checks at certain points in the code if it needs to suspend. The time in which threads will be aware of an STW phase is different for every thread. Every thread has to wait for the slowest one. The time between starting an STW phase, and the slowest thread suspension, is called time to safepoint:

JVM threads can do the work that needs the STW phase only after every thread is asleep. The time when all application threads sleep, is called safepoint operation time:

When the JVM finishes its work, application threads are wakened up:

If the application suffers from long STW phases, then most of the time, those are GC cycles, and that information can be found in the GC logs or JFR. But the situation is more tricky if the application has one thread that slows down every other from reaching the safepoint.

# preparation
curl http://localhost:8081/examples/tts/start
ab -n 100 -c 1 http://localhost:8081/examples/tts/execute

# profiling
./profiler.sh start --ttsp -f tts.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 100 -c 1 http://localhost:8081/examples/tts/execute
./profiler.sh stop -f tts.jfr first-application-0.0.1-SNAPSHOT.jar

In safepoint logs (you need to run your JVM with the -Xlog:safepoint flag), we can see:

[105,372s][info ][safepoint   ] Safepoint "ThreadDump", Time since last: 156842 ns, Reaching safepoint: 13381 ns, At safepoint: 120662 ns, Total: 134043 ns
[105,372s][info ][safepoint   ] Safepoint "ThreadDump", Time since last: 157113 ns, Reaching safepoint: 14738 ns, At safepoint: 120252 ns, Total: 134990 ns
[105,373s][info ][safepoint   ] Safepoint "ThreadDump", Time since last: 157676 ns, Reaching safepoint: 13700 ns, At safepoint: 120487 ns, Total: 134187 ns
[105,402s][info ][safepoint   ] Safepoint "ThreadDump", Time since last: 159020 ns, Reaching safepoint: 29524545 ns, At safepoint: 160702 ns, Total: 29685247 ns

Reaching safepoint contains the time to safepoint. Most of the time, it is <15 ms, but we also see one outlier: 29 ms. Async-profiler in --ttsp mode collects samples between:

• SafepointSynchronize::begin, and
• RuntimeService::record_safepoint_synchronized

During that time, our application threads are trying to reach a safepoint: (HTML)

We can see that most of the gathered samples are executing arraycopy, invoked from TtsController. The time to safepoint issues that I have approached so far are:

• arraycopy - as in our example
• old JDK + loops - since JDK 11u4 we have an loop strip mining optimization working correctly (after fixing JDK-8220374), before that if you had a counted loop, it could be compiled without any check for safepoint
• swap - when your application thread executes some work in thread_in_vm state (after calling some native method),
  and during that execution, it waits for some pages to be swapped in/out, which can slow down reaching the safepoint

The solution for the arraycopy issue is to copy larger arrays by some custom method, which might use arraycopy for smaller sub-arrays. It will be a bit slower, but it will not slow down the whole application when reaching a safepoint is required.

For the swap issue, just disable the swap.

Methods

Async-profiler can instrument a method so that we can see all the stack traces with this method on the top. To achieve that, async-profiler uses instrumentation.

Big warning: It’s already pointed out in the README of the profiler that if you are not running the profiler from agentpath, then the first instrumentation of a Java method can result in a code cache flush. It’s not the fault of the async-profiler; it’s the nature of all instrumentation-based profilers combined with JVM’s code. Here is a comment from the JVM sources:

// Deoptimize all compiled code that depends on the classes redefined. // // If the can_redefine_classes capability is obtained in the onload // phase then the compiler has recorded all dependencies from startup. // In that case we need only deoptimize and throw away all compiled code // that depends on the class. // // If can_redefine_classes is obtained sometime after the onload // phase then the dependency information may be incomplete. In that case // the first call to RedefineClasses causes all compiled code to be // thrown away. As can_redefine_classes has been obtained then // all future compilations will record dependencies so second and // subsequent calls to RedefineClasses need only throw away code // that depends on the class.

You can check the README PR discussion for more information on this topic. But let’s focus on the usage of the mode for our purposes. In this case, we could easily do it with a plain IDE debugger, but there are situations where something is happening only in one environment, or we are tracing some issues we do not know how to reproduce.

Since Spring beans are usually created during applications startup, let’s run our application that way:

java \
-agentpath:/path/to/libasyncProfiler.so=start,event="org.springframework.web.filter.AbstractRequestLoggingFilter.<init>" \
-jar first-application/target/first-application-0.0.1-SNAPSHOT.jar

The AbstractRequestLoggingFilter.<init> is simply a constructor. We are trying to find out where such an object is created. After our application is started, we can execute such a command in the profiler directory:

./profiler.sh stop first-application-0.0.1-SNAPSHOT.jar

It will print us to one stack trace:

--- Execution profile ---
Total samples       : 1

--- 1 calls (100.00%), 1 sample
  [ 0] org.springframework.web.filter.AbstractRequestLoggingFilter.<init>
  [ 1] org.springframework.web.filter.CommonsRequestLoggingFilter.<init>
  [ 2] com.example.firstapplication.examples.alloc.AllocConfiguration$1.<init>
  [ 3] com.example.firstapplication.examples.alloc.AllocConfiguration.requestLoggingFilter
  [ 4] com.example.firstapplication.examples.alloc.AllocConfiguration$$SpringCGLIB$$0.CGLIB$requestLoggingFilter$0
  [ 5] com.example.firstapplication.examples.alloc.AllocConfiguration$$SpringCGLIB$$2.invoke
...
  [24] org.springframework.beans.factory.support.AbstractBeanFactory.getBean
...
  [33] org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup
...
  [58] org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>
...
  [78] org.springframework.boot.loader.JarLauncher.main

       calls  percent  samples  top
  ----------  -------  -------  ---
           1  100.00%        1  org.springframework.web.filter.AbstractRequestLoggingFilter.<init>

We have all the information that we need. The object is created in AllocConfiguration during creation of CommonsRequestLoggingFilter bean.

One of the other use cases where I used to use method profiling was finding memory leaks. I knew which types were leaking from the heap dump, and with method profiling, I could see where objects of these types were created. Consider this a fallback when the dedicated mode does not work.

Native functions

Not only can you trace Java code with the async-profiler but also a native one. That way of profiling doesn’t cause deoptimizations.

Some native functions are worth a better look; let’s cover them quickly.

Exceptions

How many exceptions should be thrown if your application works without any outage/downtime and everything is stable? Exceptions should be thrown if something unexpected happens. Unfortunately, I saw an application that used the exception-control-flow approach more common in languages like Python. Creating a new exception is a CPU-intensive operation since, by default, it fills the stack trace. I once saw an application that consumed ~15% of its CPU time on just creating new exceptions. You can use async-profiler in method mode with event Java_java_lang_Throwable_fillInStackTrace if you want to see where exceptions are created.

Let’s start our application with profiler enabled from the start to see also how many exceptions are thrown during the startup of a Spring Boot application, just for fun:

java \
-agentpath:/path/to/libasyncProfiler.so=start,jfr,file=exceptions.jfr,event="Java_java_lang_Throwable_fillInStackTrace" \
-jar first-application/target/first-application-0.0.1-SNAPSHOT.jar

After the startup, let’s run:

ab -n 100 -c 1 http://localhost:8081/examples/exc/
./profiler.sh stop -f exceptions.jfr first-application-0.0.1-SNAPSHOT.jar

The flame graph is too large to post it here as an image, sorry. Spring Boot, in that case, threw 12478 exceptions. You can play with HTML. Let’s focus on our synthetic controller:

Source code of the controller:

@GetMapping("/")
String flowControl() {
    ThreadLocalRandom random = ThreadLocalRandom.current();
    try {
        if (!random.nextBoolean()) {
            throw new IllegalArgumentException("Random returned false");
        }
    } catch (IllegalArgumentException e) {
        return "EXC";
    }

    return "OK";
}

If you care about performance, don’t use the exception-control-flow approach. If you really need such a code, reuse exception options like ANTLR or create an exception constructor that doesn’t fill the stack trace:

/**
 * Constructs a new exception with the specified detail message,
 * cause, suppression enabled or disabled, and writable stack
 * trace enabled or disabled.
 *
 * @param  message the detail message.
 * @param cause the cause.  (A {@code null} value is permitted,
 * and indicates that the cause is nonexistent or unknown.)
 * @param enableSuppression whether or not suppression is enabled
 *                          or disabled
 * @param writableStackTrace whether or not the stack trace should
 *                           be writable
 * @since 1.7
 */
protected Exception(String message, Throwable cause,
                    boolean enableSuppression,
                    boolean writableStackTrace) {
    super(message, cause, enableSuppression, writableStackTrace);
}

Just set writableStackTrace to false. It will be rather ugly but faster.

G1GC humongous allocation

We already saw how to detect humongous objects with allocation mode. Since async-profiler can also instrument JVM code, and allocations of a humongous objects are nothing else than invocations of C++ code, we can take advantage of that. If you want to check where humongous objects are allocated, you can use native functions mode with event G1CollectedHeap::humongous_obj_allocate. This approach may have lower overhead but won’t give you sizes of allocated objects.

# little warmup
ab -n 2 -c 1 http://localhost:8081/examples/alloc/

# profiling time
./profiler.sh start -e "G1CollectedHeap::humongous_obj_allocate" -f humongous.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 1000 -c 1 http://localhost:8081/examples/alloc/
./profiler.sh stop -f humongous.jfr first-application-0.0.1-SNAPSHOT.jar

The flame graph is almost the same as in alloc mode; we can see some JVM yellow frames this time too: (HTML)

Thread start

Starting a platform thread is an expensive operation too. The number of started threads can be easily monitored with any JMX-based monitoring tool like JMC. Here is the MBean with the value of all the created threads:

java.lang:type=Threading 
attribute=TotalStartedThreadCount

If you monitor that value and the chart like that:

Then you might want to check who is creating those short-living threads: We use async-profiler with the JVM_StartThread event in native functions mode for this purpose:

# little warmup
ab -n 100 -c 1 http://localhost:8081/examples/thread/

# profiling time
./profiler.sh start -e "JVM_StartThread" -f threads.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 1000 -c 1 http://localhost:8081/examples/thread/
./profiler.sh stop -f threads.jfr first-application-0.0.1-SNAPSHOT.jar

The flame graph: (HTML)

This flame graph is not really complicated. But it is only a small example. In real life, such flame graphs are larger.

The code responsible for the thread creation observed in the flame graph is the following:

@SneakyThrows
@GetMapping("/")
String doInNewThread() {
    ExecutorService threadPool = Executors.newFixedThreadPool(1);
    return threadPool.submit(() -> {
        return "OK";
    }).get();
}

And yes, I saw such a pattern in a real production application. The intention was to have a fixed thread pool and delegate tasks to it, but by mistake, someone created that pool for every request.

Class loading

Similar to creating short-living threads, I saw an application that created plenty of short-living class definitions. I know there are some use cases for such behavior, But it has been an accident in this case. You can monitor the number of loaded classes with JMX:

java.lang:type=ClassLoading
attribute=TotalLoadedClassCount

There are some internals of the JVM, like reflection or debugging, which are used in a variety of frameworks, that can generate new class definitions during runtime: So increasing that number (even after warmup) doesn’t mean that we have a problem already. But if TotalLoadedClassCount is much higher than LoadedClassCount, then we might have a problem. You can find the creator of those classes with method mode and the event: Java_java_lang_ClassLoader_defineClass1.

To be honest, I saw such an issue only once and cannot reproduce it now. Making a synthetic example for this use-case seems wrong, so I will just keep you with the knowledge that there is such a possibility, especially if you purposefully create classes dynamically.

Perf events

Async-profiler can also help you with low-level diagnosis where you want to correlate perf events with Java code:

• context-switches - to find out which parts of your Java code do context switching
• cache-misses - which part of your code can stall due to cache misses - this information is harder to analyze if you have many context switches
• LLC-load-misses- which part of your code needs a lot of data directly from RAM which is not cached
• …

I want to describe the three in more detail in the following.

Cache misses

Let’s return to the example with matrix multiplication from the CPU - a bit harder section. I usually start by looking at basic CPU performance counters to see what our CPU is doing in the slow and the fast multiplication. This is the textbook example of cache misses and their importance for performance. I like to start with the JMH test to profile the specific code properly.

I’ve prepared such a benchmark in the jmh-suite module. Let’s run it with the perf profiler:

java -jar jmh-suite/target/benchmarks.jar -prof perf

The fast algorithm (I’ve cut the output to the most interesting metrics):

         20 544,42 msec task-clock                       #    1,008 CPUs utilized          
    49 510 157 799      L1-dcache-loads                  #    2,410 G/sec                    (38,55%)
     9 300 675 824      L1-dcache-load-misses            #   18,79% of all L1-dcache accesses  (38,55%)
     1 635 877 333      LLC-loads                        #   79,626 M/sec                    (30,80%)
        27 833 149      LLC-load-misses                  #    1,70% of all LL-cache accesses  (30,76%)

The slow one:

         22 291,74 msec task-clock                       #    1,008 CPUs utilized          
    71 632 332 204      L1-dcache-loads                  #    3,213 G/sec                    (38,51%)
    29 718 804 848      L1-dcache-load-misses            #   41,49% of all L1-dcache accesses  (38,50%)
     6 909 042 687      LLC-loads                        #  309,937 M/sec                    (30,79%)
        10 043 405      LLC-load-misses                  #    0,15% of all LL-cache accesses  (30,79%)

The slower algorithm has three times more L1 data cache misses and over four times more last-level cache loads. We can now use async-profiler in three different modes:

java -jar jmh-suite/target/benchmarks.jar -prof async:libPath=/path/to/libasyncProfiler.so\;event=cache-misses\;output=jfr
java -jar jmh-suite/target/benchmarks.jar -prof async:libPath==/path/to/libasyncProfiler.so\;event=L1-dcache-load-misses\;output=jfr
java -jar jmh-suite/target/benchmarks.jar -prof async:libPath==/path/to/libasyncProfiler.so\;event=LLC-load-misses\;output=jfr

All three flame graphs are very similar; let’s take a look at cache-misses: (HTML)

I added the line numbers this time, so we could see exactly where the problem was. ~82% of cache misses occurred in the same line:

sum += a[i][k] * b[k][j];

This line is nested inside three loops. The order of loops is i, j, k. If we unroll the last loop four times we would get the following:

sum += a[i][k + 0] * b[k + 0][j];
sum += a[i][k + 1] * b[k + 1][j];
sum += a[i][k + 2] * b[k + 2][j];
sum += a[i][k + 3] * b[k + 3][j];

Let’s look at this code from a memory layout perspective. The array a[i] is a contiguous part of memory. That’s how Java allocates arrays. Elements a[i][k + 0] … a[i][k + 3] are very close to each other and are loaded sequentially. The CPU loads small blocks of memory from RAM into the cache if the block is not already there. Accessing data in a sequential pattern is, therefore, far less expensive.

The access pattern to table b is completely different. The b[k + x] is just a pointer to a table. It is somewhere on the heap, but where exactly? Well, we cannot control that. Element b[k + 0][j] may be in a completely different place than b[k + 1][j]. That’s unfortunate for the CPU. This is why the speed difference between both matrix multiplications is not as large as expected.

Memory access patterns are the key here. The matrixMultiplyFaster algorithm accesses the table a mostly sequentially, which is why it’s faster.

I don’t want to go into detail about what is happening in the CPU with these algorithms. This post aims to teach the usage of async-profiler, not CPU architecture and algorithm engineering. If you want to go deeper with that knowledge, a very good book for a start is Denis Bakhvalov - Performance Analysis and Tuning on Modern CPUs. It’s not about Java, but I cannot recommend any Java-centric book related to CPU architecture, as it’s still a relatively niche topic. I know that two very good performance engineers are writing one now. When it is published, I will paste a link here.

Page faults

Every process running on Linux contains its own virtual memory. If a process needs more memory, it invokes functions like malloc or mmap. The OS guarantees the returned memory to be readable/writable by the current process. But this does not mean that any block of physical RAM has been reserved for the process.

The OS is smart enough to decide whether that fault should be converted into a SEGFAULT or should trigger the kernel to map RAM to the process’s virtual memory because it was previously promised to the process.

Java is a process from an OS perspective, nothing less, nothing more. Knowing that we can trace page fault events to detect why our application consumes more RAM. It may be a native memory leak or some framework/library/JVM bug.

But this is not perfect for tracing leaks since it shows every request for additional RAM, including ones that may be freed in the future. I know that Andrei Pangin is working on a native memory leak detector that will trace allocations that haven’t been freed, but for now, that feature is not in the latest release.

As an example, let’s run our application with and without -XX:+AlwaysPreTouch, forcing the JVM to access all allocated memory after requesting it from the OS. This allows us to find where Java needs more RAM after startup. We will use the heap memory leak that we used before:

java -Xmx1G -Xms1G -XX:+AlwaysPreTouch \
-jar first-application/target/first-application-0.0.1-SNAPSHOT.jar

In the other console, let’s do the following:

ab -n 10000 -c 4 http://localhost:8081/examples/leak/do-leak

./profiler.sh start -e page-faults -f page-faults-apt-on.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 1000000 -c 4 http://localhost:8081/examples/leak/do-leak
./profiler.sh stop -f page-faults-apt-on.jfr first-application-0.0.1-SNAPSHOT.jar

Now let’s do the same without -XX:+AlwaysPreTouch:

java -Xmx1G -Xms1G \
-jar first-application/target/first-application-0.0.1-SNAPSHOT.jar

In the other console, let’s execute the following:

ab -n 10000 -c 4 http://localhost:8081/examples/leak/do-leak

./profiler.sh start -e page-faults -f page-faults-apt-off.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 1000000 -c 4 http://localhost:8081/examples/leak/do-leak
./profiler.sh stop -f page-faults-apt-off.jfr first-application-0.0.1-SNAPSHOT.jar

Flame graph without -XX:+AlwaysPreTouch: (HTML)

Most of the need for additional RAM is acquired in GC threads, but there are some page faults in our Java code (big green flame). These page faults can hurt your performance and make your latency less predictable.

Flame graph with -XX:+AlwaysPreTouch: (HTML)

Almost all the frames needing additional RAM now belong to compiler threads. This is due the code heap growing with the compilation of new methods.

I was able to isolate and recreate the memory leak that I described in JDK-8240723 with that mode.

Cycles

If you need better visibility of what your kernel is doing, then you may consider choosing the cycles event instead of cpu. This may be useful for low-latency applications or while chasing bugs in the kernel (those also exist). Let’s see the difference:

# warmup
curl -v http://localhost:8081/examples/cycles/

# profiling
./profiler.sh start -e cpu -f cycles-cpu.jfr first-application-0.0.1-SNAPSHOT.jar
curl -v http://localhost:8081/examples/cycles/
./profiler.sh stop -f cycles-cpu.jfr first-application-0.0.1-SNAPSHOT.jar
./profiler.sh start -e cycles -f cycles-cycles.jfr first-application-0.0.1-SNAPSHOT.jar
curl -v http://localhost:8081/examples/cycles/
./profiler.sh stop -f cycles-cycles.jfr first-application-0.0.1-SNAPSHOT.jar

The flame graph for cpu profiling: (HTML)

Corresponding profile for cycles event: (HTML)

As we can see, the cycles profile is more detailed.

Native memory leaks

With the 4.0 Async-profiler release, we can use the new nativemem mode which:

… records malloc, realloc, calloc and free calls with the addresses, so that allocations can be matched with frees.

This mode is extremely helpful with native memory leak detection. I already wrote an article on this topic. Now let’s just focus on usage of Async-profiler for a problem that I had in the past.

Our application is run with 1GB fixed heap size:

java -Xmx1G -Xms1G -XX:+AlwaysPreTouch \
-jar first-application/target/first-application-0.0.1-SNAPSHOT.jar

This native memory “leak” I want to show you is correlated with AWS S3 uploads. To make it easier to recreate let’s run localstacks on our PC and create a proper bucket and a file to upload:

docker run --rm -it -p 4566:4566 -p 4571:4571 localstack/localstack
aws --endpoint-url=http://localhost:4566 s3 mb s3://temp-bucket
dd if=/dev/zero of=/tmp/to_upload.tmp bs=1M count=15

Let’s invoke our app:

ab -n 1  http://localhost:8081/examples/aws/upload

Now let’s check how much memory is used by the application from an OS perspective:

jcmd first-application-0.0.1-SNAPSHOT.jar GC.run
smem -c "pid command rss pss" -a -P "first-application-0.0.1-SNAPSHOT.jar"

The output:

  PID Command                                                       RSS     PSS 
18206 /home/pasq/JDK/amazon-corretto-17.0.1.12.1-linux-x64/bin/ 1388836 1372957 

Let’s invoke more of this endpoint with the profiler attached (the profiler.sh script is gone, we now use asprof executable):

./asprof start -e nativemem -f nativemem.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 200 -c 10 http://localhost:8081/examples/aws/upload
jcmd first-application-0.0.1-SNAPSHOT.jar GC.run
./asprof stop -f nativemem.jfr first-application-0.0.1-SNAPSHOT.jar

Let’s check how much memory is used now:

smem -c "pid command rss pss" -a -P "first-application-0.0.1-SNAPSHOT.jar"

The output:

  PID Command                                                       RSS     PSS 
18206 /home/pasq/JDK/amazon-corretto-17.0.1.12.1-linux-x64/bin/ 1632140 1616206 

We can clearly see that both RSS and PSS grew. Let’s see what data were gathered by the profiler. Let’s convert JFR to a flame graph first:

./jfrconv --total --nativemem --leak nativemem.jfr nativemem.html

(HTML)

We can see that nativemem mode blames the AwsService.upload method for the “leak”. In the HTML version you can see also other AWS-related allocations without any Java stack traces, but let’s focus on our code:

void upload() {
    S3CrtAsyncClientBuilder s3CrtAsyncClientBuilder = S3AsyncClient.crtBuilder()
            .endpointOverride(new URI("http://127.0.0.1:4566"))
            .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(ACCESS_KEY_ID, SECRET_ACCESS_KEY)))
            .region(REGION);

    try (S3AsyncClient s3Client = s3CrtAsyncClientBuilder.build()) {
        s3Client
                .putObject(
                        req -> req.bucket(BUCKET).key(NAME),
                        AsyncRequestBody.fromFile(Paths.get(FILE_TO_UPLOAD)))
                .join();
    }
}

We can also see (on the flame graph) that the native memory allocation was done in the DefaultS3CrtClientBuilder.build method. That line is covered with try-with-resources, so the close method on the returned object should be invoked automatically. The method is invoked, but it doesn’t clean all the native allocations done by the builder. This issue I found with AWS S3 libraries with the following versions:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.25.23</version>
</dependency>
<dependency>
    <groupId>software.amazon.awssdk.crt</groupId>
    <artifactId>aws-crt</artifactId>
    <version>0.29.14</version>
</dependency>

The behavior may vary with different versions. You can check how it looks with the newest ones.

Filtering single request

Why aggregated results are not enough

So far, we have been looking at the profile of a whole application. But what if the app works well but there are some slower requests from time to time, and we want to know why? In such an application, where one request is handled by one thread, we can extract the profile of a single request. The JFR file contains all the information needed; we just need to filter them out. To do it, we need to have a log that will tell us which thread was responsible for the execution of the request at the time of the execution. Tomcat, embedded into Spring Boot, has access logs with all that information.

I configured our example application with access logs in the format:

[%t] [%r] [%s] [%D ms] [%I]

Here is a short explanation of that magic:

• %t - time of finishing handling of the request
• %r - requested URI
• %s - response status code
• %D - duration time in milliseconds
• %I - thread that handled request

Let’s see it in action.

# warmup
ab -n 20 -c 4 http://localhost:8081/examples/filtering/

# profiling of the first request
./profiler.sh start -e wall -f filtering.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 50 -c 4 http://localhost:8081/examples/filtering/
./profiler.sh stop -f filtering.jfr first-application-0.0.1-SNAPSHOT.jar

In the access logs, we can spot faster and slower requests:

[05/Dec/2022:18:41:57 +0100] [GET /examples/filtering/ HTTP/1.0] [200] [1044 ms] [http-nio-8081-exec-2]
[05/Dec/2022:18:41:57 +0100] [GET /examples/filtering/ HTTP/1.0] [200] [2779 ms] [http-nio-8081-exec-5]
[05/Dec/2022:18:41:58 +0100] [GET /examples/filtering/ HTTP/1.0] [200] [1048 ms] [http-nio-8081-exec-8]
[05/Dec/2022:18:41:58 +0100] [GET /examples/filtering/ HTTP/1.0] [200] [1052 ms] [http-nio-8081-exec-9]
[05/Dec/2022:18:41:59 +0100] [GET /examples/filtering/ HTTP/1.0] [200] [2829 ms] [http-nio-8081-exec-7]
[05/Dec/2022:18:41:59 +0100] [GET /examples/filtering/ HTTP/1.0] [200] [1058 ms] [http-nio-8081-exec-1]

Let’s load the JFR into my viewer and look at the flame graph of the whole application: (HTML)

It’s hard to guess why some requests are slower than others. We can see two different methods executed at the top of the flame graph:

• matrixMultiplySlow()
• matrixMultiplyFaster()

We cannot conclude from that which one is responsible for worse latency. Let’s add filters to that graph to understand the latency of the second request from pasted access log:

[05/Dec/2022:18:41:57 +0100] [GET /examples/filtering/ HTTP/1.0] [200] [2779 ms] [http-nio-8081-exec-5]

• Access log filter:
  • End date - 05/Dec/2022:18:41:57 +0100
  • End date format - let’s keep the default one
  • Duration - 2779
  • Locale language - EN
• Thread filter - http-nio-8081-exec-5

Now the flame graph is obvious: (HTML)

We can check this for a few more requests and figure out the following:

• In every slow request, we executed matrixMultiplySlow()
• In every fast request, we executed matrixMultiplyFaster()

This technique is great for dealing with the tail of the latency: We can focus our work on the longest operations. That may lead us to some nice fixes.

Real-life example - DNS

The point of the previous example was to show you why aggregated results can be useless for tracing a single request. Now I want to show you a widespread issue I have diagnosed a few times.

Spring Boot has a commonly used addition called Actuator. One of the features of the Actuator is the health check endpoint. Under URI /actuator/health, you can get JSON with information about the health of your application. That endpoint is sometimes used as a load balancer probe. Let’s consider a multi-node cluster of our example application with a load balancer in front of the cluster, which:

• probes the actuator if the application is alive, expecting "status" : "UP" in the response JSON
• timeouts the probe after 1 second

Now, I will do one hack in my local configuration to make this example work. It will be explained at the end of this example.

Let’s find out what our IP is:

$ ifconfig 

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.36.53  netmask 255.255.240.0  broadcast 172.31.47.255

Let’s probe an actuator by this IP, not a localhost (without the hack described later, you cannot get the same results):

./profiler.sh start -e wall -f actuator.jfr first-application-0.0.1-SNAPSHOT.jar
ab -n 1000 http://172.31.36.53:8081/actuator/health # check your IP
./profiler.sh stop -f actuator.jfr first-application-0.0.1-SNAPSHOT.jar

The result of ab:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     0    5 158.1      0    5000
Waiting:        0    5 158.1      0    5000
Total:          0    5 158.1      0    5000

Percentage of the requests served within a specific time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      1
 100%   5000 (longest request)

So almost all the actuator endpoints returned in 0ms, but at least one lasted 5s. We can see one longer request in the access logs:

[08/DEC/2022:08:56:24 +0000] [GET /actuator/health HTTP/1.0] [200] [0 ms] [http-nio-8081-exec-7]
[08/DEC/2022:08:56:24 +0000] [GET /actuator/health HTTP/1.0] [200] [0 ms] [http-nio-8081-exec-10]
[08/DEC/2022:08:56:24 +0000] [GET /actuator/health HTTP/1.0] [200] [0 ms] [http-nio-8081-exec-3]
[08/DEC/2022:08:56:24 +0000] [GET /actuator/health HTTP/1.0] [200] [0 ms] [http-nio-8081-exec-2]
[08/DEC/2022:08:56:29 +0000] [GET /actuator/health HTTP/1.0] [200] [4999 ms] [http-nio-8081-exec-8]
[08/DEC/2022:08:56:29 +0000] [GET /actuator/health HTTP/1.0] [200] [0 ms] [http-nio-8081-exec-5]

That 5s response would make our load balancer remove that node (for some time) from a cluster. Let’s use the same technique to find out what was the reason for that latency: (HTML)

I highlighted the usage of the RemoteIpFilter class. Time for some explanations: When your requests are hitting your application, you can check the IP of the requester with the basic HttpServletRequest API. But if you have a load balancer before your application, well, you get an IP of the load balancer, not the original requester. Load balancers usually add HTTP headers to the request to avoid such confusion. The original IP is sent in the X-Forwarded-For header. The RemoteIpFilter is a tool that makes our lives easier and makes the HttpServletRequest API returns proper IP and so on.

Let’s get back to the flame graph. We can see that this filter creates an instance of XForwardedRequest that executes RequestFacade.getLocalName(), that in the end executes Inet6AddressImpl.getHostByAddr(). The last method is trying to identify the hostname by the IP address. How can it be done? Well, we just need a request to DNS, nothing more. In that case, the DNS protocol uses UDP, not TCP. UDP is a protocol that, by design, can lose packets. In Linux, the resolv.conf is responsible for configuring DNS and the related tools deal with all the retransmissions and other problems. Here is an excerpt of the manual:

timeout:n
       Sets  the amount of time the resolver will wait for a re‐
       sponse from a remote  name  server  before  retrying  the
       query  via  a different name server.  This may not be the
       total time taken by any resolver API call and there is no
       guarantee  that a single resolver API call maps to a sin‐
       gle  timeout.   Measured  in  seconds,  the  default   is
       RES_TIMEOUT (currently 5, see <resolv.h>).  The value for
       this option is silently capped to 30.

Long story short - the default timeout is 5s. If your DNS request is lost, the tools related to resolv.conf will probe the next nameserver after 5s. That’s what is happening in our example and what I observed in quite a few Java applications. DNS is commonly used for DDoS attacks. Therefore you can easily have a firewall in your infrastructure that can drop some DNS packets by design.

The funny thing about RemoteIpFilter is that the result of that DNS probing is stored in the field localName which is not used later. So we are just making DNS requests for nothing. To avoid that problem, write a filter that won’t fire DNS requests. RemoteIpFilter is open-source, so you can easily use it. There is also RemoteIpValve that can be enabled by just an entry in the Spring Boot properties. It used to have the same issue. I didn’t check if the issue is still present in Spring Boot 3; it might be fixed accidentally with this bug fix which introduced the changeLocalName property. If you want to be sure, you need to check it yourself.

This is not the only DNS request that can be done by the Actuator health check. That endpoint also probes your databases, queues, and many other things. The same may happen if that probing is done by DNS name.

About the hack. If you want to simulate not responsive DNS, you can add 72.66.115.13 (blackhole.webpagetest.org) as your nameserver. That one is designed to drop all the packets. On different Linux distributions, it is done differently. I just use an AWS instance with Amazon Linux distribution, and there I could just edit the /etc/resolv.conf file, but in distros like Ubuntu, that file is generated by other services; see ubuntu.com for more information.

Continuous profiling

Let’s now focus on a different problem: We had some performance degradation/outage in our system one hour ago. What can we do? The problem is gone, so attaching a profiler now won’t help us much. We can start profiling and wait for the problem to occur again, but maybe we can inspire ourselves with a concept used in the aviation business.

In case of an airplane disaster, what is the plane owner doing? Are they adding logs or attaching instruments to the airplane and waiting for the next crash? No, the aviation business has a flight recorder on every plane.

This box records all available data it can during the flight. After any disaster, the data are ready to be analyzed. Can we apply a similar approach to Java profiling? Yes, we can. We can have a profiler attached 24/7 dumping the data every fixed interval of time. If anything detrimental happens to our application, we have the data that we can analyze.

From my personal experience, continuous profiling is the best technique to diagnose degradations and outages efficiently. It is also handy to understand why the performance differs between two versions of the same application. You only need to get profiles of the previous version from your archives and compare them to the current one.

Here are the ways of enabling async-profiler in continuous mode:

Command line

Here is the simplest way to run async-profiler in continuous mode (dump a profile every 60 seconds in wall mode):

while true
do
	CURRENT_DATE=`date +%F_%T`
	./profiler.sh -e wall -f out-$CURRENT_DATE.jfr -d 60 <pid or application name> 
done

It looks dirt simple because it is, but I used that loop many times. Async-profiler includes this with the --loop option since version 2.6 in its profiler.sh script:

./profiler.sh -e wall --loop 1m -f profile-%t.jfr <pid or application name>

Java

I already introduced the Java API of AsyncProfiler here. To do it continuously, you can create a thread that is executing:

AsyncProfiler asyncProfiler = AsyncProfiler.getInstance();

DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd_HH:mm:ss");

while (true) {
    String date = formatter.format(LocalDateTime.now());
    asyncProfiler.execute(
        String.format("start,jfr,event=wall,file=out-%s.jfr", date)
    );
    Thread.sleep(60 * 1000);
    asyncProfiler.execute(
        String.format("stop,file=out-%s.jfr", date)
    );
}

Spring boot

If you have a Spring/SpringBoot application, you can use a starter written by Michał Rowicki and me:

<dependency>
    <groupId>com.github.krzysztofslusarski</groupId>
    <artifactId>continuous-async-profiler-spring-starter</artifactId>
    <version>2.1</version>
</dependency>

Read the README to get more details.

Contextual profiling

Continuous profiling, together with the possibility to extract a profile of a single request, is very powerful. Unfortunately, there are applications where that is not enough. Some examples:

• Any work that is delegated to a different thread will be missed in that profile
• If one request is computed by multiple threads/JVMs, we need to combine multiple profiles
• Applications in a distributed architecture, usually with microservices:
  There are usually remote calls to other services, even if every single request is processed by a single thread.

We can extract a profile for each microservice to understand the request processing behavior in such a distributed architecture. This is doable but consumes a lot of time.

All the problems mentioned above can be covered by contextual profiling. The concept is pretty simple: Whenever any thread is executing any work, that work is done in some context, usually in the context of a single request. Instead of just doing that work, we do the following:

long span = Span.start();
actualWork();
Span.end(span, contextId);

The profiler will store span details in the JFR file. Span is a very simple structure. It contains details about thread that was executing the span, start time and duration. That’s all we need to correlate the span with all samples gathered in the actualWork().

Spring Boot microservices

Let’s try to join Spring Boot microservices with contextual profiling. In Spring Boot 3.0 we have included Micrometer Tracing. One of its functionalities is generating a context ID (called traceId) for every request. That traceId is passed during the execution to other Spring Boot microservices. We just need to pass that traceId to the async-profiler and we are done.

Ok, let’s integrate it with the async-profiler. This time I will use the Java API:

public abstract class AsyncProfilerUtils {
    private static volatile AsyncProfiler asyncProfiler;
    // ...

    public static AsyncProfiler load() {
        // Lazy load with double-checked locking
        return asyncProfiler;
    }

    public static void start(String filename) throws IOException {
        load().execute("start,jfr,event=wall,file=" + filename);
    }

    public static void stop(String filename) throws IOException {
        load().execute("stop,jfr,event=wall,file=" + filename);
    }
}

I load the profiler from using AsyncProfiler.getInstance() and use wall-clock mode; I believe it is the most suitable mode for most enterprise applications.

To integrate the profiler with the Micrometer Tracing, we need to implement ObservationHandler:

public class AsyncProfilerObservationHandler implements ObservationHandler<Observation.Context> {
    private static final ThreadLocal<TraceContext> LOCAL_TRACE_CONTEXT = new ThreadLocal<>();
    private static final ThreadLocal<Long> LOCAL_SPAN_START_TIME = new ThreadLocal<>();

    @Override
    public boolean supportsContext(Observation.Context context) {
        return true;
    }

    @Override
    public void onStart(Observation.Context context) {
        TracingContext tracingContext = context.get(TracingContext.class);
        TraceContext traceContext = tracingContext.getSpan().context();
        TraceContext currentTraceContext = LOCAL_TRACE_CONTEXT.get();
        
        if (currentTraceContext == null || !currentTraceContext.traceId().equals(traceContext.traceId())) {
            LOCAL_TRACE_CONTEXT.set(traceContext);
            LOCAL_SPAN_START_TIME.set(Span.start());
        }
    }
    
    @Override
    public void onError(Observation.Context context) {
    }
    
    @Override
    public void onEvent(Observation.Event event, Observation.Context context) {
    }
    
    @Override
    public void onStop(Observation.Context context) {
        TracingContext tracingContext = context.get(TracingContext.class);
        TraceContext traceContext = tracingContext.getSpan().context();
        TraceContext currentTraceContext = LOCAL_TRACE_CONTEXT.get();
        Long spanStartTime = LOCAL_SPAN_START_TIME.get();
        if (currentTraceContext != null && spanStartTime != null && currentTraceContext.spanId().equals(traceContext.spanId())) {
            LOCAL_TRACE_CONTEXT.remove();
            LOCAL_SPAN_START_TIME.remove();
            Span.end(spanStartTime, traceContext.traceId());
        }
    }
}

Big warning: Don’t treat that class as production ready. It’s suitable for that example but will not work with any asynchronous/reactive calls. Mind that onStart/onStop can be called multiple times with the same traceId and different spanId.

Now we need to register that implementation:

@Bean
@Profile("context")
ObservedAspect observedAspect(ObservationRegistry observationRegistry) {
    observationRegistry.observationConfig().observationHandler(new AsyncProfilerObservationHandler());
    return new ObservedAspect(observationRegistry);
}

That will also register us using the @Observed aspect.

And that’s it. Let’s try it out. We need to rerun the Spring Boot applications with active profiling:

java -Xms1G -Xmx1G \
-Dspring.profiles.active=context \
-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints \
-jar first-application/target/first-application-0.0.1-SNAPSHOT.jar 

java -Xms1G -Xmx1G \
-Dspring.profiles.active=context \
-jar second-application/target/second-application-0.0.1-SNAPSHOT.jar

java -Xms1G -Xmx1G \
-Dspring.profiles.active=context \
-jar third-application/target/third-application-0.0.1-SNAPSHOT.jar

# little warmup
ab -n 24 -c 1 http://localhost:8081/examples/context/observe

# profiling time - this time, we start profiler from Java
curl -v http://localhost:8081/examples/context/start
curl -v http://localhost:8082/examples/context/start
curl -v http://localhost:8083/examples/context/start

ab -n 24 -c 1 http://localhost:8081/examples/context/observe

# stopping the profiler
curl -v http://localhost:8081/examples/context/stop
curl -v http://localhost:8082/examples/context/stop
curl -v http://localhost:8083/examples/context/stop

Let’s look at the timings during profiling. I’ve cut the output to 12 rows:

[26/lip/2026:19:13:58 +0200] [GET /examples/context/observe HTTP/1.0] [200] [1593 ms] [http-nio-8081-exec-2]
[26/lip/2026:19:14:02 +0200] [GET /examples/context/observe HTTP/1.0] [200] [3164 ms] [http-nio-8081-exec-3]
[26/lip/2026:19:14:05 +0200] [GET /examples/context/observe HTTP/1.0] [200] [3094 ms] [http-nio-8081-exec-4]
[26/lip/2026:19:14:06 +0200] [GET /examples/context/observe HTTP/1.0] [200] [1590 ms] [http-nio-8081-exec-5]
[26/lip/2026:19:14:08 +0200] [GET /examples/context/observe HTTP/1.0] [200] [1590 ms] [http-nio-8081-exec-6]
[26/lip/2026:19:14:13 +0200] [GET /examples/context/observe HTTP/1.0] [200] [4821 ms] [http-nio-8081-exec-7]
[26/lip/2026:19:14:14 +0200] [GET /examples/context/observe HTTP/1.0] [200] [1589 ms] [http-nio-8081-exec-8]
[26/lip/2026:19:14:16 +0200] [GET /examples/context/observe HTTP/1.0] [200] [1594 ms] [http-nio-8081-exec-9]
[26/lip/2026:19:14:19 +0200] [GET /examples/context/observe HTTP/1.0] [200] [3086 ms] [http-nio-8081-exec-10]
[26/lip/2026:19:14:22 +0200] [GET /examples/context/observe HTTP/1.0] [200] [3160 ms] [http-nio-8081-exec-1]
[26/lip/2026:19:14:24 +0200] [GET /examples/context/observe HTTP/1.0] [200] [1585 ms] [http-nio-8081-exec-2]
[26/lip/2026:19:14:27 +0200] [GET /examples/context/observe HTTP/1.0] [200] [3090 ms] [http-nio-8081-exec-3]

We can see that we have three groups of timings:

• ~1500ms
• ~3000ms
• ~4800ms

After we executed the script above, we had three JFR files in the /tmp directory. When we load all three files together to my viewer and check the Span stats section, we can see:

So we have similar timings from our JFR files. Looking good. Let’s filter all the samples by context ID. It’s called a Span (equals) in my viewer, let’s use a value 6a6640603d40ced403d3184933910cf8 which took 4821ms according to records in the JFR. Let’s also add an additional filename level. The filename is correlated to the application name. Here comes the flame graph: (HTML)

All three applications are on the same flame graph. This is beautiful. Just a reminder: it’s not a whole application. It’s a single request presented here. I highlighted the slowPath() method executed in the second and third app, which causes higher latency. You can play with the HTML flame graph to see what is happening there or jump into the code. I want to focus on what insights the context ID functionality gives us. Because there is more. We’ve already added an additional filename level. We can also add timestamps as another level. Let’s do that. I will present you only the bottom of the graph since that’s what is important here: (HTML)

The image may look blurred, but you can check the HTML version for clarity. At the bottom, you can see five brown rectangles. Those are timestamps truncated to seconds (in UTC and local TZ, that’s why there are 2 rows). So from left to right, we can see what was happening to our request second by second. Let’s highlight when the second application was running during that request:

And the third:

The first application is always running since it’s the entry point to our distributed architecture. The important code in the second application is the following:

@RequiredArgsConstructor
class ContextService {
    // ...
    void doSomething() {
        if (counter.incrementAndGet() % 3 == 0) {
            slowPath();
            return;
        }

        fastPath();
    }

    private void fastPath() {
        // ...
    }

    private void slowPath() {
        // ...
    }
}

And the third application:

class ContextService {
    // ...
    void doSomething() {
        if (counter.incrementAndGet() % 4 == 0) {
            blackhole = slowPath();
            return;
        }
        blackhole = fastPath();
    }

    private int slowPath() {
        // ...
    }

    private int fastPath() {
        // ...
    }
}

So the second application is slower every third request, and the third application is slower every fourth request. That means that for every twelfth request, we are slower in both of them.

I firmly believe that contextual profiling is the future in that area. Everything you see here should be taken with a grain of salt. My Spring integration and my JFR viewer are far away from something professional. I want to inspire you to search for new possibilities like that. If you find any, share it with the rest of the Java performance community.

Distributed systems

First, let’s differentiate distributed architecture from distributed systems. For this post, let’s assume the following:

• a distributed architecture is a set of applications that works together - like microservices
• a distributed system is one application that is deployed on more than one JVM to service some request it distributes the work to more than one instance

One example of a distributed system may be Hazelcast. I’ve applied the span functionality to trace the tail of the latency in SQL queries.

Sample benchmark details:

• Hazelcast cluster size: 4
• Servers with Intel Xeon CPU E5-2687W
• Heap size: 10 GB
• JDK17
• SQL query that is benchmarked: select count(*) from iMap
• iMap size – 1 million serialized Java objects
• Benchmark duration: 8 minutes

The latency distribution for that benchmark is:

The 50th percentile is 1470 ms, whereas the 99.9th is 3718 ms. Let’s now analyze the JFR file with my tool. I’ve created a table with the longest queries in the files:

Let’s analyze a single query using the span functionality. The full flame graph:

Let’s focus on the bottom rows:

Let’s start with timestamps and highlight them one by one:

Summing that up:

• 41.91% of samples were gathered between 14:47:19 and 14:47:20
• 35.79% of samples were gathered between 14:47:20 and 14:47:21
• 6.68% of samples were gathered between 14:47:21 and 14:47:22
• 12.37% of samples were gathered between 14:47:22 and 14:47:23
• 3.34% of samples were gathered between 14:47:23 and 14:47:24

Let’s highlight the filenames (which are named with the IP of the server) one by one:

A little summary:

• 25.42% of samples are from node 10.212.1.101
• 23,75% of samples are from node 10.212.1.102
• 21.07% of samples are from node 10.212.1.103
• 29.77% of samples are from node 10.212.1.104

Since brown bars are sorted alphabetically by timestamps, we can conclude that the 10.212.1.104 server is doing any work in the last seconds of processing.

I checked five more long latency requests and the results were the same, confirming that the 10.212.1.104 server is the problem. I spent a lot of time trying to figure out what was wrong with that machine. My biggest suspect was a difference in meltdown/spectre patches in the kernel. In the end, we reinstalled Linux on those machines, which solved the problem with the 10.212.1.104 server.

Stability

Attaching a profiler to a JVM can potentially cause it to crash. It’s essential to be aware of this risk, as bugs can occur not only in profilers but also in the JVM itself. The OpenJDK developers are working on improving the stability of the API used by profilers to minimize this risk. You can learn more about their work at:

• Johannes Bechberger
• Jaroslav Bachorik

In my opinion, async-profiler is a very mature product already. I know a few companies that are running async-profiler in continuous mode 24/7. Over two years, I only heard about one production crash caused by a profiler. It is working with 40 production JVMs, at least in wall-clock mode there. I have used async-profiler on multiple systems without crashes this year. While there is always some risk when attaching a profiler to a JVM, I believe the risk is minimal and can be safely ignored.

However, if you experience a profiler-related crash, I encourage you to file a GitHub issue to help improve the OpenJDK. During the crash, the hs_err.<pid> file is generated. It may be beneficial for finding the root cause of a problem.

The main problem, according to Johannes Bechberger, stated in a recent JEP proposal, is that async-profiler uses the internal AsyncGetCallTrace API for stack walking. This API was introduced in November 2002 for Sun Studio but was removed in January 2003 and demoted to an internal API (JVMTI). It is neither exported in any header nor standardized. To this date, there is only one tiny test in the whole OpenJDK: The API might be broken with any version, Johannes Bechberger caught such an issue with PR 7559 before the release. Be aware of this risk and test every JDK with your profiling setup before using it in production.

There is an ongoing effort by Johannes Bechberger, with the help of Jaroslav Bachorik and others, to improve this situation by proposing the new AsyncGetStackTrace API, that will hopefully be integrated into the OpenJDK. This API will be official, well-tested with stability and stress tests in the official OpenJDK test suite, and therefore more stable than AsyncGetCallTrace. It will also give the users of tools like async-profiler more information, like C/C++ frames between Java frames and inlining information for all Java frames. If you want to learn more, consider reading the JEP or visit the demo repository to see it in action.

Furthermore, many bugs have been found by both OpenJDK developers by using the JDK Profiling Tester to find and fix many stability issues. There are currently no known real-world stability issues.

Overhead

In the application where the profiler is running in continuous mode on production the overhead (in terms of response time) is typically between 0% and 2%. That number is a comparison of response times before and after introducing continuous profiling there. A bit of context:

• Spring and Spring Boot applications
• Mostly services that handle HTTP requests
• Not really CPU intensive - I would say that on average 60% of the request time was spent off-CPU (waiting for DB/other service)
• JDK 11 and 17 - HotSpot from various vendors
• Wall-clock event, dump of JFR every minute from Java API
• Environment provided by VMware, both VMs and Tanzu clusters
• Async-profiler 1.8.x, later 2.8.x

Johannes Bechberger shared his benchmark results with me. He used the DaCapo Benchmark Suite:

• ThreadRipper 3995WX with 128GB RAM
• Async-profiler 2.8.3
• dacapo benchmarks avrora fop h2 jython lusearch pmd -t 8 -n 3
• CPU event

Johannes’s results shows ~6% overhead on default sampling interval without jfrsync flag, and ~7.5% with jfrsync. The chart for his results:

The logarithmic-scaled X-axis is the number of samples per second, and the Y-axis is the additional overhead.

Remember: You should always measure the overhead in your application by yourself and configure the profiling interval and captured events according to your specific needs..

Random thoughts

1. You need to remember that EVERY profiler lies in some way. The async-profiler is vulnerable to JDK-8281677. There is nothing that the profiler can do; JVM is lying to the profiler, so that lie is passed to the end user. You can change the mechanism used by a profiler, but you will be lied to, maybe differently.
2. You can run an async-profiler to collect more than one event. It is allowed to gather lock and alloc together with one of the modes that gathers execution samples, like cpu, method, …
3. You can run an async-profiler with the jfrsync option that will gather more information exposed by the JVM, but be aware to use the alloc option for information on allocations. This way, you can also capture GC information and more.

If you want to know more on this topic, consider the curated collection of blogs and other resources you find here and the YouTube playlist with in-depth talks on profiling. Consider contacting Johannes Bechberger, who curates both, if you have any suggestions.

Async-profiler 2.9 was released. The brand-new feature is live mode, which can help you detect heap memory leaks.

Let’s start with the definition:

Heap memory leak happens when in your application you have Java objects that are no longer needed for program execution, but cannot be freed by garbage collector.

Why is it so hard?

If you have a heap memory leak in your application then you have two groups of objects:

• Live set - this is a group of objects that are still needed by your application
• Memory leak - this is a group of objects that are no longer needed

Garbage collectors cannot free the second group if there is at least one strong reference from live set to memory leak. The biggest problems with diagnosis memory leaks are:

• the fact that the object was created is not an issue - it was created because it was needed for something
• the fact that the mentioned reference was created is not an issue - it had some purpose too
• we need to understand why that reference was not removed by our application

The last one is not trivial. All the observability/profiling tools give us a great possibility to understand why some event has happened, but with memory leaks we need to understand why something hasn’t happened.

Tools

Since there is no tool that would tell us why something is not happening we need to use our brain to find the root cause of the problem. We can use two additional tools that can help us:

• heap dump - that shows us current state of a heap - we can find out what kind of objects are, but shouldn’t be there
• profiler - that shows us where those objects were created

There were already profilers on a market that had a great feature of detecting where not freed objects were created. Unfortunately they used instrumentation and the overhead was so big that they couldn’t be used in production. To make use of them we needed memory leaks recreated on test or local environments.

Async-profiler uses different way of finding allocated objects (quote from it’s README):

async-profiler does not use intrusive techniques like bytecode instrumentation or expensive DTrace probes which have significant performance impact. It also does not affect Escape Analysis or prevent from JIT optimizations like allocation elimination. Only actual heap allocations are measured.

The profiler features TLAB-driven sampling. It relies on HotSpot-specific callbacks to receive two kinds of notifications:

• when an object is allocated in a newly created TLAB (aqua frames in a Flame Graph);
• when an object is allocated on a slow path outside TLAB (brown frames).

This means not each allocation is counted, but only allocations every N kB, where N is the average size of TLAB. This makes heap sampling very cheap and suitable for production. On the other hand, the collected data may be incomplete, though in practice it will often reflect the top allocation sources.

I used that feature on many production systems before. The live option adds a filter to present only objects that haven’t been removed by GC.

Testing application

Let’s create a simple Spring Boot application with memory leak, that is hard to find by just a heap dump:

@RestController
@RequiredArgsConstructor
@RequestMapping("/hard")
class HardOne {
    private final LeakOne leakOne;

    @GetMapping
    void doIt() {
        String newOne = RandomStringUtils.randomAlphabetic(10000);
        leakOne.doLeak(newOne);
        LeakTwo.doLeak(newOne);
    }
}

@Component
class LeakOne {
    private final Set<String> leak = new HashSet<>();

    void doLeak(String s) {
        leak.add(s);
    }
}

class LeakTwo {
    private static final Set<String> leak = new HashSet<>();

    static void doLeak(String s) {
        leak.add(s);
    }
}

If we look at a heap dump after execution http://localhost:8080/hard 10k times we would see:

Usually I analyze heap dumps with potential memory leak in two easy steps:

• I locate objects that shouldn’t be there
• I calculate path to GC roots to understand why they are still alive

In that dump we can see the leaking Strings, but we cannot detect where they are held, since there is no dominator. There are simply two HashSets that hold a strong reference to those leaked objects, so there is no one guilty class. In such situations the allocation profiler is much more useful.

Let’s test it

First let’s try simple allocation mode to see if the live option really does any change.

./profiler.sh -e alloc --total start LeakApplication
ab -n 10000 http://localhost:8080/hard 
./profiler.sh -f live.html stop LeakApplication

The result:

Now let’s add the live option:

./profiler.sh -e alloc --live --total start LeakApplication
ab -n 10000 http://localhost:8080/hard 
./profiler.sh -f live.html stop LeakApplication

We can see that the profile looks different, but we can also see allocations other than our memory leak. This is done because GC hasn’t freed them yet. Let’s run a GC before capturing the results this time:

./profiler.sh -e alloc --live --total start LeakApplication
ab -n 10000 http://localhost:8080/hard 
jcmd LeakApplication GC.run
./profiler.sh -f live.html stop LeakApplication

Perfect. Ladies and gentlemen, we have the first profiler that can help us detect heap memory leaks in our production environment (at your own risk, of course). Thank you, Andrei Pangin.

This blog post was originally posted on Hazelcast’s website.

Hazelcast in version 5.1 contains two querying APIs:

• Predicate API
• SQL engine

Predicate API is an older Java-based API. Even though it contains sqlPredicate, which allows using SQL-like syntax for the WHERE clause, the syntax is non-standard, for example NULL handling doesn’t support the (in)famous ternary logic. It fetches the results in one batch, which limits the supported result size.

On the other hand, SQL Engine is a more modern engine, it uses standard SQL, a cost-based optimizer, and is available in all programming languages. It also supports JOIN, ORDER BY, GROUP BY or UNION operators, which don’t have an equivalent in the Predicate API. It streams the results to the client, so the result size isn’t limited (though this is also a restriction, because it’s not possible to restart the query if it fails mid-way).

In the next major release, we plan to deprecate the Predicate API. For this we need feature parity, and also match the performance of Predicate API. The performance is the focus of this blog post: we’ll describe our journey in benchmarking and fixing some performance issues we had.

Benchmark details

Here is a list of benchmarks that we did for that comparison:

The fifth benchmark was run with and without index on value field. For benchmark number three we used three different serialization methods:

• IdentifiedDataSerializable
• Portable
• HazelcastJsonValue with json-flat SQL mapping

For the rest of the benchmarks we used IdentifiedDataSerializable. If you are not familiar with our serialization options check this page.

We didn’t benchmark joins and streaming queries since they cannot be executed with Predicate API. These functionalities are available in SQL engine only.

Testing environment

All the benchmarks were run in throughput and latency modes since we cared about both. In this post I’m going to cover only the latency part of those benchmarks.

First results showed that the SQL was slower in most of them. Next step was to run those benchmarks again with the attached profiler. I used Async-profiler, and since I was tracing latency issues, I chose wall-clock mode. You can read about different modes in my previous post.

All of those benchmarks were run on our testing lab, so we knew that the results were valid (no noisy-neighbor issue). The setup for the benchmarks was:

• Hazelcast cluster size: 4
• Machines with Intel Xeon CPU E5-2687W
• Heap size: 10 GB
• JDK17

How does Hazelcast SQL work?

Each of the cluster members contains two major modules: IMDG and JET. IMDG is a module where we store our data. JET is a distributed batch and stream processing engine. Additionally, for parsing the SQL queries we use Apache Calcite.

When a client sends a query to a cluster, that query is just a string with parameters. The query is sent to a random member of the cluster. That member is called a coordinator of that query. The coordinator needs to:

• Create a query plan
• Convert the query plan to JET job
• Distribute the JET job to all the members

After that is done all the members are executing the query, and the results are streamed by the coordinator to the client.

Flame graphs

If you do sampling profiling you need to visualize the results. The results are nothing more than a set of stacktraces. My favorite way of visualization is a flame graph. The easiest way to understand what flame graphs are is to understand how they are created.

First part is to draw a rectangle for each frame of each stacktrace. The stacktraces are drawn bottom-up and sorted alphabetically. For example, such a graph:

corresponds to set of stactraces:

• 3 samples - a() -> h()
• 5 samples - b() -> d() -> e() -> f()
• 2 samples - b() -> d() -> e() -> g()
• 2 samples - b() -> d()
• 2 samples - c()

The next step is joining the rectangles with the same method name to one bar:

The flame graph usually shows you how some resource is utilized by your application. The resource is utilized by the top methods of that graph (visualized with green bar):

So in this example method b() is not utilizing the resource at all, it just invokes methods that do it. Flame graphs are commonly used to present the CPU utilization, but CPU is just one of the resources that we can visualize this way. If you use wall-clock mode then your resource is time. If you use allocation mode then your resource is heap.

Client part

Benchmark details:

• Query: SELECT __key, this FROM IMap
• Values in IMap have fields: String and int[20], serialized with IdentifiedDataSerializable
• IMap size - 100_000 entries
• Latency test - 24 queries per second

Let’s start with profiling the client side. Here is a wall-clock flame graph of the method that fetches the data on the client’s side:

Let’s look at the bottom of the flame graph, method testSelect() iterates over the results and for each row it executes three things:

• Method accept() - that executes the deserializon of the row - each SQL row travels to the client through the network as a byte array, so we need to deserialize it to the object format
• Method ClientIterator.hasNext() - that one returns the next row if it is already fetched, or waits to receive the next portion of the result set. In the flame graph we see the waiting part. At this point in time the execution is done in the cluster - nothing to improve on the client’s side.
• Some third method, let’s zoom in that part of the flame graph

Here we can see that the method getColumnValueForClient() (3rd from the top) executes LinkedList.get() which consumes a lot of time. Let’s go to the source code:

columns is a field of type List<List<?>>, and the problematic method executes columns.get(columnIndex).get(rowIndex). Executing LinkedList.get(index) is known to have bad performance, since it has O(size) complexity. The instance is not created here, it is passed in the constructor, so we need to debug the application to find the origin of that LinkedList. Two breakpoints later I found a place where it was created. One solution is to switch that List to ArrayList like this:

But it is not always that easy. The code where that list was created was done many years ago and is used not only by SQL engine. We need to be careful here. I asked colleagues from the other team if we can do it, they did their benchmarks and agreed to that change:

How did that affect our performance? Well it depends on the number of columns and whether we do deserialization on the client’s side (it is done lazily). Here is the latency distribution for two columns and no deserialization:

Was it a mistake to choose LinkedList in that code? No. It was a coherent decision made by the state of Hazelcast at the moment of creation of that code.

Thing to remember is that performance is a living organism. It evolves with new features. What was efficient yesterday could be a bottleneck tomorrow.

PR: https://github.com/hazelcast/hazelcast/pull/20398

Cluster side - sum()

Benchmark details:

• Query: SELECT sum(\”value\”) FROM IMap
• Values in IMap have fields: one Long, and one int[20], serialized with IdentifiedDataSerializable
• IMap size - 1_000_000 entries
• Latency test - 6 queries per second

In this benchmark we need to deserialize all the instances in IMap, get a single field from that deserialized object and aggregate it. The majority of work is deserialization. It was strange to me that the Predicate API had different performance than SQL.

Here is a flame graph for Predicate API:

And here is one for SQL:

Now let’s do something a bit silly. Open both graphs in separate browser tabs (right mouse click on the graph -> open graphics in a new tab) and then just switch between those two tabs very quickly and try to find a difference. Did you spot it? It is at the top. The SQL graph has a lot of yellow/red code there, the Predicate API does not. Let’s see what it is about:

The frames that we can see in SQL only are Class.forName(). We had a class definition cache in Hazelcast already, and it was strange that it wasn’t used for SQL. We can highlight all invocations of that method to see how much time it consumes:

So that loading of classes consumed ~25% of deserialization of Java objects. Let’s look at the code of our class loading:

We have four layers of caches here, each tryLoadClass() checks if the definition is in the cache, if not it loads it. If none of the classloaders loaded the class we called Class.forName().

The difference between Predicate API and SQL was that in SQL the contextClassLoader was null, since you cannot use user classes with SQL query. We decided to add another level of cache:

How did that improve the performance?

Thing to remember is that “If it looks stupid but works, it isn’t stupid” – that also applies to the performance analysis. Comparing two flame graphs by quickly switching the browser tabs looks silly, but hey, it works like a charm if you want to compare resource utilization that should look the same.

PR: https://github.com/hazelcast/hazelcast/pull/20459

Cluster side - sum() with JSON

Benchmark details:

• Query: SELECT sum(json_value(this, '$.value' returning integer)) FROM cache
• Values in IMap have two fields: Long and int[20], serialized as JSON
• IMap size - 1_000_000 entries
• Latency test - 5 queries per second

Let’s go straight to the flame graph of evaluation a value from JSON:

The big left part of that graph is acquiring a lock, it utilizes:

~70% of the time of evaluation of the value. That part of code uses Guava cache that uses ReentrantLock internally. The cache contains a mapping from string JSON path to our object that represents that path. The usage of that cache is usually single insert and multiple reads. For such a usage pattern the ReentrantLock is not the best choice, ConcurrentHashMap is better for example. Simple switch from Guava cache to CHM did this improvement:

In the end we decided to do two implementations of a cache, one for a single JSON path based on a field, the second for the rest of the cases.

Thing to remember is that the performance bottleneck may be in 3rd party libraries/frameworks. It doesn’t mean that their code is bad, usually we simply do not know the trade-offs there, and that may hurt us.

PR: https://github.com/hazelcast/hazelcast/pull/20655

Cluster side - scan for a single value by index

Benchmark details:

• Query: SELECT __key, this FROM iMap where col=...; // Index on col
• Values in IMap have two fields: String and int[20], serialized with IdentifiedDataSerializable
• IMap size - 10_000_000 entries
• Latency test - 7500 queries per second

This is a benchmark where we test a distributed deployment overhead of a job on a busy cluster. The part after the deployment is easy, we just need to take an index and fetch a single row from it. Unfortunately in such queries we will never be better than Predicate API, since the execution plan for SQL is much bigger than for Predicate API, and serialization and deployment takes more time. What we can do is to speed up that part in SQL to be as fast as possible.

When I ran the first instance of that benchmark the mean latency was around 4,5ms on a stressed cluster. To fight that kind of latency we need to focus on all the resources that are needed for that code to run. Let’s look at the heap allocation:

Over 90% of recorded allocation (during creation of the plan) was done in the constructor of SenderTasklet. That class is responsible for sending computation results to other nodes in the cluster. It created a 32k byte array as a buffer of data to send. The value 32k was ok for a task manipulating multiple entries, but for a task that processes only one row it was a waste of heap memory.

We went with the solution to have a buffer with two initial sizes:

• Small initial size: 1k
• If that buffer is too small it expands to 32k immediately

That approach didn’t hit larger tasks, they need to waste 1k array at the beginning, but allocation of such an array is much cheaper than 32k, since 32k is often allocated outside the TLAB.

That change lowered the allocation rate of that benchmark from 3,8GB/s to 2,3GB/s, but that was not the only change in that part of our engine. My colleague pointed out that in some cases we created a SenderTasklet that knew it would never send any data through the network. We can avoid the creation of unnecessary SenderTasklet. That change lowered the heap allocation rate to 1,5GB/s.

There were plenty of PRs for lowering down that deployment overhead. Current status (mean) is:

We lowered the mean latency from 4.5ms to 1.8ms on a stressed cluster, and we still have ideas how to make it even better.

Thing to remember is that allocation on heap is fast, but no allocation is faster.

PRs: https://github.com/hazelcast/hazelcast/pull/20882 and https://github.com/hazelcast/hazelcast/pull/20940

Summary

Our SQL engine in release 5.2 is currently faster with better throughput than Predicate API in most of the benchmarks. The only kind of query where Predicate API is faster is a query that normally takes just milliseconds to execute. As I’ve mentioned in the last example, we still have ideas on how to make the SQL engine faster. We are going to realize those in future releases.

In this post I focused on four different issues that can teach us something. Let me gather those four things to remember:

• Performance is a living organism. It evolves with new features. What was efficient yesterday could be a bottleneck tomorrow.
• “If it looks stupid but works, it isn’t stupid” - every flame graph analysis technique is good as long as it works for you
• The performance bottleneck may be hidden in 3rd party libraries/frameworks
• Allocation on heap is fast, but no allocation is faster

One additional “lesson learned” is that the Async-profiler is a very useful tool for finding such high-level bottlenecks. You need to remember that there are issues in which such a profiler won’t help. Some performance bottlenecks can be understood after analyzing assembly code or with Top-down Microarchitecture Analysis.

This blog post was originally posted on Hazelcast’s website.

Let’s start with the basics. It is easy these days if we want to speed up a whole application! You run a JVM profiler, gather the results, and analyze the results to identify where the time and /or resources are wasted. Equipped with this understanding, you can make the application more efficient.

But what if we want to solve the problem of knowing why a single request is slow? It is tough to find a single request in profiler results. Additionally, as we will explore in this article, this problem becomes increasingly complex as we move from monolithic applications toward distributed architectures and distributed systems.

Profiler mode

Often, the mistake made by software engineers during their initial profiling sessions is choosing the wrong profiler mode. Every profiler has multiple modes, and you need to select the one to help you solve your problem. The most common issues are:

• “my application/request is slow” – essentially, you want to measure where time is wasted; the correct mode to use is “wall-clock”
• “my application/request consumes too much CPU” – here, one would like to identify the section of code that is consuming excess CPU, and the “CPU” mode should be used
• “my application/request has a high allocation rate” – you should use “allocation” mode for this analysis

A common mistake is using “CPU” mode to analyze why an application is slow. If your application is CPU intensive, then yes, it may work. But if your application waits on IOs (like WS invocations or DB queries), locks, or waits, you will waste your time.

Since we want to solve the problem of slow requests, we’ll use the “wall-clock” profiler mode.

Profiler

Nowadays, we have plenty of profilers on the market. We need to select a profiler that supports “wall-clock” mode and runs with a small overhead so that it can even be used on a production system. The only profiler that fits these requirements that I am aware of is the Async-profiler, so we will focus on this one. It also supports dumping results in JFR format where each sample contains:

• Stacktrace
• Thread name
• Timestamp
• Thread state – if that thread consumes CPU or not

Monolithic application

Let’s start with a monolithic application’s most straightforward architecture to trace. A single thread processes the majority of the requests in a monolithic application. If we know the thread that has executed a slow request, then JFR output contains all the information we need. We need to filter that file with two filters:

• Thread name – it should equal the one that has processed the request
• Timestamp – it should be between the start and the end of the request execution

There are multiple ways of gathering this in our application. We can use our application log or application server access log.

Distributed architecture (like microservices)

The situation is more complicated in distributed architecture since one request is processed by multiple JVMs (we focus only on JVM applications here; this article will not solve problems in other execution engines). The standard approach to understanding what is happening in such an architecture is to use correlation id (in short, CID). The first application that receives a request from an outside world generates a CID passed in every request between applications in a distributed architecture. Adding that CID to our logs, we can extract the specific thread on a particular JVM that executed the request. Then like before, we need to filter the JFR files from all the applications. It is exhausting but doable. We need to have the profiler running on all the JVMs.

Distributed systems

First of all, let’s understand the difference between distributed systems and distributed architecture. Those phrases are often used interchangeably, but the distributed systems I am referring to in this article are entirely different from distributed architecture. For this article, let’s assume that a distributed system is a single system in the architecture that distributes the work to multiple instances and threads to give the response ASAP.

You can imagine how difficult it is to trace a single request with such a definition. It is executed not only on multiple threads but also on multiple JVMs. How the hell can we trace that? Is that execution slow on each node? Or maybe one node is slower? Or maybe one thread on one node has too much work? How can we distinguish that?

The first approach is creating a CID and logging the thread on which JVM executes it. It would work, but it would be even more painful than the distributed architecture problem. To troubleshoot this problem regularly, well, we need something more innovative. What if we could add the CID to the JFR file? That would be great, as we wouldn’t have to aggregate multiple data sources. We would need the CID and JFR files from all the JVMs, and the only filter we need to apply to the JFR file is equality to CID.

Bad news

Well, Async-profiler has no such feature, so we have two options:

1. “roll over and die” and accept the current state of the reality and be miserable
2. or we can create our reality and software that fulfills our needs

As a software engineer, you have an opportunity no other profession has. You can choose option two! The easiest way to modify Async-profiler was simply by contributing to the open-sourced project. I prepared a PR. When writing this article, this PR had not been merged. It may never be merged, but the PR state is good enough to solve my current problem of finding the cause of long requests using the benchmarks on my test environment.

Good news

Even if my PR is not merged, the contextual profiling feature is on the Async-profiler roadmap; it will support this functionality sooner or later.

JFR filtering and transformation

We now have a tool that gathers the needed information in JFR format; great! Next, we need a tool that filters that output and presents it in a human-readable format. My favorite format to understand profiler output is a flame graph. Previously, I had written a tool that filters and converts JFR to flame graph, which I have now extended. It was straightforward as it’s just parsing, filtering, and transforming problems. That change is already in a master branch of my tool.

We have all the tools we need; let’s try it now!

Benchmark details:

• Hazelcast cluster size: 4
• Machines with Intel Xeon CPU E5-2687W
• Heap size: 10 GB
• JDK17
• SQL query that is benchmarked: select count(*) form iMap
• iMap size – 1 million java serialized objects
• Benchmark duration: 8 minutes

The latency distribution for that benchmark:

The 50th percentile is 1470 ms, whereas the 99.9th is 3718 ms. Let’s now analyze the JFR outputs with my tool. I’ve created a table with the longest CIDs in the files:

Let’s analyze a single CID. I selected additional levels in my tool, namely the filename and timestamp. The full flame graph:

Let’s focus on the bottom rows:

The brown squares are timestamps with human-readable dates. The cyan ones are filenames from which upper stacktraces come.

Let’s start with timestamps and highlight them one by one:

Summing that up:

• 41.91% of samples were gathered between 14:47:19 and 14:47:20
• 35.79% of samples were gathered between 14:47:20 and 14:47:21
• 6.68% of samples were gathered between 14:47:21 and 14:47:22
• 12.37% of samples were gathered between 14:47:22 and 14:47:23
• 3.34% of samples were gathered between 14:47:23 and 14:47:24

Let’s highlight the filenames (which are named with the IP of the server) one by one:

A little summary:

• 25.42% of samples are from node 10.212.1.101
• 23,75% of samples are from node 10.212.1.102
• 21.07% of samples are from node 10.212.1.103
• 29.77% of samples are from node 10.212.1.104

Since brown bars are sorted alphabetically by timestamps, we can conclude that the 10.212.1.104 server doing any work in the last seconds of processing.

So, the server with IP 10.212.1.104 is problematic. Maybe it is slower. Perhaps the work between nodes was distributed without proper balance, and this server had more work to do. Maybe there were some long GC pauses on that server. That topic is out of scope for this article, so I won’t dig into it here.

I checked five more long latency requests and the results were the same, confirming that the 10.212.1.104 server is the problem. I want to point out in this article that we now have a tool that gives us a great starting point for further investigation.

Some caveats

1. The current state of my PR to Async-profiler is not merged The API can change, the whole solution may be completely different in the end. My implementation uses JNI calls when passing CID to the profiler; this may be problematic in some applications. The additional overhead for passing the CID was ~0,06% with the example above. Remember that in Hazelcast Jet, we need to give new CID very frequently since our green threads execute multiple fast tasklets. Based on this, you should consider that PR as POC for now.
2. The Async-profiler sampling interval is one sample every 10ms. This rate allows you to track latencies with hundreds of ms or more. With lower latencies, you need pieces to be gathered more often. The sampling interval can be changed, but it may degrade your performance if you sample too frequently.

This article is a continuation of this one, where we have studied following code of JVM sources:

File io_util.c:

#define BUF_SIZE 8192

void
writeBytes(JNIEnv *env, jobject this, jbyteArray bytes,
           jint off, jint len, jboolean append, jfieldID fid)
{
    ...
    if (len == 0) {
        return;
    } else if (len > BUF_SIZE) {
        buf = malloc(len);
        if (buf == NULL) {
            JNU_ThrowOutOfMemoryError(env, NULL);
            return;
        }
    } else {
        buf = stackBuf;
    }
    (*env)->GetByteArrayRegion(env, bytes, off, len, (jbyte *)buf);
    ...
    if (buf != stackBuf) {
        free(buf);
    }
}

I also mentioned that there are four essential states of a single JVM thread:

There are 4 essential states:

_thread_new : Just started, but not executed init. code yet (most likely still in OS init code)

_thread_in_native : In native code. This is a safepoint region, since all oops will be in jobject handles

_thread_in_vm : Executing in the vm

_thread_in_Java : Executing either interpreted or compiled Java code (or could be in a stub)

However, there are some understatements.

• Is a whole writeBytes is done in _thread_in_vm?
• Which call from the JVM code can cause a page fault?

We didn’t see the answer for the second question in the previous articles since JVM was compiled without debug symbols.

Our own JVM from sources

Let’s try to compile our own version of JVM from sources with debug symbols. To do it we need to execute (if you’re doing it for the first time you may need to install some dependencies, check the instruction first):

git clone --depth 1 https://github.com/openjdk/jdk11u-dev.git
cd jdk11u-dev
bash configure --with-debug-level=slowdebug
make images

After a success compilation you should see:

Creating support/demos/image/jfc/J2Ddemo/J2Ddemo.jar
Creating support/classlist.jar
Creating images/jmods/jdk.jlink.jmod
Creating images/jmods/java.base.jmod
Creating jdk image
Stopping sjavac server
Finished building target 'images' in configuration 'linux-x86_64-normal-server-slowdebug'

Let’s run our reproduction Test from previous article with our custom JVM.

./build/linux-x86_64-normal-server-slowdebug/jdk/bin/java -Xmx700M -XX:+AlwaysPreTouch -XX:+SafepointTimeout -XX:SafepointTimeoutDelay=100 -XX:+UnlockDiagnosticVMOptions -XX:+AbortVMOnSafepointTimeout Temp

This time our hs_err has much more details:

Current thread (0x00007f12b401e800):  JavaThread "main" [_thread_in_vm, id=3325, stack(0x00007f12bd06b000,0x00007f12bd16c000)]

Stack: [0x00007f12bd06b000,0x00007f12bd16c000],  sp=0x00007f12bd168148,  free space=1012k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x15baa1]  __memmove_ssse3_back+0x1b11
V  [libjvm.so+0x30bd40]  Copy::conjoint_jbytes(void const*, void*, unsigned long)+0x2b
V  [libjvm.so+0x30b6a8]  void AccessInternal::arraycopy_conjoint<signed char>(signed char*, signed char*, unsigned long)+0x2b
V  [libjvm.so+0xc03380]  EnableIf<(((!HasDecorator<64ul, 4ul>::value)&&(!(HasDecorator<64ul, 33554432ul>::value&&RawAccessBarrierArrayCopy::IsHeapWordSized<signed char>::value)))&&(!HasDecorator<64ul, 67108864ul>::value))&&(!HasDecorator<64ul, 134217728ul>::value), void>::type RawAccessBarrierArrayCopy::arraycopy<64ul, signed char>(arrayOopDesc*, unsigned long, signed char*, arrayOopDesc*, unsigned long, signed char*, unsigned long)+0x6d
V  [libjvm.so+0xc03218]  bool RawAccessBarrier<64ul>::arraycopy<signed char>(arrayOopDesc*, unsigned long, signed char*, arrayOopDesc*, unsigned long, signed char*, unsigned long)+0x48
V  [libjvm.so+0xc03119]  EnableIf<HasDecorator<2642000ul, 4096ul>::value&&AccessInternal::PreRuntimeDispatch::CanHardwireRaw<2642000ul>::value, bool>::type AccessInternal::PreRuntimeDispatch::arraycopy<2642000ul, signed char>(arrayOopDesc*, unsigned long, signed char*, arrayOopDesc*, unsigned long, signed char*, unsigned long)+0x48
V  [libjvm.so+0xc02fd2]  EnableIf<!HasDecorator<2637904ul, 4096ul>::value, bool>::type AccessInternal::PreRuntimeDispatch::arraycopy<2637904ul, signed char>(arrayOopDesc*, unsigned long, signed char*, arrayOopDesc*, unsigned long, signed char*, unsigned long)+0x59
V  [libjvm.so+0xc02e96]  bool AccessInternal::arraycopy_reduce_types<2637904ul, signed char>(arrayOopDesc*, unsigned long, signed char*, arrayOopDesc*, unsigned long, signed char*, unsigned long)+0x48
V  [libjvm.so+0xc02cf6]  bool AccessInternal::arraycopy<2621440ul, signed char>(arrayOopDesc*, unsigned long, signed char const*, arrayOopDesc*, unsigned long, signed char*, unsigned long)+0x50
V  [libjvm.so+0xc02a56]  void Access<2621440ul>::arraycopy<signed char>(arrayOopDesc*, unsigned long, signed char const*, arrayOopDesc*, unsigned long, signed char*, unsigned long)+0x4d
V  [libjvm.so+0xcc85aa]  void ArrayAccess<0ul>::arraycopy_to_native<signed char>(arrayOopDesc*, unsigned long, signed char*, unsigned long)+0x47
V  [libjvm.so+0xcbe07e]  jni_GetByteArrayRegion+0x2f4
C  [libjava.so+0x1e858]  writeBytes+0x15d
C  [libjava.so+0x10d86]  Java_java_io_FileOutputStream_writeBytes+0x5b
j  java.io.FileOutputStream.writeBytes([BIIZ)V+0 java.base
j  java.io.FileOutputStream.write([BII)V+16 java.base
j  java.io.BufferedOutputStream.write([BII)V+20 java.base
j  java.io.FilterOutputStream.write([B)V+5 java.base
j  Temp.main([Ljava/lang/String;)V+58
v  ~StubRoutines::call_stub
V  [libjvm.so+0xbebc5d]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x647
V  [libjvm.so+0x10a88dc]  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*), JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x32
V  [libjvm.so+0xbeb614]  JavaCalls::call(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0xa8
V  [libjvm.so+0xc96122]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)+0x1f0
V  [libjvm.so+0xcad204]  jni_CallStaticVoidMethod+0x36a
C  [libjli.so+0x5062]  JavaMain+0xcf7

Now we can see that the writeBytes executes jni_GetByteArrayRegion which in the end calls __memmove_ssse3_back. So it’s not the malloc, that make sense. The malloc returns a pointer returned by the OS. The page fault happens when you first time touch the page.

Thread state changes

Now let’s try to understand how the thread state changes over time. Is a whole writeBytes executed in _thread_in_vm? Or maybe we go from _thread_in_Java to _thread_in_native and then we hit _thread_in_vm?

We will use such a program to track it:

import java.io.BufferedOutputStream;
import java.io.FileOutputStream;

class Temp {
    public static void main(String[] args) throws Exception {
        byte[] arr = new byte[300 * 1024 * 1024];
        for (int i = 0; i < arr.length; i++) {
            arr[i] = (byte) i;
        }

        Thread.sleep(3000);
        try (BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream("/home/<username>/tmp.tmp"))) {
            bos.write(arr);
        }
    }
}

Let’s customize our JVM a bit and add custom debug printf.

File io_util.c:

#define BUF_SIZE 8192

void
writeBytes(JNIEnv *env, jobject this, jbyteArray bytes,
           jint off, jint len, jboolean append, jfieldID fid)
{
    printf( "WriteBytes start\n" );
    ...
    if (len == 0) {
        return;
    } else if (len > BUF_SIZE) {
        printf( "Malloc start\n" );
        buf = malloc(len);
        printf( "Malloc ended\n" );
        if (buf == NULL) {
            JNU_ThrowOutOfMemoryError(env, NULL);
            return;
        }
    } else {
        buf = stackBuf;
    }
    printf( "GetByteArrayRegion start\n" );
    (*env)->GetByteArrayRegion(env, bytes, off, len, (jbyte *)buf);
    printf( "GetByteArrayRegion ended\n" );
    ...
    if (buf != stackBuf) {
        free(buf);
    }
    printf( "WriteBytes end\n" );
}

And let’s modify thread.hpp:

  JavaThreadState thread_state() const           { return _thread_state; }
  void set_thread_state(JavaThreadState s)       {
    printf( "New thread state (%ld) %d -> %d: \n", os::current_thread_id(),  _thread_state, s );
    assert(current_or_null() == NULL || current_or_null() == this,
           "state change should only be called by the current thread");
    _thread_state = s;
  }

The output:

WriteBytes start
New thread state (346625) 4 -> 5: 
New thread state (346625) 5 -> 6: 
New thread state (346625) 6 -> 7: 
New thread state (346625) 7 -> 4: 
Malloc start
Malloc ended
GetByteArrayRegion start
New thread state (346625) 4 -> 5: 
New thread state (346625) 5 -> 6: 
New thread state (346625) 6 -> 7: 
New thread state (346625) 7 -> 4: 
GetByteArrayRegion ended
New thread state (346625) 4 -> 5: 
New thread state (346625) 5 -> 6: 
New thread state (346625) 6 -> 7: 
New thread state (346625) 7 -> 4: 
New thread state (346625) 4 -> 5: 
New thread state (346625) 5 -> 6: 
New thread state (346625) 6 -> 7: 
New thread state (346625) 7 -> 4: 
New thread state (346625) 4 -> 5: 
New thread state (346625) 5 -> 6: 
New thread state (346625) 6 -> 7: 
New thread state (346625) 7 -> 4: 
New thread state (346625) 4 -> 5: 
New thread state (346625) 5 -> 6: 
New thread state (346625) 6 -> 7: 
New thread state (346625) 7 -> 4: 
WriteBytes end

Now we have a bunch of magic numbers, the dictionary for them is in the globalDefinitions.hpp:

enum JavaThreadState {
  _thread_uninitialized     =  0, // should never happen (missing initialization)
  _thread_new               =  2, // just starting up, i.e., in process of being initialized
  _thread_new_trans         =  3, // corresponding transition state (not used, included for completness)
  _thread_in_native         =  4, // running in native code
  _thread_in_native_trans   =  5, // corresponding transition state
  _thread_in_vm             =  6, // running in VM
  _thread_in_vm_trans       =  7, // corresponding transition state
  _thread_in_Java           =  8, // running in Java or in stub code
  _thread_in_Java_trans     =  9, // corresponding transition state (not used, included for completness)
  _thread_blocked           = 10, // blocked in vm
  _thread_blocked_trans     = 11, // corresponding transition state
  _thread_max_state         = 12  // maximum thread state+1 - used for statistics allocation
};

Our thread state path looks like this (I skipped the _trans states):

_thread_in_native
WriteBytes start 
_thread_in_vm  
_thread_in_native 
Malloc start
Malloc ended
GetByteArrayRegion start 
_thread_in_vm  
_thread_in_native 
GetByteArrayRegion ended 
_thread_in_vm  
_thread_in_native  
_thread_in_vm  
_thread_in_native  
_thread_in_vm  
_thread_in_native  
_thread_in_vm  
_thread_in_native 
WriteBytes end

So the answer is that the writeBytes executes code in _thread_in_native and switches to _thread_in_vm when needed.

There is an excellent presentation on YouTube called Profilers are lying hobbitses by Nitsan Wakart. I strongly recommend you to watch it. Here is another example of lie from profilers.

Profiled application

Simple class:

import java.util.ArrayList;
import java.util.List;

/** Andrei Pangin is an author of that reproduction. Details later. */
public class ArrayListGrow {
    static final int SIZE = 2048;
    static volatile List<Object> tmp;

    static void runTest() {
        List<Object> list = new ArrayList<>(SIZE);
        fill(list);
        tmp = list;
    }

    static void fill(List<Object> list) {
        for (int i = 0; i < SIZE; i++) {
            list.add(i);
        }
    }

    static void spoil() {
        for (int i = 0; i < 1000000; i++) {
            new ArrayList<>(0).add("");
        }
    }

    public static void main(String[] args) {
        spoil();

        while (true) {
            runTest();
        }
    }
}

Let’s run that code using Amazon Corretto 1.8.0_312, and run the Async-profiler in CPU mode.

./profiler.sh -d 10 ArrayListGrow

The beginning of the output is:

Profiling for 10 seconds
Done
--- Execution profile ---
Total samples       : 1020

--- 8199255026 ns (80.37%), 820 samples
  [ 0] java.util.Arrays.copyOf
  [ 1] java.util.ArrayList.grow
  [ 2] java.util.ArrayList.ensureExplicitCapacity
  [ 3] java.util.ArrayList.ensureCapacityInternal
  [ 4] java.util.ArrayList.add
  [ 5] ArrayListGrow.fill
  [ 6] ArrayListGrow.runTest
  [ 7] ArrayListGrow.main

That means that our ArrayList executes method grow() while adding an element. That add is invoked from ArrayListGrow.fill. But how is it possible? That list is created with new ArrayList<>(SIZE) and we are adding exactly SIZE elements. The ArrayList shouldn’t grow in that case. Let’s run it in the debugger. Let’s create a breakpoint at runTest() line to skip the spoil() part first:

Now let’s remove that one and create one in ArrayList.grow():

Let’s hit continue, so debugger could stop in the new breakpoint… and it doesn’t want to stop. So the profiler shows us a huge usage of ArrayList.grow(), and the debugger shows us no usage of that method. Who should we trust?

This time the debugger has right, but we cannot blame any profiler for that lie since this is a …

JVM bug

That bug was originally spotted by me, the original discussion over that topic is here. My reconstruction was much more complicated than Andrei’s, that’s why I prefer to show his version. The bug is registered with number JDK-8281677.

This bug hurts all profilers that use AsyncGetCallTrace or PerfMapAgent, so basically every modern profilers (Async-profiler, JProfiler in async mode, Perf + PMA, eBpf profiler + PMA, …).

This bug is not just an ArrayList issue. I encountered multiple parts of my code where profiler lied to me because of that.

Context

There are situations where an application running on the JVM doesn’t respond. It can be so unresponsive that you cannot attach to it with any of the external tools, like JMX client, jcmd, jstack, jmap and so on. The first thought may be to restart the JVM. It will probably help, but during such a situation the application is in a specific state. If you restart the application, that state is lost.

The JVM is just a process

You need to understand that the JVM is just a process from an operating system perspective. The OS offers us tools to diagnose unresponsive processes. The easiest way is to fetch a core dump. You can do it with a single line:

gcore -o <filename> <pid>

It will create a very large file <filename>.<pid>. This operation may take a while.

What can we do with such a dump?

Since JDK 9 there is a tool called jhsdb delivered in every JDK. With that tool you can manipulate the core dump in multiple ways. The two basic modes are:

Heap dump

You can fetch the heap dump from a core dump using:

jhsdb jmap --binaryheap --dumpfile <output hprof file> --core <core dump file> --exe <path to java exe>

Mind that the jhsdb and java, that is pointed as a last argument, must be from exactly the same distribution as the JVM, from which the core dump has been created.

Thread dump

You can fetch the heap dump from a core dump using:

jhsdb jstack --core <core dump file> --exe <path to java exe>

There is more…

There are other modes of the jhsdb, see the help of that tool to see if they are useful to you.

You can also find it useful that the core dump is just a process memory snapshot, so you can do everything you could do with other core dumps. For example: you can attach the gdb to your dump:

gdb <path to java exe> <core dump file>

That will open you the GNU debugger. Sometimes this is useful, when you need to check where your threads have hung. Unfortunately there is an assembly code and native stacks, but it is better than nothing.

Context

Most of the applications I saw were deployed at the virtual machine. Most of them used Spring Boot with Tomcat as a web server and Hikari as a connection pool. In this article I’m going to show you metrics from that technology. If you are using others, you need to find corresponding metrics by yourself. The JMX mbean names are taken from the Java 11 with the G1GC.

Mind that some of the following metrics may have no sense in the cloud.

Foreword

This article is very long because there are a lot of metrics that you should focus on. In my applications I have all of those metrics gathered in one dashboard. It is the first place I look when there is some outage. That dashboard doesn’t tell me the exact cause of the outage, but directs me to the source of the problem.

Hardware

Let’s start from the bottom. We have four crucial hardware components that almost all applications use:

• CPU
• RAM
• Disk
• Network

Mind that there are other components, like disk controllers, bridges and so on, but I’m going to focus on those four.

CPU

CPU utilization chart (7 days)

Possible failures:

• The 100% utilization of all cores
• The 100% utilization of one core

In this situation you need to check if that resource is utilized by your JVM. The first one can be easily diagnosed by JMX, I’m going to cover that later. The second one I usually diagnose at OS level using pidstat -t <time interval>.

RAM

Available memory chart (7 days)

You need to understand the difference between available and free memory. I suggest you read the beginning of the man free at Linux. Long story short, your OS has internal caches and buffers. Those are used to help the performance of your application. Your OS can use them if there is enough memory available. If your application needs the memory used before by cache/buffer, the OS will remove that cache/buffer to make that memory available for you.

Simplifying:

• Available memory - how much memory is available for your application
• Free memory - how much memory is not used by any application and OS

Possible failures:

• 0 bytes of the available memory - this is a situation where out-of-memory killer kicks in
• The available memory drops to the level, where crucial cache/buffer is removed

The first situation is shown in the second chart above. The available memory dropped to 0, OOM killer killed the JVM. In that situation you need to check the OS logs to find out which process ate that memory.

The second situation is easy to diagnose when you look also at your next resource.

Disk

IO operations (reads/writes) chart (1 day)

Possible failures:

• Huge increase of IO operations

The IO operations don’t appear out of nowhere. If the increase of IO reads is correlated with reduction of available memory then you probably have a situation where a crucial cache/buffer is removed (shown at second chart above). In that situation you need to check RAM consumption to check which process eats the memory.

The increase of IO operations can be done by your JVM, you can check it with pidstat -d <time interval> at Linux. If that is your problem I suggest using the async-profiler in wall mode. With that profiler you can easily find the part of your code that is using the IO.

Network

Net transfer (in/out) chart (1 day)

Possible failures:

• Huge increase of the traffic
• No traffic

In the first situation first you need to check if that traffic is done by your JVM. There are multiple tools to check it, I like the nethogs. If that traffic is generated by the JVM then, again, the async-profiler at wall mode is going to show you what part of your application generates it.

If there is no traffic, then:

• You should check your health check endpoint if this is a way you inform other applications about the availability of your system
• You should check your service discovery/load balancer if you are using one, maybe there is some failure

JVM (from JMX)

CPU (again)

CPU utilization chart (7 days)

• Object name - java.lang:type=OperatingSystem
• Attributes - SystemCpuLoad and ProcessCpuLoad

The JVM reports through the JMX two metrics, the CPU utilization from OS level (which is the same as the first metric in this article), and how much of that utilization is done by that JVM. Comparing those two values can give you the information if your application is responsible for the utilization. The last chart above shows the case where the application cannot perform well, because another process consumes a whole CPU.

Heap after GC

I’ve already written why that metric is useful in the previous article.

Heap after GC chart (7 days)

The committed, and the max heap size can be obtained from:

• Object name - java.lang:type=Memory
• Attribute - HeapMemoryUsage - a complex type with max and committed attributes

The heap after GC can be obtained from:

• Object name - java.lang:name=G1 Young Generation,type=GarbageCollector
• Attribute - LastGcInfo - a complex type with memoryUsageAfterGc attribute

I use that chart to find out if there is a heap memory leak. Check the previous article for details. The last chart above shows the memory leak. If you have a memory leak in your application then you have to analyze the heap dump.

Heap before GC

I’ve already written why that metric is useful in the previous article.

Heap before GC chart (7 days)

The committed, and the max heap size can be obtained from:

• Object name - java.lang:type=Memory
• Attribute - HeapMemoryUsage - a complex type with max and committed attributes

The heap before GC can be obtained from:

• Object name - java.lang:name=G1 Young Generation,type=GarbageCollector
• Attribute - LastGcInfo - a complex type with memoryUsageBeforeGc attribute

I use that chart to find out if the heap is wasted, because the GC starts its work too early. Check the previous article for details. The second chart above shows the application where the GC runs inefficiently. If you have such a situation then you need to analyze the GC log.

Full GC count (G1GC)

Full GC count chart (7 days)

• Object name - java.lang:name=G1 Old Generation,type=GarbageCollector
• Attributes - CollectionCount

The best situation is presented at the first chart: 0 full GCs. Possible failures:

• Single Full GC (second chart) - for the most of the applications this is not a problem, but yours might be different
• Few Full GCs in a short period (third chart)
• Periodic Full GC (forth chart) - covered in the previous article
• Multiple Full GCs (fifth chart)

If you have a first situation, and it is a problem for you then you need to analyze GC logs. This is a place with information why the G1GC has to run that phase.

If you have a second situation, it usually means that your application runs a part of code that needs almost all the free heap. What I usually do in that situation is to check the access log, and the application log to find out what was going on in the time of full GCs. The continuous profiling is also useful in that case.

The last situation usually means that your application runs a part of code that needs more heap than you have. It usually ends with OutOfMemoryError. I strongly recommend you to enable -XX:+HeapDumpOnOutOfMemoryError, it will dump you a heap dump when such a situation occurs, which is the best way to find the cause of it.

Code cache size

Code cache size chart (7 days)

• Object name - java.lang:name=CodeHeap 'non-profiled nmethods',type=MemoryPool and java.lang:name=CodeHeap 'profiled nmethods',type=MemoryPool
• Attributes - Usage - a complex type with max and committed attributes

Possible failures:

• Not enough code cache for a new compilation

The code cache contains the JIT compilations. If that part of memory cannot fit a new compilation then the whole JIT disables. After that, your application starts degrading the performance. You need to remember that the code cache can be fragmented. As a rule of thumb I assume that if the application uses >=90% of that part of a memory, then that part needs to be increased.

Off-heap usage

Off-heap usage charts (7 days)

• Object name - java.nio:name=direct,type=BufferPool and java.nio:name=mapped,type=BufferPool
• Attributes - MemoryUsed

Possible failures:

• Too much memory allocated

Most of the time the off-heap is used by frameworks/libraries. Those metrics are useful for control, if there is no memory leak in this area.

To find out what part of your code allocates the memory with these two mechanisms you can use:

• Heap dump for DirectByteBuffer
• Heap dump and pmap from OS level for FileChannel

Loaded class count

Loaded class count chart (7 days)

• Object name - java.lang:type=ClassLoading
• Attributes - LoadedClassCount

Possible failures:

• Memory leak at Metaspace

This problem was common in EJB servers. If you do not use hot swap, context reload, there is a little chance you have this leak. You need to remember that it is completely normal for that chart to be ascending. There are mechanisms (like the evil reflection) which create classes at runtime. If you think there is a memory leak the first think you should look at are classloader logs.

The heap dump is also useful in that situation. It contains all class definitions and can tell you why they are alive.

Hikari active connection count

Hikari active connection count chart (7 days)

• Object name - com.zaxxer.hikari:type=Pool (<pool name>)
• Attributes - ActiveConnections

This metric is useful for scaling and sizing your application. It tells you how many active connections you need to your database. If this value is too big for you, your code may spend too much time with the opened transaction. The way I’m finding such a situation in Spring framework is to take output for async-profiler in wall mode and look for long methods covered with invokeWithinTransaction method.

Hikari threads awaiting connection

Hikari threads awaiting connection chart (7 days)

• Object name - com.zaxxer.hikari:type=Pool (<pool name>)
• Attributes - ThreadsAwaitingConnection

Possible failures:

• A rapid increase from 0
• A constant value over 0

The second situation is simple, your database connection pool has too low size to handle all the requests. Either your pool is too short, or you have a situation I’ve explained in the previous chapter.

The first situation (second chart above) means that you have one of the following issues:

• A load to your application increased
• Your database slowed down
• Your application needed more time in transaction

The last situation can be diagnosed the same way I’ve written above. The easiest way to diagnose the second option is to start from database level. There are dedicated tools to monitor that area like Oracle Enterprise Manager

Tomcat connection count

Tomcat connection count charts (7/5 days)

• Object name - Tomcat:type=ThreadPool,name="<pool name>"
• Attributes - connectionCount

Possible failures:

• Huge increase in a short period

This metric tells you how many connections are established to your server. If it is increasing rapidly then most probably you encounter one of the following situations:

• The number of requests is increasing
• The time for your application to handle the request is increasing

You need to remember that if your application slows down the number of incoming requests don’t decrease. The only way for an application server to handle the same amount of requests with slower application is to create more worker threads and accept more connections. If it is your case (the second chart above) then you need to find why your application is slower, the best way from my experiences is async-profiler in wall mode.

Created thread count

Created thread count chart (7 days)

• Object name - java.lang:type=Threading
• Attributes - TotalStartedThreadCount

If the number of created threads is increasing it means there is a lack of a thread pool. I’ve written an article explaining how to find the part of code that lacks that pool.

Current thread count

Current thread count charts (7 days)

• Object name - java.lang:type=Threading
• Attributes - ThreadCount

Possible failures:

• Thread leak
• Huge increase in a short period

There are some situations where the application has thread leak, they are very rare. That situation can be diagnosed the same way as creating too many threads. The huge increase of threads in a short period means pretty much the same as increasing the tomcat connection count. The application needs more threads to do its work. We can find the reason with async-profiler in wall mode.

Afterword

As I said in the foreword, a dashboard with all those metrics helps me handle the outages, here are some examples:

• If there is a huge CPU utilization, and it is not done by my JVM then it’s noisy neighbor situation, we need to deal with the process that eats our resource
• If there is a huge CPU utilization, and it is done by my JVM, the application runs slow, and the code cache is almost full then I look at log if the JIT compiler has been disable
• If there is a huge CPU utilization, and it is done by my JVM, the application runs slow, and the full GC is happening all the time, I need a heap dump and GC logs to diagnose what is going on
• If there is a huge CPU utilization, and it is done by my JVM, and the GC runs very frequently, starting its work too soon then I need GC logs to understand why this is happening
• If there is a huge CPU utilization, and it is done by my JVM (and it is not any of previous situations), I need CPU profiling
• If there are multiple threads awaiting database connection then I need wall profiling and access to DB performance tool
• It there was a 0 available RAM for a while, and now there is no java process then I check in OS logs if there was OOM killer activity logged

… and so on.

Those charts do not solve the outage but show you where you should focus on your diagnosis. Remember, I just covered the technology I use. You may have other metrics worth gathering (MQ, Kafka, custom thread pools and so on).

After fifteen years of handling different kinds of outages I can tell you one thing: I never start the diagnosis from the application log.