[Java][JVM][Tuning][Profiling][G1][JIT] Why is Arrays.fill 265 times slower on G1GC?

Big fat warning

This article shows some JVM tuning using JVM flags. You should never use any JVM flags without knowing what consequences they may produce. Most of the flags used here are diagnostic ones, used to understand what is going on. Only one of them is worth considering on production, and I write about it at the very end.

The benchmark

It started with a benchmark that I expected to be boring. Fill two arrays with a reference, once on G1GC, once on ParallelGC:

package pl.ks.jmh;

import org.openjdk.jmh.annotations.*;
import java.util.Arrays;
import java.util.concurrent.TimeUnit;

@State(Scope.Benchmark)
public class MyBenchmark {
    Object[] table = new Object[1024 * 1024];
    Object[] table2 = new Object[1024 * 1024];
    Object mark = new Object();

    @Benchmark
    @Fork(value = 1, warmups = 1, jvmArgsAppend = "-XX:+UseParallelGC")
    @OutputTimeUnit(TimeUnit.MICROSECONDS)
    @Warmup(iterations = 1)
    @Measurement(iterations = 2)
    @BenchmarkMode(Mode.AverageTime)
    public void parallelGC() {
        Arrays.fill(table, mark);
        Arrays.fill(table2, mark);
    }

    @Benchmark
    @Fork(value = 1, warmups = 1, jvmArgsAppend = "-XX:+UseG1GC")
    @OutputTimeUnit(TimeUnit.MICROSECONDS)
    @Warmup(iterations = 1)
    @Measurement(iterations = 2)
    @BenchmarkMode(Mode.AverageTime)
    public void g1GC() {
        Arrays.fill(table, mark);
        Arrays.fill(table2, mark);
    }
}

There is no allocation in these methods, so there are no GC cycles at all during the measurement. Whatever the difference is, it cannot be “G1 collects garbage slower”. And yet:

Benchmark               Mode  Cnt       Score   Units
MyBenchmark.g1GC        avgt    2  139019,174   us/op
MyBenchmark.parallelGC  avgt    2     525,537   us/op

139 milliseconds versus 0.5 millisecond. The same Java code, the same JDK, the same machine. G1 is 265 times slower.

This article is the story of chasing that number down to a single machine instruction, and then finding out that the instruction was only half of the answer.

Environment

Where does the time go?

The first question is always the same one: which code is hot? Since there is no GC and no allocation, I ran the benchmark with an assembly-level profiler:

java -jar target/benchmarks.jar -prof xctraceasm

On both GCs the answer is the same, 99.8% of the samples are in the JIT-compiled code of Arrays.fill. There is no VM overhead, no GC thread stealing the CPU, no safepoints. The mutator thread itself is simply running slow code.

So the difference has to be in what the JIT generated, and the profiler gives us exactly that.

A minimal assembly primer

This is an ARM64 machine, so the listings below are ARM64. If you have never read assembly, here is everything you need for this article. It is genuinely a short list.

Registers are the CPU’s working variables. Think of them as around 30 pre-declared local variables that every method shares:

Notation Meaning
x0 … x30 a register used as a full 64-bit value
w0 … w30 the same register, but only its lower 32 bits
wzr / xzr the “zero register”, reading it always gives 0
x28 HotSpot reserves this one for the current Thread*
x27 HotSpot reserves this one for the base address of the Java heap

Instructions, only the ones that show up here:

Instruction In Java terms
mov x1, #5 x1 = 5
add x1, x2, #16 x1 = x2 + 16
lsr x1, x2, #9 x1 = x2 >>> 9
eor x1, x2, x3 x1 = x2 ^ x3 (XOR)
ldr x1, [x2] x1 = memory[x2] - load 8 bytes
ldrb w1, [x2] w1 = memory[x2] - load one byte (b = byte)
str w1, [x2] memory[x2] = w1 - store 4 bytes
strb wzr, [x2] memory[x2] = 0 - store one zero byte
cbz x1, LABEL if (x1 == 0) goto LABEL
cbnz x1, LABEL if (x1 != 0) goto LABEL
cmp w1, #2 + b.ne LABEL if (w1 != 2) goto LABEL
csel x1, x2, xzr, hs x1 = cond ? x2 : 0 - a ternary without a branch
dmb ish full memory fence - stall until every pending write is visible to all cores

Two addressing shorthands: [x5, x14] means memory[x5 + x14], and add x14, x1, w10, sxtw #2 means x14 = x1 + ((long) w10) * 4, which is array indexing in a single instruction.

That is the whole vocabulary. Let’s read some code.

One caveat: all of this is ARM64

Every listing in this article is aarch64, because that is what I profiled on. On x86-64 the same barrier looks completely different - different registers, different mnemonics, and the memory fence is not a dmb ish at all. HotSpot builds a StoreLoad barrier on x86 out of a locked no-op on the stack:

lock addl $0x0,-0x40(%rsp)

Decoding the x86 version instruction by instruction is out of scope here.

What is not architecture specific is the logic. g1BarrierSetAssembler_x86.cpp and g1BarrierSetAssembler_aarch64.cpp are mirror images of one another - the same three early exits, the same young-card check, and the same StoreLoad membar at the top of the slow path. The problem described in this article is therefore not an ARM problem. Only the listings are ARM. I have not measured how large the penalty is on x86, and I would not assume it is the same number - the cost of a fence is a very microarchitecture-specific thing.

If you do want to reproduce these listings exactly as printed, you need an aarch64 machine. Those stopped being exotic a while ago:

What ParallelGC generated

Here is the hot loop from the ParallelGC run. Each Java-level table[i] = mark became this:

lsr   x14, x11, #9              ; x14 = address(table[i]) >>> 9
strb  wzr, [x5, x14, lsl #0]    ; cardTable[x14] = 0
str   w16, [x11]                ; table[i] = mark

One instruction does the actual work, and two extra instructions do something with a thing called a card table. That is the write barrier, and it is worth a short detour to explain what it is for.

What is a write barrier?

Every generational GC has the same problem. When it collects only the young generation, it must find all references pointing into the young generation, including references held by old objects. Scanning the entire old generation to find them would defeat the whole point of a young-only collection.

The solution is to make the application do the bookkeeping. The heap is divided into cards of 512 bytes, and the JVM keeps a byte array with one byte per card - the card table. Every time your code writes a reference into an object, the JIT emits a couple of extra instructions that mark that object’s card as dirty. At GC time the collector only scans the dirty cards instead of the whole old generation.

That handful of extra instructions after every reference store is the write barrier. You never wrote it, you cannot see it in your code, and it runs on every single reference assignment your application performs.

Now the ParallelGC listing reads as plain Java:

table[i] = mark;                                   // str  w16, [x11]
cardTable[address(table[i]) >>> 9] = 0;            // 0 means "dirty"

The >>> 9 is dividing by 512, the card size. Register x5 holds the base address of the card table, and it is loaded once before the loop - you will not find it recomputed anywhere inside. There are no branches, no conditions, nothing to predict. ParallelGC does not care whether the object is young or old, whether the value is null, or where it points. It marks the card and moves on.

There is one more detail in that listing worth remembering for later: the loop counter is incremented by 8, not by 1. The JIT unrolled the loop eight times, so eight elements are filled per iteration.

What G1GC generated

Now the same Java statement on G1. Same JDK, same machine, same Arrays.fill:

  0x...bff0:   add   x14, x1, w10, sxtw #2   ; x14 = arrayBase + i*4
  0x...bff4:   add   x14, x14, #0x10         ; x14 = address of table[i]
  0x...bff8:   add   w10, w10, #1            ; i++
  0x...bffc:   ldrb  w16, [x28, #0x48]       ; load a byte from the current Thread
  0x...c000:   cbnz  w16, #0x...c0a0         ; if it is not 0 -> jump away
  0x...c004:   subs  x17, x2, x27
  0x...c008:   csel  x17, x17, xzr, hs
  0x...c00c:   lsr   x17, x17, #3            ; compress the reference into 4 bytes
  0x...c010:   str   w17, [x14]              ; table[i] = mark        <-- the actual store
  0x...c014:   eor   x16, x14, x2
  0x...c018:   lsr   x16, x16, #0x17
  0x...c01c:   cbz   x16, #0x...c03c         ; exit 1
  0x...c020:   cbz   x2,  #0x...c03c         ; exit 2
  0x...c024:   lsr   x16, x14, #9
  0x...c028:   mov   x15, #0xdf310000
  0x...c02c:   add   x16, x16, x15           ; x16 = address of the card
  0x...c030:   ldrb  w15, [x16]              ; load the card byte
  0x...c034:   cmp   w15, #2
  0x...c038:   b.ne  #0x...c108              ; exit 3 -> the slow path
  0x...c03c:   cmp   w10, w13
  0x...c040:   b.lt  #0x...bff0              ; loop back

and somewhere further down, out of the main instruction stream:

  0x...c108:   dmb   ish                     ; FULL MEMORY FENCE
  0x...c10c:   ldrb  w15, [x16]              ; re-load the card byte
  0x...c110:   cbz   w15, #0x...c03c         ; already dirty -> back to the loop
  0x...c114:   strb  wzr, [x16]              ; mark the card dirty
  0x...c118:   ldr   x15, [x28, #0x50]       ; \
  0x...c11c:   cbz   x15, #0x...c134         ;  |
  0x...c120:   sub   x15, x15, #8            ;  | push the card into this thread's
  0x...c124:   str   x15, [x28, #0x50]       ;  | dirty card queue
  0x...c128:   ldr   x8,  [x28, #0x58]       ;  |
  0x...c12c:   str   x16, [x8, x15]          ; /
  0x...c130:   b     #0x...c03c

Twenty instructions instead of three, four conditional branches, and a full memory fence hiding at the end. Translated into Java, one iteration of Arrays.fill on G1 is:

// ---- pre-barrier: is concurrent marking running right now? ----
if (currentThread.satbMarkQueueActive != 0) {
    slowPathForConcurrentMarking();                 // rarely taken
}

// ---- the actual store ----
table[i] = mark;

// ---- post-barrier ----
if (((address(table[i]) ^ mark) >>> 23) == 0) { goto done; }   // exit 1: same region?
if (mark == null)                             { goto done; }   // exit 2: storing null?

cardAddr = cardTableBase + (address(table[i]) >>> 9);
if (cardTable[cardAddr] == 2)                 { goto done; }   // exit 3: young card?

// ---- slow path ----
fullMemoryFence();                                         // dmb ish
if (cardTable[cardAddr] == 0) { goto done; }               // already dirty
cardTable[cardAddr] = 0;                                   // mark it dirty
enqueueIntoDirtyCardQueue(cardAddr);
done:

G1 is doing far more work than ParallelGC, but it is also trying much harder to do nothing. It has three separate escape hatches, and in normal code one of them almost always fires, so the barrier costs a handful of cheap instructions and a well-predicted branch. That is the design.

The interesting question is: why does none of them fire in my benchmark?

How to map assembly back to the JDK source code

Before answering that, let me show the technique, because it is reusable and it is the part I find people skip.

Those three magic numbers - 23, 9, 2 - are not arbitrary. Every single instruction above comes from a specific line of HotSpot C++, and you can walk from one to the other mechanically. There are four layers.

Layer 1: machine code to the .ad file

.ad files are HotSpot’s instruction selection rules: patterns saying “when you see this shape in the compiler’s graph, emit this machine code”. A debug build of the JDK can print which rule fired:

java -XX:+UnlockDiagnosticVMOptions \
     -XX:CompileCommand=PrintOptoAssembly,java.util.Arrays::fill \
     ...

090     B7: # Loop( B7-B7 inner strip mined) Freq: 120,936
090     add R14, R1, R10, I2L #2
094 +   add R14, R14, #16
098 +   addw R10, R10, #1
09c     encode_heap_oop R17, R2
        strw  R17, [R14]        # compressed ptr
0e0 +   cmpw  R10, R13
0e4     blt B7                  // counted loop end

Those strings are copied verbatim from a format block in src/hotspot/cpu/aarch64/gc/g1/g1_aarch64.ad, belonging to a rule called g1EncodePAndStoreN. That name is the entry point into the source.

Also worth reading is the left column - those are byte offsets. The listing jumps from 09c to 0e0, which is 68 bytes of machine code that the rule does not print. That gap is the G1 barrier. It is inserted after this listing is produced, which is exactly why it is invisible at this level and very much present in the real code.

Layer 2: the .ad file to the barrier assembler

instruct g1EncodePAndStoreN(...)
  match(Set mem (StoreN mem (EncodeP src)));
  ins_encode %{
    write_barrier_pre(masm, ...);            // everything before the store
    __ encode_heap_oop($tmp1, $src);
    __ strw($tmp1$$Register, $mem$$Register);
    write_barrier_post(masm, ...);           // everything after the store
  %}

write_barrier_pre and write_barrier_post land in g1BarrierSetAssembler_aarch64.cpp. And here is the thing that makes this whole exercise easy: that file is a literal, one-line-per-instruction transcription. There is no optimizer between that C++ and the machine code. Every __ something() emits exactly one instruction, in source order:

static void generate_post_barrier_fast_path(MacroAssembler* masm, ...) {
  // Does store cross heap regions?
  __ eor(tmp1, store_addr, new_val);                     // eor  x16, x14, x2
  __ lsr(tmp1, tmp1, G1HeapRegion::LogOfHRGrainBytes);   // lsr  x16, x16, #0x17
  __ cbz(tmp1, done);                                    // cbz  x16, done
  // Crosses regions, storing null?
  if (new_val_may_be_null) {
    __ cbz(new_val, done);                               // cbz  x2, done
  }
  // Storing region crossing non-null, is card young?
  __ lsr(tmp1, store_addr, CardTable::card_shift());     // lsr  x16, x14, #9
  __ load_byte_map_base(tmp2);                           // mov  x15, #0xdf310000
  __ add(tmp1, tmp1, tmp2);                              // add  x16, x16, x15
  __ ldrb(tmp2, Address(tmp1));                          // ldrb w15, [x16]
  __ cmpw(tmp2, (int)G1CardTable::g1_young_card_val());  // cmp  w15, #2
}

static void generate_post_barrier_slow_path(MacroAssembler* masm, ...) {
  __ membar(Assembler::StoreLoad);  // StoreLoad membar    // dmb  ish
  __ ldrb(tmp2, Address(tmp1));     // tmp2 := card        // ldrb w15, [x16]
  __ cbzw(tmp2, done);                                     // cbz  w15, done
  // Storing a region crossing, non-null oop, card is clean.
  // Dirty card and log.
  STATIC_ASSERT(CardTable::dirty_card_val() == 0);
  __ strb(zr, Address(tmp1));       // *(card address) := dirty_card_val
  generate_queue_test_and_insertion(masm, ...);
  __ b(done);
}

Read it side by side with the disassembly above. It is the same code twice.

For comparison, here is the entire ParallelGC barrier, from cardTableBarrierSetAssembler_aarch64.cpp:

void CardTableBarrierSetAssembler::store_check(MacroAssembler* masm, Register obj, Address dst) {
  __ lsr(obj, obj, CardTable::card_shift());
  assert(CardTable::dirty_card_val() == 0, "must be");
  __ load_byte_map_base(rscratch1);

  if (UseCondCardMark) {
    Label L_already_dirty;
    __ ldrb(rscratch2,  Address(obj, rscratch1));
    __ cbz(rscratch2, L_already_dirty);
    __ strb(zr, Address(obj, rscratch1));
    __ bind(L_already_dirty);
  } else {
    __ strb(zr, Address(obj, rscratch1));
  }
}

UseCondCardMark is false by default, so we take the else branch: shift, and store a zero byte. That is all of it.

Layer 3: the magic numbers

Every constant you do not recognise is a JVM constant. Two of them come from -XX:+PrintFlagsFinal:

java -XX:+UseG1GC -XX:+PrintFlagsFinal -version | grep G1HeapRegionSize

size_t G1HeapRegionSize    = 8388608    {product} {ergonomic}

8388608 is 223, and lsr x16, x16, #0x17 is a shift by 23. Shifting an address right by 23 turns it into a region number, so eor followed by lsr followed by cbz is asking “are these two addresses in the same 8 MB region?”. If they are, G1 does not care about this store at all - a reference that does not leave its region needs no bookkeeping.

The third constant comes from an enum in g1CardTable.hpp:

enum G1CardValues {
    g1_young_gen = CT_MR_BS_last_reserved << 1,   // = 2
    ...
};

and from cardTable.hpp:

enum CardValues {
    clean_card             = (CardValue)-1,
    dirty_card             =  0,
    CT_MR_BS_last_reserved =  1
};

So cmp w15, #2 means “is this card in a young region?”. G1 never tracks references pointing into the young generation (it collects the whole young gen every time anyway), so a young card is an instant exit.

Layer 4: the decisions, not the code

Some differences are not in the barrier at all - they are C2 deciding differently because the barrier exists. I come back to those at the end.

Why none of the three exits fire

Now we can answer the question. Look at what the benchmark allocates:

Object[] table = new Object[1024 * 1024];

With compressed oops that is 16 bytes of header plus 4 bytes per element:

16 + 4 * 1 048 576 = 4 194 320 bytes

G1’s humongous threshold is half a region (from g1CollectedHeap.hpp):

static size_t humongous_threshold_for(size_t region_size) {
    return (region_size / 2);
}

With 8 MB regions that is 4 194 304 bytes. The array is over the limit by 16 bytes. Two words.

If you do not know what a humongous object is, I wrote about them here. The part that matters today: humongous objects skip the young generation entirely and are allocated directly into old generation regions.

Let’s verify it rather than trust the arithmetic:

public class Hum {
    static Object[] t;
    public static void main(String[] a) {
        t = new Object[1024 * 1024];
        System.gc();
    }
}

java -XX:+UseG1GC -Xlog:gc+heap=info Hum.java

[0,179s][info][gc,heap] GC(0) Humongous regions: 1->1

One humongous region. Change the array to new Object[1024 * 1024 - 8] and it reports Humongous regions: 0->0.

So, walking the three exits with this in hand:

  1. The array lives in an old region, mark lives in eden. Different regions, so the XOR test fails.
  2. mark is not null, so the null test fails.
  3. The array’s cards belong to an old region, so they are never 2. The young test fails.

Every single one of the two million stores per operation falls through to the slow path, whose first instruction is dmb ish - a full memory fence.

Proving it

An explanation that only explains is not worth much. Here is a plain main - no JMH, and thanks to the single-file source launcher you can run it straight from the .java file, in two seconds.

One thing to explain before the code, because the name of the method is famously misleading: Integer.getInteger("n", 1024*1024) does not parse the string "n". It looks up the system property called n and returns 1024*1024 when that property is not set. So the two knobs of this program are ordinary -D flags:

public class Fill {
    static Object[] table  = new Object[Integer.getInteger("n", 1024*1024)];
    static Object[] table2 = new Object[Integer.getInteger("n", 1024*1024)];
    static Object mark = new Object();

    static void work() {
        java.util.Arrays.fill(table, mark);
        java.util.Arrays.fill(table2, mark);
    }

    public static void main(String[] a) {
        int iters = Integer.getInteger("iters", 40);
        for (int i = 0; i < 3; i++) {
            work();
        }
        long t0 = System.nanoTime();
        for (int i = 0; i < iters; i++) {
            work();
        }
        System.out.printf("len=%d  %.3f ms/op%n", table.length,
                          (System.nanoTime() - t0) / 1e6 / iters);
    }
}

The four rows of the table below are these four runs:

java -XX:+UseG1GC       -Dn=1048576 -Diters=10  Fill.java   # 1024*1024, humongous
java -XX:+UseG1GC       -Dn=1048568 -Diters=200 Fill.java   # eight elements less
java -XX:+UseG1GC -XX:G1HeapRegionSize=16m -Dn=1048576 -Diters=200 Fill.java
java -XX:+UseParallelGC -Dn=1048576 -Diters=200 Fill.java
Configuration Result
G1, new Object[1024*1024] (humongous) 124.651 ms/op
G1, new Object[1024*1024 - 8] (8 elements smaller) 1.124 ms/op
G1, humongous array + -XX:G1HeapRegionSize=16m 0.982 ms/op
ParallelGC, new Object[1024*1024] 0.501 ms/op

Removing eight array elements is a 110x speedup. Raising the region size, which raises the humongous threshold to 8 MB, does the same thing.

Two more runs to confirm which of the three exits is the one that matters. Filling the very same humongous array with different values:

Value stored Result
null 1.022 ms/op
a young Object 109.480 ms/op
another humongous, old object 145.766 ms/op

Storing null takes exit 2 and is fast. Storing a reference to something in the old generation is just as slow as storing a reference to something young - which rules out the obvious alternative explanation. This has nothing to do with the young generation, remembered sets, or G1’s refinement threads. It is purely “the card is not marked young, so take the slow path”.

It is not about humongous objects

At this point it would be easy to file this under “humongous allocation strikes again” and move on. That would be a mistake. Nothing in the barrier mentions humongous objects. It asks one question: is this card young? Being humongous is just a very fast way to get an object into old generation.

An ordinary object gets there too - it just takes a few GC cycles. Same array, deliberately 8 elements under the humongous threshold, measured before and after being promoted:

public class Promote {
    static final int N = 1024*1024 - 8;    // NOT humongous
    static Object[] table;
    static Object   mark;

    static void work() { java.util.Arrays.fill(table, mark); }

    static void bench(String label, int iters) {
        for (int i = 0; i < 5; i++) {                           // warm up, see note below
            work();
        }
        long t0 = System.nanoTime();
        for (int i = 0; i < iters; i++) {
            work();
        }
        System.out.printf("%-40s %9.3f ms/op%n", label,
                          (System.nanoTime() - t0) / 1e6 / iters);
    }

    public static void main(String[] a) {
        table = new Object[N];
        mark  = new Object();
        bench("array young (fresh eden allocation)", 200);

        for (int i = 0; i < 4; i++) {              // promote it to old generation
            System.gc();
        }
        mark = new Object();                       // fresh young mark
        bench("same array, after 4x System.gc()", 30);
    }
}

A word about that warm-up loop, because five calls looks small for a JIT benchmark.

It is not the number of calls that gets Arrays.fill compiled here, it is the number of loop iterations inside it. C2 is reached through the backedge counter, and Tier4BackEdgeThreshold is 40000 - a single call over a million-element array passes that twenty-six times over, so the method is already compiled during the very first call. -XX:+PrintCompilation shows it happening:

187 1664       4       java.util.Arrays::fill (21 bytes)
187 1665 %     4       java.util.Arrays::fill @ 5 (21 bytes)

4 is C2, and % marks an on-stack-replacement compilation - the JIT swapped the method out from under the loop while it was still running. Had we been relying on invocation counts we would have needed thousands of calls, because Tier4InvocationThreshold is 5000.

The extra warm-up calls are there for a different reason, and it is worth knowing about if you reproduce this. The first fill of a freshly created array runs at about 2 ms, some forty times faster than the steady state, and it takes another two or three calls before the timings settle. That is not a JIT effect - it also happens with a humongous array that was never collected, and it survives -XX:G1ConcRefinementThreads=0. I have not chased that one to the bottom. On top of it, once we are in the slow regime the per-call time oscillates between roughly 54 ms and 100 ms, so a five-iteration measurement will cheerfully report a number 40% too low. Hence five warm-up calls and thirty measured ones.

array young (fresh eden allocation)          0,501 ms/op
same array, after 4x System.gc()            85,023 ms/op

More than a hundred times slower - I measure between 130x and 160x across runs.

And this time we do not have to take my word for where the array went, because we can watch it move. The same run with -Xlog:gc,gc+heap=info added:

[0,292s][info][gc,heap] GC(0) Eden regions: 5->0(6)
[0,292s][info][gc,heap] GC(0) Survivor regions: 0->0(0)
[0,292s][info][gc,heap] GC(0) Old regions: 1->6
[0,292s][info][gc,heap] GC(0) Humongous regions: 0->0
[0,292s][info][gc     ] GC(0) Pause Full (System.gc()) 38M->7M(160M) 3,769ms
[0,296s][info][gc,heap] GC(1) Old regions: 6->2
[0,303s][info][gc,heap] GC(2) Old regions: 2->1
[0,308s][info][gc,heap] GC(3) Eden regions: 0->0(1)
[0,308s][info][gc,heap] GC(3) Survivor regions: 0->0(0)
[0,308s][info][gc,heap] GC(3) Old regions: 1->1
[0,308s][info][gc,heap] GC(3) Humongous regions: 0->0
[0,308s][info][gc     ] GC(3) Pause Full (System.gc()) 7M->7M(32M) 5,608ms

I trimmed GC(1) and GC(2) down to the single line that changes. And if the <type> regions: <from>-><to>(<max>) notation is new to you, I explained how to read it in the humongous article.

The first cycle is the one that does the work. Eden regions: 5->0 and Old regions: 1->6 - everything that was sitting in eden, our array included, has been evacuated into the old generation. The remaining cycles only compact what is left, and by GC(3) the heap has settled at Old regions: 1->1 with an empty eden. That single 8 MB old region is where our 4 MB array now lives, and it is why the second measurement is a hundred times slower than the first.

Note also Humongous regions: 0->0 in every cycle. There is no humongous object anywhere in this test. Nothing changed except which generation the array lives in - and any long-lived reference array in your application is already in this state.

How bad is it? It depends on the array size

This is the part that surprised me, and it is the reason this article is longer than it was supposed to be.

I ran the same young-versus-old comparison across a range of array sizes, measuring nanoseconds per stored element:

elements array size young old penalty
256 1 KB 0.50 ns 0.92 ns 1.8x
1 024 4 KB 0.50 ns 1.08 ns 2.2x
4 096 16 KB 0.51 ns 2.23 ns 4.4x
16 384 64 KB 0.50 ns 7.24 ns 14x
65 536 256 KB 0.48 ns 29.00 ns 60x
262 144 1 MB 0.48 ns 50.56 ns 105x
1 048 568 4 MB 0.48 ns 66.53 ns 139x

The code being executed is identical in every row. Same instructions, same branches, same fence, same one-card-per-128-elements ratio of bookkeeping. Only the array size changes, and the penalty moves by a factor of 70.

A long-lived array of a few hundred references costs you essentially nothing. A multi-megabyte one costs you 139x.

Why the size matters

To explain that, I needed a baseline that isolates the barrier from everything else. Arrays.fill(table, null) is perfect for the job: same array, same loop, same memory traffic, but it leaves the barrier through exit 2 and never reaches the fence.

Here it is, measured on old-generation arrays from 1 KB up to 128 MB:

fill(null)   0.48 - 0.51 ns/store, at EVERY size

Dead flat. Well past the 16 MB L2 cache on this machine. That tells us the raw store loop is not memory bound at any size - the CPU keeps dozens of stores in flight at once, merges them into whole cache lines, and pipelines them out to memory. A 128 MB fill costs the same per element as a 1 KB fill.

Which is exactly what the fence destroys.

dmb ish does not have a fixed price. It stalls the core until every write already in flight has become visible to all other cores. With one fence after every store, only one store can be outstanding at a time. All that parallelism is gone, and the per-store cost becomes the completion latency of a single store to whatever level of cache owns that line:

So the curve has the shape of the memory hierarchy. Which also predicts that it should stop somewhere, and it does.

Very large arrays: it saturates

Continuing past 4 MB (all of these are humongous, so they are born in old generation - the GC log confirms 1, 2, 4 and 16 humongous regions respectively):

elements array size fill(mark) fill(null) arraycopy
2 097 136 8 MB 61.95 ns 0.48 ns 0.10 ns
4 194 272 16 MB 67.02 ns 0.49 ns 0.10 ns
8 388 544 32 MB 61.69 ns 0.49 ns 0.11 ns
33 554 176 128 MB 48.63 ns 0.50 ns 0.11 ns

The penalty plateaus at roughly 50-70 ns per store and goes no further. Making the array 32 times bigger does not make each individual store any worse - 128 MB is actually the mildest of the large sizes. There is a ceiling on how bad this gets per store. The total cost of course keeps growing with the array, but only linearly.

And the ParallelGC control at the same sizes:

ParallelGC   32 MB   fill(mark)  0,24 ns/store    fill(null)  0,24 ns/store
ParallelGC  128 MB   fill(mark)  0,24 ns/store    fill(null)  0,24 ns/store

Perfectly flat, and identical whether or not a reference is actually being stored. Two unconditional instructions, no branch, no fence, nothing to be slow about - even at 128 MB.

Two bonus effects

While reading the generated code I noticed two more differences that have nothing to do with the fence, and that are worth knowing about because they apply to every G1 application, not just this pathological one.

The barrier costs you loop unrolling

Remember that the ParallelGC loop incremented its counter by 8 and the G1 loop incremented by 1? That is not a coincidence. In src/hotspot/share/opto/loopTransform.cpp, C2 decides whether a loop is small enough to unroll:

uint body_size = _body.size();
for (uint k = 0; k < _body.size(); k++) {
    Node* n = _body.at(k);
    if (MemNode::barrier_data(n) != 0) {
      body_size += BarrierSet::barrier_set()->barrier_set_c2()->estimated_barrier_size(n);
    }
    ...
}
...
if (body_size > (uint)_local_loop_unroll_limit) {
    return false; // Loop too big.
}

C2 estimates how large the loop body will become once the barriers are expanded. For G1 that estimate is (g1BarrierSetC2.cpp):

uint G1BarrierSetC2::estimated_barrier_size(const Node* node) const {
  uint8_t barrier_data = MemNode::barrier_data(node);
  uint nodes = 0;
  if ((barrier_data & G1C2BarrierPre) != 0) {
    nodes += 6;
  }
  if ((barrier_data & G1C2BarrierPost) != 0) {
    nodes += 60;
  }
  return nodes;
}

6 + 60 = 66, and LoopUnrollLimit defaults to 60. A loop containing even a single G1 reference store is over budget before anything else is counted, so it can never be unrolled. Verified:

G1, default LoopUnrollLimit=60     1,077 ms/op
G1, -XX:LoopUnrollLimit=200        0,826 ms/op

The barrier costs you inlining

In the profiler output, ParallelGC inlined Arrays.fill into the benchmark method, while G1 compiled it as a separate method. The reason is in src/hotspot/share/opto/bytecodeInfo.cpp:

if (callee_method->has_compiled_code() &&
    callee_method->inline_instructions_size() > InlineSmallCode) {
  set_msg("already compiled into a big method");
  return true;
}

InlineSmallCode is 2500 bytes. The barrier code bloats the standalone compilation of Arrays.fill past that limit, so C2 refuses to inline it anywhere. -XX:+PrintInlining prints that exact message.

Together these two account for the 1.12 ms/op versus 0.50 ms/op difference between G1 and ParallelGC when the array is young - roughly 2x, which is the honest, unavoidable price of the G1 barrier. Everything above that came from the old generation accident.

What can we do about it?

Everything in this section applies to JDK 25 and older. If you are already on JDK 26 the barrier itself has been rewritten and the problem largely disappears - see the postscript at the end of the article.

Store null - it is free

OLD array, Arrays.fill(mark)    55.20 ns/store
OLD array, Arrays.fill(null)     0.49 ns/store

Clearing a long-lived array (Arrays.fill(a, null), ArrayList.clear(), invalidating a cache) never enters the slow path, because of exit 2. This one is free and you are probably already doing it.

Use bulk copies

System.arraycopy and Arrays.copyOf on reference arrays do not use the per-element barrier at all. They use G1BarrierSetAssembler::gen_write_ref_array_post_barrier, which dirties the whole affected card range once, after the copy is done. One barrier for the entire array instead of a fence per element.

The classic doubling trick turns a fill into a sequence of copies:

static void fill(Object[] a, Object v) {
    int len = a.length;
    if (len == 0) {
        return;
    }
    a[0] = v;
    for (int i = 1; i < len; i += i) {
        System.arraycopy(a, 0, a, i, Math.min(i, len - i));
    }
}

On a 4 MB old-generation array:

OLD    Arrays.fill                55,20 ns/store
OLD    arraycopy doubling          0,06 ns/store

900 times faster. Same result, same array, same GC.

Do not create the humongous array in the first place

Everything I wrote in the humongous article still applies, and this is one more reason to care. If your array is humongous, it is in old generation from birth and it will never get the cheap barrier path, no matter how short-lived it is.

Tuning flags - carefully

One flag genuinely changes this behaviour, and it comes with the usual warning.

-XX:G1HeapRegionSize=<size> raises the humongous threshold, which can move an array out of old generation. It also changes how G1 manages the entire heap, so please read the humongous article before touching it - and remember the maximum is 32 MB.

And one that does not work, in case you were about to try it the way I was. -XX:+UseCompactObjectHeaders shrinks the array header from 16 bytes to 12, which sounds like exactly enough to get under a threshold we missed by 16 bytes. It is not, because objects are aligned to 8 bytes and the 4 saved bytes are handed straight back as padding:

header array size after alignment humongous?
16 bytes 16 + 4 194 304 = 4 194 320 4 194 320 yes
12 bytes 12 + 4 194 304 = 4 194 316 4 194 320 yes

Both report Humongous regions: 1->1 and both run at 109 ms/op. A good reminder that when a threshold is this close, you have to measure rather than reason about it.

So there is really one flag, and I would not reach for it either. The application-level fixes above are better, for exactly the reasons I gave at the end of the humongous article: you control your code, and you do not have to understand five years of G1 internals to keep it working.

Summary

One more thing: JDK 26 fixes this

Everything above was measured on JDK 25. On JDK 26 the picture changes completely, because of JEP 522: G1 GC: Improve Throughput by Reducing Synchronization.

The JEP gives G1 a second card table. Application threads now dirty their own card table with no synchronization with the GC threads, which work on the other one. That removes the reason the StoreLoad fence existed in the first place, and with it goes the dirty card queue insertion, the runtime call, and the entire out-of-line slow path. The barrier collapses into straight-line code:

ldrb  w14, [x28, #0x48]     ; SATB pre-barrier flag, unchanged
cbnz  w14, <stub>
subs  x15, x19, x27
csel  x15, x15, xzr, hs
lsr   x15, x15, #3
str   w15, [x12]            ; the store
eor   x14, x12, x19         ; cross-region?
lsr   x14, x14, #0x17
cbz   x14, done
cbz   x19, done             ; storing null?
lsr   x14, x12, #9
ldr   x13, [x28, #0x50]     ; card table base - now taken from the thread
ldrb  w8,  [x14, x13]
tbz   w8, #0, done          ; already dirty?
strb  wzr, [x14, x13]       ; dirty it
done:

No jump to a stub, no queue, and - the whole point of this article - no dmb. I counted them in the compiled Arrays.fill: one on JDK 25, zero on JDK 26.

The benchmark this article started with:

  JDK 25 JDK 26
G1, humongous array 108,811 ms/op 1,121 ms/op
G1, array promoted to old generation 56,711 ms/op 0,527 ms/op
ParallelGC 0,532 ms/op 0,520 ms/op

And the size-dependent penalty, the curve that climbed from 1.8x to 139x, is simply gone. On JDK 26 an old generation array costs the same per store at every size I tried (both columns re-run back to back for this table, which is why the JDK 25 figures differ a little from the ones earlier in the article):

array size JDK 25, old JDK 26, old
4 KB 0.96 ns/store 0.49 ns/store
64 KB 12.72 ns/store 0.50 ns/store
1 MB 47.57 ns/store 0.51 ns/store
4 MB 51.65 ns/store 0.52 ns/store

So if you have read this far thinking “we run G1 and we do exactly that”, your fix may be an upgrade rather than a code change. It is on by default and there is no flag to turn on.

Thanks to Francesco Nigro for pointing me at JEP 522.

And the meta-lesson, which is the one I actually care about: the JMH numbers told me that G1 was 265 times slower. The assembly told me which instruction. But only the JDK source told me why that instruction was there, and only then could I design the experiments that showed the real answer was not the one I first assumed. All three layers were necessary, and the HotSpot source is far more readable than its reputation suggests - the barrier assembler is literally one line of C++ per machine instruction.