# Let's investigate a Gradle IntelliJ memory leak!

> 👋 Hi, this is P.Y., I work as an Android Engineer at [Block](https://block.xyz/). This article shares a team investigation by [Tony Robalik](https://twitter.com/AutonomousApps), [Pablo Baxter](https://github.com/pablobaxter), [Roger Hu](https://twitter.com/rogerjhu) and [myself](https://twitter.com/Piwai) into a recent **Gradle / IntelliJ memory leak**.

On September 29th, [Tony Robalik](https://twitter.com/AutonomousApps) reaches out to our friends at Gradle to report memory issues with the Gradle process when importing a project in IntelliJ IDEA. The heap size keeps climbing to new heights, reaching 60+ GB! Tony writes:

> Normally, after I start another build, the daemon gives up most of the memory it had used in the first build, i.e. it takes until that moment for the GC to run. In the past, I've been able to force the gc to run with `jcmd <pid> GC.run` and get my memory back or just run a simple build like `help`. However, right now, that's not happening.

# Dominators

The Java heap is an object graph. One useful tool we can leverage from graph theory is something called the [dominator](https://en.wikipedia.org/wiki/Dominator_\(graph_theory\)) tree:

> A node `d` dominates a node `n` if every path in the object graph from GC roots to `n` must go through `d`.

In practice, the dominator tree provides us with the list of biggest objects sorted by retained size. The retained size is the sum of the size of all the objects that would become unreachable if the dominator object was unreachable.

Tony takes a heap dump of the Gradle process and shares a screenshot from the **Biggest Objects - Dominators** tab in YourKit:

![Yourkit Biggest Objects - Dominators](https://cdn.hashnode.com/res/hashnode/image/upload/v1665566645848/9ZzZB-QqK.png align="left")

We immediately notice that 95% of the 44 GB heap is retained by `java.lang.ref.Finalizer`, which means, as YourKit gently points out, that the memory is retained by an object that is pending finalization.

# Pending Finalization

Once an object is unreachable, it can be garbage collected and its memory reclaimed. If that objects implements the `finalize()` method, then that method must be called before garbage collection. Once objects with a `finalize()` are detected as unreachable, they're put in a finalizer queue and are in a "pending finalization" state until `finalize()` is called.

![ProjectImportActionWithCustomSerializer dominator](https://cdn.hashnode.com/res/hashnode/image/upload/v1665566663404/IEaFt0zW6.png align="left")

Here we can see that the lowest dominator that retains most of the memory is `ProjectImportActionWithCustomSerializer`. It is **unreachable** & **transitively pending finalization**: even though it has no `finalize()` method, it is dominated by an object that is pending finalization, which means it is still indirectly reachable by that object which itself can still run code in its `finalize()` method. This means `ProjectImportActionWithCustomSerializer` cannot be garbage collected until its dominator is finalized.

# I Am GCroot 🌳

To understand which references exactly are keeping `ProjectImportActionWithCustomSerializer` in memory, I ask Tony to compute the shortest paths from GC Roots in YourKit:

![shortest paths from GC Roots](https://cdn.hashnode.com/res/hashnode/image/upload/v1665567126397/RJ6VUX2WW.png align="left")

Here's how to read this trace:

![1](https://cdn.hashnode.com/res/hashnode/image/upload/v1665568584070/v18zVeoNA.png align="left")

*   At the top is `ProjectImportActionWithCustomSerializer`. We want to understand why it's retained in memory.
    

![2](https://cdn.hashnode.com/res/hashnode/image/upload/v1665568608139/h-0UmNBWX.png align="left")

At the bottom is a GC root, here a `JNIGlobal` that keeps a reference to `CleanerImpl$PhantomCleanableRef`.

![3](https://cdn.hashnode.com/res/hashnode/image/upload/v1665568639643/PkZ_KeWIM.png align="left")

From the bottom to the top we see the chain of references that is retaining `ProjectImportActionWithCustomSerializer`.

![4](https://cdn.hashnode.com/res/hashnode/image/upload/v1665568681621/kFVDGTgTB.png align="left")

The bottom part of the trace is the finalizer queue. The finalizer queue is implemented as a [doubly linked list](https://en.wikipedia.org/wiki/Doubly_linked_list), where each `Finalizer` instance has a reference to the previous entry (`prev`) and next entry (`next`) in the finalizer queue, as well as a reference to the object that is pending finalization (`referent`).

![5](https://cdn.hashnode.com/res/hashnode/image/upload/v1665568710120/iuz32ggqK.png align="left")

As we move towards the top of the trace, we see that a `Finalizer` has a `referent` field referencing `Executors$FinalizableDelegatedExecutorService`. This is the object that implements `finalize()` and is pending finalization.

```java
    private static class FinalizableDelegatedExecutorService
            extends DelegatedExecutorService {
        FinalizableDelegatedExecutorService(ExecutorService executor) {
            super(executor);
        }
        @SuppressWarnings("deprecation")
        protected void finalize() {
            super.shutdown();
        }
    }
```

As you can see, `FinalizableDelegatedExecutorService` is an `ExecutorService` that automatically shuts down the thread pool when it becomes unreachable. Developers are expected to shut down thread pools manually when they stop being in use, but sometimes mistakes happen and this is a safety net.

![6](https://cdn.hashnode.com/res/hashnode/image/upload/v1665568907968/V3hcppQOn.png align="left")

The `Executors$FinalizableDelegatedExecutorService.e` field references a `ThreadPoolExecutor` instance.

![7](https://cdn.hashnode.com/res/hashnode/image/upload/v1665568930629/SYEKVIsYq.png align="left")

The `ThreadPoolExecutor.threadFactory` field references a `ProjectImportAction$1` instance. So we can assume `ProjectImportAction$1` is an anonymous class (because its name is `$1`) that implements `ThreadFactory`.

![8](https://cdn.hashnode.com/res/hashnode/image/upload/v1665568954635/sQCWkprCy.png align="left")

The `ProjectImportAction$1.this$0` field references the `ProjectImportActionWithCustomSerializer` instance. In Java, anonymous classes have a hidden reference to their outer class, compiled as a field name `this$0`.

# Reveal

At this point we can conclude that `ProjectImportActionWithCustomSerializer` is a class that extends `ProjectImportAction`, and that `ProjectImportAction` defines an anonymous class that implements `ThreadFactory` which is then passed to a `ThreadPoolExecutor`.

Let's look at the [ProjectImportAction](https://github.com/JetBrains/intellij-community/blob/ff07590cb24b25c055ce00dcd5c6f0db109e2bfa/plugins/gradle/tooling-extension-api/src/org/jetbrains/plugins/gradle/model/ProjectImportAction.java#L114-L120) sources:

```java
  myConverterExecutor =  Executors.newSingleThreadExecutor(
    new ThreadFactory() {
      @Override
      public Thread newThread(@NotNull Runnable runnable) {
        return new Thread(runnable, "idea-tooling-model-converter");
      }
    }
  );
 }
```

`ProjectImportAction` creates a single threaded executor, and passes in a `ThreadFactory` in order to set the thread name. That anonymous `ThreadFactory` doesn't actually use the hidden `this$0` reference to its `ProjectImportAction` outer class, unfortunately the Java compiler (unlike Kotlin) will still add that reference.

If we extract that anonymous class into a static class, this `this$0` reference will disappear and the `ProjectImportAction` implementation will not be retained while the thread pool executor is pending finalization.

```java
private static final class SimpleThreadFactory implements ThreadFactory {
  @Override
  public Thread newThread(@NotNull Runnable runnable) {
    return new Thread(runnable, "idea-tooling-model-converter");
  }
}
```

[Pablo Baxter](https://github.com/pablobaxter) files a [bug](https://youtrack.jetbrains.com/issue/IDEA-303282/Memory-leak-in-ProjectImportAction) and opens a [pull request](https://github.com/JetBrains/intellij-community/pull/2186/files) which is swiftly merged into the IntelliJ master branch.

[Roger Hu](https://twitter.com/rogerjhu) & [Tony Robalik](https://twitter.com/AutonomousApps) apply this fix locally by patching the `gradle-tooling-extension-api.jar` jar with [Recaf](https://www.coley.software/Recaf/) and confirm that the memory is now properly reclaimed 🎉 !

The git history shows that this bug was [introduced](https://github.com/pablobaxter/intellij-community/commit/8daa06d04f6e9ae2a2d32f2f09ea30e97c05bd24) in IntelliJ IDEA **2022.1** 221.4165.146 (that version is the base for Android Studio Electric Eel Canary 5). Last week, folks from JetBrains said they would "apply the changes and include it in next EAP of **2022.3** and next bugfix release of **2022.2** branch" while folks from Google said "we will cherry pick in EE". I love this quick turnaround!

# Are we done though?

Wait a minute, we fixed the leak, but why was the thread pool executor pending finalization for such a long time? Tony reproduces the bug a few more times and takes a peak at the finalization queue. It turns out there's a `ZipEntry` for a jar that is systematically hanging out near the head of the finalization queue. `ZipEntry` calls `close()` when finalized. We haven't quite figured out why `close()` takes so long, so we're leaving that as an exercise for you, dear reader 😘.

> Header image generated by DALL-E, prompt: "a photo of canary flying holding an elephant in the air".