[Java][JVM][Memory leak][Profiling][Spring] Spring Boot + Spring Security + Apereo CAS + Concurrent Session Control = Heap memory leak

Application configuration

Just two dependencies, nothing sophisticated:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.security</groupId>
    <artifactId>spring-security-cas</artifactId>
</dependency>

Spring security configuration:

@Override
protected void configure(final HttpSecurity http) throws Exception {
    ...
    http
        .sessionManagement()
            .maximumSessions(1)
        ...
    ;
}

The maximumSessions parameter enables Concurrent Session Control feature. After such a configuration the application let Spring to initialize CasAuthenticationFilter with its defaults.

How do I know that there is a memory leak?

What I usually do is to look at the heap after GC chart. You can generate such a chart from GC logs. Such a chart for a stateful application that has memory leak looks like this: (one week period)

alt text

When there is no memory leak the chart looks like this:

alt text

What I’m looking for is the size of the heap after GC when the application has the lowest usage. If that size grows every day then I assume that there may be heap memory leak.

Simplified application

Unfortunately I cannot show you the heap dumps of a real production system. It contains too sensitive data. I’ve created a simplified application to recreate the problem. To show the leak quicker I configured it:

server.servlet.session.timeout=1m

I’ve also created listener to gather information about session creation and destruction by Tomcat server:

@Slf4j
@WebListener
public class CustomSessionListener implements HttpSessionListener {
    @Override
    public void sessionCreated(HttpSessionEvent sessionEvent) {
        log.info("New session is created. " + sessionEvent.getSession().getId());
    }

    @Override
    public void sessionDestroyed(HttpSessionEvent sessionEvent) {
        log.info("Session destroyed. " + sessionEvent.getSession().getId());
    }
} 

Let’s run this application and see how such a session looks at the heap after creating one session. After a successful login, we can see in the log:

pl.ks.boot.leak.CustomSessionListener    : New session is created. 1CA551A1BAC6BA0988643147CE02F4E8

At the heap we can find one StandardSession object:

alt text

Path to GC root

When Garbage collector finds alive objects, it starts its work from objects called GC roots. Those objects are alive by its definition, for example following objects are GC roots:

Local variables
Static variables
Active Java threads

and many more.

If any object is alive it means that there is a path of references from any of the GC roots to it. We can look at such a path at heap dump analyzer:

alt text

The StandardSession object has two paths to GC roots:

First is managed by CAS client by its SingleSignOutHandler
Second is managed by Tomcat server by its session StandardManager

Session timeout

After we wait one minute we can see new entry in my log file:

pl.ks.boot.leak.CustomSessionListener    : Session destroyed. 0F554402B686467A4BF8FC2E2248E5FA

When we look at this log and creation one we can see a curious thing:

pl.ks.boot.leak.CustomSessionListener    : New session is created. 1CA551A1BAC6BA0988643147CE02F4E8
pl.ks.boot.leak.CustomSessionListener    : Session destroyed. 0F554402B686467A4BF8FC2E2248E5FA

The session ID has changed. Ok, this is fully supported by HttpServletRequest API:

public interface HttpServletRequest extends ServletRequest {
    ...
    String changeSessionId();
    ...
}

Let’s look at the heap dump after session timeout:

alt text

There is still one StandardSession object, which is a memory leak, because we have no active session in the application. Let’s look at the path to GC roots:

alt text

The path managed by Tomcat server has disappeared, but the one managed by CAS client still exists.

CAS client source code

Let’s look:

public final class SingleSignOutHandler {
    ...
    /** Mapping of token IDs and session IDs to HTTP sessions */
    private SessionMappingStorage sessionMappingStorage = 
                                  new HashMapBackedSessionMappingStorage();
    ...
}

public interface SessionMappingStorage {
    /**
     * Remove the HttpSession based on the mappingId.
     * 
     * @param mappingId the id the session is keyed under.
     * @return the HttpSession if it exists.
     */
    HttpSession removeSessionByMappingId(String mappingId);

    /**
     * Remove a session by its Id.
     * @param sessionId the id of the session.
     */
    void removeBySessionById(String sessionId);

    /**
     * Add a session by its mapping Id.
     * @param mappingId the id to map the session to.
     * @param session the HttpSession.
     */
    void addSessionById(String mappingId, HttpSession session);

The SessionMappingStorage.removeBySessionById() method is called from:

public final class SingleSignOutHttpSessionListener implements HttpSessionListener {
    ...
    @Override
    public void sessionDestroyed(final HttpSessionEvent event) {
        if (sessionMappingStorage == null) {
            sessionMappingStorage = getSessionMappingStorage();
        }
        final HttpSession session = event.getSession();
        sessionMappingStorage.removeBySessionById(session.getId());
    }
    ...
}

Let’s debug the CAS client code. As strange as it sounds, debugging the library/framework code is sometimes the only way to understand the reason of the memory leak. I have added two breakpoints at:

org.jasig.cas.client.session.HashMapBackedSessionMappingStorage#addSessionById
org.jasig.cas.client.session.HashMapBackedSessionMappingStorage#removeBySessionById

After the login the first method was executed, and after session timeout the second method was also executed. At the log file there was an entry:

pl.ks.boot.leak.CustomSessionListener    : New session is created. 7B0CCD97AC979672771D34562D6D6211

The view I saw after stopping on removeBySessionById method was:

alt text

So CAS client wants to remove session with id BFBEE2D66588F33DCA1C60F5CDDCFDC3, but in its storage there is only the 7B0CCD97AC979672771D34562D6D6211 entry. So it looks like the CAS client does not support session ID change.

What options do we have?

We can:

Fix CAS client by ourselves, by adding implementation of javax.servlet.http.HttpSessionIdListener which modifies the map in HashMapBackedSessionMappingStorage
Find out why our application is the changing session ID and try to make it stop

Let’s go with the second options.

Spring and Tomcat debugging

Let’s put a breakpoint at org.apache.catalina.connector.Request#changeSessionId(), and see the stacktrace:

alt text

The session id change is triggered by the Spring ChangeSessionIdAuthenticationStrategy:

public final class ChangeSessionIdAuthenticationStrategy 
             extends AbstractSessionFixationProtectionStrategy {
	@Override
	HttpSession applySessionFixation(HttpServletRequest request) {
		request.changeSessionId();
		return request.getSession();
	}
}

It is executed from onAuthentication() method from CompositeSessionAuthenticationStrategy class. Let’s put a breakpoint at the constructor of that class:

alt text

The ChangeSessionIdAuthenticationStrategy is added to delegateStrategies. The stack is initialized by the AbstractSecurityBuilder. When we look at the source of the previous frame at the stack:

public final class SessionManagementConfigurer<H extends HttpSecurityBuilder<H>>
    extends AbstractHttpConfigurer<SessionManagementConfigurer<H>, H> {
  ...
  private SessionAuthenticationStrategy getSessionAuthenticationStrategy(H http) {
    ...
    if (isConcurrentSessionControlEnabled()) {
      SessionRegistry sessionRegistry = getSessionRegistry(http);
      ConcurrentSessionControlAuthenticationStrategy concurrentSessionControlStrategy = 
            new ConcurrentSessionControlAuthenticationStrategy(sessionRegistry);
      concurrentSessionControlStrategy.setMaximumSessions(this.maximumSessions);
      concurrentSessionControlStrategy.setExceptionIfMaximumExceeded(
            this.maxSessionsPreventsLogin);
      concurrentSessionControlStrategy = postProcess(concurrentSessionControlStrategy);

      RegisterSessionAuthenticationStrategy registerSessionStrategy = 
            new RegisterSessionAuthenticationStrategy(sessionRegistry);
      registerSessionStrategy = postProcess(registerSessionStrategy);

      delegateStrategies.addAll(Arrays.asList(concurrentSessionControlStrategy,
      defaultSessionAuthenticationStrategy, registerSessionStrategy));
    } else {
      delegateStrategies.add(defaultSessionAuthenticationStrategy);
    } 
    this.sessionAuthenticationStrategy = postProcess(
          new CompositeSessionAuthenticationStrategy(delegateStrategies));
    return this.sessionAuthenticationStrategy;
  }
  ...
}

After debugging this code you can see, that ChangeSessionIdAuthenticationStrategy is added in the if (isConcurrentSessionControlEnabled()) block and its instance is stored in the defaultSessionAuthenticationStrategy field. This field is also calculated in that method:

public final class SessionManagementConfigurer<H extends HttpSecurityBuilder<H>>
        extends AbstractHttpConfigurer<SessionManagementConfigurer<H>, H> {
    ...
    private SessionAuthenticationStrategy getSessionAuthenticationStrategy(H http) {
        ...
        SessionAuthenticationStrategy defaultSessionAuthenticationStrategy;
        if (this.providedSessionAuthenticationStrategy == null) {
            // If the user did not provide a SessionAuthenticationStrategy
            // then default to sessionFixationAuthenticationStrategy
            defaultSessionAuthenticationStrategy = postProcess(
                this.sessionFixationAuthenticationStrategy);
        }
        else {
            defaultSessionAuthenticationStrategy = 
                this.providedSessionAuthenticationStrategy;
        }
        ...
    }
    ...
}

So it looks like we could override it with the other instance of AbstractSessionFixationProtectionStrategy. Let’s google how we can change it. The first result shows we can manage it by:

http.sessionManagement()
  .sessionFixation().migrateSession()

Our options are:

alt text

Which one is appropriate for your application is yours to decide. The session fixation feature is a protection against the attack, so it is good to have this feature. We can switch from changeSessionId to migrateSession, which implementation looks like:

public class SessionFixationProtectionStrategy 
       extends AbstractSessionFixationProtectionStrategy {
    ...
    @Override
    final HttpSession applySessionFixation(HttpServletRequest request) {
        HttpSession session = request.getSession();
        String originalSessionId = session.getId();
        this.logger.debug(LogMessage.of(() -> "Invalidating session with Id '" 
            + originalSessionId + "' "+ (this.migrateSessionAttributes ? "and" : "without") 
            + " migrating attributes."));
        Map<String, Object> attributesToMigrate = extractAttributes(session);
        int maxInactiveIntervalToMigrate = session.getMaxInactiveInterval();
        session.invalidate();
        session = request.getSession(true); // we now have a new session
        this.logger.debug(LogMessage.format("Started new session: %s", session.getId()));
        transferAttributes(attributesToMigrate, session);
        if (this.migrateSessionAttributes) {
            session.setMaxInactiveInterval(maxInactiveIntervalToMigrate);
        }
        return session;
    }
    ...
}

This strategy uses session.invalidate() method, which will trigger sessionDestroyed event and the CAS client will remove it from its repository.

Where is the bug?

From my perspective the bug is in the CAS client, which doesn’t support changing session id.