linux - Jboss AS 7.1.1 ejb 3 : EJB pool error

Question

We have an application running on Jboss AS 7.1.1 and uses EJB 3. We are intermittently seeing the following errors

javax.ejb.EJBException: JBAS014516: Failed to acquire a permit within 5 MINUTES at org.jboss.as.ejb3.pool.strictmax.StrictMaxPool.get(StrictMaxPool.java:109) at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:8 0) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java :41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) at org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:179) at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:179) at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43) at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:181) at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:128) at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:181) at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136) at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121) at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)

Below is the configuration for the EJB subsystem in the standalone.xml

<subsystem xmlns="urn:jboss:domain:ejb3:1.2">
        <session-bean>
            <stateless>
                <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
            </stateless>
            <stateful default-access-timeout="5000" cache-ref="simple"/>
            <singleton default-access-timeout="5000"/>
        </session-bean>
        <mdb>
            <resource-adapter-ref resource-adapter-name="hornetq-ra"/>
            <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
        </mdb>
        <pools>
            <bean-instance-pools>
                <strict-max-pool name="slsb-strict-max-pool" max-pool-size="40" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                <strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
            </bean-instance-pools>
        </pools>
        <caches>
            <cache name="simple" aliases="NoPassivationCache"/>
            <cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>
        </caches>
        <passivation-stores>
            <file-passivation-store name="file"/>
        </passivation-stores>
        <async thread-pool-name="default"/>
        <timer-service thread-pool-name="default">
            <data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>
        </timer-service>
        <remote connector-ref="remoting-connector" thread-pool-name="default"/>
        <thread-pools>
            <thread-pool name="default">
                <max-threads count="10"/>
                <keepalive-time time="100" unit="milliseconds"/>
            </thread-pool>
        </thread-pools>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:infinispan:1.2" default-cache-container="hibernate">
        <cache-container name="hibernate" default-cache="local-query">
            <local-cache name="entity">
                <transaction mode="NON_XA"/>
                <eviction strategy="LRU" max-entries="10000"/>
                <expiration max-idle="100000"/>
            </local-cache>
            <local-cache name="local-query">
                <transaction mode="NONE"/>
                <eviction strategy="LRU" max-entries="10000"/>
                <expiration max-idle="100000"/>
            </local-cache>
            <local-cache name="timestamps">
                <transaction mode="NONE"/>
                <eviction strategy="NONE"/>
            </local-cache>
        </cache-container>
    </subsystem>

We are trying to troubleshoot what may be causing this issue and a fix for it.

Answer 1

This happens, when the configured EJB pool is exhausted - means that all EJBs are currently "in use" (=processing). Look for long running invocations within your EJBs. It seems that the invocations take longer than 5 minutes in your case.