Encountering "ConcurrentAccessTimeoutException" when making high-volume calls to @Stateless classes

[Kean Erickson]

I'm using TomEE 7.0.4, EJB version 3.1
I often see this when making a high volume of calls to methods in
@Stateless classes:

javax.ejb.ConcurrentAccessTimeoutException: No instances available in
Stateless Session Bean pool. Waited 30 SECONDS

I've set maxSize to 1000 in my tomee.xml and I still receive the error just
as soon as I would without maxSize specified at all, which leads me to
believe it's not doing anything. I see recommendations on this list to use
@Singleton with a read lockmode, but that doesn't seem to make a lot of
sense with our current arrangement.



Thanks!
-Kean

[Jonathan Gallimore]

Hi

Can you get a thread dump when you're seeing this issue?

Thanks

Jon

Post by Kean Erickson
I'm using TomEE 7.0.4, EJB version 3.1
I often see this when making a high volume of calls to methods in
javax.ejb.ConcurrentAccessTimeoutException: No instances available in
Stateless Session Bean pool. Waited 30 SECONDS
I've set maxSize to 1000 in my tomee.xml and I still receive the error just
as soon as I would without maxSize specified at all, which leads me to
believe it's not doing anything. I see recommendations on this list to use
@Singleton with a read lockmode, but that doesn't seem to make a lot of
sense with our current arrangement.
Thanks!
-Kean

[Kean Erickson]

Sure, thread dump is attached. I had to censor out references to my code,
but calls in the Stateless class in question are marked with three
exclamations, "!!!"

After a few minutes, this happens for every call to a method in that
@Stateless class, which is acquired and used in non-EJB classes via
BeanManager.getReference


10-Oct-2018 10:35:50.749 SEVERE [pool-17-thread-9]
javax.ejb.ConcurrentAccessTimeoutException: No instances available in
Stateless Session Bean pool. Waited 30 SECONDS
at
org.apache.openejb.core.stateless.StatelessInstanceManager.getInstance(StatelessInstanceManager.java:226)
at
org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:204)
at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265)
at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260)
at
org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89)
at
org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347)
at (method in stateless class)
at (method in non-EJB class)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)




On Wed, Oct 10, 2018 at 1:45 AM Jonathan Gallimore <

Post by Jonathan Gallimore
Hi
Can you get a thread dump when you're seeing this issue?
Thanks
Jon

[Jonathan Gallimore]

At first glance, it looks like your threads are actually stuck trying to
borrow a database connection from the pool. What does your stateless pool
config and your datasource config look like (remove anything sensitive,
like passwords etc)?

Jon

Post by Kean Erickson
Sure, thread dump is attached. I had to censor out references to my code,
but calls in the Stateless class in question are marked with three
exclamations, "!!!"
After a few minutes, this happens for every call to a method in that
@Stateless class, which is acquired and used in non-EJB classes via
BeanManager.getReference
10-Oct-2018 10:35:50.749 SEVERE [pool-17-thread-9]
javax.ejb.ConcurrentAccessTimeoutException: No instances available in
Stateless Session Bean pool. Waited 30 SECONDS
at
org.apache.openejb.core.stateless.StatelessInstanceManager.getInstance(StatelessInstanceManager.java:226)
at
org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:204)
at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265)
at
org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260)
at
org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89)
at
org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347)
at (method in stateless class)
at (method in non-EJB class)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
On Wed, Oct 10, 2018 at 1:45 AM Jonathan Gallimore <

Post by Jonathan Gallimore
Hi
Can you get a thread dump when you're seeing this issue?
Thanks
Jon

[Kean Erickson]

That actually answered my question, I didn't have a proper stateless pool
config in my tomee.xml... I assumed everything would be configured under
<Resource> so I was putting the maxSize property there, hence the property
not working. I added a <Container> and put the prop there instead. Things
are working as expected now. Thanks!

On Wed, Oct 10, 2018 at 12:02 PM Jonathan Gallimore <

Post by Jonathan Gallimore
At first glance, it looks like your threads are actually stuck trying to
borrow a database connection from the pool. What does your stateless pool
config and your datasource config look like (remove anything sensitive,
like passwords etc)?
Jon

Post by Kean Erickson
Sure, thread dump is attached. I had to censor out references to my code,
but calls in the Stateless class in question are marked with three
exclamations, "!!!"
After a few minutes, this happens for every call to a method in that
@Stateless class, which is acquired and used in non-EJB classes via
BeanManager.getReference
10-Oct-2018 10:35:50.749 SEVERE [pool-17-thread-9]
javax.ejb.ConcurrentAccessTimeoutException: No instances available in
Stateless Session Bean pool. Waited 30 SECONDS
at

org.apache.openejb.core.stateless.StatelessInstanceManager.getInstance(StatelessInstanceManager.java:226)

Post by Kean Erickson
at

org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:204)

Post by Kean Erickson
at

org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:265)

Post by Kean Erickson
at

org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:260)

Post by Kean Erickson
at

org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:89)

Post by Kean Erickson
at

org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:347)

Post by Kean Erickson
at (method in stateless class)
at (method in non-EJB class)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

Post by Kean Erickson
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

Post by Kean Erickson
at java.lang.Thread.run(Thread.java:748)
On Wed, Oct 10, 2018 at 1:45 AM Jonathan Gallimore <

Post by Jonathan Gallimore
Hi
Can you get a thread dump when you're seeing this issue?
Thanks
Jon