TomEE Performance

Discussion:

TomEE Performance

f***@rohde-schwarz.com

2018-09-04 06:27:50 UTC

Hey,

we have been running some performance tests with our application (TomEE 7.0.5
based) and are stuck:

Until 4 core VMs (or docker containers) we see a linear increase in
performance, which is great and was anticipated.

But after 4 cores, we barely get 10% (with 8 cores) more performance of the
system.

We used jmeter based load tests (SOAP calls) with 10/20/30/40 and 100 threads,
VM to VM via 1gbit, and 4 GB of RAM.

We played around with session bean pool sizes (min set to thread count, max to
1000) and stateful bean pool settings and also with jvm heap size and GC
parameters, to no avail.

Are there any more performance parameters we can toy around with in TomEE or
Tomcat that you can recommend?

Thank you and best

Fabian

Jean-Louis Monteiro

2018-09-04 06:38:28 UTC

Permalink

Hi,

Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you are over
the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.

Jean-Louis

--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application (TomEE
Until 4 core VMs (or docker containers) we see a linear increase in
performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more performance of
the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40 and 100
threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread count,
max to 1000) and stateful bean pool settings and also with jvm heap size
and GC parameters, to no avail.
Are there any more performance parameters we can toy around with in TomEE
or Tomcat that you can recommend?
Thank you and best
Fabian

f***@rohde-schwarz.com

2018-09-04 08:53:11 UTC

Permalink

Like this:
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409de7c4 ?

-----Original Message-----
From: Jean-Louis Monteiro <***@tomitribe.com>
Sent: Tuesday, September 04, 2018 8:38 AM
To: ***@tomee.apache.org
Subject: *EXT* [Newsletter] Re: TomEE Performance

Hi,

Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you are over
the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.

Jean-Louis

--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application
(TomEE
Until 4 core VMs (or docker containers) we see a linear increase in
performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more performance
of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40 and 100
threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread
count, max to 1000) and stateful bean pool settings and also with jvm
heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around with in
TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

Jean-Louis Monteiro

2018-09-04 09:29:35 UTC

Permalink

Yes exactly.

Were you injecting anything when you took the jstack?
It seems that only one thread is working.

We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the jstack that'd be
great.
You can run `vmstat 5` on another terminal so you can see what's going on.

You are using Stateless Session beans. Any reason you aren't using a plain
singleton?
With Singleton (Lock.READ) there is no pooling involved, so you don't have
anything to configure or tune.

--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409de7c4 ?
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you are over
the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application
(TomEE
Until 4 core VMs (or docker containers) we see a linear increase in
performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more performance
of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40 and 100
threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread
count, max to 1000) and stateful bean pool settings and also with jvm
heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around with in
TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

f***@rohde-schwarz.com

2018-09-04 09:37:02 UTC

Permalink

Injecting? The jmeter was running during the threaddump. What do you mean with
injecting?

Is Singleton (Lock.READ) not even more of a bottleneck when it comes to
multiple concurrent requests? IIRC we tried Singleton before, but not sure
what the reason was why we went with @Stateless...

-----Original Message-----
From: Jean-Louis Monteiro <***@tomitribe.com>
Sent: Tuesday, September 04, 2018 11:30 AM
To: ***@tomee.apache.org
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance

Yes exactly.

Were you injecting anything when you took the jstack?
It seems that only one thread is working.

We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the jstack that'd be
great.
You can run `vmstat 5` on another terminal so you can see what's going on.

You are using Stateless Session beans. Any reason you aren't using a plain
singleton?
With Singleton (Lock.READ) there is no pooling involved, so you don't have
anything to configure or tune.

--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409de7c4 ?
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you are
over the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application
(TomEE
Until 4 core VMs (or docker containers) we see a linear increase in
performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more performance
of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40 and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread
count, max to 1000) and stateful bean pool settings and also with
jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around with in
TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

Jean-Louis Monteiro

2018-09-04 09:41:25 UTC

Permalink

Ah ok. Well I was asking if you were injecting when you took the thread
dump because from a server point of view I saw only one thread working.
If you were using jmeter with multiple virtual users, I was expecting to
see more than one thread working.

I'll double check.

@Singleton is by default using Lock WRITE which prevents multiple threads
to access the singleton.
If you don't need synchronization or if your code is thread safe (no
instance variables, etc). You can safely use @Singleton with Lock READ
which will be faster than @Stateless because there is no pool involved. And
again, there is no tuning to do

--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Injecting? The jmeter was running during the threaddump. What do you mean with
injecting?
Is Singleton (Lock.READ) not even more of a bottleneck when it comes to
multiple concurrent requests? IIRC we tried Singleton before, but not sure
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:30 AM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Yes exactly.
Were you injecting anything when you took the jstack?
It seems that only one thread is working.
We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the jstack that'd be
great.
You can run `vmstat 5` on another terminal so you can see what's going on.
You are using Stateless Session beans. Any reason you aren't using a plain
singleton?
With Singleton (Lock.READ) there is no pooling involved, so you don't have
anything to configure or tune.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409de7c4 ?
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you are
over the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application
(TomEE
Until 4 core VMs (or docker containers) we see a linear increase in
performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more performance
of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40 and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread
count, max to 1000) and stateful bean pool settings and also with
jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around with in
TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

Mark Struberg

2018-09-04 09:59:09 UTC

Permalink

Are you using @Singleton with a locking scenario or are all methods Read?
In that case you could possibly replace that @Singleton with a CDI @ApplicationScoped bean?

Of course with @Transactional if you make use of that, etc.
Would probably be worth a try.

LieGrue,
strub

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409de7c4 ?
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you are
over the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application
(TomEE
Until 4 core VMs (or docker containers) we see a linear increase in
performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more performance
of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40 and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread
count, max to 1000) and stateful bean pool settings and also with
jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around with in
TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

f***@rohde-schwarz.com

2018-09-04 09:53:44 UTC

Permalink

We have

@Resource
WebServiceContext webserviceContext;

in our SOAP api class, that should not work with @Singleton, or am I mistaken?

-----Original Message-----
From: Jean-Louis Monteiro <***@tomitribe.com>
Sent: Tuesday, September 04, 2018 11:41 AM
To: ***@tomee.apache.org
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: TomEE
Performance

Ah ok. Well I was asking if you were injecting when you took the thread dump
because from a server point of view I saw only one thread working.
If you were using jmeter with multiple virtual users, I was expecting to see
more than one thread working.

I'll double check.

@Singleton is by default using Lock WRITE which prevents multiple threads to
access the singleton.
If you don't need synchronization or if your code is thread safe (no instance
variables, etc). You can safely use @Singleton with Lock READ which will be
faster than @Stateless because there is no pool involved. And again, there is
no tuning to do

--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Injecting? The jmeter was running during the threaddump. What do you
mean with injecting?
Is Singleton (Lock.READ) not even more of a bottleneck when it comes
to multiple concurrent requests? IIRC we tried Singleton before, but
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:30 AM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Yes exactly.
Were you injecting anything when you took the jstack?
It seems that only one thread is working.
We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the jstack
that'd be great.
You can run `vmstat 5` on another terminal so you can see what's going on.
You are using Stateless Session beans. Any reason you aren't using a
plain singleton?
With Singleton (Lock.READ) there is no pooling involved, so you don't
have anything to configure or tune.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409de7c4 ?
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you
are over the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application
(TomEE
Until 4 core VMs (or docker containers) we see a linear increase
in performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more
performance of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40 and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread
count, max to 1000) and stateful bean pool settings and also with
jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around with
in TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

Jean-Louis Monteiro

2018-09-04 09:59:21 UTC

Permalink

I would need to check on this one and probably investigate a bit more what
you are doing.

That being said, I can confirm only one thread is currently working. So
either jmeter is only sending monothreaded requests or there is something
else.
But you aren't doing multiple requests in parallel

--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
We have
@Resource
WebServiceContext webserviceContext;
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:41 AM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: TomEE
Performance
Ah ok. Well I was asking if you were injecting when you took the thread dump
because from a server point of view I saw only one thread working.
If you were using jmeter with multiple virtual users, I was expecting to see
more than one thread working.
I'll double check.
@Singleton is by default using Lock WRITE which prevents multiple threads to
access the singleton.
If you don't need synchronization or if your code is thread safe (no instance
no tuning to do
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Injecting? The jmeter was running during the threaddump. What do you
mean with injecting?
Is Singleton (Lock.READ) not even more of a bottleneck when it comes
to multiple concurrent requests? IIRC we tried Singleton before, but
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:30 AM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Yes exactly.
Were you injecting anything when you took the jstack?
It seems that only one thread is working.
We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the jstack
that'd be great.
You can run `vmstat 5` on another terminal so you can see what's going

on.

Post by f***@rohde-schwarz.com
You are using Stateless Session beans. Any reason you aren't using a
plain singleton?
With Singleton (Lock.READ) there is no pooling involved, so you don't
have anything to configure or tune.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409de7c4 ?
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you
are over the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application
(TomEE
Until 4 core VMs (or docker containers) we see a linear increase
in performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more
performance of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40 and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread
count, max to 1000) and stateful bean pool settings and also with
jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around with
in TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

f***@rohde-schwarz.com

2018-09-04 10:15:32 UTC

Permalink

You were right, my student did something wrong with the first thread dump, I
uploaded a correct one:
https://gist.github.com/TrustedGate/83781f05950f245a3bfd2388c5bfd7ab

Also at the bottom there is a vmstat result, its definitely multithreaded :)

-----Original Message-----
From: Jean-Louis Monteiro <***@tomitribe.com>
Sent: Tuesday, September 04, 2018 11:59 AM
To: ***@tomee.apache.org
Subject: *EXT* [Newsletter] Re: TomEE Performance

I would need to check on this one and probably investigate a bit more what you
are doing.

That being said, I can confirm only one thread is currently working. So either
jmeter is only sending monothreaded requests or there is something else.
But you aren't doing multiple requests in parallel

--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
We have
@Resource
WebServiceContext webserviceContext;
mistaken?
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:41 AM
TomEE Performance
Ah ok. Well I was asking if you were injecting when you took the
thread dump because from a server point of view I saw only one thread
working.
If you were using jmeter with multiple virtual users, I was expecting
to see more than one thread working.
I'll double check.
@Singleton is by default using Lock WRITE which prevents multiple
threads to access the singleton.
If you don't need synchronization or if your code is thread safe (no
involved. And again, there is no tuning to do
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Injecting? The jmeter was running during the threaddump. What do you
mean with injecting?
Is Singleton (Lock.READ) not even more of a bottleneck when it comes
to multiple concurrent requests? IIRC we tried Singleton before, but
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:30 AM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Yes exactly.
Were you injecting anything when you took the jstack?
It seems that only one thread is working.
We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the jstack
that'd be great.
You can run `vmstat 5` on another terminal so you can see what's
going

on.

Post by f***@rohde-schwarz.com
You are using Stateless Session beans. Any reason you aren't using a
plain singleton?
With Singleton (Lock.READ) there is no pooling involved, so you
don't have anything to configure or tune.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 10:53 AM,

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409de7c4 ?
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you
are over the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 8:27 AM,

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application
(TomEE
Until 4 core VMs (or docker containers) we see a linear increase
in performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more
performance of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40
and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread
count, max to 1000) and stateful bean pool settings and also
with jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around with
in TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

Romain Manni-Bucau

2018-09-04 12:07:55 UTC

Permalink

Hi Fabian,

a few pointers and directions to check:

1. Ensure to test with securerandom.source=/dev/./urandom since you use a
lot of crypto
2. You use hsqldb which is known to not scale very well with the
concurrency, maybe switch to another database with a correctly configured
connection pool
3. You use DataBaseRealm which does a lookup of the connection for each
authentication which is synchronized and has no cache on the password
hashes so it can be slow at runtime

Personally i would start by using a fast realm (even if it always says
"ok") to validate this hypothesis before investigating other cases.

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>

Post by f***@rohde-schwarz.com
You were right, my student did something wrong with the first thread dump, I
https://gist.github.com/TrustedGate/83781f05950f245a3bfd2388c5bfd7ab
Also at the bottom there is a vmstat result, its definitely multithreaded :)
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:59 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
I would need to check on this one and probably investigate a bit more what you
are doing.
That being said, I can confirm only one thread is currently working. So either
jmeter is only sending monothreaded requests or there is something else.
But you aren't doing multiple requests in parallel
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Injecting? The jmeter was running during the threaddump. What do you
mean with injecting?
Is Singleton (Lock.READ) not even more of a bottleneck when it comes
to multiple concurrent requests? IIRC we tried Singleton before, but
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:30 AM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Yes exactly.
Were you injecting anything when you took the jstack?
It seems that only one thread is working.
We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the jstack
that'd be great.
You can run `vmstat 5` on another terminal so you can see what's
going

on.

Post by f***@rohde-schwarz.com
You are using Stateless Session beans. Any reason you aren't using a
plain singleton?
With Singleton (Lock.READ) there is no pooling involved, so you
don't have anything to configure or tune.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 10:53 AM,

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409de7c4

Post by f***@rohde-schwarz.com

Post by f***@rohde-schwarz.com
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when you
are over the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 8:27 AM,

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our application
(TomEE
Until 4 core VMs (or docker containers) we see a linear increase
in performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more
performance of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40
and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to thread
count, max to 1000) and stateful bean pool settings and also
with jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around with
in TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

f***@rohde-schwarz.com

2018-09-04 12:17:43 UTC

Permalink

Hi Romain,

thank you so much for your detailed feedback. After I created the second threaddump I had a look into it myself and came to the same conclusions. We will investigate further, and if there are any more TomEE related performance issues, I will get back in contact.

Thanks all for the replies!

Best
Fabian

-----Original Message-----
From: Romain Manni-Bucau <***@gmail.com>
Sent: Tuesday, September 04, 2018 2:08 PM
To: ***@tomee.apache.org
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance

Hi Fabian,

a few pointers and directions to check:

1. Ensure to test with securerandom.source=/dev/./urandom since you use a lot of crypto 2. You use hsqldb which is known to not scale very well with the concurrency, maybe switch to another database with a correctly configured connection pool 3. You use DataBaseRealm which does a lookup of the connection for each authentication which is synchronized and has no cache on the password hashes so it can be slow at runtime

Personally i would start by using a fast realm (even if it always says
"ok") to validate this hypothesis before investigating other cases.

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance>

Post by f***@rohde-schwarz.com
You were right, my student did something wrong with the first thread
https://gist.github.com/TrustedGate/83781f05950f245a3bfd2388c5bfd7ab
Also at the bottom there is a vmstat result, its definitely
multithreaded
:)
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:59 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
I would need to check on this one and probably investigate a bit more
what you are doing.
That being said, I can confirm only one thread is currently working.
So either jmeter is only sending monothreaded requests or there is
something else.
But you aren't doing multiple requests in parallel
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
We have
@Resource
WebServiceContext webserviceContext;
mistaken?
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:41 AM
TomEE Performance
Ah ok. Well I was asking if you were injecting when you took the
thread dump because from a server point of view I saw only one
thread working.
If you were using jmeter with multiple virtual users, I was
expecting to see more than one thread working.
I'll double check.
@Singleton is by default using Lock WRITE which prevents multiple
threads to access the singleton.
If you don't need synchronization or if your code is thread safe (no
involved. And again, there is no tuning to do
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 11:37 AM,

Post by f***@rohde-schwarz.com
Injecting? The jmeter was running during the threaddump. What do
you mean with injecting?
Is Singleton (Lock.READ) not even more of a bottleneck when it
comes to multiple concurrent requests? IIRC we tried Singleton
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:30 AM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Yes exactly.
Were you injecting anything when you took the jstack?
It seems that only one thread is working.
We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the jstack
that'd be great.
You can run `vmstat 5` on another terminal so you can see what's
going

on.

Post by f***@rohde-schwarz.com
You are using Stateless Session beans. Any reason you aren't using
a plain singleton?
With Singleton (Lock.READ) there is no pooling involved, so you
don't have anything to configure or tune.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 10:53 AM,

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409d
e7c4

Post by f***@rohde-schwarz.com

Post by f***@rohde-schwarz.com
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when
you are over the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro http://www.tomitribe.com
On Tue, Sep 4, 2018 at 8:27 AM,

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our
application (TomEE
Until 4 core VMs (or docker containers) we see a linear
increase in performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more
performance of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40
and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to
thread count, max to 1000) and stateful bean pool settings and
also with jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around
with in TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

Jean-Louis Monteiro

2018-09-04 12:26:17 UTC

Permalink

You are welcome.

--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hi Romain,
thank you so much for your detailed feedback. After I created the second
threaddump I had a look into it myself and came to the same conclusions. We
will investigate further, and if there are any more TomEE related
performance issues, I will get back in contact.
Thanks all for the replies!
Best
Fabian
-----Original Message-----
Sent: Tuesday, September 04, 2018 2:08 PM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Hi Fabian,
1. Ensure to test with securerandom.source=/dev/./urandom since you use a
lot of crypto 2. You use hsqldb which is known to not scale very well with
the concurrency, maybe switch to another database with a correctly
configured connection pool 3. You use DataBaseRealm which does a lookup of
the connection for each authentication which is synchronized and has no
cache on the password hashes so it can be slow at runtime
Personally i would start by using a fast realm (even if it always says
"ok") to validate this hypothesis before investigating other cases.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog <
https://rmannibucau.metawerx.net/> | Old Blog <
http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau>
| LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <
https://www.packtpub.com/application-development/java-
ee-8-high-performance>

Post by f***@rohde-schwarz.com
We have
@Resource
WebServiceContext webserviceContext;
mistaken?
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:41 AM
TomEE Performance
Ah ok. Well I was asking if you were injecting when you took the
thread dump because from a server point of view I saw only one
thread working.
If you were using jmeter with multiple virtual users, I was
expecting to see more than one thread working.
I'll double check.
@Singleton is by default using Lock WRITE which prevents multiple
threads to access the singleton.
If you don't need synchronization or if your code is thread safe (no
involved. And again, there is no tuning to do
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 11:37 AM,

@Stateless...

Post by f***@rohde-schwarz.com

Post by f***@rohde-schwarz.com
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:30 AM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Yes exactly.
Were you injecting anything when you took the jstack?
It seems that only one thread is working.
We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the jstack
that'd be great.
You can run `vmstat 5` on another terminal so you can see what's
going

on.

Post by f***@rohde-schwarz.com
You are using Stateless Session beans. Any reason you aren't using
a plain singleton?
With Singleton (Lock.READ) there is no pooling involved, so you
don't have anything to configure or tune.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 10:53 AM,

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409d
e7c4

Post by f***@rohde-schwarz.com

Post by f***@rohde-schwarz.com
-----Original Message-----
Sent: Tuesday, September 04, 2018 8:38 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
Hi,
Around SOAP, there are a couple of possible optimizations.
What would be helpful is to get into the docker container when
you are over the linear zone and get a jstack of the tomee process.
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro http://www.tomitribe.com
On Tue, Sep 4, 2018 at 8:27 AM,

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our
application (TomEE
Until 4 core VMs (or docker containers) we see a linear
increase in performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more
performance of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40
and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to
thread count, max to 1000) and stateful bean pool settings and
also with jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around
with in TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

Jonathan S. Fisher

2018-09-08 16:16:14 UTC

Permalink

Hey one more note, if you're on a virtualized Linux system and having the
random pool as a bottleneck, take a look at http://www.issihosts.com/haveged.
This is a pretty simple way to increase the entropy pool system-wide and
has solved several issues for me in the past. Be sure to read the fine
print of course for your particular business case.

Cheers!

Post by Jean-Louis Monteiro
You are welcome.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hi Romain,
thank you so much for your detailed feedback. After I created the second
threaddump I had a look into it myself and came to the same conclusions.

Post by f***@rohde-schwarz.com
will investigate further, and if there are any more TomEE related
performance issues, I will get back in contact.
Thanks all for the replies!
Best
Fabian
-----Original Message-----
Sent: Tuesday, September 04, 2018 2:08 PM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Hi Fabian,
1. Ensure to test with securerandom.source=/dev/./urandom since you use a
lot of crypto 2. You use hsqldb which is known to not scale very well

with

Post by f***@rohde-schwarz.com
the concurrency, maybe switch to another database with a correctly
configured connection pool 3. You use DataBaseRealm which does a lookup

Post by f***@rohde-schwarz.com
the connection for each authentication which is synchronized and has no
cache on the password hashes so it can be slow at runtime
Personally i would start by using a fast realm (even if it always says
"ok") to validate this hypothesis before investigating other cases.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog <
https://rmannibucau.metawerx.net/> | Old Blog <
http://rmannibucau.wordpress.com> | Github <

https://github.com/rmannibucau>

Post by f***@rohde-schwarz.com
| LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <
https://www.packtpub.com/application-development/java-
ee-8-high-performance>

Post by f***@rohde-schwarz.com
We have
@Resource
WebServiceContext webserviceContext;
mistaken?
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:41 AM
TomEE Performance
Ah ok. Well I was asking if you were injecting when you took the
thread dump because from a server point of view I saw only one
thread working.
If you were using jmeter with multiple virtual users, I was
expecting to see more than one thread working.
I'll double check.
@Singleton is by default using Lock WRITE which prevents multiple
threads to access the singleton.
If you don't need synchronization or if your code is thread safe (no
involved. And again, there is no tuning to do
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 11:37 AM,

@Stateless...

Post by f***@rohde-schwarz.com

on.

Post by f***@rohde-schwarz.com
You are using Stateless Session beans. Any reason you aren't using
a plain singleton?
With Singleton (Lock.READ) there is no pooling involved, so you
don't have anything to configure or tune.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 10:53 AM,

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd409d
e7c4

Post by f***@rohde-schwarz.com

process.

Post by f***@rohde-schwarz.com

Post by f***@rohde-schwarz.com
Post it here or in gist and put the link here.
Jean-Louis
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro http://www.tomitribe.com
On Tue, Sep 4, 2018 at 8:27 AM,

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our
application (TomEE
Until 4 core VMs (or docker containers) we see a linear
increase in performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more
performance of the system.
We used jmeter based load tests (SOAP calls) with 10/20/30/40
and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to
thread count, max to 1000) and stateful bean pool settings and
also with jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy around
with in TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

--
Jonathan | ***@gmail.com
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half
full.
Engineers, of course, understand the glass is twice as big as it needs to
be.

f***@rohde-schwarz.com

2018-09-10 06:40:03 UTC

Permalink

Thx Jonathan, but we are using haveged since 2 months into development :)

-----Original Message-----
From: Jonathan S. Fisher <***@gmail.com>
Sent: Saturday, September 08, 2018 6:16 PM
To: ***@tomee.apache.org
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: TomEE Performance

Hey one more note, if you're on a virtualized Linux system and having the random pool as a bottleneck, take a look at http://www.issihosts.com/haveged.
This is a pretty simple way to increase the entropy pool system-wide and has solved several issues for me in the past. Be sure to read the fine print of course for your particular business case.

Cheers!

Post by Jean-Louis Monteiro
You are welcome.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com

Post by f***@rohde-schwarz.com
Hi Romain,
thank you so much for your detailed feedback. After I created the
second threaddump I had a look into it myself and came to the same conclusions.

Post by f***@rohde-schwarz.com
will investigate further, and if there are any more TomEE related
performance issues, I will get back in contact.
Thanks all for the replies!
Best
Fabian
-----Original Message-----
Sent: Tuesday, September 04, 2018 2:08 PM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE Performance
Hi Fabian,
1. Ensure to test with securerandom.source=/dev/./urandom since you
use a lot of crypto 2. You use hsqldb which is known to not scale
very well

with

Post by f***@rohde-schwarz.com
the concurrency, maybe switch to another database with a correctly
configured connection pool 3. You use DataBaseRealm which does a
lookup

Post by f***@rohde-schwarz.com
the connection for each authentication which is synchronized and has
no cache on the password hashes so it can be slow at runtime
Personally i would start by using a fast realm (even if it always
says
"ok") to validate this hypothesis before investigating other cases.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog <
https://rmannibucau.metawerx.net/> | Old Blog <
http://rmannibucau.wordpress.com> | Github <

https://github.com/rmannibucau>

Post by f***@rohde-schwarz.com
| LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <
https://www.packtpub.com/application-development/java-
ee-8-high-performance>

Post by f***@rohde-schwarz.com
You were right, my student did something wrong with the first
https://gist.github.com/TrustedGate/83781f05950f245a3bfd2388c5bfd7
ab
Also at the bottom there is a vmstat result, its definitely
multithreaded
:)
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:59 AM
Subject: *EXT* [Newsletter] Re: TomEE Performance
I would need to check on this one and probably investigate a bit
more what you are doing.
That being said, I can confirm only one thread is currently working.
So either jmeter is only sending monothreaded requests or there is
something else.
But you aren't doing multiple requests in parallel
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro
http://www.tomitribe.com
On Tue, Sep 4, 2018 at 11:53 AM,

Post by f***@rohde-schwarz.com
We have
@Resource
WebServiceContext webserviceContext;
am I mistaken?
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:41 AM
TomEE Performance
Ah ok. Well I was asking if you were injecting when you took the
thread dump because from a server point of view I saw only one
thread working.
If you were using jmeter with multiple virtual users, I was
expecting to see more than one thread working.
I'll double check.
@Singleton is by default using Lock WRITE which prevents
multiple threads to access the singleton.
If you don't need synchronization or if your code is thread safe
no pool involved. And again, there is no tuning to do
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro http://www.tomitribe.com
On Tue, Sep 4, 2018 at 11:37 AM,

@Stateless...

Post by f***@rohde-schwarz.com

Post by f***@rohde-schwarz.com
-----Original Message-----
Sent: Tuesday, September 04, 2018 11:30 AM
Subject: *EXT* [Newsletter] Re: [Newsletter] Re: TomEE
Performance
Yes exactly.
Were you injecting anything when you took the jstack?
It seems that only one thread is working.
We would need you to do it when you are injecting.
If you could also give us the CPU usage when you take the
jstack that'd be great.
You can run `vmstat 5` on another terminal so you can see
what's going

on.

Post by f***@rohde-schwarz.com
You are using Stateless Session beans. Any reason you aren't
using a plain singleton?
With Singleton (Lock.READ) there is no pooling involved, so
you don't have anything to configure or tune.
--
Jean-Louis Monteiro
http://twitter.com/jlouismonteiro http://www.tomitribe.com
On Tue, Sep 4, 2018 at 10:53 AM,

Post by f***@rohde-schwarz.com
https://gist.github.com/TrustedGate/f670c079088404f42d69aabd
409d
e7c4

Post by f***@rohde-schwarz.com

process.

Post by f***@rohde-schwarz.com

Post by f***@rohde-schwarz.com
Hey,
we have been running some performance tests with our
application (TomEE
Until 4 core VMs (or docker containers) we see a linear
increase in performance, which is great and was anticipated.
But after 4 cores, we barely get 10% (with 8 cores) more
performance of the system.
We used jmeter based load tests (SOAP calls) with
10/20/30/40 and
100 threads, VM to VM via 1gbit, and 4 GB of RAM.
We played around with session bean pool sizes (min set to
thread count, max to 1000) and stateful bean pool settings
and also with jvm heap size and GC parameters, to no avail.
Are there any more performance parameters we can toy
around with in TomEE or Tomcat that you can recommend?
Thank you and best
Fabian

--
Jonathan | ***@gmail.com
Pessimists, see a jar as half empty. Optimists, in contrast, see it as half full.
Engineers, of course, understand the glass is twice as big as it needs to be.