freezing on CentOS 7

[Matthew Broadhead]

i am currently stuck on Java release
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
on CentOS 7 with TomEE 7.0.3

updating CentOS to
Version     : 1.8.0.144
Release     : 0.b01.el7_4
results in random freezes with no error in catalina.out

back story: the system automatically updated to 1.8.0.144.  TomEE
started crashing.  looked like it was still running.  no output in
catalina.out.  kept restarting and it would run for a while but same
result.  rebooted whole server.  no change.  rolled back to 1.8.0.131
and blocked yum from updating.  everything works ok now.

[Jonathan Gallimore]

Do you have a test system you can run it on, and grab the output from
jstack and jmap when it happens? How long do the freezes last - does the
system ever recover? Do you see increases in active threads or CPU?

Many thanks

Jon

On Tue, Oct 10, 2017 at 9:52 AM, Matthew Broadhead <

Post by Matthew Broadhead
i am currently stuck on Java release
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
on CentOS 7 with TomEE 7.0.3
updating CentOS to
Version : 1.8.0.144
Release : 0.b01.el7_4
results in random freezes with no error in catalina.out
back story: the system automatically updated to 1.8.0.144. TomEE started
crashing. looked like it was still running. no output in catalina.out.
kept restarting and it would run for a while but same result. rebooted
whole server. no change. rolled back to 1.8.0.131 and blocked yum from
updating. everything works ok now.

[Matthew Broadhead]

I am not sure if it recovers because in production I was forced to
restart the system.
I can set it up on my local virtual box but without much load it might
be hard to get it to crash.  I will try it and let you know

Post by Jonathan Gallimore
Do you have a test system you can run it on, and grab the output from
jstack and jmap when it happens? How long do the freezes last - does the
system ever recover? Do you see increases in active threads or CPU?
Many thanks
Jon
On Tue, Oct 10, 2017 at 9:52 AM, Matthew Broadhead <

Post by Matthew Broadhead
i am currently stuck on Java release
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
on CentOS 7 with TomEE 7.0.3
updating CentOS to
Version : 1.8.0.144
Release : 0.b01.el7_4
results in random freezes with no error in catalina.out
back story: the system automatically updated to 1.8.0.144. TomEE started
crashing. looked like it was still running. no output in catalina.out.
kept restarting and it would run for a while but same result. rebooted
whole server. no change. rolled back to 1.8.0.131 and blocked yum from
updating. everything works ok now.

[Jonathan Gallimore]

Thanks Matthew - if we're able to reproduce in a safe environment, that
will be a big help to solve your problem. One other question - is this
specific to the JDK version or were there any other changes? I note you
posted on the 7.0.4 vote on dev@ - does this only happen on 7.0.4 or are
you seeing it with 7.0.3 as well?

Cheers

Jon

On Tue, Oct 10, 2017 at 10:06 AM, Matthew Broadhead <

I am not sure if it recovers because in production I was forced to restart
the system.
I can set it up on my local virtual box but without much load it might be
hard to get it to crash. I will try it and let you know

Post by Jonathan Gallimore
Do you have a test system you can run it on, and grab the output from
jstack and jmap when it happens? How long do the freezes last - does the
system ever recover? Do you see increases in active threads or CPU?
Many thanks
Jon
On Tue, Oct 10, 2017 at 9:52 AM, Matthew Broadhead <
i am currently stuck on Java release

Post by Matthew Broadhead
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
on CentOS 7 with TomEE 7.0.3
updating CentOS to
Version : 1.8.0.144
Release : 0.b01.el7_4
results in random freezes with no error in catalina.out
back story: the system automatically updated to 1.8.0.144. TomEE started
crashing. looked like it was still running. no output in catalina.out.
kept restarting and it would run for a while but same result. rebooted
whole server. no change. rolled back to 1.8.0.131 and blocked yum from
updating. everything works ok now.

[Matthew Broadhead]

this was only on 7.0.3 in production.  only happens if i upgrade from
131 to 144

Post by Jonathan Gallimore
Thanks Matthew - if we're able to reproduce in a safe environment, that
will be a big help to solve your problem. One other question - is this
specific to the JDK version or were there any other changes? I note you
you seeing it with 7.0.3 as well?
Cheers
Jon
On Tue, Oct 10, 2017 at 10:06 AM, Matthew Broadhead <

I am not sure if it recovers because in production I was forced to restart
the system.
I can set it up on my local virtual box but without much load it might be
hard to get it to crash. I will try it and let you know

Post by Jonathan Gallimore
Do you have a test system you can run it on, and grab the output from
jstack and jmap when it happens? How long do the freezes last - does the
system ever recover? Do you see increases in active threads or CPU?
Many thanks
Jon
On Tue, Oct 10, 2017 at 9:52 AM, Matthew Broadhead <
i am currently stuck on Java release

Post by Matthew Broadhead
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
on CentOS 7 with TomEE 7.0.3
updating CentOS to
Version : 1.8.0.144
Release : 0.b01.el7_4
results in random freezes with no error in catalina.out
back story: the system automatically updated to 1.8.0.144. TomEE started
crashing. looked like it was still running. no output in catalina.out.
kept restarting and it would run for a while but same result. rebooted
whole server. no change. rolled back to 1.8.0.131 and blocked yum from
updating. everything works ok now.

[Matthew Broadhead]

i just allowed CentOS 7 to update to
Name        : java-1.8.0-openjdk
Arch        : x86_64
Epoch       : 1
Version     : 1.8.0.151
Release     : 1.b12.el7_4
Size        : 241 k
last night.  everything seems to be running ok with that version so
far.  i will let you know if there are any problems

Post by Matthew Broadhead
this was only on 7.0.3 in production.  only happens if i upgrade from
131 to 144

Post by Jonathan Gallimore
Thanks Matthew - if we're able to reproduce in a safe environment, that
will be a big help to solve your problem. One other question - is this
specific to the JDK version or were there any other changes? I note you
you seeing it with 7.0.3 as well?
Cheers
Jon
On Tue, Oct 10, 2017 at 10:06 AM, Matthew Broadhead <

I am not sure if it recovers because in production I was forced to restart
the system.
I can set it up on my local virtual box but without much load it might be
hard to get it to crash.  I will try it and let you know

Post by Jonathan Gallimore
Do you have a test system you can run it on, and grab the output from
jstack and jmap when it happens? How long do the freezes last - does the
system ever recover? Do you see increases in active threads or CPU?
Many thanks
Jon
On Tue, Oct 10, 2017 at 9:52 AM, Matthew Broadhead <
i am currently stuck on Java release

Post by Matthew Broadhead
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
on CentOS 7 with TomEE 7.0.3
updating CentOS to
Version     : 1.8.0.144
Release     : 0.b01.el7_4
results in random freezes with no error in catalina.out
back story: the system automatically updated to 1.8.0.144.  TomEE
started
crashing.  looked like it was still running.  no output in
catalina.out.
kept restarting and it would run for a while but same result. 
rebooted
whole server.  no change.  rolled back to 1.8.0.131 and blocked
yum from
updating.  everything works ok now.

[Matthew Broadhead]

to follow up i have not had any more freezing since updating to
1.8.0.151 (skipping 144)

Post by Matthew Broadhead
i just allowed CentOS 7 to update to
Name        : java-1.8.0-openjdk
Arch        : x86_64
Epoch       : 1
Version     : 1.8.0.151
Release     : 1.b12.el7_4
Size        : 241 k
last night.  everything seems to be running ok with that version so
far.  i will let you know if there are any problems

this was only on 7.0.3 in production. only happens if i upgrade from
131 to 144

Post by Jonathan Gallimore
Thanks Matthew - if we're able to reproduce in a safe environment, that
will be a big help to solve your problem. One other question - is this
specific to the JDK version or were there any other changes? I note you
you seeing it with 7.0.3 as well?
Cheers
Jon
On Tue, Oct 10, 2017 at 10:06 AM, Matthew Broadhead <

I am not sure if it recovers because in production I was forced to restart
the system.
I can set it up on my local virtual box but without much load it might be
hard to get it to crash.  I will try it and let you know

Post by Jonathan Gallimore
Do you have a test system you can run it on, and grab the output from
jstack and jmap when it happens? How long do the freezes last - does the
system ever recover? Do you see increases in active threads or CPU?
Many thanks
Jon
On Tue, Oct 10, 2017 at 9:52 AM, Matthew Broadhead <
i am currently stuck on Java release

Post by Matthew Broadhead
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
on CentOS 7 with TomEE 7.0.3
updating CentOS to
Version     : 1.8.0.144
Release     : 0.b01.el7_4
results in random freezes with no error in catalina.out
back story: the system automatically updated to 1.8.0.144.  TomEE
started
crashing.  looked like it was still running.  no output in
catalina.out.
kept restarting and it would run for a while but same result. 
rebooted
whole server.  no change.  rolled back to 1.8.0.131 and blocked
yum from
updating.  everything works ok now.

[Jonathan Gallimore]

Fantastic news. Thanks for following up, and I'm glad things are looking ok.

I'd still be interested in figuring out what was going on if you're able to
tell us a bit more about your app so we can try and reproduce.

Jon

to follow up i have not had any more freezing since updating to 1.8.0.151
(skipping 144)

Post by Matthew Broadhead
i just allowed CentOS 7 to update to
Name : java-1.8.0-openjdk
Arch : x86_64
Epoch : 1
Version : 1.8.0.151
Release : 1.b12.el7_4
Size : 241 k
last night. everything seems to be running ok with that version so far.
i will let you know if there are any problems

this was only on 7.0.3 in production. only happens if i upgrade from 131
to 144

Post by Jonathan Gallimore
Thanks Matthew - if we're able to reproduce in a safe environment, that
will be a big help to solve your problem. One other question - is this
specific to the JDK version or were there any other changes? I note you
you seeing it with 7.0.3 as well?
Cheers
Jon
On Tue, Oct 10, 2017 at 10:06 AM, Matthew Broadhead <
I am not sure if it recovers because in production I was forced to

Post by Matthew Broadhead
restart
the system.
I can set it up on my local virtual box but without much load it might be
hard to get it to crash. I will try it and let you know
Do you have a test system you can run it on, and grab the output from

Post by Jonathan Gallimore
jstack and jmap when it happens? How long do the freezes last - does the
system ever recover? Do you see increases in active threads or CPU?
Many thanks
Jon
On Tue, Oct 10, 2017 at 9:52 AM, Matthew Broadhead <
i am currently stuck on Java release

Post by Matthew Broadhead
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
on CentOS 7 with TomEE 7.0.3
updating CentOS to
Version : 1.8.0.144
Release : 0.b01.el7_4
results in random freezes with no error in catalina.out
back story: the system automatically updated to 1.8.0.144. TomEE started
crashing. looked like it was still running. no output in catalina.out.
kept restarting and it would run for a while but same result.
rebooted
whole server. no change. rolled back to 1.8.0.131 and blocked yum from
updating. everything works ok now.