ActiveMQ Connection Failures (Random to Constant)

[d***@letlocal.net]

Hi everyone,

I'm new to the forum and have been searching up and down trying new things
but am at a loss as to what my issues core problem is at this time.

I can see when TomEE starts that things are connecting and then continuous
failures. I've read a few articles in regards to network,

Here is what I'm getting out of the Catalina Log and everything else is just
kind of hung loading the application were trying to run.

Jul 10, 2016 12:50:39 PM
org.apache.activemq.transport.failover.FailoverTransport
handleTransportFailure
WARNING: Transport (tcp://TomEE1.fqdn/192.168.1.10:***@58413) failed,
attempting to automatically reconnect
java.io.IOException: Wire format negotiation timeout: peer did not send his
wire format.
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:98)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:665)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1409)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1521)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:329)
at
org.apache.activemq.ActiveMQConnection.createTopicSession(ActiveMQConnection.java:1149)
at
net.sf.ehcache.distribution.jms.JMSCacheManagerPeerProvider.init(JMSCacheManagerPeerProvider.java:161)
at net.sf.ehcache.CacheManager.init(CacheManager.java:241)
at net.sf.ehcache.CacheManager.<init>(CacheManager.java:169)
at net.sf.ehcache.CacheManager.create(CacheManager.java:454)
at
ixos.sec.um.types.PrincipalRepository.<init>(PrincipalRepository.java:215)
at
ixos.sec.um.types.PrincipalRepository.getInstance(PrincipalRepository.java:388)
at ixos.sec.um.types.UMPrincipalFactory.<init>(UMPrincipalFactory.java:65)
at
ixos.sec.um.types.UMPrincipalFactory.getInstance(UMPrincipalFactory.java:84)
at ixos.sec.um.UserManagement.getInstance(UserManagement.java:131)
at ixos.ums.web.RPC.startUpUMS(RPC.java:930)
at ixos.ums.web.RPC.init(RPC.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1284)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1197)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1087)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5266)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5554)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)

After several duplicate errors I can see the following error for a few hours
than things start on their own.

Jul 10, 2016 1:16:51 PM
org.apache.activemq.transport.failover.FailoverTransport doReconnect
WARNING: Failed to connect to [tcp://TomEE1.fqdn:61616,
tcp://TomEE2.fqdn:61616, tcp://TomEE3.fqdn:61616] after: 20 attempt(s)
continuing to retry.

I've tried changing the timeout on TomEE.xml for Failover like this
unsuccessfully:
<Resource id="JmsResourceAdapter" type="ActiveMQResourceAdapter">
BrokerXmlConfig
broker:(tcp://TomEE1.fqdn:61616)?persistent=true&amp;useJmx=false
ServerUrl =
failover:(tcp://TomEEfqdn1:61616,tcp://TomEEfqdn2:61616,tcp://TomEE3.fqdn:61616)?randomize=false
DataSource JmsDS
</Resource>

At this point I'm not sure what finally allows everything to start working
hours later but it does. The one option I thought might resolve the issue
did not and removing clustering and running on a single node presented the
same issue.

This is the change I made to the JmsREsourceAdapter:
failover:(tcp://broker1:61616?wireFormat.maxInactivityDuration=1000,tcp://broker2:61616?wireFormat.maxInactivityDuration=1000,tcp://broker3:61616?wireFormat.maxInactivityDuration=1000)

Any ideas would be sincerely appreciated.

Thanks In Advance

Don




--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[Romain Manni-Bucau]

Hi

Can you try activating activemq debug logs on all instances please?

Post by d***@letlocal.net
Hi everyone,
I'm new to the forum and have been searching up and down trying new things
but am at a loss as to what my issues core problem is at this time.
I can see when TomEE starts that things are connecting and then continuous
failures. I've read a few articles in regards to network,
Here is what I'm getting out of the Catalina Log and everything else is just
kind of hung loading the application were trying to run.
Jul 10, 2016 12:50:39 PM
org.apache.activemq.transport.failover.FailoverTransport
handleTransportFailure
attempting to automatically reconnect
java.io.IOException: Wire format negotiation timeout: peer did not send his
wire format.
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:98)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:665)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1409)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1521)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:329)
at
org.apache.activemq.ActiveMQConnection.createTopicSession(ActiveMQConnection.java:1149)
at
net.sf.ehcache.distribution.jms.JMSCacheManagerPeerProvider.init(JMSCacheManagerPeerProvider.java:161)
at net.sf.ehcache.CacheManager.init(CacheManager.java:241)
at net.sf.ehcache.CacheManager.<init>(CacheManager.java:169)
at net.sf.ehcache.CacheManager.create(CacheManager.java:454)
at
ixos.sec.um.types.PrincipalRepository.<init>(PrincipalRepository.java:215)
at
ixos.sec.um.types.PrincipalRepository.getInstance(PrincipalRepository.java:388)
at
ixos.sec.um.types.UMPrincipalFactory.<init>(UMPrincipalFactory.java:65)
at
ixos.sec.um.types.UMPrincipalFactory.getInstance(UMPrincipalFactory.java:84)
at ixos.sec.um.UserManagement.getInstance(UserManagement.java:131)
at ixos.ums.web.RPC.startUpUMS(RPC.java:930)
at ixos.ums.web.RPC.init(RPC.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1284)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1197)
at
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1087)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5266)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5554)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
After several duplicate errors I can see the following error for a few hours
than things start on their own.
Jul 10, 2016 1:16:51 PM
org.apache.activemq.transport.failover.FailoverTransport doReconnect
WARNING: Failed to connect to [tcp://TomEE1.fqdn:61616,
tcp://TomEE2.fqdn:61616, tcp://TomEE3.fqdn:61616] after: 20 attempt(s)
continuing to retry.
I've tried changing the timeout on TomEE.xml for Failover like this
<Resource id="JmsResourceAdapter" type="ActiveMQResourceAdapter">
BrokerXmlConfig
broker:(tcp://TomEE1.fqdn:61616)?persistent=true&amp;useJmx=false
ServerUrl =
failover:(tcp://TomEEfqdn1:61616,tcp://TomEEfqdn2:61616,tcp://TomEE3.fqdn:61616)?randomize=false
DataSource JmsDS
</Resource>
At this point I'm not sure what finally allows everything to start working
hours later but it does. The one option I thought might resolve the issue
did not and removing clustering and running on a single node presented the
same issue.
failover:(tcp://broker1:61616?wireFormat.maxInactivityDuration=1000,tcp://broker2:61616?wireFormat.maxInactivityDuration=1000,tcp://broker3:61616?wireFormat.maxInactivityDuration=1000)
Any ideas would be sincerely appreciated.
Thanks In Advance
Don
--
http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[d***@letlocal.net]

I didn't see any difference in the logging. Just seems to hang in the same
manner.

Jul 10, 2016 9:06:02 PM
org.apache.activemq.transport.failover.FailoverTransport
handleTransportFailure
WARNING: Transport (tcp://TomEE2.fqdn/192.168.1.2:***@51178) failed,
attempting to automatically reconnect
java.io.IOException: Wire format negotiation timeout: peer did not send his
wire format.
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:98)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:665)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1409)
at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1521)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:329)
at
org.apache.activemq.ActiveMQConnection.createTopicSession(ActiveMQConnection.java:1149)
at
net.sf.ehcache.distribution.jms.JMSCacheManagerPeerProvider.init(JMSCacheManagerPeerProvider.java:161)
at net.sf.ehcache.CacheManager.init(CacheManager.java:241)
at net.sf.ehcache.CacheManager.<init>(CacheManager.java:169)
at net.sf.ehcache.CacheManager.create(CacheManager.java:454)
at
ixos.sec.um.types.PrincipalRepository.<init>(PrincipalRepository.java:215)
at
ixos.sec.um.types.PrincipalRepository.getInstance(PrincipalRepository.java:388)
at ixos.sec.um.types.UMPrincipalFactory.<init>(UMPrincipalFactory.java:65)
at
ixos.sec.um.types.UMPrincipalFactory.getInstance(UMPrincipalFactory.java:84)
at ixos.sec.um.UserManagement.getInstance(UserManagement.java:131)
at ixos.ums.web.RPC.startUpUMS(RPC.java:930)
at ixos.ums.web.RPC.init(RPC.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1284)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1197)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1087)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5266)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5554)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

and then these events occur until the system finally starts.

Jul 10, 2016 8:57:59 PM
org.apache.activemq.transport.failover.FailoverTransport doReconnect
WARNING: Failed to connect to [tcp://TomEE1.fqdn:61616,
tcp://TomEE2.fqdn:61616] after: 10 attempt(s) continuing to retry.




--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679273.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[d***@letlocal.net]

The only other interesting events I've found were that it does connect at
start and that it begins failing but these are slightly different events I
found searching through the logs.

Jul 10, 2016 9:24:49 PM
org.apache.openejb.resource.activemq.ActiveMQ5Factory createBroker
WARNING: ActiveMQ failed to start broker within 10000 seconds - It may be
unusable

Jul 10, 2016 9:25:01 PM
org.apache.activemq.transport.failover.FailoverTransport doReconnect
INFO: Successfully connected to tcp://Tomee1.fqdn:61616
Jul 10, 2016 9:25:01 PM
org.apache.activemq.transport.failover.FailoverTransport doReconnect
INFO: Successfully connected to tcp://Tomee1.fqdn:61616
Jul 10, 2016 9:25:02 PM
org.apache.activemq.transport.failover.FailoverTransport
handleTransportFailure


WARNING: Transport (tcp://Tomee1.fqdn/192.168.1.1:***@51179) failed,
attempting to automatically reconnect
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:609)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:594)
at java.io.DataInputStream.readInt(Unknown Source)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:258)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Unknown Source)



--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679274.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[Romain Manni-Bucau]

Interesting. I'm surprised you don't get more logs in debug mode but uf the
broker(s) can't start it can explain the issue.

Maybe do a jstack on the jvm when the broker tries to start and fails.

Post by d***@letlocal.net
The only other interesting events I've found were that it does connect at
start and that it begins failing but these are slightly different events I
found searching through the logs.
Jul 10, 2016 9:24:49 PM
org.apache.openejb.resource.activemq.ActiveMQ5Factory createBroker
WARNING: ActiveMQ failed to start broker within 10000 seconds - It may be
unusable
Jul 10, 2016 9:25:01 PM
org.apache.activemq.transport.failover.FailoverTransport doReconnect
INFO: Successfully connected to tcp://Tomee1.fqdn:61616
Jul 10, 2016 9:25:01 PM
org.apache.activemq.transport.failover.FailoverTransport doReconnect
INFO: Successfully connected to tcp://Tomee1.fqdn:61616
Jul 10, 2016 9:25:02 PM
org.apache.activemq.transport.failover.FailoverTransport
handleTransportFailure
attempting to automatically reconnect
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:609)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:594)
at java.io.DataInputStream.readInt(Unknown Source)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:258)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Unknown Source)
--
http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679274.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[d***@letlocal.net]

So the one thing that seems consistent is that activeMQ doesn't seem to
connect and then just all the sudden starts connecting.

Jul 11, 2016 12:43:42 AM org.apache.activemq.ra.ActiveMQEndpointWorker$1 run
INFO: Successfully established connection to broker
[failover:(tcp://omaedcotx092.na.corp.ipgnetwork.com:61616,tcp://omaedcotx093.na.corp.ipgnetwork.com:61616)?randomize=false]
Jul 11, 2016 12:43:45 AM
org.apache.activemq.transport.failover.FailoverTransport doReconnect

I found this article:
http://activemq.apache.org/javaxjmsjmsexception-wire-format-negociation-timeout-peer-did-not-send-his-wire-format.html

However, i'm unsure as to how I can implement this properly or where. I
attempted the recommended configuration in tomee.xml but it gave me an error
for an unknown property. Maybe this wasn't set properly as I assumed it was
set on the Broker not the failover config.

This is my current default setting that will eventually start with 2 - 3
hours of activeMQ Fails and then a sudden hit.

<Resource id="JmsResourceAdapter" type="ActiveMQResourceAdapter">
BrokerXmlConfig
broker:(tcp://TomEE1.fqdn:61616)?persistent=true&amp;useJmx=false
ServerUrl =
failover:(tcp://TomEE1.fqdn:61616,tcp://TomEE2.fqdn:61616)?randomize=false
DataSource JmsDS


</Resource>



--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679280.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[Romain Manni-Bucau]

maybe tune in the broker url the value startuptimeout


Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://blog-rmannibucau.rhcloud.com> | Old Wordpress Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com> | JavaEE Factory
<https://javaeefactory-rmannibucau.rhcloud.com>

Post by d***@letlocal.net
So the one thing that seems consistent is that activeMQ doesn't seem to
connect and then just all the sudden starts connecting.
Jul 11, 2016 12:43:42 AM org.apache.activemq.ra.ActiveMQEndpointWorker$1 run
INFO: Successfully established connection to broker
[failover:(tcp://omaedcotx092.na.corp.ipgnetwork.com:61616
,tcp://omaedcotx093.na.corp.ipgnetwork.com:61616)?randomize=false]
Jul 11, 2016 12:43:45 AM
org.apache.activemq.transport.failover.FailoverTransport doReconnect
http://activemq.apache.org/javaxjmsjmsexception-wire-format-negociation-timeout-peer-did-not-send-his-wire-format.html
However, i'm unsure as to how I can implement this properly or where. I
attempted the recommended configuration in tomee.xml but it gave me an error
for an unknown property. Maybe this wasn't set properly as I assumed it was
set on the Broker not the failover config.
This is my current default setting that will eventually start with 2 - 3
hours of activeMQ Fails and then a sudden hit.
<Resource id="JmsResourceAdapter" type="ActiveMQResourceAdapter">
BrokerXmlConfig
broker:(tcp://TomEE1.fqdn:61616)?persistent=true&amp;useJmx=false
ServerUrl =
failover:(tcp://TomEE1.fqdn:61616,tcp://TomEE2.fqdn:61616)?randomize=false
DataSource JmsDS
</Resource>
--
http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679280.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[d***@letlocal.net]

So for an unknown reason it seems that the activeMQ started on it's own
without issue.

However, again today the issue is re-occuring after a reboot. I do see one
message that seems to point to the Broker not starting. I've tried setting
the starup timeout to 30 seconds with no change.

Is their a way to force the Broker to start or observe the broker directly
for events or errors with starting? I did try DEBUG but didn't get much more
than network connection errors.



--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679293.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[Romain Manni-Bucau]

Did you do a thread dump during these 30s? (Using jstack for instance)

Post by d***@letlocal.net
So for an unknown reason it seems that the activeMQ started on it's own
without issue.
However, again today the issue is re-occuring after a reboot. I do see one
message that seems to point to the Broker not starting. I've tried setting
the starup timeout to 30 seconds with no change.
Is their a way to force the Broker to start or observe the broker directly
for events or errors with starting? I did try DEBUG but didn't get much more
than network connection errors.
--
http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679293.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[d***@letlocal.net]

I have not as of yet due to the box being so random but I'll grab that the
next time I have an opportunity.

How can I validate that the Broker is starting or attempting to start in
windows?

Also I did not indicate seconds with the last timeout. Would the following
setting have started in milliseconds or with the default 10 seconds?

startupTimeout = 10

I also found that the startup and failed connections seem to be on some kind
of timer as it always starts on the hour in which seems odd. So if the
connection fails to start at 1pm it will eventually connect on an hour
interval. So for example if it fails at 1:00 PM it might start at say 3:00PM
never 3:15 or 3:45.

- Don



--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679295.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[Romain Manni-Bucau]

Post by d***@letlocal.net
I have not as of yet due to the box being so random but I'll grab that the
next time I have an opportunity.
How can I validate that the Broker is starting or attempting to start in
windows?
Also I did not indicate seconds with the last timeout. Would the following
setting have started in milliseconds or with the default 10 seconds?
startupTimeout = 10

milliseconds

Post by d***@letlocal.net
I also found that the startup and failed connections seem to be on some kind
of timer as it always starts on the hour in which seems odd. So if the
connection fails to start at 1pm it will eventually connect on an hour
interval. So for example if it fails at 1:00 PM it might start at say 3:00PM
never 3:15 or 3:45.

jstack will answer that

Post by d***@letlocal.net
- Don
--
http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679295.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[d***@letlocal.net]

I still haven't had the opportunity toget the trace done as of yet.

However, I did observe a minHeap and MaxHeap for Java 7. Sorry about putting
you all on hold. I will get this done as the opportunity comes available.



--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679328.html
Sent from the TomEE Users mailing list archive at Nabble.com.

[d***@letlocal.net]

Iv'e resolved the issue.

It seems that one of clustered machines was getting sync'd to vmware only on
shutdown and startup and that time was in the past. I'm not sure why but the
ActiveMQ lock actually got set to 6 hours in the future and the lock wasn't
being removed on restart either (Root Issue).

Again I'm not sure why this was an issue as Syncing Time was not set with
the VMWARE Tools but re-installing the drivers resolved the issue
completely.

Again, we found a single time entry being off during shutdown and startup
only.

I appreciate the time and effort everyone provided and hopefully my chime in
with the details and solution will help someone else in the future

Enjoy

- Don



--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/ActiveMQ-Connection-Failures-Random-to-Constant-tp4679269p4679470.html
Sent from the TomEE Users mailing list archive at Nabble.com.