remote site TServer losing connections to URS --intermittently

[ademince]

Dear all,

We have HA environment and the remote Tserver losing connection to URS intermittently and around 100-150 milliseconds. Naturally this is dropping all the calls to default.

Here is URS log file:

ps. Primary URS located at site A = no issues reported at site A
Tserver at site B losing connection to primary URS up to 150 milliseconds at a time therefore defaulting the calls.

I would appreciate if anyone has any idea as to why this would happen looking at the log file.

EventServerDisconnected is received for tserver TServer_Melbourne
15:30:45.542 Std 04504 Connection to TServer 'TServer_Melbourne' at host 'AASMEAP25', port 3000 lost
15:30:45.542_G_I_ [14:12] server TServer_Melbourne is being closed, socket 65209
15:30:45.636_T_W_ [14:13] calls for tserver TServer_Melbourne[Melbourne-Alcatel] will be deleted
15:30:45.636_M_I_008201bda920bf90 [17:11] VQ 00e3bbb0 first available call: 008201bda920bf90, reason=treatment
15:30:45.636_M_I_008201bda920bf90 [17:11] VQ 00e7a7f0 first available call: 008201bda920bf90, reason=treatment
    _T_I_008201bda920bf90 [14:02] event EventDiverted for virtual queue VQ_CARE_SUPER_ACCT_DETAILS is pended
    _C_W_008201bda920bf90 [14:02] can not find tserver for virtual queue varVirtualQueue for tenant Resources(101), virtual queue support is ignored
15:30:45.636_M_I_008201bda920bf90 [17:11] VQ 00e3bbb0 first available call: none, reason=binding
15:30:45.636_M_I_008201bda920bf90 [13:03] call (virtual queue 024e8020, id=18832, priority 35) doesn't wait for VQ 00e3bbb0 (name="varVirtualQueue") now
15:30:45.636_M_I_008201bda920bf90 [17:11] VQ 00e7a7f0 first available call: none, reason=binding
15:30:45.636_M_I_008201bda920bf90 [13:03] call (virtual queue 00eb90c0, id=18817, priority 35) doesn't wait for VQ 00e7a7f0 (name="VQ_CARE_SUPER_ACCT_DETAILS") now
15:30:45.636_I_I_008201bda920bf90 [01:08] call deleting truly
CALL (008201bda920bf90) HISTORY ( Dropped ):
15:30:45.636_I_I_008201bda920bf99 [01:08] call deleting truly
CALL (008201bda920bf99) HISTORY ( Dropped ):
15:30:45.636_I_I_008201bda920bf96 [01:08] call deleting truly
CALL (008201bda920bf96) HISTORY ( Dropped ):
15:30:45.636_I_I_008201bda920bf9e [01:08] call deleting truly
CALL (008201bda920bf9e) HISTORY ( Dropped ):
15:30:45.636_M_I_008201bda920bf9b [17:11] VQ 00e89f60 first available call: 008201bda920bf9b, reason=treatment
    _T_I_008201bda920bf9b [14:02] event EventDiverted for virtual queue VQ_TASPLAN is pended
15:30:45.636_M_I_008201bda920bf9b [17:11] VQ 00e89f60 first available call: none, reason=binding
15:30:45.636_M_I_008201bda920bf9b [13:03] call (virtual queue 020448f0, id=18852, priority 26) doesn't wait for VQ 00e89f60 (name="VQ_TASPLAN") now
15:30:45.636_I_I_008201bda920bf9b [01:08] call deleting truly
CALL (008201bda920bf9b) HISTORY ( Dropped ):
15:30:45.636_I_I_008201bda920bf9f [01:08] call deleting truly
CALL (008201bda920bf9f) HISTORY ( Dropped ):
15:30:45.636_I_I_008201bda920bfa1 [01:08] call deleting truly
CALL (008201bda920bfa1) HISTORY ( Dropped ):
15:30:45.636 Trc 04500 Connecting to TServer 'TServer_Melbourne' at host 'AASMEAP25', port 3000
15:30:45.636_G_I_ [14:11] try  X-connecting: ("name"="TServer_Melbourne","host"="AASMEAP25","is-server"="1","client-type"="15","server-type"="1","port"="3000","protocol"="addp","addp-timeout"="200","addp-remote-timeout"="200","addp-trace"="4","port_as_string"="3000")
addp-trace 4
(addp_xconfig) local 200000 msec, remote 200000 msec, trace both
15:30:45.636 Trc 04501 Server TServer_Melbourne contacted, socket 65210
15:30:45.636_M_I_ [10:17] STATOBJECT(0208cf60 337 2) tenant Resources name=15269@StatServer_URS.A: CHANGE OF STATE (WaitForNextCall->CallRinging)
    _M_I_ [10:37] statistic ATT for MEDIA voice (type=0), time=1256617845: busy 0=0+0
    _M_I_ [10:17] STATOBJECT(0208cf60 337 2) tenant=Resources name=15269@StatServer_URS.A: 0 ready DNs reported, dT=0
(addp-init) client side ADDP timeout=200000
-AI[t/o:200000,trace]->-512 @15:30:45.6520
-Ai[1152]-<-512 @15:30:45.6670
addp-trace 4
(addp_xconfig) local 200000 msec, remote 200000 msec, trace both
15:30:45.667_T_I_ [14:32] EventServerConnected is received for tserver TServer_Melbourne
received from 65210(TServer_Melbourne)AASMEAP25:3000(fd=) message EventLinkConnected
AttributeTimeinSecs 1256617845 (15:30:45)
AttributeTimeinuSecs 141000
AttributeServerStartTime 4ae0f302000beac8 (11:04:18.781000)
AttributeCustomerID 'Resources'
AttributeEventSequenceNumber 000000000007b1e6
AttributeRegistrationCode 0
AttributeUserData [37] 00 02 00 00..
'host-0' 'AASMEAP22'
'port-0' 3000
AttributeSessionID 8521606
AttributeApplicationName 'TServer_Melbourne'
15:30:45.667 Std 04503 Connected to TServer 'TServer_Melbourne' at host 'AASMEAP25', port 3000

[borkokrz]

Never use the same value for local and remote timeout in addp connection parameters. The remote timeout should be longer then the local timeout. Did you test ping between 2 sites ? Maybe something is wrong with connection between 2 sites ?

Earlier part of log with addp trace would be helpful in investigation.

[Fra]

It is not really related to your issue, but 200 seconds as timeout seems to me way big.

F

[Timur Karimov]

I don't think what connect remote T-Server to local URS is a good idea. Why you don't place URS on the remote site ?

WBR Thaler

[ademince]

[quote author=borkokrz link=topic=4807.msg21540#msg21540 date=1256629074]
Never use the same value for local and remote timeout in addp connection parameters. The remote timeout should be longer then the local timeout. Did you test ping between 2 sites ? Maybe something is wrong with connection between 2 sites ?

Earlier part of log with addp trace would be helpful in investigation.
[/quote]

Hi borkokrz, we made both sites the same value as a test to find the drop out which did not help. The ping test was successful. Earlier part of the log simply shows the normal routing logs.

[ademince]

[quote author=Timur Karimov link=topic=4807.msg21544#msg21544 date=1256646801]
I don't think what connect remote T-Server to local URS is a good idea. Why you don't place URS on the remote site ?

WBR Thaler

[/quote]

Hi Timur, we do have URS at remote site however its our back since we have HA environment. However when we make remote URS to be primary than the local Tserver is losing connection.

[ecki]

That sounds like you have pretty bad network issues. Have you tried to monitor the network? The easiest way would be you run infinite ping on one of the affected host  e.g. "ping <..remote host..> -t > some_logfile.log" . The best of course would be to run full network monitoring on each node involved in the path, but as common practice dictates, the network guys do blame others unless you prove they are wrong  

[cavagnaro]

a simple ping can be sometimes untrustable, increase the packet size (ping -l 800) so you really make an effort to that server

[Timur Karimov]

[quote author=ademince link=topic=4807.msg21559#msg21559 date=1256687629]
Hi Timur, we do have URS at remote site however its our back since we have HA environment. However when we make remote URS to be primary than the local Tserver is losing connection.
[/quote]

Hi there!

Did you mean what you try to create geographical distributed HA pair of you services ? And on site B you have backup designated URS for primary URS on site A? What about TServer ?

[ademince]

Hi Timur,

We do have local TServer and Tserver HA's (located on the same location) however the URS is on site A and URS_HA is on site B

cheers

[Timur Karimov]

[quote author=ademince link=topic=4807.msg21595#msg21595 date=1256865228]
Hi Timur,
We do have local TServer and Tserver HA's (located on the same location) however the URS is on site A and URS_HA is on site B
[/quote]

You answer throw me into confusion   I mean official Genesys doc said - we did not support geographical HA. But in spite of this - Why you did it? what you wanna to achieve  ?!

WBR Thaler

[ademince]

[quote author=ecki link=topic=4807.msg21560#msg21560 date=1256694573]
That sounds like you have pretty bad network issues. Have you tried to monitor the network? The easiest way would be you run infinite ping on one of the affected host  e.g. "ping <..remote host..> -t > some_logfile.log" . The best of course would be to run full network monitoring on each node involved in the path, but as common practice dictates, the network guys do blame others unless you prove they are wrong  
[/quote]

Hi Ecki,

Our network guys did a Traffic Capture (traces) and found that the TServer is sending a TCP RST bit to URS when there was no network congestion or TCP drop outs.

Now we need to find out what is causing the Tserver to send TCP RST bit to URS....any idea?

[ecki]

Sorry mate, do not know. Can you see something suspicious in the TS logs? Could be that the TCP layer received too many erroneous packets. It is probably time to raise SR. How far are those servers from each other? Are there any substantial delays in the network observed?

[Fra]

[quote author=ademince link=topic=4807.msg21650#msg21650 date=1257209483]

Our network guys did a Traffic Capture (traces) and found that the TServer is sending a TCP RST bit to URS when there was no network congestion or TCP drop outs.

Now we need to find out what is causing the Tserver to send TCP RST bit to URS....any idea?
[/quote]

I bet 2 cents on the fact it's a firewall issue.

Fra

[ademince]

RESOLVED................!!!!!!!!!!!!!!!

Dear All,

For the last 14 months we have upgraded the network, upgraded the Genesys applications (versions), increased the RAM on Tservers, changed the connection settings and none of those helped to fix the problem. And the network guys blamed the Genesys applications, Genesys blamed the network and the server team blamed both yet none of them were any help to fix this problem.

I was able fix this by reserving the listening ports 3000, 7020, 2020, 5555 on Tservers thru the registery entry. No issues reported since then. Obviously there was some sort of port clashes with some other applications yet the event viewer or any other logs were not showing anything.