AES-Tserver heart beat

[krish_kk123]

Hey,

We are having some issue with the CTI link and its heatbeats. we are running on a 8.0 t-server with AES 4.2.1, and recently we have been seeing that links drops more than 5 times a day. The reason that we noticed, is that the heart beats aren;t proper and it times out every now and then.

Can anyone of you please tell me how an heat beat is set up and where ?

1. What is the time set for a heart beat, I mean after how many seconds/mins it should attempt it ? or is it just based on the NO-Activity ?

2. Who is responsible for an heart beat initiation ? CM or Genesys ?

Thanks in Advance!

Regards
krish

[krish_kk123]

@15:52:54.5420 [tsgctm] Link [1,s4] hBeatTimerHandler - timer expired. Closing link.
@15:52:54.5420 [tsgctm] Link [1,s4] stop
@15:52:54.5420 [tsgctm] LinkReg [1, act=1] clearAll
@15:52:54.5420 [tsgctm] Link [1,s4] closeSession
@15:52:54.5420 [tsgctm] Link [1,s4] Changing state to 5
@15:52:54.5420 [asai] Link [1,s5] (sysReqSessionClose)
TP_DisconnectNotification
@15:52:54.5421 [>>] 04 00 00 00
@15:52:54.5421 [tsgctm] Link [1,s5] Changing state to 0
(link_stop) link [1] 'link-tcp' stop
(tcp_stop) closed on [11]
@15:52:54.5421 [tsgctm] PRIMARY link 1 cleared
@15:52:54.5422 [gctmi] TMsg [EventLinkDisconnected()] distributing to model
@15:52:54.5422 [gctmi] Switch [sCON] distributing EventLinkDisconnected
@15:52:54.5422 [gctmi] Switch [sCON] processLinkDisconnected
@15:52:54.5422 [0] 8.0.006.01 send_to_all: message EventLinkDisconnected

[krish_kk123]

Guys,

Any inputs, thoughts ? this is getting worst and am really finding it tough to find out why we are having cCTI links droping atleast once everyday.

Appreciate your help this

Regards
Karthik

[René]

Hi Karthik,

Does it happen randomly or during peak times? Your issue seems to be network related so I would suggest you to monitor network between TServer and AES (lost packets, current throughput etc.).

Of course, you can increase the heartbeat timeout but that is not a solution.

R.

[Gert]

This can either be related to the network between the AES and T-Server or to the Clan card. I agree with Rene if you can tell us if this normaly happens during peak or at any time. also what is the peak trafic between the two systems, how many Clans do you have configured for the link.

[MarcRobinson]

We've had a similar problem occasionally. Since the problem seems to be outside the T-Server (either in the network or the AES server or whatever), and since I don't have control over those external entities, I wrote a Powershell script on the T-Server to monitor the link. This gets run every x minutes and notifies me automatically when the link drops and stays down. It's not a perfect solution, but it's better than nothing. If you want to try this, I'll send instructions.

[krish_kk123]

Gert/Rene/MarcRobinson,


First of all, I thank you all for you prompt response.
Coming back to the issue, Well we do have peak load at this moment, We are still at the UAT phase of the project and not Live with production peak load. So I really feel the msg trasport load is very minimal.

We tried to set up sniffers and unlucky to end with no results...No packet drops even. and CTI disconnect still continues, with same pattern..."Heart beat missing".

At this point, I have quick question...sorry might not be realted and sounds silly. Can you please tell me why does the logs show "CTI Link Connected" messages every now and then, but not consistent. Me confusion was...why does it say connected when there was no disconnect at all ?

@MarcRobinson, Yes please, can you send me the script that you were talking about....that might help me at some point to atleast get a real time alert and investigate as we do not have alarms set up at the moment.

Once again, thanks!

Regards
Krish

[Fra]

Krish,

I'm not sure about TServer 8, however, up to v 7.6 the following options were configurable on TServer app:

ts-tp-enabled
Default Value: true
Valid Values: true, false
Changes Take Effect: Immediately
Enables or disables the T-Server side tunnel protocol of the Heartbeat feature.
When the option is set to true and T-Server is in High-Availability mode,
tunnel protocol heartbeat messages are sent for both primary and backup
T-Servers.

ts-tp-heartbeat-timer
Default Value: 25
Valid Values: 5–60
Changes Take Effect: Immediately
Specifies the time interval (in seconds) that T-Server waits for link activity,
before it sends a side tunnel protocol heartbeat message to the link.

ts-tp-heartbeat-timout
Default Value: 10
Valid Values: 5–60
Changes Take Effect: Immediately
Specifies the time interval (in seconds) that T-Server waits for link activity
after sending a side tunnel protocol heartbeat, before it closes the link and
restarts.

You should be able to see the polling messages in the logs and identify at exactly what time the disconnections occurred, so that you can then verify whether anything happened on AES at that istant.

In regards to your questions, how can you be sure TServer did not disconnect? After a disconnection, TServer automatically tries to reconnect to AES, therefore you may see the EventLinkConnected multiple times.

Fra

[krish_kk123]

Thanks Fra!

Yes we have not set the the heartbeat parameters that you set up and waiting for a disconnect unfortunately : )

And talking about link connected messages, I have been looking at the t-server logs and pretty sure I did not see any disconnects. All I saw was link connected message which I have not seen in the past and not sure if thats a regular thing in all t-server logs ?

Thanks!
Krish

[ademince]

Hi Krish,

Your problem sounds like mine which took me 14 month to fix...see this link and very last entry at the end of it...hope it helps you too...

http://www.sggu.com/smf/index.php/topic,4807.0.html

[krish_kk123]

@ademince

Thanks for the info and yes that really great to get it resolved after 14 months : )

Do you think this would apply for AES and T-server connectivity as well ? Because I see that your case had Genesys internal apps...like URS, so I just wanted re-confirm if this can fit into external CTI adjuncts as well.

Thanks again!

[ademince]

Krish...anything worth trying if it will fix the problem...good luck

[MarcRobinson]

Krish,

I hadn't been on in a while, so there was some delay, but I finally e-mailed you the code and instructions on how to set up a script that monitors the link.

[sunbabu15]

Hi Marc,

We are also facing a similar issue in our setup. I have followed the same points as refered below . Currently we are monitoring the link but the script would be helpful for this . Can u please send me the code or instructions on how to set up a script that monitors the link which u have refrd in the post .

Regards,