CTI no longer auto logs in our agent phones

[MarcRobinson]

We have a custom-written CTI solution using Genesys T-Server, GVP, and an Avaya PBX and AES. Always in the past when an agent logged into the CTI application, the CTI application then told the PBX to log the agent into the phone in AUX-Work. It still tells it to do that, but the phone doesn't light up, so the agent is in the CTI app but not logged into phone.

On 9/5 this apparently broke (the day before our U.S. Labor Day holiday). I came to work on Tuesday and agents were complaining about CTI not logging them into the phone automatically. The first couple of days I heard from maybe a dozen agents that they had this problem. (We have about 200 agents.) Then I started hearing from more, including at least one agent for whom CTI had logged her into her phone on 9/7. The problem was spreading. Now, 2 and a half weeks later, I know of 32 who have the problem. This is not critical, because agents simply log into their phones manually, and then they get their screen pops anyway, but it damages their faith in the software. Besides, I consider this broken, and I want to get it fixed.  Logging in manually to the phone is a workaround. Workarounds bug me.

At least the problem is consistent: once an agent gets the problem, the problem happens every time, and never goes away.

All the other I.T. people I've contacted swear up and down they made no changes that weekend: no PBX changes, no firewall changes, no patches to the agent PCs. The only thing I've been able to find is log messages that say the phone status is OUT_OF_SERVICE. I don't recall seeing that in the past, but I haven't looked in those logs for a LONG time, because CTI has been stable. But the PBX guys swear up and down they didn't change anything. Also, we're only monitoring those extensions from three servers, and we're allowed to monitor them from 4.

Now I'm down to considering stuff that's unlikely -- license expirations (checked FlexLM and it looked okay), database corruption in the CME (checked, and it looked okay), whatever. At this point I've pretty much ruled out every root cause I can think of. There is one thing did happen: the T-Server that monitors agent phones switched over to the backup machine, so the second T-Server is now primary. (They're in a hot standby arrangement.) But that happened weeks before the agents reported this problem.

I'm baffled.

[cavagnaro]

WIthout logs hard to tell, which PBX? Something must changed, even when they say no. Logs will tell the truth

[MarcRobinson]

I've looked through the logs and the only thing I found was the OUT_OF_SERVICE status I mentioned earlier.
This continues to spread -- I hear about 1 more agent a day, generally, who has he problem.

[cavagnaro]

Which PBX??? Post logs...we can't guess...

[MarcRobinson]

Per "which PBX", see the posting: "We have ... Avaya PBX and AES. "
The logs are way to big to copy into this post -- about 13 Mb.
Most forums allow files to be attached, but this forum apparently does not.
If you like, I'll zip them up and mail them to you, but some of them are custom and probably won't make a lot of sense to you.

[cavagnaro]

Not the whole logs, but a snippet of them with the issue on mention when an agent logs in and no auto login happens.
The fact that is doesn't allows login means that Genesys is sending the login order but Avaya is rejecting it. So a check of Avaya configuration is a MUST. That is why we can point the TServer event so you can go to PBX guys and say: "LOOK HERE, AVAYA IS REJECTING, NOW CHECK WHY"

[MarcRobinson]

Ordinarily, I'd make the same assumptions you're making, but in this case the assumptions are incorrect.
I had already looked through the logs and the PBX is not rejecting the event.
This is a weird problem. It started happening on a cluster of agents and has been spreading slowly ever since.

[cavagnaro]

Genesys doesn't control the login feature, just sends the CSTA event, the the PBX catches it and transforms.
Anyway, if you assume PBX is fine and Genesys is fine then a gizmo must be inside your network taking control over it.

[René]

Hi Marc,

You wrote that T-Server reports problematic DNs to be in OUT_OF_SERVICE state. In my opinion that's important information as T-Server won't be able to execute any operation on DN in out-of-service state. I would recommend you to check for reason(s) why DNs are reported to be out-of-service.

R.

[MarcRobinson]

I agree, Rene. I think the out of service status is the most important clue.
But my PBX guys are claiming that nothing's changed, and I can't log into the PBX and show them otherwise, since I'm not a telecom guy.

The T-Server thinks the phone logs in regardless of status -- i.e., the T-Server doesn't detect failures. I found this documented as a deficiency of the T-Server software.

I'd hoped that someone would recognize the root of this problem. No luck, I guess.

[MarcRobinson]

Well, it's fixed. Bouncing our .NET servers (part of our custom solution), which implement the CTI piece, solved the problem.
However, the .NET servers did not show any errors in their logs.
I still don't know what went wrong, unless they were simply moritoring the extensions too long without being bounced, but at least I know what to do next time this happens.