Genesys CTI User Forum
Genesys CTI User Forum => Genesys CTI Technical Discussion => Topic started by: MarcRobinson on March 31, 2008, 01:54:48 PM
-
We've been having a problem for a long time with stations that go unstaffed.
What happens is that a call comes in and the GVP doesn't answer it. After a couple of rings, the PBX takes the call back and marks the extension unstaffed.
This was happening a lot when we had an overloaded LAN -- call data would not get from the PBX to the T-Server, or from the T-Server to the GVP. This tended to happen more often at particular times of day, while large file transfers were in progress. We solved this problem by using a different part of our LAN for the IVR traffic. Now it looks very clean when the network guys check the statistics.
We're still getting the unstaffed stations, though. It may not happen for weeks at a time, and then we'll get unstaffing several times during one week. Some of the time, unstaffing will be on one or two or a few stations. In most such cases, we can busy and release the affected stations and they will then be able to take calls.
Once in a while, however, the busy/release from the PBX side isn't enough, and we have to reboot the GVP. We also get mass unstaffing events, with a lot of unstaffed stations (even all the stations), and these seem to require a GVP reboot.
Has anyone else had similar problems? I have some ideas on what's going on, but don't want to post them here. That way, people will give me different ideas than my own -- or the same ideas, confirming mine.
This has been going on for more than a year, since we first installed Genesys IVR, and frankly we're getting exasperated.
-
And can you see something on the GVP logs? Try to setting them to Full so you can see exactly what is going on.
Is your application WebServer the same CPU as all GVP? How is your network traffic on the ethernet interface? How is your CPU and memory consumption during those high traffic times?
I believe it could be because the call arrives GVP but at the time it tries to connect to the webserver server to fetch XML application maybe the response is very slow and therefore the application goes down. Maybe trying to configure the backup application server can help.
Maybe if you post some GVP logs we can see exactly the error.
-
I have been planning to read the GVP logs, and your post is giving me more motivation to do so.
The question about secondary URL is a smart one. However, we've got that covered. It's a one-page VXML application that runs locally on each GVP, and that gives the call back to the PBX. So the problem is prior to that -- the GVP either isn't seeing the call, or isn't responding in timely fashion on the primary URL (but not necessarily because of time required to fetch the page).
The network traffic is okay -- the network guys have looked at it the last couple of times we have a big problem, and nothing was getting dropped at the Cisco equipment. That's not to say there's no problem on the GVP itself.
As for CPU and memory, I didn't check them -- I was too busy fighting the fire. Next time, we'll just route around that GVP and I'll look at this.
I'll post some GVP logs. Thanks for the help.
-
Hum, the fact that the webserver is on the same PC as GVP isn't a secure way to avoid failures, even more, if your CPU is busy with GVP transactions IIS may not answer fast enought, even when request is locally. More and more if same CPU has TTS or ASR for example, processes will take more time to be attended. Remember that Windows is not a really multitask OS, it does a trick to make it look like one, so if one of the ticks of the clock does slower the rest will also do.
-
In the earlier posting, the secondary URL comment does not imply anything about where the primary URL points to. The primary URL is NOT fetched from the GVP. Rather, it is fetched from dedicated web servers running the normal application code. These web servers are at a different site (not where the GVPs are), more than 10 miles away. That site contains the webservers that feed the GVPs at both our call centers (in Missouri and Nebraska).
ONLY the secondary URL is on the GVP itself, as insurance against any network breakages. When the secondary URL is being used, our Voicewatch monitoring pages us so we know that the primary URL is not working. This tends to happen because of human error, not because of network problems, since the web servers are clustered and the OC3 links between sites are redundant. It is a rare event indeed when the secondary URL is used.
-
Oh ok, gotcha.
Anyway, the step between is the process of the call itself, (URS, application reading, etc etc) so still looking and setting logs to full will be good for us to help on your problem.
-
GVP/IVR port does not answer incoming call
------------------------------------------
It is hoped below information helps.
They, however, cover switch configurations and Dialogic card firmware/protocol only.
The problem was first encountered in Genesys 6.5 GVP/IVR.
- environment: Avaya CM 2.0, Dialogic card DM3 line-side T1s, Dialogic SR 6.0,
Genesys 7.1 framework and Window 2000 server
- Drop call problem was also encountered.
After upgrading firmware at Dialogic card, drop call issue was fixed.
- Problem of not answering incoming call stayed unfixed.
manual reset unstaffed or not ready, depending of Ring-no-answer action in PABX configuration, had to be performed periodically
The identical problem was found last year in other two sites, using 7.x GVP/IVR
- environment: Avaya CM 3.x, Dialogic card DM3 line-side E1s,
Genesys 7.2 framework and window 2003 server
- in GVP port status page, hung ports showed "Disconnected" call status
- attempt:
- set DisableDropcall = 1 in each of the Popgateway section of GVP
- suggested by Genesys (although the exact case was for Nortel DMS-100!)
"It happens when GVP issues gc_DropCall and there is no answer to this due to protocol limitation."
- after disableDropcall=1, frequency and occurrence of hung ports were reduced but existed still
After discussion with a GVP/IVR programmer who implemented similar components and having no hung port problem, difference was identified but not tested in production environment (yet! due to lack of testing environment)
- when installing GVP / Dialogic card, signalling protocol was "pdk_sw_e1_luls_io.cdp", the one stated in manual.
- the engineer claimed that they had problem running GVP/IVR when choosing "pdk_sw_e1_luls_io.cdp"
- the sigalling protocol "pdk_us_ls_fxs_io.cdp" was used instead
- There was no report on unstaffing / hung GVP/IVR port issue.
It may be worth to try in testing environment if available.....
-
Thanks for the reply. I have some questions. I think you're saying that you had the problem. Or do you mean that someone else had the problem? (If it was someone else, where did you find the documentation on this?)
If I understand the first part of your reply, you're saying that the Dialogic firmward upgrade did not solve the problem of ring-no-answer. Am I correct?
In the next part of your reply, I think you're saying that setting "DisableDropCall = 1" reduced the severity of the hung-port problem, but didn't entirely get rid of it. Am I correct?
In the last part of your reply, you're saying that you MAY have a solution, by changing the signalling protocol. I'll try that here on our development system. However, we never get unstaffing on that system ( ! ), so this won't tell us much. We'll have to talk about putting it on production after we've tried it on development.
Thanks for your help. This is the first time I've seen someone else who has had the problem. I suspect that many more people have experienced this, and simply learned to tolerate it.
-
Found the way to attach files, finally. It's After you hit the "Post" button.
So I've attached the files from one of our two GVPs here, zipped up.
-
I have two zipped files, each slightly over 6 meg, which I tried attaching separately and which are within the size limit. They time out, even though the connection on my end is a gigabit.
The only alternative is probably to send them by e-mail.
-
You mean below the text area where you write there is a link that says:
Additional Options
There you upload the files. Limit is 10Mb, I remember uploading a big file without issues...
-
MarcRobinson,
Yes! GVP port ring-no-answer and hung problem does exist and still unresolved in some sites I know.
1. Dialogic firmware upgrade did not solve the problem in 6.5 version
it fixed drop-call issue under heavy traffic in early version.
(since I am no long support the site, current state is unknown!)
2. correct; setting "DisableDropCall=1" helped but did not solve the problem in the specific environment/configuration
information was from Genesys SR
option was set in one of production sites having ring-no-answer problem.
3. signalling protocol configuration
information was obtained from other Genesys engineer implementing GVP/IVR in different areas.
ring-no-answer and hung port problem was never reported using the mentioned signalling protocol
Undestood that it is quite hard to simulate enough call traffic in testing envrinment!
It is one of reasons why the problem is hard to identify and resolved
It should be pointed that I'm not familar with GVP/IVR details such as call flow menu building.
Since IVR is usually placed as the first customer contacting device (i.e., customer identification and classification), remaining call flow to URS and agent desktop will be blocked if IVR ports are hung! I'm "forced" to pay attention to it. :'(
The worst case for cause of hung port issue could be mixed combination of firmware/signalling protocol, network bandwidth and server loading. If so, it was very tough to identify!
-
bcyk,
Thanks for the info. I plan to check the things you noted.
Even if those don't fix the problem, it's reassuring to know that other people have the problem and that it's not unique to us.
-
Cavagnaro,
Yes, I mean below the text area, at the "Additional Options" link.
Yes, the limit is 10Mb, and both my zipped files are slightly over 6 MB, but neither one of them will go (I tried them separately).
E-mail, or FTP, might work.
What do you say?
-
email them to cavagnaro at alcatelunleashed.com :)
I'll try to upload them as well
-
Well first thing I can see here is this:
[code]
<XMLPage>
<RESPONSE RESULT ="FAILURE" REASON="ERROR: process TTS_MRCP is not running">
</RESPONSE>
</XMLPage>
[2008/03/26 21:56:59.113] 1698 RscMgr.cpp:105 C=7:L=2:U=315 [deleting RscMgr] Failed to delete cached file: Local[<?xml version="1.0"?>
<XMLPage>
<RESPONSE RESULT ="FAILURE" REASON="ERROR: process TTS_MRCP is not running">
</RESPONSE>
</XMLPage>
[/code]
Seems that your MRCP server is too busy to accept more interactions, is this server running on same CPU as GVP?
Will check further.
-
Here is another:
[code]
[10-Dec-2007 07:02:16] PHP Fatal error: Maximum execution time of 30 seconds exceeded in F:\GVP\cn\Web\Common\utility.inc on line 10
[/code]
I will have to insist that this seems a CPU processing issue.
PLease give more details on what you have installed on GVP Cpu and how your architecture is.
-
It's quite odd that there would be a CPU problem, because we've stress tested, and the CPUs, even with maximum calls (all 48 channels busy) are using only a tiny fraction of CPU capacity -- less than 10%, with occasional peaks, but still lots of CPU to spare. Maybe there's an occasional problem with a process that hogs the CPU. We'll have to look at this further.
As for MRCP, we aren't running it. My understanding is that it's only used for ASR, and we're strictly a DTMF shop. I've seen errors for it, and was told to ignore them. MRCP errors (though maybe not the same ones you see in the logs) have been happening since we installed the GVPs in late 2006.
What we have running on our GVPs are these things:
VCS
IVR Server
IVR T-Server
local control agent
watchdog
Intel services: boardserver and system service
The GVPs are used only to run the touch-tone application. That's all.