GVP stations going unstaffed

[MarcRobinson]

We've been having a problem for a long time with stations that go unstaffed.

What happens is that a call comes in and the GVP doesn't answer it. After a couple of rings, the PBX takes the call back and marks the extension unstaffed.

This was happening a lot when we had an overloaded LAN -- call data would not get from the PBX to the T-Server, or from the T-Server to the GVP. This tended to happen more often at particular times of day, while large file transfers were in progress. We solved this problem by using a different part of our LAN for the IVR traffic. Now it looks very clean when the network guys check the statistics.

We're still getting the unstaffed stations, though. It may not happen for weeks at a time, and then we'll get unstaffing several times during one week. Some of the time, unstaffing will be on one or two or a few stations. In most such cases, we can busy and release the affected stations and they will then be able to take calls.

Once in a while, however, the busy/release from the PBX side isn't enough, and we have to reboot the GVP. We also get mass unstaffing events, with a lot of unstaffed stations (even all the stations), and these seem to require a GVP reboot.

Has anyone else had similar problems? I have some ideas on what's going on, but don't want to post them here. That way, people will give me different ideas than my own -- or the same ideas, confirming mine.

This has been going on for more than a year, since we first installed Genesys IVR, and frankly we're getting exasperated.

[cavagnaro]

And can you see something on the GVP logs? Try to setting them to Full so you can see exactly what is going on.
Is your application WebServer the same CPU as all GVP? How is your network traffic on the ethernet interface? How is your CPU and memory consumption during those high traffic times?
I believe it could be because the call arrives GVP but at the time it tries to connect to the webserver server to fetch XML application maybe the response is very slow and therefore the application goes down. Maybe trying to configure the backup application server can help.
Maybe if you post some GVP logs we can see exactly the error.

[MarcRobinson]

I have been planning to read the GVP logs, and your post is giving me more motivation to do so.

The question about secondary URL is a smart one. However, we've got that covered. It's a one-page VXML application that runs locally on each GVP, and that gives the call back to the PBX. So the problem is prior to that -- the GVP either isn't seeing the call, or isn't responding in timely fashion on the primary URL (but not necessarily because of time required to fetch the page).

The network traffic is okay -- the network guys have looked at it the last couple of times we have a big problem, and nothing was getting dropped at the Cisco equipment. That's not to say there's no problem on the GVP itself.

As for CPU and memory, I didn't check them -- I was too busy fighting the fire. Next time, we'll just route around that GVP and I'll look at this.

I'll post some GVP logs. Thanks for the help.

[cavagnaro]

Hum, the fact that the webserver is on the same PC as GVP isn't a secure way to avoid failures, even more, if your CPU is busy with GVP transactions IIS may not answer fast enought, even when request is locally. More and more if same CPU has TTS or ASR for example, processes will take more time to be attended. Remember that Windows is not a really multitask OS, it does a trick to make it look like one, so if one of the ticks of the clock does slower the rest will also do.

[MarcRobinson]

In the earlier posting, the secondary URL comment does not imply anything about where the primary URL points to. The primary URL is NOT fetched from the GVP. Rather, it is fetched from dedicated web servers running the normal application code. These web servers are at a different site (not where the GVPs are), more than 10 miles away. That site contains the webservers that feed the GVPs at both our call centers (in Missouri and Nebraska).

ONLY the secondary URL is on the GVP itself, as insurance against any network breakages. When the secondary URL is being used, our Voicewatch monitoring pages us so we know that the primary URL is not working. This tends to happen because of human error, not because of network problems, since the web servers are clustered and the OC3 links between sites are redundant. It is a rare event indeed when the secondary URL is used.

[cavagnaro]

Oh ok, gotcha.
Anyway, the step between is the process of the call itself, (URS, application reading, etc etc) so still looking and setting logs to full will be good for us to help on your problem.

[bcyk]

GVP/IVR port does not answer incoming call
------------------------------------------

It is hoped below information helps.
They, however, cover switch configurations and Dialogic card firmware/protocol only.

The problem was first encountered in Genesys 6.5 GVP/IVR.
- environment: Avaya CM 2.0, Dialogic card DM3 line-side T1s, Dialogic SR 6.0,
    Genesys 7.1 framework and Window 2000 server

- Drop call problem was also encountered.
  After upgrading firmware at Dialogic card, drop call issue was fixed.

- Problem of not answering incoming call stayed unfixed.
  manual reset unstaffed or not ready, depending of Ring-no-answer action in PABX configuration, had to be performed periodically




The identical problem was found last year in other two sites, using 7.x GVP/IVR
- environment: Avaya CM 3.x, Dialogic card DM3 line-side E1s,
Genesys 7.2 framework and window 2003 server

- in GVP port status page, hung ports showed "Disconnected" call status

- attempt:
  - set DisableDropcall = 1 in each of the Popgateway section of GVP
  - suggested by Genesys (although the exact case was for Nortel DMS-100!)
    "It happens when GVP issues gc_DropCall and there is no answer to this due to protocol limitation."
  - after disableDropcall=1, frequency and occurrence of hung ports were reduced but existed still


After discussion with a GVP/IVR programmer who implemented similar components and having no hung port problem, difference was identified but not tested in production environment (yet! due to lack of testing environment)

  - when installing GVP / Dialogic card, signalling protocol was "pdk_sw_e1_luls_io.cdp", the one stated in manual.
  - the engineer claimed that they had problem running GVP/IVR when choosing "pdk_sw_e1_luls_io.cdp"
  - the sigalling protocol "pdk_us_ls_fxs_io.cdp" was used instead
  - There was no report on unstaffing / hung GVP/IVR port issue.

It may be worth to try in testing environment if available.....

[MarcRobinson]

Thanks for the reply. I have some questions. I think you're saying that you had the problem. Or do you mean that someone else had the problem? (If it was someone else, where did you find the documentation on this?)

If I understand the first part of your reply, you're saying that the Dialogic firmward upgrade did not solve the problem of ring-no-answer. Am I correct?

In the next part of your reply, I think you're saying that setting "DisableDropCall = 1" reduced the severity of the hung-port problem, but didn't entirely get rid of it. Am I correct?

In the last part of your reply, you're saying that you MAY have a solution, by changing the signalling protocol. I'll try that here on our development system. However, we never get unstaffing on that system ( ! ), so this won't tell us much. We'll have to talk about putting it on production after we've tried it on development.

Thanks for your help. This is the first time I've seen someone else who has had the problem. I suspect that many more people have experienced this, and simply learned to tolerate it.

[MarcRobinson]

Found the way to attach files, finally. It's After you hit the "Post" button.

So I've attached the files from one of our two GVPs here, zipped up.

[MarcRobinson]

I have two zipped files, each slightly over 6 meg, which I tried attaching separately and which are within the size limit. They time out, even though the connection on my end is a gigabit.
The only alternative is probably to send them by e-mail.

[cavagnaro]

You mean below the text area where you write there is a link that says:
Additional Options

There  you upload the files. Limit is 10Mb, I remember uploading a big file without issues...

[bcyk]

MarcRobinson,


Yes! GVP port ring-no-answer and hung problem does exist and still unresolved in some sites I know.


1. Dialogic firmware upgrade did not solve the problem in 6.5 version
  it fixed drop-call issue under heavy traffic in early version.
  (since I am no long support the site, current state is unknown!)

2. correct; setting "DisableDropCall=1" helped but did not solve the problem in the specific environment/configuration
  information was from Genesys SR
  option was set in one of production sites having ring-no-answer problem.

3. signalling protocol configuration
  information was obtained from other Genesys engineer implementing GVP/IVR in different areas.
  ring-no-answer and hung port problem was never reported using the mentioned signalling protocol

  Undestood that it is quite hard to simulate enough call traffic in testing envrinment!
  It is one of reasons why the problem is hard to identify and resolved


It should be pointed that I'm not familar with GVP/IVR details such as call flow menu building.
Since IVR is usually placed as the first customer contacting device (i.e., customer identification and classification), remaining call flow to URS and agent desktop will be blocked if IVR ports are hung! I'm "forced" to pay attention to it. 

The worst case for cause of hung port issue could be mixed combination of firmware/signalling protocol, network bandwidth and server loading. If so, it was very tough to identify!

[MarcRobinson]

bcyk,

Thanks for the info. I plan to check the things you noted.
Even if those don't fix the problem, it's reassuring to know that other people have the problem and that it's not unique to us.

[MarcRobinson]

Cavagnaro,

Yes, I mean below the text area, at the "Additional Options" link.

Yes, the limit is 10Mb, and both my zipped files are slightly over 6 MB, but neither one of them will go (I tried them separately).
E-mail, or FTP, might work.
What do you say?

[cavagnaro]

email them to cavagnaro at alcatelunleashed.com
I'll try to upload them as well