CCA counts do not match expected

[jpashley]

I have an issue where the number of records entering a queue does not match the number of events leaving a queue.

The issue has to do with the Stat Server processing call events in the incorrect order.  A call comes in and the "Event Established" is processed by stat server prior to the "EventQueued" is processed.  The call is actually handled but since the latest event is the Queued event, the call is not recorded as handled.  Also this causes the "Handled" time for the call appear as "Queue" time.

The events are leaving the SIP server in the correct order but are not being processed by stat server in the correct order.

Anyone have a similar experience, or know of a way to solve this?

[René]

Hi,

I've never seen something like this and it sounds really weird. Have you checked network connection between SIP Server and Stat Server? What is current load processed by Stat Server (number of requests, CPU load etc.)

R.

[fnunezsa]

How do you know StatServer is "not processing events in the correct order"? Are the events arriving to StatServer in the correct order???

[jpashley]

We have pulled all logs for different calls.  The rate of errors increases at periods of high traffic.

We have reviewed network traffic, sniffer logs, etc. but have been unable to pinpoint the major cause.

We can see the processing of the events in the stat server logs which shows the incorrect timing of the events.

[ExodusSS]

If your StatServer is not spiking on the Server and your Network is not under load - can you advise of your StatServer version?

ESS

[jpashley]

I believe we are running version 7.6.1.0019

We have about 8 T-servers/SipServers running and 3 stat servers.  One stat server is used for routing, one for CCPulse and one for all other reporting.  The only one we are having issues with is the reporting stat server.

[fnunezsa]

How many opened stats does problematic StatServer has?

[jpashley]

about 325 stats are configured on the stat server

[cavagnaro]

and how many clients? Check the logs and see how many stats are opened

[Kevin S]

jp - have you reviewed the release notes for stat server? if you're running 7.6, the issue may have already been addressed.

Also - i *think* that your SIP server (acting as a TServer) should show the clients that the events are sent to. For example (and these were pulled from a Nortel TServer)
11:32:19.573 Trc 04542 EventDialing sent to [696] (0000001d cca_statserver_a xxx.yyy.zzz.www:50465)

Are you able to confirm that all the events that the TServers say they are sending are arriving? Are you able to identify if it is always the same TServer, or the behavior happens on each of them? Would creating dedicated Reporting Stat Servers be an option?

[jpashley]

We are seeing all the events being sent to the stat server, in the correct order but when they are processed by stat server, they are being received in an incorrect order.  The calls are coming in from multiple T-servers/SIP servers.

In our environment we currently have one stat server dedicated to CCA reporting, one for routing, and one for CCpulse.