Custom variables in reporting not updated correctly

[bogdan]

Hello guys,

We've just upgraded our current framework to 8.x and we have encountered some problems now that we didn't faced before. In each strategy that we have immediately after each succesfull target block (when a call is placed on an agents DN) we attach a KVP ("k_asnw_location") which represents the switch that the call has landed on. Based on this key we have created simple a filter PairExists("k_answ_location", "site1") that is used in our CallAnswered statistics to insert into DB the number of calls answered in each site per service. But now, after the upgrade it seems that this data is not updated everytime, somewhere around 30% this value is incremented into DMA. We have checked the URS and the key is attached everytime and also it can be seen in StatServer.

Any ideas what might be the problem, what should I check?

Regards,
Bogdan

[bogdan]

What I've tried:
1. I've added a delay after both target block and the attach block, but no luck
2. I've hardcoded the k_answ_location=site1 just before the target block, and I saw the data beeing incremented in reporting
3. I've tried to modify my filter to use RStatObjectSelected directly from the filter, but no luck here.
    note: in my case RStatObjectSelected in the VAQ which has the first 3 chars the switch

Only in the 2nd scenario the data was reported fully, I mean TotalAsnwer=answerOnSite1 + answerOnSite2...

It seems I really don't get what has happed, I don't seem to understand why I can't get this key everytime.

I've read some other post here, this key is read from VQ?! Cause on my target object I have checked the ClearTarget box. might this be the problem?

Bogdan

[cavagnaro]

can you post urs and tserver logs indicating the connid of a test call?

[bogdan]

The connId 015201f890ec81d5 can be found in all logs

Bogdan

[bogdan]

I can't upload the cca_statserver, it seems I have exceeded my limit.

What is very interesting in this log, is that I only see my key "k_ans_location" only once from 3 EventReleases:

[code]18:59:10.387 Int 04543 Interaction message "EventDiverted" received from 65200 ("buc_tserver_p")
2011-10-14 18:59:10> Server: Switch 'buc_pbx'->'EventDiverted'
        TimeStamp      '2011-10-14 18:59:10'
        ReferenceID    -1
        ThisDN 'vq_buc_cs_post_BUS_plat'
        ConnID 015201f890ec81d5 (95140709223137749)
        Call Type 'Inbound'
        This Queue 'vq_buc_cs_post_BUS_plat'
        CustomerID 'Resources'
        Media Type 'voice'
        ThirdPartyDN '91124293'
        User Data/Reasons
                'k_rona_occurred'      'false'
                'k_home_location'      'buc'
                'k_thru_cdn'    'true'
                'k_requested_service'  'cs_post_BUS_plat'
                'k_rona'        'false'
                'RTargetObjectSelected' 'vag_tim_cs_post_BUS_plat'
                'CustomerSegment'      'default'
                'ServiceType'  'default'
        Reasons
                'RTR'  356
[b]2011-10-14 18:59:10> Server: Unregistered call '015201f890ec81d5' on VQ 'vq_buc_cs_post_BUS_plat@buc_pbx' with stuck calls checker[/b]
2011-10-14 18:59:10> Action: VirtualACDQueue 'vq_buc_cs_post_BUS_plat@buc_pbx'(Monitored):
        [2011-10-14 14:06:09] [2011-10-14 14:06:09] Monitored






18:59:10.398 Int 04543 Interaction message "EventRemoteConnectionSuccess" received from 65200 ("buc_tserver_p")
2011-10-14 18:59:10> Server: Switch 'buc_pbx'->'EventRemoteConnectionSuccess'
        TimeStamp      '2011-10-14 18:59:10'
        ThisDN '4094'
        ConnID 015201f890ec81d5 (95140709223137749)
        Call Type 'Unknown'
        Media Type 'voice'



18:59:40.388 Int 04543 Interaction message "EventReleased" received from 65200 ("buc_tserver_p")
2011-10-14 18:59:40> Server: Switch 'buc_pbx'->'EventReleased'
        TimeStamp      '2011-10-14 18:59:40'
        ThisDN 'buc_pbx::'
        ConnID 015201f890ec81d5 (95140709223137749)
        Call Type 'Inbound'
        ANI '756053030'
        Media Type 'voice'
        Reliability '-1'
        User Data/Reasons
                'k_rona_occurred'      'false'
                'k_home_location'      'buc'
                'k_thru_cdn'    'true'
                'k_requested_service'  'cs_post_BUS_plat'
                'k_rona'        'false'
                'RTargetObjectSelected' 'vag_tim_cs_post_BUS_plat'
                'CustomerSegment'      'default'
                'ServiceType'  'default'
[b]                'k_answ_location'      'tim'[/b]





19:02:02.867 Int 04543 Interaction message "EventReleased" received from 65202 ("tim_tserver_p")
2011-10-14 19:02:02> Server: Switch 'tim_pbx'->'EventReleased'
        TimeStamp      '2011-10-14 19:02:02'
        ThisDN '4303'
        ConnID 015201f890ec81d5 (95140709223137749)
        FirstTransferConnID 015201f890ec81d5 (95140709223137749)
        Call Type 'Inbound'
        ANI '756053030'
        Media Type 'voice'
        Reliability '-1'
        User Data/Reasons
                'k_rona_occurred'      'false'
                'k_home_location'      'buc'
                'k_thru_cdn'    'true'
                'k_requested_service'  'cs_post_BUS_plat'
                'k_rona'        'false'
                'RTargetObjectSelected' 'vag_tim_cs_post_BUS_plat'
                'CustomerSegment'      'default'
                'ServiceType'  'default'
                'k_in_sl'      '0'
                'k_consult'    'false'
        Extensions
                'OrigAddress'  '756053030'
                'OrigAddrType'  bin: 00 00




19:02:04.866 Int 04543 Interaction message "EventReleased" received from 65202 ("tim_tserver_p")
2011-10-14 19:02:04> Server: Switch 'tim_pbx'->'EventReleased'
        TimeStamp      '2011-10-14 19:02:04'
        ThisDN 'tim_pbx::'
        ConnID 015201f890ec81d5 (95140709223137749)
        FirstTransferConnID 015201f890ec81d5 (95140709223137749)
        Call Type 'Inbound'
        ANI '756053030'
        Media Type 'voice'
        Reliability '-1'
        User Data/Reasons
                'k_rona_occurred'      'false'
                'k_home_location'      'buc'
                'k_thru_cdn'    'true'
                'k_requested_service'  'cs_post_BUS_plat'
                'k_rona'        'false'
                'RTargetObjectSelected' 'vag_tim_cs_post_BUS_plat'
                'CustomerSegment'      'default'
                'ServiceType'  'default'
                'k_in_sl'      '0'
                'k_consult'    'false'
[/code]


Bogdan

[cavagnaro]

Cant find that connid on the logs...you sure???

[bogdan]

I'm very sorry, the first archive was incorrect ...here is a fresh copy of a trace from today

[bogdan]

and the associated urs log

What I found very interesting, that might cause the problems is an ISCC error at line 133244 in clj_tserver_p.20111017_183316_155.log:  @18:36:21.3431 [ISCC] Error: UserData update collision detected. Update rejected

[code]
                'PegAGvag_clj_cs_post_BUS_plat' 1
                'k_answ_location'      'clj'
                'k_target'      'vag_buc_cs_post_BUS_plat@urs_statserver_p.GA'
                'k_from_ivr'    'false'
        ISCCAttributeConnID    015201f890ef4eca
        ISCCAttributeTrackingID 22152073 [01520389]
@18:36:21.3430 [ISCC] Origination party:
        @ c:015201f890ef4eca,0ab01ef8 @ m:0000000000000000,00000000 p:2 i:01161563 nw:0000000001144c5c t:2
          p:0000000000000000,0a923978 @ c:015201f890ef4eca,0ab01ef8 r:2 ----- n:4424:
          p:0000000000000000,0a9b25a0 @ c:015201f890ef4eca,0ab01ef8 r:1 ----- n:742086324:
        * p:015201f890ef4eca,00000000 @ c:015201f890ef4eca,0ab01ef8 r:1 ---t- n:4094:buc_tserver_p_760@buc_pbx
@18:36:21.3431 [ISCC] Error: UserData update collision detected. Update rejected
@18:36:21.8925 [ISCC] [connection 0a94fbb8] Received from connection [14]2
@18:36:21.8925 [ISCC] Received from server buc_tserver_p_760@buc_pbx/0: message ISCCEventAttachedDataChanged
        ISCCAttributeOperationUUID      '01D3US3F80F8V5OOPSOH1B5AES05H6I2'
        ISCCAttributeUpdateRevision    1442130 [00160152]
[/code]


Bogdan

[cavagnaro]

From Genesys

[quote]
Root Cause



Both T-Servers sent updates to each other at the same time and so TServer1 receiving the same update that it just sent previously created a conflicting situation (collision)

This behavior is by design and intended to prevent the data loss due to update collisions

Solution



No action is needed
The call data should be re-synchronized automatically on the next successful UserData update occurring at either of the T-Servers
[/quote]

[bogdan]

Thanks cavagnaro, I've found this on solution search, but I'm not happy with it.

In my case I think the root cause of my problem is this collision, the k_answ_location is first attached to the call on the initial Tserver (the home location on which the call has first entered), and the request to synchronize with the destination TServer (the target location on which the agent lays) fails and so the k_answ_location is not filter in the reporting.

I can't explain how this has worked on a 7.6 framework, and now after the upgrade to 8 it fails.

I've counted this errors from the logs and for a time frame of ~1.5 hours I have a large amount of errors
37 buc_tserver [ISCC] Error: UserData update collision detected. Update rejected
1337 clj_tserver [ISCC] Error: UserData update collision detected. Update rejected
1730 tim_tserver [ISCC] Error: UserData update collision detected. Update rejected

Before, on 7.6, I've counter ~5 errors per switch... this in my mind means a depreciation of a TServer.


Bogdan

[cavagnaro]

Can be...remember that G8 was re built and is not a straight upgrade from G7, that is also why you see so many upgrades over and over the weeks on the support site.
Recommendation #1: Upgrade to last versions if you havent already done it
#2: You will have to open a ticket directly with Genesys as they may indicate a new configuration option to solve this

[bogdan]

I've started the upgrade with the plan to migrate to 8.0 versions, and up to now, in 2 weeks only Tserver remained 8.0... all other components (URS, StatServers, OCS, CPD, ODS) I've upgraded because of bugs. I had really rought weeks 

One question, in theory... one scenarion:
Let's assume there is no overflow, I couldn't find any errors in Tserver, URS, StatServers, OCS, OCS_DB... which one is responsible for grabbing this KVP from the call and insert it into cca db structure? I don't really understand the flow of data in this scenario.
StatServer is listening for TServer events, and if he matches one with one if it's statistics and filters it caches the data and at the end it request the ODS to insert into CCA structure the temporary chunk. Right?

Bogdan

[cavagnaro]

Question, to which stat are you applying this filter???

[bogdan]

filters are on dedicated statservers (cca).

Today I've fixed half of the problem, or let's say a workaround... 2 days ago Genesys has released a 8.1 statserver hotfix and we've tested it today: the k_answ_location still doesn't work  , but it works if we filter PairExists("RTargetObjectSelected", "vag_[switch]*"). My virtual agent groups have the form [code]vag_[switch_name]_[service][/code], so I can filter this data correctly.

I haven't received a solution from support yet, but I will still push for a bug fix now that I've identified the problem - I'm sure the statserver is the problem.

Also for the ISCC errors, they didn't responded.

Bogdan

[cavagnaro]

hehe patience my friend That is why I still don't like the idea to go to a 8.1 version as many bugs are raising too continuously...
What I ment about my previous question was to which Metric are you applying the filter (MainMask)?