GIR generates multiple and empty mp3 files from recording

[nconiglio]

Hello guys, good morning

I am having a problem with my installation and after two weeks of throubleshooting I can't figure how to solve it.
Sorry for my bad english, it is not my first language.

A few months ago I installed GIR in a customer, and everything was working perfectly with no problems. Two months after the go live the MCPs generated the first occurrence of an empty mp3 file, but there were only a few per week. Now after a few months, the percentage of empty recording had been growing week by week. Now nearly the half or more interactions recorded has duplicated and/or empty files.

Let me explain myself better. The behaviour is the following, sometimes when an interaction gets recorded, the MCPs generates multiple mp3 for the same interaction, like multiple segments but without call transfer or anything that can generate more than one segment. There are many possible cases, the files generated for an interaction are:
- sometimes 1 mp3 ok with the call recorded fine, and one or two with the same name but empty (0 bytes)
- sometimes only 2 mp3 but both empty (this is the worst case because we lose the record of the interaction)
- sometimes the interaction get recorded fine, generating only one file with the audio ok

I discover that the problem is there, [b]when the audio file is generated[/b], and of course it generate multiple segments after all the recording process when you search at Speech Miner. Also I discover that the modification datetime of the empty mp3 files are always the datetime of the START of the call (it sholud be the end of the call, when the mp3 file is fully generated!). For example, the call started at 10:05 and ends at 10:08, the empty mp3 files has 10:05 in the modification date (in the filesystem and WebDAV).

More information. Before there was two MCPs in the Logical Resource Group, both in the same server. I tried to add another MCP (in a different server) to see if something different could happen, but the behaviour was the same. For each MCP there are fine recordings and bad recordings, so the problem is not that one MCP is failing and the others are working fine.
After that I tried to upgrade one MCP to see if it could solve it, but after the upgrade (to 8.5.185.34) the behaviour was the same, some interactions get recorded ok and some are bad.

I checked the logs of MCPs but I can't find anything weird. The only thing that I see is that when multiple files are generated, there are two (or three) posts made from the MCP in the WebDAV. This is logic, so the problem is in the MCP or before (SIP Server - RM - MCP), and not in the WebDAV or after.

I have no idea what to do, the bad behaviour looks very random, and for the worse, the percentage of incorrectly recorded interactions each seems to increase week by week(crazy!).

Could you please help me? If you need any more information I am here to bring it to you.
Thank you all very much in advance.

Regards,
Nicolás

[cavagnaro]

I would search the MP3 files into the MCP logs and try to understand who is creating them and why.
If you find it in the logs then SIP Server events can also help to understand why it would create different MP3 files

[nconiglio]

cavagnaro thank you so much for your help.

As I mentioned in the last post, I have already checked all the logs of MCP and traced the call in SIP Server, but I still can't find anything weird.

Here is the logs of SIP Server and MCP of an ocurrence of the issue:
[url=https://drive.google.com/open?id=1ZNezu5T6OsWJiseh5hReL0n0_cPwUO9K]https://drive.google.com/open?id=1ZNezu5T6OsWJiseh5hReL0n0_cPwUO9K[/url]
(You will see that you won't see it displayed because is a .rar, but if you press download you can download with no problems)

If you need I can share the logs without compressing, but the SIP Server log size is 50 MB and MCP 10 MB.

The interaction that failed has 009b02c46588f29b as ConnID.

Thank you for everything.

Regards,
Nicolás Coniglio

[cavagnaro]

What are the MP3 file names for that call?

[nconiglio]

normal_009b02c46588f29b_097531916_890_4871060_603_19-06-37_2018-07-17-00F301B4-[color=red]10002919[/color]-00000001.mp3
AND
normal_009b02c46588f29b_097531916_890_4871060_603_19-06-37_2018-07-17-00F301B4-[color=red]1000291A[/color]-00000001.mp3

The difference between them is that value, which is generated by MCP and not by recording filename setted in SIP Server configuration

[Janis]

Can it happen that there is no space on drive ocasionally?

[nconiglio]

Janice, there is a lot of space available, so that is not the problem.
Thanks anyway for your help!

Regards,
Nicolás

[Kubig]

There would be some configuration issue on RM or LRGP level as the root-cause is on selecting bad MCP instance for recording:

<description>Object does not exists: conn:F1B3F0CC-FE33-4892-7EA7-A4A2B8442A92</description>

This leads to creating mulitple recordings for one interaction.

Check your RM, MCP and LRGP configuration or post RM and MCP logs covering the issue.

[nconiglio]

Perfect Kubig! Just yesterday we tried to shutdown one of the RM (there are 2 working in PRIMARY-PRIMARY in the LRG) to see if something happened, perhaps every recording gets duplicated or perhaps every recording gets recorded fine, and we can diagnosticate that one RM was working bad.

After 1 hour every recording was recorded fine, I just asked to the customer that check if today the system is still working fine with no duplicated interactions (because we don't have remote access).

So Kubig, it seems that you are in the right way! What do you think it could be? The stopped RM configuration? Or perhaps the problem is when the 2 RMs are working together?

I've checked LRG and it's all fine (I created it from Genesys Administrator and it's a very simple wizard). Also the MCP configuration is the default configuration, and in every installation we have done, we never had to change anything, except for the listening ports of course.

I will get the old logs of the stopped RM to see if we can see something there.

Thank you all very much!
Regards,
Nicolás

[Kubig]

What load balance method do you use for RM active-active mode (external LB or SIP acts as LB)? If the second option, try to post your VoIP service obejcts configuration.

[nconiglio]

It has a LB by the VoIP service msml. This are the options:

Section    Key                                                Value
TServer    contact-list                                      172.20.32.193:5070,172.20.32.117:5070
TServer    cpd-capability                                  mediaserver
TServer    make-call-rfc3725-flow                    1
TServer    oos-check                                        5
TServer    oos-force                                        4
TServer    prefix                                              msml=
TServer    refer-enabled                                  false
TServer    ring-tone-on-make-call                    false
TServer    service-type                                    msml

The two IP:PORT are where the RM1 and RM2 listen

[nconiglio]

Here is one log of the RM that we stopped. Is from one week before the other logs, but the problem already existed at that time. We couldn't get a newer log because the log folder was moved and there were not more logs generated.

In the log I see this error ocurring a lot of times: "DataModule.cxx:2184 Replication fail reason: Call cannot be found"
Here I uploaded the log: [url=https://drive.google.com/open?id=1g2r6Jm2qJSt6ZwWOybuIPPEfn1xN7o3Q]https://drive.google.com/open?id=1g2r6Jm2qJSt6ZwWOybuIPPEfn1xN7o3Q[/url]

[Kubig]

I suppose there is problem with RM cluster inter-communication or configuration. Try to check your RM configuration on "cluster" section level or check whether the communcation between RM nodes is established properly

[nconiglio]

Both RMs are in the same LAN, so a connectivity reason is not the problem.
I discover that in RMs configuration, at cluster section, the values at member.1 and member.2 aren't the ips of the RMs servers. This is because before the RMs where on different servers. Could this be the reason of all the problem? Or perhaps another configuration is mistaken?

Here I uploaded the RMs .cfg files
https://drive.google.com/open?id=1ejNdODuKsNb4qsOQjfWqlh55UlWG0Qw9
https://drive.google.com/open?id=192SCsGJh9S-iYmX-ZaxRcBbJiC7RIwrJ

Regards,
Nicolás Coniglio

[Kubig]

IP address used in cluster configuration must reflect real state - so, this configuration is a root-cause of the issue.