Agent unable to answer Ringing call

[Junaid]

Hello,

I am trying to find root cause of particular packet lost between SIP server(on cloud) and IWS Endpoint(8.5.100.10). Agent is able to receive calls  smoothly as long 'BYE' msg received to endpoint for current call. But if BYE not received(SIP server sending showing in traces) to agent, he is unable to answer next call. Same is happening with IWS endpoint (8.0.204.xx) but Agent can answer the call. I know this pertains to Network/firewall but their justification is also valid that why this behavior is intermittent. Please advise, if anyone faced this earlier.

Let me know if my question is not clear or some logs and traces needed.

Regards,

[Fra]

What I would suggest is running those network traces also on the client side to verify whether the BYE is delivered to the agent's machine.
That is quite important to determine where the fault may lie.

Fra

[Junaid]

Yes, I captured traces on both(SIP Server and Agent machine) at same time. I saw SIP server continuously sending BYE to Agent machine but on Agent machine traces, I didn't saw BYE msg. This issue only happen with 'BYE'. If you need dumps then I can upload it


Regards,

[Kubig]

In case, you are using the WDE SipEndpoint in release 8.5, make the upgrade to the latest version or to the version 8.5.104.12, where the similiar issue as you have descibed has been fixed.

[Fra]

[quote author=Junaid link=topic=9368.msg42297#msg42297 date=1455712004]
Yes, I captured traces on both(SIP Server and Agent machine) at same time. I saw SIP server continuously sending BYE to Agent machine but on Agent machine traces, I didn't saw BYE msg. This issue only happen with 'BYE'. If you need dumps then I can upload it
[/quote]

I personally observed a similar but not quite the same scenario: BYE seen in SIP Server logs, SIP Server network traces and endpoint network traces but not processed by the endpoint.

What OS are you running the endpoint on?

As Kubig has pointed out, there were issues in the SIP endpoint stack which were fixed in 8.5.104.12. However, if you can't see the BYE in the network traces captured on the endpoint machine, I suspect the root cause should be different.

Fra

[Kubig]

Yes, Fra is right, if you do not see the SIP BYE message on NIC where the WDE SipEndpoint is running, the root-cause should be different and the upgrade of WDE SE should not help.

[hsujdik]

I'm not really an expert, but as you said that SIP Server is on cloud, maybe the address is NATed on one of the two sides? If this is the case, firewalls usually maintain a NAT connection table, where the connections expire after a certain timeout (specially UDP communication where the tracking of the connection state is not possible to be made). Try lowering the value of session-refresh-interval on TServer session of SIP Server application (and confirm that session-refresh-enforced is set to true) so SIP messages would be more frequent in a way that it would not time out the NAT connection. Or, on the endpoint side, try to reduce the value of option proxies.proxyX.reg_timeout (or sipendpoint.proxies.proxyX.reregister_in_seconds in IWS options) to something around 300

[Junaid]

Agent Machine OS: Windows 7 Professional x86

Yes Kubig and Fra. I agree that its not an endpoint bug and hoping that this will be trace out at client's firewall but I need to know why endpoint 8.0.204.xx working perfectly with out getting BYE from SIP Server. Also I forgot to mention that clients environment using NATted IP. For Instance local IP of agent machine is 10.x.x.x and its natted IP which we can see in SIP logs is: 192.168.x.x.

[Kubig]

SipEndpoint within releases 8.0 and 8.1 was based on CounterPath SW, whilst the SE in release 8.5 is based purely by Genesys. So, the interal mechanism and used OS resources should be different. Due these changes I have encountered a few issues related to upgrade from 8.x to 8.5 like:

- the firewall is droping the incoming SIP messages. The incoming rule allows the SIP communication has to be created and used
- the SE was sometimes stucked and has not processed all incoming packets at all

[Junaid]

[b]@ hsujdik[/b]

Quite close. Yes cloud SIP server has two IP address but not NATed in which one is use to communicate with client environment.
As you quote "If this is the case, firewalls usually maintain a NAT connection table, where the connections expire after a certain timeout". If I got your point then why Endpoint receives next call 'INVITE'?. I also tried your suggestion by lowering [b]session-refresh-interval[/b] from 1800 to 90 keeping [b]session-refresh-enforced[/b]= true but in results call was not routing to queue  :(production he bhai ). Can you please suggest any reference/document for NATing w.r.t Genesys  .

[b]@Kubig[/b]

"SipEndpoint within releases 8.0 and 8.1 was based on CounterPath SW, whilst the SE in release 8.5 is based purely by Genesys" that's pretty knowledgeable. Have you came to know why firewall dropping SIP messages in newer releases of SE?

BTW Thank guys for your prompt responses.

Regards,
Junaid

[hsujdik]

[quote author=Junaid link=topic=9368.msg42311#msg42311 date=1455739078]
[b]@ hsujdik[/b]
Quite close. Yes cloud SIP server has two IP address but not NATed in which one is use to communicate with client environment.
As you quote "If this is the case, firewalls usually maintain a NAT connection table, where the connections expire after a certain timeout". If I got your point then why Endpoint receives next call 'INVITE'?. I also tried your suggestion by lowering [b]session-refresh-interval[/b] from 1800 to 90 keeping [b]session-refresh-enforced[/b]= true but in results call was not routing to queue  :(production he bhai ). Can you please suggest any reference/document for NATing w.r.t Genesys  .
[/quote]

Uh oh! Well, there was a warning on the SIP Dep Guide, so my fault on that.

I don't have a documentation though. I based on some changes we had to do on both SIP and our firewall on a NATed environment, because the session (and register) refresh was bigger than our firewall bind timeout for UDP, the firewall docs and the RFC 3581

About the next INVITE, what I understood is that the agent didn't even receive it. But reading carefully again, you said that only the BYE message is not received. Perhaps there was another REGISTER between the BYE and the INVITE? That's the only thing I can think of.

Do you have traces from the current version vs. old version?

Another thing to look on NAT environment is if RPORT parameter is present on Via header of the SIP messages. RPORT needs to be used if the firewall re-maps the source port to other than your SIP port while performing the NAT conversion, and without any value - RFC 3581 - https://www.ietf.org/rfc/rfc3581.txt

Also, the reason why I commented about the options in SIP Endpoint is because of the following from this RFC
If used for UDP registrations, a client will need to frequently
      re-register in order to keep the NAT bindings fresh.  In many
      cases, these registrations will need to take place nearly one
      hundred times more frequently than the typical refresh interval of
      a registration.  This introduces load into the system and hampers
      scalability.


I have had issues with some SIP Phones + NAT, while on the same environment other SIP Phones were working as expected, depending on how they sent the REGISTER messages and re-INVITES. Finally, we had it working by tuning the options I mentioned.

[Junaid]

Thanks hsujdik. No issue I reverted it quickly . Please find my responses below.

[i]Perhaps there was another REGISTER between the BYE and the INVITE? That's the only thing I can think of.[/i]. 

No REGISTER found between them.

[i]Do you have traces from the current version vs. old version?[/i]

Currently I have only traces of new version

[i]Another thing to look on NAT environment is if RPORT parameter is present on Via header of the SIP messages
[/i]

Via: SIP/2.0/UDP 192.168.104.204:5080;branch=z9hG4bK10B0CCAE-5F20-4BFF-82EB-BB8903782782-375248
Transport: UDP
Sent-by Address: 192.168.104.204
Sent-by port: 5080
Branch: z9hG4bK10B0CCAE-5F20-4BFF-82EB-BB8903782782-375248

I can't see RPORT in via header. That's mean no remapping of source port from firewall end right?

Regards,

[hsujdik]

Not exactly that, but almost that...

RPORT means that the SIP Proxy (SIP Server in our case) needs to reply to the port where the request was originated. The firewall will usually perform this port translation when NATing, so it is almost always required (unless it is a SBC performing the NAT?).

For instance, the old IWS SIP Endpoint in our environment sends REGISTER messages like this:

REGISTER sip:xxx.xx.xx.xx:5060;transport=udp SIP/2.0
Via: SIP/2.0/ ;branch=z9hG4bK-d8754z-bf44af607bf625aa-1---d8754z-;rport
Max-Forwards: 70
Contact: <sip:310193;rinstance=b3e772935e03e172;transport=udp>
To: <sip:310193@xxx.xx.xx.xx:5060>
From: <sip:310193@xxx.xx.xx.xx:5060>;tag=b419b9d8
Call-ID: ZTQ1OWExZjJkNDQzNmJjMzhhYWMzNjA3YjVmOWRjYTY.
CSeq: 1 REGISTER
Expires: 30
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE, SUBSCRIBE, INFO
User-Agent: Genesyslab.Sip.Endpoint 8.1.100.5 F-PD-WXP32-VM13.us.int.genesyslab.com-112012-214352  Genesyslab.Sip.Endpoint.Provider.CP 8.1.100.5 F-PD-WXP32-VM13.us.int.genesyslab.com-112012-214352
Content-Length: 0

As you can see, the Expires header is very low so it will keep refreshing the NAT table, and the RPORT is present so SIP Server will know where to reply.

Well, those suggestions are the ones I can think about, considering what I have performed on our environment... It may not be applicable to your deployment.

[cavagnaro]

You sure your Firewall supports NAT for SIP?

[Junaid]

it's juniper SRX. I don't know exactly the model/OS as it's place on client side. however I searched that it support SIP. 

[u]https://www.juniper.net/documentation/en_US/junos15.1x49/topics/concept/alg-security-sip-understanding.html[/u]


@hsujdik

I got your point but it seems entirely different behavior in our case. BTW where I have to configure RPORT parameter? I want to check its impact.