We are getting below alarm
Er_ORS_Tomcat_N11, Application terminated due to internal condition
If we check on the machine we see that this application is running fine since last 25 days.
If we check the LCA logs on on this host we see the below:
04:02:44 Found(1) PID=14384 for ThirdPartyApp{887,'Er_ORS_Tomcat_N11',cmdline='/bin/ksh /appl/genesys/tomcat/bin/tomcat.sh '}
04:02:44 CREATE Application <887, Er_ORS_Tomcat_N11, pid=14384, socket=-1, confserv=:>.
04:02:44 Notify STATUS: App<887, Er_ORS_Tomcat_N11, pid=14384, APP_STATUS_RUNNING, PRIMARY.
04:02:44 ~ThirdPartyApp<887, Er_ORS_Tomcat_N11>
04:02:45 #### OS Level STOPPED: App<887, Er_ORS_Tomcat_N11, pid=14384>
04:02:45 Notify STATUS: App<887, Er_ORS_Tomcat_N11, pid=14384, APP_STATUS_STOPPED, PRIMARY.
04:02:45 CREATE ThirdPartyApp <887, Er_ORS_Tomcat_N11, confserv=:>.
04:02:45 ~Application<14384, Er_ORS_Tomcat_N11>
04:02:45 Ml_SoCoSrv_N11: 04:02:45 [RequestAppStatus] on App 'Er_ORS_Tomcat_N11' <pid=14384, dbid=887>
04:02:45 Found by AppID=887
04:02:45 Notify STATUS: App<887, Er_ORS_Tomcat_N11, pid=26393, APP_STATUS_RUNNING, PRIMARY.
04:02:45 Ml_SoCoSrv_Z11: 04:02:45 [RequestAppStatus] on App 'Er_ORS_Tomcat_N11' <pid=14384, dbid=887>
04:02:45 Found by AppID=887
04:02:45 Notify STATUS: App<887, Er_ORS_Tomcat_N11, pid=26393, APP_STATUS_RUNNING, PRIMARY.
04:02:45 Ml_SoCoSrv_Z11: 04:02:45 [RequestAppStartupInfoChanged] on App 'Er_ORS_Tomcat_N11' <pid=26393, dbid=887>
04:02:45 Ml_SoCoSrv_Z11: 04:02:45 [RequestAppHangupConf] for App 'Er_ORS_Tomcat_N11' <pid=26393, dbid=887>
04:02:45 Hangup detection is not enabled
04:02:54 Found(1) PID=26393 for ThirdPartyApp{887,'Er_ORS_Tomcat_N11',cmdline='/bin/ksh /appl/genesys/tomcat/bin/tomcat.sh '}
[u][b][b]If we see SCS logs[/b][/b][/u]
04:02:44.367 XXXXXX: REventAppLiveStatusChanged... APP_STATUS_RUNNING, mode=PRIMARY, app=Er_ORS_Tomcat_N11, PID=14384
04:02:44.394 04:02:44.394 [gsnmpalib]: Received MEvent (Successful,0,0) from managed server 'Hr_StatSrv_Z11' dbid = 105 sd = 65.
04:02:45.154 Trace XXXXXXX.xx.xxxx.com Ml_SoCoSrv_Z11 GCTI-43-30201 [Agent++] AgentXSlave: sending notify PDU (trap): (1.3.6.1.6.3.1.1.4.1.0)
04:02:45.154 ### LOCAL MESSAGE#=30201, '[Agent++] AgentXSlave: sending notify PDU (trap): (1.3.6.1.6.3.1.1.4.1.0)'
04:02:45.154 Trace XXX.xx.xxxx.com Ml_SoCoSrv_Z11 GCTI-43-30201 [Agent++] AgentX: sending agentx pdu (sd)(type)(sid)(tid)(pid)(err)(errind): (49), (12), (1), (1298911), (2378546), (0), (0)
04:02:45.154 ### LOCAL MESSAGE#=30201, '[Agent++] AgentX: sending agentx pdu (sd)(type)(sid)(tid)(pid)(err)(errind): (49), (12), (1), (1298911), (2378546), (0), (0)'
04:02:45.154 Trace XXXXXX.xx.xxxx.com Ml_SoCoSrv_Z11 GCTI-43-30201 [Agent++] AgentX: sending agentx pdu (sd)(type)(sid)(tid)(pid)(err)(errind): (49), (13), (1), (1298912), (2378548), (0), (0)
04:02:45.154 ### LOCAL MESSAGE#=30201, '[Agent++] AgentX: sending agentx pdu (sd)(type)(sid)(tid)(pid)(err)(errind): (49), (13), (1), (1298912), (2378548), (0), (0)'
[Agent++] AgentXSlave: received something on ports
04:02:45.155 ### LOCAL MESSAGE#=30202, '[Agent++] AgentXSlave: received something on ports'
04:02:45.155 Trace XXXXX.xx.xxxx.com Ml_SoCoSrv_Z11 GCTI-43-30201 [Agent++] AgentXRequestList: request received (context)(tid)(pid)(siz)(type)(err)(status): (), (1298911), (2378546), (2), (18), (0), (0)
04:02:45.155 ### LOCAL MESSAGE#=30201, '[Agent++] AgentXRequestList: request received (context)(tid)(pid)(siz)(type)(err)(status): (), (1298911), (2378546), (2), (18), (0), (0)'
[Agent++] SubAgentXMib: unhandled response (tid)(pid): (1298911), (2378546)
04:02:45.156 ### LOCAL MESSAGE#=30202, '[Agent++] SubAgentXMib: unhandled response (tid)(pid): (1298911), (2378546)'
[Agent++] AgentXSlave: received something on ports
04:02:45.156 ### LOCAL MESSAGE#=30202, '[Agent++] AgentXSlave: received something on ports'
04:02:45.156 Trace XXXXXXX.xx.xxx.com Ml_SoCoSrv_Z11 GCTI-43-30201 [Agent++] AgentXRequestList: request received (context)(tid)(pid)(siz)(type)(err)(status): (), (1298912), (2378548), (0), (18), (0), (0)
04:02:45.156 ### LOCAL MESSAGE#=30201, '[Agent++] AgentXRequestList: request received (context)(tid)(pid)(siz)(type)(err)(status): (), (1298912), (2378548), (0), (18), (0), (0)'
04:02:45.156 Trace XXXXXXX Ml_SoCoSrv_Z11 GCTI-43-30201 [Agent++] SubAgentXMib: PING response received (sid)(tid)(pid)(uptime): (1), (1298912), (2378548), (92178567)
04:02:45.156 ### LOCAL MESSAGE#=30201, '[Agent++] SubAgentXMib: PING response received (sid)(tid)(pid)(uptime): (1), (1298912), (2378548), (92178567)'
04:02:45.367 xxxxx.xx.xxx.com: REventAppLiveStatusChanged... APP_STATUS_STOPPED, mode=EXITED, app=Er_ORS_Tomcat_N11, PID=14384
04:02:45.368 [gsnmpalib]: Managed server status changed 'Er_ORS_Tomcat_N11' dbid = 887 sd = -1: status = APP_STATUS_STOPPED.
04:02:45.368 'xxxxx.xx.xxx.com': NotifyApp: status= APP_STATUS_STOPPED mode=EXITED pid=14384 {887, Er_ORS_Tomcat_N11}
04:02:45.368 Standard xxxxx.xx.xxx.com Er_ORS_Tomcat_N11 GCTI-00-05064 Application terminated due to internal condition
04:02:45.368 ### LOCAL MESSAGE#=5064, 'Application terminated due to internal condition'
04:02:45.368 CREATE ActiveAlarm <107, Application terminated> for App=<887, Er_ORS_Tomcat_N11>, createTime=1502330565 under control=1
04:02:45.368 Trace XXXXXXX Ml_SoCoSrv_Z11 GCTI-43-10612 Active Alarm 'Application terminated' created for Application 'Er_ORS_Tomcat_N11'
04:02:45.368 ### LOCAL MESSAGE#=10612, 'Active Alarm 'Application terminated' created for Application 'Er_ORS_Tomcat_N11' '
04:02:45.368 Alarm XXXXXXX Er_ORS_Tomcat_N11 GCTI-00-04115 Application terminated, alarm condition(107)
04:02:45.368 ### LOCAL MESSAGE#=4115, 'Application terminated, alarm condition(107)'
04:02:45.368 Alarm XXXXXXX Er_ORS_Tomcat_N11 GCTI-00-04200 Active Alarm "Application terminated" Created (ID=003830D8-BEC5-198B-BDFD-3CB5150AAA77)
04:02:45.368 ### LOCAL MESSAGE#=4200, 'Active Alarm "Application terminated" Created (ID=003830D8-BEC5-198B-BDFD-3CB5150AAA77)'
04:02:45.368 ActiveAlarm::SetDeleteTimer <107, Application terminated> for App=<887, Er_ORS_Tomcat_N11>, LastSetTime=1502330565, timeout=86400
04:02:45.368 Found Condition <107, Application terminated>, Detect Message = 5064
04:02:45.368 !Execute all Reaction for AlarmCondition <107, Application terminated>
04:02:45.368 Trace XXXXXXX Ml_SoCoSrv_Z11 GCTI-43-10500 Reaction 'SNMP_Send_SNMP_Trap' activated
04:02:45.368 ### LOCAL MESSAGE#=10500, 'Reaction 'SNMP_Send_SNMP_Trap' activated'
04:02:45.368 #### App{887,Er_ORS_Tomcat_N11} status changed to APP_STATUS_STOPPED mode=EXITED, [prev=APP_STATUS_RUNNING]
04:02:45.368 Trace XXXXXXX Ml_SoCoSrv_Z11 GCTI-43-20620 [Application Er_ORS_Tomcat_N11]: LiveStatus changed to APP_STATUS_STOPPED from APP_STATUS_RUNNING
04:02:45.368 ### LOCAL MESSAGE#=20620, '[Application Er_ORS_Tomcat_N11]: LiveStatus changed to APP_STATUS_STOPPED from APP_STATUS_RUNNING'
04:02:45.368 Check RunMode for {'Er_ORS_Tomcat_N11', EXITED, APP_STATUS_STOPPED}
04:02:45.368 No check for ThirdPartyServer supported
04:02:45.368 XXXXXXX: REventAppLiveStatusChanged... APP_STATUS_RUNNING, mode=PRIMARY, app=Er_ORS_Tomcat_N11, PID=26393
04:02:45.368 [gsnmpalib]: Managed server status changed 'Er_ORS_Tomcat_N11' dbid = 887 sd = -1: status = APP_STATUS_RUNNING.
04:02:45.368 Trace XXXXXXX Ml_SoCoSrv_Z11 GCTI-43-30101 [SNMP Sub Agent] Cannot establish management connection with application Er_ORS_Tomcat_N11. Application does not provide for management support.
04:02:45.368 ### LOCAL MESSAGE#=30101, '[SNMP Sub Agent] Cannot establish management connection with application Er_ORS_Tomcat_N11. Application does not provide for management support.'
04:02:45.368 'XXXXXXX': NotifyApp: status= APP_STATUS_RUNNING mode=PRIMARY pid=26393 {887, Er_ORS_Tomcat_N11}
04:02:45.368 Standard XXXXXXX Er_ORS_Tomcat_N11 GCTI-00-05060 Application started
