Dears,
I had recently upgraded the GAX 8.1 present in the BEP to GAX 8.5.1 and Pulse 8.5.1. We have HA environment where there are two instances of GAX running. Windows NLB is enabled to achieve load balancing in order to access the GAX using virtual IP.
Current versions are
GAX - 8.5.230.03
Pulse - 8.5.105.01
PulseCollector - 8.5.105.03
SS - 8.5.104.24
The GAX interface gets hanged while working on it for long or while pulse plugin is working. Suddenly the page starts loading and nothing appears after that. Now today when i was using the Pulse and when i tried to edit a widget, it got into the hang state. On checking the resource monitor of the host machine, it shows high usage of CPU during that time. To get out of this hang state, i restart GAX and then everything comes back to normal. Sometimes during the hang state the GAX throws
During the hang state, GAX interface and the resource screenshots are as follows
[img width=640 height=350]http://i.imgur.com/SOEyltF.png[/img]
[img width=495 height=480]http://i.imgur.com/tLnOyoL.png[/img]
[img width=640 height=337]http://i.imgur.com/9FLFM23.png[/img]
The GAX log starting from login to the issue period is below.
2016-11-30 12:39:11,784 INFO : com.genesyslab.gax.webservice.UserController [qtp523155230-46] - [118] Client:10.1.144.162 try to login.
2016-11-30 12:39:11,800 DEBUG: com.genesyslab.gax.core.user.LoginServiceImpl [qtp523155230-46] - [50] Log in user pulse
2016-11-30 12:39:11,800 DEBUG: com.genesyslab.gax.core.connection.ConfServiceProxyHandler [qtp523155230-46] - [107] *Request new connection
2016-11-30 12:39:11,800 WARN : com.genesyslab.gax.core.connection.ConnectorInternal [qtp523155230-46] - [142] Configuration server host 10.1.144.92 is different from gax configuration host g1-core-p
2016-11-30 12:39:11,800 INFO : com.genesyslab.gax.core.connection.ConnectorInternal [qtp523155230-46] - [271] Warmstandby.getstate: Idle
2016-11-30 12:39:11,800 INFO : com.genesyslab.gax.core.connection.ConnectorInternal [qtp523155230-46] - [272] Warmstandby service started
2016-11-30 12:39:11,800 INFO : com.genesyslab.gax.core.connection.ConnectorInternal [com.genesyslab.PCT.invoker.warmstandby] - [333] Config Server warmstandby state changed to Idle
2016-11-30 12:39:11,800 INFO : com.genesyslab.gax.core.connection.ConnectorInternal [qtp523155230-46] - [381] Opening connection to endpoint 1g2oomz34thwy1dptwryr4ppi0a - tcp://10.1.144.92:8888
2016-11-30 12:39:11,800 INFO : com.genesyslab.gax.core.connection.ConnectorInternal [qtp523155230-46] - [382] Trusted storage location: null
2016-11-30 12:39:11,800 INFO : com.genesyslab.gax.core.connection.ConnectorInternal [qtp523155230-46] - [383] TLS service properly initialized: false
2016-11-30 12:39:11,800 INFO : com.genesyslab.gax.core.connection.ConnectorInternal [qtp523155230-46] - [384] TLS Authentication mandatory: false
2016-11-30 12:39:11,893 INFO : com.genesyslab.gax.core.connection.ConnectorInternal [com.genesyslab.PCT.invoker.default] - [353] Config server (tcp://1g2oomz34thwy1dptwryr4ppi0a@10.1.144.92:8888) channel opened event. Endpoint: 1g2oomz34thwy1dptwryr4ppi0a - tcp://10.1.144.92:8888
2016-11-30 12:39:11,893 INFO : com.genesyslab.gax.core.connection.ConnectorInternal [com.genesyslab.PCT.invoker.default] - [374] Notifying waiting thread...
2016-11-30 12:39:11,909 DEBUG: com.genesyslab.gax.core.user.UserDetailsServiceImpl [qtp523155230-46] - [148] Loading role privileges for user pulse from role: PulseRoles
2016-11-30 12:39:11,909 DEBUG: com.genesyslab.gax.core.user.LoginServiceImpl [qtp523155230-46] - [78] Successfully authenticated.
2016-11-30 12:39:11,909 DEBUG: com.genesyslab.gax.core.user.LoginServiceImpl [qtp523155230-46] - [79] Security Context: org.springframework.security.web.authentication.preauth.PreAuthenticatedAuthenticationToken@1d4560b: Principal: com.genesyslab.gax.common.user.UserCustom@6611b99: Username: pulse; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: PRIVILEGE_GAPT_ACCESS_USER,PRIVILEGE_PULSE_NO_WIDGETS_LIMIT,PRIVILEGE_PULSE_READ,PRIVILEGE_PULSE_READ_ALL_LAYOUTS,PRIVILEGE_PULSE_READ_RESTRICTED,PRIVILEGE_PULSE_SHARE_TAB,PRIVILEGE_PULSE_WRITE_LAYOUT,PRIVILEGE_PULSE_WRITE_SNAPSHOT,PRIVILEGE_PULSE_WRITE_TAB,PRIVILEGE_PULSE_WRITE_TEMPLATE,PRIVILEGE_PULSE_WRITE_WIDGET,PRIVILEGE_READ_SYSTEM_PREFERENCES; Credentials: [PROTECTED]; Authenticated: true; Details: null; Granted Authorities: PRIVILEGE_GAPT_ACCESS_USER, PRIVILEGE_PULSE_NO_WIDGETS_LIMIT, PRIVILEGE_PULSE_READ, PRIVILEGE_PULSE_READ_ALL_LAYOUTS, PRIVILEGE_PULSE_READ_RESTRICTED, PRIVILEGE_PULSE_SHARE_TAB, PRIVILEGE_PULSE_WRITE_LAYOUT, PRIVILEGE_PULSE_WRITE_SNAPSHOT, PRIVILEGE_PULSE_WRITE_TAB, PRIVILEGE_PULSE_WRITE_TEMPLATE, PRIVILEGE_PULSE_WRITE_WIDGET, PRIVILEGE_READ_SYSTEM_PREFERENCES
2016-11-30 12:39:12,299 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-46] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = ksvkp4h94xrtl1b0tx0ajq0v37
2016-11-30 12:39:15,481 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-50] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b563-3c2d730e matching filter {id=000c29987dbb-8c2c-11e6-b563-3c2d730e, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\4f\000c29987dbb-8c2c-11e6-b563-3c2d730e\ using file system
2016-11-30 12:39:15,481 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-48] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b562-9e9b7798 matching filter {id=000c29987dbb-8c2c-11e6-b562-9e9b7798, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\65\000c29987dbb-8c2c-11e6-b562-9e9b7798\ using file system
2016-11-30 12:39:15,481 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-48] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\65\000c29987dbb-8c2c-11e6-b562-9e9b7798\000c29987dbb-8c2c-11e6-b562-9e9b7798_1480498754.gpb: 0.000 seconds
2016-11-30 12:39:15,481 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-48] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b562-9e9b7798 in a folder C:\GCTI\PulseCollector_backup\.\output\65\000c29987dbb-8c2c-11e6-b562-9e9b7798\ using file system
2016-11-30 12:39:15,481 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-48] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b562-9e9b7798 and layout type ltPCREGULAR
2016-11-30 12:39:15,481 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-48] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:15,481 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-51] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b562-bb9279ca matching filter {id=000c29987dbb-8c2c-11e6-b562-bb9279ca, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\69\000c29987dbb-8c2c-11e6-b562-bb9279ca\ using file system
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-51] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\69\000c29987dbb-8c2c-11e6-b562-bb9279ca\000c29987dbb-8c2c-11e6-b562-bb9279ca_1480498754.gpb: 0.000 seconds
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-51] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b562-bb9279ca in a folder C:\GCTI\PulseCollector_backup\.\output\69\000c29987dbb-8c2c-11e6-b562-bb9279ca\ using file system
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-51] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b562-bb9279ca and layout type ltPCREGULAR
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-48] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b562-9e9b7798
2016-11-30 12:39:15,481 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-49] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b562-fd48871c matching filter {id=000c29987dbb-8c2c-11e6-b562-fd48871c, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\07\000c29987dbb-8c2c-11e6-b562-fd48871c\ using file system
2016-11-30 12:39:15,481 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-50] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\4f\000c29987dbb-8c2c-11e6-b563-3c2d730e\000c29987dbb-8c2c-11e6-b563-3c2d730e_1480498754.gpb: 0.000 seconds
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-50] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b563-3c2d730e in a folder C:\GCTI\PulseCollector_backup\.\output\4f\000c29987dbb-8c2c-11e6-b563-3c2d730e\ using file system
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-51] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-46] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b563-921ccf00 matching filter {id=000c29987dbb-8c2c-11e6-b563-921ccf00, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\46\000c29987dbb-8c2c-11e6-b563-921ccf00\ using file system
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-45] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b564-18c9e3d3 matching filter {id=000c29987dbb-8c2c-11e6-b564-18c9e3d3, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\73\000c29987dbb-8c2c-11e6-b564-18c9e3d3\ using file system
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-50] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b563-3c2d730e and layout type ltPCREGULAR
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-50] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-45] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\73\000c29987dbb-8c2c-11e6-b564-18c9e3d3\000c29987dbb-8c2c-11e6-b564-18c9e3d3_1480498754.gpb: 0.000 seconds
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-45] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b564-18c9e3d3 in a folder C:\GCTI\PulseCollector_backup\.\output\73\000c29987dbb-8c2c-11e6-b564-18c9e3d3\ using file system
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-50] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b563-3c2d730e
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-49] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\07\000c29987dbb-8c2c-11e6-b562-fd48871c\000c29987dbb-8c2c-11e6-b562-fd48871c_1480498754.gpb: 0.000 seconds
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-49] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b562-fd48871c in a folder C:\GCTI\PulseCollector_backup\.\output\07\000c29987dbb-8c2c-11e6-b562-fd48871c\ using file system
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-49] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b562-fd48871c and layout type ltPCREGULAR
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-49] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-45] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b564-18c9e3d3 and layout type ltPCREGULAR
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-49] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b562-fd48871c
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-51] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b562-bb9279ca
2016-11-30 12:39:15,497 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-46] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\46\000c29987dbb-8c2c-11e6-b563-921ccf00\000c29987dbb-8c2c-11e6-b563-921ccf00_1480498754.gpb: 0.000 seconds
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-46] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b563-921ccf00 in a folder C:\GCTI\PulseCollector_backup\.\output\46\000c29987dbb-8c2c-11e6-b563-921ccf00\ using file system
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-45] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-46] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b563-921ccf00 and layout type ltPCREGULAR
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-46] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-45] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b564-18c9e3d3
2016-11-30 12:39:15,513 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-46] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b563-921ccf00
2016-11-30 12:39:16,137 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-51] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = ltwlwb0z5hwup18xr7e00em0wu
2016-11-30 12:39:25,419 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-46] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b563-3c2d730e matching filter {id=000c29987dbb-8c2c-11e6-b563-3c2d730e, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\4f\000c29987dbb-8c2c-11e6-b563-3c2d730e\ using file system
2016-11-30 12:39:25,419 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-46] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\4f\000c29987dbb-8c2c-11e6-b563-3c2d730e\000c29987dbb-8c2c-11e6-b563-3c2d730e_1480498764.gpb: 0.000 seconds
2016-11-30 12:39:25,419 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-46] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b563-3c2d730e in a folder C:\GCTI\PulseCollector_backup\.\output\4f\000c29987dbb-8c2c-11e6-b563-3c2d730e\ using file system
2016-11-30 12:39:25,419 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-46] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b563-3c2d730e and layout type ltPCREGULAR
2016-11-30 12:39:25,419 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-46] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:25,419 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-46] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b563-3c2d730e
2016-11-30 12:39:25,419 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-48] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b562-9e9b7798 matching filter {id=000c29987dbb-8c2c-11e6-b562-9e9b7798, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\65\000c29987dbb-8c2c-11e6-b562-9e9b7798\ using file system
2016-11-30 12:39:25,434 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-48] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\65\000c29987dbb-8c2c-11e6-b562-9e9b7798\000c29987dbb-8c2c-11e6-b562-9e9b7798_1480498764.gpb: 0.000 seconds
2016-11-30 12:39:25,434 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-48] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b562-9e9b7798 in a folder C:\GCTI\PulseCollector_backup\.\output\65\000c29987dbb-8c2c-11e6-b562-9e9b7798\ using file system
2016-11-30 12:39:25,434 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-48] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b562-9e9b7798 and layout type ltPCREGULAR
2016-11-30 12:39:25,434 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-48] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:25,434 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-48] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b562-9e9b7798
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-50] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b562-bb9279ca matching filter {id=000c29987dbb-8c2c-11e6-b562-bb9279ca, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\69\000c29987dbb-8c2c-11e6-b562-bb9279ca\ using file system
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-50] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\69\000c29987dbb-8c2c-11e6-b562-bb9279ca\000c29987dbb-8c2c-11e6-b562-bb9279ca_1480498764.gpb: 0.000 seconds
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-50] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b562-bb9279ca in a folder C:\GCTI\PulseCollector_backup\.\output\69\000c29987dbb-8c2c-11e6-b562-bb9279ca\ using file system
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-50] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b562-bb9279ca and layout type ltPCREGULAR
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-50] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-50] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b562-bb9279ca
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-49] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b562-fd48871c matching filter {id=000c29987dbb-8c2c-11e6-b562-fd48871c, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\07\000c29987dbb-8c2c-11e6-b562-fd48871c\ using file system
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-49] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\07\000c29987dbb-8c2c-11e6-b562-fd48871c\000c29987dbb-8c2c-11e6-b562-fd48871c_1480498764.gpb: 0.000 seconds
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-49] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b562-fd48871c in a folder C:\GCTI\PulseCollector_backup\.\output\07\000c29987dbb-8c2c-11e6-b562-fd48871c\ using file system
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-49] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b562-fd48871c and layout type ltPCREGULAR
2016-11-30 12:39:25,450 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-49] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:25,465 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-49] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b562-fd48871c
2016-11-30 12:39:25,465 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-51] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b564-18c9e3d3 matching filter {id=000c29987dbb-8c2c-11e6-b564-18c9e3d3, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\73\000c29987dbb-8c2c-11e6-b564-18c9e3d3\ using file system
2016-11-30 12:39:25,465 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-51] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\73\000c29987dbb-8c2c-11e6-b564-18c9e3d3\000c29987dbb-8c2c-11e6-b564-18c9e3d3_1480498764.gpb: 0.000 seconds
2016-11-30 12:39:25,465 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-51] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b564-18c9e3d3 in a folder C:\GCTI\PulseCollector_backup\.\output\73\000c29987dbb-8c2c-11e6-b564-18c9e3d3\ using file system
2016-11-30 12:39:25,465 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-51] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b564-18c9e3d3 and layout type ltPCREGULAR
2016-11-30 12:39:25,465 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-51] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:25,481 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-45] - [95] Read snapshots for layout 000c29987dbb-8c2c-11e6-b563-921ccf00 matching filter {id=000c29987dbb-8c2c-11e6-b563-921ccf00, extPattern=(\.lz4)|(\.gpb), start=null, end=null} in a folder C:\GCTI\PulseCollector_backup\.\output\46\000c29987dbb-8c2c-11e6-b563-921ccf00\ using file system
2016-11-30 12:39:25,481 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-51] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b564-18c9e3d3
2016-11-30 12:39:25,481 DEBUG: com.genesyslab.gax.wbrt.common.utils.Duration [qtp523155230-45] - [38] Read Snapshot from file C:\GCTI\PulseCollector_backup\.\output\46\000c29987dbb-8c2c-11e6-b563-921ccf00\000c29987dbb-8c2c-11e6-b563-921ccf00_1480498764.gpb: 0.000 seconds
2016-11-30 12:39:25,481 DEBUG: com.genesyslab.gax.wbrt.dao.AbstractSnapshotDao [qtp523155230-45] - [125] 1 snapshots are read for layout 000c29987dbb-8c2c-11e6-b563-921ccf00 in a folder C:\GCTI\PulseCollector_backup\.\output\46\000c29987dbb-8c2c-11e6-b563-921ccf00\ using file system
2016-11-30 12:39:25,481 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-45] - [531] Start filtering rows and columns for snapshot with layout id 000c29987dbb-8c2c-11e6-b563-921ccf00 and layout type ltPCREGULAR
2016-11-30 12:39:25,481 DEBUG: com.genesyslab.gax.wbrt.service.LayoutServiceImpl [qtp523155230-45] - [565] User has read all layouts privilege so skip filtration of rows
2016-11-30 12:39:25,481 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-45] - [155] Snapshot loaded for layout: 000c29987dbb-8c2c-11e6-b563-921ccf00
2016-11-30 12:39:27,431 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-48] - [64] Widget with id = 000c29987dbb-8c2c-11e6-b562-bb9e60ab loaded
2016-11-30 12:39:27,462 DEBUG: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-49] - [91] Layout loaded:000c29987dbb-8c2c-11e6-b562-bb9279ca
2016-11-30 12:39:29,163 ERROR: com.genesyslab.gax.wbrt.web.EZPulseController [qtp523155230-49] - [80] Could not write JSON: java.lang.reflect.InvocationTargetException-->null; nested exception is org.codehaus.jackson.map.JsonMappingException: java.lang.reflect.InvocationTargetException-->null - request URI: /gax/api/wbrt/layouts/000c29987dbb-8c2c-11e6-b562-bb9279ca
2016-11-30 12:42:15,318 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-48] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = m1xuh5p8tkoj0bpvki6rql4qz
2016-11-30 12:42:37,439 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-48] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = n13g5ok421oain17ate22r285zu
2016-11-30 12:42:59,201 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-45] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = oi03g3gvgho0od1whvz1qx1lx
2016-11-30 12:45:20,412 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-48] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = p94y273c63q7i184skd8tiurce
2016-11-30 12:45:31,675 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-48] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = q1t64coiwuhfal12hkl5tx6yqxl
2016-11-30 12:46:38,833 DEBUG: com.genesyslab.gax.wbrt.service.SnapshotCleaner [scheduledExecutorFactoryBean-1] - [30] Snapshot cleaning is disabled.
2016-11-30 12:46:40,425 WARN : org.cometd.websocket.server.WebSocketTransport [pool-2-thread-1] - [134]
2016-11-30 12:46:48,318 DEBUG: com.genesyslab.gax.core.configuration.warmstandby.utils.GaxWarmStandbyListener [com.genesyslab.PCT.invoker.warmstandby] - [51] WarmStandby switched channel to tcp://SCS@10.1.144.101:8010.
WarmStandby state: Waiting
Channel state: Closed
2016-11-30 12:46:50,612 DEBUG: com.genesyslab.gax.core.configuration.warmstandby.utils.GaxWarmStandbyListener [com.genesyslab.PCT.invoker.warmstandby] - [36] WarmStandby changed state to Reconnecting.
Endpoint uri: tcp://SCS@10.1.144.101:8010
State:Opening
2016-11-30 12:46:52,515 DEBUG: com.genesyslab.gax.core.configuration.warmstandby.utils.GaxWarmStandbyListener [com.genesyslab.PCT.invoker.warmstandby] - [36] WarmStandby changed state to Reconnecting.
Endpoint uri: tcp://SCS@10.1.144.101:8010
State:Opening
2016-11-30 12:47:33,236 DEBUG: com.genesyslab.gax.core.configuration.warmstandby.utils.GaxWarmStandbyListener [com.genesyslab.PCT.invoker.warmstandby] - [51] WarmStandby switched channel to tcp://SCS@10.1.144.92:8010.
WarmStandby state: Waiting
Channel state: Closed
2016-11-30 12:47:42,301 DEBUG: com.genesyslab.gax.core.configuration.warmstandby.utils.GaxWarmStandbyListener [com.genesyslab.PCT.invoker.warmstandby] - [36] WarmStandby changed state to Reconnecting.
Endpoint uri: tcp://SCS@10.1.144.92:8010
State:Opening
2016-11-30 12:47:42,862 DEBUG: com.genesyslab.gax.core.configuration.warmstandby.utils.GaxWarmStandbyListener [com.genesyslab.PCT.invoker.warmstandby] - [36] WarmStandby changed state to Reconnecting.
Endpoint uri: tcp://SCS@10.1.144.92:8010
State:Opening
2016-11-30 12:49:09,361 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-46] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = r1t9md3zapzg1aypkwvmmqfjbi
2016-11-30 12:49:30,951 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-48] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = scrh3ol7fdxsll91wrtta8pqz
2016-11-30 12:50:50,529 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-45] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = t1gj6i7tv1i8u4ffrslwwr3y61
2016-11-30 12:51:21,837 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-48] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = u1xghb9dlqx2df19gpnmso7n4x0
2016-11-30 12:51:55,610 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-50] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = v1a8znx3x6mibgkqikc4u6y850
2016-11-30 12:53:39,770 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-46] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = wcq9f5u3vag9n1cwx9wpflv2b7
2016-11-30 12:54:39,565 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-45] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = xuevvuq5bjw142powf8422vdj
2016-11-30 12:55:28,705 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-45] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = y5wmupe185cry1mqembodg6mat
2016-11-30 12:55:56,894 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-71] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = zv98feha3fw9b17kqb4e9tjc0g
2016-11-30 12:56:01,262 DEBUG: com.genesyslab.gax.wbrt.queue.RabbitSubscriber [SimpleAsyncTaskExecutor-1] - [121] Message received for layout with id = 10
2016-11-30 12:56:03,322 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-45] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 1011rbku22ram5a1hlv57b258jc9
2016-11-30 12:56:09,656 TRACE: com.genesyslab.gax.core.web.notification.CometEventService [SimpleAsyncTaskExecutor-1] - [55] CometD: Number of Subscribers=0
2016-11-30 12:56:37,035 DEBUG: com.genesyslab.gax.wbrt.queue.CometPublisher [SimpleAsyncTaskExecutor-1] - [44] Message published to channel: /wbrt/snapshots/000c29987dbb-8c2c-11e6-b563-921ccf00
2016-11-30 12:57:12,930 DEBUG: com.genesyslab.gax.wbrt.queue.RabbitSubscriber [SimpleAsyncTaskExecutor-1] - [121] Message received for layout with id = 10
2016-11-30 12:57:18,062 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-46] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 11ud4rdpyhdmltpe3jnc7tgrwf
2016-11-30 12:58:05,018 TRACE: com.genesyslab.gax.core.web.notification.CometEventService [SimpleAsyncTaskExecutor-1] - [55] CometD: Number of Subscribers=0
2016-11-30 12:58:05,018 DEBUG: com.genesyslab.gax.wbrt.queue.CometPublisher [SimpleAsyncTaskExecutor-1] - [44] Message published to channel: /wbrt/snapshots/000c29987dbb-8c2c-11e6-b563-921ccf00
2016-11-30 12:58:27,903 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-48] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 12fkg1mnummwq0j5ckoez4ej6a
2016-11-30 12:59:19,429 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-50] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 1312z5a7paihh02cqtvwxbvqfjv
2016-11-30 12:59:55,777 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-48] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 14fyz5wga8i69fi0gil57k7e8y
2016-11-30 13:00:01,627 DEBUG: com.genesyslab.gax.wbrt.queue.RabbitSubscriber [SimpleAsyncTaskExecutor-1] - [121] Message received for layout with id = 10
2016-11-30 13:00:09,630 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-45] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 151pexd4lr8uj421ny30ec3rtr0l
2016-11-30 13:00:17,539 TRACE: com.genesyslab.gax.core.web.notification.CometEventService [SimpleAsyncTaskExecutor-1] - [55] CometD: Number of Subscribers=0
2016-11-30 13:00:20,768 DEBUG: com.genesyslab.gax.wbrt.queue.CometPublisher [SimpleAsyncTaskExecutor-1] - [44] Message published to channel: /wbrt/snapshots/000c29987dbb-8c2c-11e6-b563-921ccf00
2016-11-30 13:00:36,758 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-46] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 163ogylfgmwb2hnmpuraic0mx5
2016-11-30 13:01:27,053 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-50] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 17zs4trms5dqqg1g4m4atchu7c6
2016-11-30 13:02:45,037 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-48] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 1812a3id0ueiy9p1jxsz7wi912zr
2016-11-30 13:02:52,993 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-46] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 191ebmjdqw9p59p1igjijt7jgtmv
2016-11-30 13:03:17,282 DEBUG: com.genesyslab.gax.core.web.notification.CometSecurityPolicy [qtp523155230-50] - [41] canHandshake: username = pulse, HTTP.sessionId = 1g2oomz34thwy1dptwryr4ppi0, Bayeux.sessionId = 1a1i577qnjm7xmm15rp2nrrsywd6
