[help] can datasourcer be throttled?

[shkei]

Good evening. I am scratching my head for a solution to this problem. I would think Genesys would have thought of this yet I fail to find a resolution to my peculiar problem.

Datasourcer. Datasourcer is sending over 500,000 requests to stat server and due to very antiquated equipment it physically kills our CPU. We have stat server machine CPU at 100% for any period of 30 seconds to 5 minutes every 15 minutes whenever datasourcer starts accessing those stats.

Can someone please tell me why oh why god would we need 500,000 stats takes at exactly the same time? One would think that with fifteen minutes till the next reporting interval datasourcer would spread its requests for the previous 15 minute interval. At least there would be some sort of option that would allow me to throttle how many requests it would issue per minute.

Is there a way to throttle it down?

[glow=yellow,2,300]To Genesys[/glow]: you need to think this one through. I need an option to throttle the request volume from datasourcer to statserver, because if 4-cpu sun with a sole statserver locks up there is a major flaw there. Please add it in your next release!

Has anyone done any performance testing for statserver 6.5, 7.1 and 7.2? Were there significant changes in processing speed?

[ecki]

Interesting... The server looks powerful enough. Have you checked the logging options of all Genesys apps running on this server? Check especially the option 'buffering'. It should be set to true or yes(not sure). If not try to change it, and see what happens. Sometimes apps can kill server with continuous writing of huge amount log data to disk.

e.

[Fra]

Shkei,

what about the amount of RAM? Is there only that StatServer running on that machine? Logs are set to (DebugLevel, verbose, buffering)? Is 500'000 a real number? Remember that if you open two statistic based on same stattype but different filters, then it will be two separate statistics. Last but not least keep in mind that StatServer is single-threaded..

I'd suggest anyway to upgrade to 7.x (I sound like Genesys tech support guys who advice to upgrade in any case...  ), since 7.x releases are much better with handling the CPU load and memory.

p.s. :
According to Genesys:
-StatServer 6.5 would need : MemoryReqd = 50 + (NStatistics x 0.001)  MB
-StatServer 7.0 would need : MemoryReqd = 100 + (NStatistics x 0.0012) MB

[Shkei]

Thanks for the info!

Looking at the stats I cant help but wonder: why does 7.x handle memory better if it requires more memory from the start and uses 20% more per transaction?

[Shkei]

[quote author=ecki link=topic=1769.msg5606#msg5606 date=1154404281]
Interesting... The server looks powerful enough. Have you checked the logging options of all Genesys apps running on this server? Check especially the option 'buffering'. It should be set to true or yes(not sure). If not try to change it, and see what happens. Sometimes apps can kill server with continuous writing of huge amount log data to disk.

e.
[/quote]

The problem is that we have several thousand agents active at any one time they are operating as autonomous groups meaning many kayouts with many requests. I found out that statserver is a single-threaded process and thus it does not matter if we have 1 CPU or 4. ssbackup file is another problem because when it generates it and it coincides with polling time for datasourcer statserver really slows down to a crawl if even that.

we must have a way to throttle the requests. this is the only logical way out.

[Fra]

Shkei wrote:
[quote]why does 7.x handle memory better if it requires more memory from the start and uses 20% more per transaction?[/quote]

Yeah, it requires more memory per statistic.
1.But are you running out memory? Can you check how high it is?
2.They have removed some defects in memory usage in StatServer 6.5 release. Could you post the exact release you are running?
3. Have you checked the logs options?
4. I assumed your db is not running on the same machine, am I right?
5. How big is your ssbackup?? There are some internal problems on a 6.5.x releases that bring SS to loop and to consume up to 100% of CPU

[victor]

[quote author=Fra link=topic=1769.msg5617#msg5617 date=1154439954]
Shkei wrote:
[quote]why does 7.x handle memory better if it requires more memory from the start and uses 20% more per transaction?[/quote]

Yeah, it requires more memory per statistic.
1.But are you running out memory? Can you check how high it is?
2.They have removed some defects in memory usage in StatServer 6.5 release. Could you post the exact release you are running?
3. Have you checked the logs options?
4. I assumed your db is not running on the same machine, am I right?
5. How big is your ssbackup?? There are some internal problems on a 6.5.x releases that bring SS to loop and to consume up to 100% of CPU
[/quote]

I can actually relate to Shkei here, because we are having a somewhat of a similar problem.
One of our sites that has over 2000 agents is constantly having trouble with its CCA statserver.

Statserver version 6.5.102.26 and with about 300,000 request from datasourcer the things simply freezes.
I checked CPU and CPU idle  levels drop to 20% and on an occasion or two it even hits 0%.

We are running on Suns with dual CPUs and enough memory to feed a small Polynesian island, yet once in two weeks, StatServer disconnected from T-Server with "client too slow".

I checked the network and it is operating at under 5% capacity even at peak times, so, it rules out the connection.
I then checked Solaris memory and noticed that statserver was consuming around 900 Megs, which is ok considering the specs.
I then looked at the files and noticed a huge 150MB ssbackup.00 and it hit me! When statserver spwes ssbackup.00 - it really freezes. Now, if you have your datasourcer "shock and awe" bombardment  take place at the same time, you are really doomed.

I changed the parameters specifying how often ssbackup is dumped and it significantly improved statserver performance, yet, we still get an occasional disconnect when ssbackup coincides with datasourcer bombardment.  The problem is that statserver is a single-task application.

So, unless it is doing so already, it should:

1. have an option for throttling requests from datasourcer
2. spawn a separate process for writing down ssbackup.000 file
I also think it is simply ridiculous that we cannot have a fullblown log because it affects the performance. Having a separate thread doing just that would not slow down machine performance on a multi-CPU environment. So, I add
3. if not doing so already, have logging as a separate process

StatServer 7.x is faster, really? Anyone has done any studies to confirm? I am hesitant to upgrade to 7.x our historical reporting because we would need to check all stats and this will take time...

[ecki]

Well, I did not know about StatServer's Achilles tendon.  , thanks guys!
That is really not quiet nice from developers.
But still there may be one possible workaround.
Have you ever tried to change the 15min time interval for some DMA templates with shifted restart time e.g. 00:05+0:15..??
It could help to spread the traffic, though this is not the definitive solution.

e.

[Adam G.]

Lets take the situation of reporting on thousands of agents. Why not simulate a multi-threaded stat server by sharing the load across a number of instances?

By clever use of the Genesys Persons and the security settings you can restrict the agents the TServer sends events on, to the Stat Server (you'll have to change the user that Stat Server logs into TServer in CME).

I hope this makes sense (probably not though).

[Fra]

Victor,

the biggest customer of ours had a 6.5.x StatServer with hundreds of thousands of stats calculated; I can't give you specific performance data since I haven't supported them, but I can assure you StatServer 7.x is significantly faster. But the thing has settled things once and for all is the feature introduced in 7.0.1 DMA that allows you to select specific objects for data collection, so that for example you don't have to get SS calculate the stat X for all agent groups if you need it just for some of them:

Release Number 7.0.100.07 [05/07/2004] – General
New In This Release
    * [b]DMA now enables you to select specific objects with respect to the data collected during the layout design process.[/b]
    * DMA now allows you to select individual objects, in addition to metagroups, for report layouts at the Report Layout Object Editor.

In addition to that:

Release Number 7.0.000.15 [07/31/2003] – General
Corrections and Modifications
[b]Stat Server performance has been improved for calculation of statistics based on filters.[/b] (ER# 52695)

Release Number 7.2.002.06 [07/31/06] – Hot Fix
Corrections and Modifications
[b]Stat Server no longer consumes extra memory when writing data to a backup file.[/b] Previous releases of Stat Server experienced a memory leak when writing statistics' definitions to a backup file. (ER# 47309980; ER# 45178358)

And again: have you ever tried to stop StatServer, delete ssbackup and restart StatServer? Ssbackup may be corrupted and leak memory.

[victor]

Fra - thank you for your detailed reply! 

Regarding restarting RptStatserver:

after stopping rptstatserver I have deleted its ssbackup.000 file and restarted it. The file grew back to the same size within 15 minutes.

I changed the interval for ssbackup.000 generation from 15 minutes to 50 minutes. This dramatically decreased CPU usage, yet, it is still a pain in ... well, you know.

I would love to switch to 7.2, but:

1. statistic definitions have changed between 6.5 and 7.0 (use of DNAction as opposed to AgentStatus) so it will affect several statistics. Without clear undertstanding of what will be changed and how, I am very reluctant to switch, since SVs at that site will demand fresh blood if their stats suddenly change without prior notice. (Notice would require telling them how things will change and it takes a long time to review and check everything  I wish Genesys had some simple pamphlet we could just distribute.

2. to avoid losing data if 7.2 proves to be unstable, I would need to ensure that we can always have something to go back to. So we are talking about running two datamarts : new one with data imported from the old one and old one. Buahhh. And of course, we would also use this to ask client to upgrade CCA to 7.2 from 6.5

(Wait, does 7.2 supports Japanese yet?)

So, with all this said, I will take your advice and push for upgrade.
Does anyone have a list of what stats change between 6.5 and 7.2?

Thanks,
Vic

P.S. I am afraid that I took this thread away from Shkei problem. Should I post this as a separate thread?