I need help reducing the load of my StatServer

[victor]

[color=Green]
Good morning, all,

I am in desperate need to improve the performance of our StatServer used by CCA.
This has to be done without hardware upgrades or Genesys upgrade.

My problem is as follows:

in one of our call centers that has over 3000 operators, we have our CCA StatServer periodically disconnect from the rest of Genesys due to what we as extremely high system load. Stat Server is running on a single CPU Sparc III i machine with 1002 MHz CPU and 512 MBs of RAM. There is nothing else running on this system but CCA StatServer.

prstat tells me that StatServer uses 900 MB of memory and CPU is about 80% idle.
Every 15 minutes, DataSourcer sends 250,000 requests to this StatServer. During that time, CPU idle percentage drops to around 30%. Sometimes, it even dips to 0%.

Here is the dump for prstat 1

kthr      memory            page            disk          faults      cpu
r b w  swap  free  re  mf pi po fr de sr m1 m1 m1 m2  in  sy  cs us sy id
0 0 0 1808192 15832  3  4  8  0  0  0  0  0  0  0  1  970 6450  406 96  4  0
0 0 0 1808192 15792  3  4  8  0  0  0  0  0  0  0  1  906 5721  405 96  3  1
0 0 0 1808192 15736  7  9 16  0  0  0  0  0  0  0  1  963 6069  393 96  2  2
0 1 0 1808192 15848 138 750 72 168 216 0 233 0 0 0 26 1103 7601  447 91  6  3
0 0 0 1808192 15736 12  14 16  0  0  0  0  0  0  0  2 1050 6321  406 94  6  0
0 0 0 1808192 15688  3  3  0  0  0  0  0  0  0  0  0  922 6796  426 95  5  0
0 0 0 1808192 15640  7  8  8  8  8  0  0  0  0  0  1  967 6804  422 98  1  1
kthr      memory            page            disk          faults      cpu
r b w  swap  free  re  mf pi po fr de sr m1 m1 m1 m2  in  sy  cs us sy id
0 0 0 1808184 15576  3  7 40  0  0  0  0  0  0  0  5  925 5945  419 95  2  3
0 0 0 1808184 15560 30  32 16 63 87  0 44  0  0  0  5  989 7555  412 93  5  2
0 0 0 1808184 15568  3  6 24  0 24  0 16  0  0  0  3  887 5101  393 94  4  2
0 1 0 1808184 15576  5  6  8 55 63  0 100 0  0  0  6 1284 5647  440 97  1  2
0 0 0 1808184 15568  5  5  0 55 63  0 40  0  0  0  2  988 5761  383 94  6  0
0 0 0 1808184 15576  2  4 16  0  0  0  0  0  0  0  1  901 5195  393 98  0  2
0 0 0 1808184 15600 134 731 0 40 40  0 32  0  0  0  3  976 6396  444 96  4  0
0 0 0 1808184 15592  1  1  0  0  0  0  0  0  0  0  0 1054 6071  390 94  6  0

as you can see it is pretty horrible.

Quesions:

1. how much does writing to StatusTable, AgentTable, etc add to the load?
2. is it reasonable to have 900 Megs in memory for a 3000 agent call center using standard CCA templates?
3. Is there a way for StatServer to only keep 30 minutes worth of stats in its memory and not 24? I have modified old-stats-remove-interval to 60, since I think DataSourcer only needs 15 minutes, right? But I will not know if it will reduce memory consumption until I restart the statserver.

What else can I do?
[/color]

Thanks,
Vic

[tony]

Tony steps out of the shade, finds a convienient street lamp and says;

Hi Vic,

Are you using the same statserver for CCA/Brio, or is it just for real-time?  You mention Data Sourcer, which implies historical too...?  Aside from conducting a sizing exercise, I would also look at your ETL Assistant, if you have it installed.  ETL on the same box will also chew up memory every 15 minutes, if that is what it is set for and that depends on the Table Aggregate Level ETL times...

- I think I have a CCA Sizing Guide somewhere - would that help?

Tony

[victor]

To Tony in the bright light: are you going toward it? 

Actually, this StatServer is being used exclusively for Genesys Historical Reporting.
There is nothing but this StatServer running on it.

DataSourcer, ETL, Oracle, etc are all on a different machine. CPU spikes are because of DataSourcer's 200,000+ request every 15 minutes.
Since we cannot throttle DataSourcer (100 requests/sec or something) I have no choice but to figure out how to optimize what we have.

Has anyone done any tests on how to improve StatServer performance?

On thing I was thinking about doing, is removing StatServer from DAP in order to write into LoginTable (and Queue). How much writing to StatServer DB (Status, LoginTable...) add to CPU performance? Seems to be pittance...(1~2%)

Need major ideas here (that would not break my StatServer down)

[tony]

I think you should concentrate on throttling anything but the StatServer.  DataSourcer is doing it's job by requesting and collating - StatServer is purely responding to those requests, whenever they come through.

However- I have a cunning plan...

AFAIK, DataSourcer [b]can [/b] be "throttled", by determining the collection aggregation, in the settings.  You will need to check the following detail, because I no longer have full access, but I do not believe I am far from the truth;  DataSourcer will collate whatever ETL wants - ETL is "auto"set by the requests required in the Layouts in DMA.  Big chain, but it starts in the DMA Layouts - check there first, then (if you have it) check ETL Assistant, then your settings for the DataSourcer.  Your last stop is the StatServer which, in this instance, is the dumb terminal.

Tony

[Michael]

Victor,

Have you removed all unused stats from your stat server configuration?

Mike

[Peter K.]

There is very little you can do since judging from your post the problem is not statserver but number of requests issue by d/s. Presuming that you are using standard genesys templates and brio files you are stuck with upgrading your hardware and splitting statserver with each statserver handling a particular group of templates. d/s should have a throttle factor.

To Genesys - if you are listening, add throttle factor to d/s to ensure that machine resources are utilized more efficiently and make his life easier. [and mine too ]

[victor]

And mine too!