" /> ETL parameters - Genesys CTI User Forum

Author Topic: ETL parameters  (Read 8578 times)

Mick

  • Guest
ETL parameters
« on: January 01, 1970, 12:00:00 AM »
Advertisement
When (by default) ETL transition and aggregation process should take place? Where can I find logs ETL processes? How can I impact frequency of this process?

Vic

  • Guest
ETL parameters
« Reply #1 on: January 01, 1970, 12:00:00 AM »
Mick,

let me see if I can answer this.

First of all, ETL is used to move data from ODS to final data storage and then aggregate it.

Genesys ETL itself consists of three different parts:
transfer the data from ODS into one big data aggregator database
update config data by connecting to config server and getting the latest CME info
aggregate data so that it can be viewed using Brio templates made by Genesys

To accomplish this, there are three different main programs written in Java, and you will find them under CCAnalyzer/ETL_Runtime directory. Genesys provides several batch files to get them to run.

They are runagg_only, run rans_only, and conf_reports ...
There are several variations of this programs, with the most useful being run rans_and_agg.

The behavior for each program is set in .properties files, such as agg_only.properties file or trans_and_agg.properties file, etc...

My favorite is trans_and_agg file, because it does transfer and aggregation as well as synchronization with one command.

Take a look inside and you will find a myriad of options you can play with including rescan period.

The options you want to play with are:
srcRefreshTimeout how often ETL will check for new data (default is 5 minutes... you might as well make it an hour ;) )
syncWithConfServer you want to set it, because it would be nice to have update confserver information without running run_conf script :)

Get rid of trans_once unless you want to restart your script from within something else.

The log is usually the name of the file, plus .log check "log" setting in that .properties file.

Genesys provides ETL_Runtime with CCA installation, which is a program which will run the batch files for you at predefined time intervals. It is installed using CCA wizard, and you can find its definition in CME under ETL_Runtime or ETL_Service.

Take a look at its options! in "Genesys ETL Service Starter" there is a option called Processes, and by default it is set to "ETL_Trans_Only,ETL_Agg_Only,ETL_Tracking".

Go back to options and you will find sections with identical name to the values mentioned in "Processes". This is where you set up the starting time and restart period for each process.
Please keep in mind that this will only START the script at that time, the settings for the script are inside .property files. :)


Depending on the size of your call center and number of calls you get and number of report templates activated, your transfer can finish in one minute or it can take hours, so if you have a transfer that takes 2 hours and you have your StartTime option set to 00:00+1:00 then you are asking for trouble :)

I think a lot of people will tell you different things on what to do, but, my guess is that since you still have to mess with script environment on top of ETL_Runtime variables, just use trans_agg script, set it for rescanning of tables and synch with ConfigServer and just run that on your machine. It will run, continuously do the transfer and aggregation at specific intervals, dump the log file into "trans_aggr.log" file and probably make your life easy.

Vic

Marked as best answer by on Today at 11:16:08 PM

Ritchie

  • Guest
ETL parameters
« Reply #2 on: January 01, 1970, 12:00:00 AM »
  • Undo Best Answer
  • Well, I have installed the suite during the last week, and everything seems to be working great :). Except for....
    the damn purging of the ODS DB. I have ETL_purging setup in CME and in the startup for the etl_service. I even have the 'dropTransferredTables' param in etl.properties with no value, and still it will no drop any tables from ODS.
    What am I missing anyone any ideas?

    cumi

    • Guest
    ETL parameters
    « Reply #3 on: January 01, 1970, 12:00:00 AM »
    Hi,

    Purging the DataMart and dropping the transferred ODS tables are two different issues (and databases etc).

    The ETL Purging utility is to delete "old" data records from the DataMart database (from where you pull your (BRIO) reports). This can be set up with ETL Assistant, and can be activated in CME...

    Dropping the trasferred tables from the ODS Database has to be "on". When the ETL untime transfers the chunks from ODS to DataMart, it marks them as "transferred" (see OL_CHUNK_LOG table in ODS). When all the chunks are marked as transferred AND the table size exceeds the size you set up in CME (or the default one) the data table OL_DATAxxxx will be dropped.

    hope this helps

    cumi

    Ritchie

    • Guest
    ETL parameters
    « Reply #4 on: January 01, 1970, 12:00:00 AM »
    I am aware that purging the DataMart and the ODS are completely separate issues. What I'm not aware of is that ODS tables will not delete until the table size hits max. Is this definately true?

    My understanding was that the old data would be removed after successful transfer to the DataMart. Hmmm...

    Ritchie

    • Guest
    ETL parameters
    « Reply #5 on: January 01, 1970, 12:00:00 AM »
    Excellent mate reduced the size of the table to 13000 in CME and the table was dropped! Do you know that Genesys ahs been looking at this for 2 weeks! They've had be drop the entire DaraMart schema to try and get it to work you know how it is, you gotta try what they say first.
    Anyway thanks for the tip in the right direction.