Pre-DMA chunk raw file physical position.

[Gef Buneri]

Hi All,
I would like to know the physical position (or filename/filetype to search for) of the raw files that generates the chunks I can see through the DMA client as chunks, each 15 minutes. Are they fisical, or before to becoming chunks they're some sort of volatile memory dump?

The goal is to monitor any interruptions in the writing of those files in the lesser time gap possible, to avoid any loss of chunks.


Best regards,

Gef.

[cavagnaro]

ODS writes the data to its own DB. Why would there be data lost? ODS receives from statserver, ODS writes those to tables according to your templates. ETL goes and grab those chunks and transform them to tables for extraction. ODS can retain or delete this data based on configuration files.


Enviado de meu SM-G9650 usando o Tapatalk

[cavagnaro]

And if ODS can't write to DB will generate emergency local files that will be treated as soon as DB connection goes back.
Can there be a delay in data processing? Yes... It is not intended to be real time.


Enviado de meu SM-G9650 usando o Tapatalk

[Gef Buneri]

I'm analyzing an anomaly occurred few days ago, whose symptoms were the absence of some quarters of an hour in the datamart db. There were no chunks in the dma either. Recovery files have not been written.

So, where does the statserver write data, waiting to the ods to take them?

To solve, I had to restart ods_statserver.

[cavagnaro]

Well if Statserver fails to provide info then you are out of luck. Then you should focus in Statserver and not ODS.


Enviado de meu SM-G9650 usando o Tapatalk

[Gef Buneri]

Yep, totally right there... the point is that logs from the process (ods_statserver ver. 7.1), showed no evidence of a problem, even if the verbosity was set to all, so I was guessing if it exists a way to monitor somehow the bad statserver behaviour; first idea was to locate the position/name/"whatever" were the statserver stores infos, to create some monitoring daemon on sys side.

[cavagnaro]

Well no, I have seen that behaviour in old StatServer where had a bug on writing backup data file, which has nothing to do with ETL.
An upgrade may help you, but SS doesn't store data anywhere physically, sends it directly to ODS.

[Gef Buneri]

here we are... "but SS doesn't store data anywhere physically, sends it directly to ODS", as I supposed... so if statserver has problems, bye bye data. And yes, I'm evaluating an upgrade... thanks Cav, as usual.

[cavagnaro]

Maybe HA can help your problem, however if using old version, also have seen that as SS is supposedly working fine, there is no switch or indication that HA pair needs to be up

[Gef Buneri]

HA for sure, but any upgrade to at least an 8.1.3+ ver. would be a blessing. This sayd, am I right if I say the fastest way to check if there's something wrong with data collecting, is to check ODS internal database's volatile tables? I mean, in a scenario like the one I described before, I expect no data there.

[cavagnaro]

Well yes, if no data is being sent to SQL by ODS...Houston we have a problem


Enviado de meu SM-G9650 usando o Tapatalk

[Gef Buneri]

Ok, we have a point. Now, where I can found deeper infos for that internal db? 'cause in the volatile tables no timestamp is written to understand the exact moment that record was written.

[cavagnaro]

ODS as temp DB has no information on its tables structure. Maybe you could monitor SQL via profiler or similar.


Enviado de meu SM-G9650 usando o Tapatalk

[Gef Buneri]

Roger that, I'll put something up on the os/db side. Thanksalot again.

[Gef Buneri]

G'day, people; hope everything's fine out there.

Regarding the table OL_CHUNK_LOG in the ods db... in the table I find, among others, two dates:

RECEIVE_TIME
TRANSFERRED_TIME

between which there is an interval of 2 hours. Based on what you know, that interval is real, is the process transferring data received two hours ago, or could it be more a matter of time zone? (i.e. RECEIVE is in GMT, TRANSFERRED in CEST).