integrity warning in the data logger application

[uhamid80]

hello,

if we use all the 32 channels of the delfin card for daq with sampling
rates upto 48 kHz, sometimes the integrity warning led turns on after
acquiring a certain number of data chunks (that is not fixed). we
are using the data logger application provided with the delfin card.

so what is the purpose of this integrity warning when its led gets on
and what is the way to handle this problem to avoid any data loss?

hoping to have your reply soon.

regards.

[jhenderson]

The Delfin buffering scheme is as follows;

Data is acquired on the baseboard and stored into DSP memory. Typically, a substantial portion of target memory is allocated as a FIFO - this is under DSP program control. Use of a larger portion of target memory provides greater instantaneous load-carrying capacity.

The DSP transfers data to the PC via bus-mastering through use of PciOutStream.Put() commands. EDMA is used to flow data from the target to the Host.

Data flows into a designated region of Host memory partitioned for use as bus-master memory - typically 2 MB. One-half of this region is reserved for data received from the DSP, the other half for data sent to the DSP.

During data flow, when the bus master data fills, this implies that the Host software is not responding to requests by the target to drain the bus master region. This can be the result of a host program that is performing too many calculations on each received buffer, or to a Host whose disk subsystem is too slow to sustain the current data flow rate. If the bus master region fills, the integrity warning error is displayed in the logger example. This is a warning, not an error, since the filling of the BM region does not imply that data loss has occurred; Rather, it suggests that data loss is imminent.

By contrast, the target alert mechanism is capable of monitoring the A/D FIFO within the hardware and sending an overrun message to the DSP application whenever the FIFO overflows. This is a hard error, which implies that the user application code, has not been able to keep up with the data flow rate; Data has backed up into the BM region; All DSP buffer memory has filled and the A/D FIFOs overran as a consequence.

To fix the problem, identify which of the elements in the system are at fault. If using a disk subsystem, measure its peformance to insure that it can sustain transfers at the desired sample rate continuously. For example, on a Delfin running 32-channels at 192 kHz, the max data rate is 32 * 4 * 192000 = 24 MB/s. You'll probably need a striped RAID array to handle this rate.

[uhamid80]

hello,

thank you for your technical support.

i am using Logger6x host application in Borland C++.

i am acquiring data from 32 channels with 25.6 kHz sampling rate.

i am getting the data using the following function:-

//------------------------------------------------------------------------------
// TMain::BlockFilterFilterData() -- Maintain tally of blocks received
//------------------------------------------------------------------------------
void __fastcall TMain::BlockFilterFilterData(TIIBlockFilter *Sender,
TIIBlockBuffer InBuffer, TIIBlockBuffer &OutBuffer,bool &SendOutputData)
{
// using function OutBuffer.ConstInt();
// taking data into an integer array of size 32*1024

// doing some data arrangements on the array
// doing some mathematical manipulations on the array

// Keep track of blocks read
BlocksReceived++;
}
//-------------------------------------------------------------------------------

please note: i am not using the file writing option and binview.

when i sent you the previous email i was 2MB BM region.

after going through your reply i increased the BM region to 4 MB.

now logger6x application runs fine for a longer period but still the

integrity warning LED turns on.

i have un-installed the heavy softwares on my PC.

i have disabled the startup windows services as well.

even then i am experiencing the same problem.

please provide help in this context.

hoping to have your reply soon.

with regards.

[jhenderson]

As mentioned in my previous post, an integrity warning signifies that the Host is not processing buffers at a fast enough rate. Consequently, the system is on the brink of data loss.

Try commenting out the processing code within the block filter data handler. If the system runs without any integrity warning, you must optimize your analysis code to improve performance, so that the application meets the real-time requirements.

[uhamid80]

hello,

thank you for your suggestion.

i have analysed my processing code step by step and

i have optimized it as well.

the next step is that i am transfering the data to another computer

using tcp socket.

the data sending code is also in the block filter handler function.

now the situation is as follows:-

1. my code runs fine if i dont use the data sending code i.e. integrity

warning LED remains OFF.

2. but when i run my code along with data sending code using tcp socket

the integrity warning LED turns on.

data transfer to another computer is an essetial requirement for me.

kindly provide support in this context as well.

hope to have your reply soon.

with regards.

[jhenderson]

From your previous post, you are attempting to transfer 32-channels at 25.6 kHz. This equates to 32 * 25600 * 4 bytes/s = 3.39 MB/s. This is within the capacity of a 100 Mbit Ethernet if the system is on an isolated node. In the context of a busy WAN, you may encounter too many collisions to sustain this rate. Use of jumbo frames may also help.

You should copy the data received from the board within the BlockFilter handler into a queue, then increment a semaphore.

A background thread should be used to awaken on the semaphore, then drain the data in the queue and send packets to the ethernet interface. This will perform the critical function of decoupling the transmission from the reception, allowing reception to run at full rate unimpeded by the vagaries ethernet data flow.

Jim