[DUG] Validating CDS files

[Todd]

Hmmmm.
> "Flakey wireless LAN".... ?
>
> Surely wifi (or any LAN tech for that matter) has mechanisms built in to the
> protocol to ensure this sort of thing does not occur....  certainly even
> with the flakiest of wifi connections I have never experienced corruption.
> The wifi connection is either made, however slowly as a result of the
> flakiness, or not made at all.
>
>
> Are you positive that the "corruption" is not occurring directly in the
> output of the remote server, rather than in the communication of the data ?
>
>
> Because if that's the case then checksuming will of course be a waste of
> time
>    
Agreed.

You're not using a remote data broker with Midas are you?

Todd.

>
> -----Original Message-----
> From: delphi-bounces at delphi.org.nz [mailto:delphi-bounces at delphi.org.nz] On
> Behalf Of Todd
> Sent: Monday, 17 January 2011 14:22
> To: NZ Borland Developers Group - Delphi List
> Subject: Re: [DUG] Validating CDS files
>
> Hi Matthew
>
> It sounds like comparing a MD5 hash of the CDS file prior to sending and
> subsequent to receiving the data stream would provide a sufficient check
> on its integrity.
>
> Todd.
>    
>> Hi Todd,
>>
>> The ClientDatasets initially get populated via a remote server, and it is
>> this process which in rare cases causes the corruption (e.g. customer on
>> flakey wireless lan or similar).
>>
>> Longterm we will replace this mechanism so the dataset is populated server
>> side and send back with a hash which will maintain integrity.
>>
>> For the moment, we are stuck with a mechanism which populates this cds
>> data and then writes to file.
>>
>> Appending some metadata at this point is a valid option. Ill test and see
>> if it circumvents the issue. The main issue though is that I'm not sure if
>> the ClientDataset at this point knows that the data is corrupt, and
>> therefore we could be exporting a corrupt data chunk packed with some
>> metadata, which does not help.
>>
>> Thats why we really need some protection on load.
>>
>> Cheers,
>>
>> Matt.
>>
>>
>>
>>
>>      
>>> Hi Matthew
>>>
>>> Are the CDS files being stored as disk files or in a database? How are
>>> they being corrupted? Faulty back up media? Perhaps you could add some
>>> meta-data to each file as it is saved.
>>>
>>> Todd.
>>>
>>>        
>>>> The driver for the question, is that we have some application client
>>>> datasets which are put into a defaulted state if a corrupt cds file is
>>>> loaded.
>>>>
>>>> Yes with XML, we can just validate the XML, but we use the binary format
>>>> so that solution does not apply.
>>>>
>>>> At present we basically have two solutions.
>>>>
>>>> 1. Load into a test clientdataset as suggested by Alistair. This is a
>>>> valid solution but does add considerable load time into the startup.
>>>>
>>>> 2. Can load into application clientdatasets, and dispose and reload if
>>>> error encountered. This is ok also but does require additional loading
>>>> in
>>>> case of error.
>>>>
>>>> What I'm really after is a file level test to check that file should
>>>> even
>>>> be attempted. e.g. open file stream seek start and seek end and check a
>>>> couple of bytes... that type of thing.
>>>>
>>>> Matt.
>>>>
>>>>
>>>>