[DUG] Validating CDS files

[Matthew Comb]

I think "Flakey wireless LAN" in my brain read "Outside world" or "Not
Client"

I wasn't suggesting that wireless was changing byte structure, but if you
are streaming data, and your datastream gets disconnected, then you could
end up with an incomplete transfer.

I'm not 100% sure that midas catches all scenarios when working off a
remote data instance ?

Note we use dbx4mysql + midas.

Note also that I cannot rule out the drivers either and also could be the
data out of the db server, its a tricky one to track down, as you
basically have a black box from db -> dbxmysql + midas...

What I do know is that its very rare. e.g. maybe 1 in 10,000 usages
corrupts the file and has occurred in more than 1 location, so it does not
appear to be station specific.

Logically with these stats, I can only put that down to a flakey
connection, otherwise the error rate would be more often.

Matt.

> Hmmmm.
>
> "Flakey wireless LAN".... ?
>
> Surely wifi (or any LAN tech for that matter) has mechanisms built in to
> the
> protocol to ensure this sort of thing does not occur....  certainly even
> with the flakiest of wifi connections I have never experienced corruption.
> The wifi connection is either made, however slowly as a result of the
> flakiness, or not made at all.
>
>
> Are you positive that the "corruption" is not occurring directly in the
> output of the remote server, rather than in the communication of the data
> ?
>
>
> Because if that's the case then checksuming will of course be a waste of
> time
>
>
> -----Original Message-----
> From: delphi-bounces at delphi.org.nz [mailto:delphi-bounces at delphi.org.nz]
> On
> Behalf Of Todd
> Sent: Monday, 17 January 2011 14:22
> To: NZ Borland Developers Group - Delphi List
> Subject: Re: [DUG] Validating CDS files
>
> Hi Matthew
>
> It sounds like comparing a MD5 hash of the CDS file prior to sending and
> subsequent to receiving the data stream would provide a sufficient check
> on its integrity.
>
> Todd.
>> Hi Todd,
>>
>> The ClientDatasets initially get populated via a remote server, and it
>> is
>> this process which in rare cases causes the corruption (e.g. customer on
>> flakey wireless lan or similar).
>>
>> Longterm we will replace this mechanism so the dataset is populated
>> server
>> side and send back with a hash which will maintain integrity.
>>
>> For the moment, we are stuck with a mechanism which populates this cds
>> data and then writes to file.
>>
>> Appending some metadata at this point is a valid option. Ill test and
>> see
>> if it circumvents the issue. The main issue though is that I'm not sure
>> if
>> the ClientDataset at this point knows that the data is corrupt, and
>> therefore we could be exporting a corrupt data chunk packed with some
>> metadata, which does not help.
>>
>> Thats why we really need some protection on load.
>>
>> Cheers,
>>
>> Matt.
>>
>>
>>
>>
>>> Hi Matthew
>>>
>>> Are the CDS files being stored as disk files or in a database? How are
>>> they being corrupted? Faulty back up media? Perhaps you could add some
>>> meta-data to each file as it is saved.
>>>
>>> Todd.
>>>
>>>> The driver for the question, is that we have some application client
>>>> datasets which are put into a defaulted state if a corrupt cds file is
>>>> loaded.
>>>>
>>>> Yes with XML, we can just validate the XML, but we use the binary
>>>> format
>>>> so that solution does not apply.
>>>>
>>>> At present we basically have two solutions.
>>>>
>>>> 1. Load into a test clientdataset as suggested by Alistair. This is a
>>>> valid solution but does add considerable load time into the startup.
>>>>
>>>> 2. Can load into application clientdatasets, and dispose and reload if
>>>> error encountered. This is ok also but does require additional loading
>>>> in
>>>> case of error.
>>>>
>>>> What I'm really after is a file level test to check that file should
>>>> even
>>>> be attempted. e.g. open file stream seek start and seek end and check
>>>> a
>>>> couple of bytes... that type of thing.
>>>>
>>>> Matt.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> What does "validate" mean for a CDS file ?
>>>>>
>>>>> For XML it obviously/intuitively means "is a valid XML representation
>>>>> of a
>>>>> CDS file" but for a CDS file, if you dumped it from a client data set
>>>>> in
>>>>> the
>>>>> first place, what additional validation would be needed to ensure
>>>>> that
>>>>> it
>>>>> is
>>>>> a CDS file ?
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: delphi-bounces at delphi.org.nz
>>>>> [mailto:delphi-bounces at delphi.org.nz]
>>>>> On
>>>>> Behalf Of Matthew Comb
>>>>> Sent: Monday, 17 January 2011 10:27
>>>>> To: NZ Borland Developers Group - Delphi List
>>>>> Subject: Re: [DUG] Validating CDS files
>>>>>
>>>>> Hey Cam, for space size reasons, we're using the CDS format not the
>>>>> XML
>>>>> format as the files get pretty large.
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>
>>>>>> If you have dumped the CDS as XML then you can validate it using
>>>>>> XSD.
>>>>>> I
>>>>>> haven't come across a schema for it but it shouldn't be hard to
>>>>>> make.
>>>>>>
>>>>>> Cameron Hart | Development Manager | Flow Software Limited
>>>>>> P: +64 9 476 3569 x910 | M: +64 21 222 3569 | E:
>>>>>> cameron.hart at flowsoftware.co.nz
>>>>>> PO Box 305-237, Triton Plaza, Auckland 0757, New Zealand |
>>>>>> www.flowsoftware.co.nz
>>>>>>
>>>>>> This message is intended for the addressee named above. It may
>>>>>> contain
>>>>>> privileged or confidential information. If you are not the intended
>>>>>> recipient of this message you must not use, copy, distribute or
>>>>>> disclose
>>>>>> it to anyone.
>>>>>>
>>>>>> Please consider the environment before printing this email
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: delphi-bounces at delphi.org.nz
>>>>>> [mailto:delphi-bounces at delphi.org.nz]
>>>>>> On Behalf Of Matthew Comb
>>>>>> Sent: Monday, 17 January 2011 10:03 a.m.
>>>>>> To: NZ Borland Developers Group - Delphi List
>>>>>> Cc: NZ Borland Developers Group - Delphi List
>>>>>> Subject: [DUG] Validating CDS files
>>>>>>
>>>>>> Does anyone know if there is a way to Validate a CDS file (midas)
>>>>>> before
>>>>>> attempting to load it into the dataset ?
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> _______________________________________________
>>>>>> NZ Borland Developers Group - Delphi mailing list
>>>>>> Post: delphi at delphi.org.nz
>>>>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>>>>>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
>>>>>> Subject:
>>>>>> unsubscribe
>>>>>>
>>>>>> _______________________________________________
>>>>>> NZ Borland Developers Group - Delphi mailing list
>>>>>> Post: delphi at delphi.org.nz
>>>>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>>>>>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
>>>>>> Subject:
>>>>>> unsubscribe
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> NZ Borland Developers Group - Delphi mailing list
>>>>> Post: delphi at delphi.org.nz
>>>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>>>>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
>>>>> Subject:
>>>>> unsubscribe
>>>>>
>>>>> _______________________________________________
>>>>> NZ Borland Developers Group - Delphi mailing list
>>>>> Post: delphi at delphi.org.nz
>>>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>>>>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
>>>>> Subject:
>>>>> unsubscribe
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> NZ Borland Developers Group - Delphi mailing list
>>>> Post: delphi at delphi.org.nz
>>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>>>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
>>>> Subject:
>>>> unsubscribe
>>>>
>>>>
>>> _______________________________________________
>>> NZ Borland Developers Group - Delphi mailing list
>>> Post: delphi at delphi.org.nz
>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
>>> Subject:
>>> unsubscribe
>>>
>>>
>>
>> _______________________________________________
>> NZ Borland Developers Group - Delphi mailing list
>> Post: delphi at delphi.org.nz
>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> Unsubscribe: send an email to delphi-request at delphi.org.nz with Subject:
> unsubscribe
>>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at delphi.org.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at delphi.org.nz with Subject:
> unsubscribe
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at delphi.org.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at delphi.org.nz with Subject:
> unsubscribe
>