[DUG] Validating CDS files

[Matthew Comb]

Hi Jolyon,

I almost wish I hadn't asked this question now :)

I'm well aware of the options, and setting up a webservice to retrieve a
hashed/checksumed payload set, is our preferred approach yes. This is a
legacy piece of software and doesn't adhere to our architectural patterns
and practices.

Knowing and acknowledging this is the better solution, I asked only one
very simple question.

Is there a known way to validate a binary CDS file.

All other solutions and suggestions are appreciated but really not
necessary. I may have lead the thread offtrack. Sorry for that.

Cheers,

Matt.



> I am thoroughly confused now...
>
> If you are talking about a *stream* that you theorise is being corrupted
> in
> mid-flow over the wire, then I don't see how XML really helps ensuring the
> integrity of that stream, or at least not how it is any more helpful than
> other techniques ... you can't do anything with XML until the document is
> complete as an incomplete document is not valid no matter how valid it may
> be when eventually complete.
>
> So you can't do any validation until your potentially interrupted stream
> has
> finished streaming and you have a complete file to work with.
>
> That being the case, if XML is a viable way to circumvent your apparent
> problem then why can you not simply eliminate the streaming part entirely
> ?
> Dump your output into a file on the server side of your communication,
> then
> transmit the entire file, thereby leveraging the integrity checks of the
> connection to ensure reliable transmission.
>
> Hashing files in this way is a trivially simple and relatively efficient
> matter using MD5, for example.
>
>
> I am not entirely convinced that you are solving the right problem here.
>
>
> The infrequency of the apparent corruption could surely potentially be as
> much due to an infrequent access to some data that is already corrupt on
> the
> server as it is to some sporadic wireless network corruption, no ?
>
>
> -----Original Message-----
> From: delphi-bounces at delphi.org.nz [mailto:delphi-bounces at delphi.org.nz]
> On
> Behalf Of Matthew Comb
> Sent: Monday, 17 January 2011 19:20
> To: NZ Borland Developers Group - Delphi List
> Subject: Re: [DUG] Validating CDS files
>
> Paul,
>
> Thanks for your thoughts, I was tending towards reverse engineering the
> format, I could not see any obvious tokens at footer, and wondered if
> someone had beaten me to it.
>
> I actually prefer the simplicity of your idea of compressing/encrypting
> the xml file. Thats a tidyier solution for now until we can get to the
> hashing
>
> Not sure why we haven't thought of that already....
>
> Cheers,
>
> Matt.
>
>
>> Matthew wrote:
>>
>>> I wasn't suggesting that wireless was changing byte
>>> structure, but if you are streaming data, and your datastream
>>> gets disconnected, then you could end up with an incomplete transfer.
>>>
>>> I'm not 100% sure that midas catches all scenarios when
>>> working off a remote data instance ?
>>>
>>> Note we use dbx4mysql + midas.
>>>
>>> Note also that I cannot rule out the drivers either and also
>>> could be the data out of the db server, its a tricky one to
>>> track down, as you basically have a black box from db ->
>>> dbxmysql + midas...
>>>
>>> What I do know is that its very rare. e.g. maybe 1 in 10,000
>>> usages corrupts the file and has occurred in more than 1
>>> location, so it does not appear to be station specific.
>>>
>>> Logically with these stats, I can only put that down to a
>>> flakey connection, otherwise the error rate would be more often.
>>
>> If you can't go the hash route in the short term, it sounds to me like
>> you'll need to reverse engineer the cds format to determine whether
>> there is a examinable file tail.
>>
>> I don't imagine the format will be _that_ hard to reverse engineer but
>> then I've done a fair bit of binary reverse engineering so maybe I'm
>> being a bit cavalier. If you make a number of simple and small cds files
>> and compare and contrast them, you can probably work your way up in
>> determining the encoding.
>>
>> Assuming it was implemented using some form of streaming class model,
>> there may well be a common interface and then a cds outputter and an xml
>> outputter. If so, there is probably going to be a fairly 1-to-1 mapping
>> between the binary format and the xml format in terms of logical schema.
>>
>> These xml inspited compressed binary schemas are often all pretty
>> similar - they replace the text tags and attributes with binary tokens
>> taken from a dictionary to avoid repetition and reduce the size.
>>
>> As an side, often you're better off just zipping the xml rather than
>> implementing and debugging your own tokenized 'binary xml' engine but
>> that's a whole different argument :-)
>>
>> Anyway, you might get lucky and quickly determine there is a common
>> metadata trailer or end of data stream signature in the binary schema,
>> similar to how zip files replicate their metadata directory and place it
>> at the end of the file.
>>
>> This may seem overkill for your current problem since you're aiming for
>> a short term solution, but having a cds stream viewer which can load
>> into a virtual tree view might be a useful in-house debugging tool down
>> the track.
>>
>> Cheers,
>>   Paul.
>>
>> _______________________________________________
>> NZ Borland Developers Group - Delphi mailing list
>> Post: delphi at delphi.org.nz
>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> Unsubscribe: send an email to delphi-request at delphi.org.nz with Subject:
>> unsubscribe
>>
>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at delphi.org.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at delphi.org.nz with Subject:
> unsubscribe
>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at delphi.org.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at delphi.org.nz with Subject:
> unsubscribe
>