[DUG] Validating CDS files

Tue Jan 18 08:53:19 NZDT 2011

If you have a copy of Delphi 2010 then you will get Midas source code so
it is no longer a black box.  If you take a look through the code you
will see how the binary datapackets are created.  You could load the
data packet as a stream and read the header to see if this is the same
as what you expect.

Also given the specific erro message you are receiving you will probably
be able to find the piece of code in midas that is raising this and it
will help narrow down your problem.

Cameron Hart | Development Manager | Flow Software Limited
P: +64 9 476 3569 x910 | M: +64 21 222 3569 | E:
cameron.hart at flowsoftware.co.nz
PO Box 305-237, Triton Plaza, Auckland 0757, New Zealand |
www.flowsoftware.co.nz

This message is intended for the addressee named above. It may contain
privileged or confidential information. If you are not the intended
recipient of this message you must not use, copy, distribute or disclose
it to anyone.

Please consider the environment before printing this email 

-----Original Message-----
From: delphi-bounces at delphi.org.nz [mailto:delphi-bounces at delphi.org.nz]
On Behalf Of Matthew Comb
Sent: Monday, 17 January 2011 10:34 p.m.
To: NZ Borland Developers Group - Delphi List
Cc: 'NZ Borland Developers Group - Delphi List'
Subject: Re: [DUG] Validating CDS files

Hi Jolyon,

I almost wish I hadn't asked this question now :)

I'm well aware of the options, and setting up a webservice to retrieve a
hashed/checksumed payload set, is our preferred approach yes. This is a
legacy piece of software and doesn't adhere to our architectural
patterns and practices.

Knowing and acknowledging this is the better solution, I asked only one
very simple question.

Is there a known way to validate a binary CDS file.

All other solutions and suggestions are appreciated but really not
necessary. I may have lead the thread offtrack. Sorry for that.

Cheers,

Matt.

> I am thoroughly confused now...
>
> If you are talking about a *stream* that you theorise is being 
> corrupted in mid-flow over the wire, then I don't see how XML really 
> helps ensuring the integrity of that stream, or at least not how it is

> any more helpful than other techniques ... you can't do anything with 
> XML until the document is complete as an incomplete document is not 
> valid no matter how valid it may be when eventually complete.
>
> So you can't do any validation until your potentially interrupted 
> stream has finished streaming and you have a complete file to work 
> with.
>
> That being the case, if XML is a viable way to circumvent your 
> apparent problem then why can you not simply eliminate the streaming 
> part entirely ?
> Dump your output into a file on the server side of your communication,

> then transmit the entire file, thereby leveraging the integrity checks

> of the connection to ensure reliable transmission.
>
> Hashing files in this way is a trivially simple and relatively 
> efficient matter using MD5, for example.
>
>
> I am not entirely convinced that you are solving the right problem
here.
>
>
> The infrequency of the apparent corruption could surely potentially be

> as much due to an infrequent access to some data that is already 
> corrupt on the server as it is to some sporadic wireless network 
> corruption, no ?
>
>
> -----Original Message-----
> From: delphi-bounces at delphi.org.nz 
> [mailto:delphi-bounces at delphi.org.nz]
> On
> Behalf Of Matthew Comb
> Sent: Monday, 17 January 2011 19:20
> To: NZ Borland Developers Group - Delphi List
> Subject: Re: [DUG] Validating CDS files
>
> Paul,
>
> Thanks for your thoughts, I was tending towards reverse engineering 
> the format, I could not see any obvious tokens at footer, and wondered

> if someone had beaten me to it.
>
> I actually prefer the simplicity of your idea of 
> compressing/encrypting the xml file. Thats a tidyier solution for now 
> until we can get to the hashing
>
> Not sure why we haven't thought of that already....
>
> Cheers,
>
> Matt.
>
>
>> Matthew wrote:
>>
>>> I wasn't suggesting that wireless was changing byte structure, but 
>>> if you are streaming data, and your datastream gets disconnected, 
>>> then you could end up with an incomplete transfer.
>>>
>>> I'm not 100% sure that midas catches all scenarios when working off 
>>> a remote data instance ?
>>>
>>> Note we use dbx4mysql + midas.
>>>
>>> Note also that I cannot rule out the drivers either and also could 
>>> be the data out of the db server, its a tricky one to track down, as

>>> you basically have a black box from db -> dbxmysql + midas...
>>>
>>> What I do know is that its very rare. e.g. maybe 1 in 10,000 usages 
>>> corrupts the file and has occurred in more than 1 location, so it 
>>> does not appear to be station specific.
>>>
>>> Logically with these stats, I can only put that down to a flakey 
>>> connection, otherwise the error rate would be more often.
>>
>> If you can't go the hash route in the short term, it sounds to me 
>> like you'll need to reverse engineer the cds format to determine 
>> whether there is a examinable file tail.
>>
>> I don't imagine the format will be _that_ hard to reverse engineer 
>> but then I've done a fair bit of binary reverse engineering so maybe 
>> I'm being a bit cavalier. If you make a number of simple and small 
>> cds files and compare and contrast them, you can probably work your 
>> way up in determining the encoding.
>>
>> Assuming it was implemented using some form of streaming class model,

>> there may well be a common interface and then a cds outputter and an 
>> xml outputter. If so, there is probably going to be a fairly 1-to-1 
>> mapping between the binary format and the xml format in terms of
logical schema.
>>
>> These xml inspited compressed binary schemas are often all pretty 
>> similar - they replace the text tags and attributes with binary 
>> tokens taken from a dictionary to avoid repetition and reduce the
size.
>>
>> As an side, often you're better off just zipping the xml rather than 
>> implementing and debugging your own tokenized 'binary xml' engine but

>> that's a whole different argument :-)
>>
>> Anyway, you might get lucky and quickly determine there is a common 
>> metadata trailer or end of data stream signature in the binary 
>> schema, similar to how zip files replicate their metadata directory 
>> and place it at the end of the file.
>>
>> This may seem overkill for your current problem since you're aiming 
>> for a short term solution, but having a cds stream viewer which can 
>> load into a virtual tree view might be a useful in-house debugging 
>> tool down the track.
>>
>> Cheers,
>>   Paul.
>>
>> _______________________________________________
>> NZ Borland Developers Group - Delphi mailing list
>> Post: delphi at delphi.org.nz
>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
Subject:
>> unsubscribe
>>
>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at delphi.org.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at delphi.org.nz with
Subject:
> unsubscribe
>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at delphi.org.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at delphi.org.nz with
Subject:
> unsubscribe
>

_______________________________________________
NZ Borland Developers Group - Delphi mailing list
Post: delphi at delphi.org.nz
Admin: http://delphi.org.nz/mailman/listinfo/delphi
Unsubscribe: send an email to delphi-request at delphi.org.nz with Subject:
unsubscribe