[DUG] Validating CDS files

[Matthew Comb]

Thanks Cam, thats a good idea.

Matt.

> If you have a copy of Delphi 2010 then you will get Midas source code so
> it is no longer a black box.  If you take a look through the code you
> will see how the binary datapackets are created.  You could load the
> data packet as a stream and read the header to see if this is the same
> as what you expect.
>
> Also given the specific erro message you are receiving you will probably
> be able to find the piece of code in midas that is raising this and it
> will help narrow down your problem.
>
>
>
> Cameron Hart | Development Manager | Flow Software Limited
> P: +64 9 476 3569 x910 | M: +64 21 222 3569 | E:
> cameron.hart at flowsoftware.co.nz
> PO Box 305-237, Triton Plaza, Auckland 0757, New Zealand |
> www.flowsoftware.co.nz
>
> This message is intended for the addressee named above. It may contain
> privileged or confidential information. If you are not the intended
> recipient of this message you must not use, copy, distribute or disclose
> it to anyone.
>
> Please consider the environment before printing this email
>
> -----Original Message-----
> From: delphi-bounces at delphi.org.nz [mailto:delphi-bounces at delphi.org.nz]
> On Behalf Of Matthew Comb
> Sent: Monday, 17 January 2011 10:34 p.m.
> To: NZ Borland Developers Group - Delphi List
> Cc: 'NZ Borland Developers Group - Delphi List'
> Subject: Re: [DUG] Validating CDS files
>
> Hi Jolyon,
>
> I almost wish I hadn't asked this question now :)
>
> I'm well aware of the options, and setting up a webservice to retrieve a
> hashed/checksumed payload set, is our preferred approach yes. This is a
> legacy piece of software and doesn't adhere to our architectural
> patterns and practices.
>
> Knowing and acknowledging this is the better solution, I asked only one
> very simple question.
>
> Is there a known way to validate a binary CDS file.
>
> All other solutions and suggestions are appreciated but really not
> necessary. I may have lead the thread offtrack. Sorry for that.
>
> Cheers,
>
> Matt.
>
>
>
>> I am thoroughly confused now...
>>
>> If you are talking about a *stream* that you theorise is being
>> corrupted in mid-flow over the wire, then I don't see how XML really
>> helps ensuring the integrity of that stream, or at least not how it is
>
>> any more helpful than other techniques ... you can't do anything with
>> XML until the document is complete as an incomplete document is not
>> valid no matter how valid it may be when eventually complete.
>>
>> So you can't do any validation until your potentially interrupted
>> stream has finished streaming and you have a complete file to work
>> with.
>>
>> That being the case, if XML is a viable way to circumvent your
>> apparent problem then why can you not simply eliminate the streaming
>> part entirely ?
>> Dump your output into a file on the server side of your communication,
>
>> then transmit the entire file, thereby leveraging the integrity checks
>
>> of the connection to ensure reliable transmission.
>>
>> Hashing files in this way is a trivially simple and relatively
>> efficient matter using MD5, for example.
>>
>>
>> I am not entirely convinced that you are solving the right problem
> here.
>>
>>
>> The infrequency of the apparent corruption could surely potentially be
>
>> as much due to an infrequent access to some data that is already
>> corrupt on the server as it is to some sporadic wireless network
>> corruption, no ?
>>
>>
>> -----Original Message-----
>> From: delphi-bounces at delphi.org.nz
>> [mailto:delphi-bounces at delphi.org.nz]
>> On
>> Behalf Of Matthew Comb
>> Sent: Monday, 17 January 2011 19:20
>> To: NZ Borland Developers Group - Delphi List
>> Subject: Re: [DUG] Validating CDS files
>>
>> Paul,
>>
>> Thanks for your thoughts, I was tending towards reverse engineering
>> the format, I could not see any obvious tokens at footer, and wondered
>
>> if someone had beaten me to it.
>>
>> I actually prefer the simplicity of your idea of
>> compressing/encrypting the xml file. Thats a tidyier solution for now
>> until we can get to the hashing
>>
>> Not sure why we haven't thought of that already....
>>
>> Cheers,
>>
>> Matt.
>>
>>
>>> Matthew wrote:
>>>
>>>> I wasn't suggesting that wireless was changing byte structure, but
>>>> if you are streaming data, and your datastream gets disconnected,
>>>> then you could end up with an incomplete transfer.
>>>>
>>>> I'm not 100% sure that midas catches all scenarios when working off
>>>> a remote data instance ?
>>>>
>>>> Note we use dbx4mysql + midas.
>>>>
>>>> Note also that I cannot rule out the drivers either and also could
>>>> be the data out of the db server, its a tricky one to track down, as
>
>>>> you basically have a black box from db -> dbxmysql + midas...
>>>>
>>>> What I do know is that its very rare. e.g. maybe 1 in 10,000 usages
>>>> corrupts the file and has occurred in more than 1 location, so it
>>>> does not appear to be station specific.
>>>>
>>>> Logically with these stats, I can only put that down to a flakey
>>>> connection, otherwise the error rate would be more often.
>>>
>>> If you can't go the hash route in the short term, it sounds to me
>>> like you'll need to reverse engineer the cds format to determine
>>> whether there is a examinable file tail.
>>>
>>> I don't imagine the format will be _that_ hard to reverse engineer
>>> but then I've done a fair bit of binary reverse engineering so maybe
>>> I'm being a bit cavalier. If you make a number of simple and small
>>> cds files and compare and contrast them, you can probably work your
>>> way up in determining the encoding.
>>>
>>> Assuming it was implemented using some form of streaming class model,
>
>>> there may well be a common interface and then a cds outputter and an
>>> xml outputter. If so, there is probably going to be a fairly 1-to-1
>>> mapping between the binary format and the xml format in terms of
> logical schema.
>>>
>>> These xml inspited compressed binary schemas are often all pretty
>>> similar - they replace the text tags and attributes with binary
>>> tokens taken from a dictionary to avoid repetition and reduce the
> size.
>>>
>>> As an side, often you're better off just zipping the xml rather than
>>> implementing and debugging your own tokenized 'binary xml' engine but
>
>>> that's a whole different argument :-)
>>>
>>> Anyway, you might get lucky and quickly determine there is a common
>>> metadata trailer or end of data stream signature in the binary
>>> schema, similar to how zip files replicate their metadata directory
>>> and place it at the end of the file.
>>>
>>> This may seem overkill for your current problem since you're aiming
>>> for a short term solution, but having a cds stream viewer which can
>>> load into a virtual tree view might be a useful in-house debugging
>>> tool down the track.
>>>
>>> Cheers,
>>>   Paul.
>>>
>>> _______________________________________________
>>> NZ Borland Developers Group - Delphi mailing list
>>> Post: delphi at delphi.org.nz
>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
> Subject:
>>> unsubscribe
>>>
>>
>>
>> _______________________________________________
>> NZ Borland Developers Group - Delphi mailing list
>> Post: delphi at delphi.org.nz
>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
> Subject:
>> unsubscribe
>>
>>
>> _______________________________________________
>> NZ Borland Developers Group - Delphi mailing list
>> Post: delphi at delphi.org.nz
>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> Unsubscribe: send an email to delphi-request at delphi.org.nz with
> Subject:
>> unsubscribe
>>
>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at delphi.org.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at delphi.org.nz with Subject:
> unsubscribe
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at delphi.org.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at delphi.org.nz with Subject:
> unsubscribe
>