[DUG] [computing] Re: Creating lots of files on slow machines

[John Bird]

Yes thats all been looked at - all the assembling data and writing files are 
written in one thread anyway...so I am guessing thats a "red herring"

John Bird


Just wondering of the key word in the problem description is ... "Thread".

Do you have multiple threads writing these files?

Are you sure your code is multi-thread safe?

Are you sure the VCL routines you are using are multi-thread safe (a
number are not)!!

If you force the app to have only one thread, does the problem go away?

If you force parts of the operation to run in the context of the main
thread (synchronize), does the problem go away?


Experience shows that short sleeps often change the conditions of a
threading bug.  I get especially nervous working with multi-threaded
code when I find sleeps that are there to "Fix" bugs.

Cheers

On 12/04/2012 2:59 p.m., Steve Peacocke wrote:
> Hi John,
>
> All I can say is that I came across this before and resolved it in much
> the same way. I looked up the issue at the time and there was some
> convoluted explanation but in the end I just put it down to the
> programmer (me) being so fast that even a computer couldn't keep up. I
> could live with that explanation.
>
> This was about 10 years ago.
>
> Steve
>
> On Thursday, 12 April 2012, John Bird wrote:
>
>     I came across an oddity in a program that creates lots (eg 30,000+)
>     files exporting selected data from a database.
>     Some files disappear on slower PCs  even though they are created.
>
>       * Files are created in a thread using TFilestream, in a loop
>       * Each file is being checked that it gets created using a if
>         FileExists(filename) in the loop and they always seem to be there.
>       * Files are written at the rate of around 20-30 per second and are
>         mostly around 3KB in size.
>       * At the end of the big loop a list of files created is created by
>         the usual FindNext code.
>       * On some slower PC’s (XP or Server 2003) some files are missing.
>         Faster PC’s with Win7 have all the files.  All PC’s are VMs.
>       * Files missing might range from 10 to 500, never the same number
>       * The problem can be cured by putting a 2 millisecond sleep
>         between creating each file.  1 Millisecond is not enough.
>
>     Can anyone shed any light on why this might be the underlying issue
>     and why a sleep might fix it?
>     John Bird
>
>
>
> --
>
> Steve Peacocke
> Mobile: 0220 612-611
> Linkedin Professional Profile
> <http://nz.linkedin.com/pub/steve-peacocke/1/a06/489>
>
>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at listserver.123.net.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at listserver.123.net.nz with 
> Subject: unsubscribe



_______________________________________________
NZ Borland Developers Group - Delphi mailing list
Post: delphi at listserver.123.net.nz
Admin: http://delphi.org.nz/mailman/listinfo/delphi
Unsubscribe: send an email to delphi-request at listserver.123.net.nz with 
Subject: unsubscribe