[DUG] [computing] Re: Creating lots of files on slow machines

[David Moorhouse]

Hi Peter

JC and I are working with JB on this project and it has us all stumped.

Only one thread writes these files.  Standard threaded app, with one 
thread creating the files, and then signalling another thread when it 
has finished to consume the files.

As JB says, the writer thread creates each file, then uses FileExists to 
check it is on disk.  This all works ok.

However when the consuming thread does a FileFirst/FileNext scan some of 
the files do NOT exist on the disk.

Single threading the app is not an option unfortunately :(

We are nervous about the sleep - hence the call to the list.  Any other 
help appreciated.

Cheers


D

On 12/04/12 16:07, Peter Ingham wrote:
> Just wondering of the key word in the problem description is ... "Thread".
>
> Do you have multiple threads writing these files?
>
> Are you sure your code is multi-thread safe?
>
> Are you sure the VCL routines you are using are multi-thread safe (a
> number are not)!!
>
> If you force the app to have only one thread, does the problem go away?
>
> If you force parts of the operation to run in the context of the main
> thread (synchronize), does the problem go away?
>
>
> Experience shows that short sleeps often change the conditions of a
> threading bug.  I get especially nervous working with multi-threaded
> code when I find sleeps that are there to "Fix" bugs.
>
> Cheers
>
> On 12/04/2012 2:59 p.m., Steve Peacocke wrote:
>> Hi John,
>>
>> All I can say is that I came across this before and resolved it in much
>> the same way. I looked up the issue at the time and there was some
>> convoluted explanation but in the end I just put it down to the
>> programmer (me) being so fast that even a computer couldn't keep up. I
>> could live with that explanation.
>>
>> This was about 10 years ago.
>>
>> Steve
>>
>> On Thursday, 12 April 2012, John Bird wrote:
>>
>>      I came across an oddity in a program that creates lots (eg 30,000+)
>>      files exporting selected data from a database.
>>      Some files disappear on slower PCs  even though they are created.
>>
>>        * Files are created in a thread using TFilestream, in a loop
>>        * Each file is being checked that it gets created using a if
>>          FileExists(filename) in the loop and they always seem to be there.
>>        * Files are written at the rate of around 20-30 per second and are
>>          mostly around 3KB in size.
>>        * At the end of the big loop a list of files created is created by
>>          the usual FindNext code.
>>        * On some slower PC’s (XP or Server 2003) some files are missing.
>>          Faster PC’s with Win7 have all the files.  All PC’s are VMs.
>>        * Files missing might range from 10 to 500, never the same number
>>        * The problem can be cured by putting a 2 millisecond sleep
>>          between creating each file.  1 Millisecond is not enough.
>>
>>      Can anyone shed any light on why this might be the underlying issue
>>      and why a sleep might fix it?
>>      John Bird
>>
>>
>>
>> --
>>
>> Steve Peacocke
>> Mobile: 0220 612-611
>> Linkedin Professional Profile
>> <http://nz.linkedin.com/pub/steve-peacocke/1/a06/489>
>>
>>
>>
>> _______________________________________________
>> NZ Borland Developers Group - Delphi mailing list
>> Post: delphi at listserver.123.net.nz
>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> Unsubscribe: send an email to delphi-request at listserver.123.net.nz with Subject: unsubscribe
>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at listserver.123.net.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at listserver.123.net.nz with Subject: unsubscribe