[DUG] [computing] Re: Creating lots of files on slow machines

[Stefan Mueller]

Did you try with calling the win32api FlushFileBuffers() function?
See
http://msdn.microsoft.com/en-us/library/windows/desktop/aa364439(v=vs.85).as
px

This function will make sure the file is actually written to the disk and
avoids all the system disk caching functionality.

... or you could use the win32api CreateFile() function with the
FILE_FLAG_NO_BUFFERING flag set.
http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).as
px

Kind Regards,
Stefan Mueller 
_______________________
R&D Manager
ORCL Toolbox LLP, Japan
http://www.orcl-toolbox.com

-----Original Message-----
From: delphi-bounces at listserver.123.net.nz
[mailto:delphi-bounces at listserver.123.net.nz] On Behalf Of David Moorhouse
Sent: Thursday, April 12, 2012 9:14 PM
To: delphi at listserver.123.net.nz
Subject: Re: [DUG] [computing] Re: Creating lots of files on slow machines

Hi Peter

JC and I are working with JB on this project and it has us all stumped.

Only one thread writes these files.  Standard threaded app, with one thread
creating the files, and then signalling another thread when it has finished
to consume the files.

As JB says, the writer thread creates each file, then uses FileExists to
check it is on disk.  This all works ok.

However when the consuming thread does a FileFirst/FileNext scan some of the
files do NOT exist on the disk.

Single threading the app is not an option unfortunately :(

We are nervous about the sleep - hence the call to the list.  Any other help
appreciated.

Cheers


D

On 12/04/12 16:07, Peter Ingham wrote:
> Just wondering of the key word in the problem description is ... "Thread".
>
> Do you have multiple threads writing these files?
>
> Are you sure your code is multi-thread safe?
>
> Are you sure the VCL routines you are using are multi-thread safe (a 
> number are not)!!
>
> If you force the app to have only one thread, does the problem go away?
>
> If you force parts of the operation to run in the context of the main 
> thread (synchronize), does the problem go away?
>
>
> Experience shows that short sleeps often change the conditions of a 
> threading bug.  I get especially nervous working with multi-threaded 
> code when I find sleeps that are there to "Fix" bugs.
>
> Cheers
>
> On 12/04/2012 2:59 p.m., Steve Peacocke wrote:
>> Hi John,
>>
>> All I can say is that I came across this before and resolved it in 
>> much the same way. I looked up the issue at the time and there was 
>> some convoluted explanation but in the end I just put it down to the 
>> programmer (me) being so fast that even a computer couldn't keep up. 
>> I could live with that explanation.
>>
>> This was about 10 years ago.
>>
>> Steve
>>
>> On Thursday, 12 April 2012, John Bird wrote:
>>
>>      I came across an oddity in a program that creates lots (eg 30,000+)
>>      files exporting selected data from a database.
>>      Some files disappear on slower PCs  even though they are created.
>>
>>        * Files are created in a thread using TFilestream, in a loop
>>        * Each file is being checked that it gets created using a if
>>          FileExists(filename) in the loop and they always seem to be
there.
>>        * Files are written at the rate of around 20-30 per second and are
>>          mostly around 3KB in size.
>>        * At the end of the big loop a list of files created is created by
>>          the usual FindNext code.
>>        * On some slower PC’s (XP or Server 2003) some files are missing.
>>          Faster PC’s with Win7 have all the files.  All PC’s are VMs.
>>        * Files missing might range from 10 to 500, never the same number
>>        * The problem can be cured by putting a 2 millisecond sleep
>>          between creating each file.  1 Millisecond is not enough.
>>
>>      Can anyone shed any light on why this might be the underlying issue
>>      and why a sleep might fix it?
>>      John Bird
>>
>>
>>
>> --
>>
>> Steve Peacocke
>> Mobile: 0220 612-611
>> Linkedin Professional Profile
>> <http://nz.linkedin.com/pub/steve-peacocke/1/a06/489>
>>
>>
>>
>> _______________________________________________
>> NZ Borland Developers Group - Delphi mailing list
>> Post: delphi at listserver.123.net.nz
>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> Unsubscribe: send an email to delphi-request at listserver.123.net.nz 
>> with Subject: unsubscribe
>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at listserver.123.net.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at listserver.123.net.nz 
> with Subject: unsubscribe


_______________________________________________
NZ Borland Developers Group - Delphi mailing list
Post: delphi at listserver.123.net.nz
Admin: http://delphi.org.nz/mailman/listinfo/delphi
Unsubscribe: send an email to delphi-request at listserver.123.net.nz with
Subject: unsubscribe