[DUG] [computing] Re: Creating lots of files on slow machines
Kyley Harris
Kyley at harrissoftware.com
Fri Apr 13 03:21:41 NZST 2012
One last thought. perhaps something to try, or remove, just to see if it
has an impact.
I cannot tell from your original post.
Are you Creating a file, and then immediately using fileexists to verify
that it was made.
or are you creating all the files, and then looping fileexists against a
known list of filenames you expect?
I wonder if you should remove the fileexists check from the thread, based
on the fact that
if you didn't raise an error then the files will exist. perhaps the speed
of the checks
is creating some kind of file attribute conflict. Only suggesting it as a
way of elimintaing a possible weirdness
I've certainly had a number of occasions in windows where filestreams that
I've opened and closed
cannot be deleted from explorer because it thinks I've still got the file
locked.. I put it down to an
internal windows or vcl issue.
On Fri, Apr 13, 2012 at 3:06 AM, Kyley Harris <Kyley at harrissoftware.com>wrote:
> I know this might be a stupid question.. but what filenaming convention
> are you using?
>
> its not a random name based on Time or something where file names are
> accidentally reused?
> im sure its not, but have to ask.
>
> On Fri, Apr 13, 2012 at 12:14 AM, David Moorhouse <delphi at moorhouse.net.nz
> > wrote:
>
>> Hi Peter
>>
>> JC and I are working with JB on this project and it has us all stumped.
>>
>> Only one thread writes these files. Standard threaded app, with one
>> thread creating the files, and then signalling another thread when it
>> has finished to consume the files.
>>
>> As JB says, the writer thread creates each file, then uses FileExists to
>> check it is on disk. This all works ok.
>>
>> However when the consuming thread does a FileFirst/FileNext scan some of
>> the files do NOT exist on the disk.
>>
>> Single threading the app is not an option unfortunately :(
>>
>> We are nervous about the sleep - hence the call to the list. Any other
>> help appreciated.
>>
>> Cheers
>>
>>
>> D
>>
>> On 12/04/12 16:07, Peter Ingham wrote:
>> > Just wondering of the key word in the problem description is ...
>> "Thread".
>> >
>> > Do you have multiple threads writing these files?
>> >
>> > Are you sure your code is multi-thread safe?
>> >
>> > Are you sure the VCL routines you are using are multi-thread safe (a
>> > number are not)!!
>> >
>> > If you force the app to have only one thread, does the problem go away?
>> >
>> > If you force parts of the operation to run in the context of the main
>> > thread (synchronize), does the problem go away?
>> >
>> >
>> > Experience shows that short sleeps often change the conditions of a
>> > threading bug. I get especially nervous working with multi-threaded
>> > code when I find sleeps that are there to "Fix" bugs.
>> >
>> > Cheers
>> >
>> > On 12/04/2012 2:59 p.m., Steve Peacocke wrote:
>> >> Hi John,
>> >>
>> >> All I can say is that I came across this before and resolved it in much
>> >> the same way. I looked up the issue at the time and there was some
>> >> convoluted explanation but in the end I just put it down to the
>> >> programmer (me) being so fast that even a computer couldn't keep up. I
>> >> could live with that explanation.
>> >>
>> >> This was about 10 years ago.
>> >>
>> >> Steve
>> >>
>> >> On Thursday, 12 April 2012, John Bird wrote:
>> >>
>> >> I came across an oddity in a program that creates lots (eg
>> 30,000+)
>> >> files exporting selected data from a database.
>> >> Some files disappear on slower PCs even though they are created.
>> >>
>> >> * Files are created in a thread using TFilestream, in a loop
>> >> * Each file is being checked that it gets created using a if
>> >> FileExists(filename) in the loop and they always seem to be
>> there.
>> >> * Files are written at the rate of around 20-30 per second and
>> are
>> >> mostly around 3KB in size.
>> >> * At the end of the big loop a list of files created is created
>> by
>> >> the usual FindNext code.
>> >> * On some slower PC’s (XP or Server 2003) some files are
>> missing.
>> >> Faster PC’s with Win7 have all the files. All PC’s are VMs.
>> >> * Files missing might range from 10 to 500, never the same
>> number
>> >> * The problem can be cured by putting a 2 millisecond sleep
>> >> between creating each file. 1 Millisecond is not enough.
>> >>
>> >> Can anyone shed any light on why this might be the underlying
>> issue
>> >> and why a sleep might fix it?
>> >> John Bird
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Steve Peacocke
>> >> Mobile: 0220 612-611
>> >> Linkedin Professional Profile
>> >> <http://nz.linkedin.com/pub/steve-peacocke/1/a06/489>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> NZ Borland Developers Group - Delphi mailing list
>> >> Post: delphi at listserver.123.net.nz
>> >> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> >> Unsubscribe: send an email to delphi-request at listserver.123.net.nzwith Subject: unsubscribe
>> >
>> >
>> > _______________________________________________
>> > NZ Borland Developers Group - Delphi mailing list
>> > Post: delphi at listserver.123.net.nz
>> > Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> > Unsubscribe: send an email to delphi-request at listserver.123.net.nzwith Subject: unsubscribe
>>
>>
>> _______________________________________________
>> NZ Borland Developers Group - Delphi mailing list
>> Post: delphi at listserver.123.net.nz
>> Admin: http://delphi.org.nz/mailman/listinfo/delphi
>> Unsubscribe: send an email to delphi-request at listserver.123.net.nz with
>> Subject: unsubscribe
>>
>
>
>
> --
> Kyley Harris
> Harris Software
> +64-21-671-821
>
--
Kyley Harris
Harris Software
+64-21-671-821
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://listserver.123.net.nz/pipermail/delphi/attachments/20120413/ad408083/attachment-0001.html
More information about the Delphi
mailing list