[DUG] [computing] Re: Creating lots of files on slow machines

[Kyley Harris]

I know this might be a stupid question.. but what filenaming convention are
you using?

its not a random name based on Time or something where file names are
accidentally reused?
im sure its not, but have to ask.

On Fri, Apr 13, 2012 at 12:14 AM, David Moorhouse
<delphi at moorhouse.net.nz>wrote:

> Hi Peter
>
> JC and I are working with JB on this project and it has us all stumped.
>
> Only one thread writes these files.  Standard threaded app, with one
> thread creating the files, and then signalling another thread when it
> has finished to consume the files.
>
> As JB says, the writer thread creates each file, then uses FileExists to
> check it is on disk.  This all works ok.
>
> However when the consuming thread does a FileFirst/FileNext scan some of
> the files do NOT exist on the disk.
>
> Single threading the app is not an option unfortunately :(
>
> We are nervous about the sleep - hence the call to the list.  Any other
> help appreciated.
>
> Cheers
>
>
> D
>
> On 12/04/12 16:07, Peter Ingham wrote:
> > Just wondering of the key word in the problem description is ...
> "Thread".
> >
> > Do you have multiple threads writing these files?
> >
> > Are you sure your code is multi-thread safe?
> >
> > Are you sure the VCL routines you are using are multi-thread safe (a
> > number are not)!!
> >
> > If you force the app to have only one thread, does the problem go away?
> >
> > If you force parts of the operation to run in the context of the main
> > thread (synchronize), does the problem go away?
> >
> >
> > Experience shows that short sleeps often change the conditions of a
> > threading bug.  I get especially nervous working with multi-threaded
> > code when I find sleeps that are there to "Fix" bugs.
> >
> > Cheers
> >
> > On 12/04/2012 2:59 p.m., Steve Peacocke wrote:
> >> Hi John,
> >>
> >> All I can say is that I came across this before and resolved it in much
> >> the same way. I looked up the issue at the time and there was some
> >> convoluted explanation but in the end I just put it down to the
> >> programmer (me) being so fast that even a computer couldn't keep up. I
> >> could live with that explanation.
> >>
> >> This was about 10 years ago.
> >>
> >> Steve
> >>
> >> On Thursday, 12 April 2012, John Bird wrote:
> >>
> >>      I came across an oddity in a program that creates lots (eg 30,000+)
> >>      files exporting selected data from a database.
> >>      Some files disappear on slower PCs  even though they are created.
> >>
> >>        * Files are created in a thread using TFilestream, in a loop
> >>        * Each file is being checked that it gets created using a if
> >>          FileExists(filename) in the loop and they always seem to be
> there.
> >>        * Files are written at the rate of around 20-30 per second and
> are
> >>          mostly around 3KB in size.
> >>        * At the end of the big loop a list of files created is created
> by
> >>          the usual FindNext code.
> >>        * On some slower PC’s (XP or Server 2003) some files are missing.
> >>          Faster PC’s with Win7 have all the files.  All PC’s are VMs.
> >>        * Files missing might range from 10 to 500, never the same number
> >>        * The problem can be cured by putting a 2 millisecond sleep
> >>          between creating each file.  1 Millisecond is not enough.
> >>
> >>      Can anyone shed any light on why this might be the underlying issue
> >>      and why a sleep might fix it?
> >>      John Bird
> >>
> >>
> >>
> >> --
> >>
> >> Steve Peacocke
> >> Mobile: 0220 612-611
> >> Linkedin Professional Profile
> >> <http://nz.linkedin.com/pub/steve-peacocke/1/a06/489>
> >>
> >>
> >>
> >> _______________________________________________
> >> NZ Borland Developers Group - Delphi mailing list
> >> Post: delphi at listserver.123.net.nz
> >> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> >> Unsubscribe: send an email to delphi-request at listserver.123.net.nzwith Subject: unsubscribe
> >
> >
> > _______________________________________________
> > NZ Borland Developers Group - Delphi mailing list
> > Post: delphi at listserver.123.net.nz
> > Admin: http://delphi.org.nz/mailman/listinfo/delphi
> > Unsubscribe: send an email to delphi-request at listserver.123.net.nz with
> Subject: unsubscribe
>
>
> _______________________________________________
> NZ Borland Developers Group - Delphi mailing list
> Post: delphi at listserver.123.net.nz
> Admin: http://delphi.org.nz/mailman/listinfo/delphi
> Unsubscribe: send an email to delphi-request at listserver.123.net.nz with
> Subject: unsubscribe
>



-- 
Kyley Harris
Harris Software
+64-21-671-821
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://listserver.123.net.nz/pipermail/delphi/attachments/20120413/4e5b76a0/attachment.html