atsnew is much slower that atsdev

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Here is a little reproducer

% cat loop.ats

for i in range(20):
    test(executable="/bin/ls")

This takes atsnew about 1 minute to go through.  It seems to pause for 3 
seconds after each job. 
With atsdev there is no delay, and it finishes in about 1 second.  There is 
also a very long startup cost with atsnew.  Running atsnew on an empty file 
takes about 7 seconds before anything happens.

3 seconds is quite a headache for my project, since we have about 4000+ ATS 
tests in our test suite.

Can anything be done?

Original issue reported on code.google.com by brian.mc...@gmail.com on 17 Jun 2011 at 12:06

[GoogleCodeExporter]

Are you using the one that TJ put in public? I made a change that should reduce 
the use of this pause, and another that makes it controllable by you.  

manager.machine.naptime = 1 will change it from the default 3.

The naptime second pause is used when we have checked the running jobs, and 
found that either the machine is full, or that we could not find a job eligible 
to start. 

Please see if the machine object has a naptime attribute in the one you are 
trying; and if it does let me know if you still see this problem.  It should be 
firing up your initial jobs until the machine is full without any such pauses. 
If you set naptime to zero the consequence might be excessive time use by the 
ats program as it spins without much hope starting anything new. You can try 
changing it to 1. 

I observed a one second time for ats with no input on my Mac. I suspect the 
time you saw of seven seconds was LC-related. However, even if it was seven 
seconds, it isn't clear to me that that would be a problem when you plan to run 
thousands of tests.

Original comment by pfdub...@gmail.com on 22 Jun 2011 at 7:48

[GoogleCodeExporter]

I do get the one minute timing on /bin/ls .  Something looks wrong to me, will 
investigate. It should find more than one of these jobs exiting within the nap 
period, so that more than one starts up. Using naptime = 1 changed it to 16 
seconds.

Original comment by pfdub...@gmail.com on 22 Jun 2011 at 7:56

• Changed state: Started

[GoogleCodeExporter]

Paul,

Yes, I was using the ats that TJ put in public.  

Thanks for the tip about naptime.  I can change the naptime to another value, 
and   changing it to .2 seemed to work pretty well for me.  I will play around 
with this.  I have noticed in the past that ats ran at 100% cpu according to 
top.  With naptime=.2 that seems to go down to ~20%.  I can easily add this 
into my wrapper script around the ATS.   

There is still a long startup cost that is not affected by this.   Yes, not a 
problem when running the entire suite, but when running a single test case 
(which would be a fairly common thing to), the delay is a bit painful, 
especially since I'm not sure why its necessary.  The old atsb was pretty quick 
on startup.


% time /usr/apps/atsdev/bin/atsb
0.056u 0.046s 0:00.19 47.3%     0+0k 0+0io 0pf+0w


% time /usr/apps/atsnew/bin/ats
0.079u 0.083s 0:06.68 2.2%      0+0k 0+0io 0pf+0w

Original comment by brian.mc...@gmail.com on 23 Jun 2011 at 5:23

[GoogleCodeExporter]

I may have figured out the startup delay you observe. It does not occur on my 
Mac; on alastor it is a delay that is before anything happens at all. So I 
asked myself, what happens before it even prints ATS blah blah blah, and the 
answer is, machine configuration. So I tried 

setenv MACHINE_TYPE generic

and the delay disappears. I believe the new ats also involved changing to using 
the chaosMulti machine file from chaos, so that would explain the change you 
observed. The latter was a one-none machine file we used until we got going on 
the real one.

So I speculate the delay is the chaosMulti machine file making system calls to 
inquire about available nodes / resources.  It may be that this is avoidable if 
you aren't going off node after all.  I will call this to Nu Ai's attention.

Original comment by pfdub...@gmail.com on 23 Jun 2011 at 6:55