[users at bb.net] Worker running extremely slow

[Dan Kegel]

Did all that, years ago, over and over.

- launchd does not give access to the foreground console and opengl
- restart options on terminal only work if the system is shut down
nicely via Apple apis, those settings are ignored if reboot is
triggered via shutdown -r now

Here are my notes on setting up a mac worker:

Tell it to never sleep: sudo systemsetup -setcomputersleep Never
Tell it to never sleep the display: sudo systemsetup -setdisplaysleep Never
Tell it to boot after power fail: sudo pmset autorestart 1
In System Preferences / Sharing, turn on remote login (ssh) for all users.
In System Preferences / Bluetooth, turn off bluetooth.
In System Preferences / Energy, turn off automatic graphics switching
(in hopes that will make OpenGL tests more repeatable/reliable)
In System Preferences / Date and Time, select "Set date and time
automatically".  (This is the default, but it doesn't happen unless
you visit this settings page once.)  (Failing to do this may lead to
build failures, e.g. in gst-plugins-good, whose configure script makes
sure newly created files are not older than source files.)
In Terminal.app Preferences > Profiles > Shell > "When the shell
exits:" > "Close the window", tell it to close no matter what's
running.
Set the job runner to start at boot
- do System Preferences / Users / buildbot / Login Items / + /
$HOME/src/bot/run-bslave
- do System Preferences / Users / Login Options / Automatic login / buildbot

That run-bslave script does

    # caffeinate's assertions only last 15 seconds regardless of -t,
    # so reissue constantly.
    ./bslave-watchdog &
    # what the hey, let's use it as intended
    caffeinate -d -t 100000000 sh bslave.sh run

I used to use a one-shot runner that exits after one build, but that
was back in buildbot 0.8 days, and I haven't figured out how to bring
that feature
back for buildbot 1.0... so I don't get to reboot after each build
anymore on mac.

I'm rather grumpy and bitter about all this.
- Dan

On Mon, Mar 25, 2019 at 9:45 AM Yngve N. Pettersen <yngve at vivaldi.com> wrote:
>
> Hi,
>
> We have been doing frequent reboots of Mac workers, using autologin.
>
> Until recently, when I changed to using launchd, we had occasional
> problems with the Applescript timing out on reboot, I have not seen any
> similar issues after converting to launchd, so far.
>
> Regarding terminals, you should check the restart options, which includes
> a setting for reopening windows, which should be turned off. You might
> also want to check the Terminal preference for closing the window, and
> consider selecting "close if exited cleanly"
>
> About display blanking, you may perhaps find something in the the Energy
> Saver or the Screen Saver preferences.
>
> On Mon, 25 Mar 2019 17:07:32 +0100, Dan Kegel <dank at kegel.com> wrote:
>
> > I had a hard time launching the build worker on Mac.
> > Launchd was fine until I needed them to them to
> > have access to OpenGL, at which point I had to
> > switch to setting the machine to autologin as the builder
> > account, with a start item to launch the worker.
> >
> > That fails about 1% of them time because the
> > autologin mechanism is not reliable.
> > It's also hard to keep the screen from blanking
> > and causing opengl tests to fail.  caffeinate helps,
> > but not completely.
> > And if you want to reboot after every build to clear
> > kernel bugs due to OpenGL driver problems, you're
> > even more hosed; in recent versions of Mac OS, there is
> > abolutely, positively no way for the worker
> > to trigger a reboot without stale terminals piling up,
> > and after 1000 reboots, the system is out of fds, and fails.
> >
> > The autologin reliability problem could probably be
> > worked around by having multiple autologin jobs
> > (a primary one, and a backup in case the primary
> > one didn't work).
> > The reboot problem could probably be worked around
> > but I ran out of energy after trying a dozen workarounds.
> >
> > Mac OS has been working very hard to keep me from
> > running buildbot workers on it, and I'm a bit bitter :-(
> > - Dan
> >
> > On Mon, Mar 25, 2019 at 8:55 AM Chris Mumford <cmumford at cmumford.com>
> > wrote:
> >>
> >> I'm running it from the command-line directly -- no launchd. I am
> >> getting an file not found error when building via bb and not the
> >> command-line. Looking into that now.
> >>
> >> On Sat, Mar 23, 2019 at 11:24 AM Charles Lepple <clepple at gmail.com>
> >> wrote:
> >>>
> >>> On Mar 23, 2019, at 2:06 PM, Chris Mumford <cmumford at cmumford.com>
> >>> wrote:
> >>> >
> >>> > I've been using Buildbot several years on a project without any
> >>> issues. A few months ago, for no apparent reason, my macOS builder
> >>> (Mojave) started building really slow - so slow that it times out. My
> >>> two Windows builders run just fine AND if I manually run the build
> >>> script then it also builds just fine on the mac. The process being run
> >>> is only taking about 5% of the CPU when executed from Buildbot. A
> >>> build that normally finishes in 5 min. times out after 5 hours!
> >>>
> >>> How are you starting the macOS builder? If you are starting via
> >>> launchd, what options are you using?
> >>>
> >>> In a similar vein, I was trying to figure out why I couldn't wrap a
> >>> build in "caffeinate" on an older Mac. Same sort of thing: starting
> >>> the Buildbot slave from a launchd plist would not work (timeout
> >>> exceptions when the Mac went to sleep), but running it manually via
> >>> SSH with caffeinate kept the box awake. (Still trying to figure out a
> >>> solution for that, but I use that box for streaming music now, so it's
> >>> asleep less frequently.)
> >>>
> >>> > _______________________________________________
> >>> > users mailing list
> >>> > users at buildbot.net
> >>> > https://lists.buildbot.net/mailman/listinfo/users
> >>>
> >> _______________________________________________
> >> users mailing list
> >> users at buildbot.net
> >> https://lists.buildbot.net/mailman/listinfo/users
> > _______________________________________________
> > users mailing list
> > users at buildbot.net
> > https://lists.buildbot.net/mailman/listinfo/users
>
>
> --
> Sincerely,
> Yngve N. Pettersen
> Vivaldi Technologies AS
>