Build Servers

Here's how I build everything. There are many ways to do this - see way below. Your choice is yours but lets discuss some requirements. I have a command to build everything that I can put in cron or invoke by hand. It calls scripts for each of the platforms I'm building. There is a pattern in each of those subordinate scripts - create the environment, connect to it, do a rake <os>:setup:<target>, rake, rake package, rake clobber

There are assumptions in that simple scheme you need to understand. The first one is that all the 'builders`, be they real boxes or virtual machines need to access the (1)source code and (2) they need a place to build to, and (3) we need a way to test that build and then we (4) need a way to copy it to the download site. I use an NFS server for (1) and (4) because Linux and osx machines can connect to NFS without a lot of trouble and Virtualbox VM's can use it as a Shared Folder.

Consider those Machines on the network. Some are real and some are Virtual Machines (VM) and some are/could be slightly less light weight chroots. But they are machines - with their own OS installed. If I can build Shoes for 6 (say) machines then there are 6 OS, virtual or physical, that have the build tools installed and the capability to connect to the source (1), deposit their build (4) and can be tested(3) if something goes wrong on a build an it will.

I have a physical Mac Mini with monitor/keyboard/mouse/desk-space and a Raspberry Pi3 with all the hardware it need sto be a standalone computer. I have Virtual Machines (I use Oracle's Virtualbox) for the Window 7 testing and another VM for testing Win10 and a couple more VM for testing non debian Linux like Suse or Fedora/Red Hat. One can do this in OSX or FreeBSD instead of Linux if one is willing to work hard.

A Mac system with a licensed ability to run Mac VM's (and you have valid Windows and OSX licenses) would be ideal. That is not what I have. Let's look at the cron called script in ~/bin that I use to build everything.

#!/bin/bash
# Build all binary variations of shoes. This runs under
# my crontab so user name may not be set up for chroot
schroot -c debx86 -u ccoupe -- ~/bin/nbuild-shoes-x86_64
~/bin/nbuild-shoes-xwin7
# ssh to Mac mini and build. Fragile or brittle? You decide.
~/bin/nbuild-shoes-mavericks
~/bin/nbuild-shoes-yosemite
~/bin/nbuild-shoes-rpi
~/bin/nbuild-shoes-bsd

X86_64 Patterns for all

I change to a schroot and run a ~/bin/build-shoes-x86_64 (a bash script). Inside a schroot I'm no longer running Mint 18.1 - its a minimal Ubuntu 16.04 and I do mean minimal but I did have install linux in there with debootstrap and the apt-get install all the packages Shoes needs. Aside from the huge utility of debootstrap, schroots mount the users directory /home/ccoupe/ for me and it trims the $PATH variable down - it strips out all the Mint 18.1 rvm goodness - because I don't have rvm in the chroot. We may not even have ruby and if we do it's likely to be incorrect for our usage. Lets look at the nbuild-shoes-x86_64 script.

#!/bin/bash
echo "Build Shoes X86_64 for Linux"
cd /home/ccoupe/Projects/shoes3
rm -rf build_target
rake linux:setup:x86_64_linux
rake
rake package
rake clobber
ls -ld pkg/*-x86_64.install

I change directory to where the Shoes source is, clean up any left over build_target file, and tell rake to setup for an x86_64 build in Linux. Then it does the rake. You could run the Shoes here, if building by hand.The rake package creates the ...x86_64.install file and copy's it to the pkg/ This is the pattern for all builds, all architectures, all machines.

Where is Ruby

If you've been paying attention, you should ask "If is the ruby in this chroot is bad or non existent how can rake run? How does it know where the gems are?". When you did the rake linux:setup:x86_64_linux it created that build_target file and inside that file is TGT_ARCH=x86_64-linux. When you do the simple rake it sees you have a build_target file it reads it and uses that TGT_ARCH values to look for a "#{TGT_ARCH}-custom.yaml and loads it into rake. Mine looks like this:

Ruby: /usr/local
Deps: 
Gemloc: /home/ccoupe/Projects/gems/rb23
Extloc: /home/ccoupe/Projects/gems/rb23
Exts:
Gems:
InclGems: 
  - chipmunk-6.1.3.4
  - sqlite3-1.3.13
  - mini_portile2-2.2.0
  - nokogiri-1.8.0
  - rb-readline-0.5.4
  - byebug-9.0.6
  - ffi-1.9.18
    # picky needs:
  - activesupport-5.1.2
  - concurrent-ruby-1.0.5
  - i18n-0.8.6
  - multi_json-1.12.1
  - picky-4.31.3
  - rack_fast_escape-2009.06.24
  - thread_safe-0.3.6
  - tzinfo-1.2.3
  - url_escape-2009.06.24
  - yajl-ruby-1.3.0
  # typhoeus
  - ethon-0.10.1
  - typhoeus-1.1.2
Debug: false

I've declared where my Ruby is, (/usr/local/[bin,lib,etc,share..]) which just happens to survive the $PATH rewrite of the schroot call. That also means I build the ruby by hand inside the schroot. It's not that hard to do. - See the wiki for instructions for how to to build Ruby for the machines of your interest.

"OK, I guess, Then what does that Gemloc: and the Deps: line do?"

Deps

Shoes needs many libraries (,dll, .so, .dylib) like libcairo, libpango and many more just to compile and link. In Shoes we call these the Dependencies. Remember how I said I had to do a bunch of apt-get installs when setting my schroot? That put those libraries in /usr/lib locations of the chroot (not in my outer Mint 18.1's /usr/lib) Since this Dep: line is empty it means to use the system locations. When we get to Windows and OSX this won't be so simple or pleasant so we provide a way to pre-compile them and distribute them separately. We'll talk more about this.

Gems

Gemloc: in the custom.yaml is where the gems enumerated with InclGems: are located. "Why aren't they in that /usr/local/lib/ruby you created?". They were, might still be, once you install them by hand from the schroot cmd line, gem install --no-doc . There is no way to know if Windows, OSX, or Linux users have all the tools and libraries needed to build 'native'. Most your end users will not the tools - You have to build the gems for them. Once you've done that, why not save the prebuild gems somewhere for other developers to quickly use? There are tools in gemutils/ to make that easier for you.

Blank Extloc, Ext, and Gem lines in yaml

These are historical. Some gems were actually extentions but anymore. Once you've got things working you can try deleting them from your custom.yaml and see what happens. You'll be be pretty savvy by then. Do it later.

Recap

We've take a peek at the build process - time for a recap. I export ~/home/ccoupe/Projects via NFS or VirtualBox's shared folder or schroot. Those machines can connect to and R/W to Projects/shoes3, Projects/gems, Projects/shoesdeps. Projects/shoes3 has the git clone/repo and .gitignore and other git setup, usually. Each of those machines has pre-built dependencies, pre-built Ruby and pre-built gems. I hesitate to call it a design pattern but its a design concept you should accept until have enough cross platform hardware and skills to fix. Every platform has it's unique quirks. I'm going to go through in increased complexity.

Machine Details

Don't get overwhelmed. Read once or twice or thrice.

Raspberry PI

Just another Debian right? It's actually a pi3 with its own boot dive (USB) The pi2-custom.yam has this (abbreviated)

Ruby: /home/ccoupe/shoesdeps/pi3 
Deps: 
Gemloc: /home/ccoupe/Projects/gems/rb23
Extloc: /home/ccoupe/Projects/gems/rb23
...
...

Lets parse and think. I built a Ruby on the Pi3 and it put on the Pi3's disk /home/ccoupe/shoesdeps/ which IS NOT THE SAME AS '/home/ccoupe/Projects/shoesdeps. PI compiles are slow and NFS writing just makes 'very slow' into "I may dies before this finishes!" What about the ~/bin/nbuild-shoes-rpi? Pay attention, it gets fun.

#!/bin/bash
ssh ccoupe@pi3 <<'ENDSSH'
echo "Build Shoes rpi3"
export NFS_ALTP=/home/ccoupe/build/
cd ~/Projects/shoes3
rm -f build_target
rake linux:setup:pi2
rake
rake package
rake clobber
ENDSSH
ls -ld /home/ccoupe/Projects/shoes3/pkg/*armhf.install

Whoa! We create a script from the bash script and ssh in to run that script. So we have ssh setup between 'bronco' my big Linux box and machine named pi3. Thats a pain (hint: man ssh-copy-id).

What is is that NFS_ALTP env var doing? Instead of writing the compiled files (.o, .so, ..) via slow NFS it puts them on local (to pi3) disk. Still slow but way faster than NFS. It gets sorted out in the package step which detects the env var has been set. Yes, you do need the trailing slash on that export NFS_ALTP= well see this again with OSX.

Like x86_64_linux, we compiled our own Ruby and the deps (via apt-get install) and installed gems and copied the gems to the NFS server at /home/ccoupe/Projects/gems/rb23

OSX 10.9 Mavericks

We have to build Ruby by hand (just like we have to do for Linux) and we have to install and copy the gems. But we have to build those dependencies libraries. More about below.

I have a real mac mini -the cheapest, underpowered that I cared to buy. A slow disk but it is a lot faster than NFS writing. It also uses the NFS_ALTP trick to speed up compiling. Here's the relevant part of mavericks-custom.yaml

Ruby: /Users/ccoupe/shoesdeps/10.9
Deps: /Users/ccoupe/shoesdeps/10.9
Zlib: /Users/ccoupe/shoesdeps/10.9/lib
Gemloc: /Users/ccoupe/gems/rb23
Extloc: /Users/ccoupe/gems/rb23
Exts:
Gems:
InclGems:

`/Users/ccoupe is an OSX home directory - not linux. Its on that mac mini the NFS Project is mounted as /Users/ccoupe/Projects. The build script - nbuild-osx-yosemite shows some oddities

#!/bin/bash
ssh [email protected] <<'ENDSSH'
echo "Build Shoes OSX 10.0.9 Mavericks
source .bashrc
export PATH=$PATH:/usr/local/bin
echo $PATH
rvm use 2.3.4
cd /Users/ccoupe/Projects/shoes3
rm -f build_target
rake osx:setup:yosemite
rake
rake package
rake clobber
ENDSSH
ls -ld /home/ccoupe/Projects/shoes3/pkg/*osx-10.9.tgz

It's does that clever thing of feeding a script to ssh. Why does it source .bashrc? 1) That's where the NFS_ALTP is declared and 2) that sets up rvm. so it do the rvm use. "Wait, I thought we weren't using RVM?" Not entirely true. We need a Ruby of the same major.minor (2.3) to run the rake files but we don't want to put that /Users/ccoupe/shoesdeps/10.9 in my $PATH. Just don't do it. I probably don't need the /usr/local/bin, a hangover from playing with HomeBrew which we definitely do not use. Home brew installed libraries (Shoes Deps) tend to be bleeding edge, and not stable enough for an orderly Shoes. I speak from experience.

We build our dependent libraries (Shoes Deps) from source and it is a pain in the behind so we don't do it very often or lightly. What is in shoesdeps/10.9? bin/ etc/ include/ lib/ share/ and ssl/ directories where the deps build placed them. So Ruby: /Users/ccoupe/shoesdeps/10.9/bin/ruby is where we installed the Ruby we built by hand. How to build the deps and Ruby is already documented in the Wiki (hint: there is a shoesdep/src the NFS ~/Project mount)

One saving grace in the scheme is the shoesdeps/10.9 can be zipped up and shared with other C developers.

OSX 10.10 Yosemite

Apple changes and deprecates their OSX much more often than Microsoft or Linux. In fact, Shoes OSX 10.9 is broken for 10.13 users so I created a different build that works for 10.10 up - might work for 10.9. Let's look at the top part of yosemite-custom.yaml

Ruby: /Users/ccoupe/shoesdeps/10.10
Deps: /Users/ccoupe/shoesdeps/10.10
Gemloc: /Users/ccoupe/gems/rb23
InclGems:
  - chipmunk-6.1.3.4
  - sqlite3-1.3.13

Less legacy declares. Can we modify the build script to be less legacy?

#!/bin/bash
ssh ccoupe@mini <<'ENDSSH'
echo "Build Shoes OSX 10.10 Yosemite"
source .bashrc
rvm use 2.3.4
cd /Users/ccoupe/Projects/shoes3
rm -f build_target
rake osx:setup:yosemite
rake
rake package
rake clobber
ENDSSH
ls -ld /home/ccoupe/Projects/shoes3/pkg/*osx-10.10.tgz

Important secret - I copied (cp -r) shoesdep/10.9 to shoesdeps/10.10 and then built my Ruby (2.3.6) and installed that in shoesdeps/10.10/[bin|include|lib] This not always safe since it doesn't remove the old ruby inside 10.10 (copy of of 10.9) - since both were Ruby 2.3 it's likely to succeed. I speak with experience.

Windows - Native compile

I rarely fire up the Win7 VM to build shoes. It's just an awful experience but if you want to, it can work. However, I often fire up that Win7 VM to test the cross compile. Native building is documented in the wiki. I recommend installing RubyInstaller (2.3.4) and it's msys devkit

Windows - Cross compile.

The nbuild-shoes-xwin7, gets called by the master build script. It fits into the general pattern but it is different. On Linux (or bsd, osx) build hosts you have to install the Mingw-64 toolchain. I recommend installing RubyInstaller (2.3.4) and it's msys devkit on the VM - we need to use it for getting our gems built.

Lets look at xwin7-custom.yaml (x stands for cross compile).

Deps: /home/ccoupe/Projects/shoesdeps/mingw
Ruby: /home/ccoupe/Projects/shoesdeps/mingw/ruby234
Gemloc: /home/ccoupe/Projects/gems/rb23
Extloc: /home/ccoupe/Projects/gems/rb23
Gems:
InclGems:

Don't Panic yet!

#!/bin/bash
echo "Build Shoes GTK Widgets for MinGW32 Windows"
source ~/bin/bash-chroot
source ~/.rvm/scripts/rvm
cd /home/ccoupe/Projects/shoes3
rm -f build_target
rvm use 2.3.6
rake linux:setup:xwin7
rake
rake package
rake clobber
ls -ld pkg/*.exe

We don't login to a different box, VM, or chroot - Just running on the host Linux box. You should peek at make/linux/xwin/env.rb Let's heighten the panic level one step at a time :-) shoesdeps/mingw can be downloaded or you could gain some knowledge after a day or two of building libraries from the command line. (the wiki has instructions).

Ruby is a very difficult thing to cross compile to Windows- all to often it wants to run an .exe at some step and that won't work. Note: building the gtk3 libraries also has this problem. Sometimes Ruby and Gtk fix this and sometimes they regress. It's not something those developers think about often.

As you know the Ruby that runs the build (rake tasks) does not have to be the Ruby included in Shoes. In fact they rarely are the same set of files. If you don't want to build Ruby, or it fails to build in a cross compile setup you can copy the Ruby from the RubyInstaller's ruby on your Windows VM. That's likely what I did for Ruby: /home/ccoupe/Projects/shoesdeps/mingw/ruby234 I installed all the Shoes required gems in Windows too. Then copy things to shoesdeps/ming/ruby234 and move the gems to Gemloc: /home/ccoupe/Projects/gems/rb23

FreeBSD 11.1

FreeBSD 11.1 is difficult to setup. It has dependencies several years newer that Shoes prefers and as a VirtualBox Guest OS, it has issues. It worked once and it was a lot like linux except when it wasn't. The freebsd-custom.yaml snippet:

Ruby: /opt
Deps: 
Gemloc: /usr/home/ccoupe/gems/rb23
InclGems: 

We do have to build our Ruby and since you shouldn't put user code /usr/local in freebsd, I used /opt when build ruby. Deps: are empty so we'll use whatever freebsd 11.1 pkg install <name> provides. And. we built the Shoes gems with the ruby in /opt and copied them to /usr/home/ccoupe/rb23 which is virtualbox shared folder of ~/Projects - same as the nfs export or the Win7 shared folder.

The build script has a timing problem - some time's it worked, mostly not:

#!/bin/bash
VBoxHeadless --startvm "freebsd 11.1" &
echo "waiting"
sleep 40
echo "logging in"
ssh -p 3022 [email protected] <<'ENDSSH'
echo "Build Shoes FreeBSD"
export PATH=/opt/bin:$PATH
export NFS_ALTP=/usr/home/ccoupe/build/
cd Projects/shoes3
pwd
echo $PATH
rm -rf build-target
rake bsd:setup:freebsd
rake
rake package
cp /usr/home/ccoupe/build/pkg/*freebsd*.install pkg
rm -rf /usr/home/ccoupe/build/freebsd
rake clobber
ENDSSH
ls -ld /home/ccoupe/Projects/shoes3/pkg/*freebsd*.install
VBoxManage controlvm "freebsd 11.1" powerof

What's new and interesting is we attempt to fire up a headless VM in the background, wait 40 seconds and then ssh log in running our script. We put our ruby's /opt/bin first in the $PATH. We use the NFS_ALTP/ to speed up writes but that leaves the packaged .install in /usr/home/ccoupe/build/pkg so we have to copy that to shared folder.

Work Flow and Alternatives

The original, Raisins work flow

Assume 3 people, 3 different physical machines, Win7, OSX, and Linux. All developers push their commits for next release. All developers pull any git changes. Each builds Shoes for his machines and they send there Shoes installer to the distribution website or maintainer. Conceptually, this should work if you can maintain discipline since there is no guarantee that each of 3 machines actually did sync to the git source and the branches are correct

My Workflow as documented above

One network shared copy of the sources for all builds (~/Projects for me.) that source is the github source. One person build everything for a Release. It does not preclude pull requests and commits from other developers via git and does not preclude use VMs (assuming the freebsd build script can work)

Git and VM's, Containers and ...

Assume the 'single maintainer' has (Virtual) Machines for all the platforms with the correct ruby and gems and deps. The build scripts start with a git clone <shoes> , and then a git checkout <branch>, default 'master', do the build dance: rake :setup:, rake, rake package. Repeat for all VM's. Seems simple unless you want to fix a Mac bug (say) but if you know how things work, that too can be done, if you have the discipline and know the secrets.

Don't Forget

Your build host/controller needs to send email (logs) about what worked or didn't which means to you need to configure email I use exmi4 smarthost but it can take many days and tests and head scratching to get that working.

You also need to get /etc/hosts working and all kinds of system admin stuff working for your build host and all the VM's or machines.