With embedded OSes you usually use the cross-compilation toolchain to compile the software for your target system (Galileo in this case) on your "big brother" machine, like PC or laptop.
For Galileo you can, as you already know, also create a Linux image, which would contain the tools right away, or use the cross-compilation. For the latter you'd need to generate the toolchain first - see the BSP Build Guide for instructions.