By Honza Dvorsky, iOS Developer
SwiftKey Tech Blog – posts by developers, for developers. Find more here: swiftkey.com/tech-blog
This article was originally published on my personal blog honzadvorsky.com.
On the iOS team here at SwiftKey we’re always pushing our code forward, and to ensure that it also stays stable and performant, we use Xcode Server as our continuous integration server. One feature missing from it, Pull Request testing, I filled by creating an app called Buildasaur in my spare time which we now use at SwiftKey to automatically test Pull Requests before merging them. Writing an app that talks to Xcode Server’s API gave me a chance to explore the ins and outs of Xcode Server, which is what I’m writing about today.
Back in 2013, when Xcode 5 was released, Apple referred to the system as “Xcode Bots”, but technically, it’s called Xcode Server. Each job/plan on the server is called a “Bot” and each run of a job is called an “Integration”.
Buildasaur supports the newer version of Xcode Server, released in 2014 with Xcode 6, so everything described in this article relates to a setup with Xcode 6.x, OS X Server 4.x.
This article assumes you have such an environment setup, if you want to try some of the hacks yourself. It’s fine if you don’t, I’ll be writing an article on how to set up Xcode Server from scratch in the next couple of weeks.
So, let’s get started!
First we’ll look at where Xcode Server prints its logs, because that should be the go-to place when something breaks. All the logs can be found at /Library/Developer/XcodeServer/Logs. You’ll find multiple log and pid files there.
This is the build daemon’s log, so when a Bot is building your code, this is where all the output goes. It should match what you see when you just run xcodebuild on your project from Terminal. This is a very useful log, because sometimes Xcode Server isn’t verbose enough about a build failure, at those times, head right for xcsbuildd.log!
This is the log of the main process which controls Xcode Server under the hood.
CouchDB’s log, more on the role of CouchDB in Xcode Server later.
API Server’s log, one of the most important ones. Every request, processing of it and response are logged here. This is extremely useful when trying to figure out how the API talks to Xcode or to your app. But more on xcsd later.
Contains xcsdeviced’s logs, the daemon which handles the communication with connected iOS devices. If you can’t figure out why your build isn’t being tested on your device, this is the place to go. You’ll find messages like “The device is passcode protected.” if you forgot to unlock your iPhone, etc.
Redis’s log, as with CouchDB, more on its role later.
The Logs folder also contains several .pid files, which just tell you the process ids of its running daemons. This can be useful if you need to force quit one of them.
You might have noticed that all the filenames have a prefix of xcs – I can only guess that this is an abbreviation of Xcode Server. This prefix is present in the majority of files and folders that Xcode Server keeps around.
Xcode Server runs under multiple users on your system (like _xcsbuildd, _xcsd) and it persists all files in the /Library/Developer/XcodeServer folder (which is where the logs are as well). Even if you remove both Xcode and OS X Server from your system and then reinstall them, all preferences and data will get picked up, because this location remained untouched. Now, let’s look at what’s in there for us. Except for the Logs folder, you’ll need to go into superuser mode by using $ sudo -s in Terminal before you start digging into the folders below.
Contains the server’s certificates, used for signing various resources (not code, however). I’m not yet clear on the exact purpose of what’s in there.
Contains just ota.mobileconfig, which is a signed property list allowing OTA (over the air) installations of your app, when you archive it with Xcode Server. This is one of my favorite features of Xcode Server, because it lets non-technical members of your team install nightly builds in the morning without ever touching Xcode.
As the name suggests, this is a symlink to your currently used Xcode in OS X Server. The fact that a different Xcode can be plugged in easily makes Xcode Server very flexible when it comes to the ever-changing versions of Xcode.
Contains the persisted data from CouchDB, Xcode Server’s database and a file called couch.uri, which contains the address and port of the running CouchDB instance.
- HostedRepositories, HostedRepositoriesHTTPCGIScriptSymlink
Might be useful when you host your git repo on Xcode Server. However, I have never tried that, so if anyone has experience with hosting code on Xcode Server, let me know.
Now we’re getting to the good stuff. This folder contains everything about the integration of each Bot. This is described in detail later.
This folder contains the assets of the currently running integration (useful for observing the build log of a currently running integration) and a Caches folder, which contains the latest assets for each Bot, so that if you start another integration on an existing Bot, it already has its DerivedData and source folders cached.
This folder contains Xcode Server’s keychains, which contain SSH keys and other sensitive information. If you sign your team into Xcode Server, this is where it downloads its certificates and keys.
Were described in detail at the beginning of this article.
Folder containing all provisioning profiles downloaded by the server from the developer portal. Also the place to put any provisioning profiles that you want the server to use during code signing (more on how to set this up in a future article).
Folder containing shared secrets used by Xcode Server to unlock the keychains in Keychains and to securely talk to the developer portal.
Both IntegrationAssets and Integrations folders contain something I called “integration assets”. But what are those exactly?
Integrations contains cached and running integrations, so it needs to keep the files necessary to build your project. The nice thing about Xcode Server is that it keeps DerivedData and Source files per Bot. So if one of your Bots has a broken ModuleCache (classic) in its DerivedData, it won’t affect other Bots. The file structure of Buildasaur’s Bot’s folder in Caches looks like this
| |– Build
| |– Logs
| |– ModuleCache
| |– TestResults
| |– info.plist
| |– Buildasaur
The Source folder contains the checked out repository and DerivedData we’re all too familiar with – this folder contains the intermediate files created by the compiler to speed up future compilations and also the final built products, together with logs.
In contrast to Integrations, which contained still active integrations and caches, IntegrationAssets only contains build results such as archives, test results and logs. This is exactly what you get when you hit “Download All Logs” in the Bot’s detail screen in Xcode’s Report navigator.
The structure of that folder looks like this:
| |– Archive.xcarchive.zip
| |– Session-2015-05-04_00:24:01-MKjX3a.log
| |– build.log
| |– buildService.log
| |– sourceControl.log
| |– xcodebuild_result.bundle.zip
| — 2
| |– Archive.xcarchive.zip
| |– Session-2015-05-04_13:37:55-YQAc8p.log
| |– build.log
| |– buildService.log
| |– sourceControl.log
| |– xcodebuild_result.bundle.zip
Each integration’s assets are in a folder matching its number and contains
The built Xcode archive, if you ticked the archive action as part of your Bot setup.
- Session-[timestamp]-[random string].log
Containing the log of a test run. This one is very important when hunting down a test failure.
Has the complete build log, the output of running xcodebuild on your project.
Contains information about the high level actions performed as part of your integration, such as building, testing and input parameters of those actions.
Will prove helpful if you are having problems with git authentication, such as when you have invalid SSH keys. It has the whole output of all git commands performed as part of the checkout step.
Contains results of the Bot run, mostly in binary plists, which, in my best guess, are used by Xcode to present you with the pretty printed, interactive result pane.
The unzipped structure looks like this:
| |– build.xcactivitylog
| |– action.xcactivitylog
| |– action_TestSummaries.plist
| |– build.xcactivitylog
Now that you know where Xcode Server persists its data, you might be curious where its source code lives. Well, luckily, the engineers in Cupertino are indeed using the same open source tools as we do, like Node.js, Express, Redis and CouchDB, among others. And all of that lives inside of Xcode’s bundle, specifically in /Applications/Xcode.app/Contents/Developer/usr/share/xcs
This is a goldmine of the source code of all the important parts of Xcode Server, in addition to /Applications/Xcode.app/Contents/Developer/usr/bin, where the closed source binaries live, e.g. xcscontrol, xcsbuildd, xcsbridge, xcssecurity and others.
This is a 2-level structure of the xcs directory:
│ ├── bin
│ ├── etc
│ ├── lib
│ ├── sbin
│ ├── share
│ └── var
│ ├── ArchiveXCSv1ServiceData.rb
│ ├── ExportXCSv1JSONData.rb
│ └── MigrateXCSv1HostedRepositories.rb
│ └── bin
│ └── bin
│ └── com.apple.xcs.conf
│ ├── _design
│ └── config.ini
│ ├── app.js
│ ├── bootstrap_xcs.js
│ ├── classes
│ ├── com.apple.xcsd.plist
│ ├── config
│ ├── constants.js
│ ├── diagnostics.md
│ ├── error_handler.js
│ ├── node_modules
│ ├── package.json
│ ├── public
│ ├── routes
│ ├── socket.js
│ ├── util
│ └── views
│ └── redis.conf
and I’d like to point out just a couple of interesting parts.
- CouchDB and Redis
Contain the binaries of both databases, so that Xcode Server doesn’t have to pull any dependencies when launched.
Contains scripts to migrate v1 to v2 (currently Xcode Server is on v2 and might be upgraded again to v3 this year).
Contains a binary of an old version of Node.js, specifically v0.10.26 in Xcode 6.3. But hey, who knew that Xcode contained Node.js?
Is an Apache proxy config file for the Server.
Contains the list of daemons and where to find their .pid files.
- xcscouch and xcsredis
Contain configuration files of Xcode Server’s CouchDB and Redis instances, on which ports to run etc.
- and finally xcsd
Contains the full source to xcsd’s Node.js Express app. Most interesting is the routes/routes.js file, which describes all the API endpoints that xcsd responds to, and that’s how you can write an app talking to those APIs, like I did with Buildasaur.
I have discovered a couple (possibly not all) of important subsystems of Xcode Server, which I’ll talk about now. If you know about more of them, let me know and I’ll gladly amend the article and learn more in the process.
All of Xcode Server’s daemons’ users are members of the group _xcs, which is useful when managing permissions. If you don’t care whether xcsd (the API) or xcsbuildd (the worker/builder) need to access your resource, just allow the whole group _xcs.
The API server is called xcsd and runs under the user _xcsd. The actual process is a Node.js Express app and it handles talking to all the Xcodes on your local network, in addition to any browser in which you go to your [servers-address]/xcode, to see Xcode Server’s status page. It also handles talking to its caching database, Redis and its persistence and map-reduce database, CouchDB.
- Log: /Library/Developer/XcodeServer/Logs/xcsd.log
- Process Id: /Library/Developer/XcodeServer/Logs/xcsd.pid
- Source: /Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcsd
This is the worker process, the one that actually takes the scheduled Bots from the database and runs xcodebuild on your source code. It first needs to checkout the code, based on your Bot’s blueprint. It also runs pre and post-build scripts, which can be used for running CocoaPods, version increment or any other scripts, like uploading your archives to Dropbox and emailing of build results to your team members.
- Log: /Library/Developer/XcodeServer/Logs/xcsbuildd.log
- Process Id: (seems to run on demand)
- Binary: /Applications/Xcode.app/Contents/Developer/usr/bin/xcsbuildd
Its man page says it’s a “utility for managing Xcode Server”. Basically it’s the manager of the whole Xcode Server system. It starts and stops the databases and all the Xcode Server daemons. When you turn on Xcode Server in OS X Server, that giant green button translates to $ xcrun xcscontrol –start.
- Log: /Library/Developer/XcodeServer/Logs/xcscontrol.log
- Process Id: /Library/Developer/XcodeServer/Logs/xcscontrol.pid
- Binary: /Applications/Xcode.app/Contents/Developer/usr/bin/xcscontrol
Handles connected devices and makes sure they are ready for testing. Presumably takes care of waking them up for running tests and other responsibilities. I haven’t found many more details about it.
Process Id: /Library/Developer/XcodeServer/Logs/xcsdeviced.pid
Xcode Server uses CouchDB as its main persistence and a fast lookup of items through map-reduce queries. You can actually browse the data in your web browser by going to http://localhost:10355/_utils/.
- Log: /Library/Developer/XcodeServer/Logs/xcscouch.log
- Port: /Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcscouch/config.ini and grep for “port =”, usually 10355
- Binary: /Applications/Xcode.app/Contents/Developer/usr/share/xcs/CouchDB/bin/couchdb
Xcode Server seems to use Redis as an in-memory cache only (its redis.conf doesn’t specify any save rules, so we know the data has to be inserted at runtime). Xcode engineers probably have good reasons to use Redis in addition to CouchDB, my guess would be its simplicity of the single level key-value storage, performance and great features for inter-process communication (like blocking pop on list for worker queues etc). Again, you can browse the data by running $ redis-cli -p 10356 in Terminal.
- Log: /Library/Developer/XcodeServer/Logs/xcsredis.log
- Port: /Applications/Xcode.app/Contents/Developer/usr/share/xcs/xcsredis/redis.conf and grep for “port “, usually 10356
- Binary: /Applications/Xcode.app/Contents/Developer/usr/share/xcs/Redis/bin/redis-server
When you hack too much and Xcode Server goes crazy, you need a way to reset all of its state and start over. When I was originally developing Buildasaur, I would reset Xcode Server several times a day. There is one command which stops all the running daemons of Xcode Server and deletes all the contents of /Library/Developer/XcodeServer. Be careful, all your bots and integration data will be deleted as well. During debugging it’s an indispensable tool, but be careful in production to not accidentally delete all your Bots and Integration assets.
Ok, since you now read the small print and I warned you, the command is
sudo xcrun xcscontrol –reset.
I don’t have many, just one big one. Xcode Server cannot run multiple integrations at the same time. From how much I’ve seen in its internals, there is nothing inherently preventing multiple instances of xcsbuildd running at the same time on two different devices. Especially since it uses Redis, a database perfect for managing a worker queue with multiple instances. I’d love to get my hands on the source code of xcsbuildd and find out what the problem is or if there is a way to make it work. That’s my one and only feature request!
Why would I want to know all this?
I don’t know, to be honest. I like finding out how things work under the hood. However, this might still prove useful to you if you use Xcode Server in your team and something goes south. Or if you’re like me and want to build more tools on top of its “hidden” API. However, I really appreciate that Apple has done the heavy lifting for us. Xcode Server provides an amazing, always-compatible-with-Xcode way to have a very simple CI server, for free.
That is why I wanted to build on top of Xcode Server and I’m always looking for a way to put a bit of energy into making our workflow even more efficient, without buying into a completely new 3rd party solution. Those usually take ages to support the latest Xcode and Xcode’s features. We have a simple, easy to use CI system handed to us by Apple, so you should have a damn good reason not to use it.
So whether you’re thinking of using Xcode Server’s now-unhidden APIs or you just wanted to know a bit more about the tool you depend on, I hope learned something new. Now, let’s perform an experiment – if you read all the way till the end, tweet at me with the word “gelato” used in a sentence, so that I know you got through the whole thing 🙂
Ping me on Twitter with your success/failure stories of using Xcode Server!