docker machine workers

Bidding for Builds

running jobs

This post covers the how of using EC2 spot instances for continuous integration (CI), and the why you want to do this.

Really, a whole post on that?

For a CI system to usable, it must fulfill specific needs.

  • Builds must be (largely!) reproducible
  • Providing access control
  • Delivering logs
  • Allowing build generation via multiple means
  • Accurately allocating compute resources
  • Allowing artifact archival
  • Allow arbitrary command execution

If you think about these needs for a bit, a well-developed CI system begins to look a bit like a simplistic execution platform.

Such an execution platform was required for an internal Two Six Labs project. Retrofitting CI was a good way to meet that need, and this post covers the details.

The starting point – Gitlab CI

Gitlab tightly integrates the addressing of all CI system needs. Using it allows us to centralize user permissions across both source control and our “simplistic execution platform”. Gitlab CI also provides the ability to trigger jobs via webhooks, and passing environmental variables to jobs as arguments. When jobs are triggered, environmental variables are saved, checking the “permits jobs to be replicated” box.

Gitlab CI also supports, to varying degrees, a handful of “executors”. I say “executors” as the specific executor we use, Docker Machine, is more a poorly-supported provisioner than an executor.

Docker Machine

docker machine workers

One of Docker Machine’s many features is handling most of the details involved in spinning up and tearing down EC2 Spot instances, such as setting maximum bid price.

Provided the commands you need to run can be executed within a Docker container, Docker Machine can serve as a decent provisioner, providing compute only as you need it.

Making Gitlab work well with Docker Machine

Spot instances are, as you may know, fantastically low-cost. They are low cost, because their availability both before and after provisioning is not guaranteed. Spot instance pricing changes with regional and zone demand, and provisioned spot instances can be terminated after a two minute warning.

Gitlab CI does not address spot instance termination. If a spot instance running a job is terminated, Gitlab CI eventually marks that job as failed. This is problematic because regardless of what you’re using it for, knowing if a task has failed or completed is too useful and too basic a feature to lose. The workaround to this issue we use the script

Handling Terminations is wrapped in a file and passed to the instance as Docker Machine provisions it. configures a cron job to run every thirty seconds, allowing it to cancel all jobs if the instance is marked for termination. There’s a bit more to this script, so I’m going to go through it function-by-function.


The interesting thing here is the docker package. This Docker library is unique in how high quality it is. I doubt there is any other Docker library, in any language, of such high quality.

#!/usr/bin/env python 
import docker 
import requests 
import os 
from sh import wall 
from sh import echo 
from pathlib import Path


This script checks if any jobs are in need of termination and performs the termination if so.

def main(): 
   if to_be_terminated(): 
if __name__ == '__main__': 
    gitlab_api = “” 

Check Termination

This function determines if the instance is to be terminated and if that termination needs to be addressed.
One thing to note is that the IP being checked is present on all instances on AWS and AWS clones.

def to_be_terminated(): 
   if not Path("/clean-exit-attempted").is_file(): 
           resp = requests.get('') 
           return True 
           return False 
       return False

Wall All

This function is mainly for debugging, using wall to broadcast messages to all ttys.

def wall_all(container, msg): 
   container.exec_run(f'sh -c "echo \'{msg}\' | wall"')

Terminate Jobs

This function has several roles. It first acquires the job ID of the Gitlab CI job on the runner, and then cancels that job so it is not marked as having failed. Lastly, it retries the job, allowing the job to complete without user intervention.
It also will run the script / if it exists, which is useful if your jobs are stateful in a way CI doesn’t quite support.

def terminate_jobs(): 
   client = docker.from_env() 
   for container in client.containers.list(filters = {'status': 'running',}): 
           jid = container.exec_run('sh -c "echo ${CI_JOB_ID?"NOJOB"}"')[1].decode('utf-8').strip('\n') 
           pid = container.exec_run('sh -c "echo ${CI_PROJECT_ID?"NOJOB"}"')[1].decode('utf-8').strip('\n') 
           if (pid != "NOJOB") and (jid != "NOJOB"): 
               job_container = container 
               container.exec_run('sh -c "/"') 
           wall_all(job_container,f"Giving on on clean exit and restarting job {jid} of project {pid}.") 

   kill_url = f"{gitlab_api}/projects/{pid}/jobs/{jid}/cancel" 
   retry_url = f"{gitlab_api}/projects/{pid}/jobs/{jid}/retry?scope[]=pending&scope[]=running" 
   auth_header = {'PRIVATE-TOKEN': os.environ.get('GITLAB_TOKEN')} 

   killed = False 
   tries = 20 
   while not killed and tries > 0: 
           tries -= 1 
           resp =, headers = auth_header) 
           #gitlab status code 
           killed = True 
           wall_all(job_container,'Failed to cancel job, retrying.') 

   if killed: 
       wall_all(job_container,"Cancellation successful.") 

   retried = False 
   tries = 20 
   while not retried and tries > 0: 
           tries -= 1 
           resp =, headers = auth_header) 
           #gitlab status code 
           retried = True 
           wall_all(job_container,'Failed to restart job, retrying.') 

   if retried: 
       wall_all(job_container, "Restarted job - That's all folks!!!.")

User Data

This file contains the Docker Machine has AWS execute this script on instances once they are provisioned.

cat << "EOF" > /home/ubuntu/ 



crontab << EOF 
* * * * * python3.6 /home/ubuntu/ 


A problematic bug I had to workaround was Docker Machine abandoning provisioned instances, seemingly when it is rate-limited by AWS. The percentage of machines abandoned increase as the number of machines provisioned at once does. Fortunately, when this bug occurs, the instance in question is never tagged. As we only use Docker Machine to provision instances for CI, this allowed us to find and terminate instances meeting the criterion. The script we use is

Spot Sniper

This script terminates abandoned spot instances. While this script is straightforward, this breakdown serves a vital purpose: preventing runaway AWS bills. Abandoned instances are not counted against the configured resource limits, allowing them to accumulate.


Some standard imports.

#!/usr/bin/env python3 
import boto3 
from pprint import pprint 
import toml 
import os 
import syslog


This script looks for and terminates abandoned spot instances.
There is a bug somewhere between Docker Machine and Gitlab Runner that causes instances to be abandoned.
Instances that are abandoned by this error are identifiable by the lack of a name tag while having the docker-machine security group.
This script also terminates instances abandoned by as the instances that script provisions are configured to look abandoned.

def main(): 
   regions = ['us-east-1', 'us-east-2'] 
   for region in regions: 
       ec2 = boto3.resource('ec2', region_name = region, aws_access_key_id=os.environ['AWS_KEY'],aws_secret_access_key=os.environ['AWS_SECRET']) 

       all_ci_instances = set(ec2.instances.filter(Filters = [ 
           {'Name': '', 'Values': ['docker-machine']}, 
           {'Name': 'instance-state-name', 'Values': ['running']}, 

       all_functional_ci_instances = set(ec2.instances.filter(Filters = [ 
           {'Name': '', 'Values': ['docker-machine']}, 
           {'Name': 'instance-state-name', 'Values': ['running']}, 
           {'Name': 'tag-key', 'Values': ['Name']}, 

       # This right here is how the bug somewhere between Docker Machine and Gitlab Runner expresses itself. 
       horde = all_ci_instances - all_functional_ci_instances 

       if len(horde) == 0: 
           syslog.syslog(" - No abandoned spot instances.") 

       for zombie in horde: 

if __name__ == '__main__': 

Kill With Fire

This function terminates abandoned instances. It’s main purpose is to allow zombie instances to be killed with fire, instead of simply being terminated.

def kill_with_fire(zombie): 
   syslog.syslog(f" - Terminating zombie spot instance {}.") 


This cron runs every 3 minutes. It needs to be tuned to minimize waste without causing excessive rate-limiting.

*/3 * * * * python3.6 /root/gitlab-ci-ami/

Optimizing Instances Used

As spot instance prices vary across regions, zones in those regions, instance types and time; costs can be minimized by checking across those axes. We wrote the script to do this.

Min Bid Price

While was initially intended to be a script run by cron to select the cheapest combination of region, zone and instance type; we also needed to determine instance availability. We found that requesting a few spot instances, waiting a few seconds, and checking if those instances were available was an effective way to do this.
The following breakdown details the what and why of each component of


There are a couple of interesting imports in this script:

#!/usr/bin/env python3 
from sh import sed 
from sh import systemctl 
from functools import total_ordering

The sh package is a package that wraps binaries on $PATH, allowing them to be used in as pythonic a way as is possible without having to use a dedicated library.

total_ordering is an annotation that, provided an equivalence and a comparison operator are defined, will generate the not-explicitly-defined equivalence and comparison operators.

Instance Profile Class

instance_profile obtains, stores and simplifies the sorting of pricing info from AWS.

class instance_profile: 
    def __init__(self, instance, region, zone): 
        self.instance = instance 
        self.region = region = zone 
        self.price = None 
    def determine_price(self, client): 
            resp = client.describe_spot_price_history(InstanceTypes=[self.instance],MaxResults=1,ProductDescriptions=['Linux/UNIX (Amazon VPC)'],AvailabilityZone= self.region + 
            self.price = float(resp['SpotPriceHistory'][0]['SpotPrice']) 
            return True 
            return False 
    def __eq__(self, other): 
        if self.price == other.price: 
            return True 
            return False 
    def __gt__(self, other): 
        if self.price > other.price: 
            return True 
            return False 
    def __str__(self): 
        if self.price is None: 
            return f"No price for {self.instance} {self.region}{}" 
            return f"{self.instance} at {self.region}{} costing {self.price} at {}"


There’s a fair amount going on here, so a few interruptions for the following function:

def main():

This code block specifies instances, regions and zones to be considered for use:

 instances = ['m5.xlarge', 'm4.xlarge', 'c4.2xlarge', 'c5.2xlarge'] 
regions = ['us-east-1','us-east-2'] 
zones = ['a', 'b', 'c', 'd', 'e', 'f']

Here I specify the AMI to use in each region, as AMIs are not available across regions:

region_amis = {'us-east-1': 'ami-5bc0cf24', 'us-east-2': 'ami-3de9d358'}

The following code block specifies the criteria an instance_profile must meet to be usable. It specifies that an instance_profile must enable the provisioning of 3 instances via 3 separate spot instance requests in under 10 seconds when max bid price is 0.08 cents per hour:

 max_bid = 0.08 
test_instances = 3 
wait_retries = 2

The following snippet assures the system configuration can be updated, before firing off AWS requests to find a suitable instance_profile and updating system configuration.

    if safe_to_update_config(): 
        inst_confs = get_price_list(instances,regions,zones) 
        for conf in inst_confs: 
            if spot_test(conf.region,, region_amis[conf.region], conf.instance, 
                         test_instances, max_bid, wait_retries): 
                if safe_to_update_config(): 
                    syslog.syslog(f" - Min Price: {conf}") 
                    syslog.syslog(f" - Cannot update config as jobs are running.") 
                syslog.syslog(f" - {conf} failed provisioning check.") 
        syslog.syslog(f" - Cannot update config as jobs are running.") 

if __name__ == '__main__': 

Update Config

The following block of code uses sed, via sh, to edit Gitlab Runners /etc/gitlab-runner/config.toml configuration file.
If you anticipate needing to use multiple instance types, use the toml package instead of sh and sed here.

def update_config(next_inst,region_amis): 
   sed("-i", f"s/amazonec2-zone=[a-f]/amazonec2-zone={}/g", "/etc/gitlab-runner/config.toml") 
   sed("-i", f"s/amazonec2-ami=ami-[a-z0-9]*/amazonec2-ami={region_amis[next_inst.region]}/g", "/etc/gitlab-runner/config.toml") 
   sed("-i", f"s/amazonec2-instance-type=[a-z0-9]*.[a-z0-9]*/amazonec2-instance-type={next_inst.instance}/g", "/etc/gitlab-runner/config.toml") 
   systemctl("restart", "gitlab-runner") 
   syslog.syslog(f" - Moved CI to {next_inst}")

Get Price List

The following function creates a boto3 client for each region being considered, and uses those clients to create a price-sorted list of instance_profile objects.

def get_price_list(instances, regions, zones): 
   price_list = [] 
   for region in regions: 
       for instance_type in instances: 
           for zone in zones: 
               price = instance_profile(instance_type, region, zone) 
               if price.determine_price(client): 
   return price_list

Safe to Configure

This function determines if it is safe to update system configuration. It determines this by assuring that both:

  • No CI jobs are running and,
  • No non-zombie Docker Machine instances are running
def safe_to_update_config(): 
   auth_header = {'PRIVATE-TOKEN': os.environ['GITLAB_TOKEN']} 
       resp = requests.get('', headers = auth_header) 
       syslog.syslog(' - Cannot get runner status from Something up?') 
       return False 
   if len(resp.json()) != 0: 
       return False 
       instances_running = "/root/.docker/machine/machines" 
       if os.listdir(instances_running): 
           return False 
       return True

Spot Test

This function tests instance_profile objects to determine their usability, by exploring if instances can be provisioned quickly enough for the instance_profile in question.
If you’re wondering why instance_profile is absent , it processes the components of instance_profile.

def spot_test(region, availability_zone, ami, instance_type, instances, max_bid, wait_retries): 
   client = boto3.client('ec2', region_name = region, aws_access_key_id=os.environ['AWS_KEY'],aws_secret_access_key=os.environ['AWS_SECRET']) 
   req_ids = spot_up(client, instances, max_bid, ami, availability_zone, 
                    region, instance_type) 
   usable_config = check_type_in_az(client, wait_retries, req_ids) 
   spot_stop(client, req_ids) 
   spot_down(client, req_ids) 
   if usable_config: 
       syslog.syslog(f" - {region}{availability_zone} {instance_type} wins as it spins up {instances} instances in {wait_retries*5} seconds at max_bid {max_bid}.") 
       return True 
       syslog.syslog(f" - {region}{availability_zone} {instance_type} loses as it fails to spins up {instances} instances in {wait_retries*5} seconds at max_bid {max_bid}.") 
       return False

Spot Up

This function requests the specified number of spot instances and returns a list of the IDs of those requests.

def spot_up(client, instances, max_bid, ami, availability_zone, region, instance_type): 
   responses = [] 
   for i in range(instances): 
               'ImageId': ami, 
               'InstanceType': instance_type, 
               'Placement': { 
                   'AvailabilityZone': region + availability_zone, 
           SpotPrice= str(max_bid), 
   return [x["SpotInstanceRequests"][0]["SpotInstanceRequestId"] for x in responses]

Spot Stop

This function cancels outstanding spot instance requests. It can fail when the system is being rate limited by AWS. Requests not cancelled will be fulfilled and cleaned up by This failure is permitted as it results in stderr being emailed via cron, letting us know to not slam the system with jobs for a couple minutes.

def spot_stop(client, req_ids): 
   cancellations = (client.cancel_spot_instance_requests(SpotInstanceRequestIds=[x])["CancelledSpotInstanceRequests"][0]["State"] == "cancelled" for x in req_ids) 
   while False in cancellations: 
       print(f" - Failed to cancel all spot requests, retrying") 
       cancellations = (client.cancel_spot_instance_requests(SpotInstanceRequestIds=[x])["CancelledSpotInstanceRequests"][0]["State"] == "cancelled" for x in req_ids)

Spot Down

This function terminates provisioned spot instances. It can fail when the system is being rate limited by AWS, in which case will clean up the provisioned instances during its next pass.

def spot_down(client, req_ids): 
   instances = [client.describe_spot_instance_requests(SpotInstanceRequestIds = [x]) for x in req_ids] 
   terminate_ids = [] 
   for x in instances: 
       except KeyError: 
   if len(terminate_ids) > 0: 
       client.terminate_instances(InstanceIds = terminate_ids)

Check Instance Type in AZ

This function checks the status of spot instance requests made every five seconds until either the specified number of retries are made, all requests are fulfilled, or one request will not be fulfilled.

def check_type_in_az(client, wait_retries, req_ids): 
   statuses = spot_req_status(client, req_ids) 
   while wait_retries > 0 and req_status_check(statuses) is None: 
       wait_retries -= 1 
       statuses = spot_req_status(client, req_ids) 
   if req_status_check(statuses) is None: 
       return False 
       return req_status_check(statuses)

Spot Request Status

This function returns a list of the statuses of spot requests made.

def spot_req_status(client, req_ids): 
   return [client.describe_spot_instance_requests(SpotInstanceRequestIds=[x])["SpotInstanceRequests"][0]["Status"] for x in req_ids]

Request Status Check

This function reduces a list of spot requests to a boolean once their success can be determined, returning None if their success cannot be determined.

def req_status_check(statuses): 
   for x in statuses: 
       if (x["Code"] == "pending-evaluation") or (x["Code"] == "pending-fulfillment"): 
           return None 
       elif x["Code"] != "fulfilled": 
           syslog.syslog(f"Fail req_status: {x['Code']}") 
           return False 
   return True


This crontab runs every 10 minutes.
The period of this cron needs to be tuned for your use case.
If it is too wide, it is less likely that system configuration will be updated when users are active.
If it is too narrow, the cost of determining instance availability will increase as instances are billed by the minute for their first minute.

*/10 * * * * python3.6 /root/gitlab-ci-ami/

Config, config, config…


As bandwidth costs on AWS can add up and Spot Instance usage is billed by the second (after the first minute), we pre-load a handful of images we use often into the AMI used by Docker Machine to provision instances.

Gitlab Runner Config

This is the config.toml we use for Gitlab Runner. Key points to note are the volume mounts it configures, and the max builds limitation. The volume mounts are configured to allow CI jobs to use volume mounts of their own. MaxBuilds being set to 1 prevents port conflicts from occurring and ensures that all jobs are run in a clean environment.

 concurrent = 80 
check_interval = 0 

  name = "alpine" 
  limit = 80 
  url = "“ 
  token = “XXXXX” 
  executor = "docker+machine" 
  output_limit = 16384 


tls_verify = true image = “BUILD_IMAGE_TAG” privileged = true disable_cache = true shm_size = 0 volumes = [“/var/run/docker.sock:/var/run/docker.sock”,”/builds:/builds”,”/cache:/cache”]



MachineDriver = “amazonec2” MaxBuilds = 1 MachineName = “gitlab-docker-machine-%s” OffPeakIdleCount = 0 OffPeakIdleTime = 0 IdleCount = 0 IdleTime = 0 MachineOptions = [ “amazonec2-request-spot-instance=true”, “amazonec2-spot-price=0.080”, “amazonec2-access-key=XXXXX”, “amazonec2-secret-key=XXXXX”, “amazonec2-ssh-user=ubuntu”, “amazonec2-region=us-east-2”, “amazonec2-instance-type=m4.xlarge”, “amazonec2-ami=ami-XXXXX”, “amazonec2-root-size=50”, “amazonec2-zone=a”, “amazonec2-userdata=/etc/gitlab-runner/”, “amazonec2-use-ebs-optimized-instance=true”, ]

Docker Daemon Config

The following is the contents of /etc/docker/daemon.json on all CI machines. It configures the Docker daemon to use Google’s mirror of Dockerhub when Dockerhub is down or having reliability issues. It also limits the size of Docker logs (a source of many filled disks).

   "registry-mirrors": [""], 
   "log-driver": "json-file", 
   "log-opts": {"max-size": "10m", "max-file": "3"} 


The following table contains some metrics on the cost of our configuration over the past six months:

Instance TypeInstance CountTotal Job HoursCostCost Relative to On Demand, Always On, 4Turnaround Time Relative to On Demand, Always On, 4
On demand, Always On41938.743423.84100%100%
On demand, As Needed80 Max1938.74378.8711.07%0.05%
Spot80 Max1938.7480.692.36%0.05%

The following histogram shows the durations of jobs ran since we started using CI:

job durations

The following plot shows the maximum number of jobs we’ve had running at once, over time:

max jobs

One thing these metrics do not capture is the impact of checking the availability of instance_profile has on job durations.

Before running this check, job startup times would often go as high as 6 minutes and would occasionally end up stuck in a “Pending” state due to a lack of available compute.

Job startup times now rarely exceed 1 minute and they do not get stuck “Pending”.

Changes since this was started

Docker Machine Gitlab MR

This MR, which was added in Gitlab 11.1, raised the number of CI jobs we could have running at once to at least 80. Given the performance claims and the number of jobs we could run at once before being rate limited by AWS before this MR was merged, I would guess we could run somewhere between 200 and 250 jobs at once before being rate limited by AWS.


While working on this project, Gitlab announced their Meltano project. While the goal of Meltano might not be enabling the use of CI to process versioned data, that will almost certainly be a component. As the purpose of this CI configuration was to allowing us to use CI to process versioned data, I expect that the performance and capabilities of this CI configuration will increase as bugs related to Gitlab Runner and Docker Machine are addressed for Meltano.

Spot Pricing Change

AWS recently changed how they calculate the price of spot instances, smoothing price fluctuations.
While this change reduces the benefit of the approach of finding the optimal instance_profile to use to run instances, the approach of finding the optimal instance_profile still allows us to use the cheapest instances meeting our compute, startup-time and compute capacity requirements.


That’s left us with more than we need to get our jobs ran in a timely manner.

This paragraph was initially going to be:

There are a few yet-to-be-a-problem cases these scripts have not addressed, such as ignoring sold out instance-region-zone combinations and automatically restarting jobs that are cancelled due to instance price increases and automating the generation (and use of) new pre-loaded AMIs periodically.

but, as our needs grew, all those problems had to be addressed.

See for a more copy-paste friendly version of the scripts on this page.