This benchmark compares Semaphore to GitHub Actions, GitLab CI, Buildkite, and CircleCI using the same repository, pipeline logic, versions, and equivalent machine classes. 

The goal is to measure real execution time and compute cost under identical conditions.

Repository and Workload

Repository: Redmine (Ruby on Rails application). The workload consists of dependency installation and full test execution. Runs were measured after cache warm-up to focus on steady-state execution time. After initializing the cache, 10 consecutive runs were executed per provider. No outliers were removed.

Infrastructure Configuration Setup

All providers used 2 vCPU machines, matching memory as closely as possible:

• Semaphore: f1-standard-2 (2 vCPU, 8 GB RAM)
• GitHub Actions: ubuntu-latest (Linux runner) (2 vCPU, 7 GB RAM)
• GitLab: saas-linux-small-amd64 (2 vCPU, 8 GB RAM)
• CircleCI: Docker medium (2 vCPU, 4 GB RAM)
• Buildkite: LINUX_AMD64_2X4 (2 vCPU, 4 GB RAM)

OS family, Ruby version, dependency installation strategy, and database backend are the same across the board. This benchmark measures single-job execution speed (no parallelism).

Semaphore Pipeline Overview

The Semaphore pipeline defines a single job running on f1-standard-2. It checks out the repository, restores cache, installs dependencies, sets up the database, and executes the full test suite.

version: v1.0
name: Redmine
agent:
 machine:
 type: f1-standard-2
 os_image: ubuntu2404
blocks:
 - name: Tests
 task:
 jobs:
 - name: tests
 commands:
 - checkout
 - cache restore gems-$(checksum Gemfile)-ruby-4.0
 - sudo apt-get update
 - sudo DEBIAN_FRONTEND=noninteractive apt-get install --yes --quiet build-essential pkg-config libpq-dev postgresql-client ghostscript gsfonts locales bzr cvs imagemagick
 - sudo locale-gen en_US.UTF-8
 - |
 cat > policy.xml <<'EOF'
 <policymap>
 <policy domain="coder" rights="read | write" pattern="PDF" />
 </policymap>
 EOF
 - sudo rm -f /etc/ImageMagick-6/policy.xml
 - sudo mv policy.xml /etc/ImageMagick-6/policy.xml
 - sem-version ruby "4.0"
 - sem-service start postgres 14 --db redmine_test
 - |
 cat > config/database.yml <<'EOF'
 test:
 adapter: postgresql
 database: redmine_test
 username: postgres
 password:
 host: 127.0.0.1
 EOF
 - bundle config set path 'vendor/bundle'
 - bundle install --jobs 4 --retry 3
 - cache store gems-$(checksum Gemfile)-ruby-4.0 vendor/bundle
 - export RAILS_ENV=test
 - 'export SCMS=subversion,git,git_utf8,filesystem,bazaar,cvs'
 - 'bundle exec rake ci:about'
 - 'bundle exec rake ci:setup'
 - 'bundle exec rake db:environment:set'
 - bin/rails test
 - LANG=en_US.ISO8859-1 LC_ALL=en_US.ISO8859-1 bin/rails test test/unit/repository_bazaar_test.rb
 - 'bin/rails test:autoload'

Semaphore CI/CD Competitive Benchmark

Avg build time and cost per job across leading CI/CD providers.

Benchmark Results

Cost Calculation Method

Formulas used:

• Semaphore faster by = (ProviderAvg – SemaphoreAvg) / SemaphoreAvg
• Cost per run = AverageDurationMinutes × PricePerMinute
• Cost vs Semaphore = (ProviderCost – SemaphoreCost) / SemaphoreCost

Productivity Impact

A reduction from 9-13 minutes down to 5 minutes changes feedback cycles. For a team running 100 builds per day, saving 4 minutes per build results in 400 minutes saved daily. That equals over 6.5 engineer hours regained per day.

Budget and Engineering Capacity Impact at Scale

Assume your organization consumes 1,000,000 build minutes on Semaphore for this workload.

Based on the benchmark runtime ratios, the same workload would require:

These are not abstract numbers. They represent real waiting time in feedback loops, and a slower CI means:

• Longer pull request cycles
• Slower bug detection
• Slower incident resolution
• More context switching
• Reduced deployment frequency

Now consider your internal engineering cost model:

• H = fully loaded engineering hourly cost

Then the organizational impact of slower CI is:

Additional personnel cost exposure = Extra pipeline hours × 𝐻

Without assuming a specific compensation level, the relationship is linear and direct:

• If H increases, the cost penalty increases proportionally
• If your build volume increases, the cost penalty scales proportionally

CI performance therefore has a second-order budget impact:

• Direct compute cost
• Indirect engineering time cost

The benchmark shows that under identical workload conditions, Semaphore minimizes both simultaneously.

When performance and cost efficiency move in the same direction, CI infrastructure reduces both compute waste and feedback loop waiting times.

TL;DR

Under identical workload and equivalent machine classes, and single-job execution constraints, Semaphore delivered the fastest execution time and the lowest cost per run in this benchmark configuration.

For engineering teams optimizing feedback loops and infrastructure spend, execution time, and cost per run are measurable levers. This benchmark demonstrates both.

Next Step

CI performance directly influences two variables that scale with your organization: infrastructure spend and engineering throughput.

This benchmark demonstrates measurable differences under controlled conditions. The most relevant comparison, however, is against your own repository, build frequency, and test volume. 

Create a project, run your existing pipeline, and measure the execution time and cost against your current setup.

You will have the data you need to quantify the difference where it matters.

FAQs