Hey there! Building your own CI runner is a fantastic way to demystify how tools like GitLab Runner and CircleCI work under the hood. Let’s break down the core pieces you’ll need to tackle, with a focus on Go since that’s your language of choice.

Core Components to Implement #

1. Runner ↔ CI Server Communication #

First, your runner needs to talk to a central CI server (either one you build from scratch or integrate with an existing tool). The two most common patterns here are:

• Polling: The runner regularly checks the server for pending jobs—super simple to implement with Go’s net/http package.
• Push/Streaming: The server sends jobs to the runner in real-time (use WebSockets or gRPC for this; Go has rock-solid libraries for both).

Don’t forget authentication: use API keys or OAuth tokens to verify the runner’s identity. Here’s a quick example of adding an auth header to an HTTP request in Go:

client := &http.Client{}
req, _ := http.NewRequest("GET", "https://your-ci-server.com/jobs/pending", nil)
req.Header.Add("Authorization", "Bearer YOUR_RUNNER_TOKEN")
resp, _ := client.Do(req)

2. Execution Environment Isolation #

This is the most crucial (and trickiest) feature—you need to isolate each job to prevent cross-contamination between builds. Docker containers are the standard approach here, and Go has excellent support for the Docker API via the official docker/client package.

Key tasks to handle:

• Pulling base images (e.g., Ubuntu, Node.js, Golang)
• Creating containers with resource limits (CPU/memory) to prevent resource hogging
• Mounting workspace directories to clone code or share build artifacts
• Cleaning up containers immediately after jobs finish (no orphaned containers allowed!)

Here’s a minimal snippet to run a command in a Docker container:

package main

import (
    "context"
    "fmt"
    "github.com/docker/docker/api/types"
    "github.com/docker/docker/api/types/container"
    "github.com/docker/docker/client"
    "io"
    "os"
)

func main() {
    ctx := context.Background()
    cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
    if err != nil {
        panic(err)
    }
    defer cli.Close()

    // Pull Ubuntu image
    pullReader, err := cli.ImagePull(ctx, "ubuntu:latest", types.ImagePullOptions{})
    if err != nil {
        panic(err)
    }
    io.Copy(os.Stdout, pullReader)

    // Create container with a simple echo command
    createResp, err := cli.ContainerCreate(ctx,
        &container.Config{Image: "ubuntu:latest", Cmd: []string{"echo", "Hello from custom CI!"}},
        nil, nil, nil, "",
    )
    if err != nil {
        panic(err)
    }

    // Start container
    if err := cli.ContainerStart(ctx, createResp.ID, types.ContainerStartOptions{}); err != nil {
        panic(err)
    }

    // Wait for job to finish
    waitStatus, err := cli.ContainerWait(ctx, createResp.ID, container.WaitConditionNotRunning)
    if err != nil {
        panic(err)
    }
    fmt.Printf("Job exit code: %d\n", waitStatus.StatusCode)

    // Fetch and print logs
    logsReader, err := cli.ContainerLogs(ctx, createResp.ID, types.ContainerLogsOptions{ShowStdout: true})
    if err != nil {
        panic(err)
    }
    io.Copy(os.Stdout, logsReader)

    // Clean up container
    if err := cli.ContainerRemove(ctx, createResp.ID, types.ContainerRemoveOptions{}); err != nil {
        panic(err)
    }
}

3. Job Lifecycle Management #

Every CI job follows a lifecycle: clone repository → run scripts → upload artifacts → clean up. Use Go’s context package to manage job cancellation (e.g., timeouts or user-initiated cancels) and goroutines to handle concurrent tasks smoothly.

Key implementations:

• Git Repository Cloning: Use Go’s os/exec package to run git clone, or use github.com/go-git/go-git/v5 for more programmatic control.
• Script Execution: Run multiple steps (e.g., go build, npm test) in sequence, capturing real-time output and exit codes.
• Artifact Upload: Send build outputs (binaries, test reports) back to the CI server via HTTP POST requests.

4. Configuration Parsing #

Your runner should support a YAML config file (like .custom-ci.yml) where users define jobs, steps, and settings. Use Go’s gopkg.in/yaml.v3 package to parse the config into Go structs.

Example config structure:

jobs:
  build:
    image: golang:1.21
    steps:
      - checkout
      - run: go build -o myapp
      - upload:
          source: myapp
          destination: artifacts/myapp

Corresponding Go structs:

type Config struct {
    Jobs map[string]Job `yaml:"jobs"`
}

type Job struct {
    Image string `yaml:"image"`
    Steps []Step `yaml:"steps"`
}

type Step struct {
    Checkout bool   `yaml:"checkout,omitempty"`
    Run      string `yaml:"run,omitempty"`
    Upload   Upload `yaml:"upload,omitempty"`
}

type Upload struct {
    Source      string `yaml:"source"`
    Destination string `yaml:"destination"`
}

5. Error Handling & Retries #

CI jobs fail all the time—network blips, broken scripts, etc. Implement retry logic for flaky steps (like cloning a repo) using simple loops or a lightweight retry library. Also, capture detailed error messages and send them back to the CI server so users can debug easily.

Example retry function for Git clone:

func cloneRepo(ctx context.Context, repoURL string, retries int) error {
    for i := 0; i < retries; i++ {
        cmd := exec.CommandContext(ctx, "git", "clone", repoURL)
        if err := cmd.Run(); err == nil {
            return nil
        }
        time.Sleep(time.Second * time.Duration(i+1)) // Exponential backoff
    }
    return fmt.Errorf("failed to clone repo after %d retries", retries)
}

Next Steps for Advanced Features #

Once you have the basics working, you can level up with:

• Distributed Runners: Run multiple runners across different machines, with the CI server assigning jobs based on resource availability.
• Resource Monitoring: Track CPU/memory usage of jobs using Docker API metrics or Go’s runtime package.
• Secrets Management: Inject encrypted secrets into job environments without exposing them in logs.
• Webhook Notifications: Send alerts to Slack/email when jobs pass or fail.

内容的提问来源于stack exchange，提问作者nitrocode