Been dealing with this more often lately. Tests pass on my machine, I push, and CI blows up. Usually it’s one of these:

  • Different Node/Python/whatever version
  • Missing env vars that exist in my .env but not in CI secrets
  • File system case sensitivity (macOS vs Linux)
  • Some flaky test that depends on timing

My current debugging flow is pretty basic: check the logs, compare versions, run the exact same Docker image locally if I can. But it still eats 20-30 minutes each time before I figure out the actual problem.

Anyone have a more systematic approach? Like a quick checklist you run through before you even look at the logs?

Also curious — do you replicate your CI environment locally with something like act (for GitHub Actions) or just trust the remote runner?

  • hallettj@leminal.spaceEnglish
    1·
    8 days ago

    I try to capture every detail of the build and test environments in Nix devshells. And where I can I try to encapsulate as much as possible in Nix checks and packages which run in build sandboxes - both locally and on the server. Build sandboxes don’t work for everything, but the devshells alone are great for reproducibility.

    • Wrong interpreter version? A devshell with a flake.lock file ensures every environment is using the exact same interpreter.
    • Accidentally picking up stuff from the local .env? Sandboxed checks and builds don’t get any files that aren’t version controlled, so that’s not an issue. But it’s still an issue with devshells.
    • Accidentally picking up programs or env vars in your environment? Sandboxed builds always get a clean starting environment. If you run nix develop --ignore-env you get a devshell that also gets a clean starting state.

    Nix doesn’t fix everything.

    • File system case sensitivity - depending on where this issue presents (program-generated files vs source files), I use property testing to catch this problem. In fact I was working on exactly that the other day.
    • Timing issues - that’s a good old fashioned hard problem. Try to make logical dependencies explicit. It’s really easy to get implicit order dependencies in concurrent code if you aren’t on guard. In languages that support it promises or futures are good for spelling out what needs to happen in what order.
    • Edit: Difficulty keeping secrets in sync - one option is to use Sops or Age to put encrypted secrets in version control. Then your CI only needs to be configured with one secret to decrypt the other secrets it needs.