Script works in the shell but fails as a systemd service: PATH and missing environment

Every night at 02:00 a timer on one of our app servers runs ship-reports.sh, which builds CSV reports for our partner portal with a Node.js script and uploads them to object storage. We had moved it from a crontab line to a systemd service and timer the week before, mainly to get logs in the journal. On the first morning, partners emailed that their reports were missing.

The journal showed the failure clearly:

Aug 05 02:00:01 app-03 ship-reports.sh[48211]: /opt/reports/ship-reports.sh: line 14: node: command not found
Aug 05 02:00:01 app-03 systemd[1]: reports-export.service: Main process exited, code=exited, status=127/n/a
Aug 05 02:00:01 app-03 systemd[1]: reports-export.service: Failed with result 'exit-code'.

The engineer who wrote the unit logged in as the reports user, ran /opt/reports/ship-reports.sh, and it worked. That is the classic shape of this bug.

Reproducing it

The quickest way to see what a service really gets is to run a command under systemd with the same user and print its environment:

sudo systemd-run --uid=reports --wait --pipe env

LANG=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
INVOCATION_ID=4f6d2c...
USER=reports
LOGNAME=reports
HOME=/home/reports
SHELL=/bin/bash

Compare that with a login shell:

sudo -iu reports bash -c 'echo "$PATH"; echo "$REPORTS_BUCKET"; pwd'

/home/reports/.nvm/versions/node/v22.18.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
partner-reports-prod
/home/reports

Three differences stood out: node lived under ~/.nvm, which only .bashrc adds to PATH; REPORTS_BUCKET was exported in .bashrc; and the working directory was the home directory.

What we ruled out

• Permissions. The script was executable, owned by reports, and systemctl status showed it started; it failed on line 14, not at exec time.
• The timer. systemctl list-timers showed the correct next run, and systemctl start reports-export.service failed the same way immediately.
• SELinux or AppArmor. No denials in journalctl -k or /var/log/audit, and the error was a plain shell "command not found".

Root cause

The script relied on the interactive login environment of the reports user. Under cron it had only half worked too: the crontab line started with bash -lc, which loaded the profile. systemd starts services directly, with a fixed default PATH, no shell startup files and, unless told otherwise, / as the working directory. Once we added node to PATH, the next run failed on the next assumption:

Error: ENOENT: no such file or directory, open './config/partners.yml'

and after that, the upload step failed because REPORTS_BUCKET was empty.

What is happening under the hood

systemd builds each service's environment from scratch. It does not run .bashrc, .profile or /etc/profile, and it does not inherit the environment of the shell that typed systemctl start. What a service gets:

Everything else must be declared in the unit with Environment=, EnvironmentFile=, WorkingDirectory= and absolute paths. That is a feature: the service behaves the same no matter who restarts it and from where.

Environment

Minimal reproduction

# /etc/systemd/system/env-demo.service
[Service]
Type=oneshot
User=reports
ExecStart=/bin/bash -c 'echo "PATH=$PATH"; echo "cwd=$(pwd)"; command -v node || echo "node missing"'

sudo systemctl daemon-reload
sudo systemctl start env-demo.service
journalctl -u env-demo.service -n 5 --no-pager

The journal shows the default PATH, cwd=/, and node missing.

Who is affected

Anyone moving jobs from a terminal, tmux session or bash -lc cron line into systemd, and especially anyone using per-user version managers (nvm, pyenv, rbenv, asdf, SDKMAN). Units with ExecStart=node app.js fail even earlier with status=203/EXEC, because systemd resolves a bare command name against its own fixed search path, not the user's.

The fix

We made the unit carry everything the job needs, and made the script stop depending on its working directory.

1. Install Node.js system-wide. Per-user nvm is fine for development, not for services. Debian 12's own package is Node.js 18, so we unpacked the official Node.js 22 Linux binary into /usr/local, which gives a stable, root-owned path at /usr/local/bin/node. If you must keep a version manager, reference its binary by absolute path instead of relying on PATH.

2. Move configuration into an environment file owned by root and readable only by the service user:

sudo install -d -m 0750 -o root -g reports /etc/reports
sudo install -m 0640 -o root -g reports /dev/null /etc/reports/export.env

# /etc/reports/export.env
REPORTS_BUCKET=partner-reports-prod
REPORTS_CONFIG=/opt/reports/config/partners.yml
NODE_ENV=production

3. Rewrite the unit:

 [Service]
 Type=oneshot
 User=reports
-ExecStart=/opt/reports/ship-reports.sh
+Group=reports
+WorkingDirectory=/opt/reports
+EnvironmentFile=/etc/reports/export.env
+Environment=PATH=/usr/local/bin:/usr/bin
+ExecStart=/opt/reports/ship-reports.sh
+TimeoutStartSec=30min

4. Make the script defensive so it fails loudly and never depends on pwd:

#!/usr/bin/env bash
set -euo pipefail

cd "$(dirname "$(readlink -f "$0")")"
: "${REPORTS_BUCKET:?REPORTS_BUCKET is not set}"
: "${REPORTS_CONFIG:?REPORTS_CONFIG is not set}"

node build-reports.mjs --config "$REPORTS_CONFIG" --out /var/tmp/reports
./upload.sh /var/tmp/reports "$REPORTS_BUCKET"

The :? expansions turn a missing variable into a clear one-line error in the journal instead of an upload to an empty bucket name.

Alternatives and trade-offs

1. ExecStart=/bin/bash -lc /opt/reports/ship-reports.sh. It reproduces the old cron behaviour and works, but the service now depends on whatever any profile file does, including interactive-only output that can break scripts. We used it for one night as a stopgap.
2. Environment= lines instead of a file. Fine for non-secret values. Unit files are readable by every local user, so secrets do not belong there.
3. systemd credentials (LoadCredential=). Better for secrets such as API keys: the value is exposed to the service as a file under $CREDENTIALS_DIRECTORY, not as an environment variable visible through /proc. We moved the storage key to this mechanism.

How to test the fix

sudo systemctl daemon-reload
sudo systemctl start reports-export.service
systemctl status reports-export.service --no-pager
journalctl -u reports-export.service -n 20 --no-pager

To check the exact environment the service received without reading the script:

sudo systemctl show reports-export.service -p Environment -p EnvironmentFiles -p WorkingDirectory

Environment=PATH=/usr/local/bin:/usr/bin
EnvironmentFiles=/etc/reports/export.env (ignore_errors=no)
WorkingDirectory=/opt/reports

We also ran systemd-analyze verify /etc/systemd/system/reports-export.service, which catches typos in directive names that systemd would otherwise only log as warnings.

Confirming in production

The next scheduled run finished in 4 minutes 12 seconds and uploaded 37 partner files, matching the last successful cron run. We added a final log line with counts, so the journal now ends each run with something we can alert on:

Aug 06 02:04:13 app-03 ship-reports.sh[50122]: uploaded 37 files (12.4 MiB) to partner-reports-prod
Aug 06 02:04:13 app-03 systemd[1]: reports-export.service: Deactivated successfully.

A week of runs showed the same result each night, and the partner portal's "report freshness" check stayed under 26 hours.

Regression guard

• OnFailure=notify-oncall@%n.service on the unit sends a chat alert whenever a run fails, instead of waiting for partner emails.
• A CI check runs systemd-analyze verify on every unit file in the repository and greps ExecStart= lines for non-absolute paths.
• The script's set -euo pipefail and :? guards make any missing variable a hard failure.

Security implications

The fix improved the security posture. Values that used to sit in a user's .bashrc are now in a file with explicit ownership and mode 0640, and the storage key moved to LoadCredential=, so it is no longer an environment variable that child processes inherit and that could leak into crash dumps or debug output. Do not put secrets in Environment= lines; systemctl show exposes them to any local user. Pinning PATH to system directories also prevents a writable directory early in a user's PATH from hijacking commands run by the service.

Prevention checklist

• Use absolute paths for every binary in ExecStart= and in scripts run by services.
• Set WorkingDirectory= explicitly, and never use relative file paths in service scripts.
• Put configuration in EnvironmentFile= and secrets in LoadCredential=.
• Test units with systemctl start and journalctl, never only by running the script in a shell.
• Install runtimes system-wide for services instead of using per-user version managers.

References

• systemd.exec manual, environment variables, WorkingDirectory=, EnvironmentFile= — https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html
• systemd.service manual, ExecStart= command lines — https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html
• systemd-run manual — https://www.freedesktop.org/software/systemd/man/latest/systemd-run.html
• systemd documentation, Credentials — https://systemd.io/CREDENTIALS/
• GNU Bash manual, Bash startup files — https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html