systemd Deep Dive: Units, Targets, Journals, and Troubleshooting

Essential systemctl Commands

systemctl start|stop|restart|reload nginx
systemctl status nginx
systemctl enable|disable nginx     # boot behavior
systemctl is-enabled nginx
systemctl list-units --type=service --state=failed
systemctl list-dependencies nginx

---

Writing a Unit File from Scratch

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Server
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server --port 8080
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
EnvironmentFile=/etc/myapp/env

[Install]
WantedBy=multi-user.target

systemctl daemon-reload    # required after editing unit files
systemctl enable --now myapp

---

Key Unit File Fields

Type: simple (default, ExecStart IS the service), forking (process forks), oneshot (runs and exits), notify (signals readiness via sd_notify).

Restart: no, on-failure, always.

After vs Requires: After = ordering only. Requires = hard dependency (fails if dep fails). Wants = soft dependency.

---

journalctl: Reading Logs

journalctl -u nginx -f              # follow in real time
journalctl -u nginx -n 100          # last 100 lines
journalctl -u nginx --since "1 hour ago"
journalctl -b -1 -u nginx           # previous boot (useful after crashes)
journalctl -u nginx -p err          # errors only

---

Debugging a Failed Service

# 1. Status and recent log
systemctl status myapp

# 2. Full journal since last start
journalctl -u myapp -n 50 --no-pager

# 3. Run the ExecStart command manually as the service user
sudo -u appuser /opt/myapp/bin/server --port 8080

# 4. Check dependencies
systemctl list-dependencies myapp

Common causes: wrong User (cannot access files), missing EnvironmentFile, binary not found, port already in use.

---

Targets (Replacing Runlevels)

systemctl get-default
systemctl set-default multi-user.target   # no GUI
systemctl isolate rescue.target           # switch immediately

Practice in the ShellGenius Linux Labs — the systemd-services and service-debug challenges use real broken service scenarios.