SRE Interview Prep: Linux Troubleshooting Questions You'll Actually Face

How SRE Interviews Actually Work

Most Linux troubleshooting questions in SRE and DevOps interviews are not memory tests. Interviewers know you can look up command syntax. What they are testing is whether you can reason systematically under ambiguity - whether you ask the right questions, narrow down the problem space, and do not jump to conclusions.

The best answers usually have a structure: what you would check first and why, what the output means, and what you would do next. Interviewers specifically watch for candidates who immediately guess a solution without gathering information first.

With that framing, here are 12 questions you are likely to encounter.

---

Question 1: Load average is 8.0 on a 4-core machine. Is that a problem?

Load average of 8.0 on 4 cores means processes are waiting for CPU time on average - the queues are full and then some. But this is not always a crisis.

What you need to know:

The signal that matters: load average alone tells you something is wrong. vmstat, top, and ps tell you what.

---

Question 2: How do you find which process is listening on port 8080?

ss -tlnp | grep :8080

Output:

LISTEN  0  128  0.0.0.0:8080  0.0.0.0:*  users:(("node",pid=12345,fd=20))

The process name, PID, and file descriptor are in the output. If the port is in use but nothing shows as listening (can happen with TIME_WAIT sockets), use:

ss -tnp | grep :8080

Older systems may still use netstat:

netstat -tlnp | grep :8080

If you need to know who started that process:

ls -la /proc/12345/exe        # binary path
cat /proc/12345/cmdline | tr '' ' '   # full command line with args

---

Question 3: df -h shows the filesystem is 100% full, but du -sh /* shows only 60% used. What is happening?

This is a classic inode problem, or more commonly, open file descriptors holding space that du cannot see.

Deleted files still held open: When a process opens a file and another process deletes it, the file's blocks are not freed until the file descriptor is closed. du does not count deleted files; df does.

lsof | grep deleted | awk '{print $7, $8, $9}' | sort -rn | head

This shows files that are deleted but still open. If you see a log file that a crashed process is still holding, restarting that process will free the space immediately.

Inodes full, not blocks: If df -i shows 100% inode usage with free block space, you have too many small files. Common cause: mail spool directories, cache directories, or a runaway process creating temp files.

df -i                          # show inode usage
find /var -xdev -type f | wc -l   # count files (slow but thorough)
for d in /var/*/; do echo "$(find "$d" -xdev | wc -l) $d"; done | sort -rn | head

---

Question 4: A container keeps getting OOM-killed. How do you debug it?

Start with the kernel:

dmesg -T | grep -i "oom|out of memory|killed process" | tail -20

This shows which container and which process was killed, with a timestamp.

Then check the container's memory usage versus its limit:

docker stats <container_name> --no-stream
docker inspect <container_name> | grep -A3 '"Memory"'

Look inside the container at what is consuming memory:

docker exec -it <container> /bin/sh -c "cat /proc/meminfo && ps aux --sort=-%mem | head"

The resolution depends on what you find: if it is a memory leak, fix the application. If the limit is set too conservatively for legitimate usage, increase it. If logs are accumulating in the container, add a log rotation policy to the container runtime.

---

Question 5: A process is stuck in D state (uninterruptible sleep). What does that mean and what can you do?

D state in ps output means the process is waiting for I/O and cannot be interrupted - not even by kill -9. The process is typically waiting on a disk read, a network filesystem call, or a kernel operation.

You cannot directly kill a process in D state. Options:

  1. Find what it is waiting for: strace -p <pid> (if the process accepts attach). It will block too, but it shows the system call the process is stuck in.
  2. If it is NFS-related, the mount may be stale. Unmounting or force-unmounting the NFS share (umount -f -l /mount/point) will usually release the process.
  3. If it is a disk I/O issue, check dmesg for disk errors. A failing drive can cause processes to hang in D state for minutes.
  4. If nothing else works, the solution is a reboot. A process in D state that cannot be killed indicates a kernel-level problem.

---

Question 6: A cron job is not running. How do you debug it?

# Check if cron is running
systemctl status cron    # Debian/Ubuntu
systemctl status crond   # RHEL/CentOS

# Check the cron log
grep CRON /var/log/syslog | tail -30    # Debian/Ubuntu
journalctl -u cron --since "1 hour ago"
grep cron /var/log/cron | tail -30      # RHEL

# Verify the crontab syntax
crontab -l -u <username>

Common causes:

---

Question 7: ssh user@host returns "Connection refused". Walk through your debugging steps.

# Is the host reachable at all?
ping host

# Is port 22 open? (or whatever port SSH is on)
nc -zv host 22
telnet host 22

# Check from the server side if you have other access (console, bastion)
systemctl status sshd
ss -tlnp | grep :22

# Check firewall
iptables -L INPUT -n | grep 22
firewall-cmd --list-all    # RHEL/CentOS with firewalld

# Check /etc/hosts.deny and /etc/hosts.allow
# Check /var/log/auth.log or /var/log/secure for rejection reasons

Most common causes: sshd not running, firewall blocking the port, hosts.deny rule, wrong SSH port (check /etc/ssh/sshd_config for the Port directive).

---

Question 8: System time is wrong. Why does that matter, and how do you fix it?

Time drift causes: TLS certificate validation failures (client says cert is expired or not yet valid), distributed system issues (Kafka, etcd, Cassandra all rely on synchronized time), authentication systems (Kerberos tickets and TOTP tokens have narrow time windows), and log correlation across machines becoming impossible.

Check the current status:

timedatectl status         # shows NTP sync status and offset
chronyc tracking           # if chrony is the NTP client
systemctl status chronyd

Fix: ensure NTP is running and configured:

systemctl enable --now chronyd
chronyc -a makestep         # force immediate sync instead of gradual drift correction

For large offsets, ntpdate or chronyc makestep will step the clock immediately rather than slewing slowly.

---

Question 9: A service restarts every 5 minutes automatically. What is happening?

Three possibilities: a systemd Restart=always or RestartSec=300 policy causing automatic restarts after failure, a cron job or watchdog script restarting it, or the service itself is crashing and the init system is recovering it.

# Check restart behavior in the unit file
systemctl cat <service>    # shows effective unit file after overrides
grep -i restart /lib/systemd/system/<service>.service

# Check what happened in the last hour
journalctl -u <service> --since "1 hour ago"

# Check for external watchdog scripts
grep -r "<service>" /etc/cron*
ls /etc/cron.d/

The journal (journalctl) will show the exact exit code and any last error output when the service terminated. That is usually the most direct path to the root cause.

---

Question 10: How would you find which process has a specific file open?

lsof /path/to/file            # who has this file open
lsof -u username              # all files open by a user
lsof -p <pid>                 # all files open by a process
lsof +D /var/log              # all files open under /var/log (recursive)

This is essential before unmounting a filesystem (umount will fail if anything is open on it) and for diagnosing why a deleted file's space has not been freed.

---

Question 11: Disk I/O is high. How do you identify what is causing it?

iostat -xz 1          # per-device I/O stats, 1-second refresh
iostat -xz 1 | grep -v "^
quot;   # same without blank lines

iotop -o              # like top but for disk I/O, shows processes

In iostat output, the columns that matter: %util (how saturated the device is - above 80% is busy), await (average wait time per request in ms - above 10ms for SSD or 30ms for spinning disk suggests a problem), r/s and w/s (reads and writes per second).

Once you know which device, use iotop or pidstat -d 1 to find which process is driving the I/O.

---

Question 12: Zombie processes are accumulating. What causes this and what do you do?

A zombie process (Z state in ps) is one that has finished executing but whose parent has not called wait() to read its exit status. The process entry stays in the process table until the parent acknowledges it.

Zombies themselves consume almost no resources - just a PID and a small entry. A few zombies are normal. Thousands of zombies may exhaust the PID limit.

ps aux | grep Z          # list zombies

The output shows the zombie's PPID. The parent is the process that needs to fix:

ps -o ppid= -p <zombie_pid>    # get parent PID
ps aux | grep <ppid>           # identify the parent

Solutions:

---

How Interviewers Evaluate Your Answers

The best candidates do three things: they say what they would check first and why (showing a mental model), they distinguish between symptoms and root causes, and they acknowledge when a problem could have multiple causes before committing to one.

Candidates who jump straight to "I would restart the service" without gathering data signal that they operate by cargo-culting rather than reasoning. Candidates who ask clarifying questions ("Is this a VM or a container? Is this a new issue or did it just start?") signal that they understand context changes the approach.

The ShellGenius Linux Labs and Learn track are built around exactly this kind of systematic troubleshooting - working through real scenarios with real tools, not memorizing commands in isolation.