Sysadmin Linux Theory

System Calls

Understanding system calls as the interface between user programs and the Linux kernel for requesting services

System calls are the interface between user programs and the Linux kernel. They provide a controlled way for applications to request kernel services like file operations, process management, and hardware access. Understanding system calls is crucial for debugging, performance analysis, and system programming.

Key Concepts

  • System Call Interface: Bridge between user space and kernel space
  • User Mode vs Kernel Mode: Different privilege levels for security
  • System Call Number: Unique identifier for each system call
  • Parameters: Arguments passed to system calls
  • Return Values: Success/failure codes and data returned
  • Library Wrappers: C library functions that invoke system calls

Command Syntax

strace [options] command [args]

  • Trace system calls and signals
  • Monitor program behavior at kernel level
  • Debug application issues

Common Options

-c - Count system calls and summarize -e trace=syscall - Trace specific system calls -f - Follow child processes -o file - Write output to file -p pid - Attach to running process -T - Show time spent in each system call

Practical Examples

Example 1: Basic system call tracing 

1
2
3
4
5
6
7

strace ls /home
execve("/bin/ls", ["ls", "/home"], ...) = 0
openat(AT_FDCWD, "/home", O_RDONLY|O_NONBLOCK) = 3
getdents64(3, /* 5 entries */, 32768) = 144
write(1, "user1  user2  user3\n", 20) = 20
close(3) = 0
exit_group(0) = ?

Shows system calls made by ls command

Example 2: Count system calls 

1
2
3
4
5
6
7

strace -c cat /etc/passwd
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- --------
 45.45    0.000050          10         5           read
 27.27    0.000030          15         2           write
 18.18    0.000020          10         2           openat
  9.09    0.000010          10         1           close

Summary of system call usage and timing

Example 3: Trace file operations only 

1
2
3
4
5
6
7

strace -e trace=openat,read,write,close cp file1 file2
openat(AT_FDCWD, "file1", O_RDONLY) = 3
openat(AT_FDCWD, "file2", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 4
read(3, "Hello World\n", 131072) = 12
write(4, "Hello World\n", 12) = 12
close(4) = 0
close(3) = 0

Filters to show only file-related system calls

Example 4: Monitor running process 

1
2
3
4

strace -p 1234 -e trace=network
# Attach to process PID 1234 and trace network calls
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
connect(5, {sa_family=AF_INET, sin_port=htons(80)}, 16) = 0

Traces network system calls of existing process

Common System Call Categories

File Operations

  • open/openat - Open files
  • read/write - Read/write data
  • close - Close file descriptors
  • lseek - Change file position
  • stat/fstat - Get file information

Process Management

  • fork - Create new process
  • execve - Execute program
  • wait/waitpid - Wait for child process
  • exit/exit_group - Terminate process
  • getpid/getppid - Get process IDs

Memory Management

  • mmap/munmap - Map/unmap memory
  • brk/sbrk - Change heap size
  • mprotect - Set memory protection
  • malloc (uses brk/mmap internally)

Use Cases

  • Debugging: Find why programs fail or behave unexpectedly
  • Performance Analysis: Identify bottlenecks in system calls
  • Security Auditing: Monitor file and network access
  • Learning: Understand how programs interact with kernel
  • Reverse Engineering: Analyze program behavior

ltrace - Trace library function calls perf - Performance analysis tool lsof - List open files and system calls netstat - Network connections and statistics procfs - Virtual filesystem (/proc) for process info

Tips & Troubleshooting

Performance Considerations

  • Use -c flag for summary instead of full trace
  • Filter specific system calls with -e trace=
  • Avoid tracing high-frequency calls in production

Common Issues

  • Permission denied: Use sudo for system processes
  • Too much output: Redirect to file with -o
  • Process exits quickly: Use -f to follow children

Security Notes

  • System call tracing can expose sensitive data
  • Some processes may detect being traced
  • Root privileges needed for most system processes
  • Be cautious when tracing in production environments

Useful Patterns 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Find failed system calls
strace -e trace=openat program 2>&1 | grep ENOENT

# Monitor file access
strace -e trace=file program

# Check network activity
strace -e trace=network program

# Time analysis
strace -T -c program