In today’s increasingly hostile threat landscape, SSH remains one of the primary vectors through which attackers gain persistent access to critical infrastructure. Despite proper hardening and monitoring, sophisticated adversaries can sometimes bypass defenses, making forensic analysis capabilities a crucial component of any security strategy.

When a server shows signs of compromise, rapid forensic analysis of SSH activity becomes critical to understanding the scope and nature of the breach. Security teams that can quickly identify unauthorized access patterns have a significant advantage in containing incidents and preventing future attacks. Without proper forensic capabilities, organizations often struggle to determine what happened during an incident, leading to incomplete remediation and recurring breaches.

According to recent data from the SANS Institute, approximately 67% of organizations have experienced SSH-related security incidents, yet less than 30% report having comprehensive forensic response capabilities specifically tailored to SSH investigations. This gap represents a significant blind spot in many security programs.

This guide covers six powerful SSH forensic analysis techniques that every system administrator should master to effectively respond to potential security incidents. These techniques go beyond basic log analysis to provide deeper insights into attacker activities, helping organizations build a complete picture of what happened before, during, and after a breach.

1. Analyze Authentication Logs with Advanced Filtering

Understanding the attack timeline is crucial for effective incident response. SSH authentication logs contain valuable timestamps, IP addresses, and authentication methods that can reveal both the initial compromise and subsequent attacker movements. However, these logs often contain thousands of entries, making manual analysis impractical.

The first step in any SSH forensic investigation is to examine authentication logs for suspicious patterns. While basic grep commands can help, combining multiple tools provides deeper insights. Advanced filtering techniques allow investigators to quickly identify anomalies that might indicate compromise, such as login attempts from unusual geographic locations, authentication outside normal business hours, or credential stuffing attacks targeting multiple accounts.

# Find all failed SSH login attempts with timestamps and IP addresses
grep "Failed password" /var/log/auth.log | awk '{print $1,$2,$3,$11}' | sort | uniq -c | sort -nr

# Identify successful logins from unusual geographic locations
grep "Accepted" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | 
while read count ip; do
  geo=$(geoiplookup $ip | awk -F ": " '{print $2}')
  echo "$count $ip $geo"
done | sort -nr

# Detect login attempts outside business hours (8 AM - 6 PM)
grep "sshd" /var/log/auth.log | awk '
  { time=$3; 
    split(time, t, ":"); 
    hour=t[1]; 
    if (hour < 8 || hour >= 18) 
      print $0
  }'

What’s happening here? The first command identifies all failed login attempts, counting occurrences by IP address. This helps detect brute force attacks. The second command geolocates successful logins, which can reveal access from unexpected countries or regions. The third command filters for SSH activity outside normal business hours, a common indicator of malicious activity.

2. Reconstruct SSH Session Activity with Session Recording

If you’ve enabled session recording (as recommended in security best practices), you can reconstruct an attacker’s activities:

# Install and configure session recording if not already done
apt install tlog
echo "session required pam_tlog.so" >> /etc/pam.d/sshd

# Review recorded sessions (after installation)
tlog-play -r journal -M TLOG_REC=$(journalctl -o json TLOG_REC=1 | jq -r '.TLOG_REC' | head -1)

# Extract commands executed during a specific session
aureport --tty | grep <username> | grep <date>
ausearch -i -k tty-keystrokes | grep <session_id> | awk -F "data=" '{print $2}' | xxd -r -p

Session recording creates a detailed transcript of all commands and outputs during an SSH session. This is invaluable for understanding what an attacker attempted to do on your system. The tlog-play command replays recorded sessions in real-time, while aureport and ausearch extract specific commands and keystrokes when using the Linux audit framework.

For legacy systems without session recording, check history files and command timestamps:

# Check modified timestamps on user bash history files
find /home -name ".bash_history" -exec ls -la {} \;

# Look for attempts to clear command history
grep -i "history -c\|history -w\|rm .*bash_history" /var/log/auth.log

3. Analyze SSH Key Usage and Unusual Authentication Methods

SSH keys represent one of the most common persistence mechanisms used by attackers. Unlike password-based access which generates authentication logs with each use, SSH key-based access can be much stealthier. Advanced Persistent Threat (APT) groups frequently deploy backdoor SSH keys as part of their toolkit, allowing them to maintain access even after password changes or other remediation efforts.

A 2024 study by Mandiant revealed that in 72% of investigated breaches where attackers established persistence, unauthorized SSH keys were among the primary mechanisms. These keys are particularly dangerous because they can be easily overlooked during incident response if forensic analysts don’t specifically look for them.

Sophisticated attackers often deploy unauthorized SSH keys to maintain persistence. Identify these with:

# Find all authorized_keys files and their last modification times
find / -name "authorized_keys" -exec ls -la {} \;

# Check for recently added SSH keys
for user in $(cut -f1 -d: /etc/passwd); do
  if [ -d "/home/$user/.ssh" ]; then
    echo "=== $user ==="
    find "/home/$user/.ssh" -type f -mtime -30 -exec ls -la {} \;
  fi
done

# Identify SSH agent forwarding usage, which can indicate lateral movement
grep "agent forwarding" /var/log/auth.log | awk '{print $1,$2,$3,$11}'

What’s happening here? The first command locates all authorized_keys files system-wide, which might reveal keys in unexpected locations. The second command identifies recently modified SSH key files, which could indicate an attacker adding their own access keys. The third command focuses on SSH agent forwarding, a technique often used for lateral movement through a network once initial access is gained.

4. Correlate SSH Activity with Network Traffic Analysis

SSH breaches often involve data exfiltration or connection to command-and-control servers. Correlate SSH sessions with network traffic:

# Identify all established SSH connections with destination IPs
ss -tnp | grep ssh

# Capture and analyze SSH traffic patterns (packet sizes and timing)
tcpdump -nnXSs 0 'tcp port 22' -w ssh_traffic.pcap

# Analyze the captured traffic for unusual patterns
tshark -r ssh_traffic.pcap -q -z io,stat,60,"COUNT(tcp.analysis.retransmission)tcp.analysis.retransmission"

Network analysis provides insights that might not be visible in logs alone. The first command shows all current SSH connections with process information. The second uses tcpdump to capture detailed SSH traffic for further analysis. The final command uses tshark (the command-line version of Wireshark) to analyze packet patterns, which might reveal command-and-control traffic or data exfiltration attempts.

For long-term monitoring, consider using netflow data:

# Install and configure netflow collection
apt install nfdump
nfcapd -w /var/netflow -D -p 9996

# Analyze historical netflow data for SSH connections to unusual destinations
nfdump -R /var/netflow -t 2025/02/01-2025/03/01 'proto tcp and port 22' | sort -rn -k 7

5. Detect Backdoor SSH Servers and Non-Standard Ports

When attackers gain privileged access to a system, they often deploy customized backdoor versions of SSH servers to maintain persistence. These backdoors may include modifications that bypass standard logging, authentication mechanisms, or network monitoring. They frequently operate on non-standard ports to evade detection by security tools that focus exclusively on the standard SSH port 22.

More sophisticated backdoors might include rootkit functionality that actively hides their presence from standard system utilities. Such backdoors present significant challenges for incident responders, as they may not appear in standard process listings or network connection tables. The MITRE ATT&CK framework identifies this technique (T1205.001) as a common tactic used by numerous threat actors, including APT28, APT29, and Lazarus Group.

Attackers often install backdoor SSH services on non-standard ports. Identify these with:

# Identify all processes listening on network ports
netstat -tulpn | grep "LISTEN"

# Find processes that look like SSH but run on non-standard ports
ps aux | grep ssh | grep -v "grep\|:22"

# Check for modified SSH binaries
find /usr -name "ssh*" -type f -exec md5sum {} \; > current_ssh_md5.txt
diff current_ssh_md5.txt known_good_ssh_md5.txt

This approach finds processes listening on network ports, identifies SSH-like services running on non-standard ports, and verifies the integrity of SSH-related binaries. Backdoored versions of SSH might have the same name but different checksums compared to legitimate versions.

For a deeper analysis, examine loaded modules and libraries:

# Check for unusual libraries loaded by SSH processes
for pid in $(pgrep -f ssh); do
  echo "=== Process $pid ==="
  ls -la /proc/$pid/maps | grep -v " 0 "
done

6. Create Timeline Analysis of SSH-Related Filesystem Changes

To understand when a breach might have occurred, create a timeline of filesystem changes related to SSH:

# Install sleuthkit if not already available
apt install sleuthkit

# Create a timeline of all SSH-related file activities
find / -name "*ssh*" -type f -printf "%T+ %p\n" | sort

# Use advanced forensics tools for comprehensive timeline
fls -m / -r /dev/sda1 > filesystem.body
mactime -b filesystem.body -d -y "2025-01-01..2025-03-01" > timeline.csv
grep -i ssh timeline.csv | less

Timeline analysis helps establish when SSH configurations or binaries were modified. The first command provides a simple timestamp-sorted list of SSH-related files. The more advanced approach using Sleuthkit tools creates a comprehensive filesystem timeline that can reveal subtle changes an attacker might have made.

For systems with modern filesystems, you can also leverage journal data:

# Extract journaled filesystem changes related to SSH
debugfs -R "logdump -i <inode_of_ssh_config>" /dev/sda1

The Critical Role of SSH Forensics in the Incident Response Lifecycle

SSH forensic analysis doesn’t exist in isolation—it’s a crucial component of a comprehensive incident response strategy. When integrated into the wider incident response workflow, SSH forensics provides critical insights that inform containment decisions, eradication strategies, and recovery processes.

Organizations with mature security programs typically incorporate these SSH forensic techniques into their incident response playbooks, ensuring consistent application during security events. By standardizing these approaches, security teams can more rapidly identify the full scope of compromise, reducing both the time-to-detection and time-to-remediation metrics that directly impact breach costs.

According to IBM’s Cost of a Data Breach 2024 report, organizations with established forensic capabilities reduced average breach costs by approximately 32% compared to those without such capabilities. The difference is particularly pronounced in cases involving SSH-based lateral movement, where effective forensics can prevent attackers from reaching critical assets.

Conclusion

Effective SSH forensic analysis requires both broad scanning for anomalies and deep inspection of specific components. By mastering these six techniques, you can rapidly determine whether a breach has occurred, how attackers gained access, what actions they took, and what backdoors they might have installed.

Remember that forensic analysis is most effective when you have proper baseline logging configured beforehand. Implement comprehensive logging, session recording, and file integrity monitoring before incidents occur to ensure you have the data needed for thorough investigations.

For complex environments, consider using dedicated security information and event management (SIEM) solutions that can correlate SSH events with other security telemetry for more comprehensive threat detection and investigation capabilities.

The reality of today’s threat landscape is that prevention will sometimes fail, making detection and response capabilities essential components of a defense-in-depth strategy. By building robust SSH forensic capabilities before they’re needed, organizations position themselves to respond effectively when prevention measures are bypassed, ultimately reducing both the likelihood and impact of successful attacks.