In the world of Linux system administration, SSH keys have been the gold standard for server access for over two decades. They eliminated the security risks of password-based authentication and made it possible to automate server access. But as infrastructure grows more complex and security threats evolve, traditional SSH key management has become a significant blind spot in many organizations’ security posture.

Think about your own environment for a moment. How many SSH keys are actively granting access to your servers right now? Who do those keys belong to? When was the last time each key was used? For most organizations, these questions are surprisingly difficult to answer.

The fundamental problem is that traditional SSH keys, once deployed, provide unlimited access with no expiration date. A key created years ago for a former employee might still grant access to critical systems. A contractor’s key from a past project could be silently providing persistent access long after their work was completed. Each unmanaged key represents an unmonitored entry point to your infrastructure.

This article explores how to implement a more secure approach called “Just-in-Time” SSH access, where users receive temporary access that automatically expires after a short period. We’ll focus on a straightforward implementation using HashiCorp Vault that any system administrator can set up, even without specialized security knowledge. By the end of this guide, you’ll have a practical path to significantly improve your SSH security posture while actually reducing administrative overhead.

The Problem with Traditional SSH Keys

Before we dive into solutions, let’s look at why traditional SSH key management is problematic:

• Keys don’t expire: Unlike passwords, standard SSH keys remain valid indefinitely unless manually revoked.
• Key sprawl: As your organization grows, SSH keys multiply and become impossible to track.
• Poor visibility: It’s hard to know who has access to which systems and when that access was last used.
• Difficult revocation: When an employee leaves, finding and removing all their SSH access is challenging.
• Limited accountability: It’s hard to know who accessed what and when.

Consider this scenario: A contractor worked on your systems six months ago. Are you certain their SSH access was properly removed? With traditional SSH keys, this simple question is surprisingly difficult to answer.

Understanding Zero Trust SSH

Zero Trust SSH follows a simple principle: grant access only when needed, for exactly as long as needed, and no longer. The key components of this approach are:

1. Short-lived credentials: Access automatically expires after a set time (typically hours, not years)
2. Just-in-time access: Users request access only when they need it
3. Strong authentication: Users must prove their identity before receiving access
4. Comprehensive logging: All access requests and sessions are recorded

A Practical Implementation with HashiCorp Vault

We’ll implement Zero Trust SSH using HashiCorp Vault, a widely-used open-source tool that’s relatively easy to set up and maintain. Our setup will use Vault’s SSH Certificate Authority (CA) feature.

How SSH Certificates Work

Before diving into the implementation, let’s understand how SSH certificates differ from traditional SSH keys:

Traditional SSH key authentication requires adding each user’s public key to the authorized_keys file on every server they need to access. This creates a management nightmare as you add more users and servers.

With SSH certificates:

• A trusted Certificate Authority (CA) signs a user’s public key, creating a certificate
• Servers are configured to trust the CA, not individual user keys
• Certificates include an expiration date and can restrict what the user can do
• Users request new certificates when they need access

This approach is much easier to manage because you only need to configure each server once to trust the CA.

Step 1: Install and Configure Vault

First, let’s set up Vault on a dedicated server:

# Add HashiCorp repository
sudo apt update
sudo apt install -y software-properties-common
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"

# Install Vault
sudo apt update
sudo apt install -y vault

# Create a configuration file
sudo mkdir -p /etc/vault.d
sudo tee /etc/vault.d/vault.hcl > /dev/null << EOF
storage "file" {
  path = "/opt/vault/data"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1  # Enable TLS in production!
}

api_addr = "http://127.0.0.1:8200"
ui = true
EOF

# Create data directory
sudo mkdir -p /opt/vault/data
sudo chown -R vault:vault /opt/vault

# Start Vault
sudo systemctl enable vault
sudo systemctl start vault

Important: This is a simplified setup for demonstration. For production, you should enable TLS and consider using a more robust storage backend like Consul.

Next, initialize Vault and unseal it (you’ll only do this once):

# Initialize Vault
vault operator init

# This will output 5 unseal keys and a root token
# Save these in a secure location!

# Unseal Vault (you'll need to run this 3 times with different keys)
vault operator unseal

Step 2: Configure Vault’s SSH Certificate Authority

Now that Vault is running, let’s set up the SSH certificate authority:

# Log in to Vault with your root token
vault login

# Enable the SSH secrets engine
vault secrets enable -path=ssh ssh

# Generate a new CA key pair
vault write ssh/config/ca generate_signing_key=true

Vault has now generated a CA key pair that we’ll use to sign SSH certificates. Next, let’s create a simple role that defines what kind of certificates can be issued:

# Create a role for standard users
vault write ssh/roles/user-role -<<EOF
{
  "allow_user_certificates": true,
  "allowed_users": "*",
  "default_extensions": {
    "permit-pty": ""
  },
  "key_type": "ca",
  "default_user": "ubuntu",
  "ttl": "4h"
}
EOF

This role allows:

• User certificates (as opposed to host certificates)
• Any username (you can restrict this to specific users)
• PTY allocation (interactive terminal sessions)
• A 4-hour validity period (adjust this based on your security needs)

Step 3: Configure Your Servers to Trust Vault’s CA

Next, we need to configure all your SSH servers to trust certificates signed by Vault’s CA. You’ll run these commands on each server:

# Create a directory for SSH certificates
sudo mkdir -p /etc/ssh/ca

# Get the CA public key from Vault
vault read -field=public_key ssh/config/ca > ca.pub

# Copy the public key to your server (if running this from your workstation)
scp ca.pub user@server:/tmp/
ssh user@server "sudo mv /tmp/ca.pub /etc/ssh/ca/trusted_user_ca_keys.pem"

# Configure SSH to trust certificates signed by this CA
ssh user@server "echo 'TrustedUserCAKeys /etc/ssh/ca/trusted_user_ca_keys.pem' | sudo tee -a /etc/ssh/sshd_config"

# Restart SSH to apply the changes
ssh user@server "sudo systemctl restart sshd"

For multiple servers, you can script this process or use configuration management tools like Ansible.

Step 4: Set Up User Authentication

For a complete Zero Trust implementation, users should authenticate with their existing credentials and multi-factor authentication. Here’s a simple way to set up username/password authentication with Vault:

# Enable userpass authentication
vault auth enable userpass

# Create a policy for SSH access
vault policy write ssh-access -<<EOF
path "ssh/sign/user-role" {
  capabilities = ["create", "update"]
}
EOF

# Create a test user
vault write auth/userpass/users/testuser \
    password=changeMe123 \
    policies=ssh-access

In a production environment, you’d likely use LDAP, Active Directory, or OIDC (Okta, Google, etc.) instead of userpass authentication.

Step 5: User Workflow for Accessing Servers

Now let’s see how users would request access to servers:

1. User authenticates to Vault:

# Log in to Vault
vault login -method=userpass username=testuser

2. User requests an SSH certificate:

# Request a certificate for your SSH key
vault write -field=signed_key ssh/sign/user-role \
    public_key=@$HOME/.ssh/id_rsa.pub > $HOME/.ssh/id_rsa-cert.pub

# Check the details of your certificate
ssh-keygen -Lf ~/.ssh/id_rsa-cert.pub

3. User connects to the server using the certificate:

# Connect to the server
ssh -i ~/.ssh/id_rsa -i ~/.ssh/id_rsa-cert.pub ubuntu@server-hostname

The certificate will automatically expire after the TTL (4 hours in our example), requiring the user to request a new certificate for subsequent access.

Step 6: Making the User Experience Better

To make this process more user-friendly, create a simple script that users can run:

#!/bin/bash
# get-ssh-cert.sh - Get a short-lived SSH certificate from Vault

# Check if we're logged in to Vault
if ! vault token lookup >/dev/null 2>&1; then
  echo "Not logged in to Vault. Please authenticate:"
  vault login -method=userpass username=$USER
fi

# Request a certificate
echo "Requesting SSH certificate..."
vault write -field=signed_key ssh/sign/user-role \
    public_key=@$HOME/.ssh/id_rsa.pub > $HOME/.ssh/id_rsa-cert.pub

# Check if it worked
if [ $? -eq 0 ]; then
  # Get expiration time
  expiration=$(ssh-keygen -Lf ~/.ssh/id_rsa-cert.pub | grep Valid | awk '{print $4, $5, $6, $7, $8}')
  echo "✅ Certificate issued successfully!"
  echo "Valid until: $expiration"
else
  echo "❌ Failed to get certificate"
  exit 1
fi

Users can simply run this script before accessing servers, and it will handle authentication and certificate issuance.

For an even better experience, users can add this to their SSH config:

# Add to ~/.ssh/config
Host *.example.com
    IdentityFile ~/.ssh/id_rsa
    CertificateFile ~/.ssh/id_rsa-cert.pub

With this configuration, users can use SSH normally without specifying the certificate file each time.

Monitoring and Logging

One of the key benefits of this approach is improved logging and visibility. Vault logs all certificate issuance, and you can view this activity:

# View recent certificate signing operations
vault read sys/internal/counters/activity

For comprehensive monitoring, configure Vault’s audit logging:

# Enable file audit logging
vault audit enable file file_path=/var/log/vault/audit.log

# View recent activity
tail -f /var/log/vault/audit.log | jq

These logs show who requested certificates, when, and for what purpose.

Common Questions and Challenges

How do I handle emergency access?

Create a separate role with longer TTL for emergency situations:

# Create an emergency role
vault write ssh/roles/emergency -<<EOF
{
  "allow_user_certificates": true,
  "allowed_users": "*",
  "default_extensions": {
    "permit-pty": ""
  },
  "key_type": "ca",
  "ttl": "24h"
}
EOF

# Create a policy with emergency access
vault policy write emergency-ssh -<<EOF
path "ssh/sign/emergency" {
  capabilities = ["create", "update"]
}
EOF

Assign this policy only to administrators and security personnel.

What if Vault is down?

If Vault is unavailable, users with valid certificates can still access servers until their certificates expire. For high availability:

1. Deploy Vault in a cluster configuration
2. Consider having a backup CA for true emergency situations
3. Implement longer certificate TTLs for critical staff

How do I handle automated scripts and CI/CD?

For automation, create dedicated roles with appropriate restrictions:

# Create a role for automation
vault write ssh/roles/automation -<<EOF
{
  "allow_user_certificates": true,
  "allowed_users": "jenkins,gitlab-runner",
  "default_extensions": {
    "permit-pty": ""
  },
  "key_type": "ca",
  "ttl": "1h"
}
EOF

Then have your automation system authenticate to Vault and request certificates as needed.

Benefits of Just-in-Time SSH Access

Implementing this approach provides several key benefits:

1. Automatic expiration: Access is time-limited without any manual intervention
2. Centralized management: Manage access from a single place instead of across all servers
3. Improved visibility: See who has access to what and when they last used it
4. Reduced attack surface: Even if keys are compromised, the damage is limited by short TTLs
5. Better compliance: Easily demonstrate access controls for audits and compliance

Getting Started: A Phased Approach

Don’t try to implement everything at once. Follow these steps:

1. Start small: Set up Vault and configure a few test servers
2. Pilot with IT staff: Have your technical team use it first
3. Document the process: Create clear instructions for users
4. Gradually expand: Add more servers and user groups
5. Integrate with existing systems: Connect to your identity provider
6. Enhance monitoring: Set up alerts for suspicious activity

Conclusion

Just-in-Time SSH access with certificate authentication provides a significant security improvement over traditional SSH keys with minimal ongoing management overhead. By implementing this approach, you’ll gain better control, visibility, and security across your entire infrastructure.

The best part is that users can still use standard SSH clients – there’s no need for special software or complicated procedures. Once the initial setup is complete, both administrators and users will benefit from the improved security and simplified access management.

Remember, security is a journey, not a destination. Start with this basic implementation and gradually enhance it as your needs evolve and your team becomes more comfortable with the new approach.

Ready to implement Just-in-Time SSH access in your environment? Start with a focused pilot project today, and you’ll quickly see the benefits of this more secure approach to server access.