Troubleshooting SSH host & client-side errors

Smallstep provides a growing comprehensive list of the most common problems when registering a host or when a client is attempting to connect to registered SSH hosts.

SSH via Smallstep is generally successful without changes. However, some factors and configurations can generate errors that need resolving.

The most common errors and resolutions are documented in our Smallstep SSH Professional Quick Start Guide.

The following problems are less likely to occur but have been encountered.

Related to Error Message Possible Causes Possible Fix/Resolution
Client Access Too many authentication failures.

This error is not related to Smallstep SSH Pro, but rather SSH itself.

When SSH attempts to connect to a host it runs through each of the keys it has a record for.

Linux systems typically have a configuration parameter in /etc/ssh/sshd_config called, "MaxAuthTries" which dictates how many key iterations it should make before throwing this error message. That max can be changed, but 

There are multiple approaches to resolving this, depending on the operating system.

Search for the most accurate resolution now.

Host Registration error getting authority data: authority not found (when registering a host) The values for the installation script's parameter arguments may contain incorrect characters or are not properly formatted or parsable.

Check the part of the host registration command  for the following:

  • Check if the team name is correct.
  • Check if any parameter argument values that are wrapped in quotes haven't been auto-corrected which can affect the interpretation of the quote. This is especially common on macOS with Smart Quotes and Smart Dashes.
Client Access Permission denied (publickey) 

This error is vague and requires a more detailed examination of the customer's setup.

  • This error can happen when a host is unregistered from the Smallstep Dashboard, and the end-user tries to SSH to the host. 
  • This error can happen when an Authority's third-party OIDC Provisioners are not set up correctly due to their account plan not being on Teams or Enterprise. Free accounts cannot use third-party OIDC Provisioners. Smallstep provides a free one for Free accounts.

If host registrations have been made, make sure that each client runs `step ssh logout` followed by `step ssh config --team [your_team_name] and finally `step ssh login`.

If you're unable to add or complete the setup for an OIDC Provisioner, make sure you're on a Teams or Enterprise Plan and make sure you've provided payment information and haven't skipped that stage.

Host Registration "nc: command not found" A required `nc` (netcat) package is not installed on the host. Install the `nc` package on the host and retry.
Host Registration and Client Access Permission denied (publickey).

kex_exchange_identification: Connection closed by remote host

Connection closed by UNKNOWN port 65535
This error is vague and requires a more detailed examination of the customer's setup.
  • Smallstep Host Configuration
    • Check the host configuration at https://smallstep.com/app/YOUR_TEAM_NAME/ssh/hosts to ensure the bastion settings are correct. If your host is not a bastion or behind a bastion, keep those bastion settings blank.
    • DNS - This error might be encountered if the SSH host was set up with a hostname not resolvable by external client machines. Have the client user `ping` the hostname from their machine. If necessary, add a host record on the client machine, or work with the customer's dev/sec/ops on resolving the hostname through proper DNS resolution. Another DNS test could be temporarily configuring the SSH hostname with an IP address to rule out DNS.
  • Client configuration
    • Some changes made to a host registration (e.g., uninstalling and reinstalling on the same host), may affect client access, causing "Permission denied (Public Key)." The recommendation is to have the client user run `step ssh logout` followed by `step ssh login`.



 
Client Access "permission denied (publickey)" when running `ssh ssh-test.app.smallstep.com` Possible unstable client configuration state.

Reinitialize the client side:

  • Either move or completely remove the ~/.step hidden directory in the user's home directory.
  • Run the following commands:
    • `step ssh config --team [your_team_name]
    • `step ssh login`.
  • Try accessing the test host again.
Client Access

`step ssh hosts` only returns the Smallstep SSH test host

`step ssh hosts` returns the Smallstep test SSH host (ssh-test.app.smallstep.com) and nothing else. The user expected at least one SSH host that they're permitted to access.

  • The SSH Host configuration in Smallstep is missing tags**
  • The Smallstep SSH host has a Tag, but the SSH User Group doesn't have one
  • The user doesn't belong to the Smallstep SSH user group that maps to the intended SSH host
  • Something is incorrect with the IdP configuration (etc. OKTA, etc.)
SSH Host Tags are required to map access to Smallstep SSH user groups. Ensure you have a tag for the Host to map to Smallstep SSH user groups. Also, check the Smallstep SSH user group configuration to ensure its Tag matches the desired SSH Host Tag(s). Also, check that the user belongs to a Smallstep SSH user group that they should.

ℹ️ For more detailed instructions, follow along with the Smallstep Tag documentation here.
Host Registration and Client Access

TOFU (Trust on First Use) warning when connecting via SSH to the IP of a hostname registered in Smallstep.

For example, below is a hostname, not an IP. If a user runs SSH to the IP of the hostname, it will not assume certificate authentication and will throw a TOFU warning:

 

  • The Host was not set up with principals for the IP (or Hostname if it was registered with an IP)
  • To allow users to SSH to either the hostname or the IP, the host registration must take advantage of the --principal flag either during registration or after the fact from the User/Groups settings in the Smallstep Dashboard.
  • Check out the SSH installation instructions here to ensure that tags are set up and assigned as expected. If a user cannot see a host, they should make sure they're in a group that's tagged for that host.
Client Access

When running `step ssh config --team [team]` they get, "The request lacked necessary authorization to be completed.

The account plan type might be listed as "Free" and requires a Teams or Enterprise plan.

**Note that if you exit the setup of your IdP for OIDC connections and didn't complete the credit card portion, you may get this error.

Make sure the Dashboard's account plan is Teams or Enterprise. Also, make sure there's a payment method set up under the Billing menu in settings. If not, add a payment method and have the client run the configuration again.
Host Registration

When running host installation, you get "The request lacked necessary authorization to be completed."

 

Often this is a lack of authorization on the host operating system, or the Enrollment Token wasn't correct.

  • Check that you're running the script as sudo or as root
  • Double-check the Enrollment Token that was used when running the ssh-host.sh. If the Enrollment token doesn't match, you will get this error. As soon as you use the correct token, it should work. If Enrollment Token was misplaced or lost, contact support.
Host Registration

Google Cloud Platform (GCP) only:

Embedded web-based SSH using the Compute Engine fails with, "CONNECTION FAILED - You cannot connect to the VM instance because of an unexpected error. Wait a few moments, and then try again."

Chances are, the host has already been registered and thus configured for Smallstep Certificates. The problem is, GCP VMs initially use SSH keys to allow a web-based SSH session, which subsequently breaks when you register a host for SSH. You can still connect to the VM using a serial console if it's enabled. 

  • Stop the VM. 
  • Edit the VM to turn on Console Port connections, and save the settings.
  • Start the VM, and log into the VM via a serial console. Note: You need a root password or another user/password that's capable of sudo.
  • Uninstall the Smallstep SSH software and re-register the host.
Client Access

Users SSH's to a host but it gets hung up. Using vvv it hangs up as follows, and eventually connection times out

  • The host may have been unregistered and registered in some way. Have the user logout, config, login, and SSH
  • The host may have an ephemeral IP that was changed since the user last accessed the host.
  • Ephemeral Hosts: If the host was registered with an IP address for a host name, a change like this can't be resolved on the client side, the end user should contact their company's Smallstep admin to look at the host configuration.
  • Client Confirmation: While the following doesn't guarantee resolution, users are best to run a new config below. It won't resolve an IP address change that hasn't been updated on the host in Smallstep but will deal with any breaking changes from the client side.
    • `step ssh logout`
    • `step ssh config --team [[team name]]
    • `step ssh login`
      •  SSH again

Host Registration 

open /etc/ssh/ssh_host_ecdsa_key.pub failed: no such file or directory

SSH is not installed, or the service not running. Check that SSH is installed and that the service is running.