SSH port forwarding for Beginners

Experimenting with SSH port forwarding using Docker

Port forwarding with SSH allows a client to access services running on the server-side or a server to access services running on the client-side over a secure SSH tunnel established between the client and server.

There are three types of port forwarding.

• Local
• Remote
• Dynamic

Dynamic port forwarding is not widely used. In this blog post, I will be covering experiments that use only Local and Remote port forwarding.

Prerequisites

• Basic understanding of using SSH
• Basic understanding of Docker

Basics of SSH port forwarding

What is Local Port forwarding?

Local port forwarding allows an SSH client to configure a listening port on the client machine, that forwards all incoming traffic to a service on the SSH server’s network, over an established SSH session with the server.

Image by Erik via StackExchange.

How to forward local ports using SSH?

The syntax for forwarding local ports using the ssh CLI client is as follows.

ssh -L "$local_listener:$server_side_listening_service" <other ssh options> $destination_server

The local_listener and server_side_listening_service can be either a TCP port or Unix socket.

Applications of Local Port forwarding

• To access services (like mail servers and websites) in protected secure networks, that are otherwise blocked by firewalls.
• To secure insecure services by tunnelling the traffic over an SSH session from a client to a jump host, or from a client to the target server.

What is Remote Port forwarding?

Remote port forwarding allows an SSH client to configure a listening port on the SSH server, that forwards all incoming traffic to a service on the client’s network, over an established SSH session with the server.

Image by Erik via StackExchange.

How to forward remote ports using SSH?

The syntax for forwarding remote ports using the ssh CLI client is as follows.

ssh -R "$remote_listener:$client_side_listening_service" <other ssh options> $destination_server

The remote_listener and client_side_listening_service can be either a TCP port or Unix socket.

Applications of Remote Port forwarding

• To access services (like FTP and VNC) in the client’s network or an external network like the internet, that are otherwise blocked by firewalls to internal services.

Note that in a single invocation of the ssh client, multiple local and remote forwarding options may be configured.

Experimental setup

docker-compose setup

The diagram above shows the setup I have used for my experiments. The bastion server has one interface in each of the networks since it is used as an entry-point from the external to the internal network.

All four computers have the following services running.

• A TCP server running on port 80 which is the service that will be used by forwarded ports to connect to.
• A Unix socket at /var/run/mysock.sock which is another service that will be used by forwarded ports to connect to.
• The sshd daemon running on TCP port 22.

The client1 and client2 computers are identical in all aspects. For my experiments, client1 is considered as the primary client that connects to the SSH server and client2 is an extra host in the external network.

Create the dockerfile and docker-compose file

Steps for setting up the SSH service in docker can be found in the official Docker documentation to Dockerize an SSH service. The SSH server is configured to allow root user login with a password.

I added instructions to install some extra utilities in the Dockerfile in case I need to debug inside a container.

Each container has two users.

1. root user with password toor. The socket file’s access is restricted to only the root user.
2. user user with password resu.

The environment can be set up by running the following command.

docker-compose up --build --remove-orphans --force-recreate

Experiment 1: Test local port forwarding scenarios

Aim

1. Learn how local port forwarding behaves.
2. Observe the status of the network connections for different local port forwarding scenarios.

According to the manual for ssh, local port forwarding should be used as follows.

Source: http://manpages.ubuntu.com/manpages/xenial/man1/ssh.1.html

Design

Requirements

• The session always remains established between client1 and bastion for all scenarios.

Inputs

Remote services:

1. host = nil
   port = 8080
2. host = localhost = 127.0.0.1 (loopback interface)
   port = 80
3. host = bastion = 172.28.1.2 (ethernet interface)
   port = 80
(bastion may resolve to either one of its ethernet interface IPs)
4. host = 172.28.0.2 (another ethernet interface of bastion)
   port = 80
5. host = *
   port = 80
6. host = server = 172.28.0.3 (ethernet interface)
   port = 80
7. host = google.com
   port = 80
8. remote_socket = /var/run/mysock.sock

Local listeners:

1. bind_address = nil
   port = 8080
2. bind_address = localhost = 127.0.0.1 (loopback interface)
   port = 8080
3. bind_address = client1 = 172.28.1.4 (ethernet interface)
   port = 8080
4. bind_address = *
   port = 8080
5. bind_address = client2 = 172.28.1.5 (ethernet interface)
   port = 8080
6. local_socket = /tmp/listener.sock
7. bind_address = nil
   port = 99
8. bind_address = nil
   port = 0

Users to connect to the bastion:

1. root
2. user

Users to run the ssh command on the client:

1. root
2. user

Procedure

docker exec into each container to observe the status of network connections

1. To connect from client1 to bastion, exec into client1.
   docker exec -it client1 bash
2. Test a simple ssh connection to the bastion or client2.
   ssh bastion or ssh client2
3. To observe the state of network connections, exec into each container and run either of the following commands.
   watch netstat -na — does not allow scrollback
   while sleep 2; do echo; date; netstat -na; done — allows scrollback
4. Establish a backgrounded ssh connection tobastion with local port forwarding by running the following command on client1. Variables in angle brackets are placeholders.
   ssh -L <local_listener_options>:<remote_service_options> -f <user>@bastion sleep 60
   The session is backgrounded to allow connecting to the configured local listener using the same shell.
5. Construct scenarios based on inputs mentioned in the previous section, replacing them in the command above. For each scenario, connect to the local listener using nc, and note the observations.
   To connect to a TCP local listener, run nc localhost 8080 andnc client1 8080.
   To connect to a Unix socket local listener, run nc -U /tmp/listener.sock.

If you would like to try the above scenarios yourself, now is a good time to stop reading further and make your observations.

Important Results and Conclusions

Changing remote services

• The remote host cannot be nil. It results in the error connect failed: Name or service not known.
• When the remote host is localhost or any of the bastion’s ethernet interfaces, the service on the respective interface of the bastion receives forwarded traffic. When the host is server or google.com, traffic is forwarded to the service running on server or google.com respectively. To connect to google.com with nc, pipe printf "HEAD / HTTP/1.0\r\n\r\n" into its stdin.
• When the remote host is *, the service receives traffic on the loopback interface.
• Connecting to the remote socket fails when the user is user, since user does not have permissions to read the socket file.

Changing local listeners

• When bind_address is nil, the default interface for the listener is the loopback interface, i.e., after running ssh -L 8080:localhost:80 -f bastion sleep 60, nc localhost 8080 connects, but nc client1 8080 does not. If the -o GatewayPorts=yes option is passed to the ssh command, a nil bind_address creates a listener on all interfaces.
• When bind_address is localhost or client1, the local listener is available on the respective bind_address interface. When bind_address is * the listener is available on all interfaces.
• It is logically wrong to have a bind_address that is not on one of the machine’s interfaces, since it is impossible to create a listening service on another machine’s interface without physically being inside the other machine. Therefore, if bind_address is set to client2, port forwarding fails.

I added this case just to reaffirm my understanding of port forwarding. Since a remote service can be any service accessible from the server’s network, a user is likely to assume that the same is true for the local listener and that the listener can also be configured on any machine accessible from the client’s network.

• When the listener is a socket, reusing the socket results in a bind error such as Address already in use. To overcome this, use the option flag, -o StreamLocalBindUnlink=yes , in the ssh command.

The sshd_config manual states that appending the line, StreamLocalBindUnlink yes, to the file /etc/ssh/sshd_config on the server should also solve the problem. However, this did not work for me.

Also note that killing the sshd process on the bastion, to restart it, will stop the container. For any config changes to take effect, restart bastion container after killing the sshd process, or run a new sshd daemon at a different port using /usr/sbin/sshd -D -p 2222.

• The listener socket file created by the ssh process is owned by the user that ran ssh. Therefore, the command ssh -o StreamLocalBindUnlink=yes -L /tmp/listener.sock:localhost:80 bastion -f sleep 20 fails to connect to the local socket if run by user if the socket file was already previously created by root.
• The output of netstat -na shows multiple listening Unix sockets at the same path when the socket is reused within a short interval, and multiple TCP connections to the same port in the state TIME_WAIT and LISTEN when ports are reused within a short interval. These connections are automatically removed when the backgrounded ssh connection is killed.
• A non-privileged user cannot bind to a privileged port. Therefore, user on client1 cannot create a listener on port 99.
• Binding on port 0 is not allowed. It results in the error Bad local forwarding specification ‘0:localhost:80’.

Other observations

• Any combination of local listeners and remote services can be used, as mentioned in the ssh manual.

Experiment 2: Test remote port forwarding scenarios

Aim

1. Learn how remote port forwarding behaves.
2. Observe the status of the network connections for different remote port forwarding scenarios.

According to the manual for ssh, remote port forwarding should be used as follows.

Source: http://manpages.ubuntu.com/manpages/xenial/man1/ssh.1.html

Design

Requirements

• The session always remains established between client1 and bastion for all scenarios.

Inputs

Local services:

1. host = nil
   port = 8080
2. host = localhost = 127.0.0.1 (loopback interface)
   port = 80
3. host = client1 = 172.28.1.4 (ethernet interface)
   port = 80
4. host = *
   port = 80
5. host = client2 = 172.28.1.5 (ethernet interface)
   port = 80
6. host = google.com
   port = 80
7. local_socket = /var/run/mysock.sock

Remote listeners:

1. bind_address = nil
   port = 8080
2. bind_address = localhost = 127.0.0.1 (loopback interface)
   port = 8080
3. bind_address = bastion = 172.28.1.2 (ethernet interface)
   port = 8080
4. bind_address = bastion = 172.28.0.2 (another ethernet interface of bastion)
   port = 8080
5. bind_address = *
   port = 8080
6. bind_address = server = 172.28.0.3 (ethernet interface)
   port = 8080
7. remote_socket = /tmp/listener.sock
8. bind_address = nil
   port = 99
9. bind_address = nil
   port = 0

Users to connect to the bastion:

1. root
2. user

Users to run the ssh command on the client:

1. root
2. user

Procedure

1. Follow steps 1 to 3 from the procedure of experiment 1.
2. Establish an ssh connection tobastion with remote port forwarding by running the following command on client1. Variables in angle brackets are placeholders.
   ssh -R <remote_listener_options>:<local_service_options> <user>@bastion
The session is not backgrounded now since the listener is on the server, and can be tested from within the ssh session.
3. Construct scenarios based on inputs mentioned in the previous section, replacing them in the command above. For each scenario, connect to the listener from the bastion using nc.
   To connect to a TCP local listener, run nc $bastion_ip 8080.
   To connect to a Unix socket local listener, run nc -U /tmp/listener.sock.

If you would like to try the above scenarios yourself, now is a good time to stop reading further and make your observations.

Important Results and Conclusions

Changing local services

• The host for the client-side service cannot be nil. It results in the error connect failed: Name or service not known.
• When the client-side remote host (remote to the server) is localhost, or any ethernet interface IP of client1, client2 or google.com, the server’s traffic is forwarded to the respective host.
• When the host is *, the service receives traffic on the loopback interface.
• Connecting to the client-side socket, /var/run/mysock.sock, from the bastion, succeeds with both ssh users — root and user —provided that root user on client1 started the ssh process to connect to the bastion. If client1 logs in as user user to establish the remote port forwarding session with bastion, the socket will fail to connect with a Permission denied for any user logged into the bastion.

Changing remote listeners

• When bind_address is nil, localhost, any of the bastion’s ethernet interface IPs, server, or *, the listener is always on the bastion’s loopback interface. This is mentioned in the manual.
  TCP listening sockets on the server will be bound to the loopback interface only. Specifying a remote bind_address will only succeed if the server’s GatewayPorts option is enabled.
• When GatewayPorts yes is set in the /etc/ssh/sshd_config of the bastion (note that config changes require reloading the sshd daemon or starting a new service at a different port than 22), it forces all remote port forwardings to bind to the wildcard address i.e., the configuration of bind_address is ignored and the listener is available on all interfaces at the configured port.
• When GatewayPorts clientspecified is set in the /etc/ssh/sshd_config of the bastion, nil bind_address defaults to localhost, bastion bind_address allows the listener to be set up on both its ethernet interfaces, any specific bastion IP creates the listener on the respective interface, and * defaults to the wildcard address.
• It is illogical to create the listener on a different machine, so bind_address cannot be set to server.
• When the listener is a socket, reusing the socket results in an error. Setting the option flag, -o StreamLocalBindUnlink=yes in the ssh command, like in the case of local port forwarding, does not fix the issue. Appending the line, StreamLocalBindUnlink yes to the file /etc/ssh/sshd_config on the bastion fixes it.
• The listener socket file is owned by the user that was used to connect. Therefore, the command ssh -R /tmp/listener.sock:localhost:80 user@bastion fails to connect to the socket if the socket file was already previously created by root.
• A non-privileged user cannot bind to a privileged port. Therefore, user on the bastion cannot have a listener on port 99.
• Binding to port 0 results in a dynamically allocated listening port on the bastion with a message similar to Allocated port 46307 for remote forward to localhost:80 printed on stdout.

Other observations

• Any combination of remote listeners and local services can be used, as mentioned in the ssh manual.

Experiment 3: Forward ports over jump servers

Aim

1. Learn if and how port forwarding works over jump host sessions by understanding the network connections made.
2. Compare forwarding from a bastion to forwarding directly from a server (which could either be the bastion itself or a server connected to the bastion).

Design

Requirements

• The session should be established between client1 and server (via the bastion).

Test cases

• Local port forwarding
  * listen on TCP port — forward to TCP port
  * listen on TCP port — forward to remote socket
  * listen on local socket — forward to TCP port
  * listen on local socket — forward to remote socket
• Remote port forwarding
  * listen on TCP port — forward to TCP port
  * listen on TCP port — forward to remote socket
  * listen on local socket — forward to TCP port
  * listen on local socket — forward to remote socket

Procedure

Follow the procedure in experiment 1 for local port forwarding and experiment 2 for remote port forwarding. Connect to server via the bastion as follows.

ssh -J $USER1@bastion $USER2@server

If you would like to try it out yourself, now is a good time to stop reading further and make your observations.

Results

Case 1: Local port forwarding with listener and service on TCP ports.

ssh -J bastion -L 8080:localhost:80 -f server sleep 10

Case 2: Local port forwarding with the listener on TCP port, remote service on a socket.

ssh -J bastion -L 8080:/var/run/mysock.sock -f server sleep 10

Case 3: Local port forwarding with the listener on a socket, remote service on TCP port.

ssh -J bastion -L /tmp/listener.sock:localhost:80 -f server sleep 10

Case 4: Local port forwarding with listener and service on sockets.

ssh -J bastion -L /tmp/listener.sock:/var/run/mysock.sock -f server sleep 10

Case 5: Remote port forwarding with listener and service on TCP ports.

ssh -J bastion -R 8080:localhost:80 server

Case 6: Remote port forwarding with the listener on TCP port, service on a socket.

ssh -J bastion -R 8080:/var/run/mysock.sock server

Case 7: Remote port forwarding with the listener on a socket, service on TCP port.

ssh -J bastion -R /tmp/listener.sock:localhost:80 server

Case 8: Remote port forwarding with listener and service on sockets.

ssh -J bastion -R /tmp/listener.sock:/var/run/mysock.sock server

Conclusions

• Port forwarding over jump hosts works.
• In all the cases, traffic is forwarded from the end server instead of from the bastion.
• The SSH traffic itself is forwarded from the bastion to the server in the bastion’s network. This can be observed from the netstat output on the bastion.
• It is more efficient to forward local ports by connecting to the bastion, rather than other servers in the bastion’s network, to avoid extra ssh connections.
• To forward remote server ports, connecting to the server directly via jump hosts is necessary since the listeners are created on the server. An alternative to doing this can be to configure remote port forwarding on the bastion’s ethernet interface.

Do your observations match with mine? Did you have any more interesting new observations or experiments to try? Let me know in the comments.

References

• https://docstore.mik.ua/orelly/networking_2ndEd/ssh/ch09_02.htm