SSH into server with broken or missing shell by Josh Sherman


I did the unthinkable the other day. I attempted to upgrade my Debian home server (sherver) from Debian 10 to 11. In doing so, I fat fingered something in /etc/apt/sources.list. I then proceeded to run the usual update and upgrade commands, not paying any attention to any output. Then I rebooted the box.

Upon rebooting the box, a short stack Intel NUC, I was greeted with an error about my private key not being valid or accepted. Adding in -vvv showed more of the same.

I was locked out.

At this point, I jumped to the conclusion that I had borked something during the upgrade and that caused the lock out. I figured the worst case scenario would be that the system would need to be rebooted with a rescue disk and sorted out that say.

Since it did appear that everything was running correctly on it, I waited a few days before digging out a keyboard and hooking the machine up to a monitor.

Fast forward a few days later, I plugged in an HDMI cable and keyboard, and attempted to log in.

Logging in with my user account didn’t work. A message flashed on the screen and a prompt was presented again. I was able to log into the root account, fortunately. Said root account could not be utilized over SSH as I keep
PermitRootLogin set to no on my entire server fleet.

Cool, so I’m logged into the machine now. When attempting to switch to my normal user account, I was able to see the error in all its glory. Something about
zsh being missing.

Fantastic, let’s just reinstall that and we should be good to go!

Not quite, turns out during the upgrade process, there was a mismatch between a required lib and the zsh version which resulted in zsh being removed. After a bit of troubleshooting, I tracked down where I had an error, got that sorted and was able to complete the upgrade and get zsh back.

Now I know what you’re thinking, “hey Josh, when are you going to tell me how to SSH into a server with a broken or missing shell?”

Simply put, you cannot.

I did some digging after the fact, and there’s no way to pass in a different shell to run in this scenario. If there were, things like /usr/sbin/nologin
would be rendered useless, wouldn’t they?

So that’s the hard truth of it, in a scenario where your shell is missing or broken or set to something like nologin that’s it, you can’t log in.

Fortunately for me, I had physical access to the server in question, but that’s not always the case with remote servers and cloud computing.

In those scenarios, you need to make sure you’re being more mindful of what you’re doing. Keeping an active root shell logged in while you’re running upgrades is a good start.

With an active session going, you can finish your upgrade and then attempt to log in with your normal user. If things are messed up, no big deal, just switch back to yours root session and attempt to sort things out!

Good stuff? Want more?

Weekly emails about technology, development, and sometimes sauerkraut.

100% Fresh, Grade A Content, Never Spam.



Please enter your comment!
Please enter your name here