'rm -rf' sftp python script

problem

Why does SFTP not allow for a simple rm -rf?

It’s a rhetorical question because there’s probably a reason.

But that doesn’t help me, because I need to be able to clear out and remove directories when updating this blog over SSH File Transfer Protocol and doing so manually is tedious.

So the task is simple: write a script to clear out and remove a particular directory over SFTP à la rm -rf.

paramiko: ssh and sftp in Python

Lucky for me, there’s a “pure-Python…implementation of the SSHv2 protocol” in the form of the paramiko package.

So connecting to a server over SFTP is as easy as:

from paramiko import client
ssh = client.SSHClient()
ssh.set_missing_host_key_policy(client.WarningPolicy)
ssh.connect(username='username',
            hostname='hostname',
            port='port',
            key_filename='~/.ssh/key')
sftp = ssh.open_sftp()

That’s half the problem out of the way; now for the more interesting part.

recursively deleting files

SFTP does make available some basic directory manipulation commands like cd, ls, rm and rmdir. But the last 2 have some unexpected caveats: rm only works on files and not directories, which is why rmdir exists. And rmdir can only remove empty directories, meaning there is no way to delete a directory and its contents all in one go, i.e. our problem.

But these 4 commands are all we need and they’re all conveniently a part of our sftp client object’s methods:

The only thing to note is that .listdir() returns a list of SFTPAttributes objects. It’s relevant because we need to know if an element in a directory is a file or subdirectory to dictate which kind of remove to perform on it. And we can do such file typing by passing the st_mode attribute of an SFTPAttributes object to the stat package’s .S_ISDIR() method.

With that, we can implement our recursive deletion function. It takes an SFTP client object and directory/path as argument. It first changes to the passed directory/path before iterating through its contents either deleting what files it finds, or recursing to the directories it comes across. The only subtlety is that once it has iterated through all the contents in a directory it “moves up” to remove the directory it just cleared:

def sftp_rm_rf(sftp, directory: str):
    try:
        sftp.chdir(directory)
    except FileNotFoundError as e:
        print(f"Couldn't remove {director}: {e}")

    contents = sftp.listdir()
    for filename in contents:
        f_stat = sftp.stat(filename)
        if stat.S_ISDIR(f_stat.st_mode):
            sftp_rm_rf(sftp, filename)
        else:
            try:
                sftp.remove(filename)
            except PermissionError as e:
                print(f"Couldn't remove {directory}: {e}")
    sftp.chdir('..')
    try:
        sftp.rmdir(directory)
    except (PermissionError, OSError) as e:
        print(f"Couldn't remove {directory}: {e}")

And that’s it, problem solved!

as a script

Combining the SFTP connection and recursive file deletion in a single (very basic) script:

#!/usr/bin/python3

import os
from paramiko import client
import stat
import sys

def sftp_rm_rf(sftp, directory: str):
    try:
        sftp.chdir(directory)
    except FileNotFoundError as e:
        print(f"Couldn't remove {director}: {e}")

    contents = sftp.listdir()
    for filename in contents:
        f_stat = sftp.stat(filename)
        if stat.S_ISDIR(f_stat.st_mode):
            sftp_rm_rf(sftp, filename)
        else:
            try:
                sftp.remove(filename)
            except PermissionError as e:
                print(f"Couldn't remove {directory}: {e}")
    sftp.chdir('..')
    try:
        sftp.rmdir(directory)
    except (PermissionError, OSError) as e:
        print(f"Couldn't remove {directory}: {e}")

if __name__ == '__main__':
    # expeced format: username@hostname port key_filename
    args = sys.argv[1:]
    if len(args) != 4:
        raise ValueError(f"Expected 4 arguments of form 'username@hostname port key_filename directory', got: {''.join(args)}")

    user_host, port, key_filename, directory = args
    username, hostname = user_host.split('@')
    ssh_args = {'hostname': hostname,
                'username': username,
                'port': port,
                'key_filename': key_filename}
    ssh = client.SSHClient()
    ssh.set_missing_host_key_policy(client.WarningPolicy)
    ssh.connect(**ssh_args)
    sftp = ssh.open_sftp()
    sftp_rm_rf(sftp, directory)
    sftp.close()
    ssh.close()