A basic example would be to synchronize a local directory called “MyDocs” with a remote SSH server. From the following output, you can see that this directory contains four text files.

ls -ld ~/MyDocs
drwxr-xr-x 2 gmendoza gmendoza 4096 2009-04-09 16:05 /home/gmendoza/MyDocs

ls -l ~/MyDocs
total 12
-rw-r--r-- 1 gmendoza gmendoza 31 2009-04-09 16:09 file1.txt
-rw-r--r-- 1 gmendoza gmendoza 31 2009-04-09 16:09 file2.txt
-rw-r--r-- 1 gmendoza gmendoza 31 2009-04-09 16:09 file3.txt
-rw-r--r-- 1 gmendoza gmendoza 31 2009-04-09 16:09 file4.txt

The first time you run Unison for this particular directory structure, both sides will create a local index and hash table. You’ll get a warning and will be prompted with a message, asking you to hit the space bar if you accept. If the root directory on the remote side does not exist yet, you’ll also be prompted to accept the changes.

unison MyDocs ssh://host2/MyDocs
Contacting server...
Connected [//host1//home/gmendoza/MyDocs -> //host2//home/gmendoza/MyDocs]
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
/home/gmendoza/MyDocs
//host2//home/gmendoza/MyDocs
(snipped for brevity...)
Press return to continue.[] Waiting for changes from server
Reconciling changes

local host2
dir ----> / [f]

Proceed with propagating updates? [] y
Propagating updates

UNISON 2.27.57 started propagating changes at 16:14:30 on 09 Apr 2009
[BGN] Copying from /home/gmendoza/MyDocs to //host2//home/gmendoza/MyDocs
[END] Copying
UNISON 2.27.57 finished propagating changes at 16:14:30 on 09 Apr 2009

Saving synchronizer state
Synchronization complete (1 item transferred, 0 skipped, 0 failures)

Subsequent synchronizations are shown as the following.

unison MyDocs ssh://host2/MyDocs
Contacting server...
Connected [//host1//home/gmendoza/MyDocs -> //host2//home/gmendoza/MyDocs]
Looking for changes
Waiting for changes from server
Reconciling changes
Nothing to do: replicas have not changed since last sync.

For the following example, I have modified file1.txt on host1, and file2.txt on host2. Both file3.txt and file4.txt have been modified on each side. The great thing about unison is that when there is a conflict, you have the opportunity to view the differences and select which direction you wish to synchronize. Pressing the “x” key displays some basic information about the files that differ. In this case, I have chosen the files with the most recent timestamp. You choose the file direction by pressing the greater and less-than symbols, “>” and “<".

unison MyDocs ssh://host2/MyDocs
(snipped)
local host2
changed <-?-> changed file3.txt [] x
local : changed file modified on 2009-04-09 at 16:16:29 size 50
host2 : changed file modified on 2009-04-09 at 16:16:43 size 55
changed <==== changed file3.txt [] <
changed <-?-> changed file4.txt [] x
local : changed file modified on 2009-04-09 at 16:17:20 size 56
host2 : changed file modified on 2009-04-09 at 16:16:59 size 41
changed ====> changed file4.txt [] >
changed ----> file1.txt [f]
<---- changed file2.txt [f]

Proceed with propagating updates? [] y
Propagating updates

UNISON 2.27.57 started propagating changes at 16:18:27 on 09 Apr 2009
[BGN] Updating file file3.txt from //host2//home/gmendoza/MyDocs to /home/gmendoza/MyDocs
[BGN] Updating file file4.txt from /home/gmendoza/MyDocs to //host2//home/gmendoza/MyDocs
[BGN] Updating file file1.txt from /home/gmendoza/MyDocs to //host2//home/gmendoza/MyDocs
[BGN] Updating file file2.txt from //host2//home/gmendoza/MyDocs to /home/gmendoza/MyDocs
[END] Updating file file3.txt
[END] Updating file file2.txt
[END] Updating file file4.txt
[END] Updating file file1.txt
UNISON 2.27.57 finished propagating changes at 16:18:27 on 09 Apr 2009

Saving synchronizer state
Synchronization complete (4 items transferred, 0 skipped, 0 failures)

Unison also has a GTK front end for the graphically inclined. Be sure to check out all the documentation for a full understanding of syntax.

Related posts:

1. File Synchronization with Rsync over SSH
2. Backup Delivery via SSH

I use rsync the most for synchronizing my “Music” and “Documents” folders between a number of my systems at home and at work. All of these systems have these folders in the root of my home directory.

ls -ld ~/Music ~/Documents
drwxr-xr-x 16 gmendoza gmendoza 4096 2009-04-06 23:23 /home/gmendoza/Documents
drwxr-xr-x  9 gmendoza gmendoza 4096 2009-04-06 23:23 /home/gmendoza/Music

To push my recent changes from my local system (host1) to my remote system called (host2), I use the following commands.

rsync -avPe ssh ~/Music host2:~/
rsync -avPe ssh ~/Documents host2:~/

Notice, “Music” and “Documents” are specified without a trailing “/”, e.g. “Music/” or “Documents/”. This is important, because otherwise, it would copy only the contents of the folder to the remote home directory, and not the folder itself, which is described in more detail in the rsync man page.

Instead of running the above commands twice, you can also specify multiple files all in a single line.

rsync -avPe ssh ~/Music ~/Documents host2:~/

To synchronize changes made on the remote system to my local system, just reverse the commands. Notice the periods at the end of the line, which specifies the destination as the local working directory. Also, instead of wasting space by entering the host twice, you can use standard syntax to specify ranges or sets of files. In this case, I use curly brackets to specify the two directories on the remote host that share the same parent directory should be copied to my local working directory.

rsync -avPe ssh host2:~/{Music,Documents} .

I’ll also use the “delete” option to remove any files and folders the have been removed from the source system.

rsync --delete -avPe ssh ~/Music host2:~/

By default, rsync compares files extremely fast using a “quick check” algorithm based on the file size or in the last modified time (per the rsync man page). While I was updating my Music collection, I noticed that rsync was not detecting my ID3 tag modifications. By using the “-c” option, rsync will compare files using a 128 bit MD4 checksum as a more definitive change detection method. While this will slow the process down significantly, there’s obvious accuracy benefits in using the checksum method.

rsync -acvPe ssh Music host2:~/

Also, as you may have noticed rsync is strictly a unidirectional utility. This means that it only sends or receives data in a single direction, and it will clobber or delete any file or folder with the same name in the direction your are sending the data. For a great bidirectional utility, check out unison, which I will cover in an upcoming article.

Related posts:

1. File Synchronization with Unison over SSH
2. Backup Delivery via SSH

DenyHosts is a project that adds a protective layer to an SSH server by automatically blocking malicious hosts that use brute force or dictionary attacks. If you have SSH services enabled and accessible from the internet, you will likely have thousands of failed login attempts from several sources within a very short period of time. DenyHosts monitors all login attempts, and based on a customizable rule-set can block hosts from making further connections if an attack pattern is matched.

Using tcp_wrappers, the DenyHosts service elegantly manages entries in the /etc/hosts.deny file, adding and removing hosts when thresholds are crossed. e.g. Three failed logins with unknown user accounts; Three failed logins with root account; Five failed logins with known user accounts; Unblock host after a set period of time; etc. You can also specify whether DenyHosts blocks access to SSH or ALL services, thereby mitigating any other attack vectors the offender might try next.

A most valuable feature that makes DenyHosts even more attractive is the optional centralized reporting system. The service can be configured to report all abusive hosts to the DenyHosts collection server, and automatically import a list of IP addresses that others have reported. This network of intelligence gathering and incident response helps to thwart a large number of attacks before they happen, because the attackers (most of which are automated bots) are blocked before they have a chance to move on to other protected servers.Other useful features include email notification when hosts are blocked, and counter resets after successful authentication to prevent accidental blacklisting caused by fat fingered admins. :-)

For those of you using Ubuntu 7.04 (Feisty Fawn) and above, it is available in the Universe repository:

sudo apt-get install denyhosts

Edit and customize /etc/denyhosts.conf for your desired options, and restart the service:

sudo /etc/init.d/denyhosts restart

Ubuntu 6.06.1 LTS will need a manual installation, as it is not included in the repositories.

Be sure to check out the project at http://denyhosts.sourceforge.net.

Related posts:

1. fwknop: Single Packet Authorization in Ubuntu
2. VirtualBox Wireless Bridging with DHCP
3. VLAN Hopping on Cisco Voice-enabled Switch Ports

All you need is external access to a trusted OpenSSH server, perhaps the one you have at home, work, etc. If you’re using your laptop to surf the internet at your local coffee shop, you’ll simply need to establish a connection to that external SSH server using the appropriate client variables, and configure your web browser’s proxy settings to connect to a locally defined TCP port.

From a Linux terminal session on your local computer, run the following command:

$ ssh -ND 8080 user@server.yourdomain.com

You will be prompted for a password as you would normally expect when connecting to your server. After you make a successful connection, the session does not execute your default shell, and the command will remain silent until you close the console or terminate the process.

The “-D 8080″ option opens TCP port 8080 on your local client, which you will use as the SOCKS proxy port to tunnel your connections through. This can be verified via netstat:

$ netstat -an | grep 8080
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN

Next, you change your browser (or any other application) proxy settings to connect to your localhost (127.0.0.1) port 8080. For example, here is a screenshot of the appropriate Firefox settings.

You should also make sure that your web browser is using the proxy for DNS queries. This is important because even if people cannot “see” the content of your browser traffic, they would still be able to get a feel for what your are doing by your DNS lookups. There is also a risk of a malicious user feeding you dangerous DNS answers which could send you in the wrong direction leaving you open to man in the middle attacks.

In firefox, open “about:config” in your location bar, and change the value of “network.proxy.socks_remote_dns” to “true”.

You are now ready to surf securely.

No related posts.

In the following examples, I’ll show you three ways to ship an archived folder to a remote SSH server.

Method 1: Secure Copy
Using ‘scp’, (secure copy), one can take any existing file and deliver it to an SSH server. This means that you can create a backup, store it temporarily to your “local” file system, and copy the file across the network.

In this example, one backs up a folder in their home directory called “myfiles” using tar and gzip compression, and then copies the resulting archive using scp to a folder called /archives on a remote SSH server.

$ tar -czvpf myfiles.tar.gz ~/myfiles
$ scp myfiles.tar.gz user@sshserver:/archives/
$ rm myfiles.tar.gz

Cool stuff, but the downside is two-fold:
(1) If your backup is larger than the available space on your local file system, this method obviously won’t work;
(2) If your backup is large, the entire process takes a little longer than you might find convenient, since you have to first create the backup, and then copy it across the network.

A better solution would be to start sending the backup during the file creation process, which leads us to to the next two methods.

Method 2: Concatenate to SSH
SSH can read from STDIN and print results to STDOUT, which means one can concatenate any type of “input” to a remote SSH server. For example, you could redirect the output of ‘tar’ using the following syntax:

$ tar czpvf - ~/myfiles | ssh user@sshserver "cat > /archives/myfiles.tar.gz"

As you can see, with a single command, you can both create and deliver the backup at the same time. The backup process does not take up any space on the local file system. Wicked cool!

There is however yet another way to accomplish this task as shown in the next section.

Method 3: Write to an SSH File System (SSHFS)
For those of you not familiar with SSHFS, this is a file system client based on SFTP and FUSE. This client allows you to mount any remote SSH server to a local empty directory, just as you would with other devices like CD/ROM’s, floppies, usb sticks, etc. What’s also great about this client is that it requires no server side modification. It’s resource friendly, and sending data is just as fast as any other SSH file transfer.

In Ubuntu 7.04, the fuse kernel module and utilities are installed by default, and sshfs is available in the repositories.

Once you have sshfs installed and working, the following example mounts the remote “/archives” directory to the local “~/temp-mount” folder, and then places the backup directly in the mounted file system. The file is transported across the network during the write process.

$ mkdir ~/temp-mount
$ sshfs user@sshserver:/archives ~/temp-mount
$ tar -czvpf ~/temp-mount/myfiles.tar.gz ~/myfiles

To unmount the directory,

$ fusermount -u ~/mnt

Conclusion:
As you can see, using SSH for the delivery of your backups can make your life a whole lot easier. A suggested practice would be to use DSA/RSA public key authentication for making SSH connections. This way, you don’t have rely on passwords every time the SSH client is used, which makes sense when applying any of the above examples to an automated process such as Crontab, or At.

Related posts:

1. File Synchronization with Rsync over SSH
2. File Synchronization with Unison over SSH