File transfer

File transfer using ssh-based file transfer protocols

Copying files to/from the cluster: scp

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp (secure copy). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

scp <source> <destination>

scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/

This example assumes a user that is part of the ABGC user group. See the Lustre Parallel File System layout page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the man pages.

man scp

rsync

The rsync protocol, like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.

rsync <source> <destination>

e.g.:

rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/

The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the man pages.

man scp

WinSCP

WinSCP is a free and open source (S)FTP client for Microsoft Windows. By providing the hostname (login.anunna.wur.nl), your username, and password, using SFTP protocol and port 22, you can login. After login files can be transferred between a local system (PC) and the cluster.

FileZilla

FileZilla is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, password and server type (Unix, see Site Manager;Advanced), files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on Anunna. Detailed instruction can be found on the FileZilla Wiki.

Samba/CIFS based protocols

The Common Interface File System (CIFS) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet or when connected through the VPN.

There are two mount points that are available

1. your home folder ( \\cifs.anunna.wur.nl\[username] )
2. the Lustre mount ( \\cifs.anunna.wur.nl\lustre )

You can enter these in the location bar of File Explorer.

If you have issues with logging in, the issue is probably your username. In that case, there are a couple of options you could try:

1. Use '\WUR\[userid]' as username
2. Use '[userid]' as username
3. Use '[WUR e-mail]' as username

All these can be entered if you select "connect using different credentials" in the connection box.

rclone to OneDrive

To easily transfer data to OneDrive, one can use rclone. It is usable through the modules system, so you can access it through

module load rclone

To be able to access your own OneDrive space, you will need to configure rclone. In one of the steps, rclone starts a webserver, so we will first have to use SSH to create a tunnel to the login server, so we can access the webserver on your own computer.

So first connect to anunna, with the tunnel:

ssh user@login.anunna.wur.nl -L53682:127.0.0.1:53682

• For Windows PUTTY users, please refer to: Putty SSH Tunnelling

Then load the rclone module (see above), and start the configure process by entering

rclone config

You will then have to create an new config (I will use remote as name, but you can choose what you want), and select onedrive as the type. Use defaults for next few steps, and choose global for the region. Don't start the advanced config, and do use auto config. Copy the URL to your local web browser, and enter your WUR credentials when asked. In the next steps, you will have to select onedrive, the WUR uses the Business version. If all is well, you now have a working config.

To test. do this:

rclone tree remote:

If all is well, you see the content of your own OneDrive. Now you can create folders, and copy data to and from your OneDrive:

rclone copy --create-empty-src-dirs --copy-links --progress bin/ remote:test

As soon as your are done, please remove your config:

rclone config delete remote

Anunna is relatively safe, but better safe than sorry.

See also

• Log in to Anunna
• ssh without password

External links

• WinSCP homepage
• FileZilla homepage
• The Common Interface File System (CIFS) on Wikipedia
• Info on adding Microsoft OneDrive from the rclone website