File transfer

File transfer using ssh-based file transfer protocols

Copying files to/from the cluster: scp

From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp (secure copy). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:

Syntax of the scp command requires from-to order:

<source lang='bash'> scp <source> <destination> </source>

<source lang='bash'> scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/ </source>

This example assumes a user that is part of the ABGC user group. See the Lustre Parallel File System layout page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the man pages.

<source lang='bash'> man scp </source>

rsync

The rsync protocol, like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order. <source lang='bash'> rsync <source> <destination> </source> e.g.: <source lang='bash'> rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/ </source> The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the man pages. <source lang='bash'> man scp </source>

WinSCP

WinSCP is a free and open source (S)FTP client for Microsoft Windows. By providing the hostname (login.anunna.wur.nl), your username, and password, using SFTP protocol and port 22, you can login. After login files can be transferred between a local system (PC) and the cluster.

FileZilla

FileZilla is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, password and server type (Unix, see Site Manager;Advanced), files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on Anunna. Detailed instruction can be found on the FileZilla Wiki.

Samba/CIFS based protocols

The Common Interface File System (CIFS) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet.

There are two mount points that are available

1. your home folder ( \\cifs.anunna.wur.nl\[username] )
2. the Lustre mount ( \\cifs.anunna.wur.nl\lustre )

You can enter these in the location bar of File Explorer.

rclone to OneDrive

To easily transfer data to OneDrive, one can use rclone. It is usable through the modules system, so you can access it through <source lang='bash'> module load rclone </source>

To be able to access your own OneDrive space, you will need to configure rclone. In one of the steps, rclone starts a webserver, so we will have to use SSH to create a tunnel to the login server, so we can access it on your computer.

So first connect to anunna, with the tunnel: <source lang='bash'> ssh user@login.anunna.wur.nl -L53682:127.0.0.1:53682 </source>

Then load the rclone module (see above), and start the configure process by entering <source lang='bash'> rclone config </source>

You will then have to create an new config (I will use remote as name, but you can choose what you want), and select onedrive as the type. Use defaults for next few steps, and choose global for the region. Don't start the advanced config, and do use auto config. Copy the URL to your local web browser, and enter your WUR credentials when asked. In the next steps, you will have to select onedrive, the WUR uses the Business version. If all is well, you now have a working config.

To test. do this: <source lang='bash'> rclone tree remote: </source>

If all is well, you see the content of your own OneDrive. Now you can create folders, and copy data to and from your OneDrive: <source lang='bash'> rclone copy --create-empty-src-dirs --copy-links --progress bin/ remote:test </source> As soon as your are done, please remove your config: <source lang='bash'> rclone config delete remote </source> Anunna is relatively safe, but better safe than sorry.

See also

• Log in to Anunna
• ssh without password

External links

• WinSCP homepage
• FileZilla homepage
• The Common Interface File System (CIFS) on Wikipedia