File transfer
File transfer using ssh-based file transfer protocols
Copying files to/from the cluster: scp
From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp (secure copy). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:
Syntax of the scp command requires from-to order:
scp <source> <destination>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
This example assumes a user that is part of the ABGC user group. See the Lustre Parallel File System layout page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the man pages.
man scp
rsync
The rsync protocol, like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
rsync <source> <destination>
e.g.:
rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the man pages.
man scp
WinSCP
WinSCP is a free and open source (S)FTP client for Microsoft Windows. By providing the hostname (login.anunna.wur.nl), your username, and password, using SFTP protocol and port 22, you can login. After login files can be transferred between a local system (PC) and the cluster.
FileZilla
FileZilla is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, password and server type (Unix, see Site Manager;Advanced), files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on Anunna. Detailed instruction can be found on the FileZilla Wiki.
Samba/CIFS based protocols
The Common Interface File System (CIFS) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet or when connected through the VPN.
There are two mount points that are available
- your home folder ( \\cifs.anunna.wur.nl\[username] )
- the Lustre mount ( \\cifs.anunna.wur.nl\lustre )
You can enter these in the location bar of File Explorer.
If you have issues with logging in, the issue is probably your username. In that case, there are a couple of options you could try:
- Use '\WUR\[userid]' as username
- Use '[userid]' as username
- Use '[WUR e-mail]' as username
All these can be entered if you select "connect using different credentials" in the connection box.
rclone to OneDrive
To easily transfer data to OneDrive, one can use rclone. It is usable through the modules system, so you can access it through
module load rclone
To be able to access your own OneDrive space, you will need to configure rclone. In one of the steps, rclone starts a webserver, so we will first have to use SSH to create a tunnel to the login server, so we can access the webserver on your own computer.
So first connect to anunna, with the tunnel:
ssh user@login.anunna.wur.nl -L53682:127.0.0.1:53682
- For Windows PUTTY users, please refer to: Putty SSH Tunnelling
Then load the rclone module (see above), and start the configure process by entering
rclone config
You will then have to create an new config (I will use remote as name, but you can choose what you want), and select onedrive as the type. Use defaults for next few steps, and choose global for the region. Don't start the advanced config, and do use auto config. Copy the URL to your local web browser, and enter your WUR credentials when asked. In the next steps, you will have to select onedrive, the WUR uses the Business version. If all is well, you now have a working config.
To test. do this:
rclone tree remote:
If all is well, you see the content of your own OneDrive. Now you can create folders, and copy data to and from your OneDrive:
rclone copy --create-empty-src-dirs --copy-links --progress bin/ remote:test
As soon as your are done, please remove your config:
rclone config delete remote
Anunna is relatively safe, but better safe than sorry.