File transfer: Difference between revisions
No edit summary |
m (<source> to <pre> to fix code blocks) |
||
| Line 6: | Line 6: | ||
Syntax of the scp command requires from-to order: | Syntax of the scp command requires from-to order: | ||
< | <pre> | ||
scp <source> <destination> | scp <source> <destination> | ||
</ | </pre> | ||
< | <pre> | ||
scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/ | scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/ | ||
</ | </pre> | ||
This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages]. | This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages]. | ||
< | <pre> | ||
man scp | man scp | ||
</ | </pre> | ||
=== rsync === | === rsync === | ||
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order. | The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order. | ||
< | <pre> | ||
rsync <source> <destination> | rsync <source> <destination> | ||
</ | </pre> | ||
e.g.: | e.g.: | ||
< | <pre> | ||
rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/ | rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/ | ||
</ | </pre> | ||
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages]. | The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages]. | ||
< | <pre> | ||
man scp | man scp | ||
</ | </pre> | ||
=== WinSCP === | === WinSCP === | ||
| Line 53: | Line 53: | ||
To easily transfer data to OneDrive, one can use '''''rclone'''''. | To easily transfer data to OneDrive, one can use '''''rclone'''''. | ||
It is usable through the modules system, so you can access it through | It is usable through the modules system, so you can access it through | ||
< | <pre> | ||
module load rclone | module load rclone | ||
</ | </pre> | ||
To be able to access your own OneDrive space, you will need to configure rclone. | To be able to access your own OneDrive space, you will need to configure rclone. | ||
| Line 61: | Line 61: | ||
So first connect to anunna, with the tunnel: | So first connect to anunna, with the tunnel: | ||
< | <pre> | ||
ssh user@login.anunna.wur.nl -L53682:127.0.0.1:53682 | ssh user@login.anunna.wur.nl -L53682:127.0.0.1:53682 | ||
</ | </pre> | ||
Then load the rclone module (see above), and start the configure process by entering | Then load the rclone module (see above), and start the configure process by entering | ||
< | <pre> | ||
rclone config | rclone config | ||
</ | </pre> | ||
You will then have to create an new config (I will use remote as name, but you can choose what you want), and select '''''onedrive''''' as the type. | You will then have to create an new config (I will use remote as name, but you can choose what you want), and select '''''onedrive''''' as the type. | ||
| Line 78: | Line 78: | ||
To test. do this: | To test. do this: | ||
< | <pre> | ||
rclone tree remote: | rclone tree remote: | ||
</ | </pre> | ||
If all is well, you see the content of your own OneDrive. | If all is well, you see the content of your own OneDrive. | ||
Now you can create folders, and copy data to and from your OneDrive: | Now you can create folders, and copy data to and from your OneDrive: | ||
< | <pre> | ||
rclone copy --create-empty-src-dirs --copy-links --progress bin/ remote:test | rclone copy --create-empty-src-dirs --copy-links --progress bin/ remote:test | ||
</ | </pre> | ||
As soon as your are done, please remove your config: | As soon as your are done, please remove your config: | ||
< | <pre> | ||
rclone config delete remote | rclone config delete remote | ||
</ | </pre> | ||
Anunna is relatively safe, but better safe than sorry. | Anunna is relatively safe, but better safe than sorry. | ||
Revision as of 15:44, 15 June 2023
File transfer using ssh-based file transfer protocols
Copying files to/from the cluster: scp
From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp (secure copy). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:
Syntax of the scp command requires from-to order:
scp <source> <destination>
scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
This example assumes a user that is part of the ABGC user group. See the Lustre Parallel File System layout page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the man pages.
man scp
rsync
The rsync protocol, like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
rsync <source> <destination>
e.g.:
rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the man pages.
man scp
WinSCP
WinSCP is a free and open source (S)FTP client for Microsoft Windows. By providing the hostname (login.anunna.wur.nl), your username, and password, using SFTP protocol and port 22, you can login. After login files can be transferred between a local system (PC) and the cluster.
FileZilla
FileZilla is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, password and server type (Unix, see Site Manager;Advanced), files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on Anunna. Detailed instruction can be found on the FileZilla Wiki.
Samba/CIFS based protocols
The Common Interface File System (CIFS) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet.
There are two mount points that are available
- your home folder ( \\cifs.anunna.wur.nl\[username] )
- the Lustre mount ( \\cifs.anunna.wur.nl\lustre )
You can enter these in the location bar of File Explorer.
rclone to OneDrive
To easily transfer data to OneDrive, one can use rclone. It is usable through the modules system, so you can access it through
module load rclone
To be able to access your own OneDrive space, you will need to configure rclone. In one of the steps, rclone starts a webserver, so we will have to use SSH to create a tunnel to the login server, so we can access it on your computer.
So first connect to anunna, with the tunnel:
ssh user@login.anunna.wur.nl -L53682:127.0.0.1:53682
Then load the rclone module (see above), and start the configure process by entering
rclone config
You will then have to create an new config (I will use remote as name, but you can choose what you want), and select onedrive as the type. Use defaults for next few steps, and choose global for the region. Don't start the advanced config, and do use auto config. Copy the URL to your local web browser, and enter your WUR credentials when asked. In the next steps, you will have to select onedrive, the WUR uses the Business version. If all is well, you now have a working config.
To test. do this:
rclone tree remote:
If all is well, you see the content of your own OneDrive. Now you can create folders, and copy data to and from your OneDrive:
rclone copy --create-empty-src-dirs --copy-links --progress bin/ remote:test
As soon as your are done, please remove your config:
rclone config delete remote
Anunna is relatively safe, but better safe than sorry.