Data Transfer Methods: Difference between revisions

From HPCwiki
Jump to navigation Jump to search
← Older edit
Haars0011 (talk | contribs)
Bureaucrats, Administrators
261 edits
VisualWikitext
Sun092 (talk | contribs)
37 edits
No edit summary
IA migration §3: polish (syntaxhighlight code, fix dead Lustre link + man rsync, neutralize first person, add lead/See also) (via update-page on MediaWiki MCP Server)
 
(5 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== File transfer using ssh-based file transfer protocols ==
There are several ways to move data between your own computer (or other systems) and Anunna. Pick the one that matches your operating system and workflow.
 
== File transfer over SSH ==
=== Copying files to/from the cluster: scp ===
=== Copying files to/from the cluster: scp ===


From any Posix-compliant system (Linux/MacOSX) terminal files and folder can be transferred to and from the cluster using an ssh-based file copying protocol called scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]). For instance, copying a folder containing several files from scomp1090/lx6 can be achieved like this:
From any POSIX-compliant system (Linux/macOS) terminal, files and folders can be transferred to and from the cluster using the SSH-based copy command scp ([http://en.wikipedia.org/wiki/Secure_copy secure copy]).


Syntax of the scp command requires from-to order:
The syntax of the scp command is in from-to order:


<pre>
<syntaxhighlight lang="bash">
scp <source> <destination>
scp <source> <destination>
</pre>
</syntaxhighlight>


<pre>
For example, copying a folder and its contents to the cluster:
 
<syntaxhighlight lang="bash">
scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
</pre>
</syntaxhighlight>


This example assumes a user that is part of the ABGC user group. See the [[Lustre_PFS_layout | Lustre Parallel File System layout]] page for further details. The -p flag will preserve the file metadata such as timestamps. The -r flag allows for recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
This example assumes a user in the ABGC group; see [[Compute Storage|the Lustre layout]] for how the <code>/lustre</code> paths are structured. The <code>-p</code> flag preserves file metadata such as timestamps, and <code>-r</code> allows recursive copying. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages]:


<pre>
<syntaxhighlight lang="bash">
man scp
man scp
</pre>
</syntaxhighlight>


=== rsync ===
=== rsync ===
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like the scp protocol, allow CLI-based copying of files. The rsync protocol, however, will only transfer those files between systems that have changed, i.e. it synchronises the files, hence the name. The rsync protocol is very well suited for making regular backups and file syncs between file systems. Like the scp command, syntax is in the from-to order.
The [http://en.wikipedia.org/wiki/Rsync rsync protocol], like scp, allows command-line copying of files. Unlike scp, it only transfers the files that have changed between the two systems — it synchronises them, hence the name — which makes it very well suited to regular backups and repeated syncs. Like scp, the syntax is in from-to order.
<pre>
 
<syntaxhighlight lang="bash">
rsync <source> <destination>
rsync <source> <destination>
</pre>
</syntaxhighlight>
e.g.:
 
<pre>
For example:
 
<syntaxhighlight lang="bash">
rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/
</pre>
</syntaxhighlight>
The -a flag will preserve file metadata and allows for recursive copying, amongst others. The -v flag provides verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages].
 
<pre>
The <code>-a</code> flag preserves file metadata and copies recursively, among other things, and <code>-v</code> gives verbose output. Further options can be found in the [http://en.wikipedia.org/wiki/Man_page man pages]:
man scp
 
</pre>
<syntaxhighlight lang="bash">
man rsync
</syntaxhighlight>


=== WinSCP ===
=== WinSCP ===
[http://en.wikipedia.org/wiki/WinSCP WinSCP] is a free and open source (S)FTP client for Microsoft Windows. By providing the hostname (login.anunna.wur.nl), your username, and password, using SFTP protocol and port 22, you can login. After login files can be transferred between a local system (PC) and the cluster.
[http://en.wikipedia.org/wiki/WinSCP WinSCP] is a free and open-source (S)FTP client for Microsoft Windows. By providing the hostname (<code>login.anunna.wur.nl</code>), your username and password, and using the SFTP protocol on port 22, you can log in and transfer files between your local PC and the cluster.


=== FileZilla ===
=== FileZilla ===
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open source graphical (S)FTP client. It is available for Linux, MacOSX, and Windows. By providing the address, username, password and server type (Unix, see Site Manager;Advanced), files can be transferred between a local system and the cluster. Furthermore, the graphical interface allows for easy browsing of files on Anunna. Detailed instruction can be found on the [https://wiki.filezilla-project.org/Using FileZilla Wiki].
[http://en.wikipedia.org/wiki/Filezilla FileZilla] is a free and open-source graphical (S)FTP client, available for Linux, macOS, and Windows. By providing the address, username, password, and server type (Unix — see Site Manager → Advanced), you can transfer files between a local system and the cluster. The graphical interface also makes it easy to browse files on Anunna. Detailed instructions can be found on the [https://wiki.filezilla-project.org/Using FileZilla wiki].


== Samba/CIFS based protocols ==
== Samba/CIFS based protocols ==
The Common Interface File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet or when connected through the VPN.
The Common Internet File System ([http://en.wikipedia.org/wiki/Cifs CIFS]) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet, or when connected through the VPN.


There are two mount points that are available  
There are two mount points available:
# your home folder ( '''\\cifs.anunna.wur.nl\[username]''' )  
# your home folder ( '''\\cifs.anunna.wur.nl\[username]''' )
# the Lustre mount ( '''\\cifs.anunna.wur.nl\lustre''' )
# the Lustre mount ( '''\\cifs.anunna.wur.nl\lustre''' )


You can enter these in the location bar of File Explorer.
You can enter these in the location bar of File Explorer.


If you have issues with logging in, the issue is probably your username. In that case, there are a couple of options you could try:
If you have issues with logging in, the problem is probably your username. In that case, there are a couple of options you could try:
# Use '\WUR\[userid]' as username
# Use '''\WUR\[userid]''' as username
# Use '[userid]' as username
# Use '''[userid]''' as username
# Use '[WUR e-mail]' as username
# Use '''[WUR e-mail]''' as username
 
All of these can be entered if you select "connect using different credentials" in the connection box.


All these can be entered if you select "connect using different credentials" in the connection box.
== rclone to OneDrive ==
== rclone to OneDrive ==


To easily transfer data to OneDrive, one can use '''''rclone'''''.
To easily transfer data to OneDrive, you can use '''rclone'''. It is available through the modules system:
It is usable through the modules system, so you can access it through
 
<pre>
<syntaxhighlight lang="bash">
module load rclone
module load rclone
</pre>
</syntaxhighlight>


To be able to access your own OneDrive space, you will need to configure rclone.
To access your own OneDrive space, you first need to configure rclone. In one of the steps, rclone starts a web server, so we first have to use SSH to create a tunnel to the login server so that we can reach that web server from our own computer.
In one of the steps, rclone starts a webserver, so we will '''first''' have to use SSH to create a tunnel to the login server, so we can access the webserver on your own computer.


So first connect to anunna, with the tunnel:
First, connect to Anunna with the tunnel:
<pre>
 
<syntaxhighlight lang="bash">
ssh user@login.anunna.wur.nl -L53682:127.0.0.1:53682
ssh user@login.anunna.wur.nl -L53682:127.0.0.1:53682
</pre>
</syntaxhighlight>
For Windows PUTTY users: https://wiki.anunna.wur.nl/index.php?title=Putty_SSH_Tunnelling&action=edit


Then load the rclone module (see above), and start the configure process by entering
* Windows PuTTY users: see [[Putty SSH Tunnelling]].
<pre>
 
Then load the rclone module (see above) and start the configuration:
 
<syntaxhighlight lang="bash">
rclone config
rclone config
</pre>
</syntaxhighlight>


You will then have to create an new config (I will use remote as name, but you can choose what you want), and select '''''onedrive''''' as the type.
Create a new config (we use the name <code>remote</code> here, but you can choose your own) and select '''onedrive''' as the type. Use the defaults for the next few steps, and choose '''global''' for the region. Do not start the advanced config, and do use auto config ("Use web browser to automatically authenticate rclone with remote?"). '''Copy the URL''' into your local web browser and enter your WUR credentials when asked. In the next steps select OneDrive; the WUR uses the Business version. If all is well, you now have a working config.
Use defaults for next few steps, and choose '''''global''''' for the region.
Don't start the advanced config, and do use auto config.
'''Copy the URL''' to your local web browser, and enter your WUR credentials when asked.
In the next steps, you will have to select onedrive, the WUR uses the Business version.
If all is well, you now have a working config.


To test. do this:
To test it:
<pre>
 
<syntaxhighlight lang="bash">
rclone tree remote:
rclone tree remote:
</pre>
</syntaxhighlight>


If all is well, you see the content of your own OneDrive.
If all is well, you see the contents of your own OneDrive. Now you can create folders and copy data to and from your OneDrive:
Now you can create folders, and copy data to and from your OneDrive:
 
<pre>
<syntaxhighlight lang="bash">
rclone copy --create-empty-src-dirs --copy-links --progress bin/ remote:test
rclone copy --create-empty-src-dirs --copy-links --progress bin/ remote:test
</pre>
</syntaxhighlight>
As soon as your are done, please remove your config:
 
<pre>
Another handy tool is <code>rclone ncdu</code>: https://rclone.org/commands/rclone_ncdu/
 
As soon as you are done, please remove your config:
 
<syntaxhighlight lang="bash">
rclone config delete remote
rclone config delete remote
</pre>
</syntaxhighlight>
 
Anunna is relatively safe, but better safe than sorry.
Anunna is relatively safe, but better safe than sorry.


== See also ==
== See also ==
* [[Log_in_to_Anunna | Log in to Anunna]]
* [[Login Nodes]]
* [[ssh_without_password | ssh without password]]
* [[Storage Systems Overview]]
* [[Log in to Anunna|Logging in over SSH]]
* [[ssh_without_password|SSH without password]]


== External links ==
== External links ==
* [http://winscp.net/eng/index.php WinSCP homepage]
* [http://winscp.net/eng/index.php WinSCP homepage]
* [https://filezilla-project.org FileZilla homepage]
* [https://filezilla-project.org FileZilla homepage]
* [http://en.wikipedia.org/wiki/Cifs The Common Interface File System (CIFS) on Wikipedia]
* [http://en.wikipedia.org/wiki/Cifs The Common Internet File System (CIFS) on Wikipedia]
* [https://rclone.org/onedrive/ Info on adding Microsoft OneDrive from the rclone website]
* [https://rclone.org/onedrive/ Adding Microsoft OneDrive — rclone documentation]

Latest revision as of 12:08, 18 June 2026

There are several ways to move data between your own computer (or other systems) and Anunna. Pick the one that matches your operating system and workflow.

File transfer over SSH

Copying files to/from the cluster: scp

From any POSIX-compliant system (Linux/macOS) terminal, files and folders can be transferred to and from the cluster using the SSH-based copy command scp (secure copy).

The syntax of the scp command is in from-to order: 

scp <source> <destination>

For example, copying a folder and its contents to the cluster: 

scp -pr /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/

This example assumes a user in the ABGC group; see the Lustre layout for how the /lustre paths are structured. The -p flag preserves file metadata such as timestamps, and -r allows recursive copying. Further options can be found in the man pages: 

man scp

rsync

The rsync protocol, like scp, allows command-line copying of files. Unlike scp, it only transfers the files that have changed between the two systems — it synchronises them, hence the name — which makes it very well suited to regular backups and repeated syncs. Like scp, the syntax is in from-to order. 

rsync <source> <destination>

For example: 

rsync -av /home/WUR/[username]/folder_to_transfer [username]@login.anunna.wur.nl:/lustre/scratch/WUR/ABGC/

The -a flag preserves file metadata and copies recursively, among other things, and -v gives verbose output. Further options can be found in the man pages: 

man rsync

WinSCP

WinSCP is a free and open-source (S)FTP client for Microsoft Windows. By providing the hostname (login.anunna.wur.nl), your username and password, and using the SFTP protocol on port 22, you can log in and transfer files between your local PC and the cluster.

FileZilla

FileZilla is a free and open-source graphical (S)FTP client, available for Linux, macOS, and Windows. By providing the address, username, password, and server type (Unix — see Site Manager → Advanced), you can transfer files between a local system and the cluster. The graphical interface also makes it easy to browse files on Anunna. Detailed instructions can be found on the FileZilla wiki.

Samba/CIFS based protocols

The Common Internet File System (CIFS) is commonly used in and between Windows systems for file sharing. It is only available to clients within WURnet, or when connected through the VPN.

There are two mount points available:

  1. your home folder ( \\cifs.anunna.wur.nl\[username] )
  2. the Lustre mount ( \\cifs.anunna.wur.nl\lustre )

You can enter these in the location bar of File Explorer.

If you have issues with logging in, the problem is probably your username. In that case, there are a couple of options you could try:

  1. Use \WUR\[userid] as username
  2. Use [userid] as username
  3. Use [WUR e-mail] as username

All of these can be entered if you select "connect using different credentials" in the connection box.

rclone to OneDrive

To easily transfer data to OneDrive, you can use rclone. It is available through the modules system: 

module load rclone

To access your own OneDrive space, you first need to configure rclone. In one of the steps, rclone starts a web server, so we first have to use SSH to create a tunnel to the login server so that we can reach that web server from our own computer.

First, connect to Anunna with the tunnel: 

ssh user@login.anunna.wur.nl -L53682:127.0.0.1:53682

Then load the rclone module (see above) and start the configuration: 

rclone config

Create a new config (we use the name remote here, but you can choose your own) and select onedrive as the type. Use the defaults for the next few steps, and choose global for the region. Do not start the advanced config, and do use auto config ("Use web browser to automatically authenticate rclone with remote?"). Copy the URL into your local web browser and enter your WUR credentials when asked. In the next steps select OneDrive; the WUR uses the Business version. If all is well, you now have a working config.

To test it: 

rclone tree remote:

If all is well, you see the contents of your own OneDrive. Now you can create folders and copy data to and from your OneDrive: 

rclone copy --create-empty-src-dirs --copy-links --progress bin/ remote:test

Another handy tool is rclone ncdu: https://rclone.org/commands/rclone_ncdu/

As soon as you are done, please remove your config: 

rclone config delete remote

Anunna is relatively safe, but better safe than sorry.

See also

Retrieved from "https://wiki.anunna.wur.nl/index.php?title=Data_Transfer_Methods&oldid=2824"