Shared folders

Working with shared folders in the Lustre file system

If you work in a group or team and use large volumes of data, it is useful to work within a shared space. User can thus share inputs to their models and make their outputs also easily available. This article explains how to do so within the Lustre file system, that presently supports the HPC.

There are two main methods available to you: Access Control List (ACL) access (that you can administer yourself) or group access (that is centrally administered).

ACL shared directories

You may create a folder that can be accessed by yourself and someone else in the following manner:

<source lang='bash'> cd /lustre/shared mkdir shared_folder chmod 700 shared_folder setfacl -R -m u:my_id:rwx shared_folder setfacl -R -d -m u:my_id:rwx shared_folder </source>

Then, for each person who you want to have access to this: <source lang='bash'> setfacl -R -m u:my_friend:rwx shared_folder setfacl -R -d -m u:my_friend:rwx shared_folder </source>

Adding users later can be done using the same method, but it's messy. You may have trouble updating ACLs on files you don't own, though, and you cannot chown files to yourself. Each user with files will need to update their ACLs appropriately, or you can contact your sysadmin to assist.

Group shared directories

Users access the HPC cluster with their WUR-wide account. This means that all the membership information is also available in the HPC system. To check of which groups is your user a member of, use the following command:

groups <username>

This can result in a rather long list, reflecting permissions in the overall WUR systems. Within these groups you must then identify the one that is closer to match the team or group with which you wish to collaborate.

For instance, if I wish to work together with colleagues at ISRIC, I can search within my groups an appropriate match:

groups duque004 | grep isric

In my case the group des-isric-users looked appropriate. Then next step is to confirm if the other users in my team are also members of the group.

Creating a shared folder with correct permissions

The Lustre file system is accessible in the /lustre folder and then divided into the /backup and /nobackup sections (corresponding to the different usage plans). Inside each of these folders there is a sub-folder named SHARED in which users are to create their own assets.

You start by creating a folder in this space; it is probably better if it matches the name of your group or team, e.g.:

mkdir /lustre/nobackup/SHARED/myTeamWorkspace

Or in alternative:

cd /lustre/nobackup/SHARED

mkdir myTeamWorkspace

Setting permissions

Three basic steps are involved in stepping permissions correctly:

1. Pass the ownership of the group to the team. In the example below it is applied recursively to all sub-folder and files that may exist:

chgrp -R my-team-group myTeamWorkspace

2. Concede read/write permissions to the group. This allows other members of the group to read and write in the shared folder. If you wish other team members to only read from the folder then remove the w character from the +rw bit:

chmod -R g+rw myTeamWorkspace

3. Set default ownership within the group. This guarantees that any new files or folders created within the shared folder are owned by default owned by your team group:

chmod -R g+s myTeamWorkspace

In case the contents of the shared are sensitive or private, and should be accessed by your team, you can block access from any other users with the following command:

chmod -R o-rw myTeamWorkspace

Further reading

An Introduction to Linux Permissions

Linux Users and Groups

Linux Permissions Basics and How to Use Umask on a VPS

Linux Tutorial - Managing Group Access on Linux and UNIX