Best Practices
Environment
- Do not modify your ~/.bashrc or ~/.bash_profile
- If you need to create aliases or environment variables either setup your own module or edit your ~/.bash_aliases
- Setup environment variables with your lustre folder locations ( see HPC Advanced/Preparation Exercise part 1)
Login node
The login node is meant to be a staging area for your computations, whatever work outside of editing files should be done in a job in the compute node.
shared filesystems (/home)
Shared Filesystems (/lustre)
You have a quota of 200GB in you home directory,
- Do not store datasets or large files in your home directory. You have a quota of 200 GB
- Do not install software that you intend to run in jobs in your $HOME . Install your software in lustre.
- Do not point the logs of your jobs into your $HOME, specially when running array jobs.
- Do not edit or let software edit your ~/.bashrc or ~/.bash_profile. If something goes wrong you will be locked out.
Shared Filesystems (/lustre)
Lustre is a parallel filesystem that excels are high read and write speeds but it is limited at handling metadata operations
- Avoid folder with lots of small files
- Use the compute node's /tmp directory or /dev/shm for metadata heavy operations
- jobs - general good practices about submitting jobs
- Be mindful when running array jobs that write to lustre. thousands of job are likely going to slow down lustre and consequently your jobs. Limit the number of active jobs to a couple of hundred
- Control where the log and error output locations of your job and make sure they do not point to your home folder
Educational Courses
Request assistance for your course a soon as possible, at least with a month in advance, this leaves room for customisation and testing
Do not modify your student's ~/.bashrc or ~/.bash_profile file, any environment modifications should be done via modules