<\/span><\/h3>\n\n\n\nR is CPU intensive and can allocate large amounts of RAM. This is particularly the case for parallel operations using non-shared memory, e.g., when using parallel::parLapply()<\/code> on a local PSOCK “cluster”. We therefore recommend a shared-memory approach, e.g., by setting up a local FORK “cluster” for use with parallel::parLapply()<\/code> or using parallel::mclapply()<\/code>. See package parallel<\/code>‘s documentation for more information on parallel operations in R.<\/p>\n\n\n\nStarting with R version 3.6.1, support for add-on packages requiring C++14 or higher has been added. However, this is not<\/strong> enabled by default. To start R with the support enabled, issue scl enable devtoolset-8 R<\/code> at the Bash prompt. To permanently enable the support in the Bash shell, add source scl_source enable devtoolset-8<\/code> to your ~\/.bashrc<\/code> file, log out and log in again.<\/p>\n\n\n\n<\/span>Other applications<\/span><\/h3>\n\n\n\nI\/O intensive applications or, e.g., repeated processing of large files during computations, may benefit if used under the scratch file system located at \/scratch\/tmp[^1]. This is due to the home (\/nfs\/home) and project folders (\/nfs\/projects) being situated on the NFS storage system, which runs slower (and generates a lot of backup traffic during nights). However, note that \/scratch\/tmp is intended for temporary<\/em> storage only and that files and folders older than five days will be deleted automatically.<\/p>\n\n\n\n<\/span>How to use the servers<\/span><\/h2>\n\n\n\n<\/span>Connection<\/span><\/h3>\n\n\n\nConnect\/logon to the servers is made through a secure shell (SSH) connection. Under MS Windows, the current recommendation is to use KiTTY as the SSH client together with the WinSCP file manager for drag-and-drop of files and folders. Under macOS, an SSH connection can be started in a terminal window.<\/p>\n\n\n\n
<\/span>File systems<\/span><\/h3>\n\n\n\n<\/span>Home folders<\/span><\/h4>\n\n\n\nHome folders are located under \/nfs\/home\/\\ and have quota limits of 100 GB per user. The home folders are shared<\/em> between the servers.<\/p>\n\n\n\n<\/span>Scratch<\/span><\/h4>\n\n\n\nScratch is a local file system intended for I\/O intensive temporary storage. Users need to create their own temporary folders under \/scratch\/tmp\/<username><\/code>.<\/p>\n\n\n\nFiles and folders on the scratch disk will be automatically deleted<\/em> when their modification time exceeds five days. The scratch file system has a quota limit of 500 GB per user.<\/p>\n\n\n\n<\/span>SASWORK<\/span><\/h4>\n\n\n\nSASWORK is a fast local temporary space intended for SAS and is therefore only available on Matrix. The \/saswork\/tmp<\/code> file system has a quota limit of 150 GB per user. For SAS users requiring more disk space, we recommend using the scratch disk (possibly in combination with SASWORK if necessary).<\/p>\n\n\n\n<\/span>STATATMP<\/span><\/h4>\n\n\n\nAnalogously to the above, STATATMP is a fast local temporary space intended for Stata.<\/p>\n\n\n\n
<\/span>Project folders<\/span><\/h4>\n\n\n\nProject folders are located under \/nfs\/projects and contain folders that are typically shared between several researchers working in cooperation. Project folders have quota limits between 10 GB-1 TB and are shared<\/em> between the servers.<\/p>\n\n\n\nSome projects, usually when exceeding 1 TB, have their own file systems directly under \/nfs\/<projectname><\/code>. Group access to specific project folders are granted by their respective owners.<\/p>\n\n\n\n<\/span>Windows drives Z: and P:<\/span><\/h4>\n\n\n\nThe MS Windows drives Z: and P: can be accessed via \/cifs\/Z<\/code> and \/cifs\/P<\/code>. However, in order to do so you may be required to initiate a so-called Kerberos ticket. This is done by evoking the UNIX command kinit<\/code>; if you have not setup GSSAPI integration in your favorite SSH client before logon to the server, provide your password when prompted. Kerberos tickets can be checked by issuing the UNIX command klist<\/code>.<\/p>\n\n\n\nHowever, do not work directly against these folders since the Kerberos ticket can expire during job execution causing Z: and P: to be inaccessible. Instead, copy the desired files and folders to the scratch disk (and back, if necessary, when the job is done).<\/p>\n\n\n\n
<\/span>File creation<\/span><\/h3>\n\n\n\nAll program and data files should be prepared on your local computer if possible, including testing and debugging with reduced data size.<\/p>\n\n\n\n
Please, do not experiment on the servers!<\/strong><\/p>\n\n\n\n<\/span>File transfer<\/span><\/h3>\n\n\n\nFiles can be transferred to or from the server via the secure file transfer protocol (SFTP). If desired, this can be done through a graphical user interface, e.g., WinSCP under MS Windows.<\/p>\n\n\n\n
Depending on your operating system and setup, command-line tools, like sftp and scp, can also be used.<\/p>\n\n\n\n
<\/span>Running programs<\/span><\/h2>\n\n\n\nOnce all data and command files have been copied to the server, the user needs to start their program from the server’s command line. This is done via the SSH protocol. Under MS Windows, we (again) recommend using KiTTY.<\/p>\n\n\n\n
<\/span>tmux<\/span><\/h3>\n\n\n\ntmux is a t<\/span><\/strong>erminal mu<\/span><\/strong>ltiplex<\/span><\/strong>er, i.e., a program that can be used to run applications interactively while being able to disconnect from the server and resume the session at another time. However, it is important that tmux sessions are ended when they are no longer useful, e.g., when the job has been finished.<\/p>\n\n\n\n- Start the tmux session via UNIX command
tmux<\/code>.<\/li>- Run your program.<\/li>
- Detach from tmux by pressing
Ctrl + b d<\/code>.<\/li>- Reconnect to tmux using
tmux attach<\/code>.<\/li>- To end a tmux session permanently press
Ctrl + d<\/code> while in tmux.<\/li>- Check if you have ongoing tmux sessions with
tmux ls<\/code>.<\/li><\/ol>\n\n\n\n<\/span>SLURM<\/span><\/h3>\n\n\n\nThe SLURM workload manager can submit jobs to a queue that performs the jobs on the Scalar nodes instead of interactively on the Vector node. The queue system will make sure to run the jobs without overloading the system running too many jobs simultaneously regarding CPU threads memory and ends jobs that are running to long time. The queue system does not know how many CPU threads your job eventually will use for the parallelization \/ subtasks so make sure to specify the number of CPU threads anticipated.<\/p>\n\n\n\n
Example to submit a job a simple one-task job: Create a shell script <filename>.sh with the code you want to execute. For example,<\/p>\n\n\n\n
#!\/bin\/bash\necho \"hello world\"\nsleep 60<\/code><\/pre>\n\n\n\nSubmit <filename>.sh to the queue using<\/p>\n\n\n\n
sbatch <filename>.sh<\/code><\/pre>\n\n\n\nThe default time and memory limits per job are currently set to 240 minutes and 4.8 GB, respectively. To submit a job that you know will use a maximum of 4 CPU threads and 10 GB of RAM and will be finished within max 8 hours (480 minutes)<\/p>\n\n\n\n
sbatch --ntasks 4 --time 480 --mem 10000 <filename>.sh<\/code><\/pre>\n\n\n\nTo submit jobs on specific SLURM nodes add --nodelist <nodename(s)><\/code>. Submitting jobs excluding specific SLURM nodes can be done by adding --exclude <nodename(s)><\/code>. For example,<\/p>\n\n\n\nsbatch --nodelist scalar3,scalar4 <filename>.sh<\/code><\/pre>\n\n\n\nwill run the jobs on the scalar3 and scalar4 nodes. To run interactively on one of the SLURM nodes use srun<\/code>. For example,<\/p>\n\n\n\nsrun --nodelist scalar4 --pty bash<\/code><\/pre>\n\n\n\nYou can also specify the sbatch information within the shellscript using example:<\/p>\n\n\n\n
#SBATCH --ntasks 4\n#SBATCH --time 480\n#SBATCH --mem 10000\n#SBATCH --nodelist scalar3,scalar4<\/code><\/pre>\n\n\n\nTo see state of the SLURM system run sinfo<\/code>.<\/p>\n\n\n\nTo see the SLURM job queue run squeue<\/code>.<\/p>\n\n\n\nTo see information about a job run scontrol show job <jobid><\/code>.<\/p>\n\n\n\nTo cancel running jobs use scancel <jobid><\/code>.<\/p>\n\n\n\nTo cancel all your jobs (running and pending) use scancel --user <username><\/code>.<\/p>\n\n\n\nThe Scalar nodes have access to the same NFS areas (\/nfs) as Vector. They also have a \/scratch\/tmp area but it is not shared with Vector’s \/scratch\/tmp. Hence, you need to create your own folder\/files there from within your sbatch script.<\/p>\n\n\n\n
If you use the SLURM queue you have to keep in mind that \/scratch\/tmp is local to each of the Scalar nodes.<\/p>\n\n\n\n
<\/span>Clean up<\/span><\/h3>\n\n\n\nTransfer the results to your local computer via, e.g., SFTP as above, and delete the files no longer needed on the server.<\/p>\n\n\n\n
For long-term storage of files, we recommend the MS Windows fileserver (P: or Z: drive). Ask MEB IT Support if more space is required on the fileserver.<\/p>\n\n\n\n
<\/span>Things you may find useful when working on the servers<\/span><\/h2>\n\n\n\n<\/span>Some UNIX commands<\/span><\/h3>\n\n\n\nhtop<\/code> displays all user processes (sorted by CPU usage by default). Press
h for help or q to quit.<\/p>\n\n\n\nPressing M (i.e., Shift + m<\/code> ) while in htop will sort by memory usage per process. Analogously, pressing P (i.e., Shift + p<\/code> ) will sort by CPU usage.<\/p>\n\n\n\nhtop --u <username><\/code> displays only processes belonging to \\ in the same way as above.<\/p>\n\n\n\nAlternatively, the older top<\/code> can be used instead of htop<\/code>. See below for an example running top.<\/p>\n\n\n\ndu --hs ~<\/code> will report how much space your home folder is occupying.<\/p>\n\n\n\ndu --hs \/<foldername>\/*<\/code> will report how much disk space files and folders occupy within \\.<\/p>\n\n\n\ndf --h<\/code> will report file system usage, e.g., how large and how full the