\n
Summary<\/h3>\n
<\/p>\n
- \n
- Definition<\/a><\/li>\n
- Authentication and databases<\/a><\/li>\n
- Slurm installation<\/a><\/li>\n
- Configure usage limit<\/a><\/li>\n
- Links<\/a><\/li>\n
- License<\/a><\/li>\n<\/ul>\n
\n<\/a><\/p>\n
Definition<\/h2>\n
Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters.<\/p>\n
https:\/\/slurm.schedmd.com\/<\/a><\/p>\n
\n<\/a><\/p>\n
Authentication and databases:<\/h2>\n
Create the user for munge and slurm:<\/h3>\n
Slurm and Munge require consistent UID and GID across every node in the cluster.
\nFor all the nodes, before you install Slurm or Munge:<\/p>\n$ export MUNGEUSER=1001\n$ groupadd -g $MUNGEUSER munge\n$ useradd -m -c "MUNGE Uid 'N' Gid Emporium" -d \/var\/lib\/munge -u $MUNGEUSER -g munge -s \/sbin\/nologin munge\n$ export SLURMUSER=1002\n$ groupadd -g $SLURMUSER slurm\n$ useradd -m -c "SLURM workload manager" -d \/var\/lib\/slurm -u $SLURMUSER -g slurm -s \/bin\/bash slurm<\/code><\/pre>\nMunge Installation for authentication:<\/h3>\n
$ yum install munge munge-libs munge-devel -y<\/code><\/pre>\nCreate a munge authentication key:<\/h4>\n
$ \/usr\/sbin\/create-munge-key<\/code><\/pre>\nCopy the munge authentication key on every node:<\/h4>\n
$ cp \/etc\/munge\/munge.key \/home\n$ cexec cp \/home\/munge.key \/etc\/munge <\/code><\/pre>\nSet the rights:<\/h4>\n
$ chown -R munge: \/etc\/munge\/ \/var\/log\/munge\/ \/var\/lib\/munge\/ \/run\/munge\/\n$ chmod 0700 \/etc\/munge\/ \/var\/log\/munge\/ \/var\/lib\/munge\/ \/run\/munge\/\n$ cexec chown -R munge: \/etc\/munge\/ \/var\/log\/munge\/ \/var\/lib\/munge\/ \/run\/munge\/\n$ cexec chmod 0700 \/etc\/munge\/ \/var\/log\/munge\/ \/var\/lib\/munge\/ \/run\/munge\/<\/code><\/pre>\nEnable and Start the munge service with:<\/h4>\n
$ systemctl enable munge\n$ systemctl start munge\n$ cexec systemctl enable munge\n$ cexec systemctl start munge<\/code><\/pre>\nTest munge from the master node:<\/h4>\n
$ munge -n | unmunge\n$ munge -n | ssh <somehost_in_cluster> unmunge<\/code><\/pre>\nMariadb installation and configuration<\/h3>\n
Install mariadb with the following command:<\/h4>\n
$ yum install mariadb-server -y<\/code><\/pre>\nActivate and start the mariadb service:<\/h4>\n
$ systemctl start mariadb\nsystemctl enable mariadb<\/code><\/pre>\nsecure the installation:<\/h4>\n
Launch the following command to set up the root password an secure mariadb:<\/p>\n
$ mysql_secure_installation{<\/code><\/pre>\nModify the innodb configuration:<\/h4>\n
Setting innodb_lock_wait_timeout,innodb_log_file_size and innodb_buffer_pool_size to larger values than the default is recommended.<\/p>\n
To do that, create a the file
\/etc\/my.cnf.d\/innodb.cnf<\/code> with the following lines:<\/p>\n[mysqld]\n innodb_buffer_pool_size=1024M\n innodb_log_file_size=64M\n innodb_lock_wait_timeout=900<\/code><\/pre>\nTo implement this change you have to shut down the database and move\/remove logfiles:<\/p>\n
$ systemctl stop mariadb\n mv \/var\/lib\/mysql\/ib_logfile? \/tmp\/\n systemctl start mariadb<\/code><\/pre>\n
\n<\/a><\/p>\n
Slurm installation:<\/h2>\n
Install the following prerequisites:<\/h3>\n
$ yum install openssl openssl-devel pam-devel rpmbuild numactl numactl-devel hwloc hwloc-devel lua lua-devel readline-devel rrdtool-devel ncurses-devel man2html libibmad libibumad -y<\/code><\/pre>\nRetrieve the tarball<\/h3>\n
$ wget https:\/\/download.schedmd.com\/slurm\/slurm-19.05.0.tar.bz2<\/code><\/pre>\nCreate the RPMs:<\/h3>\n
$ rpmbuild -ta slurm-19.05.0.tar.bz2<\/code><\/pre>\nRPMs are located in \/root\/rpmbuild\/RPMS\/x86_64\/<\/p>\n
Install slurm on master and nodes<\/h3>\n
In the RPMs'folder, launch the following command:<\/p>\n
$ yum --nogpgcheck localinstall slurm-*<\/code><\/pre>\nCreate and configure the slurm_acct_db database:<\/h3>\n
$ mysql -u root -p\n mysql> grant all on slurm_acct_db.* TO 'slurm'@'localhost' identified by 'some_pass' with grant option;\n mysql> create database slurm_acct_db;<\/code><\/pre>\nConfigure the slurm db backend:<\/h3>\n
Modify the
\/etc\/slurm\/slurmdbd.conf<\/code> with the following parameters:<\/p>\nAuthType=auth\/munge\n DbdAddr=192.168.1.250\n DbdHost=master0\n SlurmUser=slurm\n DebugLevel=4\n LogFile=\/var\/log\/slurm\/slurmdbd.log\n PidFile=\/var\/run\/slurmdbd.pid\n StorageType=accounting_storage\/mysql\n StorageHost=master0\n StoragePass=some_pass\n StorageUser=slurm\n StorageLoc=slurm_acct_db<\/code><\/pre>\nThen enable and start the slurmdbd service<\/p>\n
$ systemctl start slurmdbd\n$ systemctl enable slurmdbd\n$ systemctl status slurmdbd<\/code><\/pre>\nThis will populate the slurm_acct_db with tables<\/p>\n
Configuration file \/etc\/slurm\/slurm.conf:<\/h3>\n
using the command
lscpu<\/code> on each node to get processors' informations.<\/p>\nVisit http:\/\/slurm.schedmd.com\/configurator.easy.html<\/a> to make a configuration file for Slurm.<\/p>\n
Modify the following parameters in
\/etc\/slurm\/slurm.conf<\/code> to match with your cluster:<\/p>\nClusterName=IRD\n ControlMachine=master0\n ControlAddr=192.168.1.250\n SlurmUser=slurm\n AuthType=auth\/munge\n StateSaveLocation=\/var\/spool\/slurmd\n SlurmdSpoolDir=\/var\/spool\/slurmd\n SlurmctldLogFile=\/var\/log\/slurm\/slurmctld.log\n SlurmdDebug=3\n SlurmdLogFile=\/var\/log\/slurm\/slurmd.log\n AccountingStorageHost=master0\n AccountingStoragePass=3devslu!!\n AccountingStorageUser=slurm\n NodeName=node21 CPUs=16 Sockets=4 RealMemory=32004 CoresPerSocket=4 ThreadsPerCore=1 State=UNKNOWN\n PartitionName=r900 Nodes=node21 Default=YES MaxTime=INFINITE State=UP<\/code><\/pre>\nNow that the server node has the slurm.conf and slurmdbd.conf correctly filled, we need to send these filse to the other compute nodes.<\/p>\n
$ cp \/etc\/slurm\/slurm.conf \/home\n$ cp \/etc\/slurm\/slurmdbd.conf \/home\n$ cexec cp \/home\/slurm.conf \/etc\/slurm\n$ cexec cp \/home\/slurmdbd.conf \/etc\/slurm<\/code><\/pre>\nCreate the folders to host the logs<\/h3>\n
On the master node:<\/h4>\n
$ mkdir \/var\/spool\/slurmctld\n$ chown slurm:slurm \/var\/spool\/slurmctld\n$ chmod 755 \/var\/spool\/slurmctld\n$ mkdir \/var\/log\/slurm\n$ touch \/var\/log\/slurm\/slurmctld.log\n$ touch \/var\/log\/slurm\/slurm_jobacct.log \/var\/log\/slurm\/slurm_jobcomp.log\n$ chown -R slurm:slurm \/var\/log\/slurm\/<\/code><\/pre>\nOn the compute nodes:<\/p>\n
$ mkdir \/var\/spool\/slurmd\n$ chown slurm: \/var\/spool\/slurmd\n$ chmod 755 \/var\/spool\/slurmd\n$ mkdir \/var\/log\/slurm\/\n$ touch \/var\/log\/slurm\/slurmd.log\n$ chown -R slurm:slurm \/var\/log\/slurm\/slurmd.log<\/code><\/pre>\ntest the configuration:<\/h4>\n
$ slurmd -C<\/code><\/pre>\nYou should get something like:<\/p>\n
NodeName=master0 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=23938 UpTime=22-10:03:46<\/code><\/pre>\nLaunch the slurmd service on the compute nodes:<\/h4>\n
$ systemctl enable slurmd.service\n$ systemctl start slurmd.service\n$ systemctl status slurmd.service<\/code><\/pre>\nLaunch the slurmctld service on the master node:<\/h4>\n
$ systemctl enable slurmctld.service\n$ systemctl start slurmctld.service\n$ systemctl status slurmctld.service<\/code><\/pre>\nChange the state of a node from down to idle<\/h4>\n
$ scontrol update NodeName=nodeX State=RESUME<\/code><\/pre>\nWhere nodeX is the name of your node<\/p>\n
\n<\/a><\/p>\n
Configure usage limits<\/h2>\n
Modify the \/etc\/slurm\/slurm.conf file<\/h3>\n
Modify the
AccountingStorageEnforce<\/code>parameter with:<\/p>\nAccountingStorageEnforce=limits<\/code><\/pre>\nCopy the modified file to the several nodes<\/p>\n
Restart the slurmctld service to validate the modifications:<\/p>\n
$ systemctl restart slurmctld<\/code><\/pre>\nCreate a cluster:<\/h3>\n
The cluster is the name we want for your slurm cluster.<\/p>\n
It is defined in the
\/etc\/slurm\/slurm.conf<\/code> file with the line <\/p>\nClusterName=ird<\/code><\/pre>\nTo set usage limitations for your users, you first have to create an accounting cluster with the command:<\/p>\n
$sacctmgr add cluster ird<\/code><\/pre>\nCreate an accounting account<\/h3>\n
An accounting account is a group under slurm that allows the administrator to manage the users rights to use slurm.<\/p>\n
Example: you can create a account to group the bioinfo teams members <\/p>\n
$ sacctmgr add account bioinfo Description="bioinfo member"<\/code><\/pre>\nYou can create a account to group the peaople allow to use the gpu partition<\/p>\n
$ sacctmgr add account gpu_group Description="Members can use the gpu partition"<\/code><\/pre>\nCreate a user account<\/h3>\n
You have to create slurm user to make them be able to launch slurm jobs.<\/p>\n
$ sacctmgr create user name=xxx DefaultAccount=yyy<\/code><\/pre>\nModify a user account to add it to another accounting account:<\/h3>\n
$ sacctmgr add user xxx Account=zzzz<\/code><\/pre>\nModify a node definition<\/h3>\n
Add the amount of \/scratch partition<\/h4>\n
In the file
\/etc\/slurm\/slurm.conf<\/code><\/p>\nModify the TmpFS file system<\/h5>\n
$TmpFS=\/scratch<\/code><\/pre>\nAdd the TmpDisk value for \/scratch<\/h5>\n
The TmpDisk is the size of the scratch in MB , you have to add in the line starting with NodeName <\/p>\n
For example for a node with a 3TB disk:<\/p>\n
$ NodeName=node21 CPUs=16 Sockets=4 RealMemory=32004 TmpDisk=3000 CoresPerSocket=4 ThreadsPerCore=1 State=UNKNOWN<\/code><\/pre>\nModify a partition definition<\/h3>\n
You have to modify the line starting with PartitionName in the file
\/etc\/slurm\/slurm.conf<\/code> .<\/p>\nSeveral options are available according to what you want <\/p>\n
Add a time limit for running jobs (MaxTime)<\/h4>\n
A limitation time on partitions allows slurm to manage priorities between jobs on the same node.<\/p>\n
You have to add it in the PartitionName line with the amount of time in minutes.<\/p>\n
For example a partition with a 1 day max time the partition definition will be:<\/p>\n
PartitionName=short Nodes=node21,node[12-15] MaxTime=1440 State=UP<\/code><\/pre>\nAdd a Max Memory per CPU (MaxMemPerCPU)<\/h4>\n
As memory is a consumable resource MaxMemPerCPU serves not only to protect the node\u2019s memory but will also automatically increase a job\u2019s core count on submission where possible<\/p>\n
You have to add it in the PartitionName line with the amount of memory in Mb.<\/p>\n
This is normally set to MaxMem\/NumCores<\/p>\n
for example 2GB\/CPU, the partition definition will be<\/p>\n
PartitionName=normal Nodes=node21,node[12-15] MaxMemPerCPU=2000 MaxTime=4320 State=UP<\/code><\/pre>\n
\nLinks<\/h3>\n
- Authentication and databases<\/a><\/li>\n
![](\"http:\/\/creativecommons.org.nz\/wp-content\/uploads\/2012\/05\/by-nc-sa1.png\"\/)