How to use the infiniband network

How to use the infiniband network to transfer faster

Description Use the --constraint option of Slurm
Author ndomassi TANDO (
Creation Date 01/02/2024
Last Modified Date 01/02/2024

Role of the Infiniband network:

The infiniband network (  allows users to have access to an high speed network to transfer their data from the san to the nodes or between infiniband nodes.

Only certain nodes are equiped with infiniband for now :  node0, node1, node4, node5, node7, node8, node9, node11, node17, node20, node21, node22, node23, node24, node25, node26, node27

Reserving node in the Infiniband network:

To use the infiniband network, we have to add the "-ib" siffix to the alias of the machine, for example san will become san-ib, node0 will become node0-ib etc...

Use the options    -p partition ( with partition being the node partition) --constraint=infiniband to reserve one or several cores on a infiniband node. ( cf feature page ) for more explanations.
For example, we want to reserve an infiniband node on highmem partition in a interactive mode, type the following command:

srun -p highmem --constraint=infiniband --pty bash -i

You now have access to the infiniband network, to use it just change the alias san to san-ib to do your transfers .

Transfer your data via Infiniband:

There are 2 possible syntaxes:

Retrieve data from the san:

scp -r san-ib:path_to_files/file local_destination

Transfer data to the san:

scp -r /local_path_to_files/file san-ib:remote_destination