How to use the infiniband network to transfer faster
| Description | Use the --constraint option of Slurm |
|---|---|
| Author | ndomassi TANDO (ndomassi.tando@ird.fr) |
| Creation Date | 01/02/2024 |
| Last Modified Date | 07/03/2025 |
Role of the Infiniband network:
The infiniband network (192.168.4.0) allows users to have access to an high speed network to transfer their data from the san to the nodes or between infiniband nodes.
Only certain nodes are equiped with infiniband for now , check the list below:
| node | partitions | CPU | RAM total (Gb) | RAM by CPU (Gb/cpu) | scratch size (To) |
|---|---|---|---|---|---|
| node0 | normal | 24 | 144 | 6 | 13 |
| node1 | normal | 24 | 144 | 6 | 2 |
| node4 | normal | 72 | 257 | 4 | 13 |
| node5 | normal | 88 | 516 | 6 | 13 |
| node7 | normal | 24 | 145 | 6 | 3 |
| node8 | long | 24 | 48 | 2 | 13 |
| node9 | normal | 64 | 144 | 2 | 13 |
| node11 | runner | 24 | 48 | 2 | 3 |
| node17 | normal | 48 | 144 | 3 | 13 |
| node20 | normal | 20 | 64 | 3 | 13 |
| node21 | normal | 24 | 144 | 6 | 13 |
| node24 | normal | 20 | 64 | 3 | 13 |
| node25 | supermem | 40 | 1 032 | 26 | 14 |
| node26 | gpu | 24 | 192 | 8 | 14 |
| node27 | normal | 24 | 144 | 6 | 14 |
Reserving node in the Infiniband network:
To use the infiniband network, we have to add the "-ib" siffix to the alias of the machine, for example san will become san-ib, node0 will become node0-ib etc...
Use the options -p partition ( with partition being the node partition) --constraint=infiniband to reserve one or several cores on a infiniband node. ( cf feature page ) for more explanations.
For example, we want to reserve an infiniband node on highmem partition in a interactive mode, type the following command:
srun -p highmem --constraint=infiniband --pty bash -iYou now have access to the infiniband network, to use it just change the alias san to san-ib to do your transfers .
Transfer your data via Infiniband:
There are 2 possible syntaxes:
Retrieve data from the san:
scp -r san-ib:path_to_files/file local_destination
Transfer data to the san:
scp -r /local_path_to_files/file san-ib:remote_destination
