Data transfer

A basic requirement for the transfer of data is the existence of an sshkey pair, whereby the public key must be stored in the UDE SelfCare Portal, see Security -> SSH-keys in the SelfCare-Portal.

Copy files from/to local client

The command line tools scp and rsync can be used to transfer files between local computers and the file systems of the HPC systems

Due to the upstream jump host, which ensures load balancing between the front-end nodes of the cluster, it is necessary to specify this when executing the commands.

Please note that

  • in the following examples, the target directory of the data transfer is in the workspace area on the SCRATCH. Alternatively, the path to directories in HOME or HPC_HOME can also be used there.

  • the data transfer can only be initiated on the local machine, since due to IT security only incoming ssh connections on the HPC systems are allowed.

RSYNC

For a file transfer using rsync with activated zip compression (option ‘z’), recursive processing and preservation of almost all file attributes (option ‘a’) and screen output (‘v’) of the transferred file and a summary, use the following syntax:

rsync -azv -e 'ssh -J <username>@login.hpc.uni-due.de' <your-files> <username>@amplitude:/lustre/scratch/<your-workspace>/

SCP

Alternatively, the scp tool can be used with the following syntax. If the data is large, it should be reduced in size in advance using compression (e.g. zip.format).

scp -J <username>@login.hpc.uni-due.de <your-files> <username>@amplitude:/lustre/scratch/<your-workspace>/<your-files>

Please note

  • on the jump host HOME, SCRATCH, HPC_HOME are not mounted, hence the commands forward the request to frontend nodes

  • the examples show how to transfer files from your local machine to the scratch directory on the HPC system. In order to download your files from scratch switch the parameters, e.g.

# Syntax: scp with ProxyJump                                      COPY-FROM-SCRATCH                                    COPY-TO-LOCAL-DIR, e.g. ./
scp -J <username>@login.hpc.uni-due.de <username>@amplitude:/lustre/scratch/<your-workspace>/<your-files> <path-to-local-directory>

From internet sources

Due to IT security guidelines, the HPC systems have no direct connection to external UDE networks (i.e. the Internet). A web proxy proxy.hpc.uni-due.de is available to enable users to access sources on the Internet. The environment variables $HTTP_PROXY and $HTTPS_PROXY are set accordingly on the HPC system by default.

Specific internet sources are released on this server. More can be added on request at hpc-support@uni-due.de. An informal description/content of the source and a justification for the necessity of activation must be provided.