Frequently Asked Questions (FAQ) - High Performance Computing (HPC)

Page / - last modified by superadmin on Tue, 05/07/2024, 2:07 PM

High Performance Computing (HPC)

last modified by superadmin

on Tue, 05/07/2024, 2:07 PM

Table of Contents

Expand All

Accessing the HPC

+

How to generate public and private SSH keys?

Here are the procedures on how to generate public and private SSH keys.

+

What are the recommended platforms for generating SSH keys for Windows OS and MacOS users?

Both Windows OS and MacOS users can generate SSH keys via terminal/ CLI (Command Line Interface), whereas only Windows OS users can use GUI (Graphical User Interface) by way of PuTTYgen as another way to generate SSH keys.

+

How can I access the HPC if I will be using a different machine?

In case you switch to a different machine, we suggest that you generate a new set of SSH keys and have these appended by COARE team.

Package/ Program Installation in HPC

+

How to install a program or package in HPC?

Most of the programs/ packages can be installed by users. You can use the Anaconda environment to do the installation of the package. Moreover, you may refer to our Wiki for the step-by-step process.

+

How to install a GPU/ CUDA-enabled package?

You may refer to our wiki for the standard installation procedures of a CUDA-enabled build package.

+

Can I install programs directly in the frontend?

No, it is recommended to submit jobs using SLURM when installing programs.

+

Is it possible to install multiple packages? If yes, how?

Yes, you may install multiple packages. You can do this by submitting a job via SLURM. Here is the resource in installing multiple packages.

+

Which folder within HPC can I load and install my environments and software packages?

We highly suggest that you change the default location where you load and install your conda environments and packages to scratch folder because it is much faster than home folder.

+

How can I see the list of the available modules and the installed package/s in my environment?

To view the available modules, you may refer to this link for the procedures. For the installed packages in your Anaconda environment, you may check our COARE Wiki.

Running Jobs

+

Can I run test jobs in the GPU partition? How to increase priority of my job so that it will run immediately?

Yes. However, by default, user’s jobs are queued unless you are using special QOS.

+

What are the maximum resources that can be used for a job script?

You may refer to our Wiki for more information about the QOS (Quality of Service) per job.

+

How can I know which partition to be used when running jobs for a particular purpose (e.g., installing packages, running scripts, among others)?

Knowing the partition to be used is on a case-to-case basis, depending on what is stated in the software’s documentation. However, here are the some of the general information per partition:

Debug – You may use this partition for test runs, creating conda environments, among others.

Batch – This is COARE HPC’s main partition dedicated to processing multi-core/multi-node CPU jobs in parallel.

Serial – This partition is specifically for serial jobs or those that only require 1 CPU core (e.g., replications/ running 5 replicas of a certain program 5 CPUs running independently.

GPU - These are used to make quick work of numerically intensive operations. For certain workloads like image processing, training artificial neural networks and solving complex equations, etc.

+

Do you have recommendations on submitting jobs that would produce optimal results?

Here are some recommendations on job submissions that would yield optimal results.

Errors Encountered

+

How can I resolve if an error occurred when I initialized SFTP protocol via WinSCP but I was able to log-in via PuTTY?

You may refer to our Wiki.

+

Do the data/ files inside the scratch directories get deleted?

Yes, as we regularly conduct automated purging. You may refer to the COARE AUP (Acceptable Use Policy) for more details. This also serves as a reminder not to use HPC for personal storage.

+

I suddenly encountered “No supported authentication methods available” or “Server refused our key” after being able to access my account in the previous log-ins. What might be the cause and how can I resolve this?

This error might be caused by the following factors:

PuTTY can’t find the private key; you may try to log-in once again.

You probably generated the SSH keys in a different format (e.g., SSH2). Please ensure that the SSH keys are in the correct OpenSSH format.

Alternatively, you may opt to generate a new set of SSH keys and have these appended by COARE Team.

+

How can I troubleshoot if I encounter an error message: "Network Error: Connection Timed Out"?

Try logging in your account again using a different computer or network connection.

+

Is it possible to recover accidentally deleted files from scratch folders on the HPC?

Files that are accidentally deleted by the rm command can no longer be recovered. We highly recommend that you always backup your files and be mindful when deleting files and folders. You may refer to our Acceptable Use Policy (AUP) - General Conditions of Use.

+

Why is my job waiting in the queue even if there are nodes available?

This can either mean that you may have already maximized your QOS allocation, or your job has been assigned a lower priority (since the scheduler increases the priority of jobs that have been queued longer).

+

What are the possible reasons for the jobs being suddenly terminated before the expected end date?

This is a case-to-case basis. However, here are some of the possible reasons for sudden termination of jobs:

Software/ hardware issues

Storage is already full

Lack of CPU hours

+

What might be the cause if I encounter a problem installing a particular module?

You might have installed the module without using SLURM script. For the installation of mamba packages, please submit jobs using SLURM.