Remote Clusters — SSH Execution and Queue Schedulers

Two ways to run remotely

The Queue system picker in Config → SSH decides how a job reaches the remote machine. Everything else — composing the method, submitting from Run, analyzing the result — is identical to a local run.

• None (direct run): Qemica opens an SSH session, runs the engine in the foreground, and streams progress back live — the convergence charts and logs update exactly like a local job. Best for short jobs on a machine you own. The session must stay up for the whole run, so laptop sleep or a network drop kills it.
• SLURM / PBS / Custom: Qemica uploads a rendered submission script and hands it to the scheduler (sbatch, qsub, or your own command), records the remote job id, and polls the queue in the background. Submission is detached — quitting Qemica is safe, and it reattaches to every pending cluster job on the next launch. Best for long jobs and shared clusters.

One-time: install the engine on the cluster

The remote machine needs the same quchemy engine environment as a local install. Install it once into your cluster home directory (on clusters with a shared filesystem, the login node is fine — compute nodes see the same environment).

• On the cluster: curl -LsSf https://astral.sh/uv/install.sh | sh, then uv venv ~/qemica-engine --python 3.14 and uv pip install --python ~/qemica-engine/bin/python <the quchemy wheel URL shown in the app>.
• In Config → SSH, set Remote python to ~/qemica-engine/bin/python and leave Remote sidecar path empty — Qemica runs the installed engine directly (python -m quchemy.sidecar). A sidecar path is only needed for a source checkout.
• Remote work dir (default ~/qemica_jobs) is a staging directory Qemica creates automatically: the job request, the submission script, and the result land there, and are cleaned up after each job.

Connection settings (Config → SSH)

• Host, User, Port — the machine you would reach with ssh user@host -p port. Qemica uses your system's own ssh and scp binaries, so everything in your ~/.ssh/config (aliases, keys, jump hosts, multiplexing) applies here too.
• Extra SSH options — raw flags added to every ssh/scp call, e.g. -J bastion for a jump host, -i ~/.ssh/cluster_key for a specific identity, or -o ControlPath=… for connection multiplexing. Quote a value that contains spaces. Your -o flags take precedence over Qemica's built-ins.
• Strict host-key checking — off by default (a first-seen host key is trusted and pinned; a changed key still fails). Turn it on for sites whose policy requires pre-seeded ~/.ssh/known_hosts entries.
• Test connection — one click verifies reachability and authentication, the remote Python (with its version), that the quchemy engine imports on it (a missing engine is translated into 'install the engine on the server' guidance), and — when a queue system is picked — that the submit command (sbatch/qsub/custom) exists on the remote PATH. Each check is a separate status line, and the raw SSH error is shown verbatim on failure. Run it after any settings change; it is much faster than discovering a typo on a real submit.

SLURM

Pick SLURM in the Queue system picker and the built-in preset does the rest: jobs submit with sbatch, are watched with squeue, and cancel with scancel. The default submission-script template covers the common #SBATCH directives.

• Set per-job resources — cores, memory, walltime, and (optionally) the partition — directly on the Run screen at submit time; they fill the {cores}, {memory}, {walltime}, and {queue} placeholders in the template.
• Walltime uses SLURM's HH:MM:SS form and memory takes sbatch strings like 8G. Values are validated before submission, so a malformed request fails in Qemica with a clear message instead of at the scheduler.
• Site-specific needs — an account to charge (--account), GPUs (--gres), a constraint, or module load lines before the engine runs — go straight into the editable template (see below).
• A finished job that has already aged out of squeue is handled correctly: Qemica treats the scheduler's 'invalid job id' answer as job-finished and fetches the result.

PBS / Torque

Pick PBS / Torque and jobs submit with qsub, are watched with qstat -f, and cancel with qdel. The default template covers the common #PBS directives (-l nodes/ppn, -l mem, -l walltime, -q, -A).

• Per-job resources from the Run screen fill the template exactly as with SLURM; walltime is HH:MM:SS and memory takes PBS strings like 8gb.
• PBS Pro and Torque variants that phrase resources differently (e.g. -l select=1:ncpus={cores}) are a one-line edit of the template.
• A job that has aged out of qstat ('Unknown Job Id') is treated as finished and its result fetched.

Other schedulers (Custom)

Pick Custom and supply four things: the submit command, a status command, a cancel command, and a regular expression that pulls the job id out of the submit command's output ({id} in the status/cancel commands is replaced with that id). This reaches Grid Engine, LSF, or an in-house queue with no new code. Two worked examples:

• Grid Engine (SGE/UGE): Submit command qsub · Job-id regex Your job (\d+) · Status command qstat -j {id} · Cancel command qdel {id}. SGE's qstat errors for a finished job, which the generic status logic reads as job-finished.
• LSF: Submit command bsub < · Job-id regex Job <(\d+)> · Status command bjobs -noheader {id} · Cancel command bkill {id}. (bsub reads the script from stdin, hence the trailing <; the submit command runs through the remote shell, so redirection works.)
• For schedulers without built-in status parsing, Qemica uses a robust generic rule: while the status command reports the job, it counts as running; once the query comes back empty or errors, the job is presumed finished — and the definitive success test is always whether the result file exists and parses. A vanished job with no result is reported as a failure, never as an empty success.
• Pick the matching preset first and switch to Custom afterwards if you only need to tweak one command — the preset's values remain as starting points.

The submission-script template

Every queue submission renders an editable template (Config → SSH) into the actual batch script. The default for the picked scheduler is a solid starting point; Reset to preset default restores it at any time.

• Placeholders: {job_name} {cores} {memory} {walltime} {queue} {nodes} {gpus} {account} for resources, and {workdir} {python} {sidecar} {req} for the run itself. The last line of the default template — {python} {sidecar} {req} — is the engine invocation; keep it.
• A directive line whose only placeholder is empty is dropped from the rendered script — so the default template's --account and --gres lines simply vanish unless you set those values, no editing needed.
• Clusters that provide software through environment modules often need module load lines (or a source …/env.sh) added to the template before the engine line — the standard fix when a job dies immediately even though Test connection is green, because batch shells start with a minimal environment.
• Poll interval (default 30 s) sets how often Qemica asks the scheduler about your pending jobs. Raise it for long jobs on busy login nodes; lower it for fast turnaround.

Submitting, reattaching, and cancelling

• Submit works exactly like a local run: pick SSH in the Run screen's backend picker, set the per-job resources, and press Submit. The job appears in the queue as Queued, flips to Running when the scheduler starts it, and lands in Finished with the full result fetched back automatically.
• Detached lifecycle: once the scheduler accepts the job, nothing depends on your session — quit Qemica, close the laptop, and on the next launch it reattaches to every pending remote job and resumes polling. Results persist against the exact molecule and method you submitted, even if you have moved on to other work meanwhile.
• Stop on a scheduler job issues the scheduler's own cancel (scancel / qdel / your custom command) and cleans up the staged files on the remote host.
• Every failure is reported honestly with the remote console output attached — a rejected submission, an unparseable job id, a job that vanished without writing a result — so the fix is always visible, never a silent empty result.

Troubleshooting

• Permission denied (publickey): your key is not loaded (ssh-add), not authorized on the host, or the host demands two-factor — see the authentication group above.
• sbatch/qsub not found in Test connection: the scheduler binary is not on the PATH of a non-interactive shell. Either use the full path (e.g. /usr/bin/sbatch) as a Custom submit command, or ask your admin where scheduler commands live for batch/SSH sessions.
• Job dies instantly but Test connection is green: almost always the batch environment — add the module load / source lines your cluster documents to the submission-script template.
• Host key verification failed: the host's key changed (or strict checking is on and the host is unknown). Verify with your admin, update ~/.ssh/known_hosts, and retry.
• Every transport step fails fast with a bounded timeout, so a dead host answers in seconds, not minutes — check the exact SSH error text shown with the failure.