python (65.1k questions)

javascript (44.2k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (12.9k questions)

Questions - slurm

Running multiples files jobs with one sbatch

I want to run N files (N jobs) that are inside N folders that are in my pwd such : Folder_1 contains file_1 Folder_2 contains file_2 | | | Folder_N contains file_N For one file_1 i just have...
test-img

haswellrefresh

linux

file

jobs

slurm

sbatch

Votes: 0

Answers: 2

Latest Answer

I found a solution i just want to share it here, #!/bin/bash ls ./folder*/file* > file (stock paths in a file) while read p; do (read it line by line) sbatch script.sh "$p" done &l...
test-img

haswellrefresh

Using wildcards in slurm resources directive with snakemake

I'm using snakemake to create rules and submit jobs on our HPC with slurm. To make the output "prettier", I would like to be able to set the job_name argument in the resources directive so t...
test-img

PD Schloss

slurm

snakemake

Votes: 0

Answers: 1

Latest Answer

In your resource directive, you need to use an input function: resources: job_name=lambda wildcards: f"{wildcards.dataset}_clean_data" You may run into some issues with having to re...
test-img

Troy Comi

What is causing my random: "joblib.externals.loky.process_executor.TerminatedWorkerError" errors?

I'm making GIS-based data-analysis, where I calculate wide area nation wide prediction maps (e.g. weather maps etc.). Because my target area is very big (whole country) I am using supercomputers (Slur...
test-img

jjepsuomi

python

parallel-processing

slurm

joblib

supercomputers

Votes: 0

Answers: 3

Latest Answer

The error traceback indicates that a worker process was unexpectedly terminated due to a segmentation fault or because of excessive memory usage causing the Operating System to kill the worker. This t...
test-img

Pierre Vermot

CUDA OOM on Slurm but not locally, even if Slurm has more GPUs

I am working on a Slurm-based cluster. I debug my code on the login node, which has 2 GPUs. I can run it fine using model = nn.DataParallel(model), but my Slurm jobs crash because of RuntimeError: CUD...
test-img

Simon

parallel-processing

pytorch

out-of-memory

lstm

slurm

Votes: 0

Answers: 0

Posts

Questions

Blogs

Jobs