How to adjust GENSYS to different queue systems   

The programs in the GENSYS package can easily be adjusted to fit into different queue systems. A queue system is used for executing many calcuations in series, running the computers in a cluster at maximum load without overloading. To communicate with the queue system on your computer, Gensys are using the following scripts:

q_sys	=	Sending a job to the queue system
q_stat	=	Checking the status of jobs submitted to the queue system
q_del	=	Deleting a job running in the queue system

The above scripts are located in directory $gensys/bin. The communication between Gensys and the installed queue system on your machine must go via these three scripts.

Script q_sys

Sends a job to the queue system.

The programs in Gensys calls script q_sys with three arguments:

Examples showing how q_sys can look for different queue systems:

Queue system LSF:

echo "--- Sending job to queue system LSF ------------------------"
echo "File      $1"
echo "Queue     $2"
echo "Ident     $3"
bsub  -q $2  -J $3  $1
exit_status=$?

Queue system PBSPro:

echo "--- Sending job to queue system PBSPro ---------------------"
echo "File      $PWD/$1"
qsub $PWD/$1
exit_status=$?

Queue system Q-sys:

echo "--- Sending job to queue system Q-sys ----------------------"
echo "File      $1"
echo "Queue     $2"
echo "Ident     $3"
Q-sys $1 $2 $3
exit_status=$?

Queue system SLURM:

echo "--- Sending job to queue system SLURM ----------------------"
echo "File      \"$1\""
echo "Queue     $2"

echo "sbatch --output=\"qout/slurm.stdout\" \"$1\""
sbatch --output="qout/$3_slurm.stdout" --open-mode=append $1

Queue system SQS:

echo "--- Sending job to queue system SQS ------------------------"
echo "File      \"$1\""
echo "Queue     $2"
echo  "/home/gensys/bin/qsub -q $2 \"$1\""
jobnr=`/home/gensys/bin/qsub -q $2 \"$1\"| head -1`
echo "Job nr    $jobnr"
sleep 7
echo q_stat
$gensys/bin/q_stat
$gensys/bin/q_stat | grep $jobnr > /dev/null
exit_status=$?
if [ $exit_status -ne 0 ]; then
 echo " * warning * In Script `basename $0`"
 echo "             Your job has stopped after a very short time."
 echo "             Please write \"qinit stop\" in case demon qseek"
 echo "             is running in the wrong directory."
 echo "             Output from script $col_red"$1"$col_black has been written to file"
 echo "             "$col_red"qout/$3.qout"$col_black
 echo
fi

Script q_stat

Lists current jobs in queue system.

Script q_stat do not hold any arguments.

Examples showing how q_sys can look for different queue systems:

Queue system LSF:

xlsbatch &

Queue system PBSPro:

qstat

Queue system Q-sys:

Q-stat

Queue system SLURM:

squeue

Queue system SQS:

qinit show

Script q_del

Deletes a job to the queue system.

The argument list to script q_del, shall contain a list of the jobs that shall be deleted.

Examples showing how q_sys can look for different queue systems:

Queue system LSF:

bkill $@

Queue system PBSPro:

qdel $@

Queue system Q-sys:

Q-del $@

Queue system SLURM:

scancel $@

Queue system SQS:

qdel $@

Debugging SLURM

Check that all SLURM demons are up and running

ps -ef | grep munged | grep -v grep
ps -ef | grep slurmd | grep -v grep
ps -ef | grep slurmctld | grep -v grep

Display information about all partitions:

sinfo -a --long --Node

To change the node state from DRAIN, DRAINING, DOWN or REBOOT to IDLE, give the following command:

sudo scontrol update nodename=`hostname` state=resume reason="The node states has been resumed"

Error messages can also be found in the two files /var/log/slurmctld.log and /var/log/slurmd.log. To inspect the contents of these files you must open them as superuser:

sudo xed /var/log/slurmctld.log &
sudo xed /var/log/slurmd.log &

If necesssary, update the slurm configuration file:

sudo xed /etc/slurm/slurm.conf

sudo /etc/init.d/slurmd    stop
sudo /etc/init.d/slurmd    start
sudo slurmctld
sudo scontrol update nodename=`hostname` state=resume reason="File /etc/slurm/slurm.conf has been updated"

After restarting slurmd and slurmctld, wait some time before checking the state with command sinfo. It takes some time for slurm to update the status of the nodes.

If command sinfo -a --long --Node say the node states are inval. Give the command slurmd -C to see the actual hardware configuration of your nodes. Edit file /etc/slurm/slurm.conf accordingly and restart the compute node daemon slurmd and the central management daemon slurmctld (how to restart, see instructions above).

After command sinfo -a --long --Node have told you that the nodes are up and running, and their state are idle. You can send a test test job to SLURM:

sbatch --output=sbatch.out <<+
#!/bin/sh
/bin/hostname
+