How to Monitor Jobs

From Statistics Cluster
Jump to: navigation, search

Check the state of the job

Check the jobs of a specific user

Once jobs have been submitted to the cluster, monitoring can be performed using the following command in a terminal

[username@computer ~]$ condor_q -submitter <username>| less

This will display

  • ID: the process ID
  • OWNER: the owner of the job
  • SUBMITTED: the date and time it was submitted
  • RUN_TIME: how long it has been running
  • ST: its current status (run R, held H , idle I)
  • SIZE: the job size
  • CMD: program name

This is useful to monitor your own jobs to check on their status.

Check all jobs

If you want to see all of the jobs in the queue

condor_q | less

Held job?

If you check the status of your job and find it's been held, there are a couple possibilities.

Memory thrashing

Your job is using more memory than you requested and is thus causing the server to use swap (hard disk) instead of memory. This will thrash the system and is not good. A script is run periodically to check for such jobs. If a job is found that thrashes the machine, the job is held.

Check your log file for a particular job ID and find the point where it says that your job was held and that it was checkpointed. It should list the disk and memory your job used and requested. If the disk is significantly larger than requested, this job was using swap.

Submitted too many jobs

The cluster can only process so many jobs in the queue. If you submitted tens of thousands of jobs, they will be held. Our cluster cannot handle negotiating so many queued jobs at one time.

Never submit more than 5,000 jobs at a time. Remember, other users might also submit a lot of jobs too.

Check which machine the job is running on

Another useful command is condor_status which can tell you information about the cluster machines

[username@computer ~]$ condor_status [-r] | grep stat | less

This will show a list of the various machine resources and if the option -r is supplied it will only show machines with running jobs.

If there are any concerns about a specific job, please contact the main administrator.

Check the job as it runs

Check stdout

If a job is running you can execute the following command to see the tail of the stdout file

condor_tail <job_id>

Check stderr

If you want to see if there are any errors you can run the following

condor_tail -no-stdout -stderr <job_id>