The eventmon API on metaserver at Caltech-CIT

Legend:
green ball Normal status or debugging message
yellow ball Notable condition which may be a non-fatal error
orange ball Error condition not fatal to job
red ball Error condition which fatal to job
blue ball Notable condition which is not an error
purple ball Currently undefined
email Condition requires email notification of the responsible administrator of this API
telephone Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Caltech-CIT

11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP execOverload overloaded exec
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP eventmon::setDatabaseRate Max number of insertions/secs set to 1000 (-1 = no limit)
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP bgLoop Looping process eventmon_expiredJobKiller started
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP bgLoop Looping process eventmon_defunctjobs started
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP openListenSock port 10019 (operator) opened on metaserver as sock3
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP openListenSock port 10020 (emergency) opened on metaserver as sock4
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP openListenSock port 10021 (data) opened on metaserver as dsock0
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP leakLogger inital size of eventmon API: 14276 kB
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP bgLoop Looping process etchosts started
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP eventmon Trying API
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP eventmon API yes
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP eventmon Trying LDAS_SYSTEM
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 STARTUP eventmon LDAS_SYSTEM yes
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 IDLE exec exec lsof -b -p 3368
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_prepare@MountPointStatus.cc begin
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_prepare@MountPointStatus.cc end
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_parent@MountPointStatus.cc begin
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 CXX_THREAD fork_parent@MountPointStatus.cc end
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 IDLE bgLoop Looping process statpagefile started
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 IDLE bucketBrigade ::eventmon_BUCKET_CHECK_INTERVAL not defined in LDASeventmon.rsc file. I am setting it to the default value of 5 seconds.
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 IDLE bgLoop Looping process checkbuckets started
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 IDLE bgLoop Looping process killedjobreaper started
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 IDLE logVariableChange Changed ::DEBUG_DEADLOCK_DETECTOR_LEVEL from 0 to 0
11/06/09-16:52:40 PST 11/07/09-00:52:40 GMT 941590375 IDLE bgLoop Looping process deadlockdetectorlevel started
11/06/09-16:52:45 PST 11/07/09-00:52:45 GMT 941590380 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.114.9') (::FTPDIR '') (::HTTPURL 'http://131.215.114.9/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/usr1/grid/ldas') (::GRIDFTPDIR '/usr1/grid/ldas') (::LDAS_GATEWAY 'ldas-cit 131.215.114.9') (::LDAS_SYSTEM 'ldas-cit') (::RUNCODE 'LDAS-CIT')
11/06/09-16:52:45 PST 11/07/09-00:52:45 GMT 941590380 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.114.9') (::FTPDIR '') (::HTTPURL 'http://131.215.114.9/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/usr1/grid/ldas') (::GRIDFTPDIR '/usr1/grid/ldas') (::LDAS_GATEWAY 'ldas-cit 131.215.114.9') (::LDAS_SYSTEM 'ldas-cit') (::RUNCODE 'LDAS-CIT')
11/06/09-17:13:22 PST 11/07/09-01:13:22 GMT 941591617 LDAS-CIT1382450 operator_callback begin processing LDAS-CIT1382450
11/06/09-17:13:22 PST 11/07/09-01:13:22 GMT 941591617 LDAS-CIT1382450 standaloneWrapper ingesting files 3 from standalone wrapper: /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/output_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd
11/06/09-17:13:25 PST 11/07/09-01:13:25 GMT 941591620 LDAS-CIT1382450 processDataBucket bad items in data bucket: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/output_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd'
11/06/09-17:13:25 PST 11/07/09-01:13:25 GMT 941591620 LDAS-CIT1382450 ::eventmon::checkBuckets process data bucket error: processDataBucket: bad items in data bucket: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/output_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd' (item: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd') (type: 'text html file'), contents=
11/06/09-17:13:25 PST 11/07/09-01:13:25 GMT 941591620 LDAS-CIT1382450 killJob products to remove
11/06/09-17:13:25 PST 11/07/09-01:13:25 GMT 941591620 LDAS-CIT1382450 killJob LDAS-CIT1382450 processed return code=1, called with rc=1 {process data bucket error: processDataBucket: bad items in data bucket: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/output_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd' (item: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382450/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd') (type: 'text html file'), contents=}
11/06/09-17:13:25 PST 11/07/09-01:13:25 GMT 941591620 LDAS-CIT1382450 reattach done processing LDAS-CIT1382450, closing sock7