The eventmon API on metaserver at Caltech-CIT

Legend:

Normal status or debugging message

Notable condition which may be a non-fatal error

Error condition not fatal to job

Error condition which fatal to job

Notable condition which is not an error

Currently undefined

Condition requires email notification of the responsible administrator of this API

Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Caltech-CIT

941515141 CXX_THREAD fork_prepare@MountPointStatus.cc begin
941515141 CXX_THREAD fork_prepare@MountPointStatus.cc end
941515141 CXX_THREAD fork_prepare@MountPointStatus.cc begin
941515141 CXX_THREAD fork_prepare@MountPointStatus.cc end
941515141 CXX_THREAD fork_parent@MountPointStatus.cc begin
941515141 CXX_THREAD fork_parent@MountPointStatus.cc end
941515141 CXX_THREAD fork_parent@MountPointStatus.cc begin
941515141 CXX_THREAD fork_parent@MountPointStatus.cc end
941515141 STARTUP execOverload overloaded exec
941515141 STARTUP eventmon::setDatabaseRate Max number of insertions/secs set to 1000 (-1 = no limit)
941515141 STARTUP bgLoop Looping process eventmon_expiredJobKiller started
941515141 STARTUP bgLoop Looping process eventmon_defunctjobs started
941515141 STARTUP openListenSock port 10019 (operator) opened on metaserver as sock3
941515141 STARTUP openListenSock port 10020 (emergency) opened on metaserver as sock4
941515141 STARTUP openListenSock port 10021 (data) opened on metaserver as dsock0
941515141 STARTUP leakLogger inital size of eventmon API: 14276 kB
941515141 STARTUP bgLoop Looping process etchosts started
941515141 STARTUP eventmon Trying API
941515141 STARTUP eventmon API yes
941515141 STARTUP eventmon Trying LDAS_SYSTEM
941515141 STARTUP eventmon LDAS_SYSTEM yes
941515141 IDLE exec exec lsof -b -p 13661
941515141 CXX_THREAD fork_prepare@MountPointStatus.cc begin
941515141 CXX_THREAD fork_prepare@MountPointStatus.cc end
941515141 CXX_THREAD fork_prepare@MountPointStatus.cc begin
941515141 CXX_THREAD fork_prepare@MountPointStatus.cc end
941515141 CXX_THREAD fork_parent@MountPointStatus.cc begin
941515141 CXX_THREAD fork_parent@MountPointStatus.cc end
941515141 CXX_THREAD fork_parent@MountPointStatus.cc begin
941515141 CXX_THREAD fork_parent@MountPointStatus.cc end
941515141 IDLE bgLoop Looping process statpagefile started
941515141 IDLE bucketBrigade ::eventmon_BUCKET_CHECK_INTERVAL not defined in LDASeventmon.rsc file. I am setting it to the default value of 5 seconds.
941515141 IDLE bgLoop Looping process checkbuckets started
941515141 IDLE bgLoop Looping process killedjobreaper started
941515141 IDLE logVariableChange Changed ::DEBUG_DEADLOCK_DETECTOR_LEVEL from 0 to 0
941515141 IDLE bgLoop Looping process deadlockdetectorlevel started
941515145 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.114.9') (::FTPDIR '') (::HTTPURL 'http://131.215.114.9/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/usr1/grid/ldas') (::GRIDFTPDIR '/usr1/grid/ldas') (::LDAS_GATEWAY 'ldas-cit 131.215.114.9') (::LDAS_SYSTEM 'ldas-cit') (::RUNCODE 'LDAS-CIT')
941515146 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.114.9') (::FTPDIR '') (::HTTPURL 'http://131.215.114.9/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/usr1/grid/ldas') (::GRIDFTPDIR '/usr1/grid/ldas') (::LDAS_GATEWAY 'ldas-cit 131.215.114.9') (::LDAS_SYSTEM 'ldas-cit') (::RUNCODE 'LDAS-CIT')
941517559 LDAS-CIT1382389 operator_callback begin processing LDAS-CIT1382389
941517559 LDAS-CIT1382389 standaloneWrapper ingesting files 3 from standalone wrapper: /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/output_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd
941517563 LDAS-CIT1382389 processDataBucket bad items in data bucket: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/output_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd'
941517563 LDAS-CIT1382389 ::eventmon::checkBuckets process data bucket error: processDataBucket: bad items in data bucket: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/output_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd' (item: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd') (type: 'text html file'), contents=
941517563 LDAS-CIT1382389 killJob products to remove
941517563 LDAS-CIT1382389 killJob LDAS-CIT1382389 processed return code=1, called with rc=1 {process data bucket error: processDataBucket: bad items in data bucket: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/output_1.ilwd /ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd' (item: '/ldas_outgoing/jobs/LDAS-CIT_138/LDAS-CIT1382389/www.ldas-dev.ligo.caltech.edu/ldas_outgoing/test/data/putStandAlone/process_2.ilwd') (type: 'text html file'), contents=}
941517563 LDAS-CIT1382389 reattach done processing LDAS-CIT1382389, closing sock7