The mpi API on beowulf at Caltech-CIT

Legend:
green ball Normal status or debugging message
yellow ball Notable condition which may be a non-fatal error
orange ball Error condition not fatal to job
red ball Error condition fatal to job
blue ball Notable condition which is not an error
purple ball Currently undefined
email Condition requires email notification of the responsible administrator of this API
telephone Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Caltech-CIT

06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 STARTUP archiveLog file "/ldas_outgoing/logs/LDASmpi.log.html" already closed. (archived as /ldas_outgoing/logs/archive/mpiAPI/LDASmpi.834835372)
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 STARTUP closeListenSock no cid registered for service 'data'
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 STARTUP mpi::init unused data port 10021 closed
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 STARTUP mpi::init port 10021 (jobstate) opened on beowulf as sock7
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 STARTUP bgLoop Looping process watchlogs started
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 STARTUP openListenSock port 10019 (operator) opened on beowulf as sock8
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 STARTUP openListenSock port 10020 (emergency) opened on beowulf as sock9
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 STARTUP leakLogger inital size of mpi API: 23092 kB
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 STARTUP bgLoop Looping process etchosts started
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 IDLE bgLoop Looping process statpagefile started
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 IDLE bgLoop Looping process killedjobreaper started
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 IDLE bgLoop Looping process logrotate started
06/20/06-03:47:40 PDT 06/20/06-10:47:40 GMT 834835674 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.114.9') (::FTPDIR '') (::HTTPURL 'http://131.215.114.9/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/usr1/grid/ldas') (::GRIDFTPDIR '/usr1/grid/ldas') (::LDAS_GATEWAY 'ldas-cit 131.215.114.9') (::LDAS_SYSTEM 'ldas-cit') (::RUNCODE 'LDAS-CIT')
06/20/06-03:47:45 PDT 06/20/06-10:47:45 GMT 834835679 STARTUP mpi::killAllMpirun cleaning up for user ldas
06/20/06-03:47:46 PDT 06/20/06-10:47:46 GMT 834835680 STARTUP mpi::killAllMpirun ran kill 10 times in 1.120 seconds
06/20/06-03:47:46 PDT 06/20/06-10:47:46 GMT 834835680 STARTUP mpi::prestartLamds running lamboot for user search01
06/20/06-03:47:47 PDT 06/20/06-10:47:47 GMT 834835681 STARTUP mpi::prestartLamds running lamboot for user search02
06/20/06-03:47:47 PDT 06/20/06-10:47:47 GMT 834835681 STARTUP mpi::prestartLamds running lamboot for user search03
06/20/06-03:47:48 PDT 06/20/06-10:47:48 GMT 834835682 STARTUP mpi::prestartLamds running lamboot for user search04
06/20/06-03:47:49 PDT 06/20/06-10:47:49 GMT 834835683 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.114.9') (::FTPDIR '') (::HTTPURL 'http://131.215.114.9/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/usr1/grid/ldas') (::GRIDFTPDIR '/usr1/grid/ldas') (::LDAS_GATEWAY 'ldas-cit 131.215.114.9') (::LDAS_SYSTEM 'ldas-cit') (::RUNCODE 'LDAS-CIT')
06/20/06-03:47:50 PDT 06/20/06-10:47:50 GMT 834835684 STARTUP mpi::prestartLamds STARTUP search01 beowulf ok!
06/20/06-03:47:50 PDT 06/20/06-10:47:50 GMT 834835684 STARTUP mpi::prestartLamds STARTUP search02 beowulf ok!
06/20/06-03:47:51 PDT 06/20/06-10:47:51 GMT 834835685 STARTUP mpi::prestartLamds STARTUP search03 beowulf ok!
06/20/06-03:47:52 PDT 06/20/06-10:47:52 GMT 834835686 STARTUP mpi::prestartLamds STARTUP search04 beowulf ok!
06/20/06-03:47:52 PDT 06/20/06-10:47:52 GMT 834835686 STARTUP mpi::killAllMpirun {ldas@beowulf:mpirun: child process exited abnormally} {ldas@beowulf:wrapperAPI: child process exited abnormally} {ldas@beowulf:lamd: child process exited abnormally}
06/20/06-03:48:00 PDT 06/20/06-10:48:00 GMT 834835694 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
06/20/06-05:32:56 PDT 06/20/06-12:32:56 GMT 834841990 SHUTDOWN closeListenSock port 10019 (sock8) (operator) closed on beowulf
pehrens@ligo.caltech.edu 834841990 SHUTDOWN mpi::sHuTdOwN Subject: LDAS CIT mpi shutdown at 834841990 ( 06/20/06 05:32:56 PDT ); Body: mpi shutting down NOW
06/20/06-05:32:57 PDT 06/20/06-12:32:57 GMT 834841991 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search01'
06/20/06-05:32:59 PDT 06/20/06-12:32:59 GMT 834841993 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search02'
06/20/06-05:33:00 PDT 06/20/06-12:33:00 GMT 834841994 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search03'
06/20/06-05:33:01 PDT 06/20/06-12:33:01 GMT 834841995 SHUTDOWN ::mpi::atExit calling lam::halt for user 'search04'
06/20/06-05:33:04 PDT 06/20/06-12:33:04 GMT 834841998 SHUTDOWN closeListenSock port 10020 (sock9) (emergency) closed on beowulf
06/20/06-05:33:04 PDT 06/20/06-12:33:04 GMT 834841998 SHUTDOWN closeListenSock no cid registered for service 'data'
06/20/06-05:33:05 PDT 06/20/06-12:33:05 GMT 834841999 SHUTDOWN closeLog /ldas_outgoing/logs/LDASmpi.log.html (file5) closed