The mpi API on metaserver at Caltech-Dev

Legend:

Normal status or debugging message

Notable condition which may be a non-fatal error

Error condition not fatal to job

Error condition fatal to job

Notable condition which is not an error

Currently undefined

Condition requires email notification of the responsible administrator of this API

Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Caltech-Dev

911342712 STARTUP archiveLog file "/ldas_outgoing/logs/LDASmpi.log.html" already closed. (archived as /ldas_outgoing/logs/archive/mpiAPI/LDASmpi.911342291)
911342712 STARTUP execOverload overloaded exec
911342712 STARTUP closeListenSock no cid registered for service 'data'
911342712 STARTUP mpi::init unused data port 10021 closed
911342712 STARTUP mpi::init port 10021 (jobstate) opened on metaserver as sock5
911342712 STARTUP bgLoop Looping process watchlogs started
911342712 STARTUP openListenSock port 10019 (operator) opened on metaserver as sock6
911342712 STARTUP openListenSock port 10020 (emergency) opened on metaserver as sock7
911342712 STARTUP setResourceLimit vmemoryuse=unlimited; datasize=unlimited; core=unlimited; maxproc=65536; descriptors=1024; memorylocked=32768; filesize=unlimited; cputime=unlimited
911342712 STARTUP leakLogger inital size of mpi API: 14064 kB
911342712 STARTUP bgLoop Looping process etchosts started
911342712 STARTUP mpi Trying API
911342712 STARTUP mpi API yes
911342712 STARTUP mpi Trying LDAS_SYSTEM
911342712 STARTUP mpi LDAS_SYSTEM yes
911342712 IDLE exec exec /usr/sbin/lsof -b -p 31255
911342712 IDLE bgLoop Looping process statpagefile started
911342712 IDLE bgLoop Looping process killedjobreaper started
911342712 IDLE bgLoop Looping process logrotate started
911342717 STARTUP mpi::killAllMpirun cleaning up for user ldas
911342717 IDLE ::exec exec /usr/bin/ssh -n -l ldas beowulf /bin/rm -rf /tmp/lam-ldas@*
911342720 IDLE ::exec exec /usr/bin/ssh -n beowulf sudo pkill -9 -u search01,search02,search03,search04,search05,search06,search07,search08,search09,search10,search11,search12,search13,search14,search15,search16 wrapperAPI
911342723 IDLE ::exec exec /usr/bin/ssh -n beowulf sudo pkill -9 -u search01,search02,search03,search04,search05,search06,search07,search08,search09,search10,search11,search12,search13,search14,search15,search16 lamd
911342726 STARTUP mpi::killAllMpirun ran kill 10 times in 9.089 seconds
911342726 STARTUP mpi::prestartLamds running lamboot for user search01
911342726 IDLE ::exec exec scp -B conf.lam search01@metaserver:
911342726 IDLE ::exec exec scp -B conf.lam search01@metaserver:
911342726 STARTUP mpi::prestartLamds Subject: LDAS-DEV mpiAPI lamboot error!; Body: lam::boot: lam::pushSchema: search01@metaserver: Permission denied (publickey,gssapi-with-mic,password). lost connection
911342726 STARTUP mpi::killAllMpirun {mpi::cleanupTmp: ldas@beowulf: ssh: connect to host beowulf port 22: No route to host} {ldas@beowulf:wrapperAPI: ssh: connect to host beowulf port 22: No route to host} {ldas@beowulf:lamd: ssh: connect to host beowulf port 22: No route to host}
911342726 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.115.248') (::FTPDIR '') (::HTTPURL 'http://131.215.115.248/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas-dev 131.215.115.248') (::LDAS_SYSTEM 'ldas-dev') (::RUNCODE 'LDAS-DEV')
911342726 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.115.248') (::FTPDIR '') (::HTTPURL 'http://131.215.115.248/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas-dev 131.215.115.248') (::LDAS_SYSTEM 'ldas-dev') (::RUNCODE 'LDAS-DEV')
911342732 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
911342738 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
911342881 LDAS-DEV98131907 operator_callback begin processing LDAS-DEV98131907
911342881 LDAS-DEV98131907 reattach done processing LDAS-DEV98131907, closing sock3
911342884 LDAS-DEV98131905 operator_callback begin processing LDAS-DEV98131905
911342884 LDAS-DEV98131905 reattach done processing LDAS-DEV98131905, closing sock3
911342884 LDAS-DEV98131904 operator_callback begin processing LDAS-DEV98131904
911342884 LDAS-DEV98131904 reattach done processing LDAS-DEV98131904, closing sock8
911342908 LDAS-DEV98131917 operator_callback begin processing LDAS-DEV98131917
911342908 LDAS-DEV98131917 reattach done processing LDAS-DEV98131917, closing sock3
911342909 LDAS-DEV98131918 operator_callback begin processing LDAS-DEV98131918
911342909 LDAS-DEV98131918 reattach done processing LDAS-DEV98131918, closing sock3
911342926 LDAS-DEV98131922 operator_callback begin processing LDAS-DEV98131922
911342926 LDAS-DEV98131922 reattach done processing LDAS-DEV98131922, closing sock3
911342943 LDAS-DEV98131932 operator_callback begin processing LDAS-DEV98131932
911342943 LDAS-DEV98131932 reattach done processing LDAS-DEV98131932, closing sock3
911342976 LDAS-DEV98131948 operator_callback begin processing LDAS-DEV98131948
911342976 LDAS-DEV98131948 reattach done processing LDAS-DEV98131948, closing sock3
911343010 LDAS-DEV98131956 operator_callback begin processing LDAS-DEV98131956
911343010 LDAS-DEV98131956 reattach done processing LDAS-DEV98131956, closing sock3
911343029 LDAS-DEV98131963 operator_callback begin processing LDAS-DEV98131963
911343029 LDAS-DEV98131963 reattach done processing LDAS-DEV98131963, closing sock3
911343048 LDAS-DEV98131969 operator_callback begin processing LDAS-DEV98131969
911343048 LDAS-DEV98131969 reattach done processing LDAS-DEV98131969, closing sock3
911343107 LDAS-DEV98131990 operator_callback begin processing LDAS-DEV98131990
911343107 LDAS-DEV98131990 reattach done processing LDAS-DEV98131990, closing sock3
911343125 LDAS-DEV98132004 operator_callback begin processing LDAS-DEV98132004
911343125 LDAS-DEV98132004 reattach done processing LDAS-DEV98132004, closing sock3
911343140 LDAS-DEV98132013 operator_callback begin processing LDAS-DEV98132013
911343140 LDAS-DEV98132013 reattach done processing LDAS-DEV98132013, closing sock3
911343156 LDAS-DEV98132022 operator_callback begin processing LDAS-DEV98132022
911343156 LDAS-DEV98132022 reattach done processing LDAS-DEV98132022, closing sock3
911343190 LDAS-DEV98132035 operator_callback begin processing LDAS-DEV98132035
911343190 LDAS-DEV98132035 reattach done processing LDAS-DEV98132035, closing sock3
911343198 LDAS-DEV98132038 operator_callback begin processing LDAS-DEV98132038
911343198 LDAS-DEV98132038 reattach done processing LDAS-DEV98132038, closing sock3
911343207 LDAS-DEV98132041 operator_callback begin processing LDAS-DEV98132041
911343207 LDAS-DEV98132041 reattach done processing LDAS-DEV98132041, closing sock3