The wrapper API on beowulf at Caltech-CIT

Legend:
green ball Normal status or debugging message
yellow ball Notable condition which may be a non-fatal error
orange ball Error condition not fatal to job
red ball Error condition fatal to job
blue ball Notable condition which is not an error
purple ball Currently undefined
email Condition requires email notification of the responsible administrator of this API
telephone Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Caltech-CIT

12/21/05-15:30:10 PST 12/21/05-23:30:10 GMT 819243023 LDAS-CIT2367725 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/21/05-15:31:43 PST 12/21/05-23:31:43 GMT 819243116 LDAS-CIT2367726 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/21/05-15:32:54 PST 12/21/05-23:32:54 GMT 819243187 LDAS-CIT2367727 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/21/05-15:34:12 PST 12/21/05-23:34:12 GMT 819243265 LDAS-CIT2367728 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/21/05-15:35:26 PST 12/21/05-23:35:26 GMT 819243339 LDAS-CIT2367729 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/21/05-15:36:41 PST 12/21/05-23:36:41 GMT 819243414 LDAS-CIT2367730 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/21/05-15:37:55 PST 12/21/05-23:37:55 GMT 819243488 LDAS-CIT2367731 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/21/05-15:39:22 PST 12/21/05-23:39:22 GMT 819243575 LDAS-CIT2367732 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/21/05-15:40:41 PST 12/21/05-23:40:41 GMT 819243654 LDAS-CIT2367734 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/21/05-15:41:59 PST 12/21/05-23:41:59 GMT 819243732 LDAS-CIT2367735 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-15:44:25 PST 12/29/05-23:44:25 GMT 819935078 LDAS-CIT2367980 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-15:49:48 PST 12/29/05-23:49:48 GMT 819935401 LDAS-CIT2367981 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-15:50:59 PST 12/29/05-23:50:59 GMT 819935472 LDAS-CIT2367982 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-15:53:14 PST 12/29/05-23:53:14 GMT 819935607 LDAS-CIT2367983 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-15:54:30 PST 12/29/05-23:54:30 GMT 819935683 LDAS-CIT2367984 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-15:55:46 PST 12/29/05-23:55:46 GMT 819935759 LDAS-CIT2367985 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-15:57:01 PST 12/29/05-23:57:01 GMT 819935834 LDAS-CIT2367986 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-16:03:23 PST 12/30/05-00:03:23 GMT 819936216 LDAS-CIT2367987 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-16:05:01 PST 12/30/05-00:05:01 GMT 819936314 LDAS-CIT2367988 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
12/29/05-16:06:22 PST 12/30/05-00:06:22 GMT 819936395 LDAS-CIT2367989 mpi::newJobCallback Job stalled. possible lam error, see logs. Wrapper API failed to report initialisation.
01/03/06-16:22:19 PST 01/04/06-00:22:19 GMT 820369353 LDAS-CIT2368138 mpi::filterLogEntries 2368138.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) Rank3(beowulf) }
01/03/06-16:22:21 PST 01/04/06-00:22:21 GMT 820369355 LDAS-CIT2368138 mpi::filterLogEntries 2368138.1:error {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
01/03/06-16:22:22 PST 01/04/06-00:22:22 GMT 820369356 LDAS-CIT2368138 mpi::newJobCallback {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
01/03/06-16:23:44 PST 01/04/06-00:23:44 GMT 820369438 LDAS-CIT2368139 mpi::filterLogEntries 2368139.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
01/03/06-16:23:46 PST 01/04/06-00:23:46 GMT 820369440 LDAS-CIT2368139 mpi::filterLogEntries 2368139.1:progress 100.00%
01/03/06-16:23:46 PST 01/04/06-00:23:46 GMT 820369440 LDAS-CIT2368139 mpi::filterLogEntries 2368139.1:projected_ratio 0.00016
01/03/06-16:23:46 PST 01/04/06-00:23:46 GMT 820369440 LDAS-CIT2368139 mpi::filterLogEntries 2368139.2:finalizing { totalTime=1.77705 sentObjects=3; Rank0 (inPut: create=0.33003 send=0.15185 ilwd=0.22918; outPut: create=0.08006 send=0.52415 ilwd=0.04421; metadata: (un)pack=0.00017 bcast=0.01678; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01595; max memory=270336) Rank1 (inPut: create=0.29048 send=0.18473; outPut: create=0.01590 send=0.55438; metadata: (un)pack=0.00012 bcast=0.01743; IS=0.00001; CD=0.00000; AS=0.05660; FO=0.00341; FS=0.00000; LB=0.12867; max memory=184320) }
01/03/06-16:52:15 PST 01/04/06-00:52:15 GMT 820371149 LDAS-CIT2368141 mpi::filterLogEntries 2368141.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
01/03/06-16:52:16 PST 01/04/06-00:52:16 GMT 820371150 LDAS-CIT2368141 mpi::filterLogEntries 2368141.1:progress 100.00%
01/03/06-16:52:16 PST 01/04/06-00:52:16 GMT 820371150 LDAS-CIT2368141 mpi::filterLogEntries 2368141.1:projected_ratio 0.00095
01/03/06-16:52:16 PST 01/04/06-00:52:16 GMT 820371150 LDAS-CIT2368141 mpi::filterLogEntries 2368141.2:finalizing { totalTime=0.791599 sentObjects=3; Rank0 (inPut: create=0.04962 send=0.00584 ilwd=0.04277; outPut: create=0.01832 send=0.04971 ilwd=0.04426; metadata: (un)pack=0.00018 bcast=0.01665; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01595; max memory=4112384) Rank1 (inPut: create=0.04782 send=0.01073; outPut: create=0.01592 send=0.04703; metadata: (un)pack=0.00018 bcast=0.01728; IS=0.00008; CD=0.24382; AS=0.05691; FO=0.00022; FS=0.00003; LB=0.04600; max memory=8085504) }
01/03/06-17:13:56 PST 01/04/06-01:13:56 GMT 820372450 LDAS-CIT2368157 mpi::filterLogEntries 2368157.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) Rank3(beowulf) }
01/03/06-17:13:58 PST 01/04/06-01:13:58 GMT 820372452 LDAS-CIT2368157 mpi::filterLogEntries 2368157.1:error {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
01/03/06-17:13:58 PST 01/04/06-01:13:58 GMT 820372452 LDAS-CIT2368157 mpi::newJobCallback {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
01/03/06-17:14:26 PST 01/04/06-01:14:26 GMT 820372480 LDAS-CIT2368158 mpi::filterLogEntries 2368158.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) Rank3(beowulf) }
01/03/06-17:14:59 PST 01/04/06-01:14:59 GMT 820372513 LDAS-CIT2368158 mpi::filterLogEntries 2368158.1:progress 100.00%
01/03/06-17:14:59 PST 01/04/06-01:14:59 GMT 820372513 LDAS-CIT2368158 mpi::filterLogEntries 2368158.1:projected_ratio 0.00907
01/03/06-17:14:59 PST 01/04/06-01:14:59 GMT 820372513 LDAS-CIT2368158 jobstate response: '2368158.1:cont'
01/03/06-17:15:00 PST 01/04/06-01:15:00 GMT 820372514 LDAS-CIT2368158 mpi::filterLogEntries 2368158.2:progress 100.00%
01/03/06-17:15:00 PST 01/04/06-01:15:00 GMT 820372514 LDAS-CIT2368158 mpi::filterLogEntries 2368158.2:projected_ratio 0.01820
01/03/06-17:15:00 PST 01/04/06-01:15:00 GMT 820372514 LDAS-CIT2368158 mpi::filterLogEntries 2368158.3:finalizing { totalTime=33.8562 sentObjects=4; Rank0 (inPut: create=0.61130 send=0.63898 ilwd=0.38000; outPut: create=0.03254 send=0.09641 ilwd=0.08062; metadata: (un)pack=0.00040 bcast=0.03431; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12024; max memory=1335296) Rank3 (inPut: create=0.55345 send=0.84205; outPut: create=0.00000 send=0.04631; metadata: (un)pack=0.00012 bcast=0.00340; IS=0.97743; CD=0.14386; AS=0.00050; FO=0.00000; FS=0.00504; LB=0.00001; max memory=954368) Rank1 (inPut: create=0.55345 send=0.83561; outPut: create=0.01589 send=0.09473; metadata: (un)pack=0.00025 bcast=0.01751; IS=0.00021; CD=0.03047; AS=30.37700; FO=0.00004; FS=0.00006; LB=0.03192; max memory=413696) Rank2 (inPut: create=0.55530 send=0.74600; outPut: create=0.03197 send=0.35053; metadata: (un)pack=0.00038 bcast=0.03405; IS=0.98754; CD=0.14360; AS=30.20100; FO=0.00009; FS=0.03257; LB=0.00001; max memory=954368) }
01/03/06-17:15:13 PST 01/04/06-01:15:13 GMT 820372527 LDAS-CIT2368159 mpi::filterLogEntries 2368159.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
01/03/06-17:15:13 PST 01/04/06-01:15:13 GMT 820372527 LDAS-CIT2368159 mpi::filterLogEntries 2368159.1:progress 100.00%
01/03/06-17:15:13 PST 01/04/06-01:15:13 GMT 820372527 LDAS-CIT2368159 mpi::filterLogEntries 2368159.1:projected_ratio 0.00097
01/03/06-17:15:14 PST 01/04/06-01:15:14 GMT 820372528 LDAS-CIT2368159 mpi::filterLogEntries 2368159.2:finalizing { totalTime=0.778204 sentObjects=3; Rank0 (inPut: create=0.04464 send=0.00578 ilwd=0.03773; outPut: create=0.01886 send=0.08513 ilwd=0.01028; metadata: (un)pack=0.00020 bcast=0.01678; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01594; max memory=4112384) Rank1 (inPut: create=0.04280 send=0.01066; outPut: create=0.03194 send=0.05611; metadata: (un)pack=0.00018 bcast=0.03337; IS=0.00009; CD=0.24405; AS=0.05836; FO=0.00023; FS=0.00003; LB=0.05003; max memory=8085504) }
01/03/06-17:15:32 PST 01/04/06-01:15:32 GMT 820372546 LDAS-CIT2368160 mpi::filterLogEntries 2368160.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
01/03/06-17:15:33 PST 01/04/06-01:15:33 GMT 820372547 LDAS-CIT2368160 mpi::filterLogEntries 2368160.1:progress 100.00%
01/03/06-17:15:33 PST 01/04/06-01:15:33 GMT 820372547 LDAS-CIT2368160 mpi::filterLogEntries 2368160.1:projected_ratio 0.00017
01/03/06-17:15:34 PST 01/04/06-01:15:34 GMT 820372548 LDAS-CIT2368160 mpi::filterLogEntries 2368160.2:finalizing { totalTime=1.80389 sentObjects=3; Rank0 (inPut: create=0.32988 send=0.15696 ilwd=0.23035; outPut: create=0.09375 send=0.52561 ilwd=0.00989; metadata: (un)pack=0.00012 bcast=0.03274; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01594; max memory=270336) Rank1 (inPut: create=0.29489 send=0.19082; outPut: create=0.03190 send=0.55992; metadata: (un)pack=0.00013 bcast=0.03344; IS=0.00001; CD=0.00000; AS=0.06053; FO=0.00385; FS=0.00000; LB=0.12768; max memory=184320) }
01/03/06-17:16:22 PST 01/04/06-01:16:22 GMT 820372596 LDAS-CIT2368161 mpi::filterLogEntries 2368161.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
01/03/06-17:16:22 PST 01/04/06-01:16:22 GMT 820372596 LDAS-CIT2368161 mpi::filterLogEntries 2368161.1:progress 100.00%
01/03/06-17:16:22 PST 01/04/06-01:16:22 GMT 820372596 LDAS-CIT2368161 mpi::filterLogEntries 2368161.1:projected_ratio 0.00000
01/03/06-17:16:22 PST 01/04/06-01:16:22 GMT 820372596 LDAS-CIT2368161 mpi::filterLogEntries 2368161.2:finalizing { totalTime=0.279918 sentObjects=2; Rank0 (inPut: create=0.03913 send=0.00061 ilwd=0.03460; outPut: create=0.00000 send=0.01818 ilwd=0.00982; metadata: (un)pack=0.00013 bcast=0.00095; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01598; max memory=458752) Rank1 (inPut: create=0.03809 send=0.00119; outPut: create=0.00000 send=0.01673; metadata: (un)pack=0.00024 bcast=0.00169; IS=0.00010; CD=0.00052; AS=0.00013; FO=0.00000; FS=0.00004; LB=0.03190; max memory=647168) }
01/03/06-17:16:35 PST 01/04/06-01:16:35 GMT 820372609 LDAS-CIT2368162 mpi::filterLogEntries 2368162.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
01/03/06-17:16:45 PST 01/04/06-01:16:45 GMT 820372619 LDAS-CIT2368162 mpi::filterLogEntries 2368162.1:progress 68.37%
01/03/06-17:16:45 PST 01/04/06-01:16:45 GMT 820372619 LDAS-CIT2368162 mpi::filterLogEntries 2368162.1:projected_ratio 0.02519
01/03/06-17:16:45 PST 01/04/06-01:16:45 GMT 820372619 LDAS-CIT2368162 jobstate response: '2368162.1:cont'
01/03/06-17:16:45 PST 01/04/06-01:16:45 GMT 820372619 LDAS-CIT2368162 mpi::filterLogEntries 2368162.2:progress 100.00%
01/03/06-17:16:45 PST 01/04/06-01:16:45 GMT 820372619 LDAS-CIT2368162 mpi::filterLogEntries 2368162.2:projected_ratio 0.02544
01/03/06-17:16:46 PST 01/04/06-01:16:46 GMT 820372620 LDAS-CIT2368162 mpi::filterLogEntries 2368162.3:finalizing { totalTime=10.7016 sentObjects=4; Rank0 (inPut: create=0.33063 send=0.15643 ilwd=0.23033; outPut: create=0.10523 send=0.11599 ilwd=0.18761; metadata: (un)pack=0.01707 bcast=0.06464; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12176; max memory=6844416) Rank1 (inPut: create=0.29617 send=0.19005; outPut: create=0.08408 send=0.40431; metadata: (un)pack=0.01411 bcast=0.05354; IS=0.00009; CD=0.06331; AS=9.15821; FO=0.00151; FS=0.00355; LB=0.06325; max memory=348160) }
01/03/06-17:17:03 PST 01/04/06-01:17:03 GMT 820372637 LDAS-CIT2368163 mpi::filterLogEntries 2368163.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
01/03/06-17:17:03 PST 01/04/06-01:17:03 GMT 820372637 LDAS-CIT2368163 mpi::filterLogEntries 2368163.1:progress 100.00%
01/03/06-17:17:03 PST 01/04/06-01:17:03 GMT 820372637 LDAS-CIT2368163 mpi::filterLogEntries 2368163.1:projected_ratio 0.00001
01/03/06-17:17:04 PST 01/04/06-01:17:04 GMT 820372638 LDAS-CIT2368163 mpi::filterLogEntries 2368163.2:finalizing { totalTime=0.359964 sentObjects=3; Rank0 (inPut: create=0.04422 send=0.00266 ilwd=0.03825; outPut: create=0.01638 send=0.02924 ilwd=0.00864; metadata: (un)pack=0.00034 bcast=0.01669; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01597; max memory=897024) Rank1 (inPut: create=0.04243 send=0.00482; outPut: create=0.01597 send=0.03064; metadata: (un)pack=0.00035 bcast=0.01712; IS=0.00009; CD=0.00197; AS=0.00050; FO=0.00004; FS=0.00003; LB=0.03191; max memory=2060288) }
01/03/06-17:17:38 PST 01/04/06-01:17:38 GMT 820372672 LDAS-CIT2368164 mpi::filterLogEntries 2368164.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.1:progress 0.00%
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.1:projected_ratio 0.00018
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.2:progress 10.00%
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.2:projected_ratio 0.00036
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.3:progress 20.00%
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.3:projected_ratio 0.00053
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.4:progress 30.00%
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.4:projected_ratio 0.00071
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.5:progress 40.00%
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.5:projected_ratio 0.00088
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.6:progress 50.00%
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 mpi::filterLogEntries 2368164.6:projected_ratio 0.00106
01/03/06-17:17:40 PST 01/04/06-01:17:40 GMT 820372674 LDAS-CIT2368164 jobstate response: '2368164.6:cont'
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.7:progress 60.00%
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.7:projected_ratio 0.00123
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.8:progress 70.00%
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.8:projected_ratio 0.00140
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.9:progress 80.00%
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.9:projected_ratio 0.00158
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.10:progress 90.00%
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.10:projected_ratio 0.00175
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.11:progress 100.00%
01/03/06-17:17:41 PST 01/04/06-01:17:41 GMT 820372675 LDAS-CIT2368164 mpi::filterLogEntries 2368164.11:projected_ratio 0.00175
01/03/06-17:17:42 PST 01/04/06-01:17:42 GMT 820372676 LDAS-CIT2368164 mpi::filterLogEntries 2368164.12:finalizing { totalTime=3.12598 sentObjects=4; Rank0 (inPut: create=0.22232 send=0.09413 ilwd=0.15591; outPut: create=0.19279 send=0.53133 ilwd=0.02266; metadata: (un)pack=0.00056 bcast=0.19145; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12510; max memory=1413120) Rank1 (inPut: create=0.19964 send=0.11686; outPut: create=0.19595 send=0.52904; metadata: (un)pack=0.00050 bcast=0.19620; IS=0.07587; CD=0.09137; AS=1.53216; FO=0.00033; FS=0.00295; LB=0.03203; max memory=18370560) }
01/03/06-17:17:59 PST 01/04/06-01:17:59 GMT 820372693 LDAS-CIT2368165 mpi::filterLogEntries 2368165.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) Rank3(beowulf) }
01/03/06-17:18:02 PST 01/04/06-01:18:02 GMT 820372696 LDAS-CIT2368165 mpi::filterLogEntries 2368165.1:progress 0.00%
01/03/06-17:18:02 PST 01/04/06-01:18:02 GMT 820372696 LDAS-CIT2368165 mpi::filterLogEntries 2368165.1:projected_ratio 0.00000
01/03/06-17:18:02 PST 01/04/06-01:18:02 GMT 820372696 LDAS-CIT2368165 jobstate response: '2368165.1:cont'
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.2:progress 0.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.2:projected_ratio 0.00000
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.3:progress 25.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.3:projected_ratio 0.00000
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.4:progress 50.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.4:projected_ratio 0.00025
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.5:progress 50.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.5:projected_ratio 0.00055
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.6:progress 50.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.6:projected_ratio 0.00055
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.7:progress 50.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.7:projected_ratio 0.00055
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.8:progress 75.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.8:projected_ratio 0.00063
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.9:progress 75.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.9:projected_ratio 0.00090
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.10:progress 75.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.10:projected_ratio 0.00090
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.11:progress 100.00%
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.11:projected_ratio 0.00095
01/03/06-17:18:03 PST 01/04/06-01:18:03 GMT 820372697 LDAS-CIT2368165 mpi::filterLogEntries 2368165.12:finalizing { totalTime=4.11142 sentObjects=3; Rank0 (inPut: create=0.38237 send=0.44044 ilwd=0.22766; outPut: create=0.04670 send=0.28862 ilwd=0.03961; metadata: (un)pack=0.00561 bcast=0.03575; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.13024; max memory=2801664) Rank3 (inPut: create=0.34695 send=0.57565; outPut: create=0.00000 send=0.16222; metadata: (un)pack=0.00006 bcast=0.00350; IS=0.00017; CD=1.76633; AS=0.27349; FO=0.00000; FS=0.00462; LB=0.00001; max memory=917504) Rank2 (inPut: create=0.34696 send=0.57016; outPut: create=0.00000 send=0.22831; metadata: (un)pack=0.00014 bcast=0.00286; IS=0.00017; CD=1.76396; AS=0.37125; FO=0.00000; FS=0.00475; LB=0.00001; max memory=1089536) Rank1 (inPut: create=0.34680 send=0.50783; outPut: create=0.04153 send=0.62676; metadata: (un)pack=0.00466 bcast=0.03308; IS=0.00014; CD=0.85015; AS=0.03810; FO=0.00056; FS=0.00422; LB=0.06733; max memory=339968) }
01/03/06-17:18:20 PST 01/04/06-01:18:20 GMT 820372714 LDAS-CIT2368166 mpi::filterLogEntries 2368166.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) }
01/03/06-17:18:34 PST 01/04/06-01:18:34 GMT 820372728 LDAS-CIT2368166 mpi::filterLogEntries 2368166.1:error {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }
01/03/06-17:18:34 PST 01/04/06-01:18:34 GMT 820372728 LDAS-CIT2368166 mpi::newJobCallback {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }
01/04/06-08:51:40 PST 01/04/06-16:51:40 GMT 820428714 LDAS-CIT2368212 mpi::filterLogEntries 2368212.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) Rank3(beowulf) }
01/04/06-08:51:42 PST 01/04/06-16:51:42 GMT 820428716 LDAS-CIT2368212 mpi::filterLogEntries 2368212.1:error {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
01/04/06-08:51:42 PST 01/04/06-16:51:42 GMT 820428716 LDAS-CIT2368212 mpi::newJobCallback {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
01/04/06-08:53:06 PST 01/04/06-16:53:06 GMT 820428800 LDAS-CIT2368213 mpi::filterLogEntries 2368213.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
01/04/06-08:53:08 PST 01/04/06-16:53:08 GMT 820428802 LDAS-CIT2368213 mpi::filterLogEntries 2368213.1:progress 100.00%
01/04/06-08:53:08 PST 01/04/06-16:53:08 GMT 820428802 LDAS-CIT2368213 mpi::filterLogEntries 2368213.1:projected_ratio 0.00017
01/04/06-08:53:08 PST 01/04/06-16:53:08 GMT 820428802 LDAS-CIT2368213 mpi::filterLogEntries 2368213.2:finalizing { totalTime=1.79095 sentObjects=3; Rank0 (inPut: create=0.33092 send=0.15101 ilwd=0.22936; outPut: create=0.07611 send=0.52437 ilwd=0.01012; metadata: (un)pack=0.00010 bcast=0.01666; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01594; max memory=270336) Rank1 (inPut: create=0.30045 send=0.18523; outPut: create=0.01590 send=0.56160; metadata: (un)pack=0.00015 bcast=0.01737; IS=0.00001; CD=0.00000; AS=0.06035; FO=0.00423; FS=0.00000; LB=0.11120; max memory=184320) }
02/10/06-13:28:00 PST 02/10/06-21:28:00 GMT 823642094 LDAS-CIT2369081 mpi::filterLogEntries 2369081.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
02/10/06-13:28:03 PST 02/10/06-21:28:03 GMT 823642097 LDAS-CIT2369081 mpi::filterLogEntries 2369081.1:progress 100.00%
02/10/06-13:28:03 PST 02/10/06-21:28:03 GMT 823642097 LDAS-CIT2369081 mpi::filterLogEntries 2369081.1:projected_ratio 0.00016
02/10/06-13:28:03 PST 02/10/06-21:28:03 GMT 823642097 LDAS-CIT2369081 jobstate response: '2369081.1:cont'
02/10/06-13:28:06 PST 02/10/06-21:28:06 GMT 823642100 LDAS-CIT2369081 mpi::filterLogEntries 2369081.2:finalizing { totalTime=5.49651 sentObjects=3; Rank0 (inPut: create=2.14098 send=0.14619 ilwd=2.03991; outPut: create=0.07970 send=0.45996 ilwd=0.01641; metadata: (un)pack=0.00018 bcast=0.01682; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.11994; max memory=270336) Rank1 (inPut: create=2.10470 send=0.17978; outPut: create=0.01989 send=0.48892; metadata: (un)pack=0.00012 bcast=0.02144; IS=0.00001; CD=0.00000; AS=0.05685; FO=0.00324; FS=0.00000; LB=0.22092; max memory=184320) }
02/27/06-13:21:21 PST 02/27/06-21:21:21 GMT 825110495 LDAS-CIT2369603 mpi::filterLogEntries 2369603.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) Rank3(beowulf) }
02/27/06-13:21:27 PST 02/27/06-21:21:27 GMT 825110501 LDAS-CIT2369603 mpi::filterLogEntries 2369603.1:error {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
02/27/06-13:21:27 PST 02/27/06-21:21:27 GMT 825110501 LDAS-CIT2369603 mpi::newJobCallback {Rank 1 failed with error message: ( applySearch function ): Level 1: power $Id: ApplySearch.c,v 1.7 2003/12/09 17:39:03 julien Exp $ Status code 19: File error! function LALApplySearch, file ../../../../contrib/burstwrapper/src/ApplySearch.c, line 111 }
02/27/06-13:25:14 PST 02/27/06-21:25:14 GMT 825110728 LDAS-CIT2369604 mpi::filterLogEntries 2369604.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) Rank3(beowulf) }
02/27/06-13:25:50 PST 02/27/06-21:25:50 GMT 825110764 LDAS-CIT2369604 mpi::filterLogEntries 2369604.1:progress 100.00%
02/27/06-13:25:50 PST 02/27/06-21:25:50 GMT 825110764 LDAS-CIT2369604 mpi::filterLogEntries 2369604.1:projected_ratio 0.00901
02/27/06-13:25:50 PST 02/27/06-21:25:50 GMT 825110764 LDAS-CIT2369604 jobstate response: '2369604.1:cont'
02/27/06-13:25:50 PST 02/27/06-21:25:50 GMT 825110764 LDAS-CIT2369604 mpi::filterLogEntries 2369604.2:progress 100.00%
02/27/06-13:25:50 PST 02/27/06-21:25:50 GMT 825110764 LDAS-CIT2369604 mpi::filterLogEntries 2369604.2:projected_ratio 0.01807
02/27/06-13:25:51 PST 02/27/06-21:25:51 GMT 825110765 LDAS-CIT2369604 mpi::filterLogEntries 2369604.3:finalizing { totalTime=36.2778 sentObjects=4; Rank0 (inPut: create=3.24184 send=0.64147 ilwd=3.00968; outPut: create=0.03247 send=0.09295 ilwd=0.05761; metadata: (un)pack=0.00040 bcast=0.03431; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12022; max memory=1335296) Rank3 (inPut: create=3.18370 send=0.84136; outPut: create=0.00000 send=0.04829; metadata: (un)pack=0.00012 bcast=0.00351; IS=0.98512; CD=0.14610; AS=0.00047; FO=0.00000; FS=0.00481; LB=0.00001; max memory=954368) Rank1 (inPut: create=3.18371 send=0.83507; outPut: create=0.01581 send=0.07847; metadata: (un)pack=0.00031 bcast=0.01835; IS=0.00021; CD=0.03905; AS=30.15323; FO=0.00003; FS=0.00005; LB=0.03196; max memory=413696) Rank2 (inPut: create=3.18423 send=0.74687; outPut: create=0.03195 send=0.32226; metadata: (un)pack=0.00046 bcast=0.03329; IS=0.99144; CD=0.14488; AS=29.98925; FO=0.00008; FS=0.03225; LB=0.00001; max memory=954368) }
02/27/06-13:26:06 PST 02/27/06-21:26:06 GMT 825110780 LDAS-CIT2369605 mpi::filterLogEntries 2369605.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
02/27/06-13:26:07 PST 02/27/06-21:26:07 GMT 825110781 LDAS-CIT2369605 mpi::filterLogEntries 2369605.1:progress 100.00%
02/27/06-13:26:07 PST 02/27/06-21:26:07 GMT 825110781 LDAS-CIT2369605 mpi::filterLogEntries 2369605.1:projected_ratio 0.00086
02/27/06-13:26:07 PST 02/27/06-21:26:07 GMT 825110781 LDAS-CIT2369605 mpi::filterLogEntries 2369605.2:finalizing { totalTime=0.90759 sentObjects=3; Rank0 (inPut: create=0.11450 send=0.00523 ilwd=0.10821; outPut: create=0.01804 send=0.05383 ilwd=0.01000; metadata: (un)pack=0.00017 bcast=0.01662; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.01595; max memory=4112384) Rank1 (inPut: create=0.11277 send=0.00983; outPut: create=0.01594 send=0.05597; metadata: (un)pack=0.00017 bcast=0.01716; IS=0.00008; CD=0.17627; AS=0.05144; FO=0.00020; FS=0.00003; LB=0.03402; max memory=8085504) }
02/27/06-13:29:34 PST 02/27/06-21:29:34 GMT 825110988 LDAS-CIT2369606 mpi::filterLogEntries 2369606.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
02/27/06-13:29:37 PST 02/27/06-21:29:37 GMT 825110991 LDAS-CIT2369606 mpi::filterLogEntries 2369606.1:progress 100.00%
02/27/06-13:29:37 PST 02/27/06-21:29:37 GMT 825110991 LDAS-CIT2369606 mpi::filterLogEntries 2369606.1:projected_ratio 0.00016
02/27/06-13:29:37 PST 02/27/06-21:29:37 GMT 825110991 LDAS-CIT2369606 jobstate response: '2369606.1:cont'
02/27/06-13:29:41 PST 02/27/06-21:29:41 GMT 825110995 LDAS-CIT2369606 mpi::filterLogEntries 2369606.2:finalizing { totalTime=5.53201 sentObjects=3; Rank0 (inPut: create=2.13795 send=0.14742 ilwd=2.04005; outPut: create=0.09252 send=0.52338 ilwd=0.01087; metadata: (un)pack=0.00010 bcast=0.03264; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.11995; max memory=270336) Rank1 (inPut: create=2.10216 send=0.18150; outPut: create=0.03190 send=0.54221; metadata: (un)pack=0.00012 bcast=0.03329; IS=0.00001; CD=0.00000; AS=0.05588; FO=0.00323; FS=0.00000; LB=0.22492; max memory=184320) }
02/27/06-13:30:39 PST 02/27/06-21:30:39 GMT 825111053 LDAS-CIT2369611 mpi::filterLogEntries 2369611.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
02/27/06-13:30:51 PST 02/27/06-21:30:51 GMT 825111065 LDAS-CIT2369611 mpi::filterLogEntries 2369611.1:progress 68.37%
02/27/06-13:30:51 PST 02/27/06-21:30:51 GMT 825111065 LDAS-CIT2369611 mpi::filterLogEntries 2369611.1:projected_ratio 0.02511
02/27/06-13:30:51 PST 02/27/06-21:30:51 GMT 825111065 LDAS-CIT2369611 jobstate response: '2369611.1:cont'
02/27/06-13:30:51 PST 02/27/06-21:30:51 GMT 825111065 LDAS-CIT2369611 mpi::filterLogEntries 2369611.2:progress 100.00%
02/27/06-13:30:51 PST 02/27/06-21:30:51 GMT 825111065 LDAS-CIT2369611 mpi::filterLogEntries 2369611.2:projected_ratio 0.02535
02/27/06-13:30:51 PST 02/27/06-21:30:51 GMT 825111065 LDAS-CIT2369611 mpi::filterLogEntries 2369611.3:finalizing { totalTime=12.4909 sentObjects=4; Rank0 (inPut: create=2.13400 send=0.14782 ilwd=2.03673; outPut: create=0.11958 send=0.07751 ilwd=0.16268; metadata: (un)pack=0.01657 bcast=0.08051; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12203; max memory=6844416) Rank1 (inPut: create=2.09859 send=0.18172; outPut: create=0.07642 send=0.36670; metadata: (un)pack=0.01415 bcast=0.04534; IS=0.00008; CD=0.05701; AS=9.12698; FO=0.00153; FS=0.00336; LB=0.04725; max memory=348160) }
02/27/06-13:35:53 PST 02/27/06-21:35:53 GMT 825111367 LDAS-CIT2369614 mpi::filterLogEntries 2369614.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) }
02/27/06-13:35:55 PST 02/27/06-21:35:55 GMT 825111369 LDAS-CIT2369614 mpi::filterLogEntries 2369614.1:progress 0.00%
02/27/06-13:35:55 PST 02/27/06-21:35:55 GMT 825111369 LDAS-CIT2369614 mpi::filterLogEntries 2369614.1:projected_ratio 0.00017
02/27/06-13:35:55 PST 02/27/06-21:35:55 GMT 825111369 LDAS-CIT2369614 jobstate response: '2369614.1:cont'
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.2:progress 10.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.2:projected_ratio 0.00033
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.3:progress 20.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.3:projected_ratio 0.00049
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.4:progress 30.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.4:projected_ratio 0.00065
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.5:progress 40.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.5:projected_ratio 0.00081
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.6:progress 50.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.6:projected_ratio 0.00097
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.7:progress 60.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.7:projected_ratio 0.00113
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.8:progress 70.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.8:projected_ratio 0.00128
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.9:progress 80.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.9:projected_ratio 0.00144
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.10:progress 90.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.10:projected_ratio 0.00160
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.11:progress 100.00%
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.11:projected_ratio 0.00160
02/27/06-13:35:57 PST 02/27/06-21:35:57 GMT 825111371 LDAS-CIT2369614 mpi::filterLogEntries 2369614.12:finalizing { totalTime=4.1008 sentObjects=3; Rank0 (inPut: create=1.34424 send=0.08835 ilwd=1.27890; outPut: create=0.17668 send=0.52740 ilwd=0.04511; metadata: (un)pack=0.00052 bcast=0.17547; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12104; max memory=1511424) Rank1 (inPut: create=1.32082 send=0.11121; outPut: create=0.19195 send=0.51323; metadata: (un)pack=0.00044 bcast=0.19217; IS=0.05785; CD=0.08083; AS=1.40328; FO=0.00028; FS=0.00249; LB=0.03200; max memory=18370560) }
02/27/06-13:36:38 PST 02/27/06-21:36:38 GMT 825111412 LDAS-CIT2369615 mpi::filterLogEntries 2369615.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) Rank3(beowulf) }
02/27/06-13:36:42 PST 02/27/06-21:36:42 GMT 825111416 LDAS-CIT2369615 mpi::filterLogEntries 2369615.1:progress 0.00%
02/27/06-13:36:42 PST 02/27/06-21:36:42 GMT 825111416 LDAS-CIT2369615 mpi::filterLogEntries 2369615.1:projected_ratio 0.00000
02/27/06-13:36:42 PST 02/27/06-21:36:42 GMT 825111416 LDAS-CIT2369615 jobstate response: '2369615.1:cont'
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.2:progress 0.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.2:projected_ratio 0.00000
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.3:progress 25.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.3:projected_ratio 0.00000
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.4:progress 50.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.4:projected_ratio 0.00021
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.5:progress 50.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.5:projected_ratio 0.00048
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.6:progress 50.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.6:projected_ratio 0.00048
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.7:progress 50.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.7:projected_ratio 0.00048
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.8:progress 75.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.8:projected_ratio 0.00057
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.9:progress 75.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.9:projected_ratio 0.00084
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.10:progress 75.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.10:projected_ratio 0.00084
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.11:progress 100.00%
02/27/06-13:36:43 PST 02/27/06-21:36:43 GMT 825111417 LDAS-CIT2369615 mpi::filterLogEntries 2369615.11:projected_ratio 0.00089
02/27/06-13:36:44 PST 02/27/06-21:36:44 GMT 825111418 LDAS-CIT2369615 mpi::filterLogEntries 2369615.12:finalizing { totalTime=6.01799 sentObjects=3; Rank0 (inPut: create=2.18858 send=0.44059 ilwd=2.03463; outPut: create=0.04536 send=0.30215 ilwd=0.01025; metadata: (un)pack=0.00554 bcast=0.03464; IS=0.00000; CD=0.00000; AS=0.00000; FO=0.00000; FS=0.00000; LB=0.12080; max memory=2801664) Rank3 (inPut: create=2.15335 send=0.57646; outPut: create=0.00000 send=0.16210; metadata: (un)pack=0.00006 bcast=0.00360; IS=0.00016; CD=1.66835; AS=0.26030; FO=0.00000; FS=0.00443; LB=0.00001; max memory=917504) Rank2 (inPut: create=2.15335 send=0.57120; outPut: create=0.00000 send=0.24137; metadata: (un)pack=0.00013 bcast=0.00293; IS=0.00016; CD=1.66970; AS=0.34090; FO=0.00000; FS=0.00456; LB=0.00002; max memory=1089536) Rank1 (inPut: create=2.15238 send=0.50672; outPut: create=0.04163 send=0.61069; metadata: (un)pack=0.00480 bcast=0.03284; IS=0.00013; CD=0.84530; AS=0.03705; FO=0.00055; FS=0.00422; LB=0.05530; max memory=339968) }
02/27/06-13:37:08 PST 02/27/06-21:37:08 GMT 825111442 LDAS-CIT2369616 mpi::filterLogEntries 2369616.0:initializing { Master-Rank0(beowulf) Rank1(beowulf) Rank2(beowulf) }
02/27/06-13:37:23 PST 02/27/06-21:37:23 GMT 825111457 LDAS-CIT2369616 mpi::filterLogEntries 2369616.1:error {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }
02/27/06-13:37:23 PST 02/27/06-21:37:23 GMT 825111457 LDAS-CIT2369616 mpi::newJobCallback {Rank 2 failed with error message: ( applySearch function ): Level 1: $Id Status code -1: Recursive error function LALApplySearch, file ../../../../contrib/waveburst/src/Search.c, line 1075 Level 2: $Id Status code -1: Recursive error function LALSlaveWaveburstSearch, file ../../../../contrib/waveburst/src/Search.c, line 210 Level 3: $Id Status code 101: cannot open /dso-test/waveburst/usertag/timeinterval/etg function LALOutputWaveburst, file ../../../../contrib/waveburst/src/Output.c, line 1037 }