-*- mode: text -*- ------------------------------------------------------------------------------ ldas-1.12.2 ============================================================================== This is a minor bug fix to the 1.12 branch. The utility framecpp_dump_channels has been extended to dump all FrVect data types for the FrVect data elements of FrAdcData, FrEvent, FrProcData, FrSimData, and FrSimEvent structures uncompressed. ------------------------------------------------------------------------------ ldas-1.1.0 ============================================================================== a) Improvements to diskCacheAPI addressing issues associated with the Observatory file systems (Greg and Igor are waiting on these). b) Support in the createRDS user command for turning on and off the checksum calculation and timestamps validation of input frames. c) Approximately 25% overall performance boost over the 1.0.0 release d) Slightly upgraded LDCG (primarily due to a new GCC compiler) e) Standalone tarball distribution of the dataConditionAPI's C++ library with a wrapper to allow C programs to utilize the algorithm{} option (unique translators between application data types and the UDT C++ objects will be needed - examples provided) f) Several significant improvements in speed and functionality of the controlMonitorAPI's GUI. g) Close out of a few dozen open problem reports. h) Migration to DB2 Version 8.1 with fixpac 5 under RH 9 i) Removal of tape control monitoring functionality from cntlmonAPI and cmonClient ------------------------------------------------------------------------------ New/Modified/Delete resource variables ------------------------------------------------------------------------------ mpiAPI: 1) ::MPI_WRAPPER_INIT_TIMEOUT Maximum number of seconds to wait for a wrapper API to announce that it has completed it's intialization. Default is 10 seconds, which is a fairly generous value. 2) ::MPI_WRAPPER_COMMAND_TIMEOUT This is identical to the LDAS-1.0 resource variable ::MPI_WRAPPER_COMM_TIMEOUT but the name has been made more descriptive. 3) *** REMOVED *** ::MPI_WRAPPER_COMM_TIMEOUT ------------------------------------------------------------------------------ ldas-1.0.0 ============================================================================== a) Migrate to the STLPort version of the Standard Template Library - primarily for thread safty b) Rework of the diskCacheAPI to have the computationally demanding components handled in C++ - primarily for performance - addressed the need for multi-threaded support - new C++ binary object file for frame cache file on disk c) New /ldcg environment - based on STLPort - new version of LAM - new autoconfig/automake/libtools, compliant with LSC standard - new version of FFTW d) support in controlMonitorAPI's client for graphical viewing content of (i) database tables - new data viewer memu - time series - histograms + mean, median, std dev (ii) usage details of LDAS jobs - new memu under Jobs - view graphical tree of jobs by commands - view graphical tree of jobs by users In addition, controlMonitorAPI's client now works behind firewalls; multiple clients can run in one working directory; APIs can be added to hosts specified by users e) extended support for less well known frame structures in FrameCPP - accessors (FrSimData and FrSimEvent) f) migration to the newest version of FFTW (version 3) - improved performance - brings support for this library in line with LAL g) single precision bug found in IIR filtering by Julien Sylvestre fixed - Numerous test cases were added to extend the verification that the single-precision IIR filtering is equivalent to 1) converting the single-precision data to double precision 2) filtering it at double-precision and 3) converting back to single-precision. h) frameAPI's resource now allows tuning of frame file I/O method and buffer size on a per file system device basis - improved I/O performance possible with a little experimentation depending on file systems available at LDAS sites. ------------------------------------------------------------------------------ New/Modified/Delete resource variables ------------------------------------------------------------------------------ diskcacheAPI: 1.) ::CACHE_UPDATE_INTERVAL loop time, in seconds, around the ::MOUNT_PT list. 15 seconds is the default, and this is a good value generally. there is not much to be gained from tweaking this value, which is now deprecated as a tuning parameter in favor of the ::DELAY_BETWEEN_THREAD_SPAWNS_MS value. 2.) ::DELAY_BETWEEN_THREAD_SPAWNS_MS millisecond delay between ::MOUNT_PT update threads. making this very small results in thread contention and wasted cpu. a value of at least 100 (the default) is a good starting point. make it larger if too much CPU is getting used by the diskcache API. 3.) ::DEBUG_SCAN_RATE flag (0 or 1) causes messages to be printed in the diskcache API log every time that all ::MOUNT_PT directories have been updated. The message reports the number of mount points, subdirectories, and files scanned in NNNN ms. 4.) ::ERROR_REPORT_INTERVAL how often (in seconds) to bug the sys admin about badly formed directories under mount points. On a test system with intentionally bad directories, this should be at least 2 hours (7200 seconds), and at most one day (86400 seconds). 5.) ::DEBUG_CACHE_SYNCHRONIZE flag (0 or 1) whether to log each change detected by the diskcache API on a per-directory basis. Can generally be left on. 6.) ::EXCLUDE_THESE_DIRS_FROM_UPDATES list of directories that should not be touched by the diskcache API, usually because they are NFS mounted but offline, and threads pointed at them would hang forever. 7.) ::DISKCACHE_HASHFILE_NAME the name to write the binary hash file into. the default is .frame.cache2 8.) ::DIR_SLOW_UPDATE_WARNING_THRESHHOLD if any directory requires more than this many seconds to scan, send an email to the sys admin. good way to detect failed ide raid systems, or at least much better than not being able to detect them at all. 9.) ::NUMBER_OF_RUNNING_THREADS_PERMITTED absolute limit to the number of simultaneously running threads spawned via calls to the c++ extensions to the diskcache API. frameAPI: 1.) ::STREAM_BUFFER_SIZE establish the default file buffer size. If the value 0 (zero) is specified, the default buffer size of the operating system and/or compiler will be used. 2.) ::ENABLE_MEMORY_MAPPED_IO Establish use of memory mapped I/O for files if available. This does require the version of STLport specified in the "How to Build LDAS" documentation. 3.) ::DEVICE_IO_CONFIGURATION Establish a list of devices that have custom buffer sizes and memory mapped I/O characteristics. This list has to be of the form: [ list [ list ] ... ] 4.) ::CREATERDS_CHECKSUM Enable (1) or disable checksum (0) verification of frame files when generating RDSs. ------------------------------------------------------------------------------ ldas-0.7.0 ============================================================================== User Commands * a new user command is available, cacheGetFilenames. this command returns a list of the actual filenames satisfying the frame query that is passed in it's arguments. see the documentation for this command. * createRDS now supports combining channels from multiple IFOs. This feature is currently only support for input frames generated by LDAS or the framebuilder at LLO or LHO. This feature is currently only supported for input frame files that contain a single 16 second frame file. * createRDS also supports user naming of the "frame type" portion of frame files via the -usertype option. cmonClient * New test panel to submit ldas jobs to check out system * Remember user login and password when switching between ldas systems * New CPU usage plot * Fixed least square plot in memory graph for flat or step data * Have extended selection for popdown listbox for selecting Ldas cmds and dso. * Allow non-control user to abort his/her job * Tag temp files with site name to be more specific. * Force user to enter a reason for API change, node change and LDAS shutdown. cntlmonAPI * Send email for API changes, node changes and LDAS shutdown * Add new utility to create/recreate database * db2utils to get alias of database from system directory instead of local * Improvements in log filter via string match instead of regexp * Return set of completed jobs instead of running jobs for job output * Return status of user as control group or not to prevent excess login/passwd entry from cmonClient * Collect CPU data from API mem file datacondAPI * Modified the datacon Statistics class to work on complex data * Added a sum() action to sum the elements of a vector * Extended size() and mean() datacon actions to work on complex data: * Fixed several memory leaks in: o CoherenceFunction.cc o LinFiltState.cc o mtstate.cc o MixerState.cc o WaveletUDT.cc o Wavelet.cc o Matrix.hh * Fixed PR1928, oelslr produces wrong error message for bad 3rd argument type * Verified the following PR's as fixed: o 1865 output() command should treat underscore as blank or default parameter o 1629 Datacon MDC test causes occasional coredumps on ldas-test o 1434 More signal generator actions requested * Several changes and fixes to web documentation for datacon * Added new signal-generating actions: o sine() o cosine() o square() o sawtooth() * Made some changes and fixes to the Kalman filter code: o Kalman.cc o rmvmFunction.cc * Added support for 'auto-expansion' of -aliases and -algorithms options of conditionData, dataPipeline and dataStandAlone user commands via the -autoexpand flag and a simplified syntax for the -aliases and -algorithms options. this is a very powerful syntactical extension that has yet to be fully defined, but is usable and very useful. see documentation for dataPipeline -autoexpand option. diskcacheAPI * channel spec is now parsed for IFO spec. formerly the user was required to specify interferometer and channel name seperately, this change makes explicitly stating the ifo unnecessary. * data degeneracy is now handled by giving preference to directories according to the order in which they appear in the ::MOUNT_PT variable in the LDASapi.rsc. directories appearing earlier are preferred. * proper handling of NDAS style frames. a frame whose name contains a compound interferometer spec can satisfy requests for data from any one of the ifos that is contained in it. the degeneracy rule described above is applied to these files also. eventmonAPI * Use setAlert to wake up from thread calls instead of polling * Wait for threads to finish before terminating or killing a job * Process mdd data for putStandAlone * Return exception for aborted or timed out jobs * Removed extra wrapped of mdd objects sent to ligolw frameAPI * same channel spec parsing of ifo names as in the diskcache API. * "hinting" added to channel slicing syntax so that data array types TIME, FREQ, and TIMEFREQ can be discriminated between. see documentation for -framequery option in the dataPipeline user command documentation. * added support for user specified types to RDS frames via the * usertype option to createRDS. ligolwAPI * base64 encoding for Vector data in LIGO_LW documents, e.g. documents representing frame data. New option for user command: * ligolwformat { LIGO_LW base64 } * Up to date LIGO_LW DTD text file is generated as part of the ldas build: /ldas/doc/ligolwAPI/html/ligolw_dtd.txt. * Use setAlert to wake up from thread calls instead of polling * Migrated bulk of macro code from macros into tcl body * Wait for threads to finish before exiting * Support LIGO_LW base64 and add new output format for ligolw -ligolwformat managerAPI * the manager API will now return a proper email message to the user for each job that is queued but not running when the LDAS system is shut down. formerly these jobs 'disappeared'. * it is now possible to trim jobs off of the queue, or to remove the queue entirely. each job de-queued will cause an email to be sent to the user describing the conditions and stating that the job must be rerun. metadataAPI * Removed slave interp in processing * Use setAlert to wake up from thread calls instead of polling * Converted to swig 1.3 * Migrated bulk of macro code from macros into tcl body. * Removed unused bgLoop removeJobReaper mpiAPI * added support for limiting concurrent processes when ::MPI_MUTLIPLE_NODES is defined. the resource variable ::MPI_NODE_SHARE_LIMIT will be applied to limit the number of simultaneous processes. * added support for distribution of wrapper masters among the beowulf gateway machine and the nodes. this seems to improve the robustness of the mpi API. distribution is enabled by setting the resource variable ::DISTRIBUTE_WRAPPER_MASTERS to '1', and setting the resource variable ::N_WRAPPER_MASTERS_ON_GATEWAY to the number which should be started on the gateway before sending them off to the internal nodes. this extension has been implemented in a way that makes it possible to switch it on and off, or to adjust the number to start on the gateway dynamically. it has been demonstrated to behave correctly even when nodes are being dynamically added and removed. when the names of all nodes appear uniquely in the mpi API ::NODENAMES list and the flag ::MPI_MULTIPLE_NODES is not set, the wrapper master will run alone on the assigned node with no slave being assigned to that same node. wrapperAPI * disabled dlclose() on exit to run RH7.3 binaries on RH9. Developement tools * gcc-3.2.3 * autoconf-2.57 * automake-1.6.3 * libtool-1.4.3 * lam-6.5.9 * swig-1.3.19 * ObjectSpace-1.2.6 * xerces-2.2.0 ------------------------------------------------------------------------------ ============================================================================== ldas-0.2.0 ------------------------------------------------------------------------------ INSTALLATION ============================================================================== Packages removed * prngd Packages updated * GCC * ObjectSpace * openSSH Packages added * cURL Other * The lal software has moved from /ldcg to /lal. ------------------------------------------------------------------------------ USER COMMANDS ============================================================================== dataPipeline * Removed the follwoing options: -ignoreintermediates -resultname -resultcomment -framequry option * Now supports resampling. This is done by adding "!resample!q!" after the channel name or number, where q is the downsample ratio. ------------------------------------------------------------------------------ SOFTWARE ============================================================================== managerAPI: * New diagnostic functions with generated web pages available from the api status page. New diagnostic functions available for use by GUILD allowing live monitoring of job progress. * Manager no longer tries to restart mpi or diskcache API's if they are slow to start. * Manager is no longer distributed with password file for LDAS users. frameAPI: * Support for FrDetector linked list. * Frequency bandlimiting. * Removed exception specifications. * Proper handling of proc data in all user commands. diskcacheAPI (NEW): * New API replacing ad-hoc functionality formerly in the frame API. Much more complete registration of frame files... All frame files found under the ::MOUNT_PT directories are cached. datacondAPI: * Intermediate() action has been replace with output(). Modifying old action seqences requires changing intermediate to output and supplying the variable as the first argument. All arguments of the intermediate action have been shifted by one. What was the first argument of the intermediate action is the second argument of the output command, etc. Also, there is no implied output statement at the end. Each with at least one action must supply an output action as the last action. * Added support for passing detector geometry information metadataAPI: * output insertion and query statistics to file. ligolwAPI: * Support for linked list of FrDetector structures mpiAPI: * Lam Daemons are now persistent between jobs so per job startup time is significantly reduced. wrapperAPI: * Added channel name and comment fields to multiDimData structure. * Added detector geometry structure. * More persistent connection to the resultAPI( resource file has 2 parameters: number of attempts to connect and pause between retries ) eventMonitorAPI: * Added detector geometry support * Open additional 20 data socket server ports for wrapper to connect. cntlmonAPI: * add coreWatch utility and management of core file. * support pass/fail job statistics and insertion/query statistics. * establish operator port at 10000 * fix db2utils and logscan emailing * db2utils to restart db2 server and metadataAPI if db2 is down cmonClient: New features * add core file page to support debug of core files and deletion * allow tape control script to run on any ldas machine instead of just dataserver; support repeat alarms until operator cancels * modify Tree page to display job output based on jobids and move it under jobs tab * display pass/fail job statistics graph and insertion/query statistics graph * expand job info on job control page to include user, api, cmd, runtime; allow multiple selections instead of single select on job list. * added users queue info to User Admin page * added API state summary page under Status tab * added Node Balance page to Beowulf terminals ------------------------------------------------------------------------------ HARDWARE ============================================================================== * New server added internal to LDAS system to add more compute power.