The getData User Commands (Meta Macros)

All user commands have the form:

ldasJob { -name {} -password {} -email {} } { userCmd -opt1 {} ... }

Which is in the format of a Tcl command, ldasJob, with two required arguments:

1. A Tcl list of user information consisting of username, password, and e-mail address. All fields must be filled or the command will be rejected.
   Client software which would like to receive responses from the LDAS system on a port instead of through email can provide a hostname:port combination as the email argument. A further feature of this method is that the hostname can be replaced with the magic key !host!, and the LDAS system will use the IP address it finds by doing a peername lookup on the connected socket.
2. A user command in the form of a Tcl list, for which there exists a "meta" macro file which can be expanded into Tcl code which can then be evaluated by the LDAS system.
   An argument need not be provided for every option. Some options have default values and these are described below as appropriate.
   Meta macros consist of a prototype declaration of the arguments for the given user command, and a template describing the calling order of API specific blocks of Tcl code which are concatenated into a larger block comprising the complete request, which can then be distributed by the assistant manager for interpretation by the low level API's.
   These API specific blocks are maintained as API specific macro files consisting of immediately interpretable Tcl code.

• getFrameData
• getMetaData

• getFrameData

  Returns full or reduced raw frame data in several formats.

  Calling convention (all on a single line):

  ldasJob   { -name   "username"   -password   "********"   -email   "user@foobar.edu"}  {getFrameData  -returnprotocol  URL   -outputformat  {Tcl list}  -frametarget  name_of_an_api  -framequery  {Tcl list} } -compressiontype   supported_compression_mode   -compressionlevel   integer   }

  Option Descriptions:

  -returnprotocol:     http ftp gridftp mailto file port

  Optional  

  Default Value: http://results.ilwd

  Note that in most cases this option should NOT be explicitly set by the user, since it can have unexpected interactions with the output() actions in the datacond API, for example.

  The argument to -returnprotocol resembles the usual browser conventions for URL's, and is used to control the naming and disposition of output data from a user request.

  The http://, ftp://, gridftp:/, and file:/ URI types are currently supported.

  The specific format of the argument, and possibly the values of other arguments will affect the way that the -returnprotocol argument is applied by the LDAS system:

  • -returnprotocol [file|ftp|http]

    Will cause the system to return the location of output data as an ftp or http URL (the system default, in the absence of a user supplied -returnprotocol option, is to use http).
    Specifying "file" will return the local path to the file as it is seen by all LDAS API's.
    Specifying ftp or http will return a URL relative to the gateway machine of the local system (which is the same machine to which user requests are made).
    The names used for the output files will be determined by the system and are generally descriptive of the content of the files.

  • -returnprotocol [file|http|ftp]:filename.ext

    Will cause the system to return the location of output data as an ftp or http URL (the system default, in the absence of a user supplied -returnprotocol option, is to use http). And any output object which is not in frame format will be named according to the suggested pattern. When possible, indexes will be appended to the filename to differentiate between multiple output files, but sometimes this can not be managed and in that case as many levels of .ba* files will be created as required to avoid overwriting data.
    The "file" form is used to get the local name of the file relative to the LDAS installation for use as input data by subsequent jobs.
    This form is not advised when the job is one that is expected to produce many output files!
    In some special cases the suggested filenamr and extension may be ignored, as when a frame file is being produced, or an exact ilwd representation of a frame file.

  • -returnprotocol [http|ftp|file]://dirname/subdirname/

    When this form is used and dirname corresponds to an existing directory in the anonymous ftp area of the LDAS system, all output files will be copied into that directory. In some cases a filename can be specified and will be used; in this case .ba* files may be created to avoid overwriting of data.
    Note that the trailing slash is absolutely required in order for the URL to be interpreted as a local directory by the LDAS system.
    The URL returned to the user will be of the type specified.

    This option of -returnprotocol is currently is not supported. The output e.g. frame or xml, is placed in the job output directory. The user has to retrieve the output via http or ftp to to his/her local system.

  • -returnprotocol ftp://foo.bar.edu/here/there/file.ext

    When this form is used the LDAS system will attempt to copy the output data via anonymous ftp to the remote site.
    The user will need to run tests to make certain that this form works with the specified site.

    This option of -returnprotocol is currently is not supported. The output e.g. frame or xml, is placed in the job output directory. The user has to retrieve the output via http or ftp to to his/her local system.

  • -returnprotocol gridftp://here/there/file.ext

    This is a special protocol used by the dataStandAlone user command for interaction with a GLOBUS/GRID system.

    This returnprotocol is only interpreted by the datacond API for use on output data formatted for ingestion by a stand alone wrapper API.

    The file will be written to the local gridftp home directory or a subdirectory of it which is writable by the user which LDAS runs as.
    The LDAS writable subdirectory should be defined in the LDASapi.rsc resource file as the ::GRID_FTP_WRITABLE_SUBDIRECTORY, which will be joined to the grid home directory when calculating output filenames.

    Several modified forms of the gridftp URL are available:

    • If the specified URL is a left-hand substring of the default location, then data will be written into job specific subdirectories under the default location. This allows the use of:
      -returnprotocol gridftp

      as a useful shorthand notation for producing a directory hierarchy of dataStandAlone results which maps directly to the job directories normally produced by LDAS, allowing simple correlation of output data with other job resuts or messages from the LDAS system.

    • If the specified URL refers to an existing or non-existant directory under the default area:
      -returnprotocol gridftp://here/there/mydir/mysubdir/

      Then the directory hierarchy will be created as required and unique job-specific default filenames will be assigned to the datacond API's output for the wrapper, i.e.:

      LDAS-TEST1234567_wrapperdata.ilwd

    • If the specified URL does NOT end with a slash, the final element of the name will be used as the filename, with the extension .ilwd appended to the root of the final element.

    NOTES:

    1. Embedded spaces in the argument to the -returnprotocol option will cause the request to fail.

  -outputformat:     frame | {ilwd ascii} | {ilwd binary} | LIGO_LW

  Default: "{ilwd binary}"
  The argument of the -outputformat option is the data type to use in formatting the result of the user request.
  The possible output formats which the system can produce, which include binary frames, ilwd ascii, ilwd binary and LIGO_LW (an XML format) can be specified, i.e. {ilwd ascii}.

  -frametarget:     {}

  Optional: defaults to a null string
  The argument of the -target option is the name of a known LDAS API. The result output from the frame API will be registered with the named API to be used as input data for the code with the same job i.d. that the data is tagged with.
  NOTES:
  1. The target is relative to the frame API for frame commands.

  -compressiontype:     {raw, gzip, gzip_diff, zero_suppress_short, zero_suppress_int_float, or zero_suppress_otherwise_gzip}

  Optional: defaults to 'gzip'
  

  Compression Name	Description
  raw	The data is not compressed.
  gzip	The data is compressed using the gzip algorithm
  diff_gzip	The data is differentiated and then gzipped
  zero_suppress_short	Data which is composed of 2 byte integers is differentiated and then zero suppressed. All other data types are left uncompressed. This mode does not imply zero_suppress_int_float.
  zero_suppress_int_float	For 4 byte integers or float data, the data is differentiated and then zero suppressed. If the data is 2 byte integers, the compression mode is modified to be zero_suppress_short. All other data types are left uncompressed.
  zero_suppress_otherwise_gzip	All data types that support zero suppress compression are compressed using that method. All other data types are compressed using gzip.

  -compressionlevel:     {}

  Optional: defaults to '1'
  This option allows the user to specify the level of compression for compression types that support multiple levels. Gzip and gzip_diff support comprssion levels 1-6, "raw" does not support any levels. If a compression type does not support multiple levels, this option is quietly ignored.

  -framequery:     {R H {} 666666666-666666669 Adc(1,2,5-15)}

  The -framequery option is REQUIRED for getFrameData, getFrameElements, and concatFrameData user commands, but is OPTIONAL for conditionData, dataStandAlone, and dataPipeline user commands.
  The -framequery argument is a complex list of frame API query atoms. The query atoms consist of unique identifying strings (which are not case sensitive) with indices or channel names grouped in parentheses. The query atom strings consist of the unique parts of the accessor function names from the frame API c++ code. (See: frameAPI.so)

  The -framequery syntax now supports a complex associative format which allows the simultaneous retrieval of unrelated data elements from multiple sources.
  In the simplest case, a single frame repository containing frames from more than one instrument can be queried to retrieve a common time range from each specified interferometer:

  • -framequery { {} {H L} {} 600000000-600000007 Adc(0)}
    This query will return the data for Adc 0 from both Hanford and Livingston over the gps time period 600000000 to 600000007.
  • -framequery { { {} H {} 600000000-600000007 Adc(H2:LSC-AS_Q)}{ {} L {} 600000000-600000007 Adc(L1:LSC-AS_Q)}}
    This query will return the data from both Hanford and Livingston over the gps time range 600000000 to 600000007 for the H2 and L1 so called gravitational wave channels.

  The five fields of the complex framequery are:

  1. frame type
  2. interferometers
  3. frames
  4. times
  5. channels (and/or structures for which accessors exist)
  Any field which is not used must contain a pair of braces "{}" as a placeholder so that all sublists of the -framequery always consist of five elements.

  The framequery option supports the slicing syntax Adc(channel!start!range[type]!) or Proc(channel!start!range[type]!):
  The "slicing" syntax allows a subset of an Adc or Proc data channel to be obtained. Slicing is performed by appending two floating-point numeric arguments, the start and range, to the channel specification (the fifth field of the framequery), delimited by !'s. The start is the absolute starting position of the slice along the x-axis of the channel, and the range is the extent of the slice.
  Valid [type] specifications are TIME, FREQ, and TIMEFREQ. When no type specifier is given, time is assumed, and this may generate errors if frequency series data is found.
  The units of the both numbers are taken from the unitX field for the specified channel in the frame file when the data is operated on by an another API. This is usually seconds for time-series data and Hertz for frequency-series data.

  Note that the extent of the x-axis of a channel is not related to the GPS time of the frame. The first value on the x-axis is given by the startX field of the channel, and the extent is given by the number of samples in the channel times the dx field of the channel. While startX may have any value, it is usually 0, although there are cases where it could be negative, such as for a 2-sided power spectrum.

  The sliced data will contain all samples whose x-coordinate satisfy start <= x < start + range. An error will be generated if the start of the slice is less than startX, or if the slice extends past the end of the channel.

  Examples of slicing:

  Suppose we are accessing a channel from frame H-R-600000000.gwf:

  Adc(H2:LSC_AS-Q!0.3!0.4TIME!)

  The ! syntax determines a start-time and range within the channel (since startX is zero for time-series data, the start-time may be interpreted as an offset from the GPS start-time of the framequery). The syntax requires that the channel name be followed by a !, followed immediately by a valid time offset into the data array for the channel, followed by another !, followed immediately by a range, followed by a final !.

  This example will return a 0.4 second long slice of the channel data from GPS time 600000000.300000000 through 600000000.700000000.
  The ILWD object containing this data slice will have the correct calculated start-time and times-span for this slice of data, as well as the correct number of data points in the data array.

  Suppose we have a Proc frame with a Proc structure containing frequency spectrum data with Hertz as the x-axis units. We can request a specific frequency band by specifying a start-frequency and a frequency range:

  Proc(0!1024!1024FREQ!)

  This will return a Proc structure containing the data at all frequency bins greater than or equal to 1024 Hz and less than 2048 Hz, that is, all frequency data in the semi-open interval [0, 1024) Hz. Any metadata associated with the original object will be passed though unmodified.

  NOTE on frequency slicing:
  The directive Adc(CAL-CAV_GAIN!0!2048.0001FREQ!) is interpreted by LDAS as all frequencies >= 0 and < 2048.0001.
  Note the strict inequality for the upper limit. The frequency of a bin is determined by:

       f_k = startX + k*df, k = 0, 1, ..., N-1
  

  so you just need to make sure your request is consistent with this scheme. The reason some users are adding 0.0001 is becuase they have f-sequences that have a bin that they want at EXACTLY 2048 Hz (say), so they need to specify an upper limit above this ie. the upper limit must be strictly greater than 2048 and less than or equal to 2048 + df Hz. If adding 0.0001 works for you fine, but that's obviously not the only choice. This might seem a strange convention but the reason it's chosen is:
  1. to be consistent with the way we specify time intervals, which are also given as semi-open intervals.

  2. so that it's easy to pull out sub-sequences of frequency series and/or concatenate them in a non-overlapping way ie:
     [ ... ) [ ... ) [ ... )

  The framequery option support for downsampling syntax Adc(channel!resample!q!) or Proc(channel!resample!q!):
  Suppose we are asking for gravity wave channel data over an hour, but want to downsample the data by a factor of 8 before it is sent to the dataconditioning API:

  Adc(H2:LSC_AS-Q!resample!8!)

  In the case of resampling, the first field after the channel name contains the literal string "resample", and the second field contains the downsampling factor q. (see the resampling algorithm documentation.)

  It is intended that resampling be applied after data has been time sliced based on the time range part of the framequery option.
  As of the 0.2.0 release of LDAS, this has been imperfectly implemented, and it is important that users wishing to make effective use of the resampling syntax provide feedback to the developers via the problem tracking system:

  LDAS Problem Tracking System

  Note that the only supported resampling factors are 2, 4, 8, and 16. The -framequery option by-the-numbers:

  The first element of the complex -framequery option is a list of the type of the frames which should be retrieved. The type refers to the frame attribute referenced by the second field in the proposed frame spec. This filed defaults internall to R, or "Raw".
  Another example of a frame type would be mT, or "Minute Trend".
  separate output container will be created for each interferometer specified.

  The second element of the complex -framequery option is a list of the interferomenters which produced the data that is wanted. A separate output container will be created for each interferometer specified.

  The third element of the complex -framequery option is a Tcl list of frame file names or URL's.
  This option is generally only used when a single specific input frame file (or a single file from, say each of two interferometers) is needed, and is provided primarily for the application of data conditioning or pipelining of test data in the form of frame files.
  This option will become the mecahnism for the reading of calibration or other process data from frame files at a later time.

  The fourth element of the -framequery option, times, is a Tcl list of GPS timestamps and ranges. The times argument must have the form a-b where a and b are valid GPS times in whole numbers of seconds. Due to historical precedent, this range should be interpreted as a request for data including the second starting at a until the end of the second starting at b, thus the actual interval of time being requested is [a, b+1). For example, -times 666666666-666666681 represents the equivalent of specifying the names of 16 1-second frame files, and represents the 17 seconds of data from time 666666666 up to but not including 666666682.
  The syntax supports a gap filling flag: 666666666-666666681:allow_gaps which the LDAS system recognises as an indication that it should allow missing frame files, and to account for them in the data conditioning stage of the user command by filling in missing data points as specified by the fillgaps action in the algorithms option.
  Note that the -allowgaps option to the dataPipeline user command overrides all the syntactical subtleties implied here and will cause a single data segment spanning all specified time ranges to be created.

  Example:

  -framequery { {} H {} {666666666-666666669:allow_gaps 666667660-666667665} full(0)}

  Will, with the -allowgaps option specified, ultimately result in a single frame of 1000 seconds duration.

  Note that the times element of the -framequery option returns the data spanning the range of time from the top of the starting second to the bottom of the ending second.

  The fifth element of the -framequery option supports a shorthand notation for frame structure accessor methods exposed to the Tcl layer. The most commonly used methods are the Adc() and Proc() accessors, which retrieves Adc (time serialised data) or Proc (time, freq, or time-freq domain) channel structures from frames.The argument to this accessor method can be an integer (referring to the ith channel) OR the specific name of a channel:

       Adc(12)
       Adc(H2:LSC-AS_Q)
  

  An argument of "full()" as an item in the framequery option list will result in either a copy of the frame if the return format is specified as "frame", a full text dump of the frame if the return format is specified as "ilwd", and a full XML dump if the format is specified as LIGO_LW.

  getMetaData

  ---

  • getMetaData

    Returns meta data stored in the LDAS database in response to the query given by the user command; returns in several formats either by attaching the resulting files to an e-mail (-returnprotocol email) or by returning an e-mail with a URL pointing to the location of the results (-returnprotocol http or ftp).

    ldasJob   { -name   "username"   -password   "********"   -email   "user@foobar.edu"}   {getMetaData   -returnprotocol   URL   -outputformat   data_format   -sqlquery   {Tcl list}   -database   dsname   -metadatatarget   api   }

    Option Descriptions:

    -returnprotocol:     http ftp file port

    Default: http://some_sensible_name.ext
    The argument to -returnprotocol resembles the usual browser conventions for URL's, but is actually only a hinting mechanism for the user to get a URL back in e-mail or in a client.
    The mailto and port options are currently unsupported.

    The http, ftp, and file options cause a URL pointer to the location of the results to be returned by e-mail. Any directory structure in the location will be ignored, the filename and extension will be used unless they are found to break system conventions for mime typing.
    The possible formats of the argument are:

    • http://filename.ext

    • ftp://filename.ext

    • file:/filename.ext

      NOTES:

      1. Embedded spaces in the argument to the -returnprotocol option will cause the request to fail.

    -outputformat:     {ilwd ascii} | {ilwd binary} | LIGO_LW

    The argument of the -outputformat option is the name of the data type to use in formatting the result of the user request.
    The system supports ilwd ( defaults to binary ), ilwd ascii and LIGO_LW (an XML format) for the result rows.

    -sqlquery:     {any valid SQL statement}

    The argument of -sqlquery option is any valid sql statement supported by the LDAS database server ( IBM DB2 ). If the query turns out to be invalid, a database error message is returned.

    -database:     dsname

    The argument of -database option is a valid database dataset name for the site that the command applies e.g. cit_1 and cit_test are dataset names for site ldas-dev, lho_1 and lho_test are dataset names for site ldas-wa etc. For dsnames of sites, please visit the LDAS database link on each ldas site web page. If this option is not specified, retrieval is done from the default database.

    -metadatatarget:     api

    The argument of -metadatatarget option is a valid LDAS API name, usually metadata in which case the metadataAPI directs the output query results to file or to the ligolwAPI depending on returnformat specifications. This option need not be specified in a regular getMetaData command as the default is -metadatatarget metadata. When getMetaData is used as part of the dataPipeline command, setting -metadatatarget option to datacond and -outputformat to {ilwd binary} directs metadataAPI to send query results to the datacondAPI.

    Examples of getMetaData command:

    • for LDAS registered user foo to get all data from table sngl_burst in LIGO_LW (xml) format and have results delivered via ftp, submit:
      ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "ftp://www.ldas.ligo-wa.caltech.edu/pub/incoming/getMetaData1.xml"   -outputformat   LIGO_LW   -sqlquery   {select * from frameset fetch first 100 rows only} }

      Result in LIGO_LW

    • for LDAS registered user foo to get the number of rows in gds_triggers table, with output in in LIGO_LW (xml) format and have results in http URL link format, submit:
      ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   http://foo.xml   -outputformat   LIGO_LW   -sqlquery   {select count(*) from gds_triggers}   }

      Result in LIGO_LW

      Note: ldas does not do the actual http put since this is not permitted. But the users can use the result links to get the files themselves.

    • for LDAS registered user foo to get the maximum mass2 in sngl_inspiral table, with output in in LIGO_LW format and have job results notification delivered by email to a specified external port on a specified host, submit:
      ldasJob   { -name   "foo"   -password   "********"   -email   "ldas-dev:63419"}   {getMetaData   -returnprotocol   "file:maxMass2.xml"   -outputformat   LIGO_LW   -sqlquery   {select max(mass2) from sngl_inspiral}   }

      Result in LIGO_LW

    • This example shows how to use database built-in functions:

      for LDAS registered user foo to get a unique Id (guranteed unique) from the database in ilwd format. Submit:

      ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "foo@foobar.edu"   -outputformat   LIGO_LW   -sqlquery   {values(generate_unique())}   }

      Result in LIGO_LW

    • for LDAS registered user foo to get primary keys from segment in LIGO_LW format, submit:
      ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "foo@foobar.edu"   -outputformat   LIGO_LW   -sqlquery   {select t.tabname, k.colname from syscat.tabconst t,syscat.keycoluse k where t.tabname = k.tabname and t.constname = k.constname and t.tabname = 'SEGMENT' and t.type = 'P' fetch first 10 rows only}   }

      Result in LIGO_LW

    • This an example of how to use the database option to specify a database for the command instead of retrieving from the default database

      for LDAS registered user foo to count the number of gds triggers for the DMT process glitchMon in LIGO_LW format, submit:

      ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "foo@foobar.edu"   -outputformat   LIGO_LW   -sqlquery   {select count(*) from gds_trigger where process_id in (select process_id from process where program='glitchMon' }   -database   cit_1   } 

      Result in LIGO_LW

    • This an example of a complex query that could take a while to run if there are lot of rows.

      for LDAS registered user foo to further select the list of snrs selected from a subquery in LIGO_LW format, submit:

      ldasJob   { -name   "foo"   -password   "********"   -email   "foo@foobar.edu"}   {getMetaData   -returnprotocol   "file:getMeta1"   -outputformat   LIGO_LW   -sqlquery   {WITH ZZZ AS (select snr from sngl_inspiral where mass1 <= +9.89866E+008 ) select * from ZZZ where snr = (select max(snr) from ZZZ)}   }

      Result in LIGO_LW

    Examples of getMetaData command email response:

    For -returnprotocol fileURL:
    
    Message from LDAS:
    
    Your results:
    getMeta1.xml
    can be found at:
    /ldas_outgoing/jobs/NORMAL10466
    
    
    =========================================
    'LDAS API'          'CLOCK TIME(seconds)'
    =========================================
    managerAPI(queue):                0.00
    metadataAPI:                      6.48
    ligolwAPI:                        5.63
    -----------------------------------------
    managerAPI(total):               12.10
    =========================================
    
    
    
     Errors:
    
    Message from LDAS:
    
    metadataAPI:NORMAL10509: submit select * from process fetch 1 row for read only 1: Start of Error Messages
             File: ../../../../../api/metadataAPI/so/src/query.cc Line: 196
    SQLSTATE: 42601
    Native ErrorCode: -104
    [IBM][CLI Driver][DB2/SUN] SQL0104N  An unexpected token "1 row" was found following "* from process fetch" .  Expected tokens may include:  "" .  SQLSTATE=42601
    
    
    
    
    
    Return to Top