LDAS ILwd format for Metadata Insertion

The ILwd ( Internal Light weight ) format data representation is one protocol for inserting data into LDAS database via the LDAS software system. The LDAS ILwd format is described in Technical Note, LIGO-T980094-06-E, The Generic API's baseline specifications, section D.

This page describes the guidelines to follow in creating the ilwds to successfully insert meta data into the LDAS database.

• Guidelines for creating ilwds

A typical ilwd may contain the following ILwd elements:

<?ilwd?>
<ilwd comment='testing db2 6.1 with ilwd elements' name='ligo:ldas:file' size='2'>  <----  outer container ---------|
    <ilwd name='group:ilwdtypes:type_int_8s:table' size='3'>                        <----  table container -------| |
        <int_8s dims='5' name='group:ilwdtypes:type_int_8s'>1 0 3 -2 5</int_8s>     <----  array element          | |
        <ilwd name='group:ilwdtypes:type_char_u:table'>                             <----  char container   ----| | |
            <char_u dims='2' name='group:ilwdtypes:type_char_u'>1 2</char_u>        <----  array element        | | |
        </ilwd>                                                                     <----  char container   ----| | |
        <ilwd name='group:ilwdtypes:type_char_u:table'>                             <----  char_u container ----| | |
            <char_u dims='2' name='group:ilwdtypes:type_char_u'>1 2</char_u>        <----  array element        | | |
        </ilwd>                                                                     <----  char_u container ----| | |
    </ilwd>                                                                         <----  table container  ------| |																												
</ilwd>                                                                             <----  outer container ---------|

1. Outermost container
   

   This container holds all containers in the ilwd.

   1. name attribute is composed of <project>:<organization>:<medium>
      e.g. name='ligo:ldas:file'
   
   
2. Table Container
   

   This container holds all data for insertion into one table.

   1. name attribute is composed of [ <group> ]:<table>:table
      e.g. name='processgroup:process:table'

      table must be the field preceding :table identifier. Note that group is optional,

      e.g. name='process:table' is allowed.

   
   
3. Char and char_u Containers ( optional )
   

   This container holds char arrays for insertion into one column of a table.

   1. name attribute is composed of [ <group> ]:<table>:<field>
      e.g. name='processgroup:process:program'

   2. size attribute e.g. n, is the total number of its arrays, i.e. data for the column, for n rows.

   Note that group is optional so name='process:program' is also allowed. This is also true for LDAS array elements and Unique Ids explained below.

   
   

4. LDAS array elements
   

   The array element holds either character, unsigned characters (blobs), integer or floating point data for insertion into one column of a table.

   1. name attribute is composed of [ <group> ]:<table>:<field>
      e.g. name='processgroup:process:program'

   2. dims attribute n is the dimensions of its arrays, i.e. number of characters for a column.
   
   
5. Unique Ids
   

   If a table requires the database to generate a unique Id ( CHAR(13) BIT DATA ), the data for this column can be omitted in the ilwd and will be filled in at insertion time;   if the data is present for the unique Id field, this data is used instead of automatically generating it. for insertion into one column of a table.

   1. name attribute is composed of [ <group> ]:<table>:<field>
      e.g. name='processgroup:process:program'

   2. dims attribute n is the dimensions of its arrays, i.e. number of e.g. integers for n rows of a column. The default dims is 1, so if it is an empty array, dims must be set to 0 explicitly.
   
   
6. Null value representation via null mask
   

   The way to denote null values in some of the data in an array or null elements in a container is via a null mask attribute. Each item in an array or in a container is represented by a bit value: 1 for null value, 0 for not. A string of bits for the fields 1-6 would look like '011001' which indicates the 2nd, 3rd and 6th field are null. If there are more than 6 fields, the bits extends to the number of fields, e.g. '011001001001...'. In the ilwd, the mask is base64 encoded, so in the ilwd the mask is seen as a string of characters in the range [A-Za-z0-9+/=], e.g.

       mask is '1U', for bit representation 110101010100 which denotes fields 1,2,4,6,8,10 are null;
       mask is 'qo', for bit representation 101010101000 which denotes fields 1,3,5,7,9 are null.    
       
   

   Example of an hypothetical ilwd with null values
   
   Deciphering the nullmask

Examples of Single table ilwds

• Guidelines for multi-table ilwds

  Multi-table ilwds are generated when analysis is done in an online mode (vs batch) on the data: new process data etc. are to be added with new filters and the events selected via filters at the same time so the ilwd contains data for multiple tables: e.g.

      1 row of a new process 
      2 rows of new process_params 
      1 row of a new filter
      2 rows of a new filter_params used by the filter 
      3 rows of new sngl_inspiral selected by the filter 
  

  1. Containers within a table

  The guidelines remain the same as for single-table ilwds.

  2. Unique Ids

  If the child tables are referencing the parent tables within this ilwd, they must be listed via a pseudo-link in the ilwd. e.g. parent table is process, unique Id column is process_id, a pseudo representation of the unique Id for process_id is process:process_id:0

      <ilwd name='processgroup:process:table' size='14'>
          <ilwd name='processgroup:process:process_id'>
              process:process_id:0
          </ilwd>
  

  In child table process_params,

     <ilwd name='process_paramsgroup:process_params:table' size='5'>
          <ilwd name='process_paramsgroup:process_params:process_id' size='2'>
              process:process_id:0
              process:process_id:0
          </ilwd>
  

  The pseudo-link connects the 2 process_params rows to the parent process row. A unique Id is generated for the parent for the insertion; this unique Id value is filled in for the child table rows when they are inserted after the parent.

  3. Insertion order

     It is assumed that the data for the tables are to be inserted in the order each appears in the ilwd: e.g. process table data must appear before process_params table data if the latter cross-references the former's process_id column.

  Example of multi-table ilwds

  Example of multi-table xml corresponding to ilwd

  This ilwd file contains the following:

  • 2 rows for process table
  • 2 rows for process_params table using process_id and program from process data in ilwd
  • 2 rows for filter table using process_id from process data in ilwd
  • 2 rows for filter_params using filter_id from filter data in ilwd
  • 3 rows for sngl_inspiral using process_id from ilwd process data and filter_id from filter data
  • 3 rows for sngl_datasource using event_id and event_table from sngl_inspiral in ilwd

• Open Issues:
  
  
  1. Null data representation is not defined yet; currently it is assumed that data must be present for all the columns listed in the table container.
  2. There is no error indication if part of the data failed to insert; the user has to check the database via query and figure out why not all the data get inserted.