HK/TM Product¶

Data product name¶

DpdHKTMProduct

Data product custodian¶

Name of the Schema file¶

euc-soc-HktmProduct.xsd

DPDD version¶

1.1

Processing Element(s) creating/using the data product¶

Creators:

SOC

Consumers:

VIS_PF, NISP_PF

Processing function using the data product¶

VIS processing function, NISP processing function.

Data product elements¶

Header:: object of type sys:genericHeader
Data:: object of type soc-pro:hktmProductData
QualityFlags:: object of type dqc:sqfPlaceHolder
Parameters:: object of type ppr:genericKeyValueParameters

Detailed description of the data product¶

The HK/TM Product is produced by SOC from the HK/TM data received by MOC and ingested into the SOC ARES System. This is intended to be used primarily, but not exclusively, by the LE1 Teams.

The current approach for the HKTM products generation is done according to the following guidelines:

The HKTM data is extracted from the ARES system (Hadoop cluster).
The HKTM product data time span is 1 hour. That is, it contains parameter data with their time stamp included in the 1 hour period corresponding to that product.
The HKTM product is compossed of: - A series of FITS files, each of them with:
- 1 Primary HDU (Header Data Units) with a general metadata header
- 1 Extension layer (additional HDU) per parameter, each of them with one binary table with a number of rows (one per sample), and 2 columns, the first one with the sample time stamp, and the second one with the parameter value.
- One XML file with the appropriate metadata information for the HKTM product. This contains a two-way index of the parameters: (parameter) → (file, hdu) and (file, hdu) → (parameter).
For each parameter, the type used in the corresponding table in the FITS file is stated. This could be removed, since this information already appears in the table HDU (TBD).
The set of data is split into several FITS files. The current splitting criteria is: #. 1 FITS file with VIS parameters #. 1 FITS file with NISP parameters #. 1 FITS file with AOCS parameters #. 1 FITS file with Flight Dynamics related parameters #. I FITS file with the rest of parameters
All the tables have a first column with a time stamp. It has been suggested to group all this information in a common time column. But this is a problem with a number of aspects: - First, there are lots of different sampling frequencies, so in order to have a common TIMESTAMP column, we should make an effort on grouping the different parameters by a common frequency. Grouping per frequency/period several thousands of parameters is a LOT of work. - This will not be valid for those non-periodic parameters, though; those parameters that get updated only from time to time. You could think about putting them in separate files. - Then, even if one succeeds with the grouping, you will have the problem of having in separate FITS files parameters that have a common nature/origin (although a different sampling frequency).

Finally, even with a common sampling frequency, that doesn’t mean that they share a common timestamp (imagine a very simplistic scenario of two 2-hour period parameters; one could be generated at odd hours, the other at even hours). In summary: what we get from the system is not parameter values for a given set of timestamps; what we obtain are sets of pairs (timestamp, param.value). So, storing them as pairs in the FITS files is not only faster, convenient and easier to use; it is actually how the data is generated and retrieved.

Here you can see an example of the list of files generated for 1 hour that will conform the HKTM Product:

  Length      Date    Time    Name
---------  ---------- -----   ----
 52709760  01-27-2022 19:10   EUC_SOC_HKTM-AOCS_20211029T070000-20211029T080000.fits
 11687040  01-27-2022 19:10   EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits
 26786880  01-27-2022 19:10   EUC_SOC_HKTM-NISP_20211029T070000-20211029T080000.fits
 10327680  01-27-2022 19:10   EUC_SOC_HKTM-FGS_20211029T070000-20211029T080000.fits
106574400  01-27-2022 19:11   EUC_SOC_HKTM-others_20211029T070000-20211029T080000.fits
  2084434  01-28-2022 07:55   EUC_SOC_HKTM_20211029T070000-20211029T080000.xml
---------                     -------
210170194                     6 files

Below, a (reduced) view of the content of the XML Metadata file:

<?xml version="1.0" encoding="UTF-8"?>
<dpd-le1-hktm:DpdHKTMProduct xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://euclid.esa.org/schema/dpd/soc/hktm euc-test-soc-HktmProduct.xsd"
  xmlns:dpd-le1-hktm="http://euclid.esa.org/schema/dpd/soc/hktm">
  <Header>
    <ProductId>EUC_SOC_HKTM_20211029T070000-20211029T080000</ProductId>
    <ProductType>DpdHKTMProduct</ProductType>
    <SoftwareName>HKTMProductGenerator</SoftwareName>
    <SoftwareRelease>3.2.1</SoftwareRelease>
    <ProdSDC>SOC</ProdSDC>
    <DataSetRelease>TEST</DataSetRelease>
    <Purpose>TEST</Purpose>
    <PlanId/>
    <PPOId>NA</PPOId>
    <PipelineDefinitionId/>
    <PpoStatus/>
    <ManualValidationStatus>VALID</ManualValidationStatus>
    <ExpirationDate>2100-01-01T00:00:00.000Z</ExpirationDate>
    <Published>0</Published>
    <Curator>SOC</Curator>
    <CreationDate>2022-01-27T18:11:59.802Z</CreationDate>
  </Header>
  <Data>
    <DateTimeRange>
      <FromDate>2021-10-29T07:00:00.000000Z</FromDate>
      <ToDate>2021-10-29T08:00:00.000000Z</ToDate>
    </DateTimeRange>
    <ParameterList>
      <Parameter pid="3623" name="ACCT7509" type="D" prodIndex="1" hduIndex="1"/>
      <Parameter pid="3624" name="AW1T7509" type="D" prodIndex="1" hduIndex="2"/>
. . .
      <Parameter pid="27667" name="DSST1065" type="D" prodIndex="5" hduIndex="7843"/>
    </ParameterList>
    <HKTMProductList>
      <Product index="1" id="EUC_SOC_HKTM-AOCS_20211029T070000-20211029T080000">
        <HDU index="1" pid="3623" paramName="ACCT7509" type="D"/>
        <HDU index="2" pid="3624" paramName="AW1T7509" type="D"/>
. . .
        <HDU index="1733" pid="25115" paramName="ATGT6991" type="A13"/>
      </Product>
. . .
      <Product index="5" id="EUC_SOC_HKTM-others_20211029T070000-20211029T080000">
        <HDU index="1" pid="1" paramName="DJJT1100" type="X"/>
        <HDU index="2" pid="2" paramName="DJJT1139" type="X"/>
. . .
        <HDU index="7843" pid="27667" paramName="DSST1065" type="D"/>
      </Product>
    </HKTMProductList>
    <HKTMContainer>
      <File filestatus="PROPOSED">
        <FileName>EUC_SOC_HKTM-AOCS_20211029T070000-20211029T080000.fits</FileName>
      </File>
      <File filestatus="PROPOSED">
        <FileName>EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits</FileName>
      </File>
      <File filestatus="PROPOSED">
        <FileName>EUC_SOC_HKTM-NISP_20211029T070000-20211029T080000.fits</FileName>
      </File>
      <File filestatus="PROPOSED">
        <FileName>EUC_SOC_HKTM-FGS_20211029T070000-20211029T080000.fits</FileName>
      </File>
      <File filestatus="PROPOSED">
        <FileName>EUC_SOC_HKTM-others_20211029T070000-20211029T080000.fits</FileName>
      </File>
    </HKTMContainer>
  </Data>
</dpd-le1-hktm:DpdHKTMProduct>

Finally, you can see below an excerpt of the metadata keywords for the different table extensions, each corresponding to a different parameter, in the FITS files.

# HDU 0 in EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits:
SIMPLE  =                    T / conforms to FITS standard
BITPIX  =                    8 / array data type
NAXIS   =                    0 / number of array dimensions
EXTEND  =                    T
CREATOR = 'HKTMProductGenerator 3.2.1'
OBSERVER= 'Euclid Operator'
COMMENT Multitable FITS file, with data retrieved from ARES system
COMMENT File contains data for parameters in VIS group

# HDU 1 in EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits:
XTENSION= 'BINTABLE'           / binary table extension
BITPIX  =                    8 / array data type
NAXIS   =                    2 / number of array dimensions
NAXIS1  =                   16 / length of dimension 1
NAXIS2  =                  360 / length of dimension 2
PCOUNT  =                    0 / number of group parameters
GCOUNT  =                    1 / number of groups
TFIELDS =                    2 / number of table fields
TTYPE1  = 'TIMESTAMP'
TFORM1  = 'K       '
TTYPE2  = 'VMGT3035'
TFORM2  = '8A      '
EXTNAME = 'VMGT3035'           / extension name

. . .

# HDU 46 in EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits:
XTENSION= 'BINTABLE'           / binary table extension
BITPIX  =                    8 / array data type
NAXIS   =                    2 / number of array dimensions
NAXIS1  =                   16 / length of dimension 1
NAXIS2  =                 1800 / length of dimension 2
PCOUNT  =                    0 / number of group parameters
GCOUNT  =                    1 / number of groups
TFIELDS =                    2 / number of table fields
TTYPE1  = 'TIMESTAMP'
TFORM1  = 'K       '
TTYPE2  = 'VMGT3022'
TFORM2  = 'D       '
EXTNAME = 'VMGT3022'           / extension name

. . .

HK/TM Product¶

Data product name¶

Data product custodian¶

Name of the Schema file¶

DPDD version¶

Processing Element(s) creating/using the data product¶

Processing function using the data product¶

Data product elements¶

Detailed description of the data product¶

Table of Contents

Previous topic

Next topic

This Page