NOAA Logo NOAA/ OERD Autonomous Hydrophone Data File Description

Last Update: January 7th, 2016

Data Format Type 4A for the NOAA Autonomous Hydrophone Data Files from 2013 on

The Data Format Type 4A & 4B has the same settings as the Type 3.  Each data file has first 256 bytes header Information and rest of the bytes are data.  In the formats 4A & 4B, it is their header's contents of the last 192 bytes that are different than the header in the data format Type 3.

The main differences between the data format types 4A and 4B are their last 8 bytes in the header.  There the data format type 4A does not contain the NCHAN (Number of Channels) parameter whereas the data format type 4B does.  In this document, only Data Format Type 4A will be described.  See HPdataFormat4B.html for the Data Format Type 4B.

Determine the Data Format Type: 4A or 4B

The PROGNAME parameter in the header will be needed to find out which format type: 4A or 4B will be.  The PROGNAME is Program Name, Max. 12 characters long.  It is set by the technician before the hydrophone is deployed.  The PROGNAME will have the following name: CFxLogSP3izzz.c with or without the 2 suffix characters: ".c".  For example, PROGNAME = CFxLogSP3i3_4.c  or  CFxLogSP3i2_3

When the PROGNAME is equal one of the names:
CFxLogSP3i2_4.c, CFxLogSP3i3_2.c, CFxLogSP3i3_3.c or CFxLogSP3i3_4.c  with or without the last 2 suffix characters: ".c", then the data format type is 4B and it has the NCHAN parameter.
See HPdataFormat4B.html for reading the data format type 4B data files;

Otherwise, the data format is type 4A and it has No NCHAN parameters in the header.  The NCHAN will be assumed to be 1.

From here on, only Data Format Type: 4A will be described.

Key Header Information

The contents of the header are shown in between the Byte Swapping and the Decoding Instructions sections in this document below. To read the data correctly, the following parameters:

PROGNAME = Program Name, e.g. CFxLogSP3i3_1.c or CFxLogSP3i2_3 for the type 4A format.
NCHAN    = Number of Channels = 1 because of single channel data file.
TIME_GMT = Start Date & Time,
SRATEHZ  = Sample Rate in hertz,
SAMPLES  = Sample Size and type,
from the Header will be needed.

The PROGNAME (from byte positions 152 to 163 of the header with 1st byte position is counted as 0) has been described in the Determine the data format type: 4A or 4B section above and here the data format is assumed to be type 4A, a single channel data.  So the NCHAN  = 1.

TIME_GMT (from byte positions 90 to 135) contains Year, Julian Date, Hour, Minute and Seconds.  An example of TIME_GMT will be

      115 213:21:47:57:862
where the Year value needs to be adjusted by adding 1900 so that e.g. 1900 + 115 = 2015, the JDy = Julian Day ( 1 to 366 ), Hr = Hour ( 0 to 23 ), Mn = Minute ( 0 to 59 ), and Sd:xxx = Seconds as Sd + xxx/1000.  So e.g. 57:862 =  57.862.

SRATEHZ (from byte positions 196 to 199) is a Long integer value that indicate the Sample Rate, e.g. 1000 Hz. Note that SRATEHZ should not be used as the true Sample Rate. The correct Sample Rate must be computed (see Compute the Sample Rate below) and the value should be very close to SRATEHZ.

SAMPLES (from byte positions 200 to 201) is a Short integer value to indicate how the data value in the file are being store after the header. SAMPLES can be either 0, 2 or 3 where
0 indicates the data after the header are in byte values;
2 indicates the data after the header are 12-bit unsigned integers store in the short integers, i.e., 2 bytes per 1 data point and the last 4 high order bits must be set to zeros in order to interpret the value correctly; and
3 indicates 16-bit unsigned integers are stored into the short integers, i.e. 2 bytes per 1 data point.

For the data format type: 4A and 4B, the SAMPLES most likely will be equal to 3, i.e., 16-bit unsigned integers will be used.

Also, see Byte Swapping below.

Compute the Sample Rate:

Because of the hardware setup, Sample Rate (Data Points per Second) must be determined for each data file by the total data points in a file divided by the file's total time segment in seconds.

(A) Total Data Points in a given data file will be
(File Size in Bytes - Header: 256 bytes )/(Data Size)/NCHAN
where Data Size is 1 if SAMPLES = 0 and Data Size is 2 if SAMPLES = 2 or 3; NCHAN ≥ 1.

(B) Total Time is seconds will be the End Time - Start Time where the Start Time can be obtained from the given data file, e.g. 000011.DAT.  The End Time, however, must be retrieved from the next following data file, e.g. 000012.DAT and use its start time as the End Time.

Then (A)/(B) will give the correct Sample Rate for the data file: 000011.DAT. If the next data file is not available or does not exist, SRATEHZ will need to be used or use the computed Sample Rate from the very last 2 data files.

Byte (8-bit) Values: <-- Most likely it will not be used by the data format: 4A and 4B.

When SAMPLES = 0, the data are 8-bit byte numbers. To get the byte numbers, read them as unsigned 8-bit numbers: B. Then convert them into signed integers: I = B - 127.

Byte Swapping:

When SAMPLES = 2 (12-bit) or 3 (16-bit) numbers, the data are stored into the 2 bytes short integers and are generated by the Big Endian format, i.e, the most-significant, left most bit or byte. Byte Swapping for the short integers will be needed if a computer with the little endian format is used.  Then the unsigned values (UI2) need to be converted to signed integers: I2 = UI2 -  N where N = 2048 for 12-bit data or N = 32768 for 16-bit data.

Header Format for Autonomous Hydrophone Data Files Type 4A:

The header format is shown in C programming language.  Note that the 1st and 2nd 32 bytes are same as in the data format Type 3, the remaining header bytes: 192 will be different in Type 4A data format.

typedef struct {
// ----- First 32 bytes of BIR header is working variables ------
  char   BIRHdrID[4];  // "BIR\0"
  ushort BIRVersion;   // version.release * 10
  ushort BIRUserHeaderSize; // size in bytes for user header
  ushort BIRUnused;    // currently unused (zero)
  ulong  RTCsecs;      // current RTC seconds
  ushort RTCticks;     // current RTC ticks
  ulong BIRCapacityBytes;  // total capacity of current drive
  ulong BIRStartFreeBytes; // free space on current drive at start
  ulong BIRReceivedBytes;  // total number of bytes received (likely to wrap)
  ulong BIRWrittenBytes;   // total number of bytes written (likely to wrap)
// ----- Second 32 bytes of BIR header is copy of VEE settings ------
  long CFPPBSZ;     // size of CompactFlash buffer, typ. 40MB
  long RAMPPBSZ;    // size of data RAM PP buffer, typ. 16 to 64KB
  long RAMHDBFSZ;   // size of CF to HD copy buffer, typ. 16 to 64KB
  long MINFREESZ;   // minimum free space until switch to next drive
  char HDDOSDRV[4]; // DOS drive assigned to hard disk ("D:")
  short NODRVTEST;  // testing without accessing drive
  short UARTMONIT;  // sending diagnostics to RS-232 port
  short FLOGFLAG;   // flag to log for major events (startup, spinups)
  short BIADEVICE;  // device type attached to BigIDEA(s)
  short CURBIA;     // index to current BigIDEA/drive
  short CURPRTN;    // index to current DOS partition

#define    ID_LENMAX       4   //System ID length
#define   LAT_LENMAX      10   //Latitude length
#define  LONG_LENMAX      12   //Longitude max
#define   GMT_LENMAX      46   //GMT char length
#define  LOGF_NAME_MAX    14   //JHG-2003-07-28 max filename length for log file
#define     EXPID_LENMAX  16   //EXP ID, usually year
#define  PROGNAME_LENMAX  12   //Added by hm NOAA 09/30/2002

// The remaining header space 192 bytes that contain No NCHAN parameter which means
// the PROGNAME will Not equal to any 1 of the following names below:
// CFxLogSP3i2_4.c, CFxLogSP3i3_3.c, CFxLogSP3i3_3.c, CFxLogSP3i3_4.c with or w/o the ".c"
typedef struct //user header space 192 bytes 
    char     PLTFRMID[ID_LENMAX];        //System ID (e.g., G001) 4
    char     LATITUDE[LAT_LENMAX];       //Latitude in degrees  10 N45:02.356=N45deg 02.356min
    char     LONGITUDE[LONG_LENMAX];     //Longitude in degrees 12 W128:34.872=W128 deg 34.872min
    char     TIME_GMT[GMT_LENMAX];       //System ID, Lat, Lang and GMT time 46
    char     EXPID[EXPID_LENMAX];        //Exp ID   16
    char     PROGNAME[PROGNAME_LENMAX];  //Program name  12
    ushort   ACQVersion;                 // version.release * 10
    ushort   WARMUP;                     //Pre-amp warm up in sec prior to the A/D logging
    char     PROJID[ID_LENMAX];          //Project ID 4
    char     LOGFILE[LOGF_NAME_MAX];     //Filename for event logging 14 chars
    short    STARTUPS;                   //number of program resets we've seen
    short    MAXSTRTS;                   //Maximum allowable program resets up to 255
    long     MAXNUMFIL;                  //Maximum number of files
    short    GAIN;                       //additional pre-amp gain 0 to 3 with 6dB inc
    long     SRATEHZ;                    //sample rate in hertz. NEW===SRATEHZ is a long value.===NEW
    short    SAMPLES;                    //sample size and type 0=8 bit, 2=12bit(2 byte), 3=16 bit
    short    PWFILT;                     //Pre-whitening filter setting
    short    LOPASS;                     //Low pass filter cut off
    ushort   SLEEP;                      //sleep in hours before program launched
    ulong    ACTIVESEC;                  //Active logging period in second
    ulong    DUTYCYCLE;                  //Duty cycle in seconds
    short    HYDROSENS;                  //Hydrophone sensitivity
    char     PRAMPNAME[10];              //PreAmp revision up to 8-char long
    ulong    WAKEUP;                     //Time to start logging in sec since 01/01/1970 11/02/99 NOAA hm
    char     DAQNAME[10];                //DAQ board name
    char     HYDROSRN[6];                //Hydrophone serial number
    ushort   FILECOUNT;                  //File count
    short    TESTSEC;                    //Test sec prior to the real logging.  Logs all 8 channel.
    short    STANDBY;                    //Standby in second before launching program
    char     dummy[2];
    }    ACQData;

Note that the last 4 parameters: FILECOUNT, TESTSEC, STANDBY and dummy[2] in the typedef struc { ... } ACQData  above are unique to the data format type: 4A here.  They are not in the header for the data format type: 4B except the TESTSEC; but its byte position is different.

Instructions for Decoding Autonomous Hydrophone Data Files

The following instructions are one of many ways to decode the data file. The objective is to give users examples for reading the header and data. Note that the following codes only read the essential information from the header in order to compute the sample rate and read the data correctly.

Decoding Autonomous Hydrophone Data (Format 2) Block using IDL Software Package

The following IDL commands will decode an autonomous data file 1 block at a time. Note that this section is assuming readers know IDL and how to refer to the IDL manual for details.

Enter IDL ( in UNIX system, type IDL then IDL> should show up )

IDL> OPENR, 10, 'Type4AFormat.Data'    ; is the file name.
IDL> BLK = ASSOC( 10, BYTARR( 256 ) )  ; 256 is the block size in bytes
                                       ; and it is also the Header size.
;    BLK will allow users to get 256 bytes of data per read.

IDL> HDR = BLK[0]  ; Read in the Header: HDR which will be 1-D byte array
                   ; with 256 elements from 0 to 255.

IDL> PRINT, 'Latitude : ', STRING( HDR[068:077] )
IDL> PRINT, 'Longitude: ', STRING( HDR[078:089] )
IDL> PRINT, 'TIME_GMT : ', STRING( HDR[090:135] )

IDL> SRATEHZ   = LONG( HDR, 196, 1 )  ; Sample Rate in Hz. Long Integer.
IDL> SAMPLES   =  FIX( HDR, 200, 1 )  ; Sample Size: 3 means 16-Bit Short Integer.

;    If your computer is using Litte Endian Format, you must do the Byte Swapping.
;    Do the following statement to find out.

IDL> LITTLE_ENDIAN = (BYTE(1,0,2))[0] EQ 1B  ; If LITTLE_ENDIAN is 1, 
;                 then your computer is using the Little Endian Format.

IDL> BYTEORDER, SRATEHZ, /LSWAP  ; Skip this if your computer is Big Endian.
IDL> BYTEORDER, SAMPLES, /SSWAP  ; Skip this if your computer is Big Endian.

IDL> PROGNAME = STRING( HDR[152:163] )  ; Program Name='CFxLogSP3i2_3', e.g.

IDL> PRINT, 'PROGNAME: ', PROGNAME  ; Check to make sure that it is the data format: 4A.
;    i.e. the PROGNAME is Not equal to 1 of the following names:
;    CFxLogSP3i2_4.c, CFxLogSP3i3_3.c, CFxLogSP3i3_3.c, CFxLogSP3i3_4.c with or w/o the ".c"
;    If PROGNAME = 1 of the names above, Get the NCHAN parameters as NCHAN = HEADER[248]

IDL> NCHAN = 1  ; PROGNAME is not = 1 of the names above.

IDL> PRINT, 'NCHANnals         : ',  NCHAN

;    Decode data blocks.  Assuming SAMPLE Size Type is 0, i.e. 1-byte data.
IDL> B   = BLK[1]   ; Read in the 2nd block, the 2nd set of 256 bytes.
IDL> RCD = B - 127  ; Convert the data into the range between -127 to 128.
;    where RCD will be an integer array of 256 elements.
IDL> DATA = TEMPORARY( RCD )  ; Optional: Assign RCD into DATA at 1-D array.

;    Decode the next data block.
IDL> B   = BLK[2]  &  RCD = B - 127  ; Process 2 IDL statements together.

;    Note that the order of which block to read can be random, i.e.
;    B = BLK[10] & ...  and later B = BLK[5] & ... are OK.

     ::: etc :::

;    Decode data blocks with SAMPLE Size Type = 3 (Short Integer) or 2 (12-Bits Integer).
IDL> B  = BLK[1]            ; Read in the 2nd set of 256 bytes from the data file.
IDL> I2 = UINT( B, 0, 128 ) ; convert 256 bytes into 2-bytes unsigned integers
                            ; with 128 elements.
IDL> BYTEORDER, I2, /SSWAP  ; Required if Little Endian computer is used.
;    Convert I2 to a 2-bytes signed integer array.
IDL> I2 = I2 - 2048         ; if Sample Size = 2.
IDL> I2 = I2 - 32768        ; if Sample Size = 3.
IDL> DATA = TEMPORARY( I2 ) ; Optional: Assign I2 into DATA at 1-D array.

     ::: etc :::

IDL> CLOSE, 10  ; Close the data file when you are done.

Decoding Autonomous Data Block using matlab Software Package

The following matlab commands will decode all the data in an autonomous data file. Note that this section is assuming readers know matlab and how to refer to the matlab manual for details.

Note that the matlab fopen() function will read the Big Endian format to binary the data as the default (however the 'b' option still should be used see the (1) below); therefore, no bytes swapping is needed.

Enter matlab ( in UNIX system, type matlab then the matlab working window will show up )

    Open a data file (1) and Skip the 1st 68 bytes
    (BIR working and VEE setting variables) from the file (2).
>>> FID    = fopen( 'Type4AFormat.Data', 'r', 'b' );  % (1)
>>> status = fseek( FID, 68, 'bof' );                 % (2)

    Read in the next 10 bytes for the LATITUDE.
>>> a      = fread( FID, 10 );
>>> disp( char( a' ) )  % LATITUDE.

    Read in the next 12 bytes for the LONGITUDE.
>>> a      = fread( FID, 12 );
>>> disp( char( a' ) )  % LONGITUDE.

    Read in the next 46 bytes for the TIME_GMT.
>>> a      = fread( FID, 64 );
>>> disp( char( a' ) )  % Show TIME_GMT.

    Skip the 1st 196 bytes from the beginning
>>> status = fseek( FID, 196, 'bof' );

    Read in the next 4 bytes as a long integers for the Sample Rate.
>>> i      = fread( FID, 1, 'long' );    % Read in 1 long integer.
>>> i      % Shows the SRATEHZ (Sample Rate)

    Read in the next 2 bytes as a short integers for the Sample Size.
>>> i      = fread( FID, 1, 'short' );    % Read in 1 short integer.
>>> disp( ['Sample Size = ', num2str( i )] )

    Skip the 1st 152 bytes from the beginning.
>>> status = fseek( FID, 152, 'bof' );    % Move to the correct position.
    Read in the next 12 bytes for the PROGNAME
>>> a      = fread( FID, 12 );
>>> disp( char( a' ) )  % Shows the PROGNAME

    Make sure the PROGNAME is Not equal to 1 of the following names:
    CFxLogSP3i2_4.c, CFxLogSP3i3_3.c, CFxLogSP3i3_3.c, CFxLogSP3i3_4.c with or w/o the ".c"
>>> nchan  = 1;  % Data Format 4A is a single channel data file.

    Assuming SAMPLE Size Type is 0, i.e. 1-byte data
    Decode data blocks as nchan x (pts = total data points per channel).
>>> status = fseek( FID, 256, 'bof'   );  % Skip the 1st 256 header bytes.
>>> blk    = fread( FID, [ nchan inf ], 'uchar' );  % Read in All the data after the header.
>>> rcd    = blk - 127;                   % convert the range into -127 & 128.

    Assuming SAMPLE Size Type is 3, i.e. 16-bit data
    Decode data blocks as nchan x (pts = total data points per channel).
>>> status = fseek( FID, 256, 'bof'    );  % Skip the 1st 256 header bytes.
>>> blk    = fread( FID, inf, 'uint16' );  % Read in All the data after the header.
>>> rcd    = blk' - 32768;  % convert the range into -32768 & 32767.

    ::: etc :::

>>> fclose( FID );  % Close the data file.
>>> quit            % Exit out of matlab.
Other Data File Formats

Data Format Type 4B, Type 3 and Type 2. Note that Data Format Type 1 was used for experiment and it is No longer being used.