NEFSC SCS data archiving operation

Purpose/Motivation

As we, the NEFSC, become more involved with ocean observing systems, it has become more important to develop operational routines. Given multiple vessels conducting year-round sampling, we have an opportunity to make a significant contribution to the northeast regional system. To do so and to make things run smoothly, we would like to settle on some standard file formats and filenaming conventions.


The NEFSC alongtrack SCS data is a prime example of the type of data that could be used in an observing system. In order to efficiently serve this data to the community it will be best to have all shipboard technicians who handle the data during and after the cruises following the same protocol. While the SCS data has been collected and archived successfully for a few decades now, there are a few minor inconsistencies between cruises that we would like to eliminate in the future.


In order to automate the post-processing routines, it is best to have cruise names, for example, follow the same convention. This has been a problem for years with both the DELAWARE and the ALBATROSS data. Different shipboard technicians are involved and new ones come aboard all the time. They do not always follow the examples of their predecessors in the way they name the cruise directories on the NEFSC disk. We will now address the particiular issues one at a time:


Items to standardize:

1)Cruise name conventions

Recent cruises on the DELAWARE, for example, were called:


DE06015_LEG2
de06015L3
de06016 leg2
DE07-01 29JAN-08 FEB
DE07-2 07MAR-11 MAR

Having cruises named with these various forms slows down the automated post-processing routines. For the purposes of archiving the data on the NEFSC disks, would like to propose a simple form like:


DE0702-1

We ask, therefore, the NEFSC ET operators: a) never include blank spaces b) stick to upper case c) if a cruise does not have multiple legs, remove the "-1"

2a) Data subdirectories and file formats

There are a set of subdirectories and a variety of file formats that are output to the NEFSC disk after each cruise:

COMPRESS
EVENTDATA
MISC

In order to keep the subdirectory names consistent bewteen cruises, we suggest the ET simply dump the entire SCS "datalogger" directory for each cruise to the NEFSC disk rather than create new subdirectory names.

The ".RAW" datafiles should then be in the upper level directory for each cruise. 

In each of these subdirectories there are a variety of files and file formats. The NODC required files, for example, are stored in the  EVENTDATA subdirectory along with the sensorhistory files. While we make a transition in 2009 to doing without aco files, we would still like to see the compressed ACO files in the "COMPRESS" subdirectory.

2b) Eventlogs

We are now considering more regular use of  both the "HydroEvent.ELG" and "Samos.ELG" files to obtain a 1-minute merged dataset so we would like those generated with each cruise in 2009. Eventually, we hope to eliminate the need for ACO files. 


3) Data filenames

In addition to the cruise directory name, there is also the issue of naming the data files themselves. This has remained fairly constant over the years and we hope the current convention is held in the future. If each ship continues to name their files as they currently are, the routines will run smoothly. DELAWARE, for example, stores their compressed fluorometry data as RAW-Fluro.ACO while the ALBATROSS stores it as FLURO.ACO. We understand these individual sensor files are derived from the sensor config file. As long as they remain the same bewteen cruises for each ship, no problems should arise.

4) Timing and location of downloads

It is our underatanding that the shipboard technicians load the cruise data on the NEFSC disk within a day or two after each cruise (usually before leaving for the next). We currently find the data at:

/delscs/public_html
/albscs/public_html
/bigelowscs/public_html

5) Uses of the data:

a) Every few months, the oceanography branch post-processes a set of cruises. A perl and MATLAB routine has been reading several files, generates a cleaned and merged file w/one minute resolution, makes plots of both the raw and processed data, and loads the output into an ORACLE table where it is webserved. While we have been routinely using the ".ACO" files stored in the "COMPRESS" directory, beginning in 2008 we are considering the use of the "hydro event log" to obtain dataset already merged to 1-minute resolution. This is not yet a realtime operation. The data gets loaded 2-3 times per year in batches. This operation is documented at here.

b) The oceanography branch also uses the SCS data on an occassional basis to cross-check observations with the shipboard CTD and the position logged by the CTD operators. The HydroData.ELG is used in this operation.

c) The survey group does not routinely use the SCS data after it has been loaded on the NEFSC but does occassionly need to look at some files if there is an audit problem.

d) According to recent communication with Peter Gamache, the BIGELOW sends raw one minute data on a near-daily basis to Florida State University's "Shipboard Automated Meteorologic and Oceanographic System". After the other two ships are upgraded to SCS 4.0, they are scheduled to do the same. It seems reasonable that we should coordinate our efforts more closely with this SAMOS system which is already in the routine of "realtime" processing.

e) The NODC eventlog is generated for each cruise and submitted to NODC although we are not sure who is responsible for this data, how often it is sent, and what kind of quality control is conducted at NODC.

6) Acoustic Doppler Current Profiler

A renewed attempt to process NEFSC ADCP data is underway. This is being done using the UHAWAII "CODAS" system, a MATLAB based routine to post-process .LTA files. If and when a system can be developed to efficiently handle the ADCP data and the product looks accurate and reliable, ADCP might become a standard product along with the SCS data. The issue under investigation at the time of this writing is the "angle correction" that needs to be applied to provide accurate currents relative to the seafloor. This effort is documented at here but this ADCP page is no longer linked to from anywhere as it needs to be updated. In the near future, if we can resolve the processing problems with ADCP data, we may request that the technicians load the *.LTA files in a separate "ADCP" directory under each cruise.

Northeast Fisheries Science Center
Oceanography Branch

Oceanography at the NEFSC