NEFSC SCS data archiving operation


As we, the NEFSC, become more involved with ocean observing systems, it has become more important to develop operational routines. Given multiple vessels conducting year-round sampling, we have an opportunity to make a significant contribution to the northeast regional system. To do so and to make things run smoothly, we would like to settle on some standard file formats and filenaming conventions.

The NEFSC alongtrack SCS data is a prime example of the type of data that could be used in an observing system. In order to efficiently serve this data to the community it will be best to have all shipboard technicians who handle the data during and after the cruises following the same protocol. While the SCS data has been collected and archived successfully for a few decades now, there are a few minor inconsistencies between cruises that we would like to eliminate in the future.

In order to automate the post-processing routines, it is best to have cruise names, for example, follow the same convention. This has been a problem for years with both the DELAWARE and the ALBATROSS data. Different shipboard technicians are involved and new ones come aboard all the time. They do not always follow the examples of their predecessors in the way they name the cruise directories on the NEFSC disk. We will now address the particiular issues one at a time:

Items to standardize:

1)Cruise name conventions

Recent cruises on the DELAWARE, for example, were called:

de06016 leg2
DE07-01 29JAN-08 FEB
DE07-2 07MAR-11 MAR

Having cruises named with these various forms slows down the automated post-processing routines. For the purposes of archiving the data on the NEFSC disks, would like to propose a simple form like:


We ask, therefore, the NEFSC ET operators: a) never include blank spaces b) stick to upper case c) if a cruise does not have multiple legs, remove the "-1"

2a) Data subdirectories and file formats

There are a set of subdirectories and a variety of file formats that are output to the NEFSC disk after each cruise. EVENTDATA (all uppercase) is one of them and that should include a "HydroEvent.elg" file at the end of each cruise. This is the file that is used in post-processing as of March 2010. Where we had previously used the set of compressed .aco files, we now use the hydro event log and ask that that be run with each cruise.

5) Uses of the data:

a) Every few months, the oceanography branch post-processes a set of cruises. A perl and MATLAB routine had been reading several files, generating a cleaned and merged file w/one minute resolution, makes plots of both the raw and processed data, and loads the output into an ORACLE table where it is webserved. The data gets loaded 2-3 times per year in batches. This operation is documented at here.

b) The oceanography branch also uses the SCS data on an occassional basis to cross-check observations with the shipboard CTD and the position logged by the CTD operators. The HydroData.ELG is used in this operation.

c) The survey group does not routinely use the SCS data after it has been loaded on the NEFSC but does occassionly need to look at some files if there is an audit problem.

d) Both the BIGELOW and the DELAWARE routinely sends raw one minute data on a near-daily basis to Florida State University's "Shipboard Automated Meteorologic and Oceanographic System". We do not routinely use this data because it does not include salinity.

e) The NODC eventlog is generated for each cruise and submitted to NODC although we are not sure who is responsible for this data, how often it is sent, and what kind of quality control is conducted at NODC.

6) Acoustic Doppler Current Profiler

A renewed attempt to process NEFSC ADCP data is underway. This is being done using the UHAWAII "CODAS" system, a MATLAB based routine to post-process .LTA files. If and when a system can be developed to efficiently handle the ADCP data and the product looks accurate and reliable, ADCP might become a standard product along with the SCS data. The issue under investigation at the time of this writing is the "angle correction" that needs to be applied to provide accurate currents relative to the seafloor. This effort is documented at here but this ADCP page is no longer linked to from anywhere as it needs to be updated. In the near future, if we can resolve the processing problems with ADCP data, we may request that the technicians load the *.LTA files in a separate "ADCP" directory under each cruise.

Northeast Fisheries Science Center
Oceanography Branch

Oceanography at the NEFSC