NEFSC SCS data archiving operation
As we, the NEFSC, become more involved with ocean observing systems, it has
become more important to develop operational routines. Given multiple
vessels conducting year-round sampling, we have an opportunity to make a
significant contribution to the northeast regional system. To do so and to
make things run smoothly, we would like to settle on some standard file
formats and filenaming conventions.
The NEFSC alongtrack SCS data is a prime example of the type of data that
could be used in an observing system. In order to efficiently serve this
data to the community it will be best to have all shipboard technicians who
handle the data during and after the cruises following the same protocol.
While the SCS data has been collected and archived successfully for a few
decades now, there are a few minor inconsistencies between cruises that we
would like to eliminate in the future.
In order to automate the post-processing routines, it is best to have cruise
names, for example, follow the same convention. This has been a problem for
years with both the DELAWARE and the ALBATROSS data. Different shipboard
technicians are involved and new ones come aboard all the time. They do not
always follow the examples of their predecessors in the way they name the
cruise directories on the NEFSC disk. We will now address the particiular
issues one at a time:
Items to standardize:
1)Cruise name conventions
Recent cruises on the DELAWARE, for example, were called:
DE07-01 29JAN-08 FEB
DE07-2 07MAR-11 MAR
Having cruises named with these various forms slows down the automated post-processing
routines. For the purposes of archiving the data on the NEFSC disks,
would like to propose a simple form like:
We ask, therefore, the NEFSC ET operators:
a) never include blank spaces
b) stick to upper case
c) if a cruise does not have multiple legs, remove the "-1"
2a) Data subdirectories and file formats
There are a set of subdirectories and a variety of file formats that are
output to the NEFSC disk after each cruise. EVENTDATA (all uppercase)
is one of them and that should include a "HydroEvent.elg" file at the end
of each cruise. This is the file that is used in post-processing
as of March 2010. Where we had previously used the set of compressed .aco
files, we now use the hydro event log and ask that that be run with each
5) Uses of the data:
a) Every few months, the oceanography branch post-processes a set of
cruises. A perl and MATLAB routine had been reading several files,
generating a cleaned and merged file w/one minute resolution, makes plots of both the
raw and processed data, and loads the output into an ORACLE table where it
is webserved. The data gets loaded 2-3 times per year in batches.
This operation is documented at here.
b) The oceanography branch also uses the SCS data on an occassional basis to
cross-check observations with the shipboard CTD and the position logged by
the CTD operators. The HydroData.ELG is used in this operation.
c) The survey group does not routinely use the SCS data after it has been
loaded on the NEFSC but does occassionly need to look at some files if there
is an audit problem.
d) Both the BIGELOW and the DELAWARE routinely sends
raw one minute data on a near-daily basis to Florida State University's "Shipboard Automated
Meteorologic and Oceanographic System". We do not routinely use
this data because it does not include salinity.
e) The NODC eventlog is generated for each cruise and submitted to NODC
although we are not sure who is responsible for this data, how often it is
sent, and what kind of quality control is conducted at NODC.
6) Acoustic Doppler Current Profiler
A renewed attempt to process NEFSC ADCP data is underway. This is being done
using the UHAWAII "CODAS" system, a MATLAB based routine to post-process
.LTA files. If and when a system can be developed to efficiently handle the
ADCP data and the product looks accurate and reliable, ADCP might become a
standard product along with the SCS data. The issue under investigation at
the time of this writing is the "angle correction" that needs to be applied
to provide accurate currents relative to the seafloor. This effort is
documented at here but this ADCP page is no longer linked to from anywhere as it needs to be
updated. In the near future, if we can resolve the processing problems with
ADCP data, we may request that the technicians load the *.LTA files in a
separate "ADCP" directory under each cruise.
Northeast Fisheries Science Center
Oceanography at the NEFSC