Data Acquisition and Quality Control

Introduction

Stations from nearly all networks operated by the BSL transmit data continuously to the BSL facilities on the UC Berkeley campus for analysis and archive. In this chapter, we describe activities and facilities which cross-cut the individual networks described in Chapters 3 - 7, including the facilities in McCone Hall, procedures for data acquisition and quality control, sensor testing capabilities and procedures, and a collaborative experiment in early warning.

While some of these activities are continuous from year to year, we have identified changes or activities which are specific to 2003-2004.

Figure 8.1: Data flow from the BDSN, NHFN, MPBO, HRSN, and BARD network into the BSL central processing facility.

McCone Hall Facilities

The routine data acquisition, processing, and archiving activities of the BSL are carried out in McCone Hall. The BSL facilities in McCone are designed to provide air conditioning, 100-bit switched network, and reliable power with UPS and generator.

Because of the mission-critical nature of the automated earthquake processing, most computer systems operated by the BSL run on circuits with both UPS and generator power. Air conditioning is provided through both "building air" and two additional AC units.

Power

Over the years, the BSL has experienced problems with the McCone generator system, including a failure in 1999 due to a combination of a weakened power system and a leak in the water pump. In the 2001-2002 Annual Report, we described the failure of the McCone and Byerly generators in the March 7, 2002 campus-wide power outage.

While the failure of the generator at Byerly Vault was traced to PPCS human error (the generator had been left in a mode where it would not automatically start when power was lost), the failure of the McCone generator was due to poor maintenance. Similar to the situation in 1999, it failed due to problems in the power system combined with a leak in the water pump.

In the fall of 2002, BSL staff met with Eric Haemer, Sara Shirazi, and several others from PPCS to discuss maintenance and routine load testing of the McCone generator. As a result, the McCone generator is scheduled for quarterly load tests and bi-monthly run tests.

Air Conditioning

In parallel with power problems, the BSL has faced cooling problems in room 237 in the past year. As with power, the growth of the computing systems in the past year has led to an increased heat load. This came to a crisis during the fall of 2002, with peak temperatures in the computer room exceeded 85$^{\circ}$ when the AC unit failed. After consideration of several options, the BSL decided to add an additional AC unit to room 237. The new unit (which is not supported by UPS/generator power) has helped keep systems running, although the BSL held a summit with PPCS staff in February 2004 to review ongoing cooling problems. As a result, PPCC worked with the contractor to install a larger impeller on the chilled water circulation pump. This has increaseed the cooling capacity of our dry cooler on the roof. In addition, PPCS has been coming by more regularly to inspect the filters on the unit.

New Facilities

The BSL is actively working with the campus to relocate the critical operations of data acquisition, processing, archiving, and distribution to a more robust facility. With assistance from the Office of the Vice Chancellor for Research, the BSL has been granted space in 2195 Hearst, a recently completed building on the Oxford Tract. 2195 Hearst was designed to current codes and has been given special attention for post-earthquake operations. The BSL is gearing up to relocate their critical computers in the fall of 2004.

Data Acquisition

Central-site data acquisition for the BDSN/NHFN/MPBO is performed by two computer systems located at the BSL (Figure 8.1). These acquisition systems are also used for the Parkfield-Hollister electromagnetic array and for the BARD network. A third system is used primarily as data exchange system with the USNSN and receives a feed from CMB, HUMO, MOD, SAO, and WDC from the the NSN VSAT. This system also transmits data to the USNSN from HOPS, CMB, SAO, WDC, and YBH. Data acquisition for the HRSN follows a more complicated path, as described in Chapter 5.

Figure 8.2: Dataflow in the REDI processing environment, showing waveform data coming in from the Quanterra data loggers (Q) into comserv. From comserv, data are logged to disk (via datalog), distributed to other computers (mserv), fed into the CDA for REDI processing, and spooled into a trace ring for export.

Comserv

The BSL uses the comserv program for central data acquisition, which was developed by Quanterra. The comserv program receives data from a remote Quanterra data logger, and redistributes the data to one or more comserv client programs. The comserv clients used by REDI include datalog, which writes the data to disk files for archival purposes, cdafill, which writes the data to the shared memory region for REDI analysis, and other programs such as the seismic alarm process, the DAC480 system, and the feed for the Memento Mori Web page (Figure 8.2).

The two computers that perform data acquisition also serve as REDI processing systems. In order to facilitate REDI processing, each system maintains a shared memory region that contains the most recent 30 minutes of data for each channel used by the REDI analysis system. All REDI analysis routines first attempt to use data in the shared memory region, and will only revert to retrieving data from disk files if the requested data is unavailable in the shared memory region.

Most stations transmit data to only one or the other of the two REDI systems. The comserv client program cs2m receives data from a comserv and multicasts the data over a private ethernet. The program mcast, a modified version of Quanterra's comserv program, receives the multicast data from cs2m, and provides a comserv-like interface to local comserv clients. This allows each REDI system to have a comserv server for every station.

We have extended the multicasting approach to handle data received from other networks such as the NCSN and UNR. These data are received by Earthworm data exchange programs, and are then converted to MiniSEED and multicast in the same manner as the BSL data. We use mserv on both REDI computers to receive the multicast data, and handle it in an identical fashion to the BSL MiniSEED data.

BH Sampling Rate

The BSL converted most - but not quite all - of the BDSN BH data streams from 20 samples per second to 40 samples per second at the beginning of the GMT day on June 15, 2004 (day 167). This change affects: BDM, BKS, BRIB (broadband sensor only), CMB, CVS, FARB, HOPS, HUMO, JCC, JRSC, MHC, MNRC, MOD, ORV, PACP, PKD, POTR, RFSB (surface strong-motion only), SAO, SCCB, WDC, and YBH. Stations BRK, KCC, and WENL were converted at the beginning of day 168.

This change in sampling rate was made in order to be consistent with the USArray specifications. At this time, we do not intend to change the sampling rate of the BP channels associated with the BSL borehole networks - the NHFN, MPBO, and HRSN.

Seismic Noise Analysis

BSL seismic data are routinely monitored for state-of-health. An automated analysis is computed weekly to characterize the seismic noise level recorded by each broadband seismometer. The estimation of the Power Spectral Density (PSD) of the ground motion recorded at a seismic station provides an objective measure of background seismic noise characteristics over a wide range of frequencies. When used routinely, the PSD algorithm also provides an objective measure of seasonal and secular variation in the noise characteristics and aids in the early diagnoses of instrumental problems. A PSD estimation algorithm was developed in the early 1990's at the BSL for characterizing the background seismic noise and as a tool for quality control. As presently implemented, the algorithm sends the results via email to the engineering and some research staff members and generates a bargraph output which compares all the BDSN broadband stations by components. A summary of the results for 2003-2004 is displayed in Figure 3.3. Other PSD plots for the NHFN, HRSN, and MPBO are shown in Figures 4.3, 5.2, and 7.4 respectively.

Three years ago, we expanded our use of the weekly PSD results to monitor trends in the noise level at each station. In addition to the weekly bar graph, additional figures showing the analysis for the current year are produced. These cumulative PSD plots are generated for each station and show the noise level in 5 frequency bands for the broadband channels. These cumulative plots make it easier to spot certain problems, such as failure of a sensor. In addition to the station-based plots, a summary plot for each channel is produced, comparing all stations. These figures are presented as part of a noise analysis of the BDSN on the WWW at http://www.seismo.berkeley.edu/seismo/bdsn/psd/.

The PSD algorithm has been documented in previous annual reports.

Sensor Testing Facility

The BSL has set up an instrumentation test facility in the Byerly Seismographic Vault in order to systematically determine and to compare the characteristics of up to eight sensors at a time. The test equipment consists of an eight-channel Quanterra Q4120 high-resolution data logger and a custom interconnect panel that provides isolated power and preamplification when required to facilitate the connection and routing of signals from the sensors to the data logger with shielded signal lines. Upon acquisition of the 100 samples-per-second (sps) data from the instruments under test, PSD analysis and spectral phase coherency analysis are used to characterize and compare the performance of each sensor. Tilt tests and seismic signals with a sufficient signal level above the background seismic noise are also used to verify the absolute calibration of the sensors. A simple vertical shake table is used to access the linearity of a seismic sensor.

The sensor testing facility of the BSL is described in detail in the 2001-2002 Annual Report.

Instruments tested during the past year include three Keck OBS broadband seismometers, three STS-2 broadband seismometers, CITRIS accelerometers and a tiltmeter.

Figure 8.3: Closeup photo of the back of the signal/power cable connector on the STS-2 control box, installed at WDC, showing the corrosion of the wires. The subsequent leakage between the cables generated coherent long period noise.

Figure 8.4: Photo of STS-2 sensors $\char93$50205 (right) and $\char93$20022 (left) after they were installed and leveled on the seismic pier in the BKS vault. After this photo was taken, the sensors were covered with a couple layers of R-13 fiberglass batt insulation to minimize temperature variations and improve their low frequency performance.

STS-2s

On August 26th, 2003, the malfunctioning STS-2 broadband sensor ($\char93$39335), which was removed from WDC, was installed for testing. The symptom was that there is coherent long period noise on all three broadband channels and the external signal/power connector was visibly corroded (owing to water standing in the dome in which it was installed at WDC) (see Figure 8.3). The conductors in the signal/power cable were reterminated and the sensor was taken to BKS for noise testing. During the installation in the sensor testing facility, it was discovered that the bubble level on the base plate of the sensor, which is critical for leveling the seismometers, had no bubble. The fluid had leaked out of the bubble level so that it could not be used to level the instrument. On September 8th, after installing a new bubble level on its base plate, STS-2 $\char93$39335 was installed and successfully leveled for testing. The subsequent background noise PSD tests showed that the STS-2 was operating within its nominal specifications.

During the month of February 2004, a pair of STS-2 broadband seismometers ($\char93$20022 and $\char93$50205) were installed in the BKS vault (see Figure 8.4) to characterize their performance prior to installation at remote BDSN seismic stations. STS-2 $\char93$20022 is a "low power" unit, for which the factory supplied only nominal calibration information, which was originally installed at HUMO in June 2002. It was removed from service at HUMO in November 2002 because it was determined to be the source of the excessively high noise level observed on the broadband channels. As is our routine policy, it was tested after it was repaired. STS-2 $\char93$50205 is a new sensor for which the factory supplied detailed calibration information (with 5 complex zeros and 11 complex poles) for each of the three orthogonal triaxial sensors in the unit. The resulting tests showed that both sensors were operating within their nominal specifications. STS-2 $\char93$50205 was subsequently installed at FARB on August 21, 2004 and STS-2 $\char93$20022 awaits installation at a new BDSN station.

CITRIS Accelerometer

During the month of November, several calibration and noise tests were preformed on prototype CITRIS accelerometers developed by a team working with Steven Glaser in Civil Engineering. The calibration testing was done in the BSL machine shop in Room 298 McCone Hall during the first two weeks on November. The absolute calibration for each accelerometer was determined by tilting the sensor through an accurately measured angle (see Figures 8.5 and 8.6) and measuring its response. The data were transmitted to a laptop computer via a spread spectrum tranceiver and the data logging was done on the laptop with custom software written by graduate students working on the project. After the accelerometers were calibrated, the quiet background noise PSD test was done in the BKS Vault and the results are shown in Figure 8.7.

Figure: Closeup photo of the custom test apparatus assembled to rotate the CITRIS accelerometer through a range of accurately measurable angles up to +/-30 degrees. The accelerometer is an integral component on the circuit board (top) and the large pipe (bottom center) is rotated on the channel to tilt the accelerometer. A laser pointer (black tube mounted on pipe diagonal) is used to infer the rotation angle from the position of the projected laser beam (see Figure 8.6).

Figure 8.6: Photo of the laser beam projected on a meter stick located 300 cm from the center of the rotation axis of the pipe in Figure 8.5. The position of the beam can be determined to an accuracy of 0.5 mm which is equivalent to measuring the tilt angle to an accuracy of 0.6 minutes of arc.

Figure 8.7: Plot of the background noise PSD for one of the CITRIS horizontal accelerometers under test. The background noise PSD estimate from the co-sited BKS Kinemetrics FBA-23 strong motion accelerometer (dashed line) is provided for reference. The low frequency noise PSD floor of the CITRIS accelerometer averages $\sim$83 dB.

Keck OBS

The Keck OBS testing reported on last year continued through July 2004 with the application of insulation to inhibit convection induced noise.

Between March and June another round of testing of the Keck OBS sensor began. Sensor $\char93$T1046 (see Figure 8.8) was tested to characterize its performance prior to installation on the sea floor at the MOBB site in Monterey Bay. Fiberglass batt insulation (see Figure 8.9) was tied around the titanium pressure sphere to improve the thermal stability of the sensor. During the testing in early April, we found the horizontal components to be excessively noisy. Upon investigation, we found that one of the washers was sited on what appeared to be a CaCl deposit which was powdery and loose and the pier had some small loose debris on its surface. Also, the thick cable which is attached at the top of the pressure vessel is a source of tilt related noise. We swept off the seismic pier and re-installed OBS on three ceramic washers, suspended the cable from the overhead airduct girders, and reinstalled the fiberglass batt insulation. The horizontal component noise was reduced significantly.

Figure 8.8: Photo of cast titanium pressure sphere housing Keck OBS $\char93$T1046. The disc shaped base is standing on three creamic washers for stability on the seismic pier. The orange signal cable exiting at the top of the sphere is draped overhead to minimize the generation of noise at the cable responds to temperature changes.

Figure 8.9: Photo of fiberglass batt insulation surrounding the pressure sphere.

Other Tests

During June, Pinnacle Technologies tested a prototype nanoradian resolution tiltmeter in the BKS vault. They supplied the sensor and recording system and they also did all the analysis. We provided only the quiet vault environment in which they could test their sensor.

Acknowledgements

Doug Neuhauser, Bob Uhrhammer, Lind Gee, Pete Lombard, and Rick McKenzie are involved in the data acquisition and quality control of BDSN/NHFN/MBPO data.

Development of the sensor test facility and analysis system was a collaborative effort of Bob Uhrhammer, Tom McEvilly, John Friday, and Bill Karavas. IRIS and DTRA provided, in part, funding and/or incentive to set up and operate the facility and we thank them for their support.

Bob Uhrhammer led the testing and problem solving effort of the Keck sensors, with help from John Friday, Doug Neuhauser, and Bill Karavas.

Bob Uhrhammer, Lind Gee, and Doug Neuhauser contributed to the preparation of this chapter.

References

Scherbaum, Frank. Of Poles and Zeros: Fundamentals in Digital Seismology, Volume 15 of Modern Approaches in Geophysics, G. Nolet, Managing Editor, Kluwer Academic Press, Dordrecht, xi + 257 pp., 1996.

Tapley, W. C. and J. E. Tull, SAC - Seismic Analysis Code: Users Manual, Lawrence Livermore National Laboratory, Revision 4, 388 pp., March 20, 1992.