\chapter{Readout Electronics}
\section{Overview}
The goal of the \GX{} readout electronics system is to digitize and read out
the detector signals for level 1 trigger rates of up to 200 \khz\ without
incurring deadtime. A pipelined approach is required. The digitized 
information will be stored for several $\mu s$ while the level 1 trigger is 
formed.  Multiple events must be buffered within the digitizer modules and 
read out while the front ends continue to acquire new events.

A summary of the \GX{}  detector subsystems from an electronics viewpoint is
shown in figure~\ref{fig:detectors_gluex}.

Two basic types of readout electronics will be used in \GX , \fadc s and 
\tdc s.  Detectors which measure energy will be continuously sampled with 
flash \adc s while detectors which require precise time measurements will 
use a multi-hit \tdc. No currently available commercial solutions exist.  
These boards will be designed by our collaboration.  Prototypes have been 
constructed, and are being tested.

The number of channels in the \GX{} detector is not large enough to justify
the financially risky development of custom integrated circuits.  ICs developed for
other experiments will be used as well as commercially available chips.
Programmable logic devices will be extensively used for data path, memory, and control functions.

Technology is constantly evolving, and the optimum solution for
the \GX{} detector depends on when funding becomes available and the
construction schedule. Presented here is a preliminary design which could be 
implemented with currently available components.

\begin{figure}[p]\centering
\includegraphics[angle=90]{ch7_detectors_gluex.eps}
\caption[Detector subsystems]{\label{fig:detectors_gluex}
Detector subsystems}
\end{figure}  

\section{FADCs for Calorimetry}
The calorimeters will be read out with 8-bit, 250 \mhz\  \fadc s.
The 250 \mhz\ sampling clock will be derived from the 1499 \mhz\ accelerator
clock. This sampling rate and bit depth is well matched to the FEU84-3 
\pmt s used in the Forward Calorimeter, and is adequate for the silicon 
\pmt s used in the 
Barrel Calorimeter.  Additional \fadc\ channels will read out
the Photon Tagger, Backwards Veto, Start Counter, \v{C}erenkov 
Detector, and time-of-flight \pmt s.



\begin{figure}[htb]
\begin{minipage}[t]{5.25cm}
\includegraphics[height=5.25cm]{ch7_FEU84-3_puls.eps}
\caption[Digitized FEU84-3 pulse]{\label{fig:puls} Digitized FEU84-3 pulse.}
\end{minipage}
\hfill
\begin{minipage}[t]{5.25cm}
\vspace{-5.47cm}
\includegraphics[height=5.25cm]{ch7_FEU84-3_250_8.eps}
\caption[Resolutions]{\label{fig:resol} 
Dashed - calorimeter resolution, Solid - \fadc{} resolution.}
\end{minipage}
\end{figure} 


Figure ~\ref{fig:puls} shows an FEU84-3 \pmt{} pulse digitized 
by the prototype \fadc{} described in section~\ref{sec:fadc}. Note that the 
sum of the samples from 120 to 180 \nstunit{} is 1429; for this \pmt{} the 
8-bit \fadc{} is equivalent to a 10 or 11-bit conventional charge-integrating
\adc . To address resolution concerns, simulations were performed to show 
that the proposed \fadc{} provides an adequate measurement. Pulses measured 
with a digital ocilloscope were fitted to determine their functional form. 
The response of the \fadc{} was simulated using this functional form and the 
time integral of the function was compared to the summed output of the 
simulated \fadc{} for many pulses. Since the relationship between deposited 
energy and pulse height in this type of calorimeter is known, direct 
comparison of the resolution due to the \fadc\ and the resolution of the 
calorimeter is possible. Figure ~\ref{fig:resol} shows the 
result of this comparison. Clearly, above $0.15$ \eunit{} the resolution of 
the \fadc{} is small compared to the intrinsic resolution of the 
calorimeter.


The \fadc s will also give a measurement of the time a photon arrived at the
calorimeter. Previous work \cite{vavra,cussans} indicates that a time 
resolution better than the \fadc{} sampling interval can be achieved by 
fitting the \fadc{} waveform. To study how well this time could be determined
a ``library'' of pulses from phototubes of the type to be used was created 
using a digital ocilloscope with a 2.5 {\em{GHz}} sampling frequency. The 
leading edge of these sampled pulses were fitted to a 9th order polynomial 
to determine the location of various ``features'' of the pulses. The features
considered were the time the pulse achieved 10, 25, 50, 75, 90 and 100\% of 
its maximum value. These features carry the arrival time information of the 
pulses and were used as reference times.

To determine how well the \fadc{} could determine the pulse
arrival time, the samples from the digital ocilloscope (2.5 {\em{GHz}})
were averaged over 10 samples (to 250 \mhz) and quantized
to 8 bits. These transformed samples are what would be expected
from the \fadc{} system proposed here. Using only the bin
containing the pulse maximum and the two samples preceeding it
and a simple algorithm,
it was found that the 50\% crossing time could be determined
with a resolution of 160 {\em{ps}} compared to the time determined
by the detailed fitting described above. This resolution is sufficent
to determine if a pulse is in time with an event (rejecting
background) or to determine the time of the event sufficiently well
to select the beam ``bucket'' that initiated the event.


\subsection {Prototype \label{sec:fadc}}

A single channel prototype of the calorimeter \fadc\ has been designed and 
built at Indiana University.  A block diagram is shown in  
Fig.~\ref{fig:FADC_block} and a photo in Fig.~\ref{fig:FADC_photo}.
%
\begin{figure}[h!]\centering
\includegraphics[width=5.5in]{ch7_FADC_block.eps}
\caption[FADC Block Diagram]{\label{fig:FADC_block}
Block diagram of prototype \fadc\ board.}
\end{figure}  
%
\begin{figure}[h!]\centering
\includegraphics[width=0.65\textwidth]{ch7_FADC_photo.eps}
\caption[FADC Board Photo]{\label{fig:FADC_photo}
Photograph of prototype \fadc\ board.}
\end{figure}  

A differential amplifier inverts the negative \pmt{} signal and shifts the 
voltage levels to match the input range of the digitizer integrated circuit.
The digitization is performed by an SPT7721 integrated circuit manufactured 
by Fairchild Semiconductor \cite{spt}.  This IC costs about US\$20. An 8-bit value
 is produced internally every $4$\nstunit ; two samples are output every $8$\nstunit{} ($125$ \mhz).

All digital functions are performed in a Xilinx \cite{xilinx} XC2S50 
programmable gate array. This IC costs about US\$10.
A dual port RAM configured as a circular buffer stores the data for $8$ 
microseconds.  Upon receipt of a trigger signal the data from the time 
window of interest is copied to an output FIFO which can buffer the data 
from multiple events. This FIFO is interfaced to a $32$ bit, $33$ \mhz{} 
PCI bus.  More information on  this prototype is available \cite{paulrd}.

\subsection {Additional requirements for final version}

The final version of the calorimeter \fadc{} board will include pipelined
adders operating at the 125 \mhz\ digitizer output clock which 
continuously sum the digitized information from all channels on a board.
Additional pipelined adders will sum the information from all boards in a
crate, and then sum the information from all the crates associated with a
detector.
The sum of all channels will be passed through a shift register giving a time
history.  Successive samples within a programmable time window will be summed,
analgous to the gate in a conventional charge sensitive \adc.
Energy sums from the Forward and Barrel Calorimeters will be used in the level 1 trigger.
 A block diagram is shown in Fig.~\ref{fig:Energy_sum}.

\begin{figure}[t]\centering
\includegraphics[height=2.86in,width=5.4in]{ch7_Energy_sum.eps}
\caption[Energy Sum]{\label{fig:Energy_sum}
Block diagram of energy sum.}
\end{figure}  


Assuming a 100 \nstunit\ time window, each \fadc\ channel will produce 25 
bytes of data per level 1 trigger.  In the final version of the \fadc\ we
will want to suppress the readout of channels with no data.  The \fadc\ data
will be processed in real time to provide an energy and time measurement.  
 We believe that the raw \fadc\ data can be reduced to about 10 bytes per
channel.  This zero suppression and pulse shape processing may be done at the
 channel level in the gate array, 
at the board level, the crate level, the detector system level, or in some
combination of these levels.

\section{FADCs for Tracking}
The Central Tracking Drift Chamber anodes will be read out with 10 or 12-bit, 
125 \mhz\  \fadc s. The additional dynamic range is required for the {$dE/dx$} 
measurement. The Forward Tracking Drift Chamber cathodes will be read out 
with 10 or 12-bit 62.5 \mhz\ \fadc s. The exact read out electronics requirements 
for these detectors is the subject of ongoing R\&D efforts.

\section{TDCs}
The Photon Tagger, Start Counter, Forward Drift Chamber
anodes, \v{C}erenkov Detector, Barrel Calorimeter, 
and Time of Flight detectors will be read out by multi-hit \tdc s.


\subsection {Prototype}

Such a high resolution pipeline \tdc\ module has been developed for use at 
Jefferson Lab,    
and is designed to meet the requirements of current experiments, as 
well as to serve as a prototype for future experiments, including Hall D.  Fifty units
have been produced for use in Hall C at Jefferson Lab.
The design is implemented as a VME-64x module.  This bus standard was 
chosen because it is already in use at Jefferson Lab, has good (and evolving) 
data transfer capabilities, and reasonable channel densities are possible.  
A block diagram is shown in 
Fig.~\ref{fig:F1TDC_block} and a photo in Fig.~\ref{fig:F1TDC_photo}.

\begin{figure}[h!]\centering
\includegraphics[width=0.50\textwidth,angle=90]{ch7_F1TDC_block.eps}
\caption[\tdc{} Block Diagram]{\label{fig:F1TDC_block}
Block diagram of prototype \tdc\ board.}
\end{figure}  
%
\begin{figure}[h!]\centering
\includegraphics[width=0.65\textwidth]{ch7_F1TDC_photo.eps}
\caption[\tdc{} Board Photo]{\label{fig:F1TDC_photo}
Photograph of prototype \tdc\ board.}
\end{figure}  

The module is built around the TDC-F1 integrated circuit from 
Acam-Messelectronic Gmbh \cite{acam}, originally designed for the COMPASS
experiment at CERN \cite{compass}.  This chip costs about US\$130 each in 
small quantities.  It includes many features that will be useful in 
experiments at Jefferson Lab.  This chip utilizes purely digital delay 
techniques to measure time.  In normal mode the TDC-F1 chip provides 8 input 
channels with up to 120 ps resolution (LSB).  In high resolution mode 
channels are combined in pairs to yield up to 60 ps resolution for 4 input 
channels.  The dynamic range for measurement is 16 bits.  The resolution of 
the chip is tunable about its nominal value.  
A PLL circuit adjusts the core voltage of the chip to compensate for 
temperature and supply voltage variation, assuring stability of the 
resolution value.  On-chip buffering for input channels, triggers, 
and output data allows for multihit operation with nearly zero deadtime.  
The chip also has a complex trigger matching unit that can filter out hits 
unrelated to the input trigger. When enabled, only hits that are within a 
programmed timing window relative to the trigger time are kept.
       

The 8 TDC-F1 chips on our module provide 64 channels in normal mode, or 
32 channels in high resolution mode.  A 128K word deep FIFO is attached to 
each TDC-F1 chip to buffer its output data.  The module can be set up to 
interrupt the crate controller after a programmable number of triggers have 
been received.  
During read out the module will provide a block of data associated with a 
programmed number of triggers, and then terminate the transaction.  To 
enhance system performance a set of TDC modules may be read out as a single 
logical read using a multiblock read protocol.  This involves passing a token 
between modules along a VMEbus daisy-chain line.  In this setup, only the 
first module in the chain will
generate the interrupt, and only the last module in the chain will terminate 
the transaction.


The TDC module is fabricated as a single 12-layer printed circuit board.  
Six of these layers carry signals on 5 mil wide traces.  Internal layers are 
organized in a stripline configuration and have a characteristic impedance of 
50 ohms.  
High-speed ECL components are used for the front end.  Logic in a single FPGA 
chip (484-pin BGA) controls the entire module.  The high density of the 
design demands that surface mount components be installed on the front and 
back sides of the board.
%
\begin{figure}[h!]\centering
\begin{minipage}[t]{6.5cm}
\includegraphics[width=6.5cm]{ch7_tdc_low.eps}
\caption[TDC Performance]{\label{fig:ch7_tdc_low} 
TDC performance in low resolution.}
\end{minipage}
\hfill
\begin{minipage}[t]{6.5cm}
\vspace{-6.5cm}
\includegraphics[width=6.5cm]{ch7_tdc_hi.eps}
\caption[TDC Performance]{\label{fig:ch7_tdc_hi} 
TDC performance in high resolution.}
\end{minipage}
\end{figure} 
%
\subsection{Initial test results}
The layout and power sections of the bare boards were tested and verified, 
and then the programmable logic IC using a ball grid array package was 
installed by industry. Programmable logic was loaded and tested successfully.
We then proceeded with initial tests of the the F1 chip itself. The prototype
was outfitted with a single F1 chip, corresponding to eight channels of low 
(normal) resolution, or four channels of high resolution. All channels were 
found to be operational and tested for performance. Timing signals were 
generated with a Highland V851 digital delay generator with a measured jitter
of approximately 30~ps\footnote{The V851 specifications for jitter is 25 
ps.}. The time calibration at low resolution was found to be $115.1$ ps/count
(120 ps/count nominal) and $57.5$ ps/count in high resolution (60 ps/count 
nominal). The time resolution of two typical channels are shown
in Figs.\,\ref{fig:ch7_tdc_low} and \ref{fig:ch7_tdc_hi} indicating
$\sigma=$ 69~ps in low resolution and $\sigma=$ 59~ps in high resolution. 
Subtracting the contribution from the time generator we obtain $62$ 
\pstunit{} and $51$ \pstunit{} for the prototype performance in low and high 
resolution respectively. This is to be compared with the quoted rms 
resolution of the F1 chip of $40$ \pstunit .


\subsection {Additional requirements for final version}

A 62.5 \mhz{} clock phase-locked to the accelerator will be distributed to 
all \tdc\ modules so that every channel has the same time calibration.  
With this clock frequency, LSB resolutions will be 125~\pstunit{}
for the standard version and 62.5 \pstunit\ for the high resolution 
version.

A commercial \tdc\ module, the CAEN\cite{caen}v1290, based on the HPTDC 
chip developed at CERN, is being evaluated by Jefferson Lab.  It might 
be possible to use this type of \tdc\ to provide 31 \pstunit{} LSB 
resolution for the time-of-flight counters.

\section{Track Count}
The Start Counter, Barrel Calorimeter, and Time of Flight detectors will
require pipelined adders to implement a track count for use in the level 1 trigger.

\section{Clock Distribution and Pipeline Synchronization}
A 62.5\mhz{} global front end clock will be phase-locked to the 1499\mhz{} 
accelerator clock. This global clock will be distributed to all readout 
electronics crates and used directly by the \tdc{}s and Forward Drift chamber 
cathode \fadc{}s. The Calorimeter and Central Drift chamber \fadc{}s will
phase-lock their higher frequency clocks to the global front end 62.5\mhz{} 
clock.  A global reset will be distributed to synchronize all pipelines.  
The level 1 trigger as well as calibration and other special triggers will 
also be distributed globally.

\section{Discriminators and Amplifiers}
Detectors which provide precise timing (Photon Tagger, Start Counter,
Barrel Calorimeter, and Time of Flight counters)
will require ``constant fraction'' discriminators.
The Forward Drift Chamber anodes require leading-edge discriminators
mounted on the chambers inside the solenoid.  
The Central Drift Chamber anodes and Forward Drift Chamber cathodes will need
an amplifier to drive their \fadc\ inputs.  These amplifiers should be located
on the chambers. ICs developed for the Atlas \cite{atlasasd} detector are being 
considered for all chamber mounted electronics.


\section{High Voltage}
The Forward Calorimeter \pmt s will be powered by Cockcroft-Walton voltage 
multipliers \cite{cwbase} which will be built at Indiana University.
This type of base provides for the very low power consumption necessary
for such a tightly packed array and is controlled over a CAN communication
link. One hundred prototypes of these bases have been built at Indiana 
University and are being tested. The other \GX{} \pmt{}s will likely use 
conventional resistive divider bases powered by commercial HV power supplies.
The Tracking Chambers will use commercial HV power supplies with sensitive 
current monitoring.
\begin{figure}[p]
\includegraphics[angle=90]{ch7_racks.eps}
\caption[Rack Space]{\label{fig:racks}Rack space}
\end{figure}  

\section{Packaging}
The calorimeter \fadc\ circuit requires about 50 \cmlunit$^2$ of board space and adjacent
channels will need to be about 2 - 3 \cmlunit\ apart.  This implies a 
density of about 8 channels on a 6U board or possibly 16 channels on a 
9U board.

The tracking \fadc{}s will be designed at Jefferson Lab. Sixteen channels of 125 \mhz{} \fadc\
for the Central Drift chamber should fit on a 9U board.
The exact requirements for the Forward Drift cathode \fadc{}s have not been determined;
it should be possible to put 64 channels of 62.5 \mhz{} \fadc\ on a 9U or possibly even
a 6U board.

In the low resolution (125 \pstunit) version of the \tdc\, 64 channels fit 
on a 6U board.  This version of the \tdc\ is used for the Forward Drift 
Chamber anodes.

The Photon Tagger, Start Counter, Barrel Calorimeter, and Time 
of Flight counters require the high resolution (62.5\pstunit) version of
the \tdc{}; the density is 32 channels per 6U board.

Assuming a maximum of 18 boards in a crate, and 3 9U or 4 6U crates in a 
7 foot tall rack, figure~\ref{fig:racks} summarizes the space required for 
the readout electronics. Estimated counts for high voltage, gas handling,  
level 1 trigger, and other necessary racks are also included.


The readout electronics will be located as close to the detector as possible 
to minimize signal cable runs. Note that the Tagger electronics will be be
located in a separate building 80 \mlunit\ upstream of the main detector.  
The Time of Flight and Forward Calorimeter electronics will be downstream of 
these detectors.  Cabling from detectors inside the solenoid will exit at the
upstream and downstream ends of the magnet and connect to nearby electronics.
Fiber optic cables will transport the data from the crate readout processors to the
level 3 trigger processor farm in the \GX{} counting house.

\section{Readout Bus}
FASTBUS crates are no longer being manufactured, and FASTBUS is not being
considered for \GX.  
CAMAC crates are fairly slow and have limited board space and power 
available.  Some legacy devices like discriminators, trigger logic or
HV supplies which are not part of the data readout may be packaged in CAMAC,
but not the bulk of the readout electronics.

VME is popular at Jefferson Lab and the \tdc\ prototype is constructed on a 
VME64x card.  Compact PCI is used extensively in the telecommunications 
industry and can be driven directly by typical FPGA ICs without the need for
bus interface ICs.  Predefined PCI interface ``cores'' are available, 
minimizing design time.  One disadvantage of cPCI is that bridges are
 required
for a system with more than 8 slots, although commercial bridges 
which consume no slots are available.
VXI and PXI are ``instrumentation'' extensions to VME and cPCI.  Shielding, 
triggering, clock distribution, and additional power are added to the basic 
bus standard.

The \fadc s require a low skew fanout of the 250 \mhz\ clock, a 
synchronization signal, and the level 1 trigger.  The need to form
a digital global energy sum for the level 1 trigger will probably drive the 
choice of packaging for the calorimeter \fadc s.  Some sort of custom 
backplane will be required to support the trees of adders which form the
energy sum and track counts.

The telecommunications industry is moving towards 
``Switched Serial Fabrics.''
This adds a high speed serial connector to the backplane which can support 
Ethernet and other high speed serial technologies.  For VME the applicable 
standard is VXS (VITA 41) and for cPCI the standard is cPSB (PICMG 2.16). 
 For a 16 channel \fadc\ module producing 25 bytes per channel per level 1 
trigger; a level 1 trigger rate of 200\khz; and a 2\% occupancy the data
readout bandwidth required for a module is 16 Megabits per second, well
within the capability of a 100baseT Ethernet connection.



\section{Construction}
Indiana University has experience building large electronic systems for
experiments at Fermilab, Brookhaven, and Jefferson Lab.  The \GX{} experiment
is larger and more complex than past experiments and will require the
development of new techniques.  High reliability is crucial to the success of
the \GX{} experiment.  We plan to begin long term tests of \GX{} electronics
as soon as they are produced giving early identification of problems and 
failure modes.

\begin{figure}[h!]\centering
\includegraphics[width=0.75\textwidth]{ch7_spark400.eps}
\caption[Assembly Robot]{\label{fig:spark400}
Selective assembly robot.}
\end{figure}  

A robotic electronic assembly machine\cite{beamworks}  shown in figure~\ref{fig:spark400}
was purchased and used to construct 100 prototype Forward calorimeter 
Cockcroft-Walton \pmt bases. This device dispenses solder paste, picks and places 
components, and uses infrared lasers to selectively solder components  without
disturbing nearby devices.  This facilitates building a board in stages
and testing partial assemblies, a technique especially useful in producing
the Cockcroft-Walton \pmt\ bases. The infrared lasers are particularly useful 
for removing and replacing defective components. 

Producing electronics assemblies in house has several advantages over having 
a commercial firm doing the assembly.  To achieve the lowest cost, a 
commercial service would assemble a large batch all at once.  This risks 
learning about problems after it's too late to change anything. Assembling
smaller batches in house allows immediate feedback to the assembly process.

The lifetime of the \GX{} experiment will be long enough that we must plan 
for maintenance and repair of the custom electronics.  Sufficient spare parts 
must be purchased at construction time to avoid the risk of a component 
manufacturer discontinuing some crucial part.  Spreading the purchase of
components over too long a time also risks some components becoming
unavailable.

Jefferson Lab used a commercial board assembly contractor to build the 50 \tdc{}
boards used in Hall C.  No major problems were associated with this contract
assembly, although it was necessary to quickly test the first few units before
committing to assembling the remainder.




\section{Review}
The \GX{} electronics were reviewed by one internal and two outside reviewers in 
July of 2003. The report of this review is attached as an appendix.  The review 
was scheduled at an early stage in the development of \GX so that their 
recommendations could be incorporated into final designs.  The guidance of 
the committee has been extremely useful in the continued prototyping and design 
of the system.

The present design for the \GX{} electronics differs in a few ways from the 
reviewers recommendations:
\begin{list}
{\setlength{\rightmargin}{\leftmargin}}
%
\item At the time of the review it was believed that a single \fadc{} module 
could be used for all detectors. The detector characteristics have been further 
defined since the review and it seems unlikely that one type of \fadc{} is 
optimal for all detectors.  The channel count for the Forward drift cathodes has
grown substantially; the overall system cost will likely be lowered by designing 
a module specifically for this detector.  The different types of \fadc{} will 
be as similar as possible; it may be possible to use a common printed circuit 
board stuffed with different components.
%
\item The reviewers recommended locating the readout electronics in a radiation 
free area. The collaboration believes the advantages of shorter signal cables 
outweigh the access considerations and is planning to locate the electronics 
in crates adjacent to the detector.  Access to the above ground \GX{} detector 
should be easier than access to the existing underground experiments at 
Jefferson lab.
%
\end{list}
