Standard Post Maintenance Quality Checks (PMQC)

The two tests, i.e, correlator tests for stable fringes and receiver deflection tests to find out whether the receivers show expected deflections are important and fundamental. Even when all antennas are working satisfactorily, correlator and the receiver tests are essential, particularly to compare this with forthcoming feed rotation to same frequency band. It is preferable for these tests to be done after the pointing correction.

Preparing for the tests

The tests are to be run on when all antennas have been released after maintenance. Near the end of the weekly maintenance, usually a few antennas will not be in the working state either due to extended maintenance or due to some problem. A list of non-working antennas, as per the controlroom should be made around 3 pm. This list should be compared with the GCC to find out which are the antennas are under maintenance and which one has genuine problem. This will help to get back these antennas along with the antennas given for maintenance. It is preferable to wait till all antennas are released after the maintenance before starting the tests.

Pointing

It is recommended that after the weekly maintenance, both elevation and azimuth pointings be done. It should be done after loading zero offsets in respective axis. Load zero elevation offsets, run the elevation pointing, update the elevation pointing. Load zero azimuth pointing and similarly do the azimuth pointing. These values make a good database for FPS calibration.

Correlator Tests

First look at the matmon on a strong point source (3C48/3C147/3C286). Record a short stretch of lta and ltb files with shorter integration (say 8, 4 or 2s for ten min or so). Run tax on this and check the phase and amplitude. Some of the possible problems are

1. One or more antennas do not show fringes. Investigate why the antenna is 'dead'. Check (i) whether first LO is locked (ii) 30to1 output is sensible (iii) status at other sideband, other channel (iv) correlator bandshape etc

   (i) When the first LO shows unlock: Change the LO settings to 255 and come back to the required LO. Try setting slightly different LOs. If nothing works, raise a callsheet and followup.

   (ii) 30to1 output is too low: Check the attenuations, and follow this with receiver tests.

   (iii) The other sideband is working!: If the 30to1 is OK and only one sideband does not show fringes then the problem is localised to either baseband or correlator. Check the baseband output, if it is OK, then the problem is localised to correlator. Run corr_config and try. If nothing works raise a callsheet and followup.

   (iv) If the correlator bandshape is not satisfactory, run corr_config. Check if this antennas has different clock_sel.

2. One or a few antennas shows amplitude jumps similar to full scintillation. Record the data and check the bandshapes. If 130 and 175 bandshapes are showing identical selfs suspect delay rack. Check the same for other sideband also. Run newdly_config, if the problem persists, call for expert help..

   If the If 130 and 175 bandshapes are similar and highly correlated, but not identical, it is likely to be due to LO, discuss with engineers.

3. A set of antennas (usually in blocks of four, five or ten, depends on where the problem is) not showing fringes.. Run newdly_config and corr_config. If the problem does not go, call for expert help. Also check whether the tube lights are ON in the correlator room, ensure that they are OFF.

4. Some baselines showing very high cross or 9999. Run corr_config and check. Also check whether there is saturation by checking correlator and 30to1 bandshapes.

5. Antenna showing '0' for cross and self. Such problems usually comes if (i) the antenna is not set, (ii) baseband power too low or (iii) the corresponding cable is not connected to sampler in the correlator room. Inform GCC. Also check the baseband output, and the connectivity from baseband to correlator.

   (i) The antenna is not set: Run the settings once more.

   (ii) baseband power too low: Check the baseband and 30to1 output. If the baseband power is too low in one channel, check the connection from baseband to correlator.

   (iii) Cable problem: Check the cable inside correlator room.

6. 130 and 175 self bandshapes identical. This is to do with delay rack, run newdly_config.

7. Antenna bandshape replicated, channels 64 to 127 are replica of 0 to 63. Run newdly_config.

8. Some of the problems with delay rack does not go with newdly_config and require power reset. This should be done only by the correlator group or GCC.

9. Recording a short stretch (10 min) data with 128 milli-second integration on a few channels (say, every 16th channel) will also be useful. Check self power as a function of time, fringes, dropouts, etc.

Ionospheric scintillations sometimes confuse with the correlator problems. Recording a small stretch of data on a strong calibrator and looking at the amplitude and phase will help to clarify. It is possible to differentiate between ionospheric and correlator problems. Scintillations usually dominant at arm antennas (long baselines) and shortest baselines should continue to give reasonable fringes. In an given arm, the amplitude of phase change will increase with the baseline length in case of ionospheric problems. For example, C09-W03 should show larger but similar phase fluctuations than C09-W01 and C09-W06 should exhibit even larger, similar fluctuations. In the case of correlator problems, it will be independent of the baseline length.

Receiver Tests

Receiver tests are essential to find out whether the antennas give the desired amount of deflection against a strong source. This needs 30to1 to check the deflection on strong sources (CYGA, CASA, CRAB) and can be done when the correlator is not available. The second sensitivity test is done using pulsar mode of the correlator and uses relatively weak sources (point sources with flux of a few tens of Jy will do). Even when all antennas are working satisfactorily, the deflection tests are essential, particularly to compare this with forthcoming feed rotation to same frequency band.

Checking the authif table:

This is a simple and short experiment. Point the antenna to the cold sky., i.e, to a calibrator source not stronger than 30 Jy or so.. Run corresponding authif. Wait for two minutes. Record 1 dump at 30to1 and run 'difftable' on this file. This will give a list of antennas which significantly deviate from authif values. If the difference is more than 6 dB, the reason why such a large difference needs investigation. Raise a callsheet.

30to1 deflection tests:

The better way of doing this experiment is by doing power equalisation at the 30to1 output on strong source. Eventhough using authif values gives reasonable power equalisation, on strong sources a deviation of 4 dB from the operating point could lead to saturation and could lead to wrong interpretation of low sensitivity. Therefore to reduce the scatter, it is recommended to power equalise at the 30to1 output to a value of 50 dBm on a strong source.

Set the IF pre-and post attenuation to large value (say 20 + 20 dB). While running power equalisation programme, please note that there is a delay of about 1.5 to 2 minutes in updating the power levels at the 30to1 display, after the attenuations have been changed. Hence, if the power equalisation is done too quickly, it will never converge.

After the power equalisation to 50dBm, record 3 dumps on source and 3 dumps off source. For off source, go to a cold part of the sky without disturbing the attn settings. It is safer to go to a weak VLA calibrator (flux about a Jy or less).

If less than half an hour is available, this experiment can also be done by running authif +8 or 12 dB as per requirement (if the expected deflection is around 8 dB add 8, if 12 dB, add 12). In this case there is no need to do power equalise at 30to1 output. Go to strong source. Record 3 dumps per antenna for all antennas on source and off source. Sometimes this does not equalise to 50dBm at 30to1 for all antennas (on source), hence for those antennas where the power levels differ significantly from 50dBm, the results will be less reliable. While interpreting 'poor' sensitivity results, please look at the on source power level to find out about the possibility of saturation.

Run the programme 'getdefl'. To run this, login to tcc-l1 as 'elab' and move to directory '/data/cspa/data'. Make a directory for the day like 'ddmmmyy' (eg: 01jan05). Move the 30to1 data files to this directory if it is located elsewhere. 'getdefl' will give two output files, one PS file and one ASCII table giving the deflection values. The PS file contains all the informations, like bandshape, deflection, on-source power, off-source power etc. Check whether the deflection you get is as expected.. Look for some of the following problems.

1. Some antennas do not show proper deflection. Check pointing, if required re-focus the feed. If no improvement, FES/ABR is to be suspected. If only one channel is dead, try swapping. If the dead channel swaps, the problem is before swap-switch, if does not swap, the problem is after swap-switch, in the IF chain. You can also try to see the deflection with noise cal.

2. Some antennas show ripples in the passband, particularly at higher frequencies. This is known as Elevation Dependent Oscillations (EDOs). Note down the azimuth and elevation..

3. Bad bandshapes (slope, dips etc). Try RF swap to check whether the problem is before before swap-switch or not. This could also be due to gain mismatch at OTx and/or ORx.

4. Spurious spikes. Repeat the experiment with FE terminated (with attenuations adjusted appropriately to bring the power level as close to 50dBm as possible). If the spike is present even after terminating FE, it is generated internally, mostly at the antenna base. Further test can be done by changing LO by 5 MHz step and checking whether the spike moves by  times 5 MHz. These can be done by GCC, while investigating further.

Bring these to the notice of GCC. Most of the above problems can also be seen in the spectrum analyser kept in the Receiver Room. For problematic antennas, go to Receiver Room, check the bandshapes in the spectrum analyser independently. This will help to rule out any possible error introduced by 30to1 switch.

Correlator deflection tests:

This procedure allows the user to get a quick and useful estimate of the basic parameters (like Ta/Tsys, Mean/RMS and S/N) of each antenna from a  10 minute observation of a standard calibrator source, using correlator data output. The steps are as follows:

1. Track one of the standard calibrator sources of moderate strength e.g. 3c286, 3c48, 3c147 (even some extended sources like 3c353 can be used); however very strong sources like VIRGO, CYGNUS, Crab etc should NOT be used for this experiment.

2. Configure correlator in standard pulsar mode: corr_config -conf pulsar, ALC off (NOTE: if fstop has been running, just killing it is NOT sufficient: a fresh corr_config -conf pulsar HAS TO BE DONE to erase the effects of the last fringe and fstc settings!)

3. Do power equalisation of antenna signals (preferably from inside pulsar_console).

4. In pulsar console, select "Ant Cal Run" option from within the "Antenna" pop-up sub-menu. Set the correct cal source name and "Save" - this should update the appropriate file (calibration.set in /temp2/data/astro/pulsar on bhaskar) which has the sequence of commands to move the antennas OFF src (+/- 5 deg in decl) and ON src from the calibrator.

5. Set the name of the file in which you want to record the correlator self data e.g. /pulsar2/psr_sw/Tcltk/SCRATCH/cal_test.dat. NOTE : this is NOT a lta/b file! Set the "Duration" of the recording to be 8 minutes.

6. In Online, open the SAC file /temp2/data/astro/pulsar/calibration.set and start execution of it.

7. When the antennas have just moved from ON source to OFF source for the first time in the run, start the recording of the correlator data, using "Start Run" in the "Calibration Setup" window. The acquisition into the file will automatically stop after the specified duration is over. Source goes to off source.

8. The recorded data can be monitored (even while the acquisition is going on) and also analyzed using the "Plot Cal Run" option in the "Antenna" window. Here, data from any number of antennas can be plotted simultaneously, for any selected frequency channel of the correlator. Further, the transitions between the ON and OFF regions ("edges") can be identified and the signal levels for the ON and OFF regions can be computed from which quantities like Mean/RMS, Ta/Tsys and S/N for each antenna can be computed and plotted as a function of channel number. These allow for easy monitoring of the basic health parameter of each antenna.

ALL TCLTK program are available in astro2:/pulsar2/psr_sw/Tcltk/GUI. Login as pulsar and type tcl (this should take you to the required directory).

Callsheets

Whenever the problems are confirmed, callsheet should be generated with full information of RF settings, what all have been tried from control room before giving up.

Troubleshooting

While the tests are going on smoothly, some of the following problems can occur.

1. MSEB power reset: Some antennas, after MSEB power reset, gives intermittent time outs. Give reset through mcm 0. If all mcm time out, give ABC reset through telephone. If nothing works, inform GCC. Sometimes, antennas also go off due to high voltages. Inform GCC and wait for the voltage to come down.

2. RF Switch Operation: The usual medicine is to put 30+30 dB attenuations or FE termination. Sometimes these does not work. Try changing LO to a very different value (say, to the default 1st LO of another RF band). Telephone reset also can be tried. One of them will get the antenna back.

3. Command failed: Issue the command again. Note down which COMH the command failed. If occurs frequently, call GCC. It is known that command no. '0' fails, which comes with a cycle of 145 commands.

4. Radio Frequency Interference (RFI): Check which are all the antenna show strong RFI lines. Is there a pattern? Are these antennas close to housing? Controlled experiments can be done by the RFI team in coordination with MSEB whether RFI goes down of the MSEB power is switched off at certain locations.

Also refer to the attached document by K. Sanjay "No Fringe, or Fringe Vanished? Try This!".

Please send your comments to ishwar@ncra.tifr.res.in

No Fringe, or Fringe vanished? Try This!

This document is based on the experience from control room. More points can be added by individuals if they have, which are not covered in this document.

No Self?

• Make sure that RF, LO, IF & BB has been set properly, if not set it again.

• Make sure the bandshapes, from correlator o/p to see the baseband is properly set and the channel used (in matmon) to see fringe has proper power in it. Some time baseband fails to set one set of antenna and all such antennas remains at old settings. But this happens for 4 antennas (one pol[130 or 175] only) at a given time.

• If self at corr o/p is zero, and BB power is OK, try noise in correlator and make sure correlator is giving right o/p power. If it is, there is likely to be a problem with connecting cables from BB to CORR. If correlator is not being used for observation, self test can be done.

Self ok but no fringe?

• Set LO and confirm LO LOCK.(LO monitoring need to make robust!)

• Check if existing pointing offsets are not loaded.

• Find out whether feed is not rotated or is in acceptable range of counts given by theodolite experiment.

• If it is just start of new Frequency after long time. make sure swap/unswap has been taken into account, try swapping RF in such antenna and update the RF-swap table accordingly.

• If calibrator is being observed, a small pointing can be done to see if the there is pointing offset by loading different offsets w.r.t. the existing one, rather than taking antenna out of subarry. Same test can be done in case antenna is giving low fringe amplitude. User need to load pointing offsets in steps of the beamwdth of current Freq. in observation, which is usefull to save time. This happens mostly if feed is rotated for some reason after the last pointing was done.

• Make sure antenna/ABC/servo takes the proper source co-ordinate (target) and track accordingly. 'sacw' window in ondisp(ONLINE) should always show zero error below that antenna for elevation and azimuth while antenna is in tracking.

• If simultaneously five antennas are failed, and they are from same FFT control card, try corr_config, or let correlator expert attend it.

• If only CSQ antennas are showing fringes and not arm antennas, it means either 'fstop' is not running or das has been given wrong source or somehow 'fringe stop' has failed.

• If self power keeps on fluctuating, or showing constant high power, it is likely to be a RFI in that channel only and in that antenna only (in case the RFI is generated in system of same antenna). check the bandshape, and matmon in other channel.

If fringe amplitude is low

• Check phases on calibrator. If phases are winding, find out if any other antenna which shows similar winding. If such a pair exists, check the connectivity of cables in OF/BB/corr or mapping in corrsys.hdr, for whether they have got swapped. If there is such a pair for which there is no fringe amplitude and phase for no clear reason, and the pair form longer baseline, the problem is most likely to be of this sort.

• If fringe amplitude is low and phase is showing slightly more or no scatter, then check if the RF noise has left on.

• Same, if fringe amplitude is low and phase also showing somewhat more scatter, check if the antenna need RF swap and that antenna has more polarization leakage, so that the fringe amplitude user get is not real but leaked.

If fringe vanished Suddenly:

• Check if servo has applied the brakes, OR ABC has got reset.

• Check if that antenna is showing 'flat bandshape' at correlator output.

• If fringe falls gradually, it's most likely that, ONLY ABC got reset and other electronic, viz. RF, LO, IF, are still ok, but servo stopped tracking as effect of ABC reset.

• Check if RF switch has got operated, in that case, ABC starts giving ABC T.O. and self power in matmon falls to very small value and 30-1 monitoring also show the fall in power leve.

• Check if feed has got rotated, few antennas had problem of feed rotation on power on reset.(this test can be ignored if such problem does not exists.)

• Make sure the antenna setting was not got changed (by mistake) with some other antenna under maintainance.

• If fringe in all antennas has gone down by some extent, it's more likely that, while loading pointing offset in one antenna, the same offset has got loaded in all antenna by mistake.

• If fringe in all antennas suddenly stopped, check the master LO is OK, or correlator is getting STA as well as GPS clock (using irq_seq)

• Check if LO1 has been become unlock. (more technical clarification needed here, for can this really happen)

If all antennas showing no (or unstable) fringe:

• check if proper CLK_SEL was selected in 'init.hdr.new' file and the same is given to 'dlytrk' with a option '-clk_sel'. Also make sure appropriate BB bandwidth selected.

• Make sure, by using 'showband' that the channel selected for seeing fringe, in matmon is not affected by any unstable features like, RFI or band oscillations.

• Watch if the error in 'acq30' o/p is showing 'processing lagged behind'. As effect of this record does not print the 'number of STA cycles' (under title "wt") same as given (e.g. while in no error condition, if LTA1=16 & LTA2=8, the wt should be equal to 128, a product of LTA1 & LTA2, in error condition this number will be lesser than that.)

• For individual antennas, if fringe started showing unstability, check if that antenna has started showing EDO.

• If all antennas showing fluctuation in phases as well as fringe amplitude, it's well known ionosperic scintillation.