Data Reduction Recipe for an LWS01 AOT.

The case of a faint source.

Sergio Molinari


0. About this recipe...

What this recipe is: What this recipe is not: This recipe is a worked example of real LWS data reduction and analysis; it covers many of the peculiar instrumental effects that affect the quality of your data, but certainly not all. In these cases we encourage you to check the LWS and LIA FAQs; should your question remain unanswered, please contact us at if in the U.S, or if in Europe.

Recent changes: Sects 3.2 and 3.4.

1. Definitions & Requirements

2. Inspecting your Data

A preliminary inspection of your data is useful to get a feeling of what you have in your hands, and to spot potential problems you will have to take care of during the data reduction. Let's start by entering ISAP:

3. Reprocessing your Data with LIA

Let's start this Section with a recommendation: it is always advisable to have a go with LIA, also if the preliminary analysis with ISAP did not show any particular problem. It is good to have a look at the dark current and at the raw data to check that everything is all right; a trend of decreasing dark current, e.g., may reflect in an increased scatter of your grating scans (i.e. an increased noise of your averaged spectrum). Besides, the comparison of your data with the Dark Currents will tell you if you are detecting signal or you just reached the sensitivity limit of the instrument.

There are 4 steps in the LIA reprocessing of an LWS01 AOT:

3.1 Dark Currents

The tool to be used is IA_DARK. A full tutorial describing its functionality is accessible at; in the following, it is assumed that you are familiar with that document (or at least have it at hand).

While at the ISAP> prompt, type: IA_DARK, tdt='TDT', where TDT is the eight digit number attached to the filename of all you data files. If you are running ISAP in a directory which is different from the one where your data files are stored, an additional parameter has to be given; the call would then be: IA_DARK,tdt='TDT',dir='your directory'. Once the widget is up, click on detector SW1. You will see this:

Fig. 5

The red crosses are the DC measurements, while the white points are your (uncalibrated) data; everything is plotted as a function of time. If your observations was taken at a revolution number earlier than about 400 you may see DC measurements (red crosses) taken in between the observation; however only the first and the last (before and after your observation) DC measurements can actually be used.

Here the various scans of the grating can be recognized as an oscillating pattern on your data. Remember that this data is still uncalibrated at this stage and the transmission profiles (called RSRF, 1 per detector) of the instrument are still to be divided out. To check where the different scans start and end, go with the mouse on a point and click the middle button of the mouse: the text area at the bottom of the IA_DARK widget will give you all the information regarding the nearest point to the mouse actual position.  Again, it is important to remember that at this stage the instrumental transmission is not yet divided out and, if there is sufficient flux from your source, it will manifest itself as a periodic pattern throughout your dataset; it is not anything you want to model and subtract as a DC or Gain trend. For a moment you think that all the spikes that you see on your data are lines, but your illusion only lasts few tenths of seconds. Zoom wherever you want on the above plot (on your LIA widget, of course) and you will find out that most of them are 'glitches' resulting from the detector being hit by an energetic particle, each followed by its decaying trend marking the recovery to normal conditions. Most of the glitches are located on the trailing edge of a previous glitch, so you may wonder what 'normal condition' means for this detector. Let's make a note that we will have to do a lot of zapping later in ISAP. The important thing to notice here is that glitches are present also during the dark current measurements; if you bring up the first DC measurement you will see this:

Fig. 6

You will note the couple of glitch trails at the end of the measurement. The median-clipped averaged performed on the whole set of points above gives the DC estimated made by the OLP, which is visible in the box 'OLP Used...' for 'Dark 0' in Fig. 6; it is clear however that this is an overestimate because the second part of the measurement is clearly corrupted. Also the first group of 7 points in the plot above looks like a recovery from a glitch happened prior to the start of the observation. It seems that the best estimate we can do of the dark current should be based on the central group of point in Fig. 6. Now go ahead and make your DC estimate also for the second DC measurement.

When you come to the subtraction of your estimated DCs from the data, a lot of common sense is needed. Take the example in Fig. 7.

Fig. 7

The only thing you want to try here is to remove trend in DC during your observations, and so you may want to try and interpolate following the lower envelope of your data.

If you have a Raster Map you will be in the uncomfortable situation to have a signal variation during the observation. In a single pointing observation, you are sure that the intrinsic signal from the source is not varying so that you can assign trends either to DC or Gain variations (and remove them). In a raster, unless you have an extended uniform brightness source, the intrinsic signal is varying; it is practically impossible to spot a trend in the instrument behaviour in this conditions.

You should by now be convinced that the variability you see in the signal (white points) in Fig. 7 is just the relative spectral response function of the detector which has not yet been divided out; do not try to model this stuff, as it will disappear when we will recalibrate our spectrum later. The best (and most conservative) thing you want to do at this stage is to assume that the DC is constant through out the observations. In general, if the DC measurements done before and after the observation are comparable, within the noise of the single measurements, we recommend you choose the linear interpolation option which uses the average of the two measurements. This is the option applied by the OLP, and unless the two measurements are significantly different ot there is a clear DC trend in your data, you should stick with that option. The case shown in Fig. 7 is very interesting because it represents an exception to this rule. Assuming a constant DC equal to the average of the two DC measurements and interpolating this value throughout the observation, we obtain the upper full line in the above figure. Now it is clear that there is something wrong with this, because after the DC subtraction more than 1/3 of the data points would be negative. The fact itself that we see the spectral profile of the detector during the grating scans is reassuring that some flux above the DC level must be falling on the detector; the true DC value must then be lower than suggested by the DC estimates we made above (see Fig. 6). We are therefore in the unpleasant situation where we do not have a reliable independent estimate of the DC; we can only guess that, based on the considerations above (in italic), the DC must be less or equal to the minimum value of the data (white points); it cannot be higher than that because, again, the transmission profile of the detector is clearly seen and this imply that the detector is seeing flux from the source.
Tip: force a value of DC similar to the minimum value (on average) reached by your data, by typing this value in the two Dark Measures boxes visible in Fig. 6 and hit return; then interpolate the DC assuming it constant and equal to this value.
Write down that you gave your best guess for the DC of this detector, since a further adjustment might be needed when you will look at your recalibrated spectrum.

Another peculiar situation we find in the current example is found for detector LW4:

Fig. 8

Apart from the periodic pattern due to the transmission profiles, we do see a decreasing temporal trend/drift in the data. Is this a DC or a responsivity trend ? It is a tough question and you might not always be in the condition to answer. In this case, however, we have a way out.
Tip: check the peak-to-peak amplitude of each RSRF pattern. Is this decreasing with time ? If YES, this is likely to be a responsivity (gain) trend; if NO, this will be a DC trend.

As for the previous discussion about detector SW1, you are guessing DCs here; write on your Log that you guessed DCs for this detector (LW4 in this particular case) to remind that you are authorized to make further adjustments later (we will discuss this later).

Keep in mind the considerations we have been doing so far, while proceeding with the DC estimate and subtraction for the rest of the detectors.

3.2 Responsivity Drifts

With the possibility of `shifting spectral scans'  which will be available with ISAP version 2.0, the routine IA_DRIFT, used to correct for temporal responsivity drifts, is likely to become obsolete. This routine was needed because the pipeline incorrectly estimate the responsivity drift before subtracting the DC; it will be much easier and less time consuming to do this with the new ISAP functionality.

The only plausible reason to still use IA_DRIFT is when there is evidence for a non-linear responsivity drift: please refer to the tutorial available at

We have just one special recommendation for you, when you are dealing with faint sources. Be careful that the drift you will fit to your data does not cross 0. If this happens, a portion of your data will be divided by very small numbers and the calibrated spectrum will contain points with enormous flux values (both positive and negative). When the source is faint, put a lot of care in selecting the highest signal portions of your grating scans to estimate the responsivity drift. When fitting trends in an observation which is a raster map, remember what we said above.  Please note that this problem will affect OLP Version 8 data.

3.3 Absolute Responsivity Correction Factors

The absolute responsivity correction factors can revised using the routine IA_ABSCORR, whose tutorial is available at Since the estimate of these factors is based on the Illuminator Flash data, the characteristics of the source observed are not important; the tutorial has all the information you need to proceed.

3.4 Recalibration

The recalibration of your data is independent on the type of source observed, and it is performed by the routine SHORT_AAL: instructions are found in If you did not use IA_DRIFT to remove the relative responsivity drifts, remember to run SHORT_ALL with the `/nodrift` option.

4. Data Analysis with ISAP

At this time, you should have in your directory a file called LSANxxxxxxxx_SHORT.FITS produced by SHORT_AAL; load this file into ISAP. You can do a straight average and compare the result with the averaged spectrum for the original LSAN file produced by the OLP. Once you are done, we can start with the real work to analyse this data.