Skip to content

attipaci/crush

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

CRUSH

Author: Attila Kovacs <attila[AT]sigmyne.com>
Last updated: 12 July 2019


Table of Contents

  1. Getting Started

    • 1.1. Installation
    • 1.2. (optional) Java Runtime Configuration
    • 1.3. CRUSH Pipeline Configuration
    • 1.4. The Tools of the CRUSH Suite
    • 1.5 Quick Start
    • 1.6 A Brief Description of What CRUSH Does and How...
    • 1.7 Command-line Options and Scripting Support
    • 1.8 Making Sense of the Console Output
    • 1.9 Examples
  2. Advanced Topics

    • 2.1 Pointing Corrections
    • 2.2 Pixelization and Smoothing
    • 2.3 Recovery of Extended Emission
    • 2.4 Filtering corrections, transfer functions, etc.
    • 2.5 Image Processing Post-Reduction
    • 2.6 Reducing Very Large Datasets (to do...)
  3. Advanced Configuration

    • 3.1 Commands
    • 3.2 Pipeline Configuration
    • 3.3 Source (Map) Configuration
    • 3.4 Scan Configuration
    • 3.5 Instrument Configuration
    • 3.6 Advanced startup environment and Java configuration
  4. Correlated Signals

    • 4.1 Modes and Modalities
    • 4.2 Removing Correlated Signals
    • 4.3 User-defined Modalities
  5. Custom Logging Support

    • 5.1 Log Files and Versioning
    • 5.2 Decimal Formatting of Values
    • 5.3 Loggable Quantities
  6. Supported Cameras

  7. Future Developments (A Wish List...)

    • 7.1 Support for Heterodyne Receivers and Interferometry
    • 7.2 Interactive Reductions
    • 7.3 Distributed and GPU Computing
    • 7.4 Graphical User-Interface (GUI)
  8. Further information


1. Getting Started

CRUSH-2 is a reduction and imaging package for various astronomical imaging arrays. Version 2 provides a unified platform for reducing data from virtually any fast-sampling scanning camera.

1.1. Installation

Install Java (if not already installed)

Download Java, e.g. from www.java.com. If you have Java already, check that it is version 1.8.0 (a.k.a. Java 8) or later, by typing:

> java -version

Note, that The GNU java a.k.a. gij (default on some older RedHat and Fedora systems) is painfully slow and unreliable, and will not always run CRUSH correctly. If you need Java, you can download the latest JRE from

https://java.com

Download CRUSH

You can get the latest version of CRUSH from:

http://www.sigmyne.com/crush

Linux users can grab one of the distribution packages to install CRUSH via a package manager. Packages are provided for both RPM-based distributions (like Fedora, Redhat, CentOS, or SUSE), and Debian-based distributions (e.g. Ubuntu, Mint). Both will install CRUSH in /usr/share/crush. Note, if you use one of these packages you will need root privileges (e.g. via sudo) on the machine. Unprivileged users should install using the tarball.

For all others, CRUSH is distributed as a gzipped tarball (or as a zip archive). Simply unpack it in the directory, where you want the crush to reside.

A. Installation from tarball (POSIX/UNIX, incl. Mac OS X)

Unpack the tarball in the desired location (e.g. under ~/astrotools/):

> cd ~/astrotools
> tar xzf crush-2.xx-x.tar.gz

Verify that CRUSH works:

> cd crush
> ./crush

You should see a brief information screen on how to use CRUSH.

B. Installation via Linux packages

Use the package manager on your system to install. E.g., on RPM-based systems (Fedora, RHEL, CentOS, SUSE) you may type:

> sudo yum localinstall --nogpg crush-<version>-noarch.rpm

On Debian based systems (e.g. Ubuntu, Mint) the same is achieved by:

> sudo apt-get install crush-<version>.deb

Or, you may simply double click the package icon from a file browser.

C. Windows Installation

Please refer to README.windows, which may be found in the Install\windows\ subdirectory of the distribution, or online under the 'Documents' tab.

1.2. (optional) Java Configuration

CRUSH ships with a default Java configuration. On Windows and the most common POSIX platforms (Linux, Mac OS X, BSD, and Solaris), it will automatically attempt to set an optimal configuration. On other platforms, it comes with fail-safe default values (default java, 32-bit mode and 1GB of RAM use).

You can override the defaults by placing your settings in arbitrary files under /etc/crush2/startup or ~/.crush2/startup (for the equivalent configuration under Windows, please refer to README.windows)

(Any settings in the user's home under .crush2/startup will overrride the system-wide values in /etc/crush2/startup or C:\Program Data\startup. If multiple config files exist in the same location, these will be parsed in non-specific order).

E.g., placing the following lines in ~/.crush2/startup/java.conf overrides all available runtime configuration settings:

JAVA="/usr/java/latest/bin/java"
USEMB="4000"
JVM="-server"
EXTRAOPTS="-Djava.awt.headless=true"

upon startup CRUSH will find and apply these settings, so it will use /usr/java/latest/bin/java to run CRUSH with 4GB of RAM, using the HotSpot server VM, and in headless mode (without display, mouse or keyboard).

Below is a guide to the variables that you can override to set your own Java runtime configuration:

JAVA       Set to the location of the Java executable you want to use. 
           E.g. "java" to use the default Java, or
           `/usr/java/latest/bin/java` to use the latest from Oracle or 
           OpenJDK.

USEMB      Set to the maximum amount of RAM (in MB) available to 
           CRUSH. E.g. "4000" for 4GB. On a 32-bit OS, or with a 32-bit
           Java installation no more than 2GB of RAM may be accesssed. In
           practice, the maximum 32-bit OS/Java values range from around 
           "1000" (32-bit Windows / Java) to "1900" (32-bit Linux / Java).  

JVM        Usually set to `-server` for Oracle or OpenJDK. If using IBM's 
           Java, set it to "" (empty string). On ARM platforms, you 
           probably get better performance using `-jamvm` or `-avian`. To 
           see what VM options are available on your platform, run 
           `java -help`. The VM options are listed near the top of the 
           resulting help screen. 

EXTRAOPTS  Any other non-standard options you may want to pass to the Java 
           VM should go here. 

You can also specify environment variables, and add shell scripts (bash), since these configuation files are in fact sourced as bash scripts before launching Java / CRUSH. For example you can add:

 CRUSH_NO_UPDATE_CHECK="1"
 CRUSH_NO_VM_CHECK="1" 
	  
 echo "Will try to parse my own configuration now... "

 if [ -f \~/mycrushconfig.sh ] ; then
   echo -n "OK"
   source \~/mycrushconfig.sh
 else
   echo -n "Not found"
 fi

The above will disable update checking (not recommended!) and VM checking (also not recommended!) and will source the contents of ~/mycrushconfig.sh if and when such a file exists.

1.3. CRUSH Pipeline Configuration

The preferred way of creating user-specific configurations for CRUSH is to place your personalized configuration files inside a .crush2/ configuration directory within your home. This way, your personalized configurations will survive future updates to CRUSH.

You can create/edit your default configuration by editing default.cfg (either in the installation folder or ~/.crush2, or in the appropriate instrument subdirectory within it. As an example a user configuration for LABOCA data can be placed into ~/.crush2/laboca/default.cfg, with the a content:

datapath \~/data
outpath \~/images
project T-79.F-0002-2006

This tells crush that when reducing LABOCA data it should look for raw data files in ~/data, write all output to ~/images, and specifies the default project to be T-79.F-0002-2006

The tilde character ~ is used as a shorthand for the home directory (similarly to UNIX shell syntax). In your configuration you can also refer to environment variables or other settings (see more about it further below).

Of course, you can create any number of configurations, name them as you like place them where you like (practical if you have many data locations, projects etc.). You can easily invoke configuration files as needed via

> crush [...] -config=<path-to-myconfig> [...]

1.4. The Tools of the CRUSH Suite

CRUSH provides a collection of other useful tools. Here's a short description of what is there and what they are used for. Each tool, when run without options (or with the -help option) will provide you a list of available options on the console.

crush       The principal reduction tool.

imagetool   A tool for manipulating images. Can also deal with 
            images produced by BoA (and to some degree other 
            images also).

show        A simple display program for the FITS files, with 
            various useful functions for simple analysis and image
            manipulation capabilities.

coadd       Add FITS images together. Use as a last resort tool as
            it is always better to reduced scans together.

difftool    Allows to look at the difference between two images.
	(used to be named 'difference')

histogram   Write a histogram of the pixel distribution of an 
            image plane. (e.g. `flux`, `rms`, `s2n`, `weight`, or 
            `time`).

detect      A source extraction tool for maps. You should make
            sure that the noise is close enuogh to Gaussian (e.g. 
            with `histogram`) before relying on this. 

esorename	 Rename ESO archive files to their original names.

1.5. Quick Start

To reduce data, simply specify the instrument (E.g. sharc2, laboca...) and the scan numbers/ids (or names of the scan files or directories) to crush. (You may have to add ./ before crush if the current directory . is not in your path.) E.g.,

> crush laboca 10059

will reduce LABOCA scan 10059 with the default reduction parameters. (Try crush -help to see a full list of the supported instrument names.)

After the instrument name, you can specify additional options. Some of these apply to the reduction as a whole while others will affect the scan processing for those scan numbers that are listed after the option flag.

If you are new to CRUSH (or used version 1.xx before), you should be able to start working with it by the time you get to the end of this page. Nevertheless, I recommend that you read on through the entire Sections 1--2 (Getting Started & Basic Configuration), and you will become a truly well-versed user. :-)

Once you get the hang of it, and feel like you need more tweaking ability, feel free to read on yet further to see what other fine tuning capabilities exist...

Here are some quick tips:

  • Reduce as many scans together as you can. E.g.

     > crush laboca 10657-10663 10781 11233-11235 [...]
    
  • You can specify opacities, pointings, scaling etc, for each of the set of scans listed (See section for Scan-specific options). E.g.,

     > crush laboca -tau=0.32 10657-10663 10781 -tau=0.18 11233 [...]
    

    will use a zenith tau value of 0.32 for 10657-10663 and 10781, and 0.18 for the last scan (11233).

  • If you suspect that you are missing extended emission (see section on the Recovery of Extended Structures), then you can specify the 'extended' option, which will better preserve large scale structures albeit at the price of more noise. E.g.

     > crush laboca -extended 10657-10663
    
  • If your source is faint (meaning S/N < 10 in a single scan), then you may try using the faint option. E.g.

     > crush laboca -faint 10657-10663
    

    or,

     > crush laboca -faint -extended 10657-10663
    

    to also try preserve extended structures (see above).

  • For finer control of how much large scale structures are retained you can use the sourcesize option, providing a typical source scale (in arcsecs) that you'd like to see preserved. Then CRUSH will optimize its settings to do the best it can to get clean maps while keeping structures up to the specified scales more or less intact. E.g.

     > crush [...] -sourcesize=45.0 10657-10663
    

    Will optimize reduction for <~ 45 arcsec sources.

  • Finally, if your sources are not only faint, but also point-like, you should try the deep option. This will use the most aggressive filtering to yield the cleanest looking maps possible. E.g.,

     > crush laboca -deep 10657-10663
    

    With just these few tips you should be able to achieve a decent job in getting the results you seek. CRUSH can also help you reduce pointing/calibration, skydip, and pixelmap scans. E.g.:

  • To reduce a laboca pointing/calibration scan 11564:

     > crush laboca -point 11564
    

    At the end of the reduction CRUSH will suggest pointing corrections and provide detailed information on the source (such as peak and integrated fluxes, FWHM and elongation). Once you determine the appropriate pointing corrections for your science scans, you can apply these via the pointing=x,y option. E.g.:

     > crush laboca -pointing=3.2,-0.6 12069
    
  • You can also reduce skydips, for determining in-band atmospheric opacities. E.g.:

     > crush hawc+ -skydip 26648
    
  • Finally, you can derive pixel position information from appropriate scans, which are designed to ensure the source is moved over every detector, in a fully sampled manner. To reduce such beam-maps:

     > crush gismo -pixelmap 5707
    

    CRUSH will write rcp files as output, containing pixel positions source and sky gains in a standard 5 column APEX RCP format. You can feed the pixel position information to reduce other scans via the rcp option.

There are a lot more fine-tuning possibilities for the more adventurous. If interested, you can find documentation further below, as well as in the README files inside the instrument sub-directories. For a complete list of crush options, please refer to the GLOSSARY.

1.6. A Brief Description of What CRUSH Does and How...

CRUSH is a pipeline reduction that is principally meant to remove correlated signals (correlated noise) in the time-streams to arrive at clean & independent bolometer signals, which are then used to make a source model (usually an image).

As such it is not an interactive reduction software (e.g. as opposed to e.g. BoA). The term 'scripting' in CRUSH mainly means defining configuration options (in the command line or through configuration files) which are parsed in the order they are read.

During the reduction CRUSH aims to arrive at a consistent set of solutions for various correlated signals, the corresponding gains and the pixel weights, as well as tries to identify and flag problematic data.

This means a series of reduction steps, which are iterated a few times until the required self-consistent solutions are arrived at.

To learn more about the details please refer to Kovacs, A., "CRUSH: fast and scalable data reduction for imaging arrays," Proc. SPIE 7020, 45, (2008). If that does not satisfy your curiousity, then you can find yet more explanation in Kovacs, A., PhD thesis, Caltech (2006).

1.7. Command-line Options and Scripting Support

Configuration of CRUSH is available either through command line options or via scripting. You have seen scripting already in the form of 'default.cfg', which stores the default configuration values (Sec. 1.1).

The syntax is designed so that it accomodates both scripting and command-line (bash) use alike. Thus, white spaces are optional and curved brackets are avoided (unless these are placed in quotes).

For a complete guide on using the configuration engine, please read README.syntax. Below we provide only a very basic overview, and describe only what is specific to CRUSH, beyond the cofiguration engine in general.

Basic Rules

When defining settings, keys can be separated from their values either by =, : or empty spaces (or even a combination of these).

Command line options start with a dash - in front. Thus, what may look like:

key1 value1
key2 value2, value3

in a configuration script, will end up as

> crush [...] -key1=value1 -key2=value2,value3 [...] 

on the command line. Otherwise, they two ways of configuring are generally equivalent to one-another. One exception to this rule is reading scans, which is done via the read key in a script, but on command line you can simply list the scan number (or ranges or lists or names). I.e.,

[...]
read 10056                # in script

> crush [...] 10056       # on the command line.

In the next section you'll find a description of the scripting keywords. Now that you know how to use them also as command line options, you can choose scripting or command-line, or mix-and-match them to your liking and convenience...

Key/value pairs are parsed in the order they are specified. Thus, each specification may override previously defined values.

Lines that start with # designate comments that are ignored by the parser.

Dynamic Conditionals

Some conditions are interpreted dynamically. For example CRUSH, can activate settings based on observation date (or MJD) or serial number, on a scan-by-scan basis, such as:

mjd.[54555-54869] rcp {$CRUSH}/laboca/laboca-3.rcp

which loads the pixel positions (RCP) laboca-3.rcp inside the laboca subfolder in the crush installation for scans taken in the MJD range 54555 to 54869. Similarly,

date.[2007.02.28-2009.10.13] instrument.gain -1250.0

or

serial.[14203-15652] flag 112-154,173

are examples of setting activated based on dates or scan serial numbers.

You may also specify conditional settings based on the source names under the object branch. E.g.:

object.[Jupiter] bright

will invoke the bright configuration for Jupiter. The check for the source name is case insensitive, and matches all source names that begin with the specified sequence of characters. Thus, the SHARC-2 configuration line:

object.[PNT_] point

will perform a pointing fit at the end of the reduction for all sources whose catalog names begin with PNT_ (e.g. PNT_3C345).

These examples also demonstrates that conditionals can be branched just like options. (In the above cases, the conditions effectively reside in the mjd, date or serial branches). Other commonly used conditionals of this type are iteration based settings:

> crush [...] -iteration.[last-1]whiten=2.0 [...]

will activate the noise whitening in the iteration before the last one.

Startup Configuration

At launch, crush will parse config/default.cfg to establish a default configuration set. All configurations files are first parsed from the files in the config/ directory inside crush, then additional options or overriders are read also from the appropriate instrument subdirectories, if exist.

After that, crush will check if these configuration files also exists under the inside the ~/.crush2 directory of the user. If so, they will be used to override again. See more on this in the next section under the config option. In this way, the ~/.crush directory can be used as a convenient way to create user specific setting and/or overrides to global defaults.

1.8. Making sense of the console output

Don't ignore the console output. It contains a lot of information, which may be useful for diagnosing your reduction, and troubleshooting if things don't go as expected.

In the beginning of the console output, you will see information from each scan as it is being read, followed by output from the preprocessing steps (such as downsampling, scan statistics, velocity clipping, opacity used, pointing corrections applied etc.). This part is verbose enough that it should be eszsentially self-explanatory.

Once all the scans are loaded, and the source model (usually a map) has been created to accomidate the dataset, CRUSH will enter the iterated pipeline reduction. It may look cryptic at first, as it captures a lot of information in very little console space. Below is a short guide to figure out what it all means. Understanding it will help you diagnose and troubleshoot all sorts of issues that might affect the reduction of your dataset. So, don't be shy, dive in!

An example line from the console output

Let's begin with a concrete example. Here is a section from the console output from the reduction of LABOCA scan 11564 (from 2007):

$ Round 4: 
$
$  [11564] D(128) C245 W(8.44E-2)245 dN(0)245 B245 c245 Map(8.44E-2) 
$  [Source] {check} (smooth) (clip:4.0) (blank:10.0) (sync) 

Each phrase (separated by spaces around it) represents a short summary of a pipeline step. The leading characters identify the step, often followed by some quantitative capure (or figure-of-merit) summarizing what the step accomplished.

Here is how the above section is interpreted:

In the 4th iteration, the following steps are performed on scan 11564:

D(128)        1/f filtering (on 128 frame timescale)

C245          removing the correlated noise from the array with 
              245 pixels surviving the gain range criterion.

W(8.44E0-2)   Deriving pixel weights with the `rms` method. The
              average sensitivity of the array is estimated to be
              84.4 mJy sqrt(s).

dN(0)245      Despiking via the `neighbours` method, with 0 frames
              flagged as spiky, and 245 pixels surviving the
              flagging of pixels with too many spikes.

B245          Decorrelating on amplifier boxes, with 245 pixels 
              having acceptable gains to the correlated signals.

c245          Decorrelating on electronic cables, with 245 pixels
              showing acceptable gain response to these signals.

Map(8.44E-2)  Mapping scan, with estimated point source NEFD of
              84.4 mJy sqry(s).

Then, at the end of the iteration a composite source model is created. This is further processed as:

{check}       Remove invalid data from the scan maps (before adding
              to composite).

(smooth)      smooth composite by the desired amount.

(clip:4.0)    Discard map points below an S/N of 4.0 from composite.

(blank:10.0)  Do not use bolometer data in other steps, which are
              collected over the map points with S/N > 10, thus
              reducing the bias that bright sources can cause in the
              reduction steps.

(sync)        Synchronize the composite model back to the time-stream
              data.

Once the reduction is done, various files, including the source map, are written. This is appropriately echoed on the console output.

Console Output Reference Guide

Below is a reference guide for interpreting the console output for your particular reduction:

Quantities inside the brackets:
[nnnnn|m]:  processing scan nnnnn, subscan m from the list

(#.##E#)    The effective point-source NEFD (usually after 
            weighting or at the mapping step)
            shown in the effective mapping units times sqrt(s).

[]          bracketed models are solved via median estimators

(#)         Indicates the time resolution of the given step as
            the number of downsampled frames, when this is not
            the default full resolution.
Reduction-step Shorthands (alphabetically)
a          Decorrelating amplifier boards (e.g. LABOCA,GISMO).

am         Remove correlated telescope acceleration (magnitude)

ax         Remove correlated telescope x acceleration

ay         Remove correlated telescope y acceleration

B          Decorrelating electronic boxes (e.g. LABOCA).

C          Solving for correlated noise and pixel gains
           The time resolution (in frames) is shown in brackets
           if not full resolution, followed by the number of 
           unflagged pixels if gains are solved and a gain range
           is defined for flagging.

c          Decorrelating electronic cables (e.g. LABOCA)	
           or geometric columns (e.g. GISMO)

cx         Remove correlated chopper x position

cy         Remove correlated chopper y position

D          Solving for pixel drifts (1/f filtering).

dA(#)      despiking absolute deviations.	
           In brackets it shows the percentage # of frames
           flagged in the data as spiky by any method.

dG(#)      like above but proceeds gradually.

dN(#)      despiking using neighbouring samples in the timeseries.

dM(#)      despiking at multiple resolutions (up to the 1/f
           filter timescale).

E          Remove correlated SAE error signal (GISMO)	

G          Estimating atmospheric gradients accross the FOV.

J          De-jumping frames (e.g. for GISMO). Followed by two
           colon (:) separated numbers: the first, the number of
           de-jumped blocks that were re-levelled and kept; and
           the number of blocks flagged.

m          Decorrelating on multiplexers (e.g. GISMO, SCUBA-2)

Mf         Filtering principal telescope motion.

O          Solving for pixel offsets

p          Decorrelating on (virtual) amplifier pins (e.g. GISMO)

Q          Decorrelating wafers (e.g. ASZCA).

r          Decorrelating on geometric rows (e.g. SHARC-2, GISMO)

t          Solving for the twisting of band cables (e.g. LABOCA).

tx         Remove correlated telescope x position

ty         Remove correlated telescope y position

tW         Estimating time weights.

W          Estimating pixel weights.

w          Estimating pixel weights using the 'differential'
           method.

wh(x.xx)  Noise whitening. The average source throughput factor
          from the whitening is shown.	
Source model-specific:
Map         Calculating source map from scan. The effective point
            source NEV/NEFD of the instrument is shown in the 
            brackets (e.g. as Jy sqrt(s)).

[C1\~#.##]  Filtering corrections are applied directly and are
            #.## on average.

[C2=#.##]   Faint structures are corrected for filtering after
            the mapping via an average correction factor #.##.

[Source]    Solving for source map.

{check}     Discarding invalid map data.

{despike}   despiking scan maps before adding to composite.

{filter}    Spatial large-scale structure (LSS) filtering of the 
            scan maps.

(level)     Zero levelling to map median.

(smooth)    Smoothing map.

(filter)    Spatial filtering of the composite map.

(noiseclip) Clip out the excessively noisy parts of the map.

(filter)    Filtering large scale structures (i.e. sky-noise).

(exposureclip)  Clipping under-exposed map points.

(blank)    Blank excessively bright points from noise models.

(sync)     Removing the source model from the time-stream.

1.9. Examples

Reduce scans 12065-12069 and 12072 with zenith tau of 0.18:

> crush laboca -tau=0.18 12065-12069 12072

Reduce scans 10562 and 10645 together, with the first scan observed at a zenith tau of 0.21, and the second at tau of 0.35 with.

> crush laboca -tau=0.21 10562 -tau=0.35 10645

Say you realize that the pointing was off by -5.4" in AZ, and 2.4" in EL for the second scan. Then:

> crush laboca -tau=0.21 10562 -tau=0.35 -pointing=-5.4,2.4 10645

Say scan 10049 is a scan of a bright source (e.g. Saturn) and the default reduction ends up clipping much of it. Then,

> crush laboca -bright 10049

If the source still gets nipped from the resulting map, you can try disabling pixel weigting altogether (this is the likely culprit) by:

> crush laboca -bright -blacklist=weighting 10049

Perhaps you suspect there is missing extended emission (large-scale structure) in scan 10550. You can try to recover more by:

> crush laboca -extended 10550.

Try reduce some faint source (scan 10057):

> crush laboca -faint 10057

Maybe your faint source has extended structure that you want to try recover better, and willing to pay some noise penalty for it. Then try:

> crush laboca -faint -extended 10057.

You can also fine-tune, what is the largest source-scale (more-or-less) that you are interested in. Then the reduction will adjust its parameters accordingly. Say you expect your source to be a blob around 1' in diameter, then you can try:

> crush laboca -faint -sourcesize=60.0 10057.

You can also play with the blanking clipping methods above, if there is annoying negative dips remaining around your brighter peaks. Note, that you porbably want to stick with -extended, or else such dips may be the result of the aggressive filtering settings applied to your specific source size (via sourcesize).

As mentioned before, command-line options and scripting are equivalent ways of configuring the reduction. Thus, the running the script test.cfg:

faint
extended
tau 0.18
pointing -3.2,4.8
read 12065-12069
pointing 2.3,-0.5
read 12072
blank 10.0
clip 3.0
iteration.[last-1] forget clip

via

> crush laboca -config=test.cfg	

is equivalent to the command line:

> crush laboca -faint -extended -tau=0.18 \
  	-pointing=-3.2,4.8 12065-12069 \
	-pointing=2.3,-0.5 12072 \ 
	-blank=10.0 -clip=3.0 -iteration.[last-1]forget:clip  

Finally, suppose you want to reduce a dataset on a very faint point source, such as a distance galaxy, or a faint core, with a set of scans:

> crush laboca -deep 23165-23169 

And, perhaps you want to comprate the result to a random jackknife, which you can obtain with:

> crush labova -deep -jackknife 23165-23169

2. Advanced Topics

2.1. Pointing Corrections

Reducing the data with the correct pointing can be crucial (esp. when attempting to detect/calibrate faint point sources). At times the pointing may be badly determined at the time of observation. Luckily, getting the exact pointing offset wrong at the time of the observation has no serious consequences as long as the source still falls on the array, and as long as the exact pointing can be determined later. If you, at the time of the data reduction, have a better guess of what the pointing was at the time the data was taken (e.g. by fitting a model to the night's pointing data), you can specify the pointing offsets that you believe were more characteristic to the data, by using the -pointing option before listing the scans that should be reduced with the given corrections. E.g.,

> crush [...] -pointing=12.5,-7.3 <scans>

Will instruct minicrush that the true pointing was at dAZ=12.5 and dEL=-7.3 arcsec each (i.e. it works just like pcorr on APECS).

Some instruments, like SHARC-2, may allow specifying the aggregated pointing offsets (e.g. fazo and fzao) instead of the differential corrections supplied by pointing.

Obtaining Corrections

Good practice demands that you regularly observe pointing/calibration sources near your science targets, from which you can derive appropriate corrections. CRUSH provides the means for a analyzing pointing/calibration data effectively, using the point option:

> crush [...] -point [...]

At the end of the reduction, CRUSH will analyze the map, and suggest appropriate pointing corrections (to use with -pointing, or other instrument-specific options), and provide calibration information as well as some basic measures of source extent and shape.

After Reduction (a poor alternative)

You can also make pointing changes after the reduction (alas, now in RA/DEC). You can read off the apparent source position from each separately reduced scan (e.g. by using show and placing the cursor above the apparent source center/peak). Then you can use imagetool to adjust the pointing. E.g.:

> imagetool [...] -origin=3.0,-4.5 ... 

The above moves the map origin by 3" in RA and -4.5" in DEC.

Then, other crush tools (like coadd, imagetool etc.) will use these images with the proper alignment. Clearly, this method of adjusting the pointing is only practical if your source is clearly detected in the map.

2.2. Pixelization and Smoothing

There seem to be many misconceptions about the correct choice of pixelization and about the (mal)effects of smoothing. This section aims to offer a balanced account on choosing an appropriate mapping grid and on the pros and cons of applying smoothing to your maps.

Pixelization (i.e. choosing the 'right' grid)

Misconception: You should always Nyquist sample your map, with 2 pixels per beam FWHM, to ensure ideal representation.

There is more than one thing wrong with the above idea. First, there is no 'Nyquist' sampling of Gaussian beams. The Nyquist sampling theorem applies only to data, which has a natural cutoff frequency (the Nyquist frequency), above which there is no information present. Then, and only then, will sampled data (at a frequency strictly larger[!] than twice the Nyquist cutoff) preserve all information about the signals.

Two pixels per beam is almost certainly too few for the case of Gaussian beams. Gaussian beams have no true frequency cutoff -- the signal spreads across the spectrum, with information present at all frequencies, even if it is increasingly tapered at the higher frequencies. By choosing a sampling (i.e. pixel size in your map) the information above your sampling rate will be aliased into the sampled band, corrupting your pixelized data.

Thus, you can choose a practical cutoff frequency (i.e. pixelization) based on the level of information loss and corruption you are willing to tolerate. At 2.5 pixels pear FWHM, you retain ~95% of the underlying information, and corrupt it by the remaining 5%. With more pixels per beam, you get more accurate maps. (The 2-pixels per beam, that many understand to be the Nyquist sampled, is certainly too few by most standards of accuracy!)

Thus, the CRUSH default is to use around 5-pixels per FWHM, representing a compromise between completeness (~99%) with minimal corruption (<1%) and a senseless increase of map pixels (i.e. number of model parameters).

Smoothing

Misconception: You should not smooth your maps, because it does something unnatural to your raw image.

The first problem with this view is that there is no natural or raw image to start with, because there is no natural pixelization (see above). Secondly, by choosing a map pixel size, you effectively apply smoothing -- only that your smoothing kernel is a square pixel, with an abrupt edge, rather than a controlled (and directionless) taper of choice.

(To convice yourself that a map pixels apply smoothing, consider the mapping process: each pixel will represent an average of the emission from the area it covers. Thus, the pixel values are essentially samples of a moving integral under the pixel shape -- i.e. samples of the emission convolved [smoothed] by the pixel shape.)

However, smoothing (including coarse pixelization!) does have a real downside, in that it degrades the effective resolution of the image. Consider smoothing by Gaussian kernels. Then, the image resolution (imageFWHM) increases with the smoothing width (smoothFWHM) as

imageFWHM^2 = beamFWHM^2 + smoothFWHM^2

from the instrument resolution (beamFWHM). However, you can smooth a fair bit before the degradation of resolution becomes a problem or even really noticeable. If you use sub-beam smoothing with smoothFWHM < beamFWHM, then the relative widening is roughly

dW / W0 ~ 0.5 * (smoothFWHM / beamFWHM)^2

Thus, smoothing by half a beam, degrades resolution by ~12% only... At the same time, smoothing can bring many benefits:

  • Improves image appearance and S/N by rejecting pixelization noise. It is better to use a finer grid and smooth the image by the desired amount, than to pixelize coarsely -- not only for visual appearance but also for the preservation of information.

  • Smoothing can help avoid pathological solutions during the iterations, by linking nearby pixel values together.

  • Beam-smoothing is the optimal (Wiener) filtering for determining point source fluxes in a white noise background. I.e., smoothing by the beam produces the highest signal-to-noise measurement of point sources.

  • Beam-smoothing is mathematically equivalent to fitting fixed-width beams at every pixel position. The beam-smoothed flux value in each pixel represents the amplitude of a beam fitted at that position using chi-squared minimization. Thus, beam smoothed images are the bread-and- butter of point source extraction (e.g. the detect tool of CRUSH).

Given the pros and the con for smoothing, the different reduction modes (default, faint or deep) of CRUSH make different compromises. The default reduction aims to provide maps with maximal resolution (i.e. no smoothing), although a some smoothing is used during the iterations to aid convergence to a robust solution. faint mode reductions smooth by 2/3 beams (resulting in ~22% degradation of resolution), to provide better signal-to-noise at a moderate loss of spatial resolution. Finally, deep reductions yield beam smoothed images, which are ideal for point source extraction, even if some spatial details are sacrificed.

You can change/override these default settings. The smoothing during reduction is controlled by the smooth setting, which can take both a Gaussian FWHM (in arcsec) as its arguments as well as the preset values minimal, halfbeam, 2/3beam and beam. The smoothing of the final map can be controlled by 'final:smooth' (the final stands as a shorthand conditional for the last iteration). Thus,

> crush [...] -smooth=halfbeam -final:smooth=minimal [...]

smoothes by half a beam during the iterations, and only slightly for the output image, while

> crush [...] -forget=smooth -final:forget=smooth [...]

disables smoothing both during reduction and at the end. The table below summarizes the effect of the preset smoothing values, indicating both the degradation of resolution (relative to the telescope beam) and the relative peak S/N on point sources:

Table 1. Smoothing properties

smooth widening rel. S/N
minimal 6% 0.33
halfbeam 12% 0.50
2/3beam 22% 0.67
beam 41% 1.00

2.3. Recovery of Extended Emission

As a general rule, ground based instruments (in the mm and submm) are only sensitive to structures smaller than the Field-of-View (FoV) of the instrument. All scales larger than the FoV will be strongly degenerate with the bright correlated atmosphere (a.k.a. sky noise), and will be very difficult, if not outright impossible, to measure accurately. In a sense, this is analogous to the limitations of interferometric imaging, or the diffraction-limited resolution of finite apertures.

However, there is a some room for pushing this limit, just like it is possible to obtain some limited amount of super-resolution (beyond the diffraction limit) using deconvolution. The limits to recovery of scales over the FoV is similar to those of obtaining super-resolution beyond the diffraction limit. Both these methods

  1. yield imperfect answers, because the relevant spatial scales are poorly measured in the first place.

  2. are only feasible for high signal-to-noise structures.

  3. can, at best, go beyond by a factor of a few beyond the fundamental limit.

You can usually choose to recover more extended emission if you are willing to put up with more noise on those larger scales. This trade-off works backwards too -- i.e., you can get cleaner maps if you are willing to filter them more.

As a general principle, structures that are bright (>> 5-sigma) can be fully recovered to up to a few times the field-of-view (FOV) of the bolometer array. However, the fainter the structure, the more it will be affected by filtering.

Generally, the fainter the reduction mode, the more filtering of faint structures results, and the more limited the possibility of recovering extended structures becomes. The table below is a rough guide of what maximum scales you may expect for such faint feaures, and also, how noise is expected to increase on the large scales as these are added in extended mode:

Table 2. Maximum faint structure scales for S/N <~ 5

option Max. sensitive scale Noise power scaling
default / bright FOV/2 ~L^2
+ extended FOV ~L^2
faint / deep 2*sourceSize or 2*beam ~L
+ extended FOV/2 ~L

Iterating longer, will generally help recover more of the not-too-faint large-scale emission (along with the large-scale noise!). Thus,

> crush -extended -rounds=50 [...] 

will generally recover more extended emission than just the default

> crush -extended [...]

(which iterates 10 times). In general, you may expect to recover significant (>5 sigma) emission up to scales L as:

L ~= FoV + sqrt(N v T)

in terms the number of iteration N, limiting Field-of-View (FoV), scanning velocity v and correlated noise stability time-scale T. Unfortunately, the correlated noise stability of most ground-based instruments is on the order of a second or less due to a highly variable atmosphere. At the same time, the noise rms on the large scales will increase assymptotically as:

rms ~ sqrt(N)

for large numbers of iterations.

2.4. Filtering corrections, transfer functions, etc.

Most reduction steps (decorrelation, 1/f drift removal, spectral filtering) will impact different spatial frequencies of the source structure differently. In general, the low spatial frequency components are most affected (suppressed). E.g. decorrelating the full array will reject scales larger than the field-of-view, while 1/f filtering will result in 1/f spatial filtering of the map due to the scanning motion, which more or less directly maps temporal frequencies into spatial frequencies.

The approach of CRUSH is to apply appropriate point-source corrections (rescaling) such that point sources in the map yield the same fluxes no matter how (i.e. with what options exactly) the source was reduced. While the corrections will be readily sufficient for a large fraction of the science cases in this simple form, there are two notable exceptions to this rule: (i) extended emission, and (ii) small, fast maps reduced in deep mode (when the source is scanned over the same detector more than once over the stability timescale).

The better approach for these cases is to measure a transfer function, or otherwise check the reduction of a similar simulated source.

Transfer functions and simulated sources

The sources option provides a means for inserting test sources into CRUSH maps, while one of the jackknife options can be used to remove any real emission beforehand but retaining the signal and noise structure of the data otherwise (which is important in order to get a representative measure of the transfer function). E.g.

> crush [...] -jackknife.alternate -sources=test.mask ...

will apply an alternating jackknife to the input scans, and insert sources specified in the mask file test.mask (See example.mask on the format of mask files, and the GLOSSARY for more on jackknifing options).

To measure transfer functions (i.e. complete spatial characterization of the point-source response) you would want to insert a single beam-sized point source. Alternatively, you can insert one or more Gaussian-shaped source(s) with a range of FWHMs to create a simulated source structure that resembles the structure you expect your source(s) to have.

Make sure your test source is bright enough to see with high S/N, but not too bright to trigger unintended flagging, or despiking. In general, a S/N between 100 and 1000 should be fine for default reductions, and 100 to 300 for faint or deep modes.

Additionally, in faint or deep modes, you may want to disable some features which may affect your relatively bright test sources differently than your much fainter real science target(s). Thus, the following are recommended for reducing faint and deep test sources:

> crush [...] -blacklist=clip,blank,source.filter.blank

Note also that the spatial filtering (transfer function) will be varying with location on the map (since the scanning speed and directions will themselves be non-uniform over the map). Therefore, it is strongly recommended that test sources are inserted near the same locations as the real sources in the field.

2.5. Image processing post-reduction

CRUSH also provides imagetool for post-processing images after reduction. The typical usage of imagetool is:

> imagetool -name=<output.fits> [options] <input.fits>

which processes <input.fits> with the given options, and writes the result to <output.fits>.

With imagetool, you can apply the desired level of smoothing (-smooth), or spatial filtering (-extFilter), specify circular regions to be skipped by the filter (-mask=<filename>). You can adjust the clipping of noisy map edges by relative exposure (-minExp) or by relative noise (-maxNoise). You can also crop a rectangular region (-crop=dx1,dy1,dx2,dy2). There are many more image operations. See the imagetool manual (in your UNIX shell, or online) or simply run imagetool without an argument:

> imagetool

One of the useful options allows to toggle between the noise estimate from the detector time-streams (-noise=data) and from the image itself (-noise=image) using a robust estimator. For example, after spatial filtering, you probably want to re-estimate the map noise:

> imagetool [...] -extFilter=45.0 -noise=image <input.fits>

The built-in image display tool show also takes all processing options of imagetool, but rather than writing the result, it will display it in an interactive window. See the manual page of show (either inside your UNIX shell, or online), or run show without an argument.

2.6. Reducing Very Large Datasets

Coming soon...


3. Advanced Configuration

In this section, you can find information on the most useful configuration options. A complete list of available settings can be found in the text file GLOSSARY, located in the CRUSH installation directory.

Configuration options here are listed as scripting keys i.e., without a preceding dash. However, you can use the same options in the command line by adding the dash. (Also, = can be replaced by space(s) in scripting...)

3.1. Commands

There are a handful of keywords that are treated as commands by crush. (The difference being that commands are interpreted and acted on right away, whereas typical configuration keys are stored settings that are interpreted later as necessary.) The commands are:

config=<filename>	Load the configuration file filename. 
			The file is looked for in the locations in the
			following order:

				1. ./
				
				2. ./<instrument>/

				3. ~/.crush2/

				4. ~/.crush2/<instrument>/

			Whenever a matching file is found its contents
			are parsed. Because of the ordering, it is 
			convenient to create overriding configurations.
			Thus instrument specific settings can be used 
			to override default settings, and user specific
			settings placed in '~/.crush2' can override
			shipped defaults. Whenever a configuration is
			parsed, there is a note of it on the console
			output so that one always knows which files 
			were read and in what order.
			E.g. when using
			
			   > crush laboca -faint 12066

			the following configuration files will be
			loaded in order (provided they exist):

			   <crush>/config/default.cfg
			   <crush>/config/laboca/default.cfg
			   ~/.crush2/default.cfg
			   ~/.crush2/laboca/default.cfg
			  
			   <crush>/config/faint.cfg
			   <crush>/config/laboca/faint.cfg
			   ~/.crush2/faint.cfg
			   ~/.crush2/laboca/faint.cfg
			   
			Each successively loaded file may override
			the options set before it.
			When a matching configuration file is not found
			in any of the standard locations (above), CRUSH
			will make one last attempt to interpret the
			argument as a regular pathname. This allows
			users to store and invoke configuration files
			anywhere on the filesystem.
			
forget=<option>...	Forget the priorly set values for <option>
			as if it were never defined. E.g. 
	     		
				forget=outpath

			will unset the 'outpath' option.
			You can specify more than one options as a
			comma-separated list. E.g.

				forget=outpath,project
			
			With unset both the 'outpath' and 'project' 
			options.
			Additionally, the arguments 'conditions' and
			'blacklist' can be used to clear the 
			conditional or blacklisted settings 
			respectively	

recall=<option>		Undoes 'forget', and reinstates the <option>
			to its old value.	

remove=<option>		Similar to 'forget', but removes the entire
			branch. Thus '-remove=despike' unsets:

				despike
				despike.level
				despike.method
				despike.flagfraction
				...

replace=<option>	Undoes the 'remove' option, reinstating the
			<option> tree to its prior state.

blacklist=<option>...	Similar to 'forget', except it will not
			set options even if they are then specified
			at a later time. This is useful for altogether
			removing settings from the configuration.

whitelist=<option>...	Remove <option> from the blacklist, allowing
			it to be set again if desired. Whitelisting
			an option will not reinstate it to its prior
			value. After whitelisting, you must explicitly
			set it again, or 'recall' or 'replace' it
			to its prior state.

poll			Whenever unsure what options are set at any
poll=<option>		given stage, you can poll the settings.
			Without an additional argument it will list
			all options to the standard output. When
			an argument is specified it will list
			all configuration settings that start with
			the specified string. E.g.

				> crush [...] -poll=iter

			will list all iteration based options that 
			are set including all the [...] options set
			prior to '-poll' in the command line.


list.divisions		List all pixels divisions, which can be
			decorrelated for the instrument.

list.response		List all response modalities for the
			instrument (to known signals, such as
			telescope movement, or temperature data).

3.2. Pipeline Configuration

a. Source types.

The default reduction (see default.cfg) is optimized for mapping compact (up to the field-of-view or smaller) sources in the S/N range of ~10-1000. These options are useful if your source does not match these criteria.

bright		Use for bright sources (S/N > ~1000). This setting
		entirely bypasses all filtering to produce a very
		faithful map. The drawback is more noise, but
		that should not be an issue for such a bright guy :-)
		Will invoke 'bright.cfg'.

faint		Use with faint sources (S/N < ~30) when the
		source is faint but still visible in a single scan. 
		This setting applies some more aggressive filtering of 
		the timestreams, and extended structures. It invokes 
		'faint.cfg'.

deep		Use for very faint sources which are not at all 
		detected in single scans, or if you think
		there is too much residual noise (baselines) in your 
		map to your liking. This setting results in the most 
		agressive filtering. Will load the configuration from
		'deep.cfg'. The output map is optimally filtered
		(smoothed) for point sources.

extended	Try to better preserve extended structures. This
		setting can be used alone or in combination with
		the above brightness options. See also '-sourcesize=X' 
		below. With the fainter settings the recovery of 
		extended structures becomes increasingly more 
		difficult. For bright structures recovery up to FOV 
		(or beyond!) should be possible, while for faint 
		structures \~1/4 FOV - FOV scales are maximally 
		obtainable (see more on this in the section below.)

source.type=<type>	By default, crush will try to make a map from
			the data. However, some istruments may take
		data that is analyzed differently. For example, you 
		may want to use crush to reduce pixel maps (to
		determine the positions of your pixels on sky), or
		skydips (to derive appropriate opacities), or do
		point source photometry. Presently, the following
		source types are supported accross the board:
	
		   map 		Make a map of the source (default)

		   skydip	Reduced skydips, and determine 
				opacities by fitting a model to it.

		   pixelmap	Create individual maps for every
				pixel, and use it to determine their
				location in the field of view.
		
		Note, that you may also just use 'skydip' and 
		'pixelmap' shorthands to the same effect. E.g.

		  > crush [...] -skydip [...]

sourcesize=X	This option can be used instead of 'extended' in 
		conjunction with 'faint' or 'deep' to specify the 
		typical size of sources (FWHM in arcsec) that are 
		expected. The reduction then allows filtering 
		structures that are much larger than the specified 
		source-size...
		If 'sourcesize' or 'extended' is not specified, then 
		point-like compact sources are assumed.	

b. Other commonly used pipeline settings:

(in typical order of importance to users...)

outpath=<path>	Set the directory into which output files (e.g. maps) 
		will be written. Can use '\~' for home directory and 
		environment variables in {}'s. Thus,

		   outpath=\~/images

		and

		   outpath={$HOME}/images

		are equivalent


rounds=N	Iterate N times. You may want to increase the number
		of default iterations either to recover more extended
		emission (e.g. when 'extended' is set), or to go
		deeper (esp. when the 'faint' or 'deep' options are
		used).


iteration.[N]	Use as a condition to delay settings until the Nth
		iteration. E.g.

		   iteration.[3] smooth halfbeam

		or 

		   > crush [...] -iteration.[3]smooth=halfbeam [...]
		 
		to specify half-beam smoothing starting from the 3rd
		iteration.
		
iteration.[last]    Specify settings that should be interpreted only
		    at the beginning of the last iteration.

final:key=value	    A shorthand for the above :-).

iteration.[last-N]  Activate settings N iterations before the last
		    one.

iteration.[xx%]	    Activate settings as a percentage of the total
		    number of iterations (as set by 'rounds'). E.g.
		
			iteration.[50%] forget clip

		    can be used to disable the S/N clipping of the
		    source map half way through the reduction.

		Because of the flexible syntax, the same iteration
		can be referred to in different ways. Consider
		a reduction with 10 rounds. Then,

			iteration.[5] smooth 5.0
			iteration.[50%] smooth 10.0
			iteration.[last-5] smooth beam

		can all be used to define what happens in the 5th
		iteration. CRUSH will parse these conditionals in 
		the above order: first the explicit iteration settings
		then those relative to the reduction length, and
		finally the settings relative to the end of the
		reduction. Thus, in the above example the beam 
		smoothing will always override the other two settings.

idle=N		Instruct crush NOT to use N number of CPUs of the 
		machine. By default crush will try to use all 
		processors in your machine for maximum performance. 
		This option allows to modify this behavior according 
		to need. Note, that at least 1 CPU will always be used 
		by crush, independent of this setting.
		The number of actual parallel threads will be the 
		smaller of the allowed number of CPUs and the number 
		of scans processed.

3.3. Source (Map) Configuration

altaz		Shorthand for 'system=horizontal' to reduce in Alt/Az.
		It is also aliased to 'horizontal'.

grid=X		set the map pixelization to X arcsec. Pixelization
		smaller than 2/5 beam is recommended. The default is
		\~1/5 beam pixelization.

name=		Specify the output image file name, relative to the
		directory specified by 'outpath'. When not given
		minicrush will chose a file name based on the source
		name and scan number(s), which is either

			<sourcename>.<scanno>.fits

		or

			<sourcename>.<firstscan>-<lastscan>.fits

		For mapping. Other source model types (e.g. skydips
		or pixel maps) may have different default naming 
		conventions.

pixelmap	Reduce pixel map data. Instead of making a single map
		from all pixels, separate maps are created for each
		pixel (Note, this can chew up some memory if you have
		a lot of pixels). At the end of the reduction CRUSH
		determines the actual pixel offsets in the focal plane.
		This option is equivalent to 'source.type=pixelmap'.

pixelmap.writemaps	Pixel maps (above) normally only produce the
			pixel position information. Use this option
		if you want CRUSH to write individual pixel maps as
		well, so you can peek at these yourself.

projection=	Choose a map projection to use. The following 
		projections are supported:

			SFL  --  Sanson-Flamsteed
			SIN  --  Slant Orthographic
			TAN  --  Gnomonic
			ZEA  --  Zenithal Equal Area
			SFL  --  Sanson-Flamsteed
			MER  --  Mercator
			CAR  --  Plate-Carree
			AIT  --  Hammer-Aitoff
			GLS  --  Global Sinusoidal
			STG  --  Stereographic
			ARC  --  Zenithal Equidistant

skydip		Reduce skydip data, instead of trying to make an 
		impossibly large map out of it :-). This option is
		equivalent to specifying 'source.type=skydip'.

smooth=X	Smooth the map by X arcsec FWHM beam. Smoothing
		helps improve visual appearance, but is also useful
		during reduction to create more redundancy in the data
		in the intermediate reduction steps. Also, smoothing
		by the beam is optimal for point source extaction from
		deep fields. Therefore, beam smoothing is default in
		with the 'deep' option (see 'deep.cfg').
		Typically you want to use some smoothing during 
		reduction, and you may want to turn it off in the 
		final map. Thus, you may have something like:

		  smooth=9.0			# 9" smoothing at first
		  iteration.[2]smooth=12.0 	# smooth more later
		  iteration.[last]forget=smooth # no smoothing at last

		Other than specifying explicit values, you can use
		the predefined values: 'minimal', 'halfbeam', '2/3beam'
		or 'beam'. See more on smoothing in the advanced
		configuration section.

source.filter   Filter extended structures. By default the filter will
		skip over map pixels that are above the 'blanking' S/N 
		level (>6 by default). Thus any structure above this 
		significance level will remain unfiltered.
		Filtering is useful to get deeper in the map when 
		retaining the very faint extended structures is not 
		an issue. Thus filtering above 5 times the source size
		(see 'sourcesize') is default when the filter is used.
		See the advanced configuration section for further
		details on fine tuning the large-scale structure 
		filter.	

source.fixedgains	Specifies to use the fixed source gains 
		        (e.g. from an RCP file -- see 'rcp' key).
		Normally, crush calculates source gains based on the 
		correlated noise response and the specified point
		source couplings (e.g. as derived from the two gain
		columns of RCP files.)

system=<type>	Select the coordinate system for mapping. The default
		is 'equatorial'. Other possibilities are 'horizontal'
		'ecliptic', 'galactic' or 'supergalactic'. Each of 
		these values is additionally aliased to simple keys. 
		Thus, you may use:
		
		   > crush -galactic [...]
		   
		as a shorthand for '-system=galactic'.
		       
unit=xxx	Set the output to units xxx. You can use either the
		instrumental units (e.g. 'V/beam' or 'counts/beam') or
		the more typical 'Jy/beam' (default), as well as their
		common multiples (e.g. 'uJy/beam', or 'nV/beam').	

3.4. Scan Configuration

Some options relate to the scans, helping to configure and handle them These options are specified before the list or range of scans to which they apply, and remain valid for all scans read after, or until an overriding option is placed. E.g.

> ./crush -option1=x 10218-10532 12425 -option2=y 11411 \
          -option1=z 10496

will set option1 to 'x' for all scans but the last one, which will have this option set to 'z'. And the last two scans will have option2 set to 'y'.

A detailed listing of all scan specific options can be found in the 'GLOSSARY'. Here are a few of the most commonly used ones (in alphabetical order).

pointing=dx,dy	  Specify incremental pointing offsets x,y in the
		  system of the telescope mount (I.e., azimuth and
		  elevation for horizontal mounts. (Note, this option
		  works like pcorr in APECS). 

datapath=<dir>	  Start looking for raw data in directory <dir>. Some
		  instruments may also interpret it as a root directory
		  in which data may reside some specific hierarchy. E.g.
		  in <dir>/<project> for APEX bolometers. Thus, if an
		  APEX instrument defines:
			
			datapath /homes/data
			project T-79.F-0002-2007

		  then crush will try to find data first in
		  '/homes/data', then in '/homes/data/T-79.F-0002-2007'
		  This provides a convenient way for accessing
		  hierarchically stored data. See the instrument-
		  specific usage of 'datapath' in the GLOSSARY.

jackknife	  Jackkniving is a useful technique to produce accurate
		  noise maps from large datasets. When the option is 
		  used the scan signals are randomly inverted, s.t. the
		  source signals in large datasets will tend to cancel 
		  out, leaving one with pure noise maps.

project=<id>      Some instruments (e.g. APEX bolometers) may require 
		  a project ID to be set in order to locate scans by 
		  serial number. Use capitalized form when defining 
		  APEX projects. E.g.,

                          project  T-79.F-0002-2007

read <filename>	  Read the scan data from <filename>, which can be 
		  either a fully specified path, or relative to
		  'datapath'. (On the command line it is sufficient
		  to list the filename without a preceding dash.)

read scanNo	  Read scan number scanNo (scripting only).
		  in command line mode, ommit '-read='.

read from-to	  Read the range of scan numbers between from and to.
     		  (inclusive). On the command line, you can omit 
		  '-read=' and simply list the scan range.

read X Y [...]    You can combine scan numbers and different ranges
		  in a space-spearated list...
		
read...		  As mentioned, the 'read' keys only apply to scripts.
		  Thus, 

			read 10755-10762             # in script

		  and

			> ./crush [...] 10755-10762  # command line

		  are equvivalent.
			
scale=X		  Scale the fluxes by X. With this option you can apply
		  calibration corrections to your data.

scale=<filename>  Alternatively, scan specific scaling can be defined
		  by an appropriate calibration file, which among 
		  other things, contains the ISO time-stamp and	
		  the corresponding calibration values for each scan.
		  The filename may contain references to environment
		  variables enclosed in {} brackets. E.g.:

		    scale={$HOME]}/laboca/scaling.dat

scramble	  Another technique for generating noise maps, which 
		  can be used also for small datasets, for which
		  jackknifing cannot be used. This approach scrambles
		  the pixel positions, such that the source signals
		  will be smeared out in the maps.

tau=X		  Specify a zenith tau value to use. When not used
		  minicrush will try to interpolate from
		  <instrument>/<instrument>-taus.master.dat if possible
		  or use 0.0 as default.

tau=<filename>	  Alternatively, tau can also specify a file-name with 
		  lookup information (usually containing tau
		  values from the radiometer or from the skydips).
		  Tau values will be interpolated for each scan,
		  as long as the scan falls inside the interpolator's
		  range. Otherwise, tau of 0.0 will be used. The 
		  filename may contain references to environmnent 
		  variables enclosed in {} brackets. E.g.:

		    tau={$HOME}/laboca/tau.dat

3.5. Instrument Configuration

There are various instrument configuration files. These reside in the corresponding instrument subdirectories inside the main crush directory. Some types of files are commonly used among many or all instruments, while others may be strongly instrument specific.

In most cases the instrument configurations should be set correctly, and you probably can leave these settings alone. However, here you will find some of the most common instrument configuration options that you may, at times, want to adjust to your preference.

A more complete list of the available instrument-specific configuration options can be found in the GLOSSARY.

Generic Instrument Parameters

pixeldata=<filename>	Specifies a pixel data file, providing initial
			gains, weights and flags for the detectors,
		and possible other information as well depending on the
		specific instrument. Such files can be produced via the
		'write.pixeldata' option (in addition to which you
		may want to specify 'forget=pixeldata' s.t. flags are
		determined without prior bias).

rcp		Specify pixel positions (and optionally point-source
		and sky-noise gains). These are standard IRAM
		or APEX RCP files containing the information in ASCII
		columns. RCP files can be produced by the 'pixelmap' 
		option, from scans, which move a bright point source 
		over all pixels.

rcp.gains	Use gains from the RCP files. Otherwise gains may 
		come from the 'pixeldata' file, or assume default
		values, such as 'uniform'.

rcp.center=x,y	Define the center RCP position at x,y in arcseconds.
		Centering takes place immediately after the parsing
		of RCP data.

rcp.rotate=X	Rotate the RCP positions by X degrees (counter 
		clockwise). Rotations take place after centering (if
		specified).

rcp.zoom=X	Zoom (rescale) the RCP position data by the scaling
		factor X. Rescaling takes place after the centering
		(if defined).	

stability=X	Specify the instrument 1/f stability time-scale in
		seconds. This is used for determining optimal 1/f
		filtering of the time-streams in the reduction.

3.6. Advanced startup environment and Java configuration

We have seen before how to configure the startup environment and Java by placing configuration snipplets inside the directory ~/.crush2/startup/. However, what if you want certain settings to apply to just a specific program within the crush suite (e.g. to crush or show) only, but not to all of them? That too is possible, by (creating and) editing a file under the corresponding ~/.crush2/startup/<progname> directory. E.g.:

~/.crush2/startup/crush/myconfig.conf

will source the bash snippled contained in myconfig.conf for reductions (crush executable) only. As with other startup scripts, the name of the file itself is irrelevant, and all files will be parsed inside the program's startup directory (albeit in unspecified order!).

Inside this file you can set up environment variables, and add shell (bash) scripts. E.g., the crush startup configuration file may contain the lines:

CRUSH_NO_UPDATE_CHECK="1"
EXTRAOPTS="$EXTRAOPTS -Djava.awt.headless=true"

(The first disables update checking, the second adds headless mode to the list of extra Java options already defined).


4. Correlated Signals

The removal of correlated signals, atmospheric or instrumental, is really the heart of CRUSH. In its most generic form the decorrelation is a two-step process: the first step finds and removes the correlated signals, assuming some initial detector gains to it, after which the gains are estimated during the second step based on the individual detector responses to the bolometer signals. (For details, see the CRUSH paper: Kovacs 2008, Proc. SPIE, 7020, 45.) Flagging of non-responsive pixels, based on outlying gains (i.e. responses to the correlated signals) may be part of the second step of the decorrelation.

4.1. Modes and Modalities

For each instrument, crush defines a set of modalities (i.e. a set of correlated modes) on which decorrelation may be performed. Each mode in a modality affects a group of pixels. Thus, the collection of pixel groups with related modes constitutes a division. For example, SHARC-2 has 12 detector rows (each with 32 pixels). The dividing of pixels into 12 groups, each representing a detector row, is a pixel division. Pixels in a given row respond to the same correlated mode, and so there are 12 row-related modes, which are bunched together in a modality.

Some of the modalities (and pixel divisions) are common to all (or most) instruments. These are:

<modality name> Description
all All channels, regardless of their state
live All channels that aren't dead.
detectors All detectors (e.g. excluding resistors etc.)
obs-channels All observing detectors (e.g. that see sky).
gradients A focal plane gradients of 'observing' detectors.
blinds Blind detectors (if exist)
telescope-x Telescope position in the x direction (e.g. AZ)
telescope-y Telescope poistion in the y direction (e.g. EL)
accel-<dir> Acceleration in some direction (see below).
chopper-<dir> Chopper motion in some direction (see below).

Above, the motion-related modalities may have the following directionalities:

<dir> Description
x x direction.
y y direction.
x^2 square of the x coordinate.
y^2 square of the y coordinate.
` x
` y
mag vector magnitude.
norm square of vector magnitude.

4.2. Removing Correlated Signals

The decorrelation step is invoked by:

correlated.<modality-name>

For example, a decorrelation of the signals induced by the chopper displacement in x (e.g. SHARC-2) is invoked by:

correlated.chopper-x

(Of course, the same keyword must also appear in the pipeline 'ordering', for the step to actually take place during the reduction process -- otherwise crush will not know when to correlated.the chopper signals.)

You can fine-tune how CRUSH deals with each correlated mode. For example, the following lines:

correlated.obs-channels.resolution 1.0
correlated.obs-channels.gainrange 0.3--3.0

Specifies that atmospheric signals, which appear in all observing channels, should be correlated. only every second (vs. the default every available frame), and that any channel that exhibits a response outside of 0.3--3.0 times the 'average' response of the array, should be flagged, and ignored in the reduction afterwards.

4.3. User-defined Modalities

In addition to the most common modalities listed above, each instruments defines its specific ones (e.g. rows for SHARC-2 from the example above). See the README files in the instrument sub-directories of crush for a full list of predefined instrument specific modalities. You can also define your own pixel groups, divisions and modalities using the group and division keywords. Here is an example:

group.my-group-1 10,13,20-25
group.my-group-2 40-56,78,80-82
division.my-division my-group-1 my-group-2

The first two lines defines two pixel groups (my-group-1 and my-group-2) from pixel indexes and ranges, while the last line creates a division (my-division) from these pixel groupings. CRUSH will also create a correlated modality with the same name as the division. So, given the definitions above you can correlated.on my-division by:

correlated.my-division

5. Custom Logging Support

CRUSH (as of version 2.03 really) provides a poweful scan/reduction logging capability via the log and obslog keys.

log writes the log entries for each scan after the reduction, whereas obslog does not reduce data at all, only logs the scans immediately after reading them. While both logging functions work identically, some of the values are generated during reduction, and therefore may not be available to obslog.

5.1. Log Files and Versioning

You can specify the log files to use with the log.file and obslog.file keys (the default is to use <instrument>.log and <instrument>.obs.log in the outpath directory). Equivalently, you can also set the filename directly as an optional argument to log and obslog:

> crush [...] -log=myLog.log [...]

You can specify the quantities, and the formatting in which they will appear, using the log.format and obslog.format keys. Below you will find more information on how to set the number formatting of quantities and a list of values available for the logging.

A log file always has a fixed format, the one which was used when creating it. Therefore, a conflict may arise if the same log file is specified for use with a different format. The policy for resolving such conflicts can be set via the log.conflict and obslog.conflict keys, which can have one of the following values:

overwrite   Overwrites the previous log file, with a new one in the
	    newly specified format

version	    Tries to find a sub-version of the same log file (with .1,
	    .2, .3 ... extension added) in the new format, or create
	    the next available sub-version.

The default behaviour is to assume versioning, in order to preserve information in case of conflicts.

5.2. Decimal Formatting of Values

Many of the quantities you can log are floating point values, and you have the possibility of controlling how these will appear in your log files. Simply put one of the formatting directives in brackets after the value to specifts its format.

E.g. the keys RA or RAh will write the right-ascention coordinate either as radians or as hours, with the default floating point formats. However, RA(h:2) will write the value in human-readable hh:mm:ss.ss format, whereas RAh(f3) will express it as hh.hhh.

You can choose from the following formats to express various quantities. Be careful, because not all formats are appropriate to all types of data. (For example, you should not try to format angles expressed in degrees with the DMS formatting capabilities of the a angle formats. Use these only with angles expressed in radians!)

d0...d9		Integer format with 0...9 digits. E.g. 'd3' will write
		Pi (3.1415...) as 003.

e0...e9		Exponential format with 0...9 decimals. E.g. e4 will
		write Pi as 3.1415E0.

f0...f9		Floating point format with 0...9 decimals. E.g. f3 will
		write Pi as 3.141.

s0...s9		Human readable format with 0...9 significant figures 
		(floating point or exponential format,
		whichever is more compact).

a:0...a:3
as0...as3
al0...al3	Angle format with 0...3 decimals on the seconds. E.g.
		'a:1' produces angles in ddd:mm:ss.s format. Use only
		with values expressed as radians (not in degrees!).
		As such, 'a:1' will format Pi as '180:00:00.0'. The
		difference between the 'a:', 'as', and 'al' formats is
		the separators used between degrees, minutes and seconds
		(colons, symbols, and letters respectively).

h:0...h:3
hs0...hs3
hl0...hl3	Hour-angle format (e.g. for RA coordinate) with 0...3
		decimals on the seconds. E.g. 'h:2' formats angles in
		'hh:mm:ss.ss' format. Use only with values expressed
		as radians (not in degrees!). As such 'h:2' will format
		Pi as '12:00:00.0'. The difference between the 'h:', 
		'hs', and 'hl' formats is the separators used between 
		hours, minutes and seconds (colons, symbols, and 
		letters respectively).

		E.g. Pi will be:

			h:1	12:00:00.0
			hs1	12h00'00.0"
			hl1	12h00m00.0s

t:0...t:3
ts0...ts3
tf0...tf3	Time format with 0...3 decimals on the seconds. E.g.
		't:1' formats time in 'hh:mm:ss.s' format. Use only
		on time values expressed in seconds! The difference 
		between the 't:', 'ts', and 'tl' formats is the 
		separators used between hours, minutes and seconds 
		(colons, symbols, and letters respectively).

5.3. Loggable Quantities

Currently, CRUSH offers the following quantities for logging. Directives starting with '?' will log the values of configuration keys. Other quantitites reflect the various internals of the scan or instrument state. More quantities will be added to this list in the future, especially values that are specific to certain instruments only. Keep an eye out for changes/addititions :-).

?<keyword>       The value of the configuration <keyword>. If the
                 configuration option is not defined '---' is written.
                 If the keyword is set without a value, then '<true>'
                 is written.

AZ               Azymuth coordinate (in radians). E.g. 'AZ(a:1)'
                 produces ddd:mm:ss.s formatted output. See also 'AZd' 
                 and 'EL'.

AZd              Azymuth coordinate in degrees. E.g. 'AZd(f2)'
                 produces output in ddd.dd format. See also 'AZ' and 
                 'ELd'

channels         Number of channels processed in the reduction. See
                 also 'okchannels' and 'maxchannels'.

chopeff          Chopper efficiency.

chopfreq         Chopper frequency (in Hz).

chopthrow        Chopper throw (2x amplitude).

creator          Software that created the data file (as stored by
                 the FITS CREATOR keyword).

date             Date (and time) of the observation. E.g. 
                 'date(yyyy-MM-dd)'. The format follows the rules for 
                 Java's SimpleDateFormat class.
		
DEC              Declination coordinate (in radians). E.g. 'DEC(a:0)'
                 produces the declination in ddd:mm:ss format. See also
                 'RA' and 'DECd'.

DECd             Declination coordinate (in degrees). E.g. 'DECd(f1)'
                 produces output in ddd.d format. See also 'RAh' and
                 'DEC'.

descriptor       A descriptor string for the scan (e.g. the scan number
                 or file name used to invoke the scan).

dir              Scanning direction. E.g. 'HO', 'EQ', 'EC', 'GL', 'SG'.

EL               Elevation coordinate (in radians). 

ELd              Elevation coordinate (in degrees).

epoch            Full coordinate epoch, e.g. (J2000.0).

epochY           E.g. epochY(f1) -> 2000.0

frames           Number of unflagged frames in the scan.

FWHM             Mean beam FWHM of the instrument (in 'sizeunit').

gain             Instrument gain.

generation       Source model generation.

hipass           Highpass filtering timescale (in seconds)

humidity         Ambient humidity (if recorded).

id               Scan identifier (e.g. scan number)

integrations     Number of integrations (subscans) contained in the 
                 scan.

LST              Local Sidereal Time (in seconds). E.g. use 'LST(t:1)'
                 to format is as 'hh:mm:ss.s'.

LSTh             LST in hours.

maxchannels      Maximum number of channels the instrument can have
                 (effectively the number of channels stored in the
                 data file).

maxFWHM          Smallest beam of the instrument (in sizeunit).

minFWHM          Largest beam of the instrument (in sizeunit).

MJD              Modified Julian Date of the scan.

mount            The focus in which the instrument is mounted.

NEFD             Average Noise Equivalent Flux Density (Jy sqrt[s]) as a
                 measure of the array sensitivity.

object           Name of observed object.

observer         Name(s) of the observer(s).

obshours         Effective on-source time (in hours).

obsmins          Effective on-source time (in minutes).

obstime          Effective on-source time (in seconds).

okchannels       Number of good (unflagged) channels.

PA               Mean parallactic angle (in radians).

PAd              Mean parallactic angle in degrees.

pnt.angle        Source elongation angle on map (degrees).

pnt.asymX        Asymmetry in telescope X (%). 

pnt.asymY        Asymmetry in telescope Y (%). `

pnt.AZ           Azymuth pointing (arcsec).

pnt.EL           Elevation pointing (arcsec).

pnt.DEC          Declination pointing (arcsec).

pnt.dasymX       Asymmetry uncertainty in telescope X (%).

pnt.dasymY       Asymmetry uncertainty in telescope Y (%). `

pnt.dangle       Source elongation angle error (deg).

pnt.dAZ          Azymuth pointing residual (if available).

pnt.dDEC         Pointing residual in the declination dir (if available).

pnt.dEL          Elevation pointing residual (if available).

pnt.delong       Source elongation error (%).

pnt.delongX      Source elongation error in telescope X direction (%).

pnt.dNasX        Pointing residual in the Nasmyth X dir (if available).

pnt.dNasY        Pointing residual in the Nasmyth Y dir (if available).

pnt.dRA          Pointing residual in the RA direction (if available).

pnt.dX           Pointing residual in native X direction.

pnt.dY           Pointing residual in native Y direction.

pnt.elong        Source elongation (%).

pnt.elongX       Source elongation in telescope X direction (%).

pnt.RA           RA pointing (arcsec).

pnt.X            Aggregated X pointing correction (including applied
                 corrections and sometimes the pointing model too).

pnt.Y            Aggregated Y pointing correction (including applied
                 corrections and sometimes the pointing model too).

pressure         Ambient pressure (if recorded) in hPa.

project          Project name (if defined)

ptfilter         The point-source flux filtering throughput of the 
                 reduction.

RA               Right-ascention coordinate (in radians). E.g. use
                 'RA(h:2)' to format as 'hh:mm:ss.ss'.

RAd              Right-ascention coordinate (in degrees).

RAh              Righ-ascention coordinate (in hours).

rate             Sampling rate of the data (Hz). See also 'sampling'

resolution       Instrument resolution (i.e. FWHM of the main beam).
                 in 'sizeunit'.

rmsspeed         RMS fluctuation of the scanning speed (arcsec/s).

sampling         Sampling interval (seconds). The inverse of 'rate'.

scale            Scaling factor applied to the scan. 

scanspeed        Average scanning speed (arcsec/s).

serial           Serial number of the scan.

sizeunit         Size unit for measuring resolutions. E.g. 'arcsec'. 

src.a            Major axis (arcsec) of the source ellipse.

src.angle        Orientation of the source ellipse (deg).

src.b            Minor axis (arcsec) of the source ellipse.

src.da           Uncertainty of the minor axis (arcsec).

src.dangle       Uncertainty of the orientation (deg).

src.db           Uncertainty of the major axis (deg).

src.dFWHM        Uncertainty of the source FWHM (arcsec).

src.dint         Uncertainty of the integrated source flux (Jy). 

src.dpeak        Uncertainty of the peah source flux (Jy/beam).

src.FWHM         Source FWHM (arcsec).

src.int          Integrated source flux (Jy) insize and adaptive
                 aperture.

src.peak         Peak source flux (Jy/beam).

stat1f           A measure of the 1/f noise averaged over the array.
                 The configurations options '1overf.freq' and 
                 '1overf.ref' define the frequencies for the 1/f
                 measurement and white-noise reference respectively.

Tamb             Ambient temperature (if recorded) in degrees C.

tau              The in-band, line-of-sight opacity value.

tau.<id>         The zenith tau value for <id>. E.g. 'tau.225GHz' or
                 'tau.PWV' if these are defined by appropriate scaling
                 relations.

UT               UT in seconds. E.g. use 'UT(t:1)' to format is as 
                'hh:mm:ss.s'.

UTh             UT in hours.

weight          The relative weight of the scan, based on the actual
                noise of the map it generates.

winddir         Wind direction (if recorded) in degrees.

windpk          Peak wind speed (if recorded) in m/s.

windspeed       Wind speed (if recorded) im m/s.

zenithtau       In-band opacity at zenith.

Source model specific log entries

map.beams        Number of (smoothed) beams in the map.

map.contentType  Type of data stored in the map.

map.creator      Creator's name or description.

map.depth        Weighted average depth of the map in map units.

map.max          Maximum value in map units.

map.min          Minimum value in map units.

map.name         Name of map data (e.g. object's name)

map.points       Number of pixels in the map.

map.rms          Typical RMS of the map in map units.

map.size         Size of the map in pixels e.g. '121x432'.

map.sizeX        Map size in the 'x' direction (pixels).

map.sizeY        Map size in the 'y' direction (pixels).

map.system       Coordinate system id, e.g. 'HO', 'EQ', 'GL' etc.

map.unit         Name of map unit, e.g. 'Jy/beam'.

phot.flux        Photometric flux in Jy/beam (e.g. for LABOCA).

phot.dflux       Uncertainty in the photometric flux (Jy/beam).

skydip.kelvin    Data units that corresponf to a 1K load on the detectors.
                 (for skydip data, if and when fitted.)

skydip.dkelvin   Uncertainty in data units per kelvin conversion.

skydip.tau       Tau value derived (or assumed) from skydip data

skydip.dtau      Tau uncertainty from skydip data

skydip.tsky      Derived (or assumed) sky temperature in K.

skydip.dtsky     Sky temperature uncertanity (K).

smooth           Smoothing applied, in the default size unit of
                 the instrument (e.g. in arcsecs).

Telescope specific log entries

APEX: see the `crush/Documentation/README.apex.

Instrument specific log entries

See the crush/Documentation/README.<instrument>, for each <instrument>.


6. Supported Cameras

Currently, CRUSH supports the following instruments (in alphabetical order):

GISMO        (2mm) Goddard-IRAM Superconducting 2-Millimeter Observer
             www.iram.es/IRAMES/mainWiki/GoddardIramSuperconductingTwo\
             Millimeter

LABOCA       (870um) Large APEX Bolometer Camera
             www.apex-telescope.org/bolometer/laboca

PolKa        (polarimetry) Polarimeter for LABOCA
             www.mpifr-bonn.mpg.de/staff/gsiringo/laboca/laboca\
             _polarimeter.html

SABOCA       (350um) Submillimeter APEX Bolometer Camera
             www.apex-telescope.org/bolometer/saboca

SCUBA-2      (450um, 850um) Submillimetre Common User Bolometer Array 2
             www.roe.ac.uk/ukatc/projects/scubatwo

SHARC-2      (350um, 450um, 850um) The second-generation Submillimeter 
             High-Angular Resolution Camera
             Caltech, Pasadena, CA
             www.submm.caltech.edu:/~sharc

SOFIA/HAWC+  (53um, 62um, 89um, 155um, 216um)
             High-resolution Airborne Wide-angle Camera

The following instruments have legacy support only (supported by CRUSH 2.4x and earlier releases):

ASZCa        (2mm) APEX SZ Camera, from Berkeley, CA
             bolo.berkeley.edu/apexsz/instrument.html

MAKO         (350um) KID technology demonstration camera for the CSO.

MAKO-2       (350um, 850um) Second-generation KID technology demonstration
             camera for the CSO with dual-pol pixel response, and dual-band 
             imaging capability.

MUSTANG-2    (3mm) Large focal plane array for the 100m Greenbank Telescope.
             www.gb.nrao.edu/mustang/

p-ArTeMiS    (200um, 350um, 450um) 3-color camera for APEX (prototype)
             www.apex-telescope.org/instruments/pi/artemis

SHARC        (350um, 450um) Submillimeter High-Angular Resolution Camera
             Caltech, Pasadena, CA
             http://www.submm.caltech.edu/cso/sharc/cso_sharc.html

Further support for instruments is possible. If interested to use CRUSH for your bolometer array, please contact Attila Kovacs <attila[AT]sigmyne.com>.

In principle, it is possible to extend CRUSH support for other types of scanning instruments, like heterodyne receiver arrays, or single-pixel receivers that scan in frequency space, or for the reduction of line-surveys from double side-band (DSB) receivers. Such new features may appear in future releases, especially upon specific request and/or arrangement...


7. Future Developments (A Wish List...)

There are a number of plans for new features in CRUSH. Some may come in the near future, others perhaps later, depending on the resources available for implementation. Nonetheless, the following avenues of feature enrichment are being considered.

7.1. Support for Heterodyne Receivers and Interferometry

CRUSH was born as a bolometer array reduction package. However, there is no reason why many of its principles could not be applied to other types of instruments (astronomical or otherwise), especially those, which are affected by correlated signals. Besides, CRUSH also provides powerful tools for other analysis steps, like weighting, despiking, spectral filtering, and mapping.

Two obvious extensions would be heterodyne receivers (and receiver arrays) and the use of CRUSH for interferometry (e.g. ALMA) since it is well-suited to deal with immense data volumes also.

7.2. Interactive Reductions

At present CRUSH offers only a reduction pipeline. This is ideal for crunching large volumes of data in a more-or-less automated fashion, and for making reduction painless. It is also ideal for most users, who might not wish to learn about the intricacies of each and every reduction step. However, in some cases having more control may be beneficial.

Many astronomers are used to interactive reduction packages (e.g. GILDAS AIPS, BoA etc.). CRUSH should eventually offer such a capability also. This would allow step-by-step reductions, together with varous plotting capabilities to look at data at every stage. Such a capability would be very useful in the process of building pipelines for new instruments

It should be relatively simple to provide this feature. The current configuration languange of CRUSH can be easily adapted for more interactive use. Even more detailed control may be possible though a standard scripting language like JavaScript or Rhino. The main job would be to supply the essential plotting capabilities, but that too may come sooner than later...

7.3. Distributed and GPU Computing

The reduction paradigm of CRUSH is massively parallelizable. CRUSH can already make good use of any number of processing cores within a computer. It should be quite staighforward to extend implementation over several computing nodes and super-computers, allowing orders of magnitude increases in data reduction speeds and data volumes.

Another way of improving speeds may come from the use of Graphical (GPU) Computing. Today's grahics chips provide computing power well in excess of that offered by the CPUs. This can be harnessed with programming tools like CUDA or OpenCL. GPU computing is still in its infancy, and thus its languange specifications are likely very fluid. But technical details aside, GPUs clearly offer a way for boosting the number crunching performace of CRUSH.

7.4. Graphical User-Interface (GUI)

The addition of a Graphical User Interface (GUI) to CRUSH would make configurations more transparent and intuitive. It would allow users, who do not use CRUSH often, to avoid learning the complexities of command-line options and scripting support, and instead click their way through the essential configuration settings.

GUIs may also aid the more expert users, in providing a way to look at details, such as monitoring quantities, signals or maps during the reduction.


8. Further information

CRUSH-2 is the next-generation data reduction package, inheriting its DNA from the pioneering SHARC-2 specific version (crush-1.xx) and from the APEX specific minicrush implementation. It is a much more capable package than either of its predecessors. However, because the output FITS images are backward compatible with the older versions (for the most part), parts of the original CRUSH package can still be useful for manipulating images post reduction.

Mainly, the crush-1.xx distribution provides tools for displaying (show), minupulating (imagetool), and coadding (coadd) or differencing images (difftool), and producing histogram (histogram) from them.

Downloading the CRUSH package(s), and further information on the FITS images is available at:

http://www.sigmyne.com/crush

I hope you'll find this helpful and may you end up with beautiful images!!!


Copyright (C)2019 -- Attila Kovacs