xxxxxxxxxx
# Herschel-Stripe-82: Validation Report (SUBSET)

Herschel-Stripe-82: Validation Report (SUBSET)

xxxxxxxxxx
 
Master catalogue used: __master_catalogue_herschel-stripe-82_RANDOM10PCSAMPLE_20180113.fits (10% of master catalogue)__ <br>
Number of rows: 4,924,329
<br>
Surveys included:<br>
| Survey     | Telescope / Instrument      |      Filters (detection band in bold)      | Location                    |
|------------|-----------------------------|:------------------------------------------:|-----------------------------|
| HSC SSC PDR1 | Subaru/HSC                |   grizy,N921,N816                          | dmu0_HSC                    |
| PS1 3PSS   | Pan-STARRS1/Pan-STARRS1     |   grizy                                    | dmu0_PanSTARRS1-3SS         |
| DECaLS     | Blanco Telescope/Dark Energy Camera| ugrizY                              | dmu0_DECaLS                 |
| RCSLenS    | CFHT/MegaPrime/MegaCam      |   grz                                      | dmu0_RCSLenS                |
| SDSS S82   | SDSS                        |   ugriz                                    | dmu0_SDSS-S82               |
| VHS        | VISTA/VIRCAM                |   Y,J,H,K                                  | dmu0_VISTA-VHS              |
| VICS82     | VISTA+CFHT/VIRCAM+WIRCA     |   J,K                                      | dmu0_VICS82                 |
| UKIDSS LAS | UKIRT/WFCAM                 |   Y,J,H,K                                  | dmu0_UKIDSS-LAS             | 
| SHELA      | Spitzer/IRAC                |   IRAC12                                   | dmu0_SHELA                  |
| SpIES      | Spitzer/IRAC                |   IRAC12                                   | dmu0_SpIES                  |
__NB__: No error is available for VICS82 J and K magnitudes (aperture and total)

Master catalogue used: master_catalogue_herschel-stripe-82_RANDOM10PCSAMPLE_20180113.fits (10% of master catalogue)
Number of rows: 4,924,329
Surveys included:

Survey Telescope / Instrument Filters (detection band in bold) Location
HSC SSC PDR1 Subaru/HSC grizy,N921,N816 dmu0_HSC
PS1 3PSS Pan-STARRS1/Pan-STARRS1 grizy dmu0_PanSTARRS1-3SS
DECaLS Blanco Telescope/Dark Energy Camera ugrizY dmu0_DECaLS
RCSLenS CFHT/MegaPrime/MegaCam grz dmu0_RCSLenS
SDSS S82 SDSS ugriz dmu0_SDSS-S82
VHS VISTA/VIRCAM Y,J,H,K dmu0_VISTA-VHS
VICS82 VISTA+CFHT/VIRCAM+WIRCA J,K dmu0_VICS82
UKIDSS LAS UKIRT/WFCAM Y,J,H,K dmu0_UKIDSS-LAS
SHELA Spitzer/IRAC IRAC12 dmu0_SHELA
SpIES Spitzer/IRAC IRAC12 dmu0_SpIES

NB: No error is available for VICS82 J and K magnitudes (aperture and total)

 
## I. Caveats

I. Caveats

 
### I.a. Magnitude errors 

I.a. Magnitude errors

xxxxxxxxxx
At faint magnitudes (mag > 24), some surveys have very large errors (> 10) on the magnitude. These objects may be unreliable for science puposes.<br>
This in includes __SDSS aperture and total__ magnitudes (at mag > 25), __DECaLS aperture and total__ magnitudes (at m >  20), __PanSTARRS aperture and total__ magnitudes (at mag > 21). Also few sources have large errors in __VHS total__ magnitudes (at mag > 20) and __UKIDSS total__ magnitudes (at mag > 23).<br>
<img src="help_plots/Herschel-Stripe-82_magVSmagerr_SDSS_g_mag_total.png" />
In particular, __HSC-SSP aperture and total__ magnitudes can go extremely faint (up to 60 mag!!) with errors larger than $10^{13}$!
<img src="help_plots/Herschel-Stripe-82_magVSmagerr_Suprime_r_mag_total.png" />

At faint magnitudes (mag > 24), some surveys have very large errors (> 10) on the magnitude. These objects may be unreliable for science puposes.
This in includes SDSS aperture and total magnitudes (at mag > 25), DECaLS aperture and total magnitudes (at m > 20), PanSTARRS aperture and total magnitudes (at mag > 21). Also few sources have large errors in VHS total magnitudes (at mag > 20) and UKIDSS total magnitudes (at mag > 23).

In particular, HSC-SSP aperture and total magnitudes can go extremely faint (up to 60 mag!!) with errors larger than 1013!

 
### I.b. Aperture corrections

I.b. Aperture corrections

 
In most of the case when comparing the aperture magnitudes between surveys, we observed a two peak distribution in the difference between the magnitudes ($\Delta_{mag} = mag_{survey1} - mag_{survey2}$). We have one peak around 0 for point-source objects, with a small spread. And a second peak at higher $\Delta_{mag}$ with a larger spread for extended objects; implying a different aperture correction between surveys for these objects.<br>
That means that galaxies will not have the same aperture magnitude in different surveys. <br>
In the griz bands, for bright sources, there is a two peaks distribution when comparing Pan-STARRS, DECaLS, HSC-SSP and SDSS aperture magnitues. Except when comparing Pan-STARRS and DECaLS magnitudes, $\Delta_{mag}$ is similar for point-sources and extended objects.<br>
Also, in the near-IR bands, only the magnitude of extended objects between UKIDSS and VHS agreed; while with the other surveys we get again a two peak distribution on $\Delta_{mag}$.
<img src="help_plots/Herschel-Stripe-82_apcorrIssues_SDSS_r_aperture_-_GPC1_r_aperture.png" />

In most of the case when comparing the aperture magnitudes between surveys, we observed a two peak distribution in the difference between the magnitudes (Δmag=magsurvey1magsurvey2). We have one peak around 0 for point-source objects, with a small spread. And a second peak at higher Δmag with a larger spread for extended objects; implying a different aperture correction between surveys for these objects.
That means that galaxies will not have the same aperture magnitude in different surveys.

In the griz bands, for bright sources, there is a two peaks distribution when comparing Pan-STARRS, DECaLS, HSC-SSP and SDSS aperture magnitues. Except when comparing Pan-STARRS and DECaLS magnitudes, Δmag is similar for point-sources and extended objects.
Also, in the near-IR bands, only the magnitude of extended objects between UKIDSS and VHS agreed; while with the other surveys we get again a two peak distribution on Δmag.

 
## II. Flags

II. Flags

 
### II.a. Pan-STARRS aperture and total magnitudes

II.a. Pan-STARRS aperture and total magnitudes

xxxxxxxxxx
 
Few Pan-STARRS sources have exactly the same error (of <font color='blue'>0.0010860000038519502</font>) on the __aperture and total__ magnitudes in all the grizy bands. The corresponding aperture magnitude should not be trusted for these objects.<br>
<img src="help_plots/Herschel-Stripe-82_gpc1Issues_GPC1_r_mag_aperture.png" />

Few Pan-STARRS sources have exactly the same error (of 0.0010860000038519502) on the aperture and total magnitudes in all the grizy bands. The corresponding aperture magnitude should not be trusted for these objects.

 
### II.b.DECaLS total magnitudes

II.b.DECaLS total magnitudes

xxxxxxxxxx
Few DECaLS sources have excatly the same total magnitude (of <font color='blue'>14.999935</font>) in the g- and r-bands. These magnitudes should not be trusted. <br>
Also, in the grz bands, few sources have very bright or negative total magnitude but no aperture magnitude (<font color='green'>$mag_{tot} < 7$</font>).
<img src="help_plots/Herschel-Stripe-82_decamIssues_DECam_z_mag_total.png" />

Few DECaLS sources have excatly the same total magnitude (of 14.999935) in the g- and r-bands. These magnitudes should not be trusted.
Also, in the grz bands, few sources have very bright or negative total magnitude but no aperture magnitude (magtot<7).

 
### II.c. Outliers

II.c. Outliers

xxxxxxxxxx
By comparing magnitude in the same band between different surveys, we can see that some magnitudes are significanlty different could not be trusted. <br>
The outliers are identified to have a large weighted magnitude difference (equivalent of the $chi^2$).
$$chi^2 = \frac{(mag_{1}-mag_{2})^2}{magerr_{1}^2 + magerr_{2}^2}$$ 
<br>
We used the 75th and 25th percentile to flagged the objects 5$\sigma$ away on the large values tail of the $chi^2$ ditribution. (__NB:__ bright sources tend to have their errors underestimated with values as low as $10^{-6}$, which is unrealistic. So to avoid high $chi^2$ due to unrealistic small errors, we clip the error to get a minimum value of 0.1% (i.e. all errors smaller then $10^{-3}$ are set to $10^{-3}$).)
<br><br>
$$outliers == [chi^2 >  (75th \;percentile + 3.2\times (75th \;percentile - 25th \;percentile))]$$
<img src="help_plots/Herschel-Stripe-82_outliers_GPC1_r_aperture_-_DECam_r_aperture.png" />

By comparing magnitude in the same band between different surveys, we can see that some magnitudes are significanlty different could not be trusted.
The outliers are identified to have a large weighted magnitude difference (equivalent of the chi2).

chi2=(mag1mag2)2magerr12+magerr22

We used the 75th and 25th percentile to flagged the objects 5σ away on the large values tail of the chi2 ditribution. (NB: bright sources tend to have their errors underestimated with values as low as 106, which is unrealistic. So to avoid high chi2 due to unrealistic small errors, we clip the error to get a minimum value of 0.1% (i.e. all errors smaller then 103 are set to 103).)

outliers==[chi2>(75thpercentile+3.2×(75thpercentile25thpercentile))]

In [ ]: