# EGS: Validation Report (FULL)

EGS: Validation Report (FULL)

xxxxxxxxxx
Master catalogue used: __master_catalogue_egs_20180207.fits__<br>
Number of rows: 1,412,613
<br>
Surveys included:<br>
| Survey        | Telescope / Instrument  | Filters (detection band in bold)  | Location        |
|---------------|-------------------------|:---------------------------------:|-----------------|
| CANDELS-3D-HST| HST/WFC3                | F140W, F160W, F125W,F606W, F814W  |dmu0_CANDELS-3D-HST|
| CANDELS-EGS   | HST/ACS/WFC3/Megacam/WIRCAM/NEWFIRM/IRAC| F140W, F160W, F125W,F606W, F814W, u,g,r,i,z,J,H,K,IRAC1234  |dmu0_CANDELS-EGS|
| CFHT-DEEP2    | CFHT/CFHT12k            | B,R,I                             | dmu0_DEEP2      |
| HSC-SSP       | Subaru/Hypersuprime     | g,r,i,z,y                         | dmu0_HSC        | 
| PS1-3SS       | Pan-STARRS1             | g,r,i,z,y                         | dmu0_PanSTARRS1-3SS|
| Legacy Survey | BASS                    | g, r, z                           | dmu0_LegacySurvey|
| CFHTLS-WIDE   | CFHT/MegaCam            | u, g', r', i', z'                 | dmu0_CFHTLS     |
| CFHTLS-Deep   | CFHT/MegaCam            | u, g', r', i', z'                 | dmu0_CFHTLS     |
| CFHTLenS      | CFHT/MegaCam            | u,g,r,i,z                         | dmu0_CFHTLenS   |
| CFHT-WIRDS    | CFHT/WIRCAM             | J,H,Ks                            | dmu0_CFHT-WIRDS |
| AEGIS         | Palomar/WIRCS           | J,K                               | dmu0_AEGIS      |
| UHS           | UKIRT/WFACM             | J                                 | dmu0_UHS        |
| EGS-IRAC      | CFHT12k/Megacam/MMT/WIRCS/MOIRCS/IRAC           | B,R,I, u,g,r,i,z, J,K, IRAC1234       | dmu0_IRAC-EGS   |

Master catalogue used: master_catalogue_egs_20180207.fits
Number of rows: 1,412,613
Surveys included:

Survey Telescope / Instrument Filters (detection band in bold) Location
CANDELS-3D-HST HST/WFC3 F140W, F160W, F125W,F606W, F814W dmu0_CANDELS-3D-HST
CANDELS-EGS HST/ACS/WFC3/Megacam/WIRCAM/NEWFIRM/IRAC F140W, F160W, F125W,F606W, F814W, u,g,r,i,z,J,H,K,IRAC1234 dmu0_CANDELS-EGS
CFHT-DEEP2 CFHT/CFHT12k B,R,I dmu0_DEEP2
HSC-SSP Subaru/Hypersuprime g,r,i,z,y dmu0_HSC
PS1-3SS Pan-STARRS1 g,r,i,z,y dmu0_PanSTARRS1-3SS
Legacy Survey BASS g, r, z dmu0_LegacySurvey
CFHTLS-WIDE CFHT/MegaCam u, g', r', i', z' dmu0_CFHTLS
CFHTLS-Deep CFHT/MegaCam u, g', r', i', z' dmu0_CFHTLS
CFHTLenS CFHT/MegaCam u,g,r,i,z dmu0_CFHTLenS
CFHT-WIRDS CFHT/WIRCAM J,H,Ks dmu0_CFHT-WIRDS
AEGIS Palomar/WIRCS J,K dmu0_AEGIS
UHS UKIRT/WFACM J dmu0_UHS
EGS-IRAC CFHT12k/Megacam/MMT/WIRCS/MOIRCS/IRAC B,R,I, u,g,r,i,z, J,K, IRAC1234 dmu0_IRAC-EGS
 
## I. Caveats

I. Caveats

 
### I.a. Magnitude errors 

I.a. Magnitude errors

xxxxxxxxxx
At faint magnitudes (mag > 24), some surveys have very large errors on the magnitude. These objects may be unreliable for science puposes.<br>
This in includes __CFHT aperture and total__ magnitudes (at mag > 29),  __HSC-SSP aperture and total__ magnitudes (at m > 25, also the magnitude could be as faint as 60 with corresponding error of 10$^{13}$), __PanSTARRS aperture and total__ magnitudes (at mag > 23), __Legacy Survey aperture and total__ magnitudes (at mag > 25), __WIRCAM aperture and total__ magnitudes (at mag > 26), __NEWFIRM total__ magnitudes (at mag > 27; no aperture magnitude available) and __IRAC total__ magnitudes (at mag > 23; no aperture magnitude available).<br>
<img src="help_plots/EGS_magVSmagerr_Megacam_u_mag_total.png" />

At faint magnitudes (mag > 24), some surveys have very large errors on the magnitude. These objects may be unreliable for science puposes.
This in includes CFHT aperture and total magnitudes (at mag > 29), HSC-SSP aperture and total magnitudes (at m > 25, also the magnitude could be as faint as 60 with corresponding error of 1013), PanSTARRS aperture and total magnitudes (at mag > 23), Legacy Survey aperture and total magnitudes (at mag > 25), WIRCAM aperture and total magnitudes (at mag > 26), NEWFIRM total magnitudes (at mag > 27; no aperture magnitude available) and IRAC total magnitudes (at mag > 23; no aperture magnitude available).

 
### I.b. Aperture corrections

I.b. Aperture corrections

xxxxxxxxxx
In most of the case when comparing the aperture magnitudes between surveys, we observed a two peak distribution in the difference between the magnitudes ($\Delta_{mag} = mag_{survey1} - mag_{survey2}$). We have one peak around 0 for point-source objects, with a small spread. And a second peak at higher $\Delta_{mag}$ with a larger spread for extended objects; implying a different aperture correction between surveys for these objects.<br>
That means that galaxies will not have the same aperture magnitude in different surveys. <br>
In the griz bands, for bright sources, there is a two peaks distribution when comparing Pan-STARRS, CFHT, HSC-SSP and Legacy Survey aperture magnitues.<br>
<img src="help_plots/EGS_apcorrIssues_Megacam_z_aperture_-_GPC1_z_aperture.png" />

In most of the case when comparing the aperture magnitudes between surveys, we observed a two peak distribution in the difference between the magnitudes (Δmag=magsurvey1magsurvey2). We have one peak around 0 for point-source objects, with a small spread. And a second peak at higher Δmag with a larger spread for extended objects; implying a different aperture correction between surveys for these objects.
That means that galaxies will not have the same aperture magnitude in different surveys.

In the griz bands, for bright sources, there is a two peaks distribution when comparing Pan-STARRS, CFHT, HSC-SSP and Legacy Survey aperture magnitues.

 
## II. Flags

II. Flags

 
### II.a. Pan-STARRS aperture and total magnitudes
Few Pan-STARRS sources have exactly the same error (of <font color='blue'>0.0010860000038519502</font>) on the __aperture and total__ magnitudes in all the grizy bands. The corresponding aperture magnitude should not be trusted for these objects.<br>
<img src="help_plots/EGS_gpc1Issues_GPC1_g_mag_aperture.png" />

II.a. Pan-STARRS aperture and total magnitudes

Few Pan-STARRS sources have exactly the same error (of 0.0010860000038519502) on the aperture and total magnitudes in all the grizy bands. The corresponding aperture magnitude should not be trusted for these objects.

xxxxxxxxxx
 
### II.d WIRCS aperture and total K-band magnitudes
Few WIRCS sources have very large magnitudes (<font color='blue'>>100</font>) which are unreliable. The corresponding aperture magnitude should not be trusted for these objects.<br>
<img src="help_plots/EGS_wircsIssues.png" />

II.d WIRCS aperture and total K-band magnitudes

Few WIRCS sources have very large magnitudes (>100) which are unreliable. The corresponding aperture magnitude should not be trusted for these objects.

 
### II.c. Outliers

II.c. Outliers

xxxxxxxxxx
By comparing magnitude in the same band between different surveys, we can see that some magnitudes are significanlty different could not be trusted. <br>
The outliers are identified to have a large weighted magnitude difference (equivalent of the $chi^2$).
$$chi^2 = \frac{(mag_{1}-mag_{2})^2}{magerr_{1}^2 + magerr_{2}^2}$$ 
<br>
We used the 75th and 25th percentile to flagged the objects 5$\sigma$ away on the large values tail of the $chi^2$ ditribution. (__NB:__ bright sources tend to have their errors underestimated with values as low as $10^{-6}$, which is unrealistic. So to avoid high $chi^2$ due to unrealistic small errors, we clip the error to get a minimum value of 0.1% (i.e. all errors smaller then $10^{-3}$ are set to $10^{-3}$).)
<br><br>
$$outliers == [chi^2 >  (75th \;percentile + 3.2\times (75th \;percentile - 25th \;percentile))]$$
<img src="help_plots/EGS_outliers_Megacam_g_total_-_GPC1_g_total.png" />

By comparing magnitude in the same band between different surveys, we can see that some magnitudes are significanlty different could not be trusted.
The outliers are identified to have a large weighted magnitude difference (equivalent of the chi2).

chi2=(mag1mag2)2magerr12+magerr22

We used the 75th and 25th percentile to flagged the objects 5σ away on the large values tail of the chi2 ditribution. (NB: bright sources tend to have their errors underestimated with values as low as 106, which is unrealistic. So to avoid high chi2 due to unrealistic small errors, we clip the error to get a minimum value of 0.1% (i.e. all errors smaller then 103 are set to 103).)

outliers==[chi2>(75thpercentile+3.2×(75thpercentile25thpercentile))]