Herschel Stripe 82 master catalogue¶

This notebook presents the merge of the various pristine catalogues to produce HELP mater catalogue on Herschel Stripe 82.

In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
This notebook was run with herschelhelp_internal version: 
04829ed (Thu Nov 2 16:57:19 2017 +0000) [with local modifications]
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))

import os
import time

from astropy import units as u
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np
from pymoc import MOC

from herschelhelp_internal.masterlist import merge_catalogues, nb_merge_dist_plot
from herschelhelp_internal.utils import coords_to_hpidx, ebv, gen_help_id, inMoc
/Users/rs548/anaconda/envs/herschelhelp_internal/lib/python3.6/site-packages/seaborn/apionly.py:6: UserWarning: As seaborn no longer sets a default style on import, the seaborn.apionly module is deprecated. It will be removed in a future version.
  warnings.warn(msg, UserWarning)
In [3]:
TMP_DIR = os.environ.get('TMP_DIR', "./data_tmp")
OUT_DIR = os.environ.get('OUT_DIR', "./data")
SUFFIX = os.environ.get('SUFFIX', time.strftime("_%Y%m%d"))

try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

I - Reading the prepared pristine catalogues¶

In [4]:
hsc = Table.read("{}/HSC-SSP.fits".format(TMP_DIR))
vhs = Table.read("{}/VISTA-VHS.fits".format(TMP_DIR))
vics82 = Table.read("{}/VICS82.fits".format(TMP_DIR))
las = Table.read("{}/UKIDSS-LAS.fits".format(TMP_DIR))
ps1 = Table.read("{}/PS1.fits".format(TMP_DIR))
shela = Table.read("{}/SHELA.fits".format(TMP_DIR))
spies= Table.read("{}/SpIES.fits".format(TMP_DIR))
decals = Table.read("{}/DECaLS.fits".format(TMP_DIR))
rcs = Table.read("{}/RCSLenS.fits".format(TMP_DIR))
sdss = Table.read("{}/SDSS-S82.fits".format(TMP_DIR))

II - Merging tables¶

We first merge the optical catalogues and then add the infrared ones: HSC, VHS, VICS82, UKIDSS-LAS, PanSTARRS, SHELA, SpIES.

At every step, we look at the distribution of the distances separating the sources from one catalogue to the other (within a maximum radius) to determine the best cross-matching radius.

HSC¶

In [5]:
master_catalogue = hsc
master_catalogue['hsc_ra'].name = 'ra'
master_catalogue['hsc_dec'].name = 'dec'
del hsc

Add VHS¶

In [6]:
nb_merge_dist_plot(
    SkyCoord(master_catalogue['ra'], master_catalogue['dec']),
    SkyCoord(vhs['vhs_ra'], vhs['vhs_dec'])
)
HELP Warning: There weren't any cross matches. The two surveys probably don't overlap.
In [7]:
# Given the graph above, we use 0.8 arc-second radius
master_catalogue = merge_catalogues(master_catalogue, vhs, "vhs_ra", "vhs_dec", radius=0.8*u.arcsec)
del vhs

Add VICS82¶

In [8]:
nb_merge_dist_plot(
    SkyCoord(master_catalogue['ra'], master_catalogue['dec']),
    SkyCoord(vics82['vics82_ra'], vics82['vics82_dec'])
)
HELP Warning: There weren't any cross matches. The two surveys probably don't overlap.
In [9]:
# Given the graph above, we use 0.8 arc-second radius
master_catalogue = merge_catalogues(master_catalogue, vics82, "vics82_ra", "vics82_dec", radius=0.8*u.arcsec)
del vics82

Add LAS¶

In [10]:
nb_merge_dist_plot(
    SkyCoord(master_catalogue['ra'], master_catalogue['dec']),
    SkyCoord(las['las_ra'], las['las_dec'])
)
In [11]:
# Given the graph above, we use 0.8 arc-second radius
master_catalogue = merge_catalogues(master_catalogue, las, "las_ra", "las_dec", radius=0.8*u.arcsec)
del las

Add PanSTARRS¶

In [12]:
nb_merge_dist_plot(
    SkyCoord(master_catalogue['ra'], master_catalogue['dec']),
    SkyCoord(ps1['ps1_ra'], ps1['ps1_dec'])
)
In [13]:
# Given the graph above, we use 0.8 arc-second radius
master_catalogue = merge_catalogues(master_catalogue, ps1, "ps1_ra", "ps1_dec", radius=0.8*u.arcsec)
del ps1

Add SDSS¶

We are waiting for a new SDSS-82 catalogue, which does not suffer from the issue of multiple sources per object due to including all exposure extractions.

In [14]:
nb_merge_dist_plot(
    SkyCoord(master_catalogue['ra'], master_catalogue['dec']),
    SkyCoord(sdss['sdss_ra'], sdss['sdss_dec'])
)
In [15]:
# Given the graph above, we use 0.8 arc-second radius
master_catalogue = merge_catalogues(master_catalogue, sdss, "sdss_ra", "sdss_dec", radius=0.8*u.arcsec)
del sdss

Add DECaLS¶

In [16]:
nb_merge_dist_plot(
    SkyCoord(master_catalogue['ra'], master_catalogue['dec']),
    SkyCoord(decals['decals_ra'], decals['decals_dec'])
)
In [17]:
# Given the graph above, we use 0.8 arc-second radius
master_catalogue = merge_catalogues(master_catalogue, decals, "decals_ra", "decals_dec", radius=0.8*u.arcsec)
del decals

Add RCSLenS¶

In [18]:
nb_merge_dist_plot(
    SkyCoord(master_catalogue['ra'], master_catalogue['dec']),
    SkyCoord(rcs['rcs_ra'], rcs['rcs_dec'])
)
In [19]:
# Given the graph above, we use 0.8 arc-second radius
master_catalogue = merge_catalogues(master_catalogue, rcs, "rcs_ra", "rcs_dec", radius=0.8*u.arcsec)
del rcs

Add SHELA¶

In [20]:
nb_merge_dist_plot(
    SkyCoord(master_catalogue['ra'], master_catalogue['dec']),
    SkyCoord(shela['shela_ra'], shela['shela_dec'])
)
HELP Warning: There weren't any cross matches. The two surveys probably don't overlap.
In [21]:
# Given the graph above, we use 1 arc-second radius
master_catalogue = merge_catalogues(master_catalogue, shela, "shela_ra", "shela_dec", radius=1.*u.arcsec)
del shela

Add SpIES¶

In [22]:
nb_merge_dist_plot(
    SkyCoord(master_catalogue['ra'], master_catalogue['dec']),
    SkyCoord(spies['spies_ra'], spies['spies_dec'])
)
In [23]:
# Given the graph above, we use 1 arc-second radius
master_catalogue = merge_catalogues(master_catalogue, spies, "spies_ra", "spies_dec", radius=1.*u.arcsec)
del spies

Cleaning¶

When we merge the catalogues, astropy masks the non-existent values (e.g. when a row comes only from a catalogue and has no counterparts in the other, the columns from the latest are masked for that row). We indicate to use NaN for masked values for floats columns, False for flag columns and -1 for ID columns.

In [24]:
for col in master_catalogue.colnames:
    if "m_" in col or "merr_" in col or "f_" in col or "ferr_" in col or "stellarity" in col:
        master_catalogue[col].fill_value = np.nan
    elif "flag" in col:
        master_catalogue[col].fill_value = 0
    elif "id" in col:
        master_catalogue[col].fill_value = -1
        
master_catalogue = master_catalogue.filled()
In [25]:
master_catalogue[:10].show_in_notebook()
Out[25]:
<Table length=10>
idxhsc_idradecm_ap_suprime_gmerr_ap_suprime_gm_suprime_gmerr_suprime_gm_ap_suprime_rmerr_ap_suprime_rm_suprime_rmerr_suprime_rm_ap_suprime_imerr_ap_suprime_im_suprime_imerr_suprime_im_ap_suprime_zmerr_ap_suprime_zm_suprime_zmerr_suprime_zm_ap_suprime_ymerr_ap_suprime_ym_suprime_ymerr_suprime_ym_ap_suprime_n816merr_ap_suprime_n816m_suprime_n816merr_suprime_n816m_ap_suprime_n921merr_ap_suprime_n921m_suprime_n921merr_suprime_n921f_ap_suprime_gferr_ap_suprime_gf_suprime_gferr_suprime_gflag_suprime_gf_ap_suprime_rferr_ap_suprime_rf_suprime_rferr_suprime_rflag_suprime_rf_ap_suprime_iferr_ap_suprime_if_suprime_iferr_suprime_iflag_suprime_if_ap_suprime_zferr_ap_suprime_zf_suprime_zferr_suprime_zflag_suprime_zf_ap_suprime_yferr_ap_suprime_yf_suprime_yferr_suprime_yflag_suprime_yf_ap_suprime_n816ferr_ap_suprime_n816f_suprime_n816ferr_suprime_n816flag_suprime_n816f_ap_suprime_n921ferr_ap_suprime_n921f_suprime_n921ferr_suprime_n921flag_suprime_n921hsc_flag_cleanedhsc_flag_gaiaflag_mergedvhs_idvhs_stellaritym_vhs_ymerr_vhs_ym_ap_vhs_ymerr_ap_vhs_ym_vhs_jmerr_vhs_jm_ap_vhs_jmerr_ap_vhs_jm_vhs_hmerr_vhs_hm_ap_vhs_hmerr_ap_vhs_hm_vhs_kmerr_vhs_km_ap_vhs_kmerr_ap_vhs_kf_vhs_yferr_vhs_yflag_vhs_yf_ap_vhs_yferr_ap_vhs_yf_vhs_jferr_vhs_jflag_vhs_jf_ap_vhs_jferr_ap_vhs_jf_vhs_hferr_vhs_hflag_vhs_hf_ap_vhs_hferr_ap_vhs_hf_vhs_kferr_vhs_kflag_vhs_kf_ap_vhs_kferr_ap_vhs_kvhs_flag_cleanedvhs_flag_gaiavics82_idvics82_stellaritym_vics82_km_ap_vics82_km_vics82_jm_ap_vics82_jmerr_vics82_kf_vics82_kferr_vics82_kflag_vics82_kmerr_ap_vics82_kf_ap_vics82_kferr_ap_vics82_kmerr_vics82_jf_vics82_jferr_vics82_jflag_vics82_jmerr_ap_vics82_jf_ap_vics82_jferr_ap_vics82_jvics82_flag_cleanedvics82_flag_gaialas_idm_ukidss_ymerr_ukidss_ym_ap_ukidss_ymerr_ap_ukidss_ym_ukidss_jmerr_ukidss_jm_ap_ukidss_jmerr_ap_ukidss_jm_ap_ukidss_hmerr_ap_ukidss_hm_ukidss_hmerr_ukidss_hm_ap_ukidss_kmerr_ap_ukidss_km_ukidss_kmerr_ukidss_klas_stellarityf_ukidss_yferr_ukidss_yflag_ukidss_yf_ap_ukidss_yferr_ap_ukidss_yf_ukidss_jferr_ukidss_jflag_ukidss_jf_ap_ukidss_jferr_ap_ukidss_jf_ap_ukidss_hferr_ap_ukidss_hf_ukidss_hferr_ukidss_hflag_ukidss_hf_ap_ukidss_kferr_ap_ukidss_kf_ukidss_kferr_ukidss_kflag_ukidss_klas_flag_cleanedlas_flag_gaiaps1_idm_ap_gpc1_gmerr_ap_gpc1_gm_gpc1_gmerr_gpc1_gm_ap_gpc1_rmerr_ap_gpc1_rm_gpc1_rmerr_gpc1_rm_ap_gpc1_imerr_ap_gpc1_im_gpc1_imerr_gpc1_im_ap_gpc1_zmerr_ap_gpc1_zm_gpc1_zmerr_gpc1_zm_ap_gpc1_ymerr_ap_gpc1_ym_gpc1_ymerr_gpc1_yf_ap_gpc1_gferr_ap_gpc1_gf_gpc1_gferr_gpc1_gflag_gpc1_gf_ap_gpc1_rferr_ap_gpc1_rf_gpc1_rferr_gpc1_rflag_gpc1_rf_ap_gpc1_iferr_ap_gpc1_if_gpc1_iferr_gpc1_iflag_gpc1_if_ap_gpc1_zferr_ap_gpc1_zf_gpc1_zferr_gpc1_zflag_gpc1_zf_ap_gpc1_yferr_ap_gpc1_yf_gpc1_yferr_gpc1_yflag_gpc1_yps1_flag_cleanedps1_flag_gaiasdss_idsdss_stellaritym_sdss_um_sdss_gm_sdss_rm_sdss_im_sdss_zmerr_sdss_umerr_sdss_gmerr_sdss_rmerr_sdss_imerr_sdss_zm_ap_sdss_um_ap_sdss_gm_ap_sdss_rm_ap_sdss_im_ap_sdss_zmerr_ap_sdss_umerr_ap_sdss_gmerr_ap_sdss_rmerr_ap_sdss_imerr_ap_sdss_zf_sdss_uferr_sdss_uflag_sdss_uf_sdss_gferr_sdss_gflag_sdss_gf_sdss_rferr_sdss_rflag_sdss_rf_sdss_iferr_sdss_iflag_sdss_if_sdss_zferr_sdss_zflag_sdss_zf_ap_sdss_uferr_ap_sdss_uf_ap_sdss_gferr_ap_sdss_gf_ap_sdss_rferr_ap_sdss_rf_ap_sdss_iferr_ap_sdss_if_ap_sdss_zferr_ap_sdss_zsdss_flag_cleanedsdss_flag_gaiadecals_idf_decam_gf_decam_rf_decam_zferr_decam_gferr_decam_rferr_decam_zf_ap_decam_gf_ap_decam_rf_ap_decam_zferr_ap_decam_gferr_ap_decam_rferr_ap_decam_zm_decam_gmerr_decam_gflag_decam_gm_decam_rmerr_decam_rflag_decam_rm_decam_zmerr_decam_zflag_decam_zm_ap_decam_gmerr_ap_decam_gm_ap_decam_rmerr_ap_decam_rm_ap_decam_zmerr_ap_decam_zdecals_stellaritydecals_flag_cleaneddecals_flag_gaiarcs_idrcs_stellaritym_rcs_gmerr_rcs_gm_rcs_rmerr_rcs_rm_rcs_imerr_rcs_im_rcs_zmerr_rcs_zm_rcs_ymerr_rcs_yf_rcs_gferr_rcs_gf_ap_rcs_gferr_ap_rcs_gm_ap_rcs_gmerr_ap_rcs_gflag_rcs_gf_rcs_rferr_rcs_rf_ap_rcs_rferr_ap_rcs_rm_ap_rcs_rmerr_ap_rcs_rflag_rcs_rf_rcs_iferr_rcs_if_ap_rcs_iferr_ap_rcs_im_ap_rcs_imerr_ap_rcs_iflag_rcs_if_rcs_zferr_rcs_zf_ap_rcs_zferr_ap_rcs_zm_ap_rcs_zmerr_ap_rcs_zflag_rcs_zf_rcs_yferr_rcs_yf_ap_rcs_yferr_ap_rcs_ym_ap_rcs_ymerr_ap_rcs_yflag_rcs_yrcs_flag_cleanedrcs_flag_gaiashela_intidf_shela_irac1ferr_shela_irac1f_ap_shela_irac1ferr_ap_shela_irac1f_shela_irac2ferr_shela_irac2f_ap_shela_irac2ferr_ap_shela_irac2m_shela_irac1merr_shela_irac1flag_shela_irac1m_ap_shela_irac1merr_ap_shela_irac1m_shela_irac2merr_shela_irac2flag_shela_irac2m_ap_shela_irac2merr_ap_shela_irac2shela_flag_cleanedshela_flag_gaiaspies_intidf_ap_spies_irac1ferr_ap_spies_irac1f_spies_irac1ferr_spies_irac1spies_stellarity_irac1f_ap_spies_irac2ferr_ap_spies_irac2f_spies_irac2ferr_spies_irac2spies_stellarity_irac2m_ap_spies_irac1merr_ap_spies_irac1m_spies_irac1merr_spies_irac1flag_spies_irac1m_ap_spies_irac2merr_ap_spies_irac2m_spies_irac2merr_spies_irac2flag_spies_irac2spies_flag_cleanedspies_flag_gaia
degdeg
042697064458515903353.7500524360.18843248065819.20880.00076160119.38420.00064052718.80570.00072267518.88460.00058783218.69020.00072674718.67340.00049034118.64030.0009998718.73830.00071541218.63430.0018041518.61240.0012006618.65270.0014992618.64290.0011264418.65970.0013920118.65210.00099159575.24530.052781564.02050.0377687False109.0750.072601101.4290.0549153False121.320.0812064123.2060.0556424False127.0260.11698116.0580.0764728False127.7290.212245130.3350.144131False125.5770.173405126.720.131471False124.7690.159965125.6460.114751FalseFalse3False-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0
142697064458508303353.652565850.1161450501121.72160.0034550121.45070.0040667920.55120.0018722120.29970.0022255720.02290.001340419.75510.001641119.74370.0022193419.46260.0026826519.58030.0033218219.28370.0040917819.92730.002971919.67790.0035274919.66730.0025234219.39820.003085237.436020.02366279.544130.035749False21.85430.037684927.54960.056472False35.55040.043888845.49430.0687648False45.97630.093979659.55910.147159False53.44110.16350470.23060.264676False38.82060.1062648.84570.158697False49.32470.11463863.20190.179594FalseFalse0False-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0
242697060163548288353.6761145470.052636049136320.66860.0020112419.24080.0020056220.12980.001756218.71840.0018298919.74150.0013417618.37660.001593319.6070.0022038818.18260.0025435219.38980.0034901517.96250.004216619.75830.0037475118.34850.0040840119.57350.0032165718.18110.0037781219.61360.036332773.06310.134965False32.2170.0521118118.2040.199221False46.06780.0569311161.9460.237654False52.14150.105839193.6190.453585False63.69380.204747237.1390.920962False45.35980.156563166.1810.625091False53.77780.159321193.8880.674687FalseFalse0False-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0
342697068753489548353.6431851020.45944719921121.44320.0029085921.16380.003677320.5160.0017662920.26440.0023179220.10490.0016128719.86350.0021807219.7390.0024353419.47890.0032371619.72180.004162719.42170.0056187520.01060.003755519.77070.0050073219.83710.0031947219.59020.004330249.609520.02574312.43010.0420996False22.57420.03672428.45960.060758False32.96530.048970241.17180.0826942False46.17340.10356858.67210.174933False46.90980.17985261.84970.320076False35.95640.12437144.84360.206815False42.18410.12412552.95650.211206FalseFalse0False-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0
442697073048455227353.689474590.62128561956120.7210.0021243219.7540.0032166220.26990.0019454519.20780.0029275119.90680.0016164418.76350.0025067619.82430.003071518.63610.0044665719.67610.0043087218.39050.0063658319.9220.0036768418.7570.0053178519.8310.0036502118.66610.0055881318.68980.036567945.54020.134918False28.31630.050737875.31250.203068False39.56050.0588975113.3980.261814False42.68670.120759127.5170.524588False48.92660.194164159.8780.937387False39.0110.132111114.0740.558728False42.42240.142623124.0360.638394FalseFalse0False-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0
542697210487404068353.5162841250.5322078481621.32620.0026987321.40280.0023961920.0060.0014283620.01790.0012857719.65770.0017702819.63040.0017322118.84760.0011623518.87250.0010242518.72790.0018475318.6750.0015719319.04950.0022514419.00660.0020911518.83170.001565418.81090.0014105310.7030.02660369.974090.0220126False36.10740.047501635.71540.0422955False49.76490.081141151.03340.08142False104.9480.112353102.5690.09676False117.1770.199394123.0240.178114False87.14030.18069990.65090.174596False106.4930.15354108.5490.14102FalseFalse3True-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0
642697068753484122353.6512638650.38848466302521.35340.0026846520.98510.003173120.67650.0020689820.35770.0026828120.32810.0017862320.02740.0024688920.32520.003798520.00760.0051954120.17760.0058779619.87150.0083796619.83550.0030149319.52220.0037545220.23440.0039422119.94010.0055728610.43810.025809914.65390.0428266False19.47090.037103726.11570.0645309False26.83930.044155535.40160.0805007False26.91020.094146536.05290.172519False30.82990.16690740.86990.315432False42.24870.11731956.3790.194961False29.25780.10623238.36610.196925FalseFalse0False-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0
742696922724569878353.9232881430.079605060212526.21720.58185827.6940.89400826.62191.0832326.73290.47317919.16650.0018649219.1090.00111996nannannannannannannannan25.66180.55126127.67451.3734925.48940.46785727.50221.165680.118340.06341960.03036770.0250051False0.08151270.08132450.07359450.0320735False78.23590.13438282.48870.0850887FalsenannannannanFalsenannannannanFalse0.1973780.1002150.03091630.03911False0.2313390.09968670.03623570.0389039FalseFalse0False-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0
842697197602501563353.5171786040.016699036528523.5020.016997123.70540.011792322.15090.0073600822.23720.0048211120.48860.0020166520.4850.001276119.79660.0024487619.89650.0016266619.51220.0032111919.50210.0020092920.01510.0035770619.9960.0025032419.64860.0027577419.65630.001886781.442770.02258641.196330.0129935False5.007760.0339474.624930.0205366False23.15060.04323.22740.0272998False43.78960.098762839.93770.0598351False56.90320.16829857.4310.106283False35.80750.11797136.44190.0840192False50.18370.12746549.83090.0865957FalseFalse0False-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0
941628364926116242353.768172259-0.16895823233219.31180.0016986219.59140.0016282519.27050.0018890419.34180.0015118419.26750.0020793419.23910.0014163419.48590.0042338519.53580.0026758719.31640.0065126519.31550.0040756338.250347350.0infnannannaninfnan68.43250.10706252.89880.0793307False71.08870.12368566.56930.0926948False71.28440.1365273.17760.0954598False58.29840.22733655.67840.137223False68.14830.40877968.20350.256021False1.81923e-060.07933820.0nanFalsenannan0.0nanFalseFalse0False-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannanFalsenannannannannannanFalsenannannanFalse0-1nannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalsenannannannanFalseFalse0-1nannannannannannannannannannannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannanFalsenannanFalsenannannannannannannannannannanFalse0-1nannannannannannannannannannannannannannanFalsenannanFalsenannanFalsenannannannannannannanFalse0-1nannannannannannannannannannannannannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalsenannannannannannanFalseFalse0-1nannannannannannannannannannanFalsenannannannanFalsenannanFalse0-1nannannannannannannannannannannannannannanFalsenannannannanFalseFalse0

III - Merging flags and stellarity¶

Each pristine catalogue contains a flag indicating if the source was associated to a another nearby source that was removed during the cleaning process. We merge these flags in a single one.

In [26]:
flag_cleaned_columns = [column for column in master_catalogue.colnames
                        if 'flag_cleaned' in column]

flag_column = np.zeros(len(master_catalogue), dtype=bool)
for column in flag_cleaned_columns:
    flag_column |= master_catalogue[column]
    
master_catalogue.add_column(Column(data=flag_column, name="flag_cleaned"))
master_catalogue.remove_columns(flag_cleaned_columns)

Each pristine catalogue contains a flag indicating the probability of a source being a Gaia object (0: not a Gaia object, 1: possibly, 2: probably, 3: definitely). We merge these flags taking the highest value.

In [27]:
flag_gaia_columns = [column for column in master_catalogue.colnames
                     if 'flag_gaia' in column]

master_catalogue.add_column(Column(
    data=np.max([master_catalogue[column] for column in flag_gaia_columns], axis=0),
    name="flag_gaia"
))
master_catalogue.remove_columns(flag_gaia_columns)

Each prisitine catalogue may contain one or several stellarity columns indicating the probability (0 to 1) of each source being a star. We merge these columns taking the highest value.

In [28]:
stellarity_columns = [column for column in master_catalogue.colnames
                      if 'stellarity' in column]

master_catalogue.add_column(Column(
    data=np.nanmax([master_catalogue[column] for column in stellarity_columns], axis=0),
    name="stellarity"
))
master_catalogue.remove_columns(stellarity_columns)
/Users/rs548/anaconda/envs/herschelhelp_internal/lib/python3.6/site-packages/ipykernel/__main__.py:5: RuntimeWarning: All-NaN axis encountered

IV - Adding E(B-V) column¶

In [30]:
master_catalogue.add_column(
    ebv(master_catalogue['ra'], master_catalogue['dec'])
)

V - Adding HELP unique identifiers and field columns¶

In [31]:
master_catalogue.add_column(Column(gen_help_id(master_catalogue['ra'], master_catalogue['dec']),
                                   name="help_id"))
master_catalogue.add_column(Column(np.full(len(master_catalogue), "Herschel-Stripe-82", dtype='<U18'),
                                   name="field"))
In [32]:
# Check that the HELP Ids are unique
if len(master_catalogue) != len(np.unique(master_catalogue['help_id'])):
    print("The HELP IDs are not unique!!!")
else:
    print("OK!")
OK!

VI - Choosing between multiple values for the same filter¶

Both SHELA and SpIES provide IRAC1 and IRAC2 fluxes. SpIES seems to go deeper and neither apear to suffer from the bright drop off that affects both SERVS and SWIRE.

In [33]:
seip = Table.read("../../dmu0/dmu0_SEIP/data/SEIP_Herschel-Stripe-82.fits")
seip_coords = SkyCoord(seip['ra'], seip['dec'])
idx, d2d, _ = seip_coords.match_to_catalog_sky(SkyCoord(master_catalogue['ra'], master_catalogue['dec']))
mask = d2d <= 2 * u.arcsec
In [34]:
# servs -> shela and swire -> spies
fig, ax = plt.subplots()
ax.scatter(seip['i1_f_ap1'][mask], master_catalogue[idx[mask]]['f_ap_shela_irac1'], label="SHELA", s=2.)
ax.scatter(seip['i1_f_ap1'][mask], master_catalogue[idx[mask]]['f_ap_spies_irac1'], label="SpIES", s=2.)
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlabel("SEIP flux [μJy]")
ax.set_ylabel("SHELA/SpIES flux [μJy]")
ax.set_title("IRAC 1")
ax.legend()
ax.axvline(2000, color="black", linestyle="--", linewidth=1.)
ax.plot(seip['i1_f_ap1'][mask], seip['i1_f_ap1'][mask], linewidth=.1, color="black", alpha=.5);
In [35]:
fig, ax = plt.subplots()
ax.scatter(seip['i2_f_ap1'][mask], master_catalogue[idx[mask]]['f_ap_shela_irac2'], label="SHELA", s=2.)
ax.scatter(seip['i2_f_ap1'][mask], master_catalogue[idx[mask]]['f_ap_spies_irac2'], label="SpIES", s=2.)
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlabel("SEIP flux [μJy]")
ax.set_ylabel("SHELA/SpIES flux [μJy]")
ax.set_title("IRAC 2")
ax.legend()
ax.axvline(2000, color="black", linestyle="--", linewidth=1.)

ax.plot(seip['i1_f_ap2'][mask], seip['i1_f_ap2'][mask], linewidth=.1, color="black", alpha=.5);

When both SHELA and SpIES fluxes are provided, we use the SpIES flux.

We create a table indicating for each source the origin on the IRAC1 and IRAC2 fluxes that will be saved separately.

In [36]:
irac_origin = Table()
irac_origin.add_column(master_catalogue['help_id'])
In [37]:
# IRAC1 aperture flux and magnitudes
has_shela = ~np.isnan(master_catalogue['f_ap_shela_irac1'])
has_spies = ~np.isnan(master_catalogue['f_ap_spies_irac1'])
has_both = has_shela & has_spies

print("{} sources with SHELA flux".format(np.sum(has_shela)))
print("{} sources with SpIES flux".format(np.sum(has_spies)))
print("{} sources with SHELA and SpIES flux".format(np.sum(has_both)))

use_shela = has_shela
use_spies = (has_spies & ~has_shela)

print("{} sources for which we use SHELA".format(np.sum(use_shela)))
print("{} sources for which we use SpIES".format(np.sum(use_spies)))

f_ap_irac = np.full(len(master_catalogue), np.nan)
f_ap_irac[use_shela] = master_catalogue['f_ap_shela_irac1'][use_shela]
f_ap_irac[use_spies] = master_catalogue['f_ap_spies_irac1'][use_spies]

ferr_ap_irac = np.full(len(master_catalogue), np.nan)
ferr_ap_irac[use_shela] = master_catalogue['ferr_ap_shela_irac1'][use_shela]
ferr_ap_irac[use_spies] = master_catalogue['ferr_ap_spies_irac1'][use_spies]

m_ap_irac = np.full(len(master_catalogue), np.nan)
m_ap_irac[use_shela] = master_catalogue['m_ap_shela_irac1'][use_shela]
m_ap_irac[use_spies] = master_catalogue['m_ap_spies_irac1'][use_spies]

merr_ap_irac = np.full(len(master_catalogue), np.nan)
merr_ap_irac[use_shela] = master_catalogue['merr_ap_shela_irac1'][use_shela]
merr_ap_irac[use_spies] = master_catalogue['merr_ap_spies_irac1'][use_spies]

master_catalogue.add_column(Column(data=f_ap_irac, name="f_ap_irac_i1"))
master_catalogue.add_column(Column(data=ferr_ap_irac, name="ferr_ap_irac_i1"))
master_catalogue.add_column(Column(data=m_ap_irac, name="m_ap_irac_i1"))
master_catalogue.add_column(Column(data=merr_ap_irac, name="merr_ap_irac_i1"))

master_catalogue.remove_columns(['f_ap_shela_irac1', 'f_ap_spies_irac1', 
                                 'ferr_ap_shela_irac1', 'ferr_ap_spies_irac1', 
                                 'm_ap_shela_irac1', 'm_ap_spies_irac1',
                                 'merr_ap_shela_irac1', 'merr_ap_spies_irac1'])

origin = np.full(len(master_catalogue), '     ', dtype='<U5')
origin[use_shela] = "SHELA"
origin[use_spies] = "SpIES"
irac_origin.add_column(Column(data=origin, name="IRAC1_ap"))
67868 sources with SHELA flux
62528 sources with SpIES flux
21709 sources with SHELA and SpIES flux
67868 sources for which we use SHELA
40819 sources for which we use SpIES
In [38]:
# IRAC1 total flux and magnitudes
has_shela = ~np.isnan(master_catalogue['f_shela_irac1'])
has_spies = ~np.isnan(master_catalogue['f_spies_irac1'])
has_both = has_shela & has_spies

print("{} sources with SHELA total flux".format(np.sum(has_shela)))
print("{} sources with SpIES total flux".format(np.sum(has_spies)))
print("{} sources with SHELA and SpIES total flux".format(np.sum(has_both)))

use_shela = has_shela
use_spies = (has_spies & ~has_shela)

print("{} sources for which we use SHELA".format(np.sum(use_shela)))
print("{} sources for which we use SpIES".format(np.sum(use_spies)))

f_ap_irac = np.full(len(master_catalogue), np.nan)
f_ap_irac[use_shela] = master_catalogue['f_shela_irac1'][use_shela]
f_ap_irac[use_spies] = master_catalogue['f_spies_irac1'][use_spies]

ferr_ap_irac = np.full(len(master_catalogue), np.nan)
ferr_ap_irac[use_shela] = master_catalogue['ferr_shela_irac1'][use_shela]
ferr_ap_irac[use_spies] = master_catalogue['ferr_spies_irac1'][use_spies]

flag_irac = np.full(len(master_catalogue), False, dtype=bool)
flag_irac[use_shela] = master_catalogue['flag_shela_irac1'][use_shela]
flag_irac[use_spies] = master_catalogue['flag_spies_irac1'][use_spies]

m_ap_irac = np.full(len(master_catalogue), np.nan)
m_ap_irac[use_shela] = master_catalogue['m_shela_irac1'][use_shela]
m_ap_irac[use_spies] = master_catalogue['m_spies_irac1'][use_spies]

merr_ap_irac = np.full(len(master_catalogue), np.nan)
merr_ap_irac[use_shela] = master_catalogue['merr_shela_irac1'][use_shela]
merr_ap_irac[use_spies] = master_catalogue['merr_spies_irac1'][use_spies]

master_catalogue.add_column(Column(data=f_ap_irac, name="f_irac_i1"))
master_catalogue.add_column(Column(data=ferr_ap_irac, name="ferr_irac_i1"))
master_catalogue.add_column(Column(data=m_ap_irac, name="m_irac_i1"))
master_catalogue.add_column(Column(data=merr_ap_irac, name="merr_irac_i1"))
master_catalogue.add_column(Column(data=flag_irac, name="flag_irac_i1"))

master_catalogue.remove_columns(['f_shela_irac1', 'f_spies_irac1', 
                                 'ferr_shela_irac1', 'ferr_spies_irac1', 
                                 'm_shela_irac1', 'm_spies_irac1',
                                 'merr_shela_irac1', 'merr_spies_irac1',
                                'flag_shela_irac1', 'flag_spies_irac1'])

origin = np.full(len(master_catalogue), '     ', dtype='<U5')
origin[use_shela] = "SHELA"
origin[use_spies] = "SpIES"
irac_origin.add_column(Column(data=origin, name="IRAC1_total"))
67868 sources with SHELA total flux
62528 sources with SpIES total flux
21709 sources with SHELA and SpIES total flux
67868 sources for which we use SHELA
40819 sources for which we use SpIES
In [39]:
# IRAC2 aperture flux and magnitudes
has_shela = ~np.isnan(master_catalogue['f_ap_shela_irac2'])
has_spies = ~np.isnan(master_catalogue['f_ap_spies_irac2'])
has_both = has_shela & has_spies

print("{} sources with SHELA flux".format(np.sum(has_shela)))
print("{} sources with SpIES flux".format(np.sum(has_spies)))
print("{} sources with SHELA and SpIES flux".format(np.sum(has_both)))

use_shela = has_shela
use_spies = (has_spies & ~has_shela)

print("{} sources for which we use SHELA".format(np.sum(use_shela)))
print("{} sources for which we use SpIES".format(np.sum(use_spies)))

f_ap_irac = np.full(len(master_catalogue), np.nan)
f_ap_irac[use_shela] = master_catalogue['f_ap_shela_irac2'][use_shela]
f_ap_irac[use_spies] = master_catalogue['f_ap_spies_irac2'][use_spies]

ferr_ap_irac = np.full(len(master_catalogue), np.nan)
ferr_ap_irac[use_shela] = master_catalogue['ferr_ap_shela_irac2'][use_shela]
ferr_ap_irac[use_spies] = master_catalogue['ferr_ap_spies_irac2'][use_spies]

m_ap_irac = np.full(len(master_catalogue), np.nan)
m_ap_irac[use_shela] = master_catalogue['m_ap_shela_irac2'][use_shela]
m_ap_irac[use_spies] = master_catalogue['m_ap_spies_irac2'][use_spies]

merr_ap_irac = np.full(len(master_catalogue), np.nan)
merr_ap_irac[use_shela] = master_catalogue['merr_ap_shela_irac2'][use_shela]
merr_ap_irac[use_spies] = master_catalogue['merr_ap_spies_irac2'][use_spies]

master_catalogue.add_column(Column(data=f_ap_irac, name="f_ap_irac_i2"))
master_catalogue.add_column(Column(data=ferr_ap_irac, name="ferr_ap_irac_i2"))
master_catalogue.add_column(Column(data=m_ap_irac, name="m_ap_irac_i2"))
master_catalogue.add_column(Column(data=merr_ap_irac, name="merr_ap_irac_i2"))

master_catalogue.remove_columns(['f_ap_shela_irac2', 'f_ap_spies_irac2', 
                                 'ferr_ap_shela_irac2', 'ferr_ap_spies_irac2', 
                                 'm_ap_shela_irac2', 'm_ap_spies_irac2',
                                 'merr_ap_shela_irac2', 'merr_ap_spies_irac2'])

origin = np.full(len(master_catalogue), '     ', dtype='<U5')
origin[use_shela] = "SHELA"
origin[use_spies] = "SpIES"
irac_origin.add_column(Column(data=origin, name="IRAC2_ap"))
67868 sources with SHELA flux
62528 sources with SpIES flux
21709 sources with SHELA and SpIES flux
67868 sources for which we use SHELA
40819 sources for which we use SpIES
In [40]:
# IRAC2 total flux and magnitudes
has_shela = ~np.isnan(master_catalogue['f_shela_irac2'])
has_spies = ~np.isnan(master_catalogue['f_spies_irac2'])
has_both = has_shela & has_spies

print("{} sources with SHELA total flux".format(np.sum(has_shela)))
print("{} sources with SpIES total flux".format(np.sum(has_spies)))
print("{} sources with SHELA and SpIES total flux".format(np.sum(has_both)))

use_shela = has_shela
use_spies = (has_spies & ~has_shela)

print("{} sources for which we use SHELA".format(np.sum(use_shela)))
print("{} sources for which we use SpIES".format(np.sum(use_spies)))

f_ap_irac = np.full(len(master_catalogue), np.nan)
f_ap_irac[use_shela] = master_catalogue['f_shela_irac2'][use_shela]
f_ap_irac[use_spies] = master_catalogue['f_spies_irac2'][use_spies]

ferr_ap_irac = np.full(len(master_catalogue), np.nan)
ferr_ap_irac[use_shela] = master_catalogue['ferr_shela_irac2'][use_shela]
ferr_ap_irac[use_spies] = master_catalogue['ferr_spies_irac2'][use_spies]

flag_irac = np.full(len(master_catalogue), False, dtype=bool)
flag_irac[use_shela] = master_catalogue['flag_shela_irac2'][use_shela]
flag_irac[use_spies] = master_catalogue['flag_spies_irac2'][use_spies]

m_ap_irac = np.full(len(master_catalogue), np.nan)
m_ap_irac[use_shela] = master_catalogue['m_shela_irac2'][use_shela]
m_ap_irac[use_spies] = master_catalogue['m_spies_irac2'][use_spies]

merr_ap_irac = np.full(len(master_catalogue), np.nan)
merr_ap_irac[use_shela] = master_catalogue['merr_shela_irac2'][use_shela]
merr_ap_irac[use_spies] = master_catalogue['merr_spies_irac2'][use_spies]

master_catalogue.add_column(Column(data=f_ap_irac, name="f_irac_i2"))
master_catalogue.add_column(Column(data=ferr_ap_irac, name="ferr_irac_i2"))
master_catalogue.add_column(Column(data=m_ap_irac, name="m_irac_i2"))
master_catalogue.add_column(Column(data=merr_ap_irac, name="merr_irac_i2"))
master_catalogue.add_column(Column(data=flag_irac, name="flag_irac_i2"))

master_catalogue.remove_columns(['f_shela_irac2', 'f_spies_irac2', 
                                 'ferr_shela_irac2', 'ferr_spies_irac2', 
                                 'm_shela_irac2', 'm_spies_irac2',
                                 'merr_shela_irac2', 'merr_spies_irac2',
                                'flag_shela_irac2', 'flag_spies_irac2'])

origin = np.full(len(master_catalogue), '     ', dtype='<U5')
origin[use_shela] = "SHELA"
origin[use_spies] = "SpIES"
irac_origin.add_column(Column(data=origin, name="IRAC2_total"))
67868 sources with SHELA total flux
62528 sources with SpIES total flux
21709 sources with SHELA and SpIES total flux
67868 sources for which we use SHELA
40819 sources for which we use SpIES
In [41]:
irac_origin.write("{}/herschel-stripe-82_irac_fluxes_origins{}.fits".format(OUT_DIR, SUFFIX), overwrite = True)

VII.a Wavelength domain coverage¶

We add a binary flag_optnir_obs indicating that a source was observed in a given wavelength domain:

  • 1 for observation in optical;
  • 2 for observation in near-infrared;
  • 4 for observation in mid-infrared (IRAC).

It's an integer binary flag, so a source observed both in optical and near-infrared by not in mid-infrared would have this flag at 1 + 2 = 3.

Note 1: The observation flag is based on the creation of multi-order coverage maps from the catalogues, this may not be accurate, especially on the edges of the coverage.

Note 2: Being on the observation coverage does not mean having fluxes in that wavelength domain. For sources observed in one domain but having no flux in it, one must take into consideration de different depths in the catalogue we are using.

In [42]:
hsc_moc = MOC(filename="../../dmu0/dmu0_HSC/data/HSC-PDR1_deep_Herschel-Stripe-82_MOC.fits")
vhs_moc = MOC(filename="../../dmu0/dmu0_VISTA-VHS/data/VHS_Herschel-Stripe-82_MOC.fits")
vics82_moc = MOC(filename="../../dmu0/dmu0_VICS82/data/VICS82_FULL_SDSS_FEB2017_K22_HELP-coverage_intIDs_MOC.fits")
las_moc = MOC(filename="../../dmu0/dmu0_UKIDSS-LAS/data/UKIDSS-LAS_Herschel-Stripe-82_MOC.fits")
ps1_moc = MOC(filename="../../dmu0/dmu0_PanSTARRS1-3SS/data/PanSTARRS1-3SS_Herschel-Stripe-82_MOC.fits")
shela_moc = MOC(filename="../../dmu0/dmu0_SHELA/data/shela_irac_v1.3_flux_cat_MOC.fits")
spies_moc = MOC(filename="../../dmu0/dmu0_SpIES/data/SpIES_ch1andch2_HELP-coverage_MOC.fits")
decals_moc = MOC(filename="../../dmu0/dmu0_DECaLS/data/DECaLS_Herschel-Stripe-82_MOC.fits")
rcs_moc = MOC(filename="../../dmu0/dmu0_RCSLenS/data/RCSLenS_Herschel-Stripe-82_MOC.fits")
In [43]:
was_observed_optical = inMoc(
    master_catalogue['ra'], master_catalogue['dec'],
    hsc_moc + ps1_moc + decals_moc + rcs_moc) 

was_observed_nir = inMoc(
    master_catalogue['ra'], master_catalogue['dec'],
    las_moc + vics82_moc + vhs_moc
)

was_observed_mir = inMoc(
    master_catalogue['ra'], master_catalogue['dec'],
    shela_moc + spies_moc
)
In [44]:
master_catalogue.add_column(
    Column(
        1 * was_observed_optical + 2 * was_observed_nir + 4 * was_observed_mir,
        name="flag_optnir_obs")
)

VII.b Wavelength domain detection¶

We add a binary flag_optnir_det indicating that a source was detected in a given wavelength domain:

  • 1 for detection in optical;
  • 2 for detection in near-infrared;
  • 4 for detection in mid-infrared (IRAC).

It's an integer binary flag, so a source detected both in optical and near-infrared by not in mid-infrared would have this flag at 1 + 2 = 3.

Note 1: We use the total flux columns to know if the source has flux, in some catalogues, we may have aperture flux and no total flux.

To get rid of artefacts (chip edges, star flares, etc.) we consider that a source is detected in one wavelength domain when it has a flux value in at least two bands. That means that good sources will be excluded from this flag when they are on the coverage of only one band.

In [45]:
#TODO check detection bands
nb_optical_flux = (
    1 * ~np.isnan(master_catalogue['f_suprime_g']) +
    1 * ~np.isnan(master_catalogue['f_suprime_r']) +
    1 * ~np.isnan(master_catalogue['f_suprime_i']) +
    1 * ~np.isnan(master_catalogue['f_suprime_z']) +
    1 * ~np.isnan(master_catalogue['f_suprime_y']) +
    1 * ~np.isnan(master_catalogue['f_suprime_n921']) +
    1 * ~np.isnan(master_catalogue['f_suprime_n816']) +
    1 * ~np.isnan(master_catalogue['f_gpc1_g']) +
    1 * ~np.isnan(master_catalogue['f_gpc1_r']) +
    1 * ~np.isnan(master_catalogue['f_gpc1_i']) +
    1 * ~np.isnan(master_catalogue['f_gpc1_z']) +
    1 * ~np.isnan(master_catalogue['f_gpc1_y']) +
    1 * ~np.isnan(master_catalogue['f_decam_g']) +
    1 * ~np.isnan(master_catalogue['f_decam_r']) +
    1 * ~np.isnan(master_catalogue['f_decam_z']) 
)

nb_nir_flux = (
    1 * ~np.isnan(master_catalogue['f_ukidss_y']) +
    1 * ~np.isnan(master_catalogue['f_ukidss_j']) +
    1 * ~np.isnan(master_catalogue['f_ukidss_h']) +
    1 * ~np.isnan(master_catalogue['f_ukidss_k']) +
    1 * ~np.isnan(master_catalogue['f_vics82_j']) +
    1 * ~np.isnan(master_catalogue['f_vics82_k']) 
)

nb_mir_flux = (
    1 * ~np.isnan(master_catalogue['f_irac_i1']) +
    1 * ~np.isnan(master_catalogue['f_irac_i2']) 
)
In [46]:
has_optical_flux = nb_optical_flux >= 2
has_nir_flux = nb_nir_flux >= 2
has_mir_flux = nb_mir_flux >= 2

master_catalogue.add_column(
    Column(
        1 * has_optical_flux + 2 * has_nir_flux + 4 * has_mir_flux,
        name="flag_optnir_det")
)

VIII - Cross-identification table¶

We are producing a table associating to each HELP identifier, the identifiers of the sources in the pristine catalogue. This can be used to easily get additional information from them.

In [47]:
#TODO: ADD SDSS normal ids
In [48]:
id_names = []
for col in master_catalogue.colnames:
    if '_id' in col:
        id_names += [col]
    if '_intid' in col:
        id_names += [col]
        
print(id_names)
['hsc_id', 'vhs_id', 'vics82_id', 'las_id', 'ps1_id', 'sdss_id', 'decals_id', 'rcs_id', 'shela_intid', 'spies_intid', 'help_id']
In [49]:
master_catalogue[id_names].write(
    "{}/master_list_cross_ident_herschel-stripe-82{}.fits".format(OUT_DIR, SUFFIX), overwrite=True)
id_names.remove('help_id')
master_catalogue.remove_columns(id_names)

IX - Adding HEALPix index¶

We are adding a column with a HEALPix index at order 13 associated with each source.

In [50]:
master_catalogue.add_column(Column(
    data=coords_to_hpidx(master_catalogue['ra'], master_catalogue['dec'], order=13),
    name="hp_idx"
))

IX - Saving the catalogue¶

In [51]:
columns = ["help_id", "field", "ra", "dec", "hp_idx"]

bands = [column[5:] for column in master_catalogue.colnames if 'f_ap' in column]
for band in bands:
    columns += ["f_ap_{}".format(band), "ferr_ap_{}".format(band),
                "m_ap_{}".format(band), "merr_ap_{}".format(band),
                "f_{}".format(band), "ferr_{}".format(band),
                "m_{}".format(band), "merr_{}".format(band),
                "flag_{}".format(band)]    
    
columns += ["stellarity", "flag_cleaned", "flag_merged", "flag_gaia", "flag_optnir_obs", "flag_optnir_det", "ebv"]
In [52]:
# We check for columns in the master catalogue that we will not save to disk.
print("Missing columns: {}".format(set(master_catalogue.colnames) - set(columns)))
Missing columns: set()
In [53]:
master_catalogue[columns].write("{}/master_catalogue_herschel-stripe-82{}.fits".format(OUT_DIR, SUFFIX), overwrite=True)