HELP - Bootes Master List Creation¶

This notebook presents the creation of the HELP master list on the Bootes field. This field was originally ingested into HeDAM based on a masterlist produced by Eduardo Gonzales-Solares. Ken Duncan subsequently retrieved Michael Brown's precompiled masterlist for the Bootes field and therefore this notebook only contains some simple naming changes and astrometry correction. The original catalogue is described in DMU_0.

In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))

This notebook was run with herschelhelp_internal version: 
33f5ec7 (Wed Dec 6 16:56:17 2017 +0000)
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))

from collections import OrderedDict, Counter
import os
import time

from astropy import units as u
from astropy.coordinates import SkyCoord
from astropy.table import Column, hstack, Table, vstack
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, remove_duplicates, nb_merge_dist_plot
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux, gen_help_id
In [3]:
TMP_DIR = os.environ.get('TMP_DIR', "./data_tmp")
OUT_DIR = os.environ.get('OUT_DIR', "./data")
SUFFIX = os.environ.get('SUFFIX', time.strftime("_%Y%m%d"))
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "bootes_ra"
DEC_COL = "botes_dec"

I - Column selection¶

In [4]:
#imported_columns = OrderedDict({
#        "objID": "ps1_id",
#        "raMean": "ps1_ra",
#        "decMean": "ps1_dec",
#
#    })


i_catalogue = Table.read("../../dmu0/dmu0_Bootes_Brown/data/Bootes_merged_Icorr_2014a_all_ap2_07112017.fits") #[list(imported_columns)]
irac_i2_catalogue = Table.read("../../dmu0/dmu0_Bootes_Brown/data/bootes_merged_ch2corr_2014a_all_ap2_07112017.fits")
#for column in imported_columns:
#    catalogue[column].name = imported_columns[column]

#epoch = 2012

# Clean table metadata
#catalogue.meta = None
In [5]:
i_catalogue['ALPHA_J2000'].name = "ra"
i_catalogue['DELTA_J2000'].name = "dec"
i_catalogue['ra'].unit = u.deg
i_catalogue['dec'].unit = u.deg
In [6]:
i_catalogue[:10].show_in_notebook()
Out[6]:
<Table length=10>
idxidradecFLAG_DEEPCLASS_STARIMAFLAGSSEGFLAGSm_ap_lbc_umerr_ap_lbc_uf_ap_lbc_uferr_ap_lbc_um_ap_mosaic_bwmerr_ap_mosaic_bwf_ap_mosaic_bwferr_ap_mosaic_bwm_ap_mosaic_rmerr_ap_mosaic_rf_ap_mosaic_rferr_ap_mosaic_rm_ap_mosaic_imerr_ap_mosaic_if_ap_mosaic_iferr_ap_mosaic_im_ap_prime90_zmerr_ap_prime90_zf_ap_prime90_zferr_ap_prime90_zm_ap_suprime_zmerr_ap_suprime_zf_ap_suprime_zferr_ap_suprime_zm_ap_lbc_ymerr_ap_lbc_yf_ap_lbc_yferr_ap_lbc_ym_ap_newfirm_jmerr_ap_newfirm_jf_ap_newfirm_jferr_ap_newfirm_jm_ap_newfirm_hmerr_ap_newfirm_hf_ap_newfirm_hferr_ap_newfirm_hm_ap_onis_kmerr_ap_onis_kf_ap_onis_kferr_ap_onis_km_ap_newfirm_kmerr_ap_newfirm_kf_ap_newfirm_kferr_ap_newfirm_km_ap_irac_ch1merr_ap_irac_ch1f_ap_irac_ch1ferr_ap_irac_ch1m_ap_irac_ch2merr_ap_irac_ch2f_ap_irac_ch2ferr_ap_irac_ch2m_ap_irac_ch3merr_ap_irac_ch3f_ap_irac_ch3ferr_ap_irac_ch3m_ap_irac_ch4merr_ap_irac_ch4f_ap_irac_ch4ferr_ap_irac_ch4
degdeguJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJyuJy
01216.47386332.282596300.641025.3294-99.00.07659110.0957339-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.68710.17731.216580.198666-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.544-99.00.1149630.63656123.1026-99.00.1205750.98189121.0877-99.03.420724.9558920.3206-99.08.530479.24751
12216.617609832.282743300.6612125.2766-99.00.06741220.107004-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.5630.18011.363890.22624-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.428-99.01.857711.01104-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.7807-99.00.5608931.1213323.5937-99.0-0.8412741.0836221.172-99.0-1.248876.7929421.0885-99.0-4.716429.01942
23216.622926132.282758700.951027.024-99.0-0.1724960.11439-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.60790.23581.308640.284211-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.40420.31293.965871.14293-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.0248-99.00.05826381.0904123.3361-99.0-0.4502961.0656321.9414-99.0-6.45836.2659520.8749-99.0-1.740258.9797
34216.617047332.282759800.811026.4913-99.0-0.1244790.108206-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.39390.15411.593750.226203-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.94290.45472.414461.01116-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.6654-99.00.974271.0716923.2841-99.0-0.299791.031621.5816-99.0-5.150376.8052621.1196-99.0-4.894328.92056
45216.610197532.282767400.941225.7669-99.0-0.03709690.108127-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.59120.15691.328930.192043-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.95630.34422.385060.756112-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.7029-99.00.8550661.0784622.7077-99.00.7057611.1463921.3186-99.0-5.78848.2832620.5633-99.02.944789.33336
56216.546554532.282793900.7111024.74050.20360.4611250.0864713-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.06350.07826352.160630.155745-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.7609-99.0-0.3592350.74796-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.021.9110.12646.245960.72714722.24140.24744.607421.0498621.1104-99.0-0.3323226.6948820.5462-99.08.460046.74708
67216.622066632.282794800.961025.4731-99.00.009508190.112663-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.56710.22711.358750.284206-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.4394-99.01.611161.11399-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.4441-99.0-0.6589911.0903924.4186-99.0-1.510981.0655921.2629-99.0-1.648466.497222.4853-99.0-14.27758.97886
78216.637339432.282798300.951025.7453-99.0-0.06084960.121802-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.77050.49081.126630.509285-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.62130.36243.247131.08383-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.7164-99.0-0.8834041.0338222.8762-99.00.4523731.0575822.0309-99.0-8.236796.9149720.0444-99.016.1049.37401
89216.631266432.282809600.591024.9184-99.00.1587150.116346-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.26690.25481.791510.420432-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.09630.23085.266221.11947-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.9343-99.0-1.050611.0097423.5596-99.0-0.8083071.0882822.0855-99.0-8.511346.9149921.1481-99.0-5.347778.97964
910216.62843332.282811900.971925.3115-99.00.03980870.116357-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.023.47030.25631.485460.350659-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.44050.30683.835481.0838-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.025.6997-99.0-1.833491.0120423.5431-99.0-0.769021.0790922.4252-99.0-8.642486.2661220.3689-99.06.956219.44634
In [7]:
irac_i2_catalogue['ALPHA_J2000'].unit = u.deg
irac_i2_catalogue['DELTA_J2000'].unit = u.deg
In [8]:
irac_i2_catalogue[:10].show_in_notebook()
Out[8]:
<Table length=10>
idxidALPHA_J2000DELTA_J2000IMAFLAGSSEGFLAGSm_ap_lbc_umerr_ap_lbc_uf_ap_lbc_uferr_ap_lbc_um_ap_mosaic_bwmerr_ap_mosaic_bwf_ap_mosaic_bwferr_ap_mosaic_bwm_ap_mosaic_rmerr_ap_mosaic_rf_ap_mosaic_rferr_ap_mosaic_rm_ap_mosaic_imerr_ap_mosaic_if_ap_mosaic_iferr_ap_mosaic_im_ap_prime90_zmerr_ap_prime90_zf_ap_prime90_zferr_ap_prime90_zm_ap_suprime_zmerr_ap_suprime_zf_ap_suprime_zferr_ap_suprime_zm_ap_lbc_ymerr_ap_lbc_yf_ap_lbc_yferr_ap_lbc_ym_ap_newfirm_jmerr_ap_newfirm_jf_ap_newfirm_jferr_ap_newfirm_jm_ap_newfirm_hmerr_ap_newfirm_hf_ap_newfirm_hferr_ap_newfirm_hm_ap_onis_kmerr_ap_onis_kf_ap_onis_kferr_ap_onis_km_ap_newfirm_kmerr_ap_newfirm_kf_ap_newfirm_kferr_ap_newfirm_km_ap_irac_ch1merr_ap_irac_ch1f_ap_irac_ch1ferr_ap_irac_ch1m_ap_irac_ch2merr_ap_irac_ch2f_ap_irac_ch2ferr_ap_irac_ch2m_ap_irac_ch3merr_ap_irac_ch3f_ap_irac_ch3ferr_ap_irac_ch3m_ap_irac_ch4merr_ap_irac_ch4f_ap_irac_ch4ferr_ap_irac_ch4
degdegmaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJy
01217.566537132.11720250-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.021.82275663820.31336.774813507241.95493994077-99.0-99.0-99.0-99.019.3435866264-99.031.641181949317.4097925571
12216.192470432.107189660-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.020.78895663820.094717.55566741541.53123907719-99.0-99.0-99.0-99.020.1495976355-99.05.5163612375613.0590683324
23216.209738432.108072640-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.021.92495663820.23926.166196277481.35848297506-99.0-99.0-99.0-99.020.1281753971-99.011.532745229310.3660594718
34216.204857232.108289560-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.48165663820.42113.692643187761.4321821816-99.0-99.0-99.0-99.020.7232067928-99.0-6.7672574380512.7094542684
45216.59273832.113139950-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.04095663820.38825.541372495051.98129231866-99.0-99.0-99.0-99.020.1004531708-99.0-1.6968861231817.3980901074
56216.589393632.113292350-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.020.97085663820.127614.84763711971.74495286071-99.0-99.0-99.0-99.019.5326385559-99.015.943097355719.9483978362
67216.688755132.113344450-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.019.31170163380.030768.44147278371.9352336278819.75765663820.044145.38761316281.8435359636319.53870444260.407855.528791647220.856485336419.7560252021-99.016.894008101114.2809280627
78217.074687332.117215360-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.021.75335663820.26077.221997682511.73409967134-99.0-99.0-99.0-99.020.3092585617-99.0-1.8599132917914.5841666987
89216.208635132.109948650-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.021.87635663820.25346.448479814711.50501086533-99.0-99.0-99.0-99.020.0678162101-99.013.281670387310.4138426546
910216.194204932.110369360-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.0-99.022.00345663820.31725.73610833091.67581550158-99.0-99.0-99.0-99.020.0951555976-99.07.4509679980212.9051101537

II - Removal of duplicated sources¶

We remove duplicated objects from the input catalogues.

In [9]:
"""SORT_COLS = ['merr_ap_gpc1_r', 'merr_ap_gpc1_g', 'merr_ap_gpc1_i', 'merr_ap_gpc1_z', 'merr_ap_gpc1_y']
FLAG_NAME = 'ps1_flag_cleaned'

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(catalogue, RA_COL, DEC_COL,  sort_col=SORT_COLS, flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))"""
Out[9]:
'SORT_COLS = [\'merr_ap_gpc1_r\', \'merr_ap_gpc1_g\', \'merr_ap_gpc1_i\', \'merr_ap_gpc1_z\', \'merr_ap_gpc1_y\']\nFLAG_NAME = \'ps1_flag_cleaned\'\n\nnb_orig_sources = len(catalogue)\n\ncatalogue = remove_duplicates(catalogue, RA_COL, DEC_COL,  sort_col=SORT_COLS, flag_name=FLAG_NAME)\n\nnb_sources = len(catalogue)\n\nprint("The initial catalogue had {} sources.".format(nb_orig_sources))\nprint("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))\nprint("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))'

III - Astrometry correction¶

We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.

In [10]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_Bootes.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [11]:
nb_astcor_diag_plot(i_catalogue["ra"], i_catalogue["dec"], 
                    gaia_coords.ra, gaia_coords.dec)
In [12]:
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(i_catalogue["ra"], i_catalogue["dec"]),
    gaia_coords
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))

RA correction: -0.0735978168904694 arcsec
Dec correction: -0.12698051759514328 arcsec
In [13]:
i_catalogue["ra"] = i_catalogue["ra"] + delta_ra.to(u.deg)
i_catalogue["dec"] = i_catalogue["dec"] + delta_dec.to(u.deg)
In [14]:
nb_astcor_diag_plot(i_catalogue["ra"], i_catalogue["dec"], 
                    gaia_coords.ra, gaia_coords.dec)

We then correct the other catalogue

In [15]:
nb_astcor_diag_plot(irac_i2_catalogue["ALPHA_J2000"], irac_i2_catalogue["DELTA_J2000"], 
                    gaia_coords.ra, gaia_coords.dec)
In [16]:
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(irac_i2_catalogue["ALPHA_J2000"], irac_i2_catalogue["DELTA_J2000"]),
    gaia_coords
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))

RA correction: -0.07916888162640134 arcsec
Dec correction: -0.08063874360573209 arcsec
In [17]:
irac_i2_catalogue["ALPHA_J2000"] = irac_i2_catalogue["ALPHA_J2000"] + delta_ra.to(u.deg)
irac_i2_catalogue["DELTA_J2000"] = irac_i2_catalogue["DELTA_J2000"] + delta_dec.to(u.deg)
In [18]:
nb_astcor_diag_plot(irac_i2_catalogue["ALPHA_J2000"], irac_i2_catalogue["DELTA_J2000"], 
                    gaia_coords.ra, gaia_coords.dec)

Merging¶

In [19]:
nb_merge_dist_plot(
    SkyCoord(i_catalogue['ra'], i_catalogue['dec']),
    SkyCoord(irac_i2_catalogue["ALPHA_J2000"], irac_i2_catalogue["DELTA_J2000"] )
)
In [20]:
radius = 1.0*u.arcsec
coords_1 = SkyCoord(i_catalogue['ra'], i_catalogue['dec'])

coords_2 = SkyCoord(irac_i2_catalogue["ALPHA_J2000"], irac_i2_catalogue["DELTA_J2000"])

# Search for sources in second catalogue matching the sources in the first
# one.
idx_2, idx_1, d2d, _ = coords_1.search_around_sky(coords_2, radius)

# We want to flag the possible mis-associations, i.e. the sources in each
# catalogue that are associated to several sources in the other one, but
# also all the sources that are associated to a problematic source in the
# other catalogue (e.g. if two sources in the first catalogue are
# associated to the same source in the second catalogue, they must be
# flagged as potentially problematic).
#
# Search for duplicate associations
toflag_idx_1 = np.unique([item for item, count in Counter(idx_1).items()
                              if count > 1])
toflag_idx_2 = np.unique([item for item, count in Counter(idx_2).items()
                              if count > 1])
# Flagging the sources associated to duplicates
dup_associated_in_idx1 = np.in1d(idx_2, toflag_idx_2)
dup_associated_in_idx2 = np.in1d(idx_1, toflag_idx_1)
toflag_idx_1 = np.unique(np.concatenate(
    (toflag_idx_1, idx_1[dup_associated_in_idx1])
))
toflag_idx_2 = np.unique(np.concatenate(
    (toflag_idx_2, idx_2[dup_associated_in_idx2])
))

# Adding the flags to the catalogue.  In the second catalogue, the column
# is named "flag_merged_2" and will be combined to the flag_merged column
# one the merge is done.
try:
    i_catalogue["flag_merged"] |= np.in1d(np.arange(len(i_catalogue), dtype=int),
                                        toflag_idx_1)
except KeyError:
    i_catalogue.add_column(Column(
        data=np.in1d(np.arange(len(i_catalogue), dtype=int), toflag_idx_1),
        name="flag_merged"
    ))
irac_i2_catalogue.add_column(Column(
    data=np.in1d(np.arange(len(irac_i2_catalogue), dtype=int), toflag_idx_2),
       name="flag_merged_2"
))

# Now that we have flagged the maybe spurious associations, we want to
# associate each source of each catalogue to at most one source in the
# other one.

# We sort the indices by the distance to take the nearest counterparts in
# the following steps.
sort_idx = np.argsort(d2d)
idx_1 = idx_1[sort_idx]
idx_2 = idx_2[sort_idx]

# These array will contain the indexes of the matching sources in both
# catalogues.
match_idx_1 = np.array([], dtype=int)
match_idx_2 = np.array([], dtype=int)

while len(idx_1) > 0:

    both_first_idx = np.sort(np.intersect1d(
         np.unique(idx_1, return_index=True)[1],
         np.unique(idx_2, return_index=True)[1],
    ))

    new_match_idx_1 = idx_1[both_first_idx]
    new_match_idx_2 = idx_2[both_first_idx]

    match_idx_1 = np.concatenate((match_idx_1, new_match_idx_1))
    match_idx_2 = np.concatenate((match_idx_2, new_match_idx_2))

    # We remove the matching sources in both catalogues.
    to_remove = (np.in1d(idx_1, new_match_idx_1) |
                     np.in1d(idx_2, new_match_idx_2))
    idx_1 = idx_1[~to_remove]
    idx_2 = idx_2[~to_remove]

# Indices of un-associated object in both catalogues.
unmatched_idx_1 = np.delete(np.arange(len(i_catalogue), dtype=int),match_idx_1)
unmatched_idx_2 = np.delete(np.arange(len(irac_i2_catalogue), dtype=int),match_idx_2)

# Sources only in cat_1
only_in_cat_1 = i_catalogue[unmatched_idx_1]

# Sources only in cat_2
only_in_cat_2 = irac_i2_catalogue[unmatched_idx_2]
# We are using the ra and dec columns from cat_2 for the position.
only_in_cat_2["ALPHA_J2000"].name = "ra"
only_in_cat_2["DELTA_J2000"].name = "dec"

# Where we have an association we take the values from the i selected catalogue.
both_in_cat_1_and_cat_2 = i_catalogue[match_idx_1] #hstack([cat_1[match_idx_1], cat_2[match_idx_2]])
# We don't need the positions from the second catalogue anymore.
#both_in_cat_1_and_cat_2.remove_columns([racol_2, decol_2])

# Logging the number of rows
print("There are {} sources only in the first catalogue".format(len(only_in_cat_1)))
print("There are {} sources only in the second catalogue".format(len(only_in_cat_2)))
print("There are {} sources in both catalogues".format(len(both_in_cat_1_and_cat_2)))

merged_catalogue = vstack([only_in_cat_1,  both_in_cat_1_and_cat_2,
                               only_in_cat_2])

# When vertically stacking the catalogues, some values in the flag columns
# are masked because they did not exist in the catalogue some row originate
# from. We must set them to the appropriate value.
for colname in merged_catalogue.colnames:
    if 'flag' in colname:
        merged_catalogue[colname][merged_catalogue[colname].mask] = False

# We combined the flag_merged flags
merged_catalogue['flag_merged'] |= merged_catalogue['flag_merged_2']
merged_catalogue.remove_column('flag_merged_2')

There are 1580405 sources only in the first catalogue
There are 185203 sources only in the second catalogue
There are 486485 sources in both catalogues
In [21]:
# Replace -99 values
for col in merged_catalogue.colnames:
    if col.startswith('m_'):
        errcol = "merr{}".format(col[1:])
        merged_catalogue[col][merged_catalogue[col] < -90.] = np.nan
        merged_catalogue[errcol][merged_catalogue[errcol] < -90.] = np.nan
    if col.startswith('f_'):
        errcol = "ferr{}".format(col[1:])
        merged_catalogue[col][merged_catalogue[col] < -90.] = np.nan
        merged_catalogue[errcol][merged_catalogue[errcol] < -90.] = np.nan
        

/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/ipykernel/__main__.py:5: RuntimeWarning: invalid value encountered in less
In [22]:
merged_catalogue['CLASS_STAR'].name = 'stellarity'
merged_catalogue.remove_columns(['FLAG_DEEP', 'IMAFLAGS', 'SEGFLAGS'])
In [23]:
merged_catalogue[:10].show_in_notebook()
Out[23]:
<Table masked=True length=10>
idxidradecstellaritym_ap_lbc_umerr_ap_lbc_uf_ap_lbc_uferr_ap_lbc_um_ap_mosaic_bwmerr_ap_mosaic_bwf_ap_mosaic_bwferr_ap_mosaic_bwm_ap_mosaic_rmerr_ap_mosaic_rf_ap_mosaic_rferr_ap_mosaic_rm_ap_mosaic_imerr_ap_mosaic_if_ap_mosaic_iferr_ap_mosaic_im_ap_prime90_zmerr_ap_prime90_zf_ap_prime90_zferr_ap_prime90_zm_ap_suprime_zmerr_ap_suprime_zf_ap_suprime_zferr_ap_suprime_zm_ap_lbc_ymerr_ap_lbc_yf_ap_lbc_yferr_ap_lbc_ym_ap_newfirm_jmerr_ap_newfirm_jf_ap_newfirm_jferr_ap_newfirm_jm_ap_newfirm_hmerr_ap_newfirm_hf_ap_newfirm_hferr_ap_newfirm_hm_ap_onis_kmerr_ap_onis_kf_ap_onis_kferr_ap_onis_km_ap_newfirm_kmerr_ap_newfirm_kf_ap_newfirm_kferr_ap_newfirm_km_ap_irac_ch1merr_ap_irac_ch1f_ap_irac_ch1ferr_ap_irac_ch1m_ap_irac_ch2merr_ap_irac_ch2f_ap_irac_ch2ferr_ap_irac_ch2m_ap_irac_ch3merr_ap_irac_ch3f_ap_irac_ch3ferr_ap_irac_ch3m_ap_irac_ch4merr_ap_irac_ch4f_ap_irac_ch4ferr_ap_irac_ch4flag_merged
degdegmaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJymaguJyuJy
01216.47384255632.28256102760.6425.3294239044nan0.07659106701610.0957338809967nannannannannannannannan23.68714904790.1773000061511.216578125950.198666423559nannannannannannannannannannannannannannannannannannannannannannannannannannannannan23.5439605713nan0.1149633675810.63656055927323.1025695801nan0.1205748915670.98189139366121.087720871nan3.420719385154.9558925628720.3205661774nan8.530465126049.24751186371False
12216.61758935632.28270802760.6625.2766113281nan0.06741215288640.107003986835nannannannannannannannan23.56304740910.1800999939441.363894104960.226240366697nannannannannannannannan22.4279766083nan1.857714533811.01104068756nannannannannannannannannannannannannannannannan22.7807235718nan0.5608925223351.1213344335623.5936756134nan-0.8412736654281.083616137521.1719741821nan-1.248873114596.792944431321.0885429382nan-4.716422557839.01941776276False
23216.62290565632.28272342760.9527.0240478516nan-0.1724956929680.114389590919nannannannannannannannan23.60794639590.2357999980451.308642864230.284210830927nannannannannannannannan22.40415191650.3129000067713.965874433521.14293158054nannannannannannannannannannannannannannannannan23.0248279572nan0.0582638494671.0904052257523.336107254nan-0.4502956271171.0656269788721.9413833618nan-6.458296775826.2659516334520.8749294281nan-1.740251898778.97970199585False
34216.61702685632.28272452760.8126.4913158417nan-0.1244794055820.10820645839nannannannannannannannan23.39394950870.1541000008581.593749523160.226203039289nannannannannannannannan22.9429492950.4546999931342.414459943771.0111618042nannannannannannannannannannannannannannannannan22.6654319763nan0.974269568921.0716878175723.284116745nan-0.2997899949551.0315999984721.5815544128nan-5.150365352636.8052577972421.119594574nan-4.894322872168.92055988312False
45216.61017705632.28273212760.9425.7669200897nan-0.03709691762920.108126543462nannannannannannannannan23.59124755860.1569000035521.328925609590.192043378949nannannannannannannannan22.95625114440.3441999852662.38506078720.756111741066nannannannannannannannannannannannannannannannan22.7028656006nan0.8550655245781.0784634351722.7077236176nan0.7057609558111.1463925838521.3186416626nan-5.788395881658.2832555770920.563287735nan2.944777011879.333360672False
57216.62204615632.28275952760.9625.4730987549nan0.009508190676570.112662665546nannannannannannannannan23.56714820860.2270999997851.35875260830.284206002951nannannannannannannannan22.439414978nan1.611160755161.11398911476nannannannannannannannannannannannannannannannan23.4441127777nan-0.6589908003811.0903910398524.4186477661nan-1.510975480081.065593957921.2628974915nan-1.648458242426.4972023963922.4853134155nan-14.27750110638.97886371613False
68216.63731895632.28276302760.9525.7453346252nan-0.06084961816670.121801674366nannannannannannannannan23.77054786680.4907999932771.126628041270.50928491354nannannannannannannannan22.62125205990.3623999953273.247127056121.08383488655nannannannannannannannannannannannannannannannan23.7164077759nan-0.8834044933321.0338181257222.8762054443nan0.4523729085921.0575848817822.0308589935nan-8.236787796026.9149694442720.0444316864nan16.10395812999.37401008606False
79216.63124595632.28277432760.5924.918428421nan0.1587149202820.116346128285nannannannannannannannan23.26694870.2547999918461.791514635090.420431703329nannannannannannannannan22.09625244140.2308000028135.266223430631.11946570873nannannannannannannannannannannannannannannannan23.9343395233nan-1.050605654721.009736061123.5595855713nan-0.8083074688911.0882790088722.0854988098nan-8.511343955996.9149918556221.1480808258nan-5.347772121438.97964286804False
810216.62841255632.28277662760.9725.3114948273nan0.0398086532950.116356767714nannannannannannannannan23.47034835820.2563000023371.485458612440.350658893585nannannannannannannannan22.44045066830.306800007823.835475683211.08380281925nannannannannannannannannannannannannannannannan25.6997261047nan-1.833492517471.012043356923.5431251526nan-0.7690201997761.0790855884622.4251899719nan-8.642475128176.2661228179920.368894577nan6.956206798559.44633960724False
911216.61473065632.28277992760.8625.14245605470.3582000136380.3184325098990.105055473745nannannannannannannannan19.55484962460.0031987659167554.70919036870.161182701588nannannannannannannannan19.60264968870.016318619251352.35276794430.786862313747nannannannannannannannannannannannannannannannan19.87150192260.028920920565740.86950683591.0886472463620.36445617680.045524459332225.95479011541.0882734060319.9997043610.22269999980936.31766891487.449272632620.065202713nan16.30392837528.94381141663False

IV - Flagging Gaia objects¶

In [24]:
merged_catalogue.add_column(
    gaia_flag_column(SkyCoord(merged_catalogue["ra"], merged_catalogue["dec"]), 2012, gaia)
)
In [25]:
GAIA_FLAG_NAME = "bootes_flag_gaia"

merged_catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(merged_catalogue[GAIA_FLAG_NAME] > 0)))

38129 sources flagged.

V - Adding HELP unique identifiers and field columns¶

In [26]:
merged_catalogue.add_column(Column(gen_help_id(merged_catalogue['ra'], merged_catalogue['dec']),
                                   name="help_id"))
merged_catalogue.add_column(Column(np.full(len(merged_catalogue), "Bootes", dtype='<U18'),
                                   name="field"))
In [27]:
# Check that the HELP Ids are unique
if len(merged_catalogue) != len(np.unique(merged_catalogue['help_id'])):
    print("The HELP IDs are not unique!!!")
else:
    print("OK!")

The HELP IDs are not unique!!!
In [ ]:
#merged_catalogue['help_id',  'id'].write(
#    "{}/master_list_cross_ident_bootes{}.fits".format(OUT_DIR, SUFFIX))
#merged_catalogue.remove_columns(['id'])

VI - Saving to disk¶

In [29]:
merged_catalogue.write("{}/master_catalogue_bootes{}.fits".format(OUT_DIR, SUFFIX), overwrite=True)