COSMOS master catalogue¶

Preparation of Hyper Suprime-Cam Subaru Strategic Program Catalogues (HSC-SSP) data¶

This catalogue comes from dmu0_HSC.

In the catalogue, we keep:

  • The object_id as unique object identifier;
  • The position;
  • The g, r, i, z, y (no N921) aperture magnitude in 2” that we aperture correct;
  • The g, r, i, z, y (no N921) kron fluxes and magnitudes.
  • The extended flag that we convert to a stellariy.

Note: On ELAIS-N1 the HSC-SSP catalogue does not contain any N816 magnitudes.

We use 2016 as the epoch.

In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
This notebook was run with herschelhelp_internal version: 
33f5ec7 (Wed Dec 6 16:56:17 2017 +0000)
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
plt.style.use('ggplot')

from collections import OrderedDict
import os

from astropy import units as u
from astropy import visualization as vis
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, nb_plot_mag_ap_evol, \
    nb_plot_mag_vs_apcor, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux, aperture_correction
In [3]:
OUT_DIR =  os.environ.get('TMP_DIR', "./data_tmp")
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "hsc-udeep_ra"
DEC_COL = "hsc-udeep_dec"
In [4]:
# Pristine HSC catalogue
orig_hsc = Table.read("../../dmu0/dmu0_HSC/data/HSC-PDR1_uDeep_COSMOS.fits")

I - Aperture correction¶

To compute aperture correction we need to dertermine two parametres: the target aperture and the range of magnitudes for the stars that will be used to compute the correction.

Target aperture: To determine the target aperture, we simulate a curve of growth using the provided apertures and draw two figures:

  • The evolution of the magnitudes of the objects by plotting on the same plot aperture number vs the mean magnitude.
  • The mean gain (loss when negative) of magnitude is each aperture compared to the previous (except for the first of course).

As target aperture, we should use the smallest (i.e. less noisy) aperture for which most of the flux is captures.

Magnitude range: To know what limits in aperture to use when doing the aperture correction, we plot for each magnitude bin the correction that is computed and its RMS. We should then use the wide limits (to use more stars) where the correction is stable and with few dispersion.

In [5]:
bands = ["g", "r", "i", "z", "y", "n921"]
apertures = ["10", "15", "20", "30", "40", "57", "84", "118", "168", "235"]

magnitudes = {}
magnitude_errors ={}
stellarities = {}

for band in bands:
    magnitudes[band] = np.array(
        [orig_hsc["{}mag_aperture{}".format(band, aperture)] for aperture in apertures]
    )
    # Some sources have an infinite magnitude
    mask = np.isinf(magnitudes[band])
    magnitudes[band][mask] = np.nan
    try:
        magnitude_errors[band] = np.array(
        [orig_hsc["{}mag_aperture{}_err".format(band, aperture)] for aperture in apertures]
        )
        magnitude_errors[band][mask] = np.nan
    except KeyError:
        print("No error column for a " + band + " band aperture magnitude.")
        
    stellarities[band] = 1 - np.array(orig_hsc["{}classification_extendedness".format(band)])
    
mag_corr = {}

I.a - g band¶

In [6]:
nb_plot_mag_ap_evol(magnitudes['g'], stellarities['g'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [7]:
nb_plot_mag_vs_apcor(orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'], stellarities['g'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We will use magnitudes between 18.5 and 20.8

In [8]:
# Aperture correction
mag_corr['g'], num, std = aperture_correction(
    orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'], 
    stellarities['g'],
    mag_min=18.5, mag_max=20.8)
print("Aperture correction for g band:")
print("Correction: {}".format(mag_corr['g']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for g band:
Correction: -0.15904808044433594
Number of source used: 969
RMS: 0.018600222036784176
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.b - r band¶

In [9]:
nb_plot_mag_ap_evol(magnitudes['r'], stellarities['r'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [10]:
nb_plot_mag_vs_apcor(orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'], stellarities['r'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17.6 and 19.7.

In [11]:
# Aperture correction
mag_corr['r'], num, std = aperture_correction(
    orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'], 
    stellarities['r'],
    mag_min=17.6, mag_max=19.7)
print("Aperture correction for r band:")
print("Correction: {}".format(mag_corr['r']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for r band:
Correction: -0.09056663513183594
Number of source used: 349
RMS: 0.00958719667442224
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.c - i band¶

In [12]:
nb_plot_mag_ap_evol(magnitudes['i'], stellarities['i'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [13]:
nb_plot_mag_vs_apcor(orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'], stellarities['i'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 18.5 and 19.8.

In [14]:
# Aperture correction
mag_corr['i'], num, std = aperture_correction(
    orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'], 
    stellarities['i'],
    mag_min=18.5, mag_max=19.8)
print("Aperture correction for i band:")
print("Correction: {}".format(mag_corr['i']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for i band:
Correction: -0.10589122772216797
Number of source used: 2146
RMS: 0.016738303023859657
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.d - z band¶

In [15]:
nb_plot_mag_ap_evol(magnitudes['z'], stellarities['z'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [16]:
nb_plot_mag_vs_apcor(orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'], stellarities['z'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17.5 and 19.8.

In [17]:
# Aperture correction
mag_corr['z'], num, std = aperture_correction(
    orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'], 
    stellarities['z'],
    mag_min=17.5, mag_max=19.8)
print("Aperture correction for z band:")
print("Correction: {}".format(mag_corr['z']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for z band:
Correction: -0.09363174438476562
Number of source used: 3208
RMS: 0.011914830630386885
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.e - y band¶

In [18]:
nb_plot_mag_ap_evol(magnitudes['y'], stellarities['y'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [19]:
nb_plot_mag_vs_apcor(orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'], stellarities['y'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17 and 18.7.

In [20]:
# Aperture correction
mag_corr['y'], num, std = aperture_correction(
    orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'], 
    stellarities['y'],
    mag_min=17, mag_max=18.7)
print("Aperture correction for y band:")
print("Correction: {}".format(mag_corr['y']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for y band:
Correction: -0.17114639282226562
Number of source used: 764
RMS: 0.020499810124057233
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.f - n921 band¶

In [21]:
nb_plot_mag_ap_evol(magnitudes['n921'], stellarities['n921'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [22]:
nb_plot_mag_vs_apcor(orig_hsc['n921mag_aperture20'], orig_hsc['n921mag_aperture40'], stellarities['n921'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17 and 18.7.

In [23]:
# Aperture correction
mag_corr['n921'], num, std = aperture_correction(
    orig_hsc['n921mag_aperture20'], orig_hsc['n921mag_aperture40'], 
    stellarities['n921'],
    mag_min=17, mag_max=18.7)
print("Aperture correction for n921 band:")
print("Correction: {}".format(mag_corr['n921']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for n921 band:
Correction: -0.1354379653930664
Number of source used: 533
RMS: 0.008487969070401766
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

II - Stellarity¶

HSC does not provide a 0 to 1 stellarity value but a 0/1 extended flag in each band. We are using the same method as UKIDSS (cf this page) to compute a stellarity based on the class in each band:

\begin{equation*} P(star) = \frac{ \prod_{i} P(star)_i }{ \prod_{i} P(star)_i + \prod_{i} P(galaxy)_i } \end{equation*}

where $i$ is the band, and with using the same probabilities as UKDISS:

HSC flag UKIDSS flag Meaning P(star) P(galaxy) P(noise) P(saturated)
-9 Saturated 0.0 0.0 5.0 95.0
-3 Probable galaxy 25.0 70.0 5.0 0.0
-2 Probable star 70.0 25.0 5.0 0.0
0 -1 Star 90.0 5.0 5.0 0.0
0 Noise 5.0 5.0 90.0 0.0
1 +1 Galaxy 5.0 90.0 5.0 0.0
In [24]:
# We are creating an array containing the extended flag in all band.
# Some sources have no flag in some band, there will be NaN in the array.
hsc_ext_flag = np.array([
    orig_hsc[colname] for colname in 
    ['gclassification_extendedness',
     'rclassification_extendedness',
     'iclassification_extendedness',
     'zclassification_extendedness',
     'yclassification_extendedness',
    'n921classification_extendedness']
])
In [25]:
hsc_pstar = 0.9 * (hsc_ext_flag == 0) + 0.05 * (hsc_ext_flag == 1)
hsc_pgal = 0.05 * (hsc_ext_flag == 0) + 0.9 * (hsc_ext_flag == 1)

# We put back the NaN values
hsc_pstar[np.isnan(hsc_ext_flag)] = np.nan
hsc_pgal[np.isnan(hsc_ext_flag)] = np.nan
In [26]:
stellarity = np.nanprod(hsc_pstar, axis=0) / np.nansum(
    [np.nanprod(hsc_pgal, axis=0), np.nanprod(hsc_pstar, axis=0)], axis=0)

stellarity = np.round(stellarity, 3)
In [27]:
vis.hist(stellarity, bins='scott');
In [28]:
orig_hsc.add_column(Column(data=stellarity, name="stellarity"))

II - Column selection¶

In [29]:
imported_columns = OrderedDict({
        "object_id": "hsc-udeep_id",
        "ra": "hsc-udeep_ra",
        "dec": "hsc-udeep_dec",
        "gmag_aperture20": "m_ap_hsc-udeep_g",
        "gmag_aperture20_err": "merr_ap_hsc-udeep_g",
        "gmag_kron": "m_hsc-udeep_g",
        "gmag_kron_err": "merr_hsc-udeep_g",
        "rmag_aperture20": "m_ap_hsc-udeep_r",
        "rmag_aperture20_err": "merr_ap_hsc-udeep_r",
        "rmag_kron": "m_hsc-udeep_r",
        "rmag_kron_err": "merr_hsc-udeep_r",
        "imag_aperture20": "m_ap_hsc-udeep_i",
        "imag_aperture20_err": "merr_ap_hsc-udeep_i",
        "imag_kron": "m_hsc-udeep_i",
        "imag_kron_err": "merr_hsc-udeep_i",
        "zmag_aperture20": "m_ap_hsc-udeep_z",
        "zmag_aperture20_err": "merr_ap_hsc-udeep_z",
        "zmag_kron": "m_hsc-udeep_z",
        "zmag_kron_err": "merr_hsc-udeep_z",
        "ymag_aperture20": "m_ap_hsc-udeep_y",
        "ymag_aperture20_err": "merr_ap_hsc-udeep_y",
        "ymag_kron": "m_hsc-udeep_y",
        "ymag_kron_err": "merr_hsc-udeep_y",
            "n921mag_aperture20": "m_ap_hsc-udeep_n921",
        "n921mag_aperture20_err": "merr_ap_hsc-udeep_n921",
        "n921mag_kron": "m_hsc-udeep_n921",
        "n921mag_kron_err": "merr_hsc-udeep_n921",
        "stellarity": "hsc-udeep_stellarity"
    })


catalogue = orig_hsc[list(imported_columns)]
for column in imported_columns:
    catalogue[column].name = imported_columns[column]

epoch = 2017

# Clean table metadata
catalogue.meta = None
In [30]:
# Aperture correction
for band in bands:
    catalogue["m_ap_hsc-udeep_{}".format(band)] += mag_corr[band]
In [31]:
# Adding flux and band-flag columns
for col in catalogue.colnames:
    if col.startswith('m_'):
        
        errcol = "merr{}".format(col[1:])
        flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))
        
        # Fluxes are added in µJy
        catalogue.add_column(Column(flux * 1.e6, name="f{}".format(col[1:])))
        catalogue.add_column(Column(error * 1.e6, name="f{}".format(errcol[1:])))
        
        # Band-flag column
        if 'ap' not in col:
            catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))
        
# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting.
In [32]:
catalogue[:10].show_in_notebook()
Out[32]:
<Table masked=True length=10>
idxhsc-udeep_idhsc-udeep_rahsc-udeep_decm_ap_hsc-udeep_gmerr_ap_hsc-udeep_gm_hsc-udeep_gmerr_hsc-udeep_gm_ap_hsc-udeep_rmerr_ap_hsc-udeep_rm_hsc-udeep_rmerr_hsc-udeep_rm_ap_hsc-udeep_imerr_ap_hsc-udeep_im_hsc-udeep_imerr_hsc-udeep_im_ap_hsc-udeep_zmerr_ap_hsc-udeep_zm_hsc-udeep_zmerr_hsc-udeep_zm_ap_hsc-udeep_ymerr_ap_hsc-udeep_ym_hsc-udeep_ymerr_hsc-udeep_ym_ap_hsc-udeep_n921merr_ap_hsc-udeep_n921m_hsc-udeep_n921merr_hsc-udeep_n921hsc-udeep_stellarityf_ap_hsc-udeep_gferr_ap_hsc-udeep_gf_hsc-udeep_gferr_hsc-udeep_gflag_hsc-udeep_gf_ap_hsc-udeep_rferr_ap_hsc-udeep_rf_hsc-udeep_rferr_hsc-udeep_rflag_hsc-udeep_rf_ap_hsc-udeep_iferr_ap_hsc-udeep_if_hsc-udeep_iferr_hsc-udeep_iflag_hsc-udeep_if_ap_hsc-udeep_zferr_ap_hsc-udeep_zf_hsc-udeep_zferr_hsc-udeep_zflag_hsc-udeep_zf_ap_hsc-udeep_yferr_ap_hsc-udeep_yf_hsc-udeep_yferr_hsc-udeep_yflag_hsc-udeep_yf_ap_hsc-udeep_n921ferr_ap_hsc-udeep_n921f_hsc-udeep_n921ferr_hsc-udeep_n921flag_hsc-udeep_n921
042089339471003655150.2910549911.4129307707724.19920.061388923.68850.080273128.10161.7949926.34170.77494723.23790.044185422.83080.069907523.17940.066752822.56940.088521823.42840.18018622.86890.23375323.1830.073652522.97180.1339070.00.7591260.04292191.215040.0898334False0.02086150.03448920.1055180.0753136False1.840010.07488162.677230.17238False1.942030.1193993.405910.277689False1.543980.2562352.58490.556516False1.935560.1313012.351130.289971False
142089339471003658150.307162741.4127490945525.78970.25635525.75670.34124628.02431.6712127.18731.0956925.01220.25100824.9480.34859524.95130.37206924.62720.4127223.68170.35999823.30910.35652924.86010.36998224.66110.4398020.00.1754340.0414220.1808450.0568394False0.02240220.03448230.0484240.0488678False0.3590120.08299890.3808760.122287False0.3797480.1301350.5118130.194555False1.222710.4054161.723340.565902False0.4130080.1407390.4960840.20095False
242089339471003661150.3679036141.4128973407927.85071.590927.96561.6089127.76881.3213127.83941.3080725.32040.30516325.54450.36658924.34950.19291624.60850.24437824.44040.6798724.1260.46664325.23060.50500925.01940.3897380.00.02628570.03851560.02364580.0350398False0.02834550.03449560.026560.0319988False0.2702840.07596740.2198910.0742442False0.6609770.1174440.5207110.117202False0.6079330.3806770.812090.349032False0.2936060.1365650.3566460.128022False
342089339471003662150.3082807381.413241322623.56590.03636623.48740.041445828.01521.6582327.41951.2133922.64780.026366422.59450.032712222.46770.034706722.41140.043565222.49330.11287322.37280.12718822.60420.042954622.59470.05477760.01.360280.04556171.462360.0558228False0.0225910.03450280.03910060.0436977False3.168720.07695053.328240.100277False3.740440.1195673.939490.158072False3.65340.3798074.081990.478183False3.298460.1304963.327430.167875False
442089339471003663150.3563344921.4131246951525.99720.31377825.91240.35200727.81161.3734427.27211.0321424.24760.124524.01620.12897624.35360.20976224.46890.30562923.98050.47294723.52690.37787624.12370.18060124.36180.2821020.00.1449140.04188010.156690.0508007False0.02724850.03446890.04478980.0425788False0.7260720.08325770.8985120.106736False0.6585130.1272230.5921550.166689False0.9285080.4044591.410040.490745False0.8138330.1353720.6535660.169813False
542089339471003664150.3172855341.4130312856725.35180.174973nannan27.97531.59669nannan25.25750.284711nannan24.76270.292322nannan24.5330.715277nannan24.46720.238107nannan0.00.2625920.0423183nannanFalse0.02343620.0344654nannanFalse0.2864070.075104nannanFalse0.4517920.12164nannanFalse0.5582030.367741nannanFalse0.5930920.130068nannanFalse
642089339471003665150.3407892891.4130857130726.49380.47211626.4030.42258727.87391.4550327.80571.3540324.54980.15496424.46030.14900824.27610.19772524.23380.20164623.38210.25166123.53620.28526824.2030.19409324.2920.2103960.00.09172370.03988470.09972280.0388139False0.02573030.03448210.02739860.034169False0.5496510.07845030.5968530.081913False0.7071990.1287890.7353590.136573False1.611310.3734841.398010.367314False0.7564720.1352310.6969350.135053False
742089339471003669150.350036281.4131955073125.50720.19290925.90050.3648927.8361.4048227.12050.97329225.11390.25045225.0390.33154724.76850.30074424.19510.25421825.20071.3118nannan27.15162.9774325.68881.039810.00.2275770.0404350.1584180.0532406False0.02664410.03447440.05149730.046164False0.3269170.07541140.3502660.106959False0.4493820.1244770.762010.17842False0.3017950.364633nannanFalse0.05004470.1372390.1925280.184384False
842089339471003670150.2951291231.4131660559625.53620.21957425.84030.19570728.07931.7593428.72172.2550425.0170.23019325.13650.18811624.62110.24638924.63850.18435625.61382.0079125.15390.89824724.99090.38083925.04330.2838090.00.2215780.04481090.1674510.0301835False0.02129460.03450610.01178490.0244768False0.3574330.07578130.3201740.0554736False0.5147060.1168040.5065420.08601False0.2062990.381520.3150850.260675False0.3661340.1284270.3488720.0911944False
942089339471003671150.3357013631.4131784493325.58180.215166nannan27.89441.48267nannan25.75520.449599nannan26.24461.15113nannannannannannan25.59570.703606nannan0.00.2124690.0421061nannanFalse0.02524860.0344793nannanFalse0.18110.0749928nannanFalse0.115390.12234nannanFalsenannannannanFalse0.209750.135927nannanFalse

III - Removal of duplicated sources¶

We remove duplicated objects from the input catalogues.

In [33]:
SORT_COLS = [
        'merr_ap_hsc-udeep_i', 'merr_ap_hsc-udeep_r', 'merr_ap_hsc-udeep_z',
        'merr_ap_hsc-udeep_y', 'merr_ap_hsc-udeep_g','merr_ap_hsc-udeep_n921']
FLAG_NAME = 'hsc-udeep_flag_cleaned'

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(
    catalogue, RA_COL, DEC_COL, 
    sort_col= SORT_COLS,
    flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
The initial catalogue had 1307594 sources.
The cleaned catalogue has 1307525 sources (69 removed).
The cleaned catalogue has 67 sources flagged as having been cleaned

III - Astrometry correction¶

We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.

In [34]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_COSMOS.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [35]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)
In [36]:
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
    gaia_coords
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))
RA correction: -0.061218833292286945 arcsec
Dec correction: -0.03624856686692546 arcsec
In [37]:
catalogue[RA_COL] +=  delta_ra.to(u.deg)
catalogue[DEC_COL] += delta_dec.to(u.deg)
In [38]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)