COSMOS master catalogue¶

Preparation of Hyper Suprime-Cam Subaru Strategic Program Catalogues (HSC-SSP) data¶

This catalogue comes from dmu0_HSC.

In the catalogue, we keep:

  • The object_id as unique object identifier;
  • The position;
  • The g, r, i, z, y (no N921) aperture magnitude in 2” that we aperture correct;
  • The g, r, i, z, y (no N921) kron fluxes and magnitudes.
  • The extended flag that we convert to a stellariy.

Note: On ELAIS-N1 the HSC-SSP catalogue does not contain any N816 magnitudes.

We use 2016 as the epoch.

In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
This notebook was run with herschelhelp_internal version: 
33f5ec7 (Wed Dec 6 16:56:17 2017 +0000)
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
plt.style.use('ggplot')

from collections import OrderedDict
import os

from astropy import units as u
from astropy import visualization as vis
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, nb_plot_mag_ap_evol, \
    nb_plot_mag_vs_apcor, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux, aperture_correction
In [3]:
OUT_DIR =  os.environ.get('TMP_DIR', "./data_tmp")
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "hsc-deep_ra"
DEC_COL = "hsc-deep_dec"
In [4]:
# Pristine HSC catalogue
orig_hsc = Table.read("../../dmu0/dmu0_HSC/data/HSC-PDR1_deep_COSMOS.fits")

I - Aperture correction¶

To compute aperture correction we need to dertermine two parametres: the target aperture and the range of magnitudes for the stars that will be used to compute the correction.

Target aperture: To determine the target aperture, we simulate a curve of growth using the provided apertures and draw two figures:

  • The evolution of the magnitudes of the objects by plotting on the same plot aperture number vs the mean magnitude.
  • The mean gain (loss when negative) of magnitude is each aperture compared to the previous (except for the first of course).

As target aperture, we should use the smallest (i.e. less noisy) aperture for which most of the flux is captures.

Magnitude range: To know what limits in aperture to use when doing the aperture correction, we plot for each magnitude bin the correction that is computed and its RMS. We should then use the wide limits (to use more stars) where the correction is stable and with few dispersion.

In [5]:
bands = ["g", "r", "i", "z", "y", "n921"]
apertures = ["10", "15", "20", "30", "40", "57", "84", "118", "168", "235"]

magnitudes = {}
magnitude_errors ={}
stellarities = {}

for band in bands:
    magnitudes[band] = np.array(
        [orig_hsc["{}mag_aperture{}".format(band, aperture)] for aperture in apertures]
    )
    # Some sources have an infinite magnitude
    mask = np.isinf(magnitudes[band])
    magnitudes[band][mask] = np.nan
    try:
        magnitude_errors[band] = np.array(
        [orig_hsc["{}mag_aperture{}_err".format(band, aperture)] for aperture in apertures]
        )
        magnitude_errors[band][mask] = np.nan
    except KeyError:
        print("No error column for a " + band + " band aperture magnitude.")
        
    stellarities[band] = 1 - np.array(orig_hsc["{}classification_extendedness".format(band)])
    
mag_corr = {}

I.a - g band¶

In [6]:
nb_plot_mag_ap_evol(magnitudes['g'], stellarities['g'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [7]:
nb_plot_mag_vs_apcor(orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'], stellarities['g'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We will use magnitudes between 18.5 and 20.8

In [8]:
# Aperture correction
mag_corr['g'], num, std = aperture_correction(
    orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'], 
    stellarities['g'],
    mag_min=20.8, mag_max=22.0)
print("Aperture correction for g band:")
print("Correction: {}".format(mag_corr['g']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for g band:
Correction: -0.20549774169921875
Number of source used: 3768
RMS: 0.017911572957685837
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.b - r band¶

In [9]:
nb_plot_mag_ap_evol(magnitudes['r'], stellarities['r'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [10]:
nb_plot_mag_vs_apcor(orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'], stellarities['r'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17.6 and 19.7.

In [11]:
# Aperture correction
mag_corr['r'], num, std = aperture_correction(
    orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'], 
    stellarities['r'],
    mag_min=20.0, mag_max=21.0)
print("Aperture correction for r band:")
print("Correction: {}".format(mag_corr['r']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for r band:
Correction: -0.08256912231445312
Number of source used: 3891
RMS: 0.011581382492618938
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.c - i band¶

In [12]:
nb_plot_mag_ap_evol(magnitudes['i'], stellarities['i'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [13]:
nb_plot_mag_vs_apcor(orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'], stellarities['i'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 18.5 and 19.8.

In [14]:
# Aperture correction
mag_corr['i'], num, std = aperture_correction(
    orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'], 
    stellarities['i'],
    mag_min=18.5, mag_max=19.8)
print("Aperture correction for i band:")
print("Correction: {}".format(mag_corr['i']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for i band:
Correction: -0.10167789459228516
Number of source used: 3474
RMS: 0.01759486506775622
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.d - z band¶

In [15]:
nb_plot_mag_ap_evol(magnitudes['z'], stellarities['z'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [16]:
nb_plot_mag_vs_apcor(orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'], stellarities['z'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17.5 and 19.8.

In [17]:
# Aperture correction
mag_corr['z'], num, std = aperture_correction(
    orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'], 
    stellarities['z'],
    mag_min=17.5, mag_max=19.8)
print("Aperture correction for z band:")
print("Correction: {}".format(mag_corr['z']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for z band:
Correction: -0.06690597534179688
Number of source used: 5966
RMS: 0.008133258602144438
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.e - y band¶

In [18]:
nb_plot_mag_ap_evol(magnitudes['y'], stellarities['y'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [19]:
nb_plot_mag_vs_apcor(orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'], stellarities['y'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17 and 18.7.

In [20]:
# Aperture correction
mag_corr['y'], num, std = aperture_correction(
    orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'], 
    stellarities['y'],
    mag_min=17, mag_max=18.7)
print("Aperture correction for y band:")
print("Correction: {}".format(mag_corr['y']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for y band:
Correction: -0.13036632537841797
Number of source used: 1008
RMS: 0.008730060428994603
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.f - n921 band¶

In [21]:
nb_plot_mag_ap_evol(magnitudes['n921'], stellarities['n921'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [22]:
nb_plot_mag_vs_apcor(orig_hsc['n921mag_aperture20'], orig_hsc['n921mag_aperture40'], stellarities['n921'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17 and 18.7.

In [23]:
# Aperture correction
mag_corr['n921'], num, std = aperture_correction(
    orig_hsc['n921mag_aperture20'], orig_hsc['n921mag_aperture40'], 
    stellarities['n921'],
    mag_min=17, mag_max=18.7)
print("Aperture correction for n921 band:")
print("Correction: {}".format(mag_corr['n921']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for n921 band:
Correction: -0.11112499237060547
Number of source used: 419
RMS: 0.0083674661823173
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

II - Stellarity¶

HSC does not provide a 0 to 1 stellarity value but a 0/1 extended flag in each band. We are using the same method as UKIDSS (cf this page) to compute a stellarity based on the class in each band:

\begin{equation*} P(star) = \frac{ \prod_{i} P(star)_i }{ \prod_{i} P(star)_i + \prod_{i} P(galaxy)_i } \end{equation*}

where $i$ is the band, and with using the same probabilities as UKDISS:

HSC flag UKIDSS flag Meaning P(star) P(galaxy) P(noise) P(saturated)
-9 Saturated 0.0 0.0 5.0 95.0
-3 Probable galaxy 25.0 70.0 5.0 0.0
-2 Probable star 70.0 25.0 5.0 0.0
0 -1 Star 90.0 5.0 5.0 0.0
0 Noise 5.0 5.0 90.0 0.0
1 +1 Galaxy 5.0 90.0 5.0 0.0
In [24]:
# We are creating an array containing the extended flag in all band.
# Some sources have no flag in some band, there will be NaN in the array.
hsc_ext_flag = np.array([
    orig_hsc[colname] for colname in 
    ['gclassification_extendedness',
     'rclassification_extendedness',
     'iclassification_extendedness',
     'zclassification_extendedness',
     'yclassification_extendedness',
     'n921classification_extendedness']
])
In [25]:
hsc_pstar = 0.9 * (hsc_ext_flag == 0) + 0.05 * (hsc_ext_flag == 1)
hsc_pgal = 0.05 * (hsc_ext_flag == 0) + 0.9 * (hsc_ext_flag == 1)

# We put back the NaN values
hsc_pstar[np.isnan(hsc_ext_flag)] = np.nan
hsc_pgal[np.isnan(hsc_ext_flag)] = np.nan
In [26]:
stellarity = np.nanprod(hsc_pstar, axis=0) / np.nansum(
    [np.nanprod(hsc_pgal, axis=0), np.nanprod(hsc_pstar, axis=0)], axis=0)

stellarity = np.round(stellarity, 3)
In [27]:
vis.hist(stellarity, bins='scott');
In [28]:
orig_hsc.add_column(Column(data=stellarity, name="stellarity"))

II - Column selection¶

In [29]:
imported_columns = OrderedDict({
        "object_id": "hsc-deep_id",
        "ra": "hsc-deep_ra",
        "dec": "hsc-deep_dec",
        "gmag_aperture20": "m_ap_hsc-deep_g",
        "gmag_aperture20_err": "merr_ap_hsc-deep_g",
        "gmag_kron": "m_hsc-deep_g",
        "gmag_kron_err": "merr_hsc-deep_g",
        "rmag_aperture20": "m_ap_hsc-deep_r",
        "rmag_aperture20_err": "merr_ap_hsc-deep_r",
        "rmag_kron": "m_hsc-deep_r",
        "rmag_kron_err": "merr_hsc-deep_r",
        "imag_aperture20": "m_ap_hsc-deep_i",
        "imag_aperture20_err": "merr_ap_hsc-deep_i",
        "imag_kron": "m_hsc-deep_i",
        "imag_kron_err": "merr_hsc-deep_i",
        "zmag_aperture20": "m_ap_hsc-deep_z",
        "zmag_aperture20_err": "merr_ap_hsc-deep_z",
        "zmag_kron": "m_hsc-deep_z",
        "zmag_kron_err": "merr_hsc-deep_z",
        "ymag_aperture20": "m_ap_hsc-deep_y",
        "ymag_aperture20_err": "merr_ap_hsc-deep_y",
        "ymag_kron": "m_hsc-deep_y",
        "ymag_kron_err": "merr_hsc-deep_y",
            "n921mag_aperture20": "m_ap_hsc-deep_n921",
        "n921mag_aperture20_err": "merr_ap_hsc-deep_n921",
        "n921mag_kron": "m_hsc-deep_n921",
        "n921mag_kron_err": "merr_hsc-deep_n921",
        "stellarity": "hsc-deep_stellarity"
    })


catalogue = orig_hsc[list(imported_columns)]
for column in imported_columns:
    catalogue[column].name = imported_columns[column]

epoch = 2017

# Clean table metadata
catalogue.meta = None
In [30]:
# Aperture correction
for band in bands:
    catalogue["m_ap_hsc-deep_{}".format(band)] += mag_corr[band]
In [31]:
# Adding flux and band-flag columns
for col in catalogue.colnames:
    if col.startswith('m_'):
        
        errcol = "merr{}".format(col[1:])
        flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))
        
        # Fluxes are added in µJy
        catalogue.add_column(Column(flux * 1.e6, name="f{}".format(col[1:])))
        catalogue.add_column(Column(error * 1.e6, name="f{}".format(errcol[1:])))
        
        # Band-flag column
        if 'ap' not in col:
            catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))
        
# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting.
In [32]:
catalogue[:10].show_in_notebook()
Out[32]:
<Table masked=True length=10>
idxhsc-deep_idhsc-deep_rahsc-deep_decm_ap_hsc-deep_gmerr_ap_hsc-deep_gm_hsc-deep_gmerr_hsc-deep_gm_ap_hsc-deep_rmerr_ap_hsc-deep_rm_hsc-deep_rmerr_hsc-deep_rm_ap_hsc-deep_imerr_ap_hsc-deep_im_hsc-deep_imerr_hsc-deep_im_ap_hsc-deep_zmerr_ap_hsc-deep_zm_hsc-deep_zmerr_hsc-deep_zm_ap_hsc-deep_ymerr_ap_hsc-deep_ym_hsc-deep_ymerr_hsc-deep_ym_ap_hsc-deep_n921merr_ap_hsc-deep_n921m_hsc-deep_n921merr_hsc-deep_n921hsc-deep_stellarityf_ap_hsc-deep_gferr_ap_hsc-deep_gf_hsc-deep_gferr_hsc-deep_gflag_hsc-deep_gf_ap_hsc-deep_rferr_ap_hsc-deep_rf_hsc-deep_rferr_hsc-deep_rflag_hsc-deep_rf_ap_hsc-deep_iferr_ap_hsc-deep_if_hsc-deep_iferr_hsc-deep_iflag_hsc-deep_if_ap_hsc-deep_zferr_ap_hsc-deep_zf_hsc-deep_zferr_hsc-deep_zflag_hsc-deep_zf_ap_hsc-deep_yferr_ap_hsc-deep_yf_hsc-deep_yferr_hsc-deep_yflag_hsc-deep_yf_ap_hsc-deep_n921ferr_ap_hsc-deep_n921f_hsc-deep_n921ferr_hsc-deep_n921flag_hsc-deep_n921
042089330881070315150.3038148991.0453525091225.92330.19280726.19480.17036826.32510.31941426.20320.22861725.3830.16818825.41540.13928625.20090.19918925.31420.17940127.01214.5255126.2261.71336nannannannan0.00.155120.02754660.1208050.0189562False0.1071450.03152120.1198760.0252416False0.2551620.03952630.247650.0317703False0.3017370.05535650.2718460.0449182False0.05690840.2372030.1173790.185231FalsenannannannanFalse
142089330881070330150.2938986181.0458400277825.66180.18029525.59930.27829725.72750.21102925.20550.26057425.26650.15479524.72110.18482225.25930.22369826.94122.0866924.97020.69290526.38624.85603nannannannan0.00.1973690.03277470.2090630.0535873False0.1857750.03610810.3004670.0721111False0.284060.04049870.4694090.0799061False0.2859440.05891410.06074510.116747False0.3731920.2381670.1012820.452989FalsenannannannanFalse
242089330881070357150.2908847461.0463655169527.24130.7487326.49870.47833126.01930.28142425.50350.25748125.73170.23548625.40260.26006525.60180.31074525.20820.32316626.31242.42388nannannannannannan0.9470.04607510.03177370.09130990.0402274False0.1419970.03680580.2283520.0541535False0.1850590.04013750.2505840.060022False0.208590.05969970.2997360.0892155False0.1084010.242001nannanFalsenannannannanFalse
342089330881070386150.2903560431.0470105464225.41370.14233925.71730.1474725.08570.11907625.18940.11973124.80260.1020124.85920.098651124.53760.11624724.63270.11823825.6021.2620325.1610.750718nannannannan0.00.2480490.03251890.1875340.0254717False0.3355110.03679640.3049510.0336289False0.4354860.04091620.4133450.0375569False0.5558710.05951580.5092140.0554541False0.2085440.2424060.3130360.216444FalsenannannannanFalse
442089330881070454150.3035620061.048444350526.58530.34580526.58060.19724425.18150.11003725.38890.089142524.78760.098859324.87930.072065724.78120.13900524.97160.11286524.58270.50137124.73460.385842nannannannan0.00.08431160.0268530.08467710.0153832False0.3071790.03113180.2537810.0208363False0.4415480.04020410.4057760.0269334False0.444140.05686230.3726990.0387428False0.5332320.2462350.4636370.164764FalsenannannannanFalse
542089330881070462150.3051137141.0484175873426.87240.454256nannan26.3470.326979nannan26.21760.335125nannan26.14360.466893nannannannannannannannannannan0.0530.06472250.0270789nannanFalse0.1050030.0316227nannanFalse0.1182960.0365135nannanFalse0.1266410.0544589nannanFalsenannannannanFalsenannannannanFalse
642089330881070470150.2903526971.0486668932726.23710.30490825.94530.25074125.2260.13869325.12680.15798524.79730.10055124.73890.12066224.54880.11586324.49960.14200824.32240.38939724.07250.378107nannannannan0.00.1161830.03262770.1520120.0351059False0.2948560.03766530.3230450.0470061False0.4375860.04052520.4617660.0513177False0.5501470.05870840.5756460.0752913False0.6777130.243060.8531010.297092FalsenannannannanFalse
742089330881070516150.3059339841.0497284968824.76120.065812727.37133.824124.42470.0561313nannan24.08730.0476167nannan23.88120.0585796nannan23.42870.16337423.97311.6669nannannannan0.00.4523810.02742140.0408780.143977False0.6167860.0318871nannanFalse0.8415510.0369075nannanFalse1.01750.0548978nannanFalse1.543530.2322590.9348981.43532FalsenannannannanFalse
842089330881070517150.2961409851.0496379448725.40720.13886nannan25.26640.142958nannan25.76290.251108nannan25.26160.223341nannan28.317414.6183nannannannannannan0.00.2495190.0319122nannanFalse0.2840720.0374035nannanFalse0.1798150.0415876nannanFalse0.2853480.0586974nannanFalse0.01710180.230259nannanFalsenannannannanFalse
942089330881070526150.2984052461.0498030639525.36250.13362325.32260.15553725.57390.1847225.46830.24022725.56450.20500625.42850.25654925.60270.30439226.00050.637337nannannannannannannannan0.00.2600060.03199930.2697580.0386443False0.2140090.03641020.2358670.0521872False0.2158770.04076150.2446760.0578145False0.208410.05842880.1444810.0848115FalsenannannannanFalsenannannannanFalse

III - Removal of duplicated sources¶

We remove duplicated objects from the input catalogues.

In [33]:
SORT_COLS = [
        'merr_ap_hsc-deep_i', 'merr_ap_hsc-deep_r', 'merr_ap_hsc-deep_z',
        'merr_ap_hsc-deep_y', 'merr_ap_hsc-deep_g', 'merr_ap_hsc-deep_n921']
FLAG_NAME = 'hsc-deep_flag_cleaned'

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(
    catalogue, RA_COL, DEC_COL, 
    sort_col= SORT_COLS,
    flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
The initial catalogue had 1966391 sources.
The cleaned catalogue has 1966275 sources (116 removed).
The cleaned catalogue has 99 sources flagged as having been cleaned

III - Astrometry correction¶

We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.

In [34]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_COSMOS.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [35]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)