EGS master catalogue

Preparation of Hyper Suprime-Cam Subaru Strategic Program Catalogues (HSC-SSP) wide data

This catalogue comes from dmu0_HSC. We only have n921 and n816 photometry on the ultradeep field.

In the catalogue, we keep:

  • The object_id as unique object identifier;
  • The position;
  • The g, r, i, z, y aperture magnitude in 2” that we aperture correct;
  • The g, r, i, z, y kron fluxes and magnitudes.
  • The extended flag that we convert to a stellariy.

We don't know when the maps have been observed. We will use the year of the reference paper.

In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
This notebook was run with herschelhelp_internal version: 
44f1ae0 (Thu Nov 30 18:27:54 2017 +0000)
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
plt.style.use('ggplot')

from collections import OrderedDict
import os

from astropy import units as u
from astropy import visualization as vis
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, nb_plot_mag_ap_evol, \
    nb_plot_mag_vs_apcor, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux, aperture_correction
In [3]:
OUT_DIR =  os.environ.get('TMP_DIR', "./data_tmp")
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "hsc_ra"
DEC_COL = "hsc_dec"
In [4]:
# Pritine HSC catalogue
orig_hsc = Table.read("../../dmu0/dmu0_HSC/data/HSC-PDR1_wide_EGS_v2.fits")

I - Aperture correction

To compute aperture correction we need to dertermine two parametres: the target aperture and the range of magnitudes for the stars that will be used to compute the correction.

Target aperture: To determine the target aperture, we simulate a curve of growth using the provided apertures and draw two figures:

  • The evolution of the magnitudes of the objects by plotting on the same plot aperture number vs the mean magnitude.
  • The mean gain (loss when negative) of magnitude is each aperture compared to the previous (except for the first of course).

As target aperture, we should use the smallest (i.e. less noisy) aperture for which most of the flux is captures.

Magnitude range: To know what limits in aperture to use when doing the aperture correction, we plot for each magnitude bin the correction that is computed and its RMS. We should then use the wide limits (to use more stars) where the correction is stable and with few dispersion.

In [5]:
bands = ["g", "r", "i", "z", "y"]
apertures = ["10", "15", "20", "30",  "57", "84", "118", "168"] #Removed "40" and "235" because they lack errors

magnitudes = {}
magnitude_errors ={}
stellarities = {}

for band in bands:
    magnitudes[band] = np.array(
        [orig_hsc["{}mag_aperture{}".format(band, aperture)] for aperture in apertures]
    )
    magnitude_errors[band] = np.array(
        [orig_hsc["{}mag_aperture{}_err".format(band, aperture)] for aperture in apertures]
    )
    stellarities[band] = 1 - np.array(orig_hsc["{}classification_extendedness".format(band)])
    
    # Some sources have an infinite magnitude
    mask = np.isinf(magnitudes[band])
    magnitudes[band][mask] = np.nan
    magnitude_errors[band][mask] = np.nan
    
mag_corr = {}

I.a - g band

In [6]:
nb_plot_mag_ap_evol(magnitudes['g'], stellarities['g'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 57 as target.

In [7]:
nb_plot_mag_vs_apcor(orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture57'], stellarities['g'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We will use magnitudes between 18.5 and 20.8

In [8]:
# Aperture correction
mag_corr['g'], num, std = aperture_correction(
    orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture57'], 
    stellarities['g'],
    mag_min=18.5, mag_max=20.8)
print("Aperture correction for g band:")
print("Correction: {}".format(mag_corr['g']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for g band:
Correction: -0.1536693572998047
Number of source used: 477
RMS: 0.013336935836438763
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.b - r band

In [9]:
nb_plot_mag_ap_evol(magnitudes['r'], stellarities['r'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 57 as target.

In [10]:
nb_plot_mag_vs_apcor(orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture57'], stellarities['r'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 19.5 and 20.5.

In [11]:
# Aperture correction
mag_corr['r'], num, std = aperture_correction(
    orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture57'], 
    stellarities['r'],
    mag_min=19.5, mag_max=20.5)
print("Aperture correction for r band:")
print("Correction: {}".format(mag_corr['r']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for r band:
Correction: -0.11204338073730469
Number of source used: 606
RMS: 0.012112590040337011
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.c - i band

In [12]:
nb_plot_mag_ap_evol(magnitudes['i'], stellarities['i'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 57 as target.

In [13]:
nb_plot_mag_vs_apcor(orig_hsc['imag_aperture20'], orig_hsc['imag_aperture57'], stellarities['i'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 18.5 and 19.8.

In [14]:
# Aperture correction
mag_corr['i'], num, std = aperture_correction(
    orig_hsc['imag_aperture20'], orig_hsc['imag_aperture57'], 
    stellarities['i'],
    mag_min=18.5, mag_max=19.8)
print("Aperture correction for i band:")
print("Correction: {}".format(mag_corr['i']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for i band:
Correction: -0.06093025207519531
Number of source used: 831
RMS: 0.007157356612245586
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.d - z band

In [15]:
nb_plot_mag_ap_evol(magnitudes['z'], stellarities['z'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 57 as target.

In [16]:
nb_plot_mag_vs_apcor(orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture57'], stellarities['z'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17.5 and 18.8.

In [17]:
# Aperture correction
mag_corr['z'], num, std = aperture_correction(
    orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture57'], 
    stellarities['z'],
    mag_min=17.5, mag_max=18.8)
print("Aperture correction for z band:")
print("Correction: {}".format(mag_corr['z']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for z band:
Correction: -0.1141510009765625
Number of source used: 333
RMS: 0.00688200402082919
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.e - y band

In [18]:
nb_plot_mag_ap_evol(magnitudes['y'], stellarities['y'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 57 as target.

In [19]:
nb_plot_mag_vs_apcor(orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture57'], stellarities['y'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17.6 and 18.7.

In [20]:
# Aperture correction
mag_corr['y'], num, std = aperture_correction(
    orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture57'], 
    stellarities['y'],
    mag_min=17.6, mag_max=18.7)
print("Aperture correction for y band:")
print("Correction: {}".format(mag_corr['y']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for y band:
Correction: -0.15555763244628906
Number of source used: 334
RMS: 0.015017790515423833
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

II - Stellarity

HSC does not provide a 0 to 1 stellarity value but a 0/1 extended flag in each band. We are using the same method as UKIDSS (cf this page) to compute a stellarity based on the class in each band:

\begin{equation*} P(star) = \frac{ \prod_{i} P(star)_i }{ \prod_{i} P(star)_i + \prod_{i} P(galaxy)_i } \end{equation*}

where $i$ is the band, and with using the same probabilities as UKDISS:

HSC flag UKIDSS flag Meaning P(star) P(galaxy) P(noise) P(saturated)
-9 Saturated 0.0 0.0 5.0 95.0
-3 Probable galaxy 25.0 70.0 5.0 0.0
-2 Probable star 70.0 25.0 5.0 0.0
0 -1 Star 90.0 5.0 5.0 0.0
0 Noise 5.0 5.0 90.0 0.0
1 +1 Galaxy 5.0 90.0 5.0 0.0
In [21]:
# We are creating an array containing the extended flag in all band.
# Some sources have no flag in some band, there will be NaN in the array.
hsc_ext_flag = np.array([
    orig_hsc[colname] for colname in 
    ['gclassification_extendedness',
     'rclassification_extendedness',
     'iclassification_extendedness',
     'zclassification_extendedness',
     'yclassification_extendedness']
])
In [22]:
hsc_pstar = 0.9 * (hsc_ext_flag == 0) + 0.05 * (hsc_ext_flag == 1)
hsc_pgal = 0.05 * (hsc_ext_flag == 0) + 0.9 * (hsc_ext_flag == 1)

# We put back the NaN values
hsc_pstar[np.isnan(hsc_ext_flag)] = np.nan
hsc_pgal[np.isnan(hsc_ext_flag)] = np.nan
In [23]:
stellarity = np.nanprod(hsc_pstar, axis=0) / np.nansum(
    [np.nanprod(hsc_pgal, axis=0), np.nanprod(hsc_pstar, axis=0)], axis=0)

stellarity = np.round(stellarity, 3)
In [24]:
vis.hist(stellarity, bins='scott');
In [25]:
orig_hsc.add_column(Column(data=stellarity, name="stellarity"))

II - Column selection

In [26]:
imported_columns = OrderedDict({
        "object_id": "hsc_id",
        "ra": "hsc_ra",
        "dec": "hsc_dec",
        "gmag_aperture20": "m_ap_suprime_g",
        "gmag_aperture20_err": "merr_ap_suprime_g",
        "gmag_kron": "m_suprime_g",
        "gmag_kron_err": "merr_suprime_g",
        "rmag_aperture20": "m_ap_suprime_r",
        "rmag_aperture20_err": "merr_ap_suprime_r",
        "rmag_kron": "m_suprime_r",
        "rmag_kron_err": "merr_suprime_r",
        "imag_aperture20": "m_ap_suprime_i",
        "imag_aperture20_err": "merr_ap_suprime_i",
        "imag_kron": "m_suprime_i",
        "imag_kron_err": "merr_suprime_i",
        "zmag_aperture20": "m_ap_suprime_z",
        "zmag_aperture20_err": "merr_ap_suprime_z",
        "zmag_kron": "m_suprime_z",
        "zmag_kron_err": "merr_suprime_z",
        "ymag_aperture20": "m_ap_suprime_y",
        "ymag_aperture20_err": "merr_ap_suprime_y",
        "ymag_kron": "m_suprime_y",
        "ymag_kron_err": "merr_suprime_y",
        "stellarity": "hsc_stellarity"
    })


catalogue = orig_hsc[list(imported_columns)]
for column in imported_columns:
    catalogue[column].name = imported_columns[column]

epoch = 2017

# Clean table metadata
catalogue.meta = None
In [27]:
# Aperture correction
for band in bands:
    catalogue["m_ap_suprime_{}".format(band)] += mag_corr[band]
In [28]:
# Adding flux and band-flag columns
for col in catalogue.colnames:
    if col.startswith('m_'):
        
        errcol = "merr{}".format(col[1:])
        flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))
        
        # Fluxes are added in µJy
        catalogue.add_column(Column(flux * 1.e6, name="f{}".format(col[1:])))
        catalogue.add_column(Column(error * 1.e6, name="f{}".format(errcol[1:])))
        
        # Band-flag column
        if 'ap' not in col:
            catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))
        
# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting.
In [29]:
catalogue[:10].show_in_notebook()
Out[29]:
<Table masked=True length=10>
idxhsc_idhsc_rahsc_decm_ap_suprime_gmerr_ap_suprime_gm_suprime_gmerr_suprime_gm_ap_suprime_rmerr_ap_suprime_rm_suprime_rmerr_suprime_rm_ap_suprime_imerr_ap_suprime_im_suprime_imerr_suprime_im_ap_suprime_zmerr_ap_suprime_zm_suprime_zmerr_suprime_zm_ap_suprime_ymerr_ap_suprime_ym_suprime_ymerr_suprime_yhsc_stellarityf_ap_suprime_gferr_ap_suprime_gf_suprime_gferr_suprime_gflag_suprime_gf_ap_suprime_rferr_ap_suprime_rf_suprime_rferr_suprime_rflag_suprime_rf_ap_suprime_iferr_ap_suprime_if_suprime_iferr_suprime_iflag_suprime_if_ap_suprime_zferr_ap_suprime_zf_suprime_zferr_suprime_zflag_suprime_zf_ap_suprime_yferr_ap_suprime_yf_suprime_yferr_suprime_yflag_suprime_y
073979570428051653213.86508062351.783517567924.14350.043108124.0480.05865723.42510.040911923.27740.054920122.82930.019780422.65310.027401522.17640.027418722.00380.034752221.8650.046889221.7970.06644180.00.7990980.03172740.8725860.0471415False1.548740.05835851.774360.0897529False2.680890.04884153.15320.0795795False4.891510.1235285.734430.183548False6.516060.2814066.937150.42452False
173979570428051665213.86685131851.783572859125.27790.12050125.56660.32040124.99510.17565624.91990.33482624.48240.090678524.2680.16293124.11610.15956223.5080.18459723.41280.19805522.8250.2361850.00.2810740.03119520.2154480.0635787False0.3647210.05900670.3908620.120536False0.5848380.04884450.7124960.10692False0.8194870.1204331.434820.243948False1.566310.2857192.691440.585482False
273979570428051673213.82127404251.784249544522.63120.011638522.76210.010909721.40220.0068566821.48330.0064283720.17160.0019096620.17720.0017910619.66850.0028995519.77470.0027522319.4550.0053665719.55610.005002781.03.217470.03448952.852170.0286592False9.979930.06302569.261440.0548346False31.00080.054526130.83920.0508731False49.27270.13158744.68080.113261False59.98010.29646954.64520.25179False
373979570428051683213.8386870251.784177791925.98090.23471726.43551.1999826.09940.48232425.05920.6461625.83740.32861824.18970.25611625.55730.60559524.09290.53974924.71560.651893nannan0.00.1471150.03180370.09678270.106966False0.1319040.05859650.3438120.204615False0.1678910.05081510.7658150.180649False0.2173140.1212120.8372160.416203False0.4717990.283276nannanFalse
473979570428051685213.89203519651.783767496726.68660.444565nannan25.91750.41158425.02060.78380325.17250.17027925.60911.1928926.24251.1122224.7091.19628nannannannan0.00.07680110.031447nannanFalse0.1559550.05911990.3562410.257173False0.3097390.04857720.2071910.227639False0.1156160.1184360.4746980.52303FalsenannannannanFalse
573979570428051689213.79005147751.784725410227.70341.1964nannan26.43050.66878427.08510.83556526.40070.60687925.98250.29032625.98631.0421925.96920.68556927.29448.4671427.23125.40350.0530.03010420.0331726nannanFalse0.0972320.05989230.0532040.0409449False0.09993240.05585780.1468860.0392772False0.1463810.140510.1486970.0938919False0.0438790.3421920.04650740.231459False
673979570428051692213.7988568451.784817063425.69290.1848525.79260.18391225.56630.30018325.61950.3006325.02010.14609325.01250.14695225.02180.37053925.03790.353096nannannannan0.00.1917970.0326540.1749610.0296366False0.2155080.05958320.2052070.0568199False0.356410.04795720.3589180.0485789False0.3558560.1214460.3506350.114031FalsenannannannanFalse
773979570428051696213.89477958951.783934675125.70610.18030425.73470.21556325.64420.31037625.75750.41521225.64540.26367325.27040.23680526.62141.5904126.55721.7571425.44041.3059725.46091.536910.00.1894850.03146710.1845450.0366399False0.2005860.05734080.1807150.0691098False0.2003820.0486630.2830390.0617322False0.0815510.1194570.08652060.140024False0.2420070.2910960.2374850.33617False
873979570428051711213.80024498351.785309502724.00650.039747223.90280.043826523.85910.062584523.84650.077275923.71050.045367523.59450.054606423.17090.067521323.11230.079941422.9760.13128823.00550.1691250.00.9065360.03318690.9974440.0402626False1.03840.05985581.050510.0747688False1.190660.04975181.324940.0666372False1.957160.1217152.065770.1521False2.34210.2832082.279370.355058False
973979570428051714213.86114018351.784884320424.11560.044368823.84650.06424923.24760.035391423.08010.060800922.89880.021142322.63860.035675222.75380.044672122.40270.065984822.66240.10228422.43520.1600160.00.819920.03350621.050550.0621671False1.823760.05944852.127880.119161False2.514670.04896753.195630.105002False2.874020.118253.971050.241337False3.12650.294543.854180.56803False

III - Removal of duplicated sources

We remove duplicated objects from the input catalogues.

In [30]:
SORT_COLS = [
        'merr_ap_suprime_i', 'merr_ap_suprime_r', 'merr_ap_suprime_z',
        'merr_ap_suprime_y', 'merr_ap_suprime_g']
FLAG_NAME = 'hsc_flag_cleaned'

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(
    catalogue, RA_COL, DEC_COL, 
    sort_col= SORT_COLS,
    flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
The initial catalogue had 444602 sources.
The cleaned catalogue has 444573 sources (29 removed).
The cleaned catalogue has 29 sources flagged as having been cleaned

III - Astrometry correction

We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.

In [31]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_EGS.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [32]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)
In [33]:
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
    gaia_coords
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))
RA correction: -0.03406401734764586 arcsec
Dec correction: -0.03037861431209876 arcsec
In [34]:
catalogue[RA_COL] +=  delta_ra.to(u.deg)
catalogue[DEC_COL] += delta_dec.to(u.deg)
In [35]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)

IV - Flagging Gaia objects

In [36]:
catalogue.add_column(
    gaia_flag_column(SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]), epoch, gaia)
)
In [37]:
GAIA_FLAG_NAME = "hsc_flag_gaia"

catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(catalogue[GAIA_FLAG_NAME] > 0)))
2264 sources flagged.

V - Saving to disk

In [38]:
catalogue.write("{}/HSC.fits".format(OUT_DIR), overwrite=True)