GAMA-15 master catalogue

Preparation of Hyper Suprime-Cam Subaru Strategic Program Catalogues (HSC-SSP) data

This catalogue comes from dmu0_HSC.

In the catalogue, we keep:

  • The object_id as unique object identifier;
  • The position;
  • The g, r, i, z, y (no N921) aperture magnitude in 2” that we aperture correct;
  • The g, r, i, z, y (no N921) kron fluxes and magnitudes.
  • The extended flag that we convert to a stellariy.

Note: On ELAIS-N1 the HSC-SSP catalogue does not contain any N816 magnitudes.

We use 2016 as the epoch.

In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
This notebook was run with herschelhelp_internal version: 
44f1ae0 (Thu Nov 30 18:27:54 2017 +0000)
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
plt.style.use('ggplot')

from collections import OrderedDict
import os

from astropy import units as u
from astropy import visualization as vis
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, nb_plot_mag_ap_evol, \
    nb_plot_mag_vs_apcor, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux, aperture_correction
In [3]:
OUT_DIR =  os.environ.get('TMP_DIR', "./data_tmp")
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "hsc_ra"
DEC_COL = "hsc_dec"
In [4]:
# Pritine HSC catalogue
orig_hsc = Table.read("../../dmu0/dmu0_HSC/data/HSC-PDR1_wide_GAMA-15_v2.fits")

I - Aperture correction

To compute aperture correction we need to dertermine two parametres: the target aperture and the range of magnitudes for the stars that will be used to compute the correction.

Target aperture: To determine the target aperture, we simulate a curve of growth using the provided apertures and draw two figures:

  • The evolution of the magnitudes of the objects by plotting on the same plot aperture number vs the mean magnitude.
  • The mean gain (loss when negative) of magnitude is each aperture compared to the previous (except for the first of course).

As target aperture, we should use the smallest (i.e. less noisy) aperture for which most of the flux is captures.

Magnitude range: To know what limits in aperture to use when doing the aperture correction, we plot for each magnitude bin the correction that is computed and its RMS. We should then use the wide limits (to use more stars) where the correction is stable and with few dispersion.

In [5]:
bands = ["g", "r", "i", "z", "y"] #, "n921"]
apertures = ["10", "15", "20", "30", "40", "57", "84", "118", "168", "235"]

magnitudes = {}
magnitude_errors ={}
stellarities = {}

for band in bands:
    magnitudes[band] = np.array(
        [orig_hsc["{}mag_aperture{}".format(band, aperture)] for aperture in apertures]
    )
    # Some sources have an infinite magnitude
    mask = np.isinf(magnitudes[band])
    magnitudes[band][mask] = np.nan
    try:
        magnitude_errors[band] = np.array(
        [orig_hsc["{}mag_aperture{}_err".format(band, aperture)] for aperture in apertures]
        )
        magnitude_errors[band][mask] = np.nan
    except KeyError:
        print("No error column for a " + band + " band aperture magnitude.")
        
    stellarities[band] = 1 - np.array(orig_hsc["{}classification_extendedness".format(band)])
    
mag_corr = {}
No error column for a y band aperture magnitude.

I.a - g band

In [6]:
nb_plot_mag_ap_evol(magnitudes['g'], stellarities['g'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [7]:
nb_plot_mag_vs_apcor(orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'], stellarities['g'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We will use magnitudes between 18.5 and 20.8

In [8]:
# Aperture correction
mag_corr['g'], num, std = aperture_correction(
    orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'], 
    stellarities['g'],
    mag_min=18.5, mag_max=20.8)
print("Aperture correction for g band:")
print("Correction: {}".format(mag_corr['g']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for g band:
Correction: -0.10141181945800781
Number of source used: 15286
RMS: 0.027612784472519653
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.b - r band

In [9]:
nb_plot_mag_ap_evol(magnitudes['r'], stellarities['r'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [10]:
nb_plot_mag_vs_apcor(orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'], stellarities['r'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17.6 and 19.7.

In [11]:
# Aperture correction
mag_corr['r'], num, std = aperture_correction(
    orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'], 
    stellarities['r'],
    mag_min=17.6, mag_max=19.7)
print("Aperture correction for r band:")
print("Correction: {}".format(mag_corr['r']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for r band:
Correction: -0.10055923461914062
Number of source used: 9943
RMS: 0.03515977580827814
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.c - i band

In [12]:
nb_plot_mag_ap_evol(magnitudes['i'], stellarities['i'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [13]:
nb_plot_mag_vs_apcor(orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'], stellarities['i'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 18.5 and 19.8.

In [14]:
# Aperture correction
mag_corr['i'], num, std = aperture_correction(
    orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'], 
    stellarities['i'],
    mag_min=18.5, mag_max=19.8)
print("Aperture correction for i band:")
print("Correction: {}".format(mag_corr['i']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for i band:
Correction: -0.0619354248046875
Number of source used: 24834
RMS: 0.00930339810580967
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.d - z band

In [15]:
nb_plot_mag_ap_evol(magnitudes['z'], stellarities['z'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [16]:
nb_plot_mag_vs_apcor(orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'], stellarities['z'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17.5 and 19.8.

In [17]:
# Aperture correction
mag_corr['z'], num, std = aperture_correction(
    orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'], 
    stellarities['z'],
    mag_min=17.5, mag_max=19.8)
print("Aperture correction for z band:")
print("Correction: {}".format(mag_corr['z']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for z band:
Correction: -0.08566570281982422
Number of source used: 34829
RMS: 0.01121795488087638
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

I.e - y band

In [18]:
nb_plot_mag_ap_evol(magnitudes['y'], stellarities['y'], labels=apertures)
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:850: RuntimeWarning: invalid value encountered in greater
  mags = magnitudes[:, stellarity > stel_threshold].copy()

We will use aperture 40 as target.

In [19]:
nb_plot_mag_vs_apcor(orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'], stellarities['y'])
/opt/herschelhelp_internal/herschelhelp_internal/masterlist.py:903: RuntimeWarning: invalid value encountered in greater
  mask = stellarity > .9
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

We use magnitudes between 17 and 18.7.

In [20]:
# Aperture correction
mag_corr['y'], num, std = aperture_correction(
    orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'], 
    stellarities['y'],
    mag_min=17, mag_max=18.7)
print("Aperture correction for y band:")
print("Correction: {}".format(mag_corr['y']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for y band:
Correction: -0.11461448669433594
Number of source used: 9295
RMS: 0.015907751991162174
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:129: RuntimeWarning: invalid value encountered in greater
  mask &= (stellarity > 0.9)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:131: RuntimeWarning: invalid value encountered in greater_equal
  mask &= (mag >= mag_min)
/opt/herschelhelp_internal/herschelhelp_internal/utils.py:133: RuntimeWarning: invalid value encountered in less_equal
  mask &= (mag <= mag_max)

II - Stellarity

HSC does not provide a 0 to 1 stellarity value but a 0/1 extended flag in each band. We are using the same method as UKIDSS (cf this page) to compute a stellarity based on the class in each band:

\begin{equation*} P(star) = \frac{ \prod_{i} P(star)_i }{ \prod_{i} P(star)_i + \prod_{i} P(galaxy)_i } \end{equation*}

where $i$ is the band, and with using the same probabilities as UKDISS:

HSC flag UKIDSS flag Meaning P(star) P(galaxy) P(noise) P(saturated)
-9 Saturated 0.0 0.0 5.0 95.0
-3 Probable galaxy 25.0 70.0 5.0 0.0
-2 Probable star 70.0 25.0 5.0 0.0
0 -1 Star 90.0 5.0 5.0 0.0
0 Noise 5.0 5.0 90.0 0.0
1 +1 Galaxy 5.0 90.0 5.0 0.0
In [21]:
# We are creating an array containing the extended flag in all band.
# Some sources have no flag in some band, there will be NaN in the array.
hsc_ext_flag = np.array([
    orig_hsc[colname] for colname in 
    ['gclassification_extendedness',
     'rclassification_extendedness',
     'iclassification_extendedness',
     'zclassification_extendedness',
     'yclassification_extendedness']
])
In [22]:
hsc_pstar = 0.9 * (hsc_ext_flag == 0) + 0.05 * (hsc_ext_flag == 1)
hsc_pgal = 0.05 * (hsc_ext_flag == 0) + 0.9 * (hsc_ext_flag == 1)

# We put back the NaN values
hsc_pstar[np.isnan(hsc_ext_flag)] = np.nan
hsc_pgal[np.isnan(hsc_ext_flag)] = np.nan
In [23]:
stellarity = np.nanprod(hsc_pstar, axis=0) / np.nansum(
    [np.nanprod(hsc_pgal, axis=0), np.nanprod(hsc_pstar, axis=0)], axis=0)

stellarity = np.round(stellarity, 3)
In [24]:
vis.hist(stellarity, bins='scott');
In [25]:
orig_hsc.add_column(Column(data=stellarity, name="stellarity"))

II - Column selection

In [26]:
imported_columns = OrderedDict({
        "object_id": "hsc_id",
        "ra": "hsc_ra",
        "dec": "hsc_dec",
        "gmag_aperture20": "m_ap_suprime_g",
        "gmag_aperture20_err": "merr_ap_suprime_g",
        "gmag_kron": "m_suprime_g",
        "gmag_kron_err": "merr_suprime_g",
        "rmag_aperture20": "m_ap_suprime_r",
        "rmag_aperture20_err": "merr_ap_suprime_r",
        "rmag_kron": "m_suprime_r",
        "rmag_kron_err": "merr_suprime_r",
        "imag_aperture20": "m_ap_suprime_i",
        "imag_aperture20_err": "merr_ap_suprime_i",
        "imag_kron": "m_suprime_i",
        "imag_kron_err": "merr_suprime_i",
        "zmag_aperture20": "m_ap_suprime_z",
        "zmag_aperture20_err": "merr_ap_suprime_z",
        "zmag_kron": "m_suprime_z",
        "zmag_kron_err": "merr_suprime_z",
        "ymag_aperture20": "m_ap_suprime_y",
        "ymag_aperture20_err": "merr_ap_suprime_y",
        "ymag_kron": "m_suprime_y",
        "ymag_kron_err": "merr_suprime_y",
        "stellarity": "hsc_stellarity"
    })


catalogue = orig_hsc[list(imported_columns)]
for column in imported_columns:
    catalogue[column].name = imported_columns[column]

epoch = 2017

# Clean table metadata
catalogue.meta = None
In [27]:
# Aperture correction
for band in bands:
    catalogue["m_ap_suprime_{}".format(band)] += mag_corr[band]
In [28]:
# Adding flux and band-flag columns
for col in catalogue.colnames:
    if col.startswith('m_'):
        
        errcol = "merr{}".format(col[1:])
        flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))
        
        # Fluxes are added in µJy
        catalogue.add_column(Column(flux * 1.e6, name="f{}".format(col[1:])))
        catalogue.add_column(Column(error * 1.e6, name="f{}".format(errcol[1:])))
        
        # Band-flag column
        if 'ap' not in col:
            catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))
        
# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting.
In [29]:
catalogue[:10].show_in_notebook()
Out[29]:
<Table masked=True length=10>
idxhsc_idhsc_rahsc_decm_ap_suprime_gmerr_ap_suprime_gm_suprime_gmerr_suprime_gm_ap_suprime_rmerr_ap_suprime_rm_suprime_rmerr_suprime_rm_ap_suprime_imerr_ap_suprime_im_suprime_imerr_suprime_im_ap_suprime_zmerr_ap_suprime_zm_suprime_zmerr_suprime_zm_ap_suprime_ymerr_ap_suprime_ym_suprime_ymerr_suprime_yhsc_stellarityf_ap_suprime_gferr_ap_suprime_gf_suprime_gferr_suprime_gflag_suprime_gf_ap_suprime_rferr_ap_suprime_rf_suprime_rferr_suprime_rflag_suprime_rf_ap_suprime_iferr_ap_suprime_if_suprime_iferr_suprime_iflag_suprime_if_ap_suprime_zferr_ap_suprime_zf_suprime_zferr_suprime_zflag_suprime_zf_ap_suprime_yferr_ap_suprime_yf_suprime_yferr_suprime_yflag_suprime_y
041209704398979309214.038904962-1.2103412564727.53170.662073nannan26.38360.365409nannan26.24850.431549nannan25.10250.264895nannan24.21240.311641nannan0.00.03526240.0215028nannanFalse0.1015190.0341667nannanFalse0.1149760.0456995nannanFalse0.3303590.0805999nannanFalse0.7499520.21526nannanFalse
141209704398979312214.034272374-1.2102378475326.36330.224626.20110.25328525.31890.13444825.18750.15822425.12120.15461425.09560.21100325.6440.44920825.74160.66366424.4980.38091224.09740.3577010.00.1034340.02139670.1201080.0280194False0.2706660.03351690.3054830.0445178False0.3247330.04624370.3324720.0646128False0.2006370.08301060.1833830.112094False0.5765190.2022620.8337320.274677False
241209704398979314214.053083715-1.2101460683326.41990.24708526.33520.29577225.30170.13628125.29520.17686125.28240.17075725.47670.28114225.04680.25686624.80750.27332224.84410.55817424.37170.4873290.00.0981850.02234430.1061460.0289158False0.2749920.03451690.2766490.0450648False0.2799220.04402420.2340630.0606086False0.3477730.0822770.4335030.109129False0.419140.2154790.647610.290678False
341209704398979320214.035263153-1.2100466169625.70780.12554625.4510.15340525.63150.17941925.25860.19922525.28480.17269924.83170.18579424.68390.18389424.550.25681324.7540.50132524.19940.4680891.00.1891770.02187490.2396720.0338637False0.2029570.03353890.2861320.0525033False0.2793020.04442620.4239660.0725503False0.4857830.08227810.5495590.129989False0.4553940.2102720.7590010.327225False
441209704398979323214.017759272-1.2099227545426.50380.29948926.71660.30027325.75480.23561525.71730.18965424.68520.091436824.75450.08538824.51420.14959424.65590.14440323.85960.18687223.97820.1752940.00.0908810.02506860.07470340.0206601False0.1811620.03931390.1875340.0327581False0.485210.04086260.4552020.0357995False0.5679490.07825280.4984780.0662976False1.037940.1786460.9304960.15023False
541209704398979327214.045703206-1.2097534449825.5330.10357825.14160.13872725.11610.10993824.5890.13348224.80720.097154224.41960.14133824.86720.20826724.3460.255501nannan25.33181.606180.00.2222370.02120120.3186790.0407182False0.326270.03303690.5301750.0651807False0.4336250.03880170.6196530.0806645False0.4103050.07870520.6631540.156057Falsenannan0.2674750.395688False
641209704398979334213.989195362-1.2096818717526.29140.219505nannan26.3780.380161nannan25.41510.170057nannan25.45240.331199nannannannannannan0.00.1105250.022345nannanFalse0.1020460.0357304nannanFalse0.2477220.0388002nannanFalse0.2393630.0730168nannanFalsenannannannanFalse
741209704398979347214.027982555-1.2094331321525.20220.0780124.88410.11106625.65160.18574824.92590.18829425.85670.2584525.65910.4410725.35350.32690925.02710.47989426.38421.8874823.99010.4104490.00.3013750.02165380.4039790.0413252False0.1992380.03408560.3887140.0674129False0.1649330.03926090.1978550.0803766False0.262170.07893790.3541290.156524False0.1014630.1763870.920370.347935False
841209704398979361214.050204194-1.2092124337625.92910.15407725.7570.15391725.60430.1839725.30520.16487525.24050.17017625.14760.19313525.22480.30310525.3740.41862225.64161.1837726.3722.783690.00.1542940.02189590.1807960.0256302False0.2081020.03526140.2741160.0416261False0.2909350.04560060.3169140.0563739False0.2951820.08240610.2572790.0991978False0.2010770.2192320.1026140.263088False
941209704398979362214.026991288-1.209322076627.56660.67368628.2170.86386626.57780.43849726.6350.32804426.75190.55434926.4060.30085226.58220.97590726.65170.75524225.74821.0576926.23111.185530.00.03414780.02118830.0187580.0149248False0.08489120.0342850.08053940.0243342False0.07231380.03692160.09944520.0275558False0.08455180.07599890.0793120.0551697False0.1822670.1775590.116830.127569False

III - Removal of duplicated sources

We remove duplicated objects from the input catalogues.

In [30]:
SORT_COLS = [
        'merr_ap_suprime_i', 'merr_ap_suprime_r', 'merr_ap_suprime_z',
        'merr_ap_suprime_y', 'merr_ap_suprime_g']
FLAG_NAME = 'hsc_flag_cleaned'

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(
    catalogue, RA_COL, DEC_COL, 
    sort_col= SORT_COLS,
    flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
The initial catalogue had 6952860 sources.
The cleaned catalogue has 6952601 sources (259 removed).
The cleaned catalogue has 258 sources flagged as having been cleaned

III - Astrometry correction

We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.

In [31]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_GAMA-15.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [32]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)
In [33]:
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
    gaia_coords
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))
RA correction: -0.10107775475489689 arcsec
Dec correction: -0.06822208740950853 arcsec
In [34]:
catalogue[RA_COL] +=  delta_ra.to(u.deg)
catalogue[DEC_COL] += delta_dec.to(u.deg)
In [35]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)

IV - Flagging Gaia objects

In [36]:
catalogue.add_column(
    gaia_flag_column(SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]), epoch, gaia)
)
In [37]:
GAIA_FLAG_NAME = "hsc_flag_gaia"

catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(catalogue[GAIA_FLAG_NAME] > 0)))
43661 sources flagged.

V - Flagging objects near bright stars

VI - Saving to disk

In [38]:
catalogue.write("{}/HSC-SSP.fits".format(OUT_DIR), overwrite=True)