SSDF master catalogue¶

Preparation of DES data¶

Blanco DES catalogue: the catalogue comes from dmu0_DES.

In the catalogue, we keep:

  • The identifier (it's unique in the catalogue);
  • The position;
  • The G band stellarity;
  • The magnitude for each band.
  • The auto/kron magnitudes/fluxes to be used as total magnitude.
  • The aperture magnitudes, which are used to compute a corrected 2 arcsec aperture magnitude.

We don't know when the maps have been observed. We will take the final observation date as 2017.

In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
This notebook was run with herschelhelp_internal version: 
33f5ec7 (Wed Dec 6 16:56:17 2017 +0000)
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))

from collections import OrderedDict
import os

from astropy import units as u
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, remove_duplicates, \
                                            nb_plot_mag_ap_evol, nb_plot_mag_vs_apcor 
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux, flux_to_mag, aperture_correction
In [3]:
OUT_DIR =  os.environ.get('TMP_DIR', "./data_tmp")
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "des_ra"
DEC_COL = "des_dec"
In [4]:
# Pristine HSC catalogue
orig_des = Table.read("../../dmu0/dmu0_DES/data/DES-DR1_SSDF.fits")

1 - Aperture correction¶

To compute aperture correction we need to dertermine two parametres: the target aperture and the range of magnitudes for the stars that will be used to compute the correction.

Target aperture: To determine the target aperture, we simulate a curve of growth using the provided apertures and draw two figures:

The evolution of the magnitudes of the objects by plotting on the same plot aperture number vs the mean magnitude. The mean gain (loss when negative) of magnitude is each aperture compared to the previous (except for the first of course). As target aperture, we should use the smallest (i.e. less noisy) aperture for which most of the flux is captures.

Magnitude range: To know what limits in aperture to use when doing the aperture correction, we plot for each magnitude bin the correction that is computed and its RMS. We should then use the wide limits (to use more stars) where the correction is stable and with few dispersion.

In [5]:
bands = ["g", "r", "i", "z", "y"]
apertures = ["1", "2", "3", "4",  "5", "6", "7", "8", "9", "10", "11"] #Removed "40" and "235" because they lack errors

magnitudes = {}
stellarities = {}

for band in bands:
    magnitudes[band] = np.array(
        [orig_des["MAG_APER_{}_{}".format(aperture, band.upper())] for aperture in apertures]
    )

    stellarities[band] = 1 - np.array(orig_des["CLASS_STAR_{}".format(band.upper())])
    
    # Some sources have an infinite magnitude
    mask = np.isclose(magnitudes[band], 99.)
    magnitudes[band][mask] = np.nan

    
mag_corr = {}

I.a - g band¶

In [6]:
nb_plot_mag_ap_evol(magnitudes['g'], stellarities['g'], labels=apertures)

We will use aperture 10 as target.

In [7]:
nb_plot_mag_vs_apcor(orig_des['MAG_APER_4_G'], orig_des['MAG_APER_10_G'], stellarities['g'])

We will use magnitudes between 18.0 and 21.0

In [8]:
# Aperture correction
mag_corr['g'], num, std = aperture_correction(
    orig_des['MAG_APER_4_G'], orig_des['MAG_APER_10_G'], 
    stellarities['g'],
    mag_min=18.0, mag_max=21.0)
print("Aperture correction for g band:")
print("Correction: {}".format(mag_corr['g']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for g band:
Correction: -1.2006874084472656
Number of source used: 40414
RMS: 0.4168256512303092

I.b - r band¶

In [9]:
nb_plot_mag_ap_evol(magnitudes['r'], stellarities['r'], labels=apertures)

We will use aperture 10 as target.

In [10]:
nb_plot_mag_vs_apcor(orig_des['MAG_APER_4_R'], orig_des['MAG_APER_10_R'], stellarities['r'])

We use magnitudes between 17.0 and 20.0.

In [11]:
# Aperture correction
mag_corr['r'], num, std = aperture_correction(
    orig_des['MAG_APER_4_R'], orig_des['MAG_APER_10_R'], 
    stellarities['r'],
    mag_min=17.0, mag_max=20.0)
print("Aperture correction for r band:")
print("Correction: {}".format(mag_corr['r']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for r band:
Correction: -1.0379772186279297
Number of source used: 44493
RMS: 0.2995299623881016

I.b - i band¶

In [12]:
nb_plot_mag_ap_evol(magnitudes['i'], stellarities['i'], labels=apertures)

We will use aperture 10 as target.

In [13]:
nb_plot_mag_vs_apcor(orig_des['MAG_APER_4_I'], orig_des['MAG_APER_10_I'], stellarities['i'])

We use magnitudes between 17.5 and 20.0.

In [14]:
# Aperture correction
mag_corr['i'], num, std = aperture_correction(
    orig_des['MAG_APER_4_I'], orig_des['MAG_APER_10_I'], 
    stellarities['i'],
    mag_min=17.5, mag_max=20.0)
print("Aperture correction for i band:")
print("Correction: {}".format(mag_corr['i']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for i band:
Correction: -0.9119529724121094
Number of source used: 86369
RMS: 0.2773256722395292

I.b - z band¶

In [15]:
nb_plot_mag_ap_evol(magnitudes['z'], stellarities['z'], labels=apertures)

We will use aperture 57 as target.

In [16]:
nb_plot_mag_vs_apcor(orig_des['MAG_APER_4_Z'], orig_des['MAG_APER_10_Z'], stellarities['z'])

We use magnitudes between 17.5 and 20.0.

In [17]:
# Aperture correction
mag_corr['z'], num, std = aperture_correction(
    orig_des['MAG_APER_4_Z'], orig_des['MAG_APER_10_Z'], 
    stellarities['z'],
    mag_min=17.5, mag_max=20.0)
print("Aperture correction for z band:")
print("Correction: {}".format(mag_corr['z']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for z band:
Correction: -0.8320159912109375
Number of source used: 134595
RMS: 0.26560741563295165

I.b - y band¶

In [18]:
nb_plot_mag_ap_evol(magnitudes['y'], stellarities['y'], labels=apertures)

We will use aperture 10 as target.

In [19]:
nb_plot_mag_vs_apcor(orig_des['MAG_APER_4_Y'], orig_des['MAG_APER_10_Y'], stellarities['y'])

We use magnitudes between 17.0 and 19.0.

In [20]:
# Aperture correction
mag_corr['y'], num, std = aperture_correction(
    orig_des['MAG_APER_4_Y'], orig_des['MAG_APER_10_Y'], 
    stellarities['y'],
    mag_min=17.0, mag_max=19.0)
print("Aperture correction for y band:")
print("Correction: {}".format(mag_corr['y']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
Aperture correction for y band:
Correction: -1.0476741790771484
Number of source used: 33133
RMS: 0.3080793013624895

2 - Column selection¶

In [21]:
imported_columns = OrderedDict({
        'COADD_OBJECT_ID': "des_id",
        'RA': "des_ra",
        'DEC': "des_dec",
        'CLASS_STAR_G':  "des_stellarity",
    
        'MAG_AUTO_G': "m_decam_g", 
        'MAGERR_AUTO_G': "merr_decam_g", 
        'MAG_APER_4_G': "m_ap_decam_g", 
        'MAGERR_APER_4_G': "merr_ap_decam_g",
    
        'MAG_AUTO_R': "m_decam_r", 
        'MAGERR_AUTO_R': "merr_decam_r", 
        'MAG_APER_4_R': "m_ap_decam_r", 
        'MAGERR_APER_4_R': "merr_ap_decam_r",
    
        'MAG_AUTO_I': "m_decam_i", 
        'MAGERR_AUTO_I': "merr_decam_i", 
        'MAG_APER_4_I': "m_ap_decam_i", 
        'MAGERR_APER_4_I': "merr_ap_decam_i",
    
        'MAG_AUTO_Z': "m_decam_z", 
        'MAGERR_AUTO_Z': "merr_decam_z", 
        'MAG_APER_4_Z': "m_ap_decam_z", 
        'MAGERR_APER_4_Z': "merr_ap_decam_z",
    
        'MAG_AUTO_Y': "m_decam_y", 
        'MAGERR_AUTO_Y': "merr_decam_y", 
        'MAG_APER_4_Y': "m_ap_decam_y", 
        'MAGERR_APER_4_Y': "merr_ap_decam_y",

    })


catalogue = Table.read("../../dmu0/dmu0_DES/data/DES-DR1_SSDF.fits")[list(imported_columns)]
for column in imported_columns:
    catalogue[column].name = imported_columns[column]

epoch = 2017

# Clean table metadata
catalogue.meta = None
In [22]:
# Aperture correction
for band in bands:
    catalogue["m_ap_decam_{}".format(band)] += mag_corr[band]
In [23]:
# Adding flux and band-flag columns
for col in catalogue.colnames:

            
    if col.startswith('m_'):
        
        errcol = "merr{}".format(col[1:])
        
        # Some objects have -99.0 values
        mask = (np.isclose(catalogue[col], 99.) )
        catalogue[col][mask] = np.nan
        catalogue[errcol][mask] = np.nan
        
        flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))
            
        # Fluxes are added in µJy
        catalogue.add_column(Column(flux * 1.e6, name="f{}".format(col[1:])))
        catalogue.add_column(Column(error * 1.e6, name="f{}".format(errcol[1:])))
        
        # Band-flag column
        if 'ap' not in col:
            catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))
        
# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting.            
        
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
In [24]:
catalogue[:10].show_in_notebook()
Out[24]:
<Table masked=True length=10>
idxdes_iddes_rades_decdes_stellaritym_decam_gmerr_decam_gm_ap_decam_gmerr_ap_decam_gm_decam_rmerr_decam_rm_ap_decam_rmerr_ap_decam_rm_decam_imerr_decam_im_ap_decam_imerr_ap_decam_im_decam_zmerr_decam_zm_ap_decam_zmerr_ap_decam_zm_decam_ymerr_decam_ym_ap_decam_ymerr_ap_decam_yf_decam_gferr_decam_gflag_decam_gf_ap_decam_gferr_ap_decam_gf_decam_rferr_decam_rflag_decam_rf_ap_decam_rferr_ap_decam_rf_decam_iferr_decam_iflag_decam_if_ap_decam_iferr_ap_decam_if_decam_zferr_decam_zflag_decam_zf_ap_decam_zferr_ap_decam_zf_decam_yferr_decam_yflag_decam_yf_ap_decam_yferr_ap_decam_y
0123745523348.658766-51.299340.019635869190123.8303680420.14580872654923.12972640990.096942469477723.59319686890.14014747738823.04453659060.092983216047323.21147727970.16282241046422.89616775510.1189310699723.44438934330.4197209179422.41566085820.14778035879123.39300537111.1468827724522.01399040220.3558495342731.06623462860.143189766278False2.032869200540.1815094307091.326542756260.171230930795False2.198798154440.1883066282061.885424247970.282747613516False2.520768121420.2761238397621.52140298040.588139782659False3.924060158060.5341067337861.595136495831.68497150482False5.680725957781.86185528082
1123742696348.687463-51.2537820.0014523714780823.85732078550.16260027885422.99834442140.083155773580123.34067726140.12058778107222.84501266480.078244559466823.30065727230.22441723942822.43020057680.089810080826322.48163986210.20141784846821.69412422180.085413157939924.8192348485.1961088180521.9995841980.3974223434931.040091831960.155764559724False2.29436353380.1757236920821.673898404680.185912280972False2.642377934580.1904253505871.736749134290.358978953255False3.87186110310.3202731539153.692700244620.685042771024False7.626939673480.5999993369240.4288506382752.05239041804False5.756603545062.10714431499
2123742882348.51251-51.2572760.11451173573723.7794170380.1207077428723.08726310730.096985429525423.23491859440.091776408255122.63273620610.068872496485722.90360832210.11024572700322.2595195770.07068111747522.89826965330.22514185309422.11795806880.11786350607923.85533142091.659741759321.68161201480.2935592234131.117463083810.124235008464False2.113950224370.1888325816231.845153758760.155969363382False3.212961470570.2038107199712.503552315930.25421086996False4.530980250460.2949655228922.515892837160.52170386606False5.161985407090.5603660407941.041999310631.59288210288False7.715342216382.08605908091
3123742919348.361679-51.2617020.98287886381118.35867500310.0016582537209617.44635391240.0015003649750717.025012970.00069362740032416.37044525150.00071686244336916.46584701540.00060305616352715.81916809080.00053981150267616.20308876040.00076403986895515.54391288760.00054523482685916.12745285030.002091233152915.36807060240.00142450188287164.6379685650.251452906137False381.4682834180.527146141976562.3346076220.359249963562False1027.594803550.678474309357941.155086740.522750713669False1707.390118710.8488883594341198.849028970.843638124048False2200.061620631.104826579011285.342476582.47569417697False2586.853052113.39398928329
4123778078344.479975-53.9297740.438133925228.18745040896.5980501174924.9188442230.42949414253223.59972000120.12120802700523.41844367980.11689946800522.72288894650.091576479375422.63768386840.085893243551322.3019905090.12887194752722.07214736940.0981250256321.48551940920.20118865370821.26628303530.1885882020.01927612955010.117141593221False0.3912571713350.1547730126121.318596745530.147203813634False1.558197589170.1677686241112.956952998660.24940436186False3.198353425670.2530236613194.357162893020.517175410799False5.384446327520.4866273516319.242559101081.71266097069False11.31062974391.96461287515
5123797593344.689434-54.2213750.4346899092225.12724494930.30812731385223.96723556520.22461634874324.30704689030.16782763600323.49369812010.14480593800525.18005752560.64304608106624.16863060.41102990508124.41154670720.6142989993123.33981513980.3449282944223.84097480771.1754416227322.53244781490.6471475958820.3229252617440.0916448148642False0.9399522113060.1944566578620.6873552337780.106247911301False1.453853853650.1939022297290.3075933838010.18217751158False0.7808143027260.2955948482960.6242827246160.353213102167False1.675228080370.5322043807661.05586909141.14310683641False3.523878078792.10038977873
6123742082348.443963-51.2469310.01780888065721.78992652890.028192378580621.39229965210.021531561389620.77898216250.014393014833320.59169960020.011447476223120.46083641050.01692745275820.28239440920.011907064355920.13199234010.025526633486219.93891334530.016164693981420.13951110840.078433528542519.74150085450.04963187873366.982796546130.18131627454False10.07117492310.19972450045617.71769148040.234873807443False21.05329907440.22197577967523.75009962950.370282106937False27.99253541470.30698886083732.15163507860.755913822377False38.4091469570.57184434296931.9297528292.30660373016False46.06803182672.10589179867
7123744383348.621006-51.2834480.014471908099923.28887557980.092736057937122.66569328310.061082586646122.44982337950.052912801504122.10454559330.039283089339722.24391555790.07289266586321.88133621220.047413788735921.9529991150.11178229004121.60070610050.067329756915621.25277709960.17393939197121.47796440120.2172559350731.755697812650.149959532859False3.116898692410.1753540442073.802512480820.185313510992False5.226148701380.189087612114.59656892670.308598130395False6.418972500620.2803146885156.008974402890.618655725592False8.312230155790.51546631065911.45220623031.83469039689False9.30709698461.86235113957
8123896321348.315129-51.638810.97844815254220.24869918820.0075934180058519.37826156620.0052809636108618.74668121340.0025478331372118.07016181950.0021039217244817.45180320740.0011979741975716.8042716980.00095151161076516.89762687680.0013550549047116.23191452030.00093014223966716.76587295530.0041120941750715.99164772030.0028223211411428.87488908680.20194509656False64.37175853550.313100836928115.1668569760.270255273083False214.7510382690.416141022828379.558495970.41879535991False689.1143930290.603922336971632.3379533860.789190451062False1167.438985051.00013641202713.9220865032.70389330345False1456.602033323.78636974143
9124506908348.348611-51.7449040.51449441909824.25160789490.1984181255123.32379150390.11589226126723.870180130.17263753712223.4038810730.13183274865223.92576026920.29872298240723.44480705260.19997321069223.32144546510.34249630570422.62122917180.17389008402823.70215988161.4323134422322.42949104310.5238009095190.7233639180810.132194645493False1.700134962170.1814736253261.027845758340.163432663923False1.579237946370.1917549385690.9765531659710.268683012447False1.520817771940.2801074620151.703812554330.537468956928False3.247194713720.5200664389291.199875117921.58288657306False3.87439220941.86915583567

II - Removal of duplicated sources¶

We remove duplicated objects from the input catalogues.

In [25]:
SORT_COLS = ['merr_ap_decam_g', 'merr_ap_decam_r','merr_ap_decam_i','merr_ap_decam_z','merr_ap_decam_y']
FLAG_NAME = 'des_flag_cleaned'

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(catalogue, RA_COL, DEC_COL, sort_col=SORT_COLS,flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
The initial catalogue had 9307507 sources.
The cleaned catalogue has 9307386 sources (121 removed).
The cleaned catalogue has 121 sources flagged as having been cleaned

III - Astrometry correction¶

We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.

In [26]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_SSDF.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [27]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec, near_ra0=True )
In [28]:
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
    gaia_coords
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))
RA correction: 0.25125262735627985 arcsec
Dec correction: -0.10440259524102657 arcsec
In [29]:
catalogue[RA_COL] = catalogue[RA_COL] + delta_ra.to(u.deg)
catalogue[DEC_COL] = catalogue[DEC_COL] + delta_dec.to(u.deg)
In [30]:
catalogue[RA_COL].unit = u.deg
catalogue[DEC_COL].unit = u.deg
In [31]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec, near_ra0=True)

IV - Flagging Gaia objects¶

In [32]:
catalogue.add_column(
    gaia_flag_column(SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]), epoch, gaia)
)
In [33]:
GAIA_FLAG_NAME = "des_flag_gaia"

catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(catalogue[GAIA_FLAG_NAME] > 0)))
454986 sources flagged.

V - Flagging objects near bright stars¶

VI - Saving to disk¶

In [34]:
catalogue.write("{}/DES.fits".format(OUT_DIR), overwrite=True)