CDFS-SWIRE Master List Creation¶
Preparation of VIDEO/VISTA/VIRCAM data¶
The catalogue comes from dmu0_VISTA_VIDEO-private.
There is an old public version of the catalogue but we are using the newer private version in the hope that it will be public by the time we publish the masterlist.
Filters: Z, Y, J, H, Ks
In the catalogue, we keep:
- The identifier (it's unique in the catalogue);
- The position (degrees);
- The stellarity;
- The magnitude for each band in apertude 3, which is 2 arcsec (rs548 presumes same for private catalogue).
- The “auto” magnitude is provided, we presume this is standard Sextractor units etc.
Yannick said the dates of observation for VIDEO are from 2009/11 to 2016/12. There is a paper from 2012 (Jarvis et al). So will use 2012.
In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
from collections import OrderedDict
import os
from astropy import units as u
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np
from herschelhelp_internal.flagging import gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux
In [3]:
import locale
locale.setlocale(locale.LC_ALL, 'en_GB')
Out[3]:
In [4]:
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
In [5]:
OUT_DIR = os.environ.get('TMP_DIR', "./data_tmp")
try:
os.makedirs(OUT_DIR)
except FileExistsError:
pass
RA_COL = "video_ra"
DEC_COL = "video_dec"
I - Column selection¶
In [6]:
# TODO: Check that the aperture magnitudes are aperture corrected.
imported_columns = OrderedDict({
'ID': 'video_id',
'ALPHA_J2000': 'video_ra',
'DELTA_J2000': 'video_dec',
'J_CLASS_STAR': 'video_stellarity',
'Z_MAG_APER_3': 'm_ap_video_z',
'Z_MAGERR_APER_3': 'merr_ap_video_z',
'Z_MAG_AUTO': 'm_video_z',
'Z_MAGERR_AUTO': 'merr_video_z',
'Z_FLUX_APER_3': 'f_ap_video_z',
'Z_FLUXERR_APER_3':'ferr_ap_video_z',
'Z_FLUX_AUTO': 'f_video_z',
'Z_FLUXERR_AUTO': 'ferr_video_z',
'Y_MAG_APER_3': 'm_ap_video_y',
'Y_MAGERR_APER_3': 'merr_ap_video_y',
'Y_MAG_AUTO': 'm_video_y',
'Y_MAGERR_AUTO': 'merr_video_y',
'Y_FLUX_APER_3': 'f_ap_video_y',
'Y_FLUXERR_APER_3':'ferr_ap_video_y',
'Y_FLUX_AUTO': 'f_video_y',
'Y_FLUXERR_AUTO': 'ferr_video_y',
'J_MAG_APER_3': 'm_ap_video_j',
'J_MAGERR_APER_3': 'merr_ap_video_j',
'J_MAG_AUTO': 'm_video_j',
'J_MAGERR_AUTO': 'merr_video_j',
'J_FLUX_APER_3': 'f_ap_video_j',
'J_FLUXERR_APER_3':'ferr_ap_video_j',
'J_FLUX_AUTO': 'f_video_j',
'J_FLUXERR_AUTO': 'ferr_video_j',
'H_MAG_APER_3': 'm_ap_video_h',
'H_MAGERR_APER_3': 'merr_ap_video_h',
'H_MAG_AUTO': 'm_video_h',
'H_MAGERR_AUTO': 'merr_video_h',
'H_FLUX_APER_3': 'f_ap_video_h',
'H_FLUXERR_APER_3':'ferr_ap_video_h',
'H_FLUX_AUTO': 'f_video_h',
'H_FLUXERR_AUTO': 'ferr_video_h',
'K_MAG_APER_3': 'm_ap_video_k',
'K_MAGERR_APER_3': 'merr_ap_video_k',
'K_MAG_AUTO': 'm_video_k',
'K_MAGERR_AUTO': 'merr_video_k',
'K_FLUX_APER_3': 'f_ap_video_k',
'K_FLUXERR_APER_3':'ferr_ap_video_k',
'K_FLUX_AUTO': 'f_video_k',
'K_FLUXERR_AUTO': 'ferr_video_k'
})
catalogue = Table.read("../../dmu0/dmu0_VISTA-VIDEO-private/data/VIDEO-all_2017-02-12_fullcat_errfix_CDFS-SWIRE.fits"
)[list(imported_columns)]
for column in imported_columns:
catalogue[column].name = imported_columns[column]
epoch = 2012
# Clean table metadata
catalogue.meta = None
In [7]:
# Correct wrong values.
# The catalogue contains negative fluxes and 99 in the associated magnitude. These
# values may be considered as upper limits but we don't want this information in the
# master list. Also, some sources may have a very low signal to noise with the error
# on magnitude superior to 99. In all these cases, we set the magnitude, the flux, and
# the associated errors to NaN.
band_names = [_[8:] for _ in catalogue.colnames if _.startswith("merr_ap_")]
for band in band_names:
# Total fluxes and magnitudes
mask = catalogue["m_{}".format(band)] > 90.
mask &= catalogue["merr_{}".format(band)] > 90.
mask &= catalogue["f_{}".format(band)] < 0.
catalogue["m_{}".format(band)][mask] = np.nan
catalogue["merr_{}".format(band)][mask] = np.nan
catalogue["f_{}".format(band)][mask] = np.nan
catalogue["ferr_{}".format(band)][mask] = np.nan
# Aperture fluxes and magnitudes
mask = catalogue["m_ap_{}".format(band)] > 90.
mask &= catalogue["merr_ap_{}".format(band)] > 90.
mask &= catalogue["f_ap_{}".format(band)] < 0.
catalogue["m_ap_{}".format(band)][mask] = np.nan
catalogue["merr_ap_{}".format(band)][mask] = np.nan
catalogue["f_ap_{}".format(band)][mask] = np.nan
catalogue["ferr_ap_{}".format(band)][mask] = np.nan
# Adding the band-flag column
catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag_{}".format(band)))
In [8]:
catalogue[:10].show_in_notebook()
Out[8]:
II - Correct z band fluxes and magnitudes¶
We discovered that the catalogue contains some wrong z magnitudes in the CDFS-SWIRE field. Strangely, Sextractor affected some magnitudes to sources which are not on the z image. Boris found a way to get rid of these magnitudes: all the wrong sources have a Z_MAGERR_AUTO to 0.
But we have to look at these sources in another catalogue because the VIDEO catalogue we use has been processed to correct for wrong error. We use the video_id column (which is unique within a field) to find the sources identified in the other catalogue and set their flux, magnitudes, and associated errors to NaN.
In [9]:
not_errfix_catalogue = Table.read("../../dmu0/dmu0_VISTA-VIDEO-private/data/VIDEO-cdfs_2017-02-12_fullcat.fits")
In [10]:
# Identifiers of the sources with bad z measurements
bad_z_sources_ids = not_errfix_catalogue['ID'][not_errfix_catalogue['Z_MAGERR_AUTO'] == 0]
# Mask of the bad z sources in our catalogue
bad_z_sources_mask = np.in1d(catalogue['video_id'], bad_z_sources_ids)
# Check that we caught all the bad sources
assert len(bad_z_sources_ids) == np.sum(bad_z_sources_mask)
In [11]:
del not_errfix_catalogue
In [12]:
# Set the z flux, magnitudes, and associated error of the bad z sources to NaN
catalogue['m_video_z'][bad_z_sources_mask] = np.nan
catalogue['merr_video_z'][bad_z_sources_mask] = np.nan
catalogue['f_video_z'][bad_z_sources_mask] = np.nan
catalogue['ferr_video_z'][bad_z_sources_mask] = np.nan
catalogue['m_ap_video_z'][bad_z_sources_mask] = np.nan
catalogue['merr_ap_video_z'][bad_z_sources_mask] = np.nan
catalogue['f_ap_video_z'][bad_z_sources_mask] = np.nan
catalogue['ferr_ap_video_z'][bad_z_sources_mask] = np.nan
III - Removal of duplicated sources¶
We remove duplicated objects from the input catalogues.
In [13]:
SORT_COLS = ['merr_ap_video_z', 'merr_ap_video_y', 'merr_ap_video_j', 'merr_ap_video_h', 'merr_ap_video_k']
FLAG_NAME = 'video_flag_cleaned'
nb_orig_sources = len(catalogue)
catalogue = remove_duplicates(catalogue, RA_COL, DEC_COL, sort_col=SORT_COLS,flag_name=FLAG_NAME)
nb_sources = len(catalogue)
print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))
IV - Astrometry correction¶
We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.
In [14]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_CDFS-SWIRE.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [15]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL],
gaia_coords.ra, gaia_coords.dec)
In [16]:
delta_ra, delta_dec = astrometric_correction(
SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
gaia_coords
)
print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))
In [17]:
catalogue[RA_COL] += delta_ra.to(u.deg)
catalogue[DEC_COL] += delta_dec.to(u.deg)
In [18]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL],
gaia_coords.ra, gaia_coords.dec)
V - Flagging Gaia objects¶
In [19]:
catalogue.add_column(
gaia_flag_column(SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]), epoch, gaia)
)
In [20]:
GAIA_FLAG_NAME = "video_flag_gaia"
catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(catalogue[GAIA_FLAG_NAME] > 0)))
VI - Saving to disk¶
In [21]:
catalogue.write("{}/VISTA-VIDEO.fits".format(OUT_DIR), overwrite=True)