AKARI-NEP master catalogue

Preparation of Pan-STARRS1 - 3pi Steradian Survey (3SS) data

This catalogue comes from dmu0_PanSTARRS1-3SS.

In the catalogue, we keep:

  • The uniquePspsSTid as unique object identifier;
  • The r-band position which is given for all the sources;
  • The grizy <band>FApMag aperture magnitude (see below);
  • The grizy <band>FKronMag as total magnitude.

The Pan-STARRS1-3SS catalogue provides for each band an aperture magnitude defined as “In PS1, an 'optimal' aperture radius is determined based on the local PSF. The wings of the same analytic PSF are then used to extrapolate the flux measured inside this aperture to a 'total' flux.”

The observations used for the catalogue where done between 2010 and 2015 (ref).

TODO: Check if the detection flag can be used to know in which bands an object was detected to construct the coverage maps.

TODO: Check for stellarity.

In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
This notebook was run with herschelhelp_internal version: 
33f5ec7 (Wed Dec 6 16:56:17 2017 +0000)
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))

from collections import OrderedDict
import os

from astropy import units as u
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux
In [3]:
OUT_DIR =  os.environ.get('TMP_DIR', "./data_tmp")
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "ps1_ra"
DEC_COL = "ps1_dec"

I - Column selection

In [4]:
imported_columns = OrderedDict({
        "objID": "ps1_id",
        "raMean": "ps1_ra",
        "decMean": "ps1_dec",
        "gFApMag": "m_ap_gpc1_g",
        "gFApMagErr": "merr_ap_gpc1_g",
        "gFKronMag": "m_gpc1_g",
        "gFKronMagErr": "merr_gpc1_g",
        "rFApMag": "m_ap_gpc1_r",
        "rFApMagErr": "merr_ap_gpc1_r",
        "rFKronMag": "m_gpc1_r",
        "rFKronMagErr": "merr_gpc1_r",
        "iFApMag": "m_ap_gpc1_i",
        "iFApMagErr": "merr_ap_gpc1_i",
        "iFKronMag": "m_gpc1_i",
        "iFKronMagErr": "merr_gpc1_i",
        "zFApMag": "m_ap_gpc1_z",
        "zFApMagErr": "merr_ap_gpc1_z",
        "zFKronMag": "m_gpc1_z",
        "zFKronMagErr": "merr_gpc1_z",
        "yFApMag": "m_ap_gpc1_y",
        "yFApMagErr": "merr_ap_gpc1_y",
        "yFKronMag": "m_gpc1_y",
        "yFKronMagErr": "merr_gpc1_y"
    })


catalogue = Table.read("../../dmu0/dmu0_PanSTARRS1-3SS/data/PanSTARRS1-3SS_AKARI-NEP_v2.fits")[list(imported_columns)]
for column in imported_columns:
    catalogue[column].name = imported_columns[column]

epoch = 2012

# Clean table metadata
catalogue.meta = None
In [5]:
# Adding flux and band-flag columns
for col in catalogue.colnames:
    if col.startswith('m_'):
        
        errcol = "merr{}".format(col[1:])
        
        # -999 is used for missing values
        catalogue[col][catalogue[col] < -900] = np.nan
        catalogue[errcol][catalogue[errcol] < -900] = np.nan     
        
        flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))
        
        # Fluxes are added in µJy
        catalogue.add_column(Column(flux * 1.e6, name="f{}".format(col[1:])))
        catalogue.add_column(Column(error * 1.e6, name="f{}".format(errcol[1:])))
        
        # Band-flag column
        if "ap" not in col:
            catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))
        
# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting.
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
In [6]:
catalogue[:10].show_in_notebook()
Out[6]:
<Table masked=True length=10>
idxps1_idps1_raps1_decm_ap_gpc1_gmerr_ap_gpc1_gm_gpc1_gmerr_gpc1_gm_ap_gpc1_rmerr_ap_gpc1_rm_gpc1_rmerr_gpc1_rm_ap_gpc1_imerr_ap_gpc1_im_gpc1_imerr_gpc1_im_ap_gpc1_zmerr_ap_gpc1_zm_gpc1_zmerr_gpc1_zm_ap_gpc1_ymerr_ap_gpc1_ym_gpc1_ymerr_gpc1_yf_ap_gpc1_gferr_ap_gpc1_gf_gpc1_gferr_gpc1_gflag_gpc1_gf_ap_gpc1_rferr_ap_gpc1_rf_gpc1_rferr_gpc1_rflag_gpc1_rf_ap_gpc1_iferr_ap_gpc1_if_gpc1_iferr_gpc1_iflag_gpc1_if_ap_gpc1_zferr_ap_gpc1_zf_gpc1_zferr_gpc1_zflag_gpc1_zf_ap_gpc1_yferr_ap_gpc1_yf_gpc1_yferr_gpc1_yflag_gpc1_y
0185492687854425344268.7855556364.578773822.8477001190.31158500909822.9389991760.25231501460121.65250015260.088345997035521.75810050960.096868000924620.81419944760.049297999590620.92499923710.060635000467320.53459930420.10186400264520.80789947510.1269920021320.08020019530.13998700678320.09140014650.117770999672.635845511420.7564359962012.423261763130.563143537505False7.925012191030.6448556292657.190511681190.641528291246False17.15221535920.77879866403315.48817707260.864966664938False22.19015206192.0818848062917.25203020982.01786570949False33.72251230994.3479378681533.37643456953.62037854817False
1185492688278625547268.8278431564.5789155323.13220024110.29037299752223.39019966130.24460500478722.57379913330.1339409947422.59830093380.13502100110121.65019989010.13862900435921.78529930110.23417900502721.02820014950.20019699633121.00110054020.20130899548520.87159919740.35138699412320.03359985350.5866299867632.0282426160.5424401300511.599263905240.360297421678False3.39219138360.4184750276133.316497135010.412436060761False7.941820078931.014027721947.012619573121.51252971677False14.08380295222.5968877482314.43975364792.67730991117False16.2689798425.2652825788535.201409919719.0195395021False
2185492688210751638268.8210529164.5757774922.30890083310.1657020002622.39999961850.14244599640421.24340057370.058403000235621.37439918520.054303001612420.6296997070.050345998257420.72699928280.063524998724520.59810066220.13680300116520.58110046390.079989001154920.25480079650.17385800182820.31800079350.2018460035324.32951916820.6607590160423.981073104270.522307281095False11.55153692010.62137046780910.23859409550.512082390952False20.32919198070.94267231722618.58661430391.08747838082False20.92954249672.637126968821.25983132181.5662668931False28.71307341634.5977996760527.08941850165.03611359611False
3185492688927873847268.8927511164.5775977122.17950057980.24636699259322.60560035710.21235099434921.60330009460.11012399941721.86770057680.13005499541821.60149955750.091140002012321.72150039670.076228998601421.47570037840.19404600560721.56119918820.27096399664920.51519966130.3094519972820.36890029910.233154997234.877527964921.106771507423.294275051040.644302529178False8.292394638310.8410806034766.500095977490.778614500338False8.306157789350.6972441688127.437035273130.522150545363False9.326524783711.666864264588.62025920032.15133277754False22.59020309916.4385653138725.84876988825.55086018617False
4185492689652905288268.9648948264.5786562922.45859909060.25620600581222.30850028990.18502299487621.57929992680.082406997680721.63570022580.13946700096121.33539962770.068906001746721.28560066220.090690001845421.24279975890.16849799454221.49489974980.15882299840520.64459991460.33908501267420.54829978940.6070680022243.771901680540.8900724316374.33111668280.738076317523False8.477738751510.6434574020578.04859194471.03387266223False10.61304850080.67355471368511.11116853640.928100113685False11.55793097351.793702910119.163050922271.34038380191False20.05210796016.2624505400221.911902370412.2516083913False
5185492689220731725268.9220530664.5758551621.71500015260.14932699501521.86590003970.11551800370220.71570014950.051456999033720.91889953610.044617999345119.47299957280.012536999769519.55599975590.016403999179618.93950080870.030724000185719.03879928590.022605000063818.58130073550.028168000280918.71339988710.05573000013837.481693953641.028996611546.510884393460.692732142644False18.78105308310.89010261066515.57543502910.640067768656False58.99295772460.68119189003354.6512497410.825706144813False96.42722672672.7286831910787.99951642971.8321476854False134.1157262873.47945642705118.7517573256.09543591078False
6185492689653457577268.9653274764.5806764321.11829948430.088636003434721.28210067750.06434399634620.9148998260.073963001370420.9727001190.048578999936620.89669990540.065652996301720.92849922180.061785001307721.02370071410.15589900314821.03770065310.096256002783820.40360069270.23955200612520.35670089720.21631799638312.96224441751.0581958450611.14704434930.660607408815False15.63291870951.0649525942614.82244857890.663199518832False15.89717802940.96128080264815.43832969360.878534966532False14.14228921092.0306670999413.96110324421.23772237386False25.03569908295.5237652698426.14084615085.20820383286False
7185492687458386376268.7457749264.5796801522.07889938350.068832002580222.17490005490.084790997207221.00779914860.061129998415721.18079948430.066915996372720.78499984740.086314000189320.90830039980.045770999044220.55960083010.081426002085220.57530021670.097470000386220.19379997250.16463199257920.34149932860.1657799929385.351065248280.339239435294.89823906960.382529912377False14.35093973170.80799822205112.23714782340.754198796886False17.61976294041.4007382411215.72822959070.663049439086False21.68501205771.6262917541921.37371003811.91878608868False30.3724629524.6054272422726.50942260364.0476976569False
8185492687956560408268.7956077164.5747254521.95789909360.086245000362422.06539916990.11650999635520.65119934080.038637001067420.78140068050.050671998411419.36199951170.014156999997819.43619918820.021075999364318.74480056760.01981700025518.82570075990.020810000598418.48049926760.028056999668518.60580062870.03398500010375.981916720570.4751710475095.418016608960.581405586912False19.93059497820.7092500149417.67826858320.825056025249False65.34316935130.85201473830361.02676161151.18463396844False115.3665148192.10568482802107.0827944412.05242581635False147.1635626313.80292009224131.1232631294.10433408365False
9185492689322031033268.9321634964.5752757122.57180023190.26637199521122.47949981690.17037799954421.15119934080.047088999301221.16819953920.047437999397519.80249977110.031514000147619.91769981380.025315999984719.21850013730.034692998975519.35079956050.048142001032819.03619956970.075157999992419.16049957280.06511700153353.398442350990.833765942113.699985935720.580616359026False12.57535528130.54540034076412.3799868770.54090653645False43.55119654951.2640938088639.16697893880.913252440023False74.5761475882.3829639285666.0207077612.92738601448False88.21047732066.1062005897378.66837347254.71813371756False

II - Removal of duplicated sources

We remove duplicated objects from the input catalogues.

In [7]:
SORT_COLS = ['merr_ap_gpc1_r', 'merr_ap_gpc1_g', 'merr_ap_gpc1_i', 'merr_ap_gpc1_z', 'merr_ap_gpc1_y']
FLAG_NAME = 'ps1_flag_cleaned'

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(catalogue, RA_COL, DEC_COL,  sort_col=SORT_COLS, flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))
/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  ma.MaskedArray.__setitem__(self, index, value)
The initial catalogue had 221353 sources.
The cleaned catalogue has 221306 sources (47 removed).
The cleaned catalogue has 44 sources flagged as having been cleaned

III - Astrometry correction

We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.

In [8]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_AKARI-NEP.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [9]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)
In [10]:
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
    gaia_coords
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))
RA correction: 0.00029263319447636604 arcsec
Dec correction: -0.001360161365937529 arcsec
In [11]:
catalogue[RA_COL] +=  delta_ra.to(u.deg)
catalogue[DEC_COL] += delta_dec.to(u.deg)
In [12]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)

IV - Flagging Gaia objects

In [13]:
catalogue.add_column(
    gaia_flag_column(SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]), epoch, gaia)
)
In [14]:
GAIA_FLAG_NAME = "ps1_flag_gaia"

catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(catalogue[GAIA_FLAG_NAME] > 0)))
61878 sources flagged.

V - Saving to disk

In [15]:
catalogue.write("{}/PS1.fits".format(OUT_DIR), overwrite=True)