schain Commit - r1339:2c655c5faa65 · Jicamarca Repository

HDFWriter create metadata if not given, update setup files

jespinoza -

r1339:2c655c5faa65 v3.0.0b4

parent child

Context file:

r1339:2c655c5faa65

CHANGELOG.md +22 -24

             # CHANGELOG
             ## 3.0.0
             * Python 3.x & 2.X compatible
             * New architecture with multiprocessing support
             * Add @MPDecorator for multiprocessing Operations (Plots, Writers and Publishers)
             * Added new type of operation `external` for non-locking operations
             * New plotting architecture with buffering/throttle capabilities to speed up plots
             * Clean controller to optimize scripts (format & optype are no longer required)
-            * Replace ParamReader and ParamWriter with new flexible HDFReader and HDFWriter
+            * Replace `ParamReader` and `ParamWriter` with new flexible `HDFReader` and `HDFWriter`
-            * New GUI with dinamic load of Units and operations (use Kivy framework)
+            * New GUI with dynamic load of Units and operations (use Kivy framework)
             * Clean code
             ## 2.3
             * Added support for Madrigal formats (reading/writing).
             * Added support for reading BLTR parameters (*.sswma).
             * Added support for reading Julia format (*.dat).
             * Added high order function `MPProject` for multiprocessing scripts.
             * Added two new Processing Units `PublishData` and `ReceiverData` for receiving and sending dataOut through multiple ways (tcp, ipc, inproc).
             * Added a new graphics Processing Unit `PlotterReceiver`. It is decoupled from normal processing sequence with support for data generated by multiprocessing scripts.
             * Added support for sending realtime graphic to web server.
             * GUI command `schain` is now `schainGUI`.
             * Added a CLI tool named `schain`.
               * Scripts templates can be now generated with `schain generate`.
               * Now it is possible to search Processing Units and Operations with `schain search [module]` to get the right name and its allowed parameters.
               * `schain xml` to run xml scripts.
             * Added suggestions when parameters are poorly written.
             * `Controller.start()` now runs in a different process than the process calling it.
             * Added `schainpy.utils.log` for log standarization.
             * Running script on online mode no longer ignores date and hour. Issue #1109.
             * Added support for receving voltage data directly from JARS (tcp, ipc).
             * Updated README for MAC OS GUI installation.
             * Setup now installs numpy.
             ## 2.2.6
             * Graphics generated by the GUI are now the same as generated by scripts. Issue #1074.
             * Added support for C extensions.
             * Function `hildebrand_sehkon` optimized with a C wrapper.
             * Numpy version updated.
             * Migration to GIT.
-            ## 2.2.5:
+            ## 2.2.5
             * splitProfiles and combineProfiles modules were added to VoltageProc and Signal Chain GUI.
             * nProfiles of USRP data (hdf5) is the number of profiles thera are in one second.
             * jroPlotter works directly with data objects instead of dictionaries
             * script "schain" was added to Signal Chain installer
-            ## 2.2.4.1:
+            ## 2.2.4.1
             * jroIO_usrp.py is update to read Sandra's data
             * decimation in Spectra and RTI plots is always enabled.
             * time* window option added to GUI
-            ## 2.2.4:
+            ## 2.2.4
             * jroproc_spectra_lags.py added to schainpy
             * Bug fixed in schainGUI: ProcUnit was created with the same id in some cases.
             * Bug fixed in jroHeaderIO: Header size validation.
-            ## 2.2.3.1:
+            ## 2.2.3.1
             * Filtering block by time has been added.
             * Bug fixed plotting RTI, CoherenceMap and others using xmin and xmax parameters. The first day worked
             properly but the next days did not.
-            ## 2.2.3:
+            ## 2.2.3
             * Bug fixed in GUI: Error getting(reading) Code value
             * Bug fixed in GUI: Flip option always needs channelList field
             * Bug fixed in jrodata: when one branch modified a value in "dataOut" (example: dataOut.code) this value
             was modified for every branch (because this was a reference). It was modified in data.copy()
             * Bug fixed in jroproc_voltage.profileSelector(): rangeList replaces to profileRangeList.
-            ## 2.2.2:
+            ## 2.2.2
             * VoltageProc: ProfileSelector, Reshape, Decoder with nTxs!=1 and getblock=True was tested
             * Rawdata and testRawdata.py added to Signal Chain project
-            ## 2.2.1:
+            ## 2.2.1
             * Bugs fixed in GUI
             * Views were improved in GUI
             * Support to MST* ISR experiments
             * Bug fixed getting noise using hyldebrant. (minimum number of points > 20%)
             * handleError added to jroplotter.py
-            ## 2.2.0:
+            ## 2.2.0
             * GUI: use of external plotter
             * Compatible with matplotlib 1.5.0
-            ## 2.1.5:
+            ## 2.1.5
             * serializer module added to Signal Chain
             * jroplotter.py added to Signal Chain
-            ## 2.1.4.2:
+            ## 2.1.4.2
             * A new Plotter Class was added
             * Project.start() does not accept filename as a parameter anymore
-            ## 2.1.4.1:
+            ## 2.1.4.1
             * Send notifications when an error different to ValueError is detected
-            ## 2.1.4:
+            ## 2.1.4
             * Sending error notifications to signal chain administrator
             * Login to email server added
-            ## 2.1.3.3:
+            ## 2.1.3.3
             * Colored Button Icons were added to GUI
-            ## 2.1.3.2:
+            ## 2.1.3.2
             * GUI: user interaction enhanced
             * controller_api.py: Safe access to ControllerThead
-            ## 2.1.3.1:
+            ## 2.1.3.1
             * GUI: every icon were resized
             * jroproc_voltage.py: Print a message when "Read from code" option is selected and the code is not defined inside data file
-            ## 2.1.3:
+            ## 2.1.3
             * jroplot_heispectra.py: SpectraHeisScope was not showing the right channels
-            * jroproc_voltage.py: Bug fixed selecting profiles (self.nProfiles took a wrong value),
+            * jroproc_voltage.py: Bug fixed selecting profiles (self.nProfiles took a wrong value), Bug fixed selecting heights by block (selecting profiles instead heights)
-            					 Bug fixed selecting heights by block (selecting profiles instead heights)
             * jroproc_voltage.py: New feature added: decoding data by block using FFT.
             * jroIO_heispectra.py: Bug fixed in FitsReader. Using local Fits instance instead schainpy.mode.data.jrodata.Fits.
             * jroIO_heispectra.py: Channel index list does not exist.
-            ## 2.1.2:
+            ## 2.1.2
-            * jroutils_ftp.py: Bug fixed, Any error sending file stopped the Server Thread
+            * jroutils_ftp.py: Bug fixed, Any error sending file stopped the Server Thread Server thread opens and closes remote server each time file list is sent
-            				  Server thread opens and closes remote server each time file list is sent
             * jroplot_spectra.py: Noise path was not being created when noise data is saved.
-            * jroIO_base.py: startTime can be greater than endTime. Example: SpreadF [18:00 *  07:00]
  No newline at end of file
+            * jroIO_base.py: startTime can be greater than endTime. Example: SpreadF

README.md +4 -2

             # Signal Chain
             Signal Chain is a radar data processing library wich includes modules to read,
             and write different files formats, besides modules to process and visualize the
             data.
             ## Dependencies
             - GCC (gcc or gfortran)
             - Python.h (python-dev or python-devel)
             - Python-TK (python-tk)
             - HDF5 libraries (libhdf5-dev)
             ## Installation
-            To get started the easiest way to install it is through [PyPI](https://pypi.org/project/schainpy/) with pip:
+            To get started the easiest way to install it is through
+            [PyPI](https://pypi.org/project/schainpy/) with pip. We strongly recommend to
+            use an virtual environment like virtualenv or anaconda.
             ```bash
             pip install schainpy
             ```
             ### From source
             First, ensure that you have the above-listed dependencies installed, then clone
             the repository and install as normal python package:
             ```bash
-            git clone https://github.com/JRO-Peru/schain.git
+            git clone https://github.com/JRO-Peru/schainpy.git
             cd schain
             git checkout `branch-name` (optional)
             sudo pip install ./
             ```
             ### Using Docker
             Download Dockerfile from the repository, and create a docker image:
             ```bash
             docker build -t schain .
             ```
             You can run a container using an xml file or a schain script also you need to
             mount a volume for the data input and for the output files/plots:
             ```bash
             docker run -it --rm --volume /path/to/host/data:/data schain xml /data/test.xml
             docker run -it --rm --volume /path/to/host/data:/data --entrypoint /urs/local/bin/python schain /data/test.py
             ```
             ## CLI (command line interface)
             Signal Chain provides the following commands:
             - schainGUI: Open the GUI
             - schain: Signal chain command line
             ## Example
             Here you can find an script to read Spectra data (.pdata), remove dc and plot
             self-spectra & RTI:
             ```python
             #!/usr/bin/python
             from schainpy.controller import Project
             prj = Project()
             read_unit = prj.addReadUnit(
                 datatype='Spectra',
                 path='/path/to/pdata/',
                 startDate='2014/01/31',
                 endDate='2014/03/31',
                 startTime='00:00:00',
                 endTime='23:59:59',
                 online=0,
                 walk=0
                 )
             proc_unit = prj.addProcUnit(
                 datatype='Spectra',
                 inputId=read_unit.getId()
                 )
             op = proc_unit.addOperation(name='selectChannels')
             op.addParameter(name='channelList', value='0,1')
             op = proc_unit.addOperation(name='selectHeights')
             op.addParameter(name='minHei', value='80')
             op.addParameter(name='maxHei', value='200')
             op = proc_unit.addOperation(name='removeDC')
             op = proc_unit.addOperation(name='SpectraPlot')
             op.addParameter(name='wintitle', value='Spectra', format='str')
             op = proc_unit.addOperation(name='RTIPlot')
             op.addParameter(name='wintitle', value='RTI', format='str')
             prj.start()
             ```

schainpy/__init__.py +1 -1

             """Signal chain python package"""
             try:
                 from .controller import Project
             except:
                 pass
-            __version__ = '3.0.0b3'
+            __version__ = '3.0.0b4'

schainpy/model/io/jroIO_param.py +11 -5

             import os
             import time
             import datetime
             import numpy
             import h5py
             import schainpy.admin
             from schainpy.model.data.jrodata import *
             from schainpy.model.proc.jroproc_base import ProcessingUnit, Operation, MPDecorator
             from schainpy.model.io.jroIO_base import *
             from schainpy.utils import log
             class HDFReader(Reader, ProcessingUnit):
                 """Processing unit to read HDF5 format files
                 This unit reads HDF5 files created with `HDFWriter` operation contains
                 by default two groups Data and Metadata all variables would be saved as `dataOut`
                 attributes.
                 It is possible to read any HDF5 file by given the structure in the `description`
                 parameter, also you can add extra values to metadata with the parameter `extras`.
                 Parameters:
                 -----------
                 path : str
                     Path where files are located.
                 startDate : date
                     Start date of the files
                 endDate : list
                     End date of the files
                 startTime : time
                     Start time of the files
                 endTime : time
                     End time of the files
                 description : dict, optional
                     Dictionary with the description of the HDF5 file
                 extras : dict, optional
                     Dictionary with extra metadata to be be added to `dataOut`
                 Examples
                 --------
                 desc = {
                     'Data': {
                         'data_output': ['u', 'v', 'w'],
                         'utctime': 'timestamps',
                     }  ,
                     'Metadata': {
                         'heightList': 'heights'
                     }
                 }
                 desc = {
                     'Data': {
                         'data_output': 'winds',
                         'utctime': 'timestamps'
                     },
                     'Metadata': {
                         'heightList': 'heights'
                     }
                 }
                 extras = {
                     'timeZone': 300
                 }
                 reader = project.addReadUnit(
                     name='HDFReader',
                     path='/path/to/files',
                     startDate='2019/01/01',
                     endDate='2019/01/31',
                     startTime='00:00:00',
                     endTime='23:59:59',
                     # description=json.dumps(desc),
                     # extras=json.dumps(extras),
                     )
                 """
                 __attrs__ = ['path', 'startDate', 'endDate', 'startTime', 'endTime', 'description', 'extras']
                 def __init__(self):
                     ProcessingUnit.__init__(self)
                     self.dataOut = Parameters()
                     self.ext = ".hdf5"
                     self.optchar = "D"
                     self.meta = {}
                     self.data = {}
                     self.open_file = h5py.File
                     self.open_mode = 'r'
                     self.description = {}
                     self.extras = {}
                     self.filefmt = "*%Y%j***"
                     self.folderfmt = "*%Y%j"
                 def setup(self, **kwargs):
                     self.set_kwargs(**kwargs)
                     if not self.ext.startswith('.'):
                         self.ext = '.{}'.format(self.ext)
                     if self.online:
                         log.log("Searching files in online mode...", self.name)
                         for nTries in range(self.nTries):
                             fullpath = self.searchFilesOnLine(self.path, self.startDate,
                                 self.endDate, self.expLabel, self.ext, self.walk,
                                 self.filefmt, self.folderfmt)
                             try:
                                 fullpath = next(fullpath)
                             except:
                                 fullpath = None
                             if fullpath:
                                 break
                             log.warning(
                                 'Waiting {} sec for a valid file in {}: try {} ...'.format(
                                     self.delay, self.path, nTries + 1),
                                 self.name)
                             time.sleep(self.delay)
                         if not(fullpath):
                             raise schainpy.admin.SchainError(
                                 'There isn\'t any valid file in {}'.format(self.path))
                         pathname, filename = os.path.split(fullpath)
                         self.year = int(filename[1:5])
                         self.doy = int(filename[5:8])
                         self.set = int(filename[8:11]) - 1
                     else:
                         log.log("Searching files in {}".format(self.path), self.name)
                         self.filenameList = self.searchFilesOffLine(self.path, self.startDate,
                             self.endDate, self.expLabel, self.ext, self.walk, self.filefmt, self.folderfmt)
                     self.setNextFile()
                     return
                 def readFirstHeader(self):
                     '''Read metadata and data'''
                     self.__readMetadata()
                     self.__readData()
                     self.__setBlockList()
                     if 'type' in self.meta:
                         self.dataOut = eval(self.meta['type'])()
                     for attr in self.meta:
                         setattr(self.dataOut, attr, self.meta[attr])
                     self.blockIndex = 0
                     return
                 def __setBlockList(self):
                     '''
                     Selects the data within the times defined
                     self.fp
                     self.startTime
                     self.endTime
                     self.blockList
                     self.blocksPerFile
                     '''
                     startTime = self.startTime
                     endTime = self.endTime
                     thisUtcTime = self.data['utctime']
                     self.interval = numpy.min(thisUtcTime[1:] - thisUtcTime[:-1])
                     thisDatetime = datetime.datetime.utcfromtimestamp(thisUtcTime[0])
                     thisDate = thisDatetime.date()
                     thisTime = thisDatetime.time()
                     startUtcTime = (datetime.datetime.combine(thisDate, startTime) - datetime.datetime(1970, 1, 1)).total_seconds()
                     endUtcTime = (datetime.datetime.combine(thisDate, endTime) - datetime.datetime(1970, 1, 1)).total_seconds()
                     ind = numpy.where(numpy.logical_and(thisUtcTime >= startUtcTime, thisUtcTime < endUtcTime))[0]
                     self.blockList = ind
                     self.blocksPerFile = len(ind)
                     return
                 def __readMetadata(self):
                     '''
                     Reads Metadata
                     '''
                     meta = {}
                     if self.description:
                         for key, value in self.description['Metadata'].items():
                             meta[key] = self.fp[value].value
                     else:
                         grp = self.fp['Metadata']
                         for name in grp:
                             meta[name] = grp[name].value
                     if self.extras:
                         for key, value in self.extras.items():
                             meta[key] = value
                     self.meta = meta
                     return
                 def __readData(self):
                     data = {}
                     if self.description:
                         for key, value in self.description['Data'].items():
                             if isinstance(value, str):
                                 if isinstance(self.fp[value], h5py.Dataset):
                                     data[key] = self.fp[value].value
                                 elif isinstance(self.fp[value], h5py.Group):
                                     array = []
                                     for ch in self.fp[value]:
                                         array.append(self.fp[value][ch].value)
                                     data[key] = numpy.array(array)
                             elif isinstance(value, list):
                                 array = []
                                 for ch in value:
                                     array.append(self.fp[ch].value)
                                 data[key] = numpy.array(array)
                     else:
                         grp = self.fp['Data']
                         for name in grp:
                             if isinstance(grp[name], h5py.Dataset):
                                 array = grp[name].value
                             elif isinstance(grp[name], h5py.Group):
                                 array = []
                                 for ch in grp[name]:
                                     array.append(grp[name][ch].value)
                                 array = numpy.array(array)
                             else:
                                 log.warning('Unknown type: {}'.format(name))
                             if name in self.description:
                                 key = self.description[name]
                             else:
                                 key = name
                             data[key] = array
                     self.data = data
                     return
                 def getData(self):
                     for attr in self.data:
                         if self.data[attr].ndim == 1:
                             setattr(self.dataOut, attr, self.data[attr][self.blockIndex])
                         else:
                             setattr(self.dataOut, attr, self.data[attr][:, self.blockIndex])
                     self.dataOut.flagNoData = False
                     self.blockIndex += 1
                     log.log("Block No. {}/{} -> {}".format(
                         self.blockIndex,
                         self.blocksPerFile,
                         self.dataOut.datatime.ctime()), self.name)
                     return
                 def run(self, **kwargs):
                     if not(self.isConfig):
                         self.setup(**kwargs)
                         self.isConfig = True
                     if self.blockIndex == self.blocksPerFile:
                         self.setNextFile()
                     self.getData()
                     return
             @MPDecorator
             class HDFWriter(Operation):
                 """Operation to write HDF5 files.
                 The HDF5 file contains by default two groups Data and Metadata where
                 you can save any `dataOut` attribute specified by `dataList` and `metadataList`
                 parameters, data attributes are normaly time dependent where the metadata
                 are not.
                 It is possible to customize the structure of the HDF5 file with the
                 optional description parameter see the examples.
                 Parameters:
                 -----------
                 path : str
                     Path where files will be saved.
                 blocksPerFile : int
                     Number of blocks per file
                 metadataList : list
                     List of the dataOut attributes that will be saved as metadata
                 dataList : int
                     List of the dataOut attributes that will be saved as data
                 setType : bool
                     If True the name of the files corresponds to the timestamp of the data
                 description : dict, optional
                     Dictionary with the desired description of the HDF5 file
                 Examples
                 --------
                 desc = {
                     'data_output': {'winds': ['z', 'w', 'v']},
                     'utctime': 'timestamps',
                     'heightList': 'heights'
                 }
                 desc = {
                     'data_output': ['z', 'w', 'v'],
                     'utctime': 'timestamps',
                     'heightList': 'heights'
                 }
                 desc = {
                     'Data': {
                         'data_output': 'winds',
                         'utctime': 'timestamps'
                     },
                     'Metadata': {
                         'heightList': 'heights'
                     }
                 }
                 writer = proc_unit.addOperation(name='HDFWriter')
                 writer.addParameter(name='path', value='/path/to/file')
                 writer.addParameter(name='blocksPerFile', value='32')
                 writer.addParameter(name='metadataList', value='heightList,timeZone')
                 writer.addParameter(name='dataList',value='data_output,utctime')
                 # writer.addParameter(name='description',value=json.dumps(desc))
                 """
                 ext = ".hdf5"
                 optchar = "D"
                 filename = None
                 path = None
                 setFile = None
                 fp = None
                 firsttime = True
                 #Configurations
                 blocksPerFile = None
                 blockIndex = None
                 dataOut = None
                 #Data Arrays
                 dataList = None
                 metadataList = None
                 currentDay = None
                 lastTime = None
                 def __init__(self):
                     Operation.__init__(self)
                     return
                 def setup(self, path=None, blocksPerFile=10, metadataList=None, dataList=None, setType=None, description=None):
                     self.path = path
                     self.blocksPerFile = blocksPerFile
                     self.metadataList = metadataList
-                    self.dataList = dataList
+                    self.dataList = [s.strip() for s in dataList]
                     self.setType = setType
                     self.description = description
-                    for s in ['type', 'timeZone', 'useLocalTime']:
+                    if self.metadataList is None:
-                        if s not in self.metadataList:
+                        self.metadataList = self.dataOut.metadata_list
-                            self.metadataList.append(s)
                     tableList = []
                     dsList = []
                     for i in range(len(self.dataList)):
                         dsDict = {}
+                        if hasattr(self.dataOut, self.dataList[i]):
                             dataAux = getattr(self.dataOut, self.dataList[i])
                             dsDict['variable'] = self.dataList[i]
+                        else:
+                            log.warning('Attribute {} not found in dataOut', self.name)
+                            continue
                         if dataAux is None:
                             continue
                         elif isinstance(dataAux, (int, float, numpy.integer, numpy.float)):
                             dsDict['nDim'] = 0
                         else:
                             dsDict['nDim'] = len(dataAux.shape)
                             dsDict['shape'] = dataAux.shape
                             dsDict['dsNumber'] = dataAux.shape[0]
                             dsDict['dtype'] = dataAux.dtype
                         dsList.append(dsDict)
                     self.dsList = dsList
                     self.currentDay = self.dataOut.datatime.date()
                 def timeFlag(self):
                     currentTime = self.dataOut.utctime
                     timeTuple = time.localtime(currentTime)
                     dataDay = timeTuple.tm_yday
                     if self.lastTime is None:
                         self.lastTime = currentTime
                         self.currentDay = dataDay
                         return False
                     timeDiff = currentTime - self.lastTime
                     #Si el dia es diferente o si la diferencia entre un dato y otro supera la hora
                     if dataDay != self.currentDay:
                         self.currentDay = dataDay
                         return True
                     elif timeDiff > 3*60*60:
                         self.lastTime = currentTime
                         return True
                     else:
                         self.lastTime = currentTime
                         return False
-                def run(self, dataOut, path, blocksPerFile=10, metadataList=[],
+                def run(self, dataOut, path, blocksPerFile=10, metadataList=None,
                         dataList=[], setType=None, description={}):
                     self.dataOut = dataOut
                     if not(self.isConfig):
                         self.setup(path=path, blocksPerFile=blocksPerFile,
                                    metadataList=metadataList, dataList=dataList,
                                    setType=setType, description=description)
                         self.isConfig = True
                         self.setNextFile()
                     self.putData()
                     return
                 def setNextFile(self):
                     ext = self.ext
                     path = self.path
                     setFile = self.setFile
                     timeTuple = time.localtime(self.dataOut.utctime)
                     subfolder = 'd%4.4d%3.3d' % (timeTuple.tm_year,timeTuple.tm_yday)
                     fullpath = os.path.join(path, subfolder)
                     if os.path.exists(fullpath):
                         filesList = os.listdir(fullpath)
                         filesList = [k for k in filesList if k.startswith(self.optchar)]
                         if len( filesList ) > 0:
                             filesList = sorted(filesList, key=str.lower)
                             filen = filesList[-1]
                             # el filename debera tener el siguiente formato
                             # 0 1234 567 89A BCDE (hex)
                             # x YYYY DDD SSS .ext
                             if isNumber(filen[8:11]):
                                 setFile = int(filen[8:11]) #inicializo mi contador de seteo al seteo del ultimo file
                             else:
                                 setFile = -1
                         else:
                             setFile = -1 #inicializo mi contador de seteo
                     else:
                         os.makedirs(fullpath)
                         setFile = -1 #inicializo mi contador de seteo
                     if self.setType is None:
                         setFile += 1
                         file = '%s%4.4d%3.3d%03d%s' % (self.optchar,
                                                        timeTuple.tm_year,
                                                        timeTuple.tm_yday,
                                                        setFile,
                                                        ext )
                     else:
                         setFile = timeTuple.tm_hour*60+timeTuple.tm_min
                         file = '%s%4.4d%3.3d%04d%s' % (self.optchar,
                                                        timeTuple.tm_year,
                                                        timeTuple.tm_yday,
                                                        setFile,
                                                        ext )
                     self.filename = os.path.join( path, subfolder, file )
                     #Setting HDF5 File
                     self.fp = h5py.File(self.filename, 'w')
                     #write metadata
                     self.writeMetadata(self.fp)
                     #Write data
                     self.writeData(self.fp)
                 def getLabel(self, name, x=None):
                     if x is None:
                         if 'Data' in self.description:
                             data = self.description['Data']
                             if 'Metadata' in self.description:
                                 data.update(self.description['Metadata'])
                         else:
                             data = self.description
                         if name in data:
                             if isinstance(data[name], str):
                                 return data[name]
                             elif isinstance(data[name], list):
                                 return None
                             elif isinstance(data[name], dict):
                                 for key, value in data[name].items():
                                     return key
                         return name
                     else:
                         if 'Metadata' in self.description:
                             meta = self.description['Metadata']
                         else:
                             meta = self.description
                         if name in meta:
                             if isinstance(meta[name], list):
                                 return meta[name][x]
                             elif isinstance(meta[name], dict):
                                 for key, value in meta[name].items():
                                     return value[x]
+                        if 'cspc' in name:
+                            return 'pair{:02d}'.format(x)
+                        else:
                             return 'channel{:02d}'.format(x)
                 def writeMetadata(self, fp):
                     if self.description:
                         if 'Metadata' in self.description:
                             grp = fp.create_group('Metadata')
                         else:
                             grp = fp
                     else:
                         grp = fp.create_group('Metadata')
                     for i in range(len(self.metadataList)):
                         if not hasattr(self.dataOut, self.metadataList[i]):
                             log.warning('Metadata: `{}` not found'.format(self.metadataList[i]), self.name)
                             continue
                         value = getattr(self.dataOut, self.metadataList[i])
                         if isinstance(value, bool):
                             if value is True:
                                 value = 1
                             else:
                                 value = 0
                         grp.create_dataset(self.getLabel(self.metadataList[i]), data=value)
                     return
                 def writeData(self, fp):
                     if self.description:
                         if 'Data' in self.description:
                             grp = fp.create_group('Data')
                         else:
                             grp = fp
                     else:
                         grp = fp.create_group('Data')
                     dtsets = []
                     data = []
                     for dsInfo in self.dsList:
                         if dsInfo['nDim'] == 0:
                             ds = grp.create_dataset(
                                 self.getLabel(dsInfo['variable']),
                                 (self.blocksPerFile, ),
                                 chunks=True,
                                 dtype=numpy.float64)
                             dtsets.append(ds)
                             data.append((dsInfo['variable'], -1))
                         else:
                             label = self.getLabel(dsInfo['variable'])
                             if label is not None:
                                 sgrp = grp.create_group(label)
                             else:
                                 sgrp = grp
                             for i in range(dsInfo['dsNumber']):
                                 ds = sgrp.create_dataset(
                                     self.getLabel(dsInfo['variable'], i),
                                     (self.blocksPerFile, ) + dsInfo['shape'][1:],
                                     chunks=True,
                                     dtype=dsInfo['dtype'])
                                 dtsets.append(ds)
                                 data.append((dsInfo['variable'], i))
                     fp.flush()
                     log.log('Creating file: {}'.format(fp.filename), self.name)
                     self.ds = dtsets
                     self.data = data
                     self.firsttime = True
                     self.blockIndex = 0
                     return
                 def putData(self):
                     if (self.blockIndex == self.blocksPerFile) or self.timeFlag():
                         self.closeFile()
                         self.setNextFile()
                     for i, ds in enumerate(self.ds):
                         attr, ch = self.data[i]
                         if ch == -1:
                             ds[self.blockIndex] = getattr(self.dataOut, attr)
                         else:
                             ds[self.blockIndex] = getattr(self.dataOut, attr)[ch]
                     self.fp.flush()
                     self.blockIndex += 1
                     log.log('Block No. {}/{}'.format(self.blockIndex, self.blocksPerFile), self.name)
                     return
                 def closeFile(self):
                     if self.blockIndex != self.blocksPerFile:
                         for ds in self.ds:
                             ds.resize(self.blockIndex, axis=0)
                     self.fp.flush()
                     self.fp.close()
                 def close(self):
                     self.closeFile()

setup.py +2 -2

             # Copyright (c) 2012-2020 Jicamarca Radio Observatory
             # All rights reserved.
             #
             # Distributed under the terms of the BSD 3-clause license.
             """schainpy is an open source library to read, write and process radar data
             Signal Chain is a radar data processing library wich includes modules to read,
             and write different files formats, besides modules to process and visualize the
             data.
             """
             import os
             from setuptools import setup, Extension
             from setuptools.command.build_ext import build_ext as _build_ext
             from schainpy import __version__
             DOCLINES = __doc__.split("\n")
             class build_ext(_build_ext):
                 def finalize_options(self):
                     _build_ext.finalize_options(self)
                     # Prevent numpy from thinking it is still in its setup process:
                     __builtins__.__NUMPY_SETUP__ = False
                     import numpy
                     self.include_dirs.append(numpy.get_include())
             setup(
                 name = "schainpy",
                 version = __version__,
                 description = DOCLINES[0],
                 long_description = "\n".join(DOCLINES[2:]),
-                url = "https://github.com/JRO-Peru/schain",
+                url = "https://github.com/JRO-Peru/schainpy",
                 author = "Jicamarca Radio Observatory",
-                author_email = "jro-developers@igp.gob.pe",
+                author_email = "jro-developers@jro.igp.gob.pe",
                 license="BSD-3-Clause",
                 classifiers=[
                     "Development Status :: 4 - Beta",
                     "Environment :: Console",
                     "Intended Audience :: Science/Research",
                     "License :: OSI Approved :: BSD License",
                     "Operating System :: MacOS :: MacOS X",
                     "Operating System :: POSIX :: Linux",
                     "Programming Language :: Python :: 2",
                     "Programming Language :: Python :: 2.7",
                     "Programming Language :: Python :: 3",
                     "Programming Language :: Python :: 3.5",
                     "Programming Language :: Python :: 3.6",
                     "Programming Language :: Python :: 3.7",
                     "Topic :: Scientific/Engineering",
                 ],
                 packages = {
                     'schainpy',
                     'schainpy.model',
                     'schainpy.model.data',
                     'schainpy.model.graphics',
                     'schainpy.model.io',
                     'schainpy.model.proc',
                     'schainpy.model.utils',
                     'schainpy.utils',
                     'schainpy.gui',
                     'schainpy.cli',
                     },
                 package_data = {'': ['schain.conf.template'],
                                 'schainpy.files': ['*.oga']
                                 },
                 include_package_data = False,
                 scripts = ['schainpy/gui/schainGUI'],
                 entry_points = {
                     'console_scripts': [
                         'schain = schainpy.cli.cli:main',
                         ],
                     },
                 cmdclass = {'build_ext': build_ext},
                 ext_modules=[
                     Extension("schainpy.model.data._noise", ["schainc/_noise.c"]),
                     ],
                 setup_requires = ["numpy"],
                 install_requires = [
                     "scipy",
                     "h5py",
                     "matplotlib",
                     "pyzmq",
                     "fuzzywuzzy",
                     "click",
                     ],
             )

General Comments 0

Write
Preview

You need to be logged in to leave comments. Login now

No TODOs yet

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages