##// END OF EJS Templates
schain
RSS
completado op clean Rayleigh, modificada la escritura y lectura de hdf5 para potencia, cambio en gráficos para uso del ChannelList, añadir en otros pendiente
completado op clean Rayleigh, modificada la escritura y lectura de hdf5 para potencia, cambio en gráficos para uso del ChannelList, añadir en otros pendiente
joabAM - - Load All Authors

File last commit:

r1371:603b419b2641
r1397:9109c52a497c
Show More
jroproc_base.py
203 lines | 5.0 KiB | text/x-python | PythonLexer
/ schainpy / model / proc / jroproc_base.py
History | Source | Raw |Copy content |Copy permalink
Daniel Valdez
This is the new organization by packages and scripts for Signal Chain, this version contains new features and bugs fixed until August 2014
 r487 '''
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 Base clases to create Processing units and operations, the MPDecorator
must be used in plotting and writing operations to allow to run as an
external process.
Daniel Valdez
This is the new organization by packages and scripts for Signal Chain, this version contains new features and bugs fixed until August 2014
 r487 '''
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187
José Chávez
checking misspelled kwargs in operations/processing units
 r929 import inspect
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 import zmq
import time
import pickle
jespinoza
Errors handling and gracefully terminate main process
 r1241 import traceback
Juan C. Espinoza
Add input queues for processing units and external operations
 r1235 from threading import Thread
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 from multiprocessing import Process, Queue
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 from schainpy.utils import log
José Chávez
checking misspelled kwargs in operations/processing units
 r929
Daniel Valdez
This is the new organization by packages and scripts for Signal Chain, this version contains new features and bugs fixed until August 2014
 r487
Miguel Valdez
Merge with branch schain_julia_drifts from rev. 803 to 995....
 r568 class ProcessingUnit(object):
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 '''
Base class to create Signal Chain Units
'''
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
jespinoza
Update ParamReader to support diferent HDF5 files, fix Read/Write Madrigal files
 r1254 proc_type = 'processing'
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def __init__(self):
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Daniel Valdez
This is the new organization by packages and scripts for Signal Chain, this version contains new features and bugs fixed until August 2014
 r487 self.dataIn = None
Miguel Valdez
A new SendToServer Unit has been created to upload files to a remote server....
 r573 self.dataOut = None
Daniel Valdez
This is the new organization by packages and scripts for Signal Chain, this version contains new features and bugs fixed until August 2014
 r487 self.isConfig = False
Juan C. Espinoza
Review MP changes, three types of operations: self, other and external
 r1177 self.operations = []
joabAM
merged branches
 r1370
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 def setInput(self, unit):
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 self.dataIn = unit.dataOut
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
José Chávez
checking misspelled kwargs in operations/processing units
 r929 def getAllowedArgs(self):
Juan C. Espinoza
Add __attrs__ attribute to Process clasess to improve CLI finder
 r1097 if hasattr(self, '__attrs__'):
return self.__attrs__
else:
return inspect.getargspec(self.run).args
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187 def addOperation(self, conf, operation):
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 '''
'''
joabAM
Modificación a kmamisr para ejecutarse en la versión 3, creación de scripts con terminación v3 para difereciarlos, se comentó la linea #720 de JroIO_param.py debido a que reiniciaba la lista de archivos, ocasionando la reescritura del archivo hdf5. Alguna otra modificación aparente es producto de algunas variaciones en espacios al usar la función print()
 r1279
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 self.operations.append((operation, conf.type, conf.getKwargs()))
George Yong
Multiprocessing for Spectra (all operation) working
 r1171
Miguel Valdez
r577 def getOperationObj(self, objId):
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
Review MP changes, three types of operations: self, other and external
 r1177 if objId not in list(self.operations.keys()):
Miguel Valdez
r577 return None
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
Review MP changes, three types of operations: self, other and external
 r1177 return self.operations[objId]
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 def call(self, **kwargs):
'''
'''
try:
if self.dataIn is not None and self.dataIn.flagNoData and not self.dataIn.error:
return self.dataIn.isReady()
elif self.dataIn is None or not self.dataIn.error:
self.run(**kwargs)
elif self.dataIn.error:
self.dataOut.error = self.dataIn.error
self.dataOut.flagNoData = True
except:
joabAM
merged branches
 r1370 err = traceback.format_exc()
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 if 'SchainWarning' in err:
log.warning(err.split('SchainWarning:')[-1].split('\n')[0].strip(), self.name)
elif 'SchainError' in err:
log.error(err.split('SchainError:')[-1].split('\n')[0].strip(), self.name)
else:
Juan C. Espinoza
Fix ScpecraWriter and CohInt attribute
 r1310 log.error(err, self.name)
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 self.dataOut.error = True
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 for op, optype, opkwargs in self.operations:
if optype == 'other' and not self.dataOut.flagNoData:
self.dataOut = op.run(self.dataOut, **opkwargs)
elif optype == 'external' and not self.dataOut.flagNoData:
op.queue.put(self.dataOut)
joabAM
merged branches
 r1370 elif optype == 'external' and self.dataOut.error:
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 op.queue.put(self.dataOut)
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 return 'Error' if self.dataOut.error else self.dataOut.isReady()
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def setup(self):
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 raise NotImplementedError
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def run(self):
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 raise NotImplementedError
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def close(self):
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 return
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 class Operation(object):
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 '''
'''
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
jespinoza
Update ParamReader to support diferent HDF5 files, fix Read/Write Madrigal files
 r1254 proc_type = 'operation'
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def __init__(self):
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187 self.id = None
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 self.isConfig = False
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 if not hasattr(self, 'name'):
self.name = self.__class__.__name__
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def getAllowedArgs(self):
if hasattr(self, '__attrs__'):
return self.__attrs__
else:
return inspect.getargspec(self.run).args
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def setup(self):
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 self.isConfig = True
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 raise NotImplementedError
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def run(self, dataIn, **kwargs):
Daniel Valdez
This is the new organization by packages and scripts for Signal Chain, this version contains new features and bugs fixed until August 2014
 r487 """
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 Realiza las operaciones necesarias sobre la dataIn.data y actualiza los
atributos del objeto dataIn.
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Miguel Valdez
Merge with branch schain_julia_drifts from rev. 803 to 995....
 r568 Input:
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 dataIn : objeto del tipo JROData
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 Return:
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 None
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 Affected:
__buffer : buffer de recepcion de datos.
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 """
if not self.isConfig:
self.setup(**kwargs)
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 raise NotImplementedError
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def close(self):
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187 return
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
joabAM
Modificación a kmamisr para ejecutarse en la versión 3, creación de scripts con terminación v3 para difereciarlos, se comentó la linea #720 de JroIO_param.py debido a que reiniciaba la lista de archivos, ocasionando la reescritura del archivo hdf5. Alguna otra modificación aparente es producto de algunas variaciones en espacios al usar la función print()
 r1279
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def MPDecorator(BaseClass):
"""
Juan C. Espinoza
Review MP changes, three types of operations: self, other and external
 r1177 Multiprocessing class decorator
Juan C. Valdez
Version 2.2.5 Fixed several bugs, add jro colormap for spectra/rti, add ParamWriter, TODO: Fix decimation currently disabled
 r860
joabAM
merged branches
 r1370 This function add multiprocessing features to a BaseClass.
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 """
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 class MPClass(BaseClass, Process):
Juan C. Espinoza
Review MP changes, three types of operations: self, other and external
 r1177
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 def __init__(self, *args, **kwargs):
super(MPClass, self).__init__()
Process.__init__(self)
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 self.args = args
self.kwargs = kwargs
jespinoza
Use of delays instead of input queue to keep dataouts and avoid loose of them
 r1245 self.t = time.time()
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 self.op_type = 'external'
Juan C. Espinoza
Review MP changes, three types of operations: self, other and external
 r1177 self.name = BaseClass.__name__
jespinoza
Update ParamReader to support diferent HDF5 files, fix Read/Write Madrigal files
 r1254 self.__doc__ = BaseClass.__doc__
joabAM
Modificación a kmamisr para ejecutarse en la versión 3, creación de scripts con terminación v3 para difereciarlos, se comentó la linea #720 de JroIO_param.py debido a que reiniciaba la lista de archivos, ocasionando la reescritura del archivo hdf5. Alguna otra modificación aparente es producto de algunas variaciones en espacios al usar la función print()
 r1279
George Yong
Fix bug in CrossSpectraPlot
 r1201 if 'plot' in self.name.lower() and not self.name.endswith('_'):
self.name = '{}{}'.format(self.CODE.upper(), 'Plot')
joabAM
Modificación a kmamisr para ejecutarse en la versión 3, creación de scripts con terminación v3 para difereciarlos, se comentó la linea #720 de JroIO_param.py debido a que reiniciaba la lista de archivos, ocasionando la reescritura del archivo hdf5. Alguna otra modificación aparente es producto de algunas variaciones en espacios al usar la función print()
 r1279
self.start_time = time.time()
joabAM
Separación de datos antes del 17 de Julio del 2021, donde se modificó los apuntes y la lectura de estos
 r1371 #self.err_queue = args[2]
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 self.queue = Queue(maxsize=1)
self.myrun = BaseClass.run
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 def run(self):
joabAM
Modificación a kmamisr para ejecutarse en la versión 3, creación de scripts con terminación v3 para difereciarlos, se comentó la linea #720 de JroIO_param.py debido a que reiniciaba la lista de archivos, ocasionando la reescritura del archivo hdf5. Alguna otra modificación aparente es producto de algunas variaciones en espacios al usar la función print()
 r1279
George Yong
Multiprocessing for Spectra (all operation) working
 r1171 while True:
Juan C. Espinoza
Rewrite controller, remove MPDecorator to units (keep for plots an writers) use of queues for interproc comm instead of zmq, self operations are no longer supported
 r1287 dataOut = self.queue.get()
George Yong
Multiprocessing for writing Units(Spectral, Voltage and Parameters)
 r1179
jespinoza
Errors handling and gracefully terminate main process
 r1241 if not dataOut.error:
jespinoza
Fix excessive memory RAM consumption
 r1268 try:
BaseClass.run(self, dataOut, **self.kwargs)
except:
joabAM
merged branches
 r1370 err = traceback.format_exc()
Juan C. Espinoza
Fix ScpecraWriter and CohInt attribute
 r1310 log.error(err, self.name)
jespinoza
Errors handling and gracefully terminate main process
 r1241 else:
joabAM
Modificación a kmamisr para ejecutarse en la versión 3, creación de scripts con terminación v3 para difereciarlos, se comentó la linea #720 de JroIO_param.py debido a que reiniciaba la lista de archivos, ocasionando la reescritura del archivo hdf5. Alguna otra modificación aparente es producto de algunas variaciones en espacios al usar la función print()
 r1279 break
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187
Juan C. Espinoza
Review MP changes, three types of operations: self, other and external
 r1177 self.close()
def close(self):
Juan C. Espinoza
New plotting architecture with buffering/throttle capabilities
 r1187
BaseClass.close(self)
log.success('Done...(Time:{:4.2f} secs)'.format(time.time()-self.start_time), self.name)
return MPClass