##// END OF EJS Templates
schain-jc
RSS
merge with v2.3
jespinoza -
r1063:ecbe1f26c35b merge
parent child
Show More
Side by Side Unified
Context file:
r1063:ecbe1f26c35b Loading diff...:
Show file before | Show file after | Hide comments
@@ -0,0 +1,109
1 ## CHANGELOG:
2
3 ### 2.3
4 * Added high order function `multiSchain` for multiprocessing scripts.
5 * Added two new Processing Units `PublishData` and `ReceiverData` for receiving and sending dataOut through multiple ways (tcp, ipc, inproc).
6 * Added a new graphics Processing Unit `PlotterReceiver`. It is decoupled from normal processing sequence with support for data generated by multiprocessing scripts.
7 * Added support for sending realtime graphic to web server.
8 * GUI command `schain` is now `schainGUI`.
9 * Added a CLI tool named `schain`.
10 * Scripts templates can be now generated with `schain generate`.
11 * Now it is possible to search Processing Units and Operations with `schain search [module]` to get the right name and its allowed parameters.
12 * `schain xml` to run xml scripts.
13 * Added suggestions when parameters are poorly written.
14 * `Controller.start()` now runs in a different process than the process calling it.
15 * Added `schainpy.utils.log` for log standarization.
16 * Running script on online mode no longer ignores date and hour. Issue #1109.
17 * Added support for receving voltage data directly from JARS (tcp, ipc).
18 * Updated README for MAC OS GUI installation.
19 * Setup now installs numpy.
20
21 ### 2.2.6
22 * Graphics generated by the GUI are now the same as generated by scripts. Issue #1074.
23 * Added support for C extensions.
24 * function `hildebrand_sehkon` optimized with a C wrapper.
25 * Numpy version updated.
26 * Migration to GIT.
27
28 ### 2.2.5:
29 * splitProfiles and combineProfiles modules were added to VoltageProc and Signal Chain GUI.
30 * nProfiles of USRP data (hdf5) is the number of profiles thera are in one second.
31 * jroPlotter works directly with data objects instead of dictionaries
32 * script "schain" was added to Signal Chain installer
33
34 ### 2.2.4.1:
35 * jroIO_usrp.py is update to read Sandra's data
36 * decimation in Spectra and RTI plots is always enabled.
37 * time* window option added to GUI
38
39 ### 2.2.4:
40 * jroproc_spectra_lags.py added to schainpy
41 * Bug fixed in schainGUI: ProcUnit was created with the same id in some cases.
42 * Bug fixed in jroHeaderIO: Header size validation.
43
44 ### 2.2.3.1:
45 * Filtering block by time has been added.
46 * Bug fixed plotting RTI, CoherenceMap and others using xmin and xmax parameters. The first day worked
47 properly but the next days did not.
48
49 ### 2.2.3:
50 * Bug fixed in GUI: Error getting(reading) Code value
51 * Bug fixed in GUI: Flip option always needs channelList field
52 * Bug fixed in jrodata: when one branch modified a value in "dataOut" (example: dataOut.code) this value
53 was modified for every branch (because this was a reference). It was modified in data.copy()
54 * Bug fixed in jroproc_voltage.profileSelector(): rangeList replaces to profileRangeList.
55
56 ### 2.2.2:
57 * VoltageProc: ProfileSelector, Reshape, Decoder with nTxs!=1 and getblock=True was tested
58 * Rawdata and testRawdata.py added to Signal Chain project
59
60 ### 2.2.1:
61 * Bugs fixed in GUI
62 * Views were improved in GUI
63 * Support to MST* ISR experiments
64 * Bug fixed getting noise using hyldebrant. (minimum number of points > 20%)
65 * handleError added to jroplotter.py
66
67 ### 2.2.0:
68 * GUI: use of external plotter
69 * Compatible with matplotlib 1.5.0
70
71 ### 2.1.5:
72 * serializer module added to Signal Chain
73 * jroplotter.py added to Signal Chain
74
75 ### 2.1.4.2:
76 * A new Plotter Class was added
77 * Project.start() does not accept filename as a parameter anymore
78
79 ### 2.1.4.1:
80 * Send notifications when an error different to ValueError is detected
81
82 ### 2.1.4:
83 * Sending error notifications to signal chain administrator
84 * Login to email server added
85
86 ### 2.1.3.3:
87 * Colored Button Icons were added to GUI
88
89 ### 2.1.3.2:
90 * GUI: user interaction enhanced
91 * controller_api.py: Safe access to ControllerThead
92
93 ### 2.1.3.1:
94 * GUI: every icon were resized
95 * jroproc_voltage.py: Print a message when "Read from code" option is selected and the code is not defined inside data file
96
97 ### 2.1.3:
98 * jroplot_heispectra.py: SpectraHeisScope was not showing the right channels
99 * jroproc_voltage.py: Bug fixed selecting profiles (self.nProfiles took a wrong value),
100 Bug fixed selecting heights by block (selecting profiles instead heights)
101 * jroproc_voltage.py: New feature added: decoding data by block using FFT.
102 * jroIO_heispectra.py: Bug fixed in FitsReader. Using local Fits instance instead schainpy.mode.data.jrodata.Fits.
103 * jroIO_heispectra.py: Channel index list does not exist.
104
105 ### 2.1.2:
106 * jroutils_ftp.py: Bug fixed, Any error sending file stopped the Server Thread
107 Server thread opens and closes remote server each time file list is sent
108 * jroplot_spectra.py: Noise path was not being created when noise data is saved.
109 * jroIO_base.py: startTime can be greater than endTime. Example: SpreadF [18:00 * 07:00] No newline at end of file
Show file before | Show file after | Hide comments
@@ -0,0 +1,95
1
2 import os, sys
3
4 from schainpy.controller import Project
5
6 controllerObj = Project()
7 controllerObj.setup(id = '002', name='script02', description="JASMET Meteor Detection")
8
9 #-------------------------------------- Setup -----------------------------------------
10 #Verificar estas variables
11
12 #Path para los archivos
13 # path = '/mnt/jars/2016_08/NOCHE'
14 # path = '/media/joscanoa/DATA_JASMET/JASMET/2016_08/DIA'
15 # path = '/media/joscanoa/DATA_JASMET/JASMET/2016_08/NOCHE'
16 path = '/home/nanosat/data/jasmet'
17
18 #Path para los graficos
19 pathfig = os.path.join(os.environ['HOME'],'Pictures/JASMET30/201608/graphics')
20
21 #Path para los archivos HDF5 de meteoros
22 pathfile = os.path.join(os.environ['HOME'],'Pictures/JASMET30/201608/meteor')
23
24 #Fechas para busqueda de archivos
25 startDate = '2010/08/29'
26 endDate = '2017/09/11'
27 #Horas para busqueda de archivos
28 startTime = '00:00:00'
29 endTime = '23:59:59'
30
31
32 #------------------------------ Voltage Reading Unit ----------------------------------
33
34 readUnitConfObj = controllerObj.addReadUnit(datatype='VoltageReader',
35 path=path,
36 startDate=startDate,
37 endDate=endDate,
38 startTime=startTime,
39 endTime=endTime,
40 online=0,
41 delay=30,
42 walk=1,
43 getblock=1,
44 blocktime=100)
45
46 opObj11 = readUnitConfObj.addOperation(name='printNumberOfBlock')
47
48 #-------------------------- Voltage Processing Unit ------------------------------------
49
50 procUnitConfObj0 = controllerObj.addProcUnit(datatype='VoltageProc', inputId=readUnitConfObj.getId())
51
52 opObj00 = procUnitConfObj0.addOperation(name='selectChannels')
53 opObj00.addParameter(name='channelList', value='0,1,2,3,4', format='intlist')
54
55 opObj01 = procUnitConfObj0.addOperation(name='setRadarFrequency')
56 opObj01.addParameter(name='frequency', value='30.e6', format='float')
57
58 # opObj01 = procUnitConfObj0.addOperation(name='interpolateHeights')
59 # opObj01.addParameter(name='topLim', value='73', format='int')
60 # opObj01.addParameter(name='botLim', value='71', format='int')
61
62 opObj02 = procUnitConfObj0.addOperation(name='Decoder', optype='other')
63
64 opObj03 = procUnitConfObj0.addOperation(name='CohInt', optype='other')
65 opObj03.addParameter(name='n', value='2', format='int')
66
67 procUnitConfObj1 = controllerObj.addProcUnit(datatype='SpectraProc', inputId=procUnitConfObj0.getId())
68 opObj11 = procUnitConfObj1.addOperation(name='RTIPlot', optype='other')
69 opObj11.addParameter(name='id', value='237', format='int')
70 opObj11.addParameter(name='xmin', value='9.0', format='float')
71 opObj11.addParameter(name='xmax', value='16.0', format='float')
72 opObj11.addParameter(name='zmin', value='15.0', format='float')
73 opObj11.addParameter(name='zmax', value='50.0', format='float')
74
75 #--------------------------- Parameters Processing Unit ------------------------------------
76
77 procUnitConfObj1 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=procUnitConfObj0.getId())
78 #
79 opObj10 = procUnitConfObj1.addOperation(name='SMDetection', optype='other')
80 opObj10.addParameter(name='azimuth', value='45', format='float')
81 opObj10.addParameter(name='hmin', value='60', format='float')
82 opObj10.addParameter(name='hmax', value='120', format='float')
83
84 opObj12 = procUnitConfObj1.addOperation(name='ParamWriter', optype='other')
85 opObj12.addParameter(name='path', value=pathfile)
86 opObj12.addParameter(name='blocksPerFile', value='1000', format='int')
87 opObj12.addParameter(name='metadataList',value='type,heightList,paramInterval,timeZone',format='list')
88 opObj12.addParameter(name='dataList',value='data_param,utctime',format='list')
89 opObj12.addParameter(name='mode',value='2',format='int')
90
91 #--------------------------------------------------------------------------------------------------
92
93 controllerObj.start()
94
95
Show file before | Show file after | Hide comments
@@ -0,0 +1,68
1 import os, sys
2
3 path = os.path.split(os.getcwd())[0]
4 path = os.path.split(path)[0]
5
6 sys.path.insert(0, path)
7
8 from schainpy.controller import Project
9
10 controllerObj = Project()
11 controllerObj.setup(id = '004', name='script04', description="JASMET Phase Calibration")
12
13 #-------------------------------------- Setup -----------------------------------------
14 #Verificar estas variables
15
16 #Path donde estan los archivos HDF5 de meteoros
17 path = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/meteor')
18
19 #Path para los graficos
20 pathfig = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/graphics')
21
22 #Path donde se almacenaran las fases calculadas
23 pathfile = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/phase')
24
25 #Fechas para busqueda de archivos
26 startDate = '2016/08/29'
27 endDate = '2016/09/11'
28 #Horas para busqueda de archivos
29 startTime = '00:00:00'
30 endTime = '23:59:59'
31
32 #------------------------------------------------------------------------------------------------
33 readUnitConfObj = controllerObj.addReadUnit(datatype='ParamReader',
34 path=path,
35 startDate=startDate,
36 endDate=endDate,
37 startTime=startTime,
38 endTime=endTime,
39 walk=1)
40
41 #--------------------------------------------------------------------------------------------------
42
43 procUnitConfObj1 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=readUnitConfObj.getId())
44 # #
45 opObj31 = procUnitConfObj1.addOperation(name='SMPhaseCalibration', optype='other')
46 opObj31.addParameter(name='nHours', value='1', format='float')
47 opObj31.addParameter(name='hmin', value='60', format='float')
48 opObj31.addParameter(name='hmax', value='120', format='float')
49 # opObj31.addParameter(name='channelPositions', value='(2.5,0),(0,2.5),(0,0),(0,4.5),(-2,0)', format='pairslist')
50
51 opObj32 = procUnitConfObj1.addOperation(name='PhasePlot', optype='other')
52 opObj32.addParameter(name='id', value='201', format='int')
53 opObj32.addParameter(name='wintitle', value='PhaseCalibration', format='str')
54 opObj32.addParameter(name='save', value='1', format='bool')
55 opObj32.addParameter(name='xmin', value='0', format='float')
56 opObj32.addParameter(name='xmax', value='24', format='float')
57 opObj32.addParameter(name='ymin', value='-180', format='float')
58 opObj32.addParameter(name='ymax', value='180', format='float')
59 opObj32.addParameter(name='figpath', value=pathfig, format='str')
60 # #
61 opObj33 = procUnitConfObj1.addOperation(name='ParamWriter', optype='other')
62 opObj33.addParameter(name='path', value=pathfile)
63 opObj33.addParameter(name='blocksPerFile', value='1000', format='int')
64 opObj33.addParameter(name='metadataList',value='type,outputInterval,timeZone',format='list')
65 opObj33.addParameter(name='dataList',value='data_output,utctime',format='list')
66 # # opObj25.addParameter(name='mode',value='1,0,0',format='intlist')
67
68 controllerObj.start() No newline at end of file
Show file before | Show file after | Hide comments
@@ -0,0 +1,85
1 import os, sys
2
3 path = os.path.split(os.getcwd())[0]
4 path = os.path.split(path)[0]
5
6 sys.path.insert(0, path)
7
8 from schainpy.controller import Project
9
10 controllerObj = Project()
11 controllerObj.setup(id = '005', name='script05', description="JASMET Wind Estimation")
12
13 #-------------------------------------- Setup -----------------------------------------
14 #Verificar estas variables
15
16 #Path donde estan los archivos HDF5 de meteoros
17 path = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/meteor')
18
19 #Path para los graficos
20 pathfig = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/graphics')
21
22 #Path donde se almacenaran las estimaciones de vientos
23 pathfile = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/phase')
24
25 #Fechas para busqueda de archivos
26 startDate = '2016/08/29'
27 endDate = '2016/09/11'
28 #Horas para busqueda de archivos
29 startTime = '00:00:00'
30 endTime = '23:59:59'
31
32 #Offsets optimos obtenidos con OptimumOffset.py
33 phaseOffsets = '-2.84, -1.77, 11.94, 9.71'
34 phaseOffsets = '-5.86, -0.93, -7.29, 23.35'
35 #------------------------------------------------------------------------------------------------
36 readUnitConfObj = controllerObj.addReadUnit(datatype='ParamReader',
37 path=path,
38 startDate=startDate,
39 endDate=endDate,
40 startTime=startTime,
41 endTime=endTime,
42 walk=1)
43 #--------------------------------------------------------------------------------------------------
44
45 procUnitConfObj1 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=readUnitConfObj.getId())
46 opObj10 = procUnitConfObj1.addOperation(name='CorrectSMPhases',optype='other')
47 opObj10.addParameter(name='phaseOffsets', value=phaseOffsets, format='floatlist')
48
49 opObj13 = procUnitConfObj1.addOperation(name='SkyMapPlot', optype='other')
50 opObj13.addParameter(name='id', value='1', format='int')
51 opObj13.addParameter(name='wintitle', value='Sky Map', format='str')
52 opObj13.addParameter(name='save', value='1', format='bool')
53 opObj13.addParameter(name='figpath', value=pathfig, format='str')
54 opObj13.addParameter(name='ftp', value='1', format='int')
55 opObj13.addParameter(name='exp_code', value='15', format='int')
56 opObj13.addParameter(name='sub_exp_code', value='1', format='int')
57 opObj13.addParameter(name='tmin', value='0', format='int')
58 opObj13.addParameter(name='tmax', value='24', format='int')
59
60 opObj22 = procUnitConfObj1.addOperation(name='WindProfiler', optype='other')
61 opObj22.addParameter(name='technique', value='Meteors', format='str')
62 opObj22.addParameter(name='nHours', value='1', format='float')
63 opObj22.addParameter(name='hmin', value='70', format='float')
64 opObj22.addParameter(name='hmax', value='120', format='float')
65
66 opObj23 = procUnitConfObj1.addOperation(name='WindProfilerPlot', optype='other')
67 opObj23.addParameter(name='id', value='2', format='int')
68 opObj23.addParameter(name='wintitle', value='Wind Profiler', format='str')
69 opObj23.addParameter(name='save', value='1', format='bool')
70 opObj23.addParameter(name='figpath', value = pathfig, format='str')
71 opObj23.addParameter(name='zmin', value='-140', format='int')
72 opObj23.addParameter(name='zmax', value='140', format='int')
73 opObj23.addParameter(name='xmin', value='0', format='float')
74 opObj23.addParameter(name='xmax', value='24', format='float')
75 opObj23.addParameter(name='ymin', value='70', format='float')
76 opObj23.addParameter(name='ymax', value='110', format='float')
77
78 opObj33 = procUnitConfObj1.addOperation(name='ParamWriter', optype='other')
79 opObj33.addParameter(name='path', value=pathfile)
80 opObj33.addParameter(name='blocksPerFile', value='1000', format='int')
81 opObj33.addParameter(name='metadataList',value='type,outputInterval,timeZone',format='list')
82 opObj33.addParameter(name='dataList',value='data_output,utctime',format='list')
83 #--------------------------------------------------------------------------------------------------
84
85 controllerObj.start() No newline at end of file
Show file before | Show file after | Hide comments
@@ -0,0 +1,97
1 import argparse
2
3 from schainpy.controller import Project, multiSchain
4
5 desc = "HF_EXAMPLE"
6
7 def fiber(cursor, skip, q, dt):
8
9 controllerObj = Project()
10
11 controllerObj.setup(id='191', name='test01', description=desc)
12
13 readUnitConfObj = controllerObj.addReadUnit(datatype='SpectraReader',
14 path='/home/nanosat/data/sp1_f0',
15 startDate=dt,
16 endDate=dt,
17 startTime="00:00:00",
18 endTime="23:59:59",
19 online=0,
20 #set=1426485881,
21 walk=1,
22 queue=q,
23 cursor=cursor,
24 skip=skip,
25 verbose=1
26 #timezone=-5*3600
27 )
28
29 # #opObj11 = readUnitConfObj.addOperation(name='printNumberOfBlock')
30 #
31 procUnitConfObj2 = controllerObj.addProcUnit(datatype='Spectra', inputId=readUnitConfObj.getId())
32 # procUnitConfObj2.addParameter(name='nipp', value='5', format='int')
33
34 procUnitConfObj3 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=readUnitConfObj.getId())
35 opObj11 = procUnitConfObj3.addOperation(name='SpectralMoments', optype='other')
36
37 #
38 # opObj11 = procUnitConfObj1.addOperation(name='SpectraPlot', optype='other')
39 # opObj11.addParameter(name='id', value='1000', format='int')
40 # opObj11.addParameter(name='wintitle', value='HF_Jicamarca_Spc', format='str')
41 # opObj11.addParameter(name='channelList', value='0', format='intlist')
42 # opObj11.addParameter(name='zmin', value='-120', format='float')
43 # opObj11.addParameter(name='zmax', value='-70', format='float')
44 # opObj11.addParameter(name='save', value='1', format='int')
45 # opObj11.addParameter(name='figpath', value=figpath, format='str')
46
47 # opObj11 = procUnitConfObj3.addOperation(name='Parameters1Plot', optype='other')
48 # opObj11.addParameter(name='channelList', value='0', format='intList')
49
50 # opObj11.addParameter(name='id', value='2000', format='int')
51 # # opObj11.addParameter(name='colormap', value='0', format='bool')
52 # opObj11.addParameter(name='onlySNR', value='1', format='bool')
53 # opObj11.addParameter(name='DOP', value='0', format='bool')
54 # # opObj11.addParameter(name='showSNR', value='1', format='bool')
55 # # opObj11.addParameter(name='SNRthresh', value='0', format='int')
56 # opObj11.addParameter(name='SNRmin', value='-10', format='int')
57 # opObj11.addParameter(name='SNRmax', value='30', format='int')
58
59 # opObj11.addParameter(name='showSNR', value='1', format='int')
60 # # opObj11.addParameter(name='channelList', value='0', format='intlist')
61 # # opObj11.addParameter(name='xmin', value='0', format='float')
62 # opObj11.addParameter(name='xmin', value='0', format='float')
63 # opObj11.addParameter(name='xmax', value='24', format='float')
64
65 # opObj11.addParameter(name='zmin', value='-110', format='float')
66 # opObj11.addParameter(name='zmax', value='-70', format='float')
67 # opObj11.addParameter(name='save', value='0', format='int')
68 # # opObj11.addParameter(name='figpath', value='/tmp/', format='str')
69 #
70 opObj12 = procUnitConfObj3.addOperation(name='PublishData', optype='other')
71 opObj12.addParameter(name='zeromq', value=1, format='int')
72 opObj12.addParameter(name='verbose', value=0, format='bool')
73
74
75 # opObj13 = procUnitConfObj3.addOperation(name='PublishData', optype='other')
76 # opObj13.addParameter(name='zeromq', value=1, format='int')
77 # opObj13.addParameter(name='server', value="juanca", format='str')
78
79 opObj12.addParameter(name='delay', value=0, format='int')
80
81
82 # print "Escribiendo el archivo XML"
83 # controllerObj.writeXml(filename)
84 # print "Leyendo el archivo XML"
85 # controllerObj.readXml(filename)
86
87
88 # timeit.timeit('controllerObj.run()', number=2)
89
90 controllerObj.start()
91
92
93 if __name__ == '__main__':
94 parser = argparse.ArgumentParser(description='Set number of parallel processes')
95 parser.add_argument('--nProcess', default=1, type=int)
96 args = parser.parse_args()
97 multiSchain(fiber, nProcess=args.nProcess, startDate='2017/01/26', endDate='2017/01/26')
Show file before | Show file after | Hide comments
@@ -0,0 +1,94
1 import argparse
2
3 from schainpy.controller import Project, multiSchain
4
5 desc = "HF_EXAMPLE"
6
7 def fiber(cursor, skip, q, dt):
8
9 controllerObj = Project()
10
11 controllerObj.setup(id='191', name='test01', description=desc)
12
13 readUnitConfObj = controllerObj.addReadUnit(datatype='SpectraReader',
14 path='/home/nanosat/data/julia',
15 startDate=dt,
16 endDate=dt,
17 startTime="00:00:00",
18 endTime="23:59:59",
19 online=0,
20 #set=1426485881,
21 delay=10,
22 walk=1,
23 queue=q,
24 cursor=cursor,
25 skip=skip,
26 #timezone=-5*3600
27 )
28
29 # #opObj11 = readUnitConfObj.addOperation(name='printNumberOfBlock')
30 #
31 procUnitConfObj2 = controllerObj.addProcUnit(datatype='Spectra', inputId=readUnitConfObj.getId())
32 # procUnitConfObj2.addParameter(name='nipp', value='5', format='int')
33
34 # procUnitConfObj3 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=readUnitConfObj.getId())
35 # opObj11 = procUnitConfObj3.addOperation(name='SpectralMoments', optype='other')
36
37 #
38 opObj11 = procUnitConfObj2.addOperation(name='RTIPlot', optype='other')
39 opObj11.addParameter(name='id', value='1000', format='int')
40 opObj11.addParameter(name='wintitle', value='HF_Jicamarca_Spc', format='str')
41 opObj11.addParameter(name='xmin', value='0', format='int')
42 opObj11.addParameter(name='xmax', value='24', format='int')
43
44 # opObj11 = procUnitConfObj3.addOperation(name='Parameters1Plot', optype='other')
45 # opObj11.addParameter(name='channelList', value='0', format='intList')
46 #
47 # opObj11.addParameter(name='id', value='2000', format='int')
48 # # opObj11.addParameter(name='colormap', value='0', format='bool')
49 # opObj11.addParameter(name='onlySNR', value='1', format='bool')
50 # opObj11.addParameter(name='DOP', value='0', format='bool')
51 # opObj11.addParameter(name='showSNR', value='1', format='bool')
52 # opObj11.addParameter(name='SNRthresh', value='0', format='int')
53 # opObj11.addParameter(name='SNRmin', value='-10', format='int')
54 # opObj11.addParameter(name='SNRmax', value='30', format='int')
55
56 # opObj11.addParameter(name='showSNR', value='1', format='int')
57 # # opObj11.addParameter(name='channelList', value='0', format='intlist')
58 # # opObj11.addParameter(name='xmin', value='0', format='float')
59 # opObj11.addParameter(name='xmin', value='0', format='float')
60 # opObj11.addParameter(name='xmax', value='24', format='float')
61
62 # opObj11.addParameter(name='zmin', value='-110', format='float')
63 # opObj11.addParameter(name='zmax', value='-70', format='float')
64 # opObj11.addParameter(name='save', value='0', format='int')
65 # # opObj11.addParameter(name='figpath', value='/tmp/', format='str')
66 #
67 # opObj12 = procUnitConfObj2.addOperation(name='PublishData', optype='other')
68 # opObj12.addParameter(name='zeromq', value=1, format='int')
69 # opObj12.addParameter(name='server', value='tcp://10.10.10.82:7000', format='str')
70
71
72 # opObj13 = procUnitConfObj3.addOperation(name='PublishData', optype='other')
73 # opObj13.addParameter(name='zeromq', value=1, format='int')
74 # opObj13.addParameter(name='server', value="juanca", format='str')
75
76 # opObj12.addParameter(name='delay', value=1, format='int')
77
78
79 # print "Escribiendo el archivo XML"
80 # controllerObj.writeXml(filename)
81 # print "Leyendo el archivo XML"
82 # controllerObj.readXml(filename)
83
84
85 # timeit.timeit('controllerObj.run()', number=2)
86
87 controllerObj.start()
88
89
90 if __name__ == '__main__':
91 parser = argparse.ArgumentParser(description='Set number of parallel processes')
92 parser.add_argument('--nProcess', default=1, type=int)
93 args = parser.parse_args()
94 multiSchain(fiber, nProcess=args.nProcess, startDate='2016/08/19', endDate='2016/08/19')
Show file before | Show file after | Hide comments
@@ -0,0 +1,1
1 <Project description="Claire" id="002" name="script02"><ReadUnit datatype="VoltageReader" id="21" inputId="0" name="VoltageReader"><Operation id="211" name="run" priority="1" type="self"><Parameter format="str" id="2111" name="datatype" value="VoltageReader" /><Parameter format="str" id="2112" name="path" value="/media/nanosat/0BDE10E00BDE10E0/CLAIRE" /><Parameter format="date" id="2113" name="startDate" value="2017/07/26" /><Parameter format="date" id="2114" name="endDate" value="2017/07/26" /><Parameter format="time" id="2115" name="startTime" value="15:00:00" /><Parameter format="time" id="2116" name="endTime" value="16:00:00" /><Parameter format="int" id="2118" name="delay" value="30" /><Parameter format="int" id="2119" name="walk" value="1" /><Parameter format="int" id="2120" name="online" value="0" /></Operation><Operation id="212" name="printNumberOfBlock" priority="2" type="self" /></ReadUnit><ProcUnit datatype="VoltageProc" id="22" inputId="21" name="VoltageProc"><Operation id="221" name="run" priority="1" type="self" /></ProcUnit><ProcUnit datatype="SpectraProc" id="23" inputId="22" name="SpectraProc"><Operation id="231" name="run" priority="1" type="self"><Parameter format="int" id="2311" name="nFFTPoints" value="128" /><Parameter format="int" id="2312" name="nProfiles" value="128" /><Parameter format="pairslist" id="2313" name="pairsList" value="(0,1),(0,2),(1,2)" /></Operation><Operation id="232" name="setRadarFrequency" priority="2" type="self"><Parameter format="float" id="2321" name="frequency" value="445000000.0" /></Operation><Operation id="233" name="IncohInt" priority="3" type="other"><Parameter format="float" id="2331" name="timeInterval" value="128" /></Operation><Operation id="234" name="removeDC" priority="4" type="self"><Parameter format="int" id="2341" name="mode" value="2" /></Operation><Operation id="235" name="RTIPlot" priority="5" type="other"><Parameter format="int" id="2351" name="id" value="237" /><Parameter format="float" id="2352" name="xmin" value="9.0" /><Parameter format="float" id="2353" name="xmax" value="16.0" /><Parameter format="float" id="2354" name="zmin" value="15.0" /><Parameter format="float" id="2355" name="zmax" value="50.0" /></Operation></ProcUnit></Project> No newline at end of file
Show file before | Show file after | Hide comments
@@ -0,0 +1,81
1 import schainpy
2 from schainpy.model import Operation, ProcessingUnit
3 from importlib import import_module
4 from pydoc import locate
5
6 def clean_modules(module):
7 noEndsUnder = [x for x in module if not x.endswith('__')]
8 noStartUnder = [x for x in noEndsUnder if not x.startswith('__')]
9 noFullUpper = [x for x in noStartUnder if not x.isupper()]
10 return noFullUpper
11
12 def check_module(possible, instance):
13 def check(x):
14 try:
15 instancia = locate('schainpy.model.{}'.format(x))
16 return isinstance(instancia(), instance)
17 except Exception as e:
18 return False
19 clean = clean_modules(possible)
20 return [x for x in clean if check(x)]
21
22
23 def getProcs():
24 module = dir(import_module('schainpy.model'))
25 procs = check_module(module, ProcessingUnit)
26 try:
27 procs.remove('ProcessingUnit')
28 except Exception as e:
29 pass
30 return procs
31
32 def getOperations():
33 module = dir(import_module('schainpy.model'))
34 noProcs = [x for x in module if not x.endswith('Proc')]
35 operations = check_module(noProcs, Operation)
36 try:
37 operations.remove('Operation')
38 except Exception as e:
39 pass
40 return operations
41
42 def getArgs(op):
43 module = locate('schainpy.model.{}'.format(op))
44 args = module().getAllowedArgs()
45 try:
46 args.remove('self')
47 except Exception as e:
48 pass
49 try:
50 args.remove('dataOut')
51 except Exception as e:
52 pass
53 return args
54
55 def getAll():
56 allModules = dir(import_module('schainpy.model'))
57 modules = check_module(allModules, Operation)
58 modules.extend(check_module(allModules, ProcessingUnit))
59 return modules
60
61 def formatArgs(op):
62 args = getArgs(op)
63
64 argsAsKey = ["\t'{}'".format(x) for x in args]
65 argsFormatted = ": 'string',\n".join(argsAsKey)
66
67 print op
68 print "parameters = { \n" + argsFormatted + ": 'string',\n }"
69 print '\n'
70
71
72 if __name__ == "__main__":
73 getAll()
74 [formatArgs(x) for x in getAll()]
75
76 '''
77 parameters = {
78 'id': ,
79 'wintitle': ,
80 }
81 ''' No newline at end of file
Show file before | Show file after | Hide comments
@@ -105,9 +105,11 ENV/
105
105
106 .vscode
106 .vscode
107
107
108 schainpy/scripts/
109 schaingui/node_modules/
108 .svn/
110 .svn/
109 *.png
111 *.png
110 *.pyc
112 *.pyc
111 schainpy/scripts
113 *.xml
112
114 *.log
113 schaingui/node_modules
115 trash No newline at end of file
Show file before | Show file after | Hide comments
@@ -8,24 +8,39 Signal Chain (SCh) is a radar data processing library developed using [Python](w
8
8
9 Install system dependencies, clone the latest version from [git](http://jro-dev.igp.gob.pe/rhodecode/schain/) and install it as a normal python package.
9 Install system dependencies, clone the latest version from [git](http://jro-dev.igp.gob.pe/rhodecode/schain/) and install it as a normal python package.
10
10
11 ### Linux based system
11 ```
12 ```
12 $ sudo apt-get install python-pip python-dev gfortran libpng-dev freetype* libblas-dev liblapack-dev libatlas-base-dev python-qt4 python-tk libssl-dev libhdf5-dev
13 $ sudo apt-get install python-pip python-dev gfortran libpng-dev freetype* libblas-dev liblapack-dev libatlas-base-dev python-qt4 python-tk libssl-dev libhdf5-dev
13 $ sudo pip install numpy
14 $ sudo pip install numpy
14 $ git clone http://jro-dev.igp.gob.pe/rhodecode/schain/
15 $ git clone http://jro-dev.igp.gob.pe/rhodecode/schain/
15 $ cd schain
16 $ cd schain
16 $ sudo pip install ./
17 $ sudo pip install ./
17 ```
18
19 **Its recommended to install schain in a virtual environment**
20
18
21 ```
19 ```
20 **It is recommended to install schain in a virtual environment**
21 ```
22 $ sudo pip install virtualenv
22 $ sudo pip install virtualenv
23 $ virtualenv /path/to/virtual --system-site-packages
23 $ virtualenv /path/to/virtual --system-site-packages
24 $ source /path/to/virtual/bin/activate
24 $ source /path/to/virtual/bin/activate
25 (virtual) $ cd schain
25 (virtual) $ cd schain
26 (virtual) $ pip install ./
26 (virtual) $ pip install ./
27
27 ```
28 ```
28
29
30 ### MAC Os
31 ```
32 $ brew install cartr/qt4/pyqt
33 $ git clone http://jro-dev.igp.gob.pe/rhodecode/schain/
34 $ cd schain
35 $ pip install ./
36 ```
37
38 if ```pip install ./``` does not work, install a proper python enviroment, and repeat the steps.
39 ```
40 $ brew install python
41 ```
42
43
29 ## First Script
44 ## First Script
30
45
31 Read Spectra data (.pdata) - remove dc - plot spectra & RTI
46 Read Spectra data (.pdata) - remove dc - plot spectra & RTI
Show file before | Show file after | Hide comments
@@ -8,12 +8,13 save_stdout = sys.stdout
8 sys.stdout = open('trash', 'w')
8 sys.stdout = open('trash', 'w')
9 from multiprocessing import cpu_count
9 from multiprocessing import cpu_count
10 from schaincli import templates
10 from schaincli import templates
11 from schainpy import controller_api
11 from schainpy.controller import Project
12 from schainpy.model import Operation, ProcessingUnit
12 from schainpy.model import Operation, ProcessingUnit
13 from schainpy.utils import log
13 from schainpy.utils import log
14 from importlib import import_module
14 from importlib import import_module
15 from pydoc import locate
15 from pydoc import locate
16 from fuzzywuzzy import process
16 from fuzzywuzzy import process
17 from schainpy.utils import paramsFinder
17 sys.stdout = save_stdout
18 sys.stdout = save_stdout
18
19
19
20
@@ -75,47 +76,25 def search(nextcommand):
75 if nextcommand is None:
76 if nextcommand is None:
76 log.error('There is no Operation/ProcessingUnit to search')
77 log.error('There is no Operation/ProcessingUnit to search')
77 elif nextcommand == 'procs':
78 elif nextcommand == 'procs':
78 module = dir(import_module('schainpy.model'))
79 procs = paramsFinder.getProcs()
79 procs = check_module(module, ProcessingUnit)
80 try:
81 procs.remove('ProcessingUnit')
82 except Exception as e:
83 pass
84 log.success('Current ProcessingUnits are:\n\033[1m{}\033[0m'.format('\n'.join(procs)))
80 log.success('Current ProcessingUnits are:\n\033[1m{}\033[0m'.format('\n'.join(procs)))
85
81
86 elif nextcommand == 'operations':
82 elif nextcommand == 'operations':
87 module = dir(import_module('schainpy.model'))
83 operations = paramsFinder.getOperations()
88 noProcs = [x for x in module if not x.endswith('Proc')]
89 operations = check_module(noProcs, Operation)
90 try:
91 operations.remove('Operation')
92 except Exception as e:
93 pass
94 log.success('Current Operations are:\n\033[1m{}\033[0m'.format('\n'.join(operations)))
84 log.success('Current Operations are:\n\033[1m{}\033[0m'.format('\n'.join(operations)))
95 else:
85 else:
96 try:
86 try:
97 module = locate('schainpy.model.{}'.format(nextcommand))
87 args = paramsFinder.getArgs(nextcommand)
98 args = module().getAllowedArgs()
99 log.warning('Use this feature with caution. It may not return all the allowed arguments')
88 log.warning('Use this feature with caution. It may not return all the allowed arguments')
100 try:
101 args.remove('self')
102 except Exception as e:
103 pass
104 try:
105 args.remove('dataOut')
106 except Exception as e:
107 pass
108 if len(args) == 0:
89 if len(args) == 0:
109 log.success('{} has no arguments'.format(nextcommand))
90 log.success('{} has no arguments'.format(nextcommand))
110 else:
91 else:
111 log.success('Showing arguments of {} are:\n\033[1m{}\033[0m'.format(nextcommand, '\n'.join(args)))
92 log.success('Showing arguments of {} are:\n\033[1m{}\033[0m'.format(nextcommand, '\n'.join(args)))
112 except Exception as e:
93 except Exception as e:
113 log.error('Module {} does not exists'.format(nextcommand))
94 log.error('Module {} does not exists'.format(nextcommand))
114 allModules = dir(import_module('schainpy.model'))
95 allModules = paramsFinder.getAll()
115 module = check_module(allModules, Operation)
96 similar = process.extractOne(nextcommand, allModules)[0]
116 module.extend(check_module(allModules, ProcessingUnit))
97 log.success('Showing {} instead'.format(similar))
117 similar = process.extractOne(nextcommand, module)[0]
118 log.success('Searching {} instead'.format(similar))
119 search(similar)
98 search(similar)
120
99
121
100
@@ -171,18 +150,8 def test():
171
150
172
151
173 def runFromXML(filename):
152 def runFromXML(filename):
174 controller = controller_api.ControllerThread()
153 controller = Project()
175 if not controller.readXml(filename):
154 if not controller.readXml(filename):
176 return
155 return
177
178 plotterObj = controller.useExternalPlotter()
179
180 controller.start()
156 controller.start()
181 plotterObj.start()
182
183 cliLogger("Finishing all processes")
184
185 controller.join(5)
186
187 cliLogger("End of script")
188 return
157 return
Show file before | Show file after | Hide comments
1 NO CONTENT: modified file
NO CONTENT: modified file
Show file before | Show file after | Hide comments
@@ -9,53 +9,53 import datetime
9 import traceback
9 import traceback
10 import math
10 import math
11 import time
11 import time
12 from multiprocessing import Process, Queue, cpu_count
12 from multiprocessing import Process, cpu_count
13
14 import schainpy
15 import schainpy.admin
16
13
17 from xml.etree.ElementTree import ElementTree, Element, SubElement, tostring
14 from xml.etree.ElementTree import ElementTree, Element, SubElement, tostring
18 from xml.dom import minidom
15 from xml.dom import minidom
19
16
17 import schainpy
18 import schainpy.admin
20 from schainpy.model import *
19 from schainpy.model import *
21 from time import sleep
20 from schainpy.utils import log
22
21
23 def prettify(elem):
22 DTYPES = {
24 """Return a pretty-printed XML string for the Element.
23 'Voltage': '.r',
25 """
24 'Spectra': '.pdata'
26 rough_string = tostring(elem, 'utf-8')
25 }
27 reparsed = minidom.parseString(rough_string)
28 return reparsed.toprettyxml(indent=" ")
29
26
30 def multiSchain(child, nProcess=cpu_count(), startDate=None, endDate=None, by_day=False):
27 def MPProject(project, n=cpu_count()):
31 skip = 0
28 '''
32 cursor = 0
29 Project wrapper to run schain in n processes
33 nFiles = None
30 '''
34 processes = []
31
35 dt1 = datetime.datetime.strptime(startDate, '%Y/%m/%d')
32 rconf = project.getReadUnitObj()
36 dt2 = datetime.datetime.strptime(endDate, '%Y/%m/%d')
33 op = rconf.getOperationObj('run')
34 dt1 = op.getParameterValue('startDate')
35 dt2 = op.getParameterValue('endDate')
37 days = (dt2 - dt1).days
36 days = (dt2 - dt1).days
38
37
39 for day in range(days+1):
38 for day in range(days+1):
40 skip = 0
39 skip = 0
41 cursor = 0
40 cursor = 0
42 q = Queue()
43 processes = []
41 processes = []
44 dt = (dt1 + datetime.timedelta(day)).strftime('%Y/%m/%d')
42 dt = dt1 + datetime.timedelta(day)
45 firstProcess = Process(target=child, args=(cursor, skip, q, dt))
43 dt_str = dt.strftime('%Y/%m/%d')
46 firstProcess.start()
44 reader = JRODataReader()
47 if by_day:
45 paths, files = reader.searchFilesOffLine(path=rconf.path,
48 continue
46 startDate=dt,
49 nFiles = q.get()
47 endDate=dt,
48 ext=DTYPES[rconf.datatype])
49 nFiles = len(files)
50 if nFiles==0:
50 if nFiles == 0:
51 continue
51 continue
52 firstProcess.terminate()
52 skip = int(math.ceil(nFiles/n))
53 skip = int(math.ceil(nFiles/nProcess))
53 while nFiles > cursor*skip:
54 while True:
54 rconf.update(startDate=dt_str, endDate=dt_str, cursor=cursor,
55 processes.append(Process(target=child, args=(cursor, skip, q, dt)))
55 skip=skip)
56 processes[cursor].start()
56 p = project.clone()
57 if nFiles < cursor*skip:
57 p.start()
58 break
58 processes.append(p)
59 cursor += 1
59 cursor += 1
60
60
61 def beforeExit(exctype, value, trace):
61 def beforeExit(exctype, value, trace):
@@ -72,7 +72,6 def multiSchain(child, nProcess=cpu_count(), startDate=None, endDate=None, by_da
72
72
73 time.sleep(3)
73 time.sleep(3)
74
74
75
76 class ParameterConf():
75 class ParameterConf():
77
76
78 id = None
77 id = None
@@ -109,7 +108,7 class ParameterConf():
109 return self.__formated_value
108 return self.__formated_value
110
109
111 if value == '':
110 if value == '':
112 raise ValueError, "%s: This parameter value is empty" %self.name
111 raise ValueError, '%s: This parameter value is empty' %self.name
113
112
114 if format == 'list':
113 if format == 'list':
115 strList = value.split(',')
114 strList = value.split(',')
@@ -119,10 +118,10 class ParameterConf():
119 return self.__formated_value
118 return self.__formated_value
120
119
121 if format == 'intlist':
120 if format == 'intlist':
122 """
121 '''
123 Example:
122 Example:
124 value = (0,1,2)
123 value = (0,1,2)
125 """
124 '''
126
125
127 new_value = ast.literal_eval(value)
126 new_value = ast.literal_eval(value)
128
127
@@ -134,10 +133,10 class ParameterConf():
134 return self.__formated_value
133 return self.__formated_value
135
134
136 if format == 'floatlist':
135 if format == 'floatlist':
137 """
136 '''
138 Example:
137 Example:
139 value = (0.5, 1.4, 2.7)
138 value = (0.5, 1.4, 2.7)
140 """
139 '''
141
140
142 new_value = ast.literal_eval(value)
141 new_value = ast.literal_eval(value)
143
142
@@ -167,38 +166,38 class ParameterConf():
167 return self.__formated_value
166 return self.__formated_value
168
167
169 if format == 'pairslist':
168 if format == 'pairslist':
170 """
169 '''
171 Example:
170 Example:
172 value = (0,1),(1,2)
171 value = (0,1),(1,2)
173 """
172 '''
174
173
175 new_value = ast.literal_eval(value)
174 new_value = ast.literal_eval(value)
176
175
177 if type(new_value) not in (tuple, list):
176 if type(new_value) not in (tuple, list):
178 raise ValueError, "%s has to be a tuple or list of pairs" %value
177 raise ValueError, '%s has to be a tuple or list of pairs' %value
179
178
180 if type(new_value[0]) not in (tuple, list):
179 if type(new_value[0]) not in (tuple, list):
181 if len(new_value) != 2:
180 if len(new_value) != 2:
182 raise ValueError, "%s has to be a tuple or list of pairs" %value
181 raise ValueError, '%s has to be a tuple or list of pairs' %value
183 new_value = [new_value]
182 new_value = [new_value]
184
183
185 for thisPair in new_value:
184 for thisPair in new_value:
186 if len(thisPair) != 2:
185 if len(thisPair) != 2:
187 raise ValueError, "%s has to be a tuple or list of pairs" %value
186 raise ValueError, '%s has to be a tuple or list of pairs' %value
188
187
189 self.__formated_value = new_value
188 self.__formated_value = new_value
190
189
191 return self.__formated_value
190 return self.__formated_value
192
191
193 if format == 'multilist':
192 if format == 'multilist':
194 """
193 '''
195 Example:
194 Example:
196 value = (0,1,2),(3,4,5)
195 value = (0,1,2),(3,4,5)
197 """
196 '''
198 multiList = ast.literal_eval(value)
197 multiList = ast.literal_eval(value)
199
198
200 if type(multiList[0]) == int:
199 if type(multiList[0]) == int:
201 multiList = ast.literal_eval("(" + value + ")")
200 multiList = ast.literal_eval('(' + value + ')')
202
201
203 self.__formated_value = multiList
202 self.__formated_value = multiList
204
203
@@ -260,7 +259,7 class ParameterConf():
260
259
261 def printattr(self):
260 def printattr(self):
262
261
263 print "Parameter[%s]: name = %s, value = %s, format = %s" %(self.id, self.name, self.value, self.format)
262 print 'Parameter[%s]: name = %s, value = %s, format = %s' %(self.id, self.name, self.value, self.format)
264
263
265 class OperationConf():
264 class OperationConf():
266
265
@@ -369,6 +368,8 class OperationConf():
369
368
370 def addParameter(self, name, value, format='str'):
369 def addParameter(self, name, value, format='str'):
371
370
371 if value is None:
372 return None
372 id = self.__getNewId()
373 id = self.__getNewId()
373
374
374 parmConfObj = ParameterConf()
375 parmConfObj = ParameterConf()
@@ -428,7 +429,7 class OperationConf():
428
429
429 def printattr(self):
430 def printattr(self):
430
431
431 print "%s[%s]: name = %s, type = %s, priority = %s" %(self.ELEMENTNAME,
432 print '%s[%s]: name = %s, type = %s, priority = %s' %(self.ELEMENTNAME,
432 self.id,
433 self.id,
433 self.name,
434 self.name,
434 self.type,
435 self.type,
@@ -441,12 +442,11 class OperationConf():
441
442
442
443
443 if self.type == 'self':
444 if self.type == 'self':
444 raise ValueError, "This operation type cannot be created"
445 raise ValueError, 'This operation type cannot be created'
445
446
446 if self.type == 'plotter':
447 if self.type == 'plotter':
447 #Plotter(plotter_name)
448 if not plotter_queue:
448 if not plotter_queue:
449 raise ValueError, "plotter_queue is not defined. Use:\nmyProject = Project()\nmyProject.setPlotterQueue(plotter_queue)"
449 raise ValueError, 'plotter_queue is not defined. Use:\nmyProject = Project()\nmyProject.setPlotterQueue(plotter_queue)'
450
450
451 opObj = Plotter(self.name, plotter_queue)
451 opObj = Plotter(self.name, plotter_queue)
452
452
@@ -561,7 +561,7 class ProcUnitConf():
561
561
562 #Compatible with old signal chain version
562 #Compatible with old signal chain version
563 if datatype==None and name==None:
563 if datatype==None and name==None:
564 raise ValueError, "datatype or name should be defined"
564 raise ValueError, 'datatype or name should be defined'
565
565
566 if name==None:
566 if name==None:
567 if 'Proc' in datatype:
567 if 'Proc' in datatype:
@@ -592,7 +592,7 class ProcUnitConf():
592
592
593 def addParameter(self, **kwargs):
593 def addParameter(self, **kwargs):
594 '''
594 '''
595 Add parameters to "run" operation
595 Add parameters to 'run' operation
596 '''
596 '''
597 opObj = self.opConfObjList[0]
597 opObj = self.opConfObjList[0]
598
598
@@ -630,11 +630,11 class ProcUnitConf():
630 self.datatype = upElement.get('datatype')
630 self.datatype = upElement.get('datatype')
631 self.inputId = upElement.get('inputId')
631 self.inputId = upElement.get('inputId')
632
632
633 if self.ELEMENTNAME == "ReadUnit":
633 if self.ELEMENTNAME == 'ReadUnit':
634 self.datatype = self.datatype.replace("Reader", "")
634 self.datatype = self.datatype.replace('Reader', '')
635
635
636 if self.ELEMENTNAME == "ProcUnit":
636 if self.ELEMENTNAME == 'ProcUnit':
637 self.datatype = self.datatype.replace("Proc", "")
637 self.datatype = self.datatype.replace('Proc', '')
638
638
639 if self.inputId == 'None':
639 if self.inputId == 'None':
640 self.inputId = '0'
640 self.inputId = '0'
@@ -650,7 +650,7 class ProcUnitConf():
650
650
651 def printattr(self):
651 def printattr(self):
652
652
653 print "%s[%s]: name = %s, datatype = %s, inputId = %s" %(self.ELEMENTNAME,
653 print '%s[%s]: name = %s, datatype = %s, inputId = %s' %(self.ELEMENTNAME,
654 self.id,
654 self.id,
655 self.name,
655 self.name,
656 self.datatype,
656 self.datatype,
@@ -751,12 +751,12 class ReadUnitConf(ProcUnitConf):
751
751
752 return self.ELEMENTNAME
752 return self.ELEMENTNAME
753
753
754 def setup(self, id, name, datatype, path='', startDate="", endDate="", startTime="",
754 def setup(self, id, name, datatype, path='', startDate='', endDate='',
755 endTime="", parentId=None, queue=None, server=None, **kwargs):
755 startTime='', endTime='', parentId=None, server=None, **kwargs):
756
756
757 #Compatible with old signal chain version
757 #Compatible with old signal chain version
758 if datatype==None and name==None:
758 if datatype==None and name==None:
759 raise ValueError, "datatype or name should be defined"
759 raise ValueError, 'datatype or name should be defined'
760
760
761 if name==None:
761 if name==None:
762 if 'Reader' in datatype:
762 if 'Reader' in datatype:
@@ -775,39 +775,28 class ReadUnitConf(ProcUnitConf):
775 self.endDate = endDate
775 self.endDate = endDate
776 self.startTime = startTime
776 self.startTime = startTime
777 self.endTime = endTime
777 self.endTime = endTime
778
779 self.inputId = '0'
778 self.inputId = '0'
780 self.parentId = parentId
779 self.parentId = parentId
781 self.queue = queue
782 self.server = server
780 self.server = server
783 self.addRunOperation(**kwargs)
781 self.addRunOperation(**kwargs)
784
782
785 def update(self, datatype, path, startDate, endDate, startTime, endTime, parentId=None, name=None, **kwargs):
783 def update(self, **kwargs):
786
787 #Compatible with old signal chain version
788 if datatype==None and name==None:
789 raise ValueError, "datatype or name should be defined"
790
784
791 if name==None:
785 if 'datatype' in kwargs:
786 datatype = kwargs.pop('datatype')
792 if 'Reader' in datatype:
787 if 'Reader' in datatype:
793 name = datatype
788 self.name = datatype
794 else:
789 else:
795 name = '%sReader' %(datatype)
790 self.name = '%sReader' %(datatype)
791 self.datatype = self.name.replace('Reader', '')
796
792
797 if datatype==None:
793 attrs = ('path', 'startDate', 'endDate', 'startTime', 'endTime', 'parentId')
798 datatype = name.replace('Reader','')
799
794
800 self.datatype = datatype
795 for attr in attrs:
801 self.name = name
796 if attr in kwargs:
802 self.path = path
797 setattr(self, attr, kwargs.pop(attr))
803 self.startDate = startDate
804 self.endDate = endDate
805 self.startTime = startTime
806 self.endTime = endTime
807
798
808 self.inputId = '0'
799 self.inputId = '0'
809 self.parentId = parentId
810
811 self.updateRunOperation(**kwargs)
800 self.updateRunOperation(**kwargs)
812
801
813 def removeOperations(self):
802 def removeOperations(self):
@@ -828,7 +817,7 class ReadUnitConf(ProcUnitConf):
828 opObj.addParameter(name='endDate' , value=self.endDate, format='date')
817 opObj.addParameter(name='endDate', value=self.endDate, format='date')
829 opObj.addParameter(name='startTime' , value=self.startTime, format='time')
818 opObj.addParameter(name='startTime', value=self.startTime, format='time')
830 opObj.addParameter(name='endTime' , value=self.endTime, format='time')
819 opObj.addParameter(name='endTime', value=self.endTime, format='time')
831 opObj.addParameter(name='queue' , value=self.queue, format='obj')
820
832 for key, value in kwargs.items():
821 for key, value in kwargs.items():
833 opObj.addParameter(name=key, value=value, format=type(value).__name__)
822 opObj.addParameter(name=key, value=value, format=type(value).__name__)
834 else:
823 else:
@@ -854,17 +843,6 class ReadUnitConf(ProcUnitConf):
854
843
855 return opObj
844 return opObj
856
845
857 # def makeXml(self, projectElement):
858 #
859 # procUnitElement = SubElement(projectElement, self.ELEMENTNAME)
860 # procUnitElement.set('id', str(self.id))
861 # procUnitElement.set('name', self.name)
862 # procUnitElement.set('datatype', self.datatype)
863 # procUnitElement.set('inputId', str(self.inputId))
864 #
865 # for opConfObj in self.opConfObjList:
866 # opConfObj.makeXml(procUnitElement)
867
868 def readXml(self, upElement):
846 def readXml(self, upElement):
869
847
870 self.id = upElement.get('id')
848 self.id = upElement.get('id')
@@ -872,8 +850,8 class ReadUnitConf(ProcUnitConf):
872 self.datatype = upElement.get('datatype')
850 self.datatype = upElement.get('datatype')
873 self.inputId = upElement.get('inputId')
851 self.inputId = upElement.get('inputId')
874
852
875 if self.ELEMENTNAME == "ReadUnit":
853 if self.ELEMENTNAME == 'ReadUnit':
876 self.datatype = self.datatype.replace("Reader", "")
854 self.datatype = self.datatype.replace('Reader', '')
877
855
878 if self.inputId == 'None':
856 if self.inputId == 'None':
879 self.inputId = '0'
857 self.inputId = '0'
@@ -894,10 +872,10 class ReadUnitConf(ProcUnitConf):
894 self.startTime = opConfObj.getParameterValue('startTime')
872 self.startTime = opConfObj.getParameterValue('startTime')
895 self.endTime = opConfObj.getParameterValue('endTime')
873 self.endTime = opConfObj.getParameterValue('endTime')
896
874
897 class Project():
875 class Project(Process):
898
876
899 id = None
877 id = None
900 name = None
878 # name = None
901 description = None
879 description = None
902 filename = None
880 filename = None
903
881
@@ -909,8 +887,9 class Project():
909
887
910 def __init__(self, plotter_queue=None):
888 def __init__(self, plotter_queue=None):
911
889
890 Process.__init__(self)
912 self.id = None
891 self.id = None
913 self.name = None
892 # self.name = None
914 self.description = None
893 self.description = None
915
894
916 self.plotterQueue = plotter_queue
895 self.plotterQueue = plotter_queue
@@ -962,17 +941,26 class Project():
962
941
963 self.procUnitConfObjDict = newProcUnitConfObjDict
942 self.procUnitConfObjDict = newProcUnitConfObjDict
964
943
965 def setup(self, id, name, description):
944 def setup(self, id, name='', description=''):
966
945
946 print
947 print '*'*60
948 print ' Starting SIGNAL CHAIN PROCESSING v%s ' % schainpy.__version__
949 print '*'*60
950 print
967 self.id = str(id)
951 self.id = str(id)
968 self.name = name
969 self.description = description
952 self.description = description
970
953
971 def update(self, name, description):
954 def update(self, name, description):
972
955
973 self.name = name
974 self.description = description
956 self.description = description
975
957
958 def clone(self):
959
960 p = Project()
961 p.procUnitConfObjDict = self.procUnitConfObjDict
962 return p
963
976 def addReadUnit(self, id=None, datatype=None, name=None, **kwargs):
964 def addReadUnit(self, id=None, datatype=None, name=None, **kwargs):
977
965
978 if id is None:
966 if id is None:
@@ -1012,7 +1000,7 class Project():
1012 def getReadUnitObj(self):
1000 def getReadUnitObj(self):
1013
1001
1014 for obj in self.procUnitConfObjDict.values():
1002 for obj in self.procUnitConfObjDict.values():
1015 if obj.getElementName() == "ReadUnit":
1003 if obj.getElementName() == 'ReadUnit':
1016 return obj
1004 return obj
1017
1005
1018 return None
1006 return None
@@ -1057,20 +1045,20 class Project():
1057 if self.filename:
1045 if self.filename:
1058 filename = self.filename
1046 filename = self.filename
1059 else:
1047 else:
1060 filename = "schain.xml"
1048 filename = 'schain.xml'
1061
1049
1062 if not filename:
1050 if not filename:
1063 print "filename has not been defined. Use setFilename(filename) for do it."
1051 print 'filename has not been defined. Use setFilename(filename) for do it.'
1064 return 0
1052 return 0
1065
1053
1066 abs_file = os.path.abspath(filename)
1054 abs_file = os.path.abspath(filename)
1067
1055
1068 if not os.access(os.path.dirname(abs_file), os.W_OK):
1056 if not os.access(os.path.dirname(abs_file), os.W_OK):
1069 print "No write permission on %s" %os.path.dirname(abs_file)
1057 print 'No write permission on %s' %os.path.dirname(abs_file)
1070 return 0
1058 return 0
1071
1059
1072 if os.path.isfile(abs_file) and not(os.access(abs_file, os.W_OK)):
1060 if os.path.isfile(abs_file) and not(os.access(abs_file, os.W_OK)):
1073 print "File %s already exists and it could not be overwriten" %abs_file
1061 print 'File %s already exists and it could not be overwriten' %abs_file
1074 return 0
1062 return 0
1075
1063
1076 self.makeXml()
1064 self.makeXml()
@@ -1084,13 +1072,13 class Project():
1084 def readXml(self, filename = None):
1072 def readXml(self, filename = None):
1085
1073
1086 if not filename:
1074 if not filename:
1087 print "filename is not defined"
1075 print 'filename is not defined'
1088 return 0
1076 return 0
1089
1077
1090 abs_file = os.path.abspath(filename)
1078 abs_file = os.path.abspath(filename)
1091
1079
1092 if not os.path.isfile(abs_file):
1080 if not os.path.isfile(abs_file):
1093 print "%s file does not exist" %abs_file
1081 print '%s file does not exist' %abs_file
1094 return 0
1082 return 0
1095
1083
1096 self.projectElement = None
1084 self.projectElement = None
@@ -1099,7 +1087,7 class Project():
1099 try:
1087 try:
1100 self.projectElement = ElementTree().parse(abs_file)
1088 self.projectElement = ElementTree().parse(abs_file)
1101 except:
1089 except:
1102 print "Error reading %s, verify file format" %filename
1090 print 'Error reading %s, verify file format' %filename
1103 return 0
1091 return 0
1104
1092
1105 self.project = self.projectElement.tag
1093 self.project = self.projectElement.tag
@@ -1136,7 +1124,7 class Project():
1136
1124
1137 def printattr(self):
1125 def printattr(self):
1138
1126
1139 print "Project[%s]: name = %s, description = %s" %(self.id,
1127 print 'Project[%s]: name = %s, description = %s' %(self.id,
1140 self.name,
1128 self.name,
1141 self.description)
1129 self.description)
1142
1130
@@ -1170,7 +1158,7 class Project():
1170
1158
1171 self.__connect(puObjIN, thisPUObj)
1159 self.__connect(puObjIN, thisPUObj)
1172
1160
1173 def __handleError(self, procUnitConfObj, send_email=True):
1161 def __handleError(self, procUnitConfObj, send_email=False):
1174
1162
1175 import socket
1163 import socket
1176
1164
@@ -1178,33 +1166,33 class Project():
1178 sys.exc_info()[1],
1166 sys.exc_info()[1],
1179 sys.exc_info()[2])
1167 sys.exc_info()[2])
1180
1168
1181 print "***** Error occurred in %s *****" %(procUnitConfObj.name)
1169 print '***** Error occurred in %s *****' %(procUnitConfObj.name)
1182 print "***** %s" %err[-1]
1170 print '***** %s' %err[-1]
1183
1171
1184 message = "".join(err)
1172 message = ''.join(err)
1185
1173
1186 sys.stderr.write(message)
1174 sys.stderr.write(message)
1187
1175
1188 if not send_email:
1176 if not send_email:
1189 return
1177 return
1190
1178
1191 subject = "SChain v%s: Error running %s\n" %(schainpy.__version__, procUnitConfObj.name)
1179 subject = 'SChain v%s: Error running %s\n' %(schainpy.__version__, procUnitConfObj.name)
1192
1180
1193 subtitle = "%s: %s\n" %(procUnitConfObj.getElementName() ,procUnitConfObj.name)
1181 subtitle = '%s: %s\n' %(procUnitConfObj.getElementName() ,procUnitConfObj.name)
1194 subtitle += "Hostname: %s\n" %socket.gethostbyname(socket.gethostname())
1182 subtitle += 'Hostname: %s\n' %socket.gethostbyname(socket.gethostname())
1195 subtitle += "Working directory: %s\n" %os.path.abspath("./")
1183 subtitle += 'Working directory: %s\n' %os.path.abspath('./')
1196 subtitle += "Configuration file: %s\n" %self.filename
1184 subtitle += 'Configuration file: %s\n' %self.filename
1197 subtitle += "Time: %s\n" %str(datetime.datetime.now())
1185 subtitle += 'Time: %s\n' %str(datetime.datetime.now())
1198
1186
1199 readUnitConfObj = self.getReadUnitObj()
1187 readUnitConfObj = self.getReadUnitObj()
1200 if readUnitConfObj:
1188 if readUnitConfObj:
1201 subtitle += "\nInput parameters:\n"
1189 subtitle += '\nInput parameters:\n'
1202 subtitle += "[Data path = %s]\n" %readUnitConfObj.path
1190 subtitle += '[Data path = %s]\n' %readUnitConfObj.path
1203 subtitle += "[Data type = %s]\n" %readUnitConfObj.datatype
1191 subtitle += '[Data type = %s]\n' %readUnitConfObj.datatype
1204 subtitle += "[Start date = %s]\n" %readUnitConfObj.startDate
1192 subtitle += '[Start date = %s]\n' %readUnitConfObj.startDate
1205 subtitle += "[End date = %s]\n" %readUnitConfObj.endDate
1193 subtitle += '[End date = %s]\n' %readUnitConfObj.endDate
1206 subtitle += "[Start time = %s]\n" %readUnitConfObj.startTime
1194 subtitle += '[Start time = %s]\n' %readUnitConfObj.startTime
1207 subtitle += "[End time = %s]\n" %readUnitConfObj.endTime
1195 subtitle += '[End time = %s]\n' %readUnitConfObj.endTime
1208
1196
1209 adminObj = schainpy.admin.SchainNotify()
1197 adminObj = schainpy.admin.SchainNotify()
1210 adminObj.sendAlert(message=message,
1198 adminObj.sendAlert(message=message,
@@ -1219,15 +1207,15 class Project():
1219 return 0
1207 return 0
1220
1208
1221 def runController(self):
1209 def runController(self):
1222 """
1210 '''
1223 returns 0 when this process has been stopped, 1 otherwise
1211 returns 0 when this process has been stopped, 1 otherwise
1224 """
1212 '''
1225
1213
1226 if self.isPaused():
1214 if self.isPaused():
1227 print "Process suspended"
1215 print 'Process suspended'
1228
1216
1229 while True:
1217 while True:
1230 sleep(0.1)
1218 time.sleep(0.1)
1231
1219
1232 if not self.isPaused():
1220 if not self.isPaused():
1233 break
1221 break
@@ -1235,10 +1223,10 class Project():
1235 if self.isStopped():
1223 if self.isStopped():
1236 break
1224 break
1237
1225
1238 print "Process reinitialized"
1226 print 'Process reinitialized'
1239
1227
1240 if self.isStopped():
1228 if self.isStopped():
1241 print "Process stopped"
1229 print 'Process stopped'
1242 return 0
1230 return 0
1243
1231
1244 return 1
1232 return 1
@@ -1249,23 +1237,22 class Project():
1249
1237
1250 def setPlotterQueue(self, plotter_queue):
1238 def setPlotterQueue(self, plotter_queue):
1251
1239
1252 raise NotImplementedError, "Use schainpy.controller_api.ControllerThread instead Project class"
1240 raise NotImplementedError, 'Use schainpy.controller_api.ControllerThread instead Project class'
1253
1241
1254 def getPlotterQueue(self):
1242 def getPlotterQueue(self):
1255
1243
1256 raise NotImplementedError, "Use schainpy.controller_api.ControllerThread instead Project class"
1244 raise NotImplementedError, 'Use schainpy.controller_api.ControllerThread instead Project class'
1257
1245
1258 def useExternalPlotter(self):
1246 def useExternalPlotter(self):
1259
1247
1260 raise NotImplementedError, "Use schainpy.controller_api.ControllerThread instead Project class"
1248 raise NotImplementedError, 'Use schainpy.controller_api.ControllerThread instead Project class'
1261
1249
1262 def run(self):
1250 def run(self):
1263
1251
1264 print
1252 log.success('Starting {}'.format(self.name))
1265 print "*"*60
1253
1266 print " Starting SIGNAL CHAIN PROCESSING v%s " %schainpy.__version__
1254 self.createObjects()
1267 print "*"*60
1255 self.connectObjects()
1268 print
1269
1256
1270 keyList = self.procUnitConfObjDict.keys()
1257 keyList = self.procUnitConfObjDict.keys()
1271 keyList.sort()
1258 keyList.sort()
@@ -1275,7 +1262,6 class Project():
1275 is_ok = False
1262 is_ok = False
1276
1263
1277 for procKey in keyList:
1264 for procKey in keyList:
1278 # print "Running the '%s' process with %s" %(procUnitConfObj.name, procUnitConfObj.id)
1279
1265
1280 procUnitConfObj = self.procUnitConfObjDict[procKey]
1266 procUnitConfObj = self.procUnitConfObjDict[procKey]
1281
1267
@@ -1286,19 +1272,18 class Project():
1286 is_ok = False
1272 is_ok = False
1287 break
1273 break
1288 except ValueError, e:
1274 except ValueError, e:
1289 sleep(0.5)
1275 time.sleep(0.5)
1290 self.__handleError(procUnitConfObj, send_email=True)
1276 self.__handleError(procUnitConfObj, send_email=True)
1291 is_ok = False
1277 is_ok = False
1292 break
1278 break
1293 except:
1279 except:
1294 sleep(0.5)
1280 time.sleep(0.5)
1295 self.__handleError(procUnitConfObj)
1281 self.__handleError(procUnitConfObj)
1296 is_ok = False
1282 is_ok = False
1297 break
1283 break
1298
1284
1299 #If every process unit finished so end process
1285 #If every process unit finished so end process
1300 if not(is_ok):
1286 if not(is_ok):
1301 # print "Every process unit have finished"
1302 break
1287 break
1303
1288
1304 if not self.runController():
1289 if not self.runController():
@@ -1309,11 +1294,4 class Project():
1309 procUnitConfObj = self.procUnitConfObjDict[procKey]
1294 procUnitConfObj = self.procUnitConfObjDict[procKey]
1310 procUnitConfObj.close()
1295 procUnitConfObj.close()
1311
1296
1312 print "Process finished"
1297 log.success('{} finished'.format(self.name))
1313
1314 def start(self, filename=None):
1315
1316 self.writeXml(filename)
1317 self.createObjects()
1318 self.connectObjects()
1319 self.run()
Show file before | Show file after | Hide comments
1 NO CONTENT: modified file
NO CONTENT: modified file
Show file before | Show file after | Hide comments
@@ -6218,3 +6218,6 class ShowMeConsole(QtCore.QObject):
6218 text = text[:-1]
6218 text = text[:-1]
6219
6219
6220 self.textWritten.emit(str(text))
6220 self.textWritten.emit(str(text))
6221
6222 def flush(self):
6223 pass
Show file before | Show file after | Hide comments
1 NO CONTENT: modified file
NO CONTENT: modified file
Show file before | Show file after | Hide comments
1 NO CONTENT: modified file
NO CONTENT: modified file
Show file before | Show file after | Hide comments
@@ -61,7 +61,7 class Figure(Operation):
61 figfile = None
61 figfile = None
62
62
63 created = False
63 created = False
64
64 parameters = {}
65 def __init__(self, **kwargs):
65 def __init__(self, **kwargs):
66
66
67 Operation.__init__(self, **kwargs)
67 Operation.__init__(self, **kwargs)
Show file before | Show file after | Hide comments
@@ -2,11 +2,10 import os
2 import datetime
2 import datetime
3 import numpy
3 import numpy
4 import copy
4 import copy
5
5 from schainpy.model import *
6 from figure import Figure, isRealtime
6 from figure import Figure, isRealtime
7
7
8 class CorrelationPlot(Figure):
8 class CorrelationPlot(Figure):
9
10 isConfig = None
9 isConfig = None
11 __nsubplots = None
10 __nsubplots = None
12
11
Show file before | Show file after | Hide comments
This diff has been collapsed as it changes many lines, (1113 lines changed) Show them Hide them
@@ -1,30 +1,33
1
1
2 import os
2 import os
3 import zmq
4 import time
3 import time
5 import numpy
4 import glob
6 import datetime
5 import datetime
7 import numpy as np
6 from multiprocessing import Process
7
8 import zmq
9 import numpy
8 import matplotlib
10 import matplotlib
9 import glob
10 matplotlib.use('TkAgg')
11 import matplotlib.pyplot as plt
11 import matplotlib.pyplot as plt
12 from mpl_toolkits.axes_grid1 import make_axes_locatable
12 from mpl_toolkits.axes_grid1 import make_axes_locatable
13 from matplotlib.ticker import FuncFormatter, LinearLocator
13 from matplotlib.ticker import FuncFormatter, LinearLocator, MultipleLocator
14 from multiprocessing import Process
15
14
16 from schainpy.model.proc.jroproc_base import Operation
15 from schainpy.model.proc.jroproc_base import Operation
17
16 from schainpy.utils import log
18 plt.ion()
19
17
20 func = lambda x, pos: ('%s') %(datetime.datetime.fromtimestamp(x).strftime('%H:%M'))
18 func = lambda x, pos: ('%s') %(datetime.datetime.fromtimestamp(x).strftime('%H:%M'))
21
19
22 d1970 = datetime.datetime(1970,1,1)
20 d1970 = datetime.datetime(1970, 1, 1)
23
21
22
24 class PlotData(Operation, Process):
23 class PlotData(Operation, Process):
24 '''
25 Base class for Schain plotting operations
26 '''
25
27
26 CODE = 'Figure'
28 CODE = 'Figure'
27 colormap = 'jro'
29 colormap = 'jro'
30 bgcolor = 'white'
28 CONFLATE = False
31 CONFLATE = False
29 __MAXNUMX = 80
32 __MAXNUMX = 80
30 __missing = 1E30
33 __missing = 1E30
@@ -35,54 +38,143 class PlotData(Operation, Process):
35 Process.__init__(self)
38 Process.__init__(self)
36 self.kwargs['code'] = self.CODE
39 self.kwargs['code'] = self.CODE
37 self.mp = False
40 self.mp = False
38 self.dataOut = None
41 self.data = None
39 self.isConfig = False
42 self.isConfig = False
40 self.figure = None
43 self.figures = []
41 self.axes = []
44 self.axes = []
45 self.cb_axes = []
42 self.localtime = kwargs.pop('localtime', True)
46 self.localtime = kwargs.pop('localtime', True)
43 self.show = kwargs.get('show', True)
47 self.show = kwargs.get('show', True)
44 self.save = kwargs.get('save', False)
48 self.save = kwargs.get('save', False)
45 self.colormap = kwargs.get('colormap', self.colormap)
49 self.colormap = kwargs.get('colormap', self.colormap)
46 self.colormap_coh = kwargs.get('colormap_coh', 'jet')
50 self.colormap_coh = kwargs.get('colormap_coh', 'jet')
47 self.colormap_phase = kwargs.get('colormap_phase', 'RdBu_r')
51 self.colormap_phase = kwargs.get('colormap_phase', 'RdBu_r')
48 self.showprofile = kwargs.get('showprofile', True)
52 self.colormaps = kwargs.get('colormaps', None)
49 self.title = kwargs.get('wintitle', '')
53 self.bgcolor = kwargs.get('bgcolor', self.bgcolor)
54 self.showprofile = kwargs.get('showprofile', False)
55 self.title = kwargs.get('wintitle', self.CODE.upper())
56 self.cb_label = kwargs.get('cb_label', None)
57 self.cb_labels = kwargs.get('cb_labels', None)
50 self.xaxis = kwargs.get('xaxis', 'frequency')
58 self.xaxis = kwargs.get('xaxis', 'frequency')
51 self.zmin = kwargs.get('zmin', None)
59 self.zmin = kwargs.get('zmin', None)
52 self.zmax = kwargs.get('zmax', None)
60 self.zmax = kwargs.get('zmax', None)
61 self.zlimits = kwargs.get('zlimits', None)
53 self.xmin = kwargs.get('xmin', None)
62 self.xmin = kwargs.get('xmin', None)
63 if self.xmin is not None:
64 self.xmin += 5
54 self.xmax = kwargs.get('xmax', None)
65 self.xmax = kwargs.get('xmax', None)
55 self.xrange = kwargs.get('xrange', 24)
66 self.xrange = kwargs.get('xrange', 24)
56 self.ymin = kwargs.get('ymin', None)
67 self.ymin = kwargs.get('ymin', None)
57 self.ymax = kwargs.get('ymax', None)
68 self.ymax = kwargs.get('ymax', None)
58 self.__MAXNUMY = kwargs.get('decimation', 80)
69 self.xlabel = kwargs.get('xlabel', None)
59 self.throttle_value = 5
70 self.__MAXNUMY = kwargs.get('decimation', 100)
60 self.times = []
71 self.showSNR = kwargs.get('showSNR', False)
61 #self.interactive = self.kwargs['parent']
72 self.oneFigure = kwargs.get('oneFigure', True)
73 self.width = kwargs.get('width', None)
74 self.height = kwargs.get('height', None)
75 self.colorbar = kwargs.get('colorbar', True)
76 self.factors = kwargs.get('factors', [1, 1, 1, 1, 1, 1, 1, 1])
77 self.titles = ['' for __ in range(16)]
78
79 def __setup(self):
80 '''
81 Common setup for all figures, here figures and axes are created
82 '''
83
84 self.setup()
85
86 if self.width is None:
87 self.width = 8
88
89 self.figures = []
90 self.axes = []
91 self.cb_axes = []
92 self.pf_axes = []
93 self.cmaps = []
94
95 size = '15%' if self.ncols==1 else '30%'
96 pad = '4%' if self.ncols==1 else '8%'
97
98 if self.oneFigure:
99 if self.height is None:
100 self.height = 1.4*self.nrows + 1
101 fig = plt.figure(figsize=(self.width, self.height),
102 edgecolor='k',
103 facecolor='w')
104 self.figures.append(fig)
105 for n in range(self.nplots):
106 ax = fig.add_subplot(self.nrows, self.ncols, n+1)
107 ax.tick_params(labelsize=8)
108 ax.firsttime = True
109 self.axes.append(ax)
110 if self.showprofile:
111 cax = self.__add_axes(ax, size=size, pad=pad)
112 cax.tick_params(labelsize=8)
113 self.pf_axes.append(cax)
114 else:
115 if self.height is None:
116 self.height = 3
117 for n in range(self.nplots):
118 fig = plt.figure(figsize=(self.width, self.height),
119 edgecolor='k',
120 facecolor='w')
121 ax = fig.add_subplot(1, 1, 1)
122 ax.tick_params(labelsize=8)
123 ax.firsttime = True
124 self.figures.append(fig)
125 self.axes.append(ax)
126 if self.showprofile:
127 cax = self.__add_axes(ax, size=size, pad=pad)
128 cax.tick_params(labelsize=8)
129 self.pf_axes.append(cax)
62
130
131 for n in range(self.nrows):
132 if self.colormaps is not None:
133 cmap = plt.get_cmap(self.colormaps[n])
134 else:
135 cmap = plt.get_cmap(self.colormap)
136 cmap.set_bad(self.bgcolor, 1.)
137 self.cmaps.append(cmap)
138
139 def __add_axes(self, ax, size='30%', pad='8%'):
63 '''
140 '''
64 this new parameter is created to plot data from varius channels at different figures
141 Add new axes to the given figure
65 1. crear una lista de figuras donde se puedan plotear las figuras,
66 2. dar las opciones de configuracion a cada figura, estas opciones son iguales para ambas figuras
67 3. probar?
68 '''
142 '''
69 self.ind_plt_ch = kwargs.get('ind_plt_ch', False)
143 divider = make_axes_locatable(ax)
70 self.figurelist = None
144 nax = divider.new_horizontal(size=size, pad=pad)
145 ax.figure.add_axes(nax)
146 return nax
71
147
72
148
73 def fill_gaps(self, x_buffer, y_buffer, z_buffer):
149 def setup(self):
150 '''
151 This method should be implemented in the child class, the following
152 attributes should be set:
74
153
154 self.nrows: number of rows
155 self.ncols: number of cols
156 self.nplots: number of plots (channels or pairs)
157 self.ylabel: label for Y axes
158 self.titles: list of axes title
159
160 '''
161 raise(NotImplementedError, 'Implement this method in child class')
162
163 def fill_gaps(self, x_buffer, y_buffer, z_buffer):
164 '''
165 Create a masked array for missing data
166 '''
75 if x_buffer.shape[0] < 2:
167 if x_buffer.shape[0] < 2:
76 return x_buffer, y_buffer, z_buffer
168 return x_buffer, y_buffer, z_buffer
77
169
78 deltas = x_buffer[1:] - x_buffer[0:-1]
170 deltas = x_buffer[1:] - x_buffer[0:-1]
79 x_median = np.median(deltas)
171 x_median = numpy.median(deltas)
80
172
81 index = np.where(deltas > 5*x_median)
173 index = numpy.where(deltas > 5*x_median)
82
174
83 if len(index[0]) != 0:
175 if len(index[0]) != 0:
84 z_buffer[::, index[0], ::] = self.__missing
176 z_buffer[::, index[0], ::] = self.__missing
85 z_buffer = np.ma.masked_inside(z_buffer,
177 z_buffer = numpy.ma.masked_inside(z_buffer,
86 0.99*self.__missing,
178 0.99*self.__missing,
87 1.01*self.__missing)
179 1.01*self.__missing)
88
180
@@ -100,107 +192,114 class PlotData(Operation, Process):
100
192
101 return x, y, z
193 return x, y, z
102
194
195 def format(self):
103 '''
196 '''
104 JM:
197 Set min and max values, labels, ticks and titles
105 elimana las otras imagenes generadas debido a que lso workers no llegan en orden y le pueden
106 poner otro tiempo a la figura q no necesariamente es el ultimo.
107 Solo se realiza cuando termina la imagen.
108 Problemas:
109
110 File "/home/ci-81/workspace/schainv2.3/schainpy/model/graphics/jroplot_data.py", line 145, in __plot
111 for n, eachfigure in enumerate(self.figurelist):
112 TypeError: 'NoneType' object is not iterable
113
114 '''
198 '''
115 def deleteanotherfiles(self):
116 figurenames=[]
117 if self.figurelist != None:
118 for n, eachfigure in enumerate(self.figurelist):
119 #add specific name for each channel in channelList
120 ghostfigname = os.path.join(self.save, '{}_{}_{}'.format(self.titles[n].replace(' ',''),self.CODE,
121 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d')))
122 figname = os.path.join(self.save, '{}_{}_{}.png'.format(self.titles[n].replace(' ',''),self.CODE,
123 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
124
199
125 for ghostfigure in glob.glob(ghostfigname+'*'): #ghostfigure will adopt all posible names of figures
200 if self.xmin is None:
126 if ghostfigure != figname:
201 xmin = self.min_time
127 os.remove(ghostfigure)
202 else:
128 print 'Removing GhostFigures:' , figname
203 if self.xaxis is 'time':
204 dt = datetime.datetime.fromtimestamp(self.min_time)
205 xmin = (datetime.datetime.combine(dt.date(),
206 datetime.time(int(self.xmin), 0, 0))-d1970).total_seconds()
207 else:
208 xmin = self.xmin
209
210 if self.xmax is None:
211 xmax = xmin+self.xrange*60*60
212 else:
213 if self.xaxis is 'time':
214 dt = datetime.datetime.fromtimestamp(self.min_time)
215 xmax = (datetime.datetime.combine(dt.date(),
216 datetime.time(int(self.xmax), 0, 0))-d1970).total_seconds()
129 else :
217 else:
130 '''Erasing ghost images for just on******************'''
218 xmax = self.xmax
131 ghostfigname = os.path.join(self.save, '{}_{}'.format(self.CODE,datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d')))
219
132 figname = os.path.join(self.save, '{}_{}.png'.format(self.CODE,datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
220 ymin = self.ymin if self.ymin else numpy.nanmin(self.y)
133 for ghostfigure in glob.glob(ghostfigname+'*'): #ghostfigure will adopt all posible names of figures
221 ymax = self.ymax if self.ymax else numpy.nanmax(self.y)
134 if ghostfigure != figname:
222
135 os.remove(ghostfigure)
223 ystep = 200 if ymax>= 800 else 100 if ymax>=400 else 50 if ymax>=200 else 20
136 print 'Removing GhostFigures:' , figname
224
225 for n, ax in enumerate(self.axes):
226 if ax.firsttime:
227 ax.set_facecolor(self.bgcolor)
228 ax.yaxis.set_major_locator(MultipleLocator(ystep))
229 if self.xaxis is 'time':
230 ax.xaxis.set_major_formatter(FuncFormatter(func))
231 ax.xaxis.set_major_locator(LinearLocator(9))
232 if self.xlabel is not None:
233 ax.set_xlabel(self.xlabel)
234 ax.set_ylabel(self.ylabel)
235 ax.firsttime = False
236 if self.showprofile:
237 self.pf_axes[n].set_ylim(ymin, ymax)
238 self.pf_axes[n].set_xlim(self.zmin, self.zmax)
239 self.pf_axes[n].set_xlabel('dB')
240 self.pf_axes[n].grid(b=True, axis='x')
241 [tick.set_visible(False) for tick in self.pf_axes[n].get_yticklabels()]
242 if self.colorbar:
243 cb = plt.colorbar(ax.plt, ax=ax, pad=0.02)
244 cb.ax.tick_params(labelsize=8)
245 if self.cb_label:
246 cb.set_label(self.cb_label, size=8)
247 elif self.cb_labels:
248 cb.set_label(self.cb_labels[n], size=8)
249
250 ax.set_title('{} - {} UTC'.format(
251 self.titles[n],
252 datetime.datetime.fromtimestamp(self.max_time).strftime('%H:%M:%S')),
253 size=8)
254 ax.set_xlim(xmin, xmax)
255 ax.set_ylim(ymin, ymax)
256
137
257
138 def __plot(self):
258 def __plot(self):
259 '''
260 '''
261 log.success('Plotting', self.name)
139
262
140 print 'plotting...{}'.format(self.CODE)
141 if self.ind_plt_ch is False : #standard
142 if self.show:
143 self.figure.show()
144 self.plot()
263 self.plot()
145 plt.tight_layout()
264 self.format()
146 self.figure.canvas.manager.set_window_title('{} {} - {}'.format(self.title, self.CODE.upper(),
265
147 datetime.datetime.fromtimestamp(self.max_time).strftime('%Y/%m/%d')))
266 for n, fig in enumerate(self.figures):
148 else :
267 if self.nrows == 0 or self.nplots == 0:
149 print 'len(self.figurelist): ',len(self.figurelist)
268 log.warning('No data', self.name)
150 for n, eachfigure in enumerate(self.figurelist):
269 continue
151 if self.show:
270 if self.show:
152 eachfigure.show()
271 fig.show()
153
272
154 self.plot()
273 fig.tight_layout()
155 eachfigure.tight_layout() # ajuste de cada subplot
274 fig.canvas.manager.set_window_title('{} - {}'.format(self.title,
156 eachfigure.canvas.manager.set_window_title('{} {} - {}'.format(self.title[n], self.CODE.upper(),
157 datetime.datetime.fromtimestamp(self.max_time).strftime('%Y/%m/%d')))
275 datetime.datetime.fromtimestamp(self.max_time).strftime('%Y/%m/%d')))
276 # fig.canvas.draw()
158
277
159 # if self.save:
278 if self.save and self.data.ended:
160 # if self.ind_plt_ch is False : #standard
279 channels = range(self.nrows)
161 # figname = os.path.join(self.save, '{}_{}.png'.format(self.CODE,
280 if self.oneFigure:
162 # datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
281 label = ''
163 # print 'Saving figure: {}'.format(figname)
164 # self.figure.savefig(figname)
165 # else :
166 # for n, eachfigure in enumerate(self.figurelist):
167 # #add specific name for each channel in channelList
168 # figname = os.path.join(self.save, '{}_{}_{}.png'.format(self.titles[n],self.CODE,
169 # datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
170 #
171 # print 'Saving figure: {}'.format(figname)
172 # eachfigure.savefig(figname)
173
174 if self.ind_plt_ch is False :
175 self.figure.canvas.draw()
176 else :
177 for eachfigure in self.figurelist:
178 eachfigure.canvas.draw()
179
180 if self.save:
181 if self.ind_plt_ch is False : #standard
182 figname = os.path.join(self.save, '{}_{}.png'.format(self.CODE,
183 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
184 print 'Saving figure: {}'.format(figname)
185 self.figure.savefig(figname)
186 else :
282 else:
187 for n, eachfigure in enumerate(self.figurelist):
283 label = '_{}'.format(channels[n])
188 #add specific name for each channel in channelList
284 figname = os.path.join(
189 figname = os.path.join(self.save, '{}_{}_{}.png'.format(self.titles[n].replace(' ',''),self.CODE,
285 self.save,
190 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
286 '{}{}_{}.png'.format(
191
287 self.CODE,
288 label,
289 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')
290 )
291 )
192 print 'Saving figure: {}'.format(figname)
292 print 'Saving figure: {}'.format(figname)
193 eachfigure.savefig(figname)
293 fig.savefig(figname)
194
195
294
196 def plot(self):
295 def plot(self):
197
296 '''
198 print 'plotting...{}'.format(self.CODE.upper())
297 '''
199 return
298 raise(NotImplementedError, 'Implement this method in child class')
200
299
201 def run(self):
300 def run(self):
202
301
203 print '[Starting] {}'.format(self.name)
302 log.success('Starting', self.name)
204
303
205 context = zmq.Context()
304 context = zmq.Context()
206 receiver = context.socket(zmq.SUB)
305 receiver = context.socket(zmq.SUB)
@@ -212,150 +311,102 class PlotData(Operation, Process):
212 else:
311 else:
213 receiver.connect("ipc:///tmp/zmq.plots")
312 receiver.connect("ipc:///tmp/zmq.plots")
214
313
215 seconds_passed = 0
216
217 while True:
314 while True:
218 try:
315 try:
219 self.data = receiver.recv_pyobj(flags=zmq.NOBLOCK)#flags=zmq.NOBLOCK
316 self.data = receiver.recv_pyobj(flags=zmq.NOBLOCK)
220 self.started = self.data['STARTED']
221 self.dataOut = self.data['dataOut']
222
223 if (len(self.times) < len(self.data['times']) and not self.started and self.data['ENDED']):
224 continue
225
317
226 self.times = self.data['times']
318 self.min_time = self.data.times[0]
227 self.times.sort()
319 self.max_time = self.data.times[-1]
228 self.throttle_value = self.data['throttle']
229 self.min_time = self.times[0]
230 self.max_time = self.times[-1]
231
320
232 if self.isConfig is False:
321 if self.isConfig is False:
233 print 'setting up'
322 self.__setup()
234 self.setup()
235 self.isConfig = True
323 self.isConfig = True
236 self.__plot()
237
324
238 if self.data['ENDED'] is True:
239 print '********GRAPHIC ENDED********'
240 self.ended = True
241 self.isConfig = False
242 self.__plot()
243 self.deleteanotherfiles() #CLPDG
244 elif seconds_passed >= self.data['throttle']:
245 print 'passed', seconds_passed
246 self.__plot()
325 self.__plot()
247 seconds_passed = 0
248
326
249 except zmq.Again as e:
327 except zmq.Again as e:
250 print 'Waiting for data...'
328 log.log('Waiting for data...')
251 plt.pause(2)
329 if self.data:
252 seconds_passed += 2
330 plt.pause(self.data.throttle)
331 else:
332 time.sleep(2)
253
333
254 def close(self):
334 def close(self):
255 if self.dataOut:
335 if self.data:
256 self.__plot()
336 self.__plot()
257
337
258
338
259 class PlotSpectraData(PlotData):
339 class PlotSpectraData(PlotData):
340 '''
341 Plot for Spectra data
342 '''
260
343
261 CODE = 'spc'
344 CODE = 'spc'
262 colormap = 'jro'
345 colormap = 'jro'
263 CONFLATE = False
264
346
265 def setup(self):
347 def setup(self):
266
348 self.nplots = len(self.data.channels)
267 ncolspan = 1
349 self.ncols = int(numpy.sqrt(self.nplots)+ 0.9)
268 colspan = 1
350 self.nrows = int((1.0*self.nplots/self.ncols) + 0.9)
269 self.ncols = int(numpy.sqrt(self.dataOut.nChannels)+0.9)
351 self.width = 3.4*self.ncols
270 self.nrows = int(self.dataOut.nChannels*1./self.ncols + 0.9)
352 self.height = 3*self.nrows
271 self.width = 3.6*self.ncols
353 self.cb_label = 'dB'
272 self.height = 3.2*self.nrows
273 if self.showprofile:
354 if self.showprofile:
274 ncolspan = 3
355 self.width += 0.8*self.ncols
275 colspan = 2
276 self.width += 1.2*self.ncols
277
356
278 self.ylabel = 'Range [Km]'
357 self.ylabel = 'Range [Km]'
279 self.titles = ['Channel {}'.format(x) for x in self.dataOut.channelList]
280
281 if self.figure is None:
282 self.figure = plt.figure(figsize=(self.width, self.height),
283 edgecolor='k',
284 facecolor='w')
285 else:
286 self.figure.clf()
287
288 n = 0
289 for y in range(self.nrows):
290 for x in range(self.ncols):
291 if n >= self.dataOut.nChannels:
292 break
293 ax = plt.subplot2grid((self.nrows, self.ncols*ncolspan), (y, x*ncolspan), 1, colspan)
294 if self.showprofile:
295 ax.ax_profile = plt.subplot2grid((self.nrows, self.ncols*ncolspan), (y, x*ncolspan+colspan), 1, 1)
296
297 ax.firsttime = True
298 self.axes.append(ax)
299 n += 1
300
358
301 def plot(self):
359 def plot(self):
302
303 if self.xaxis == "frequency":
360 if self.xaxis == "frequency":
304 x = self.dataOut.getFreqRange(1)/1000.
361 x = self.data.xrange[0]
305 xlabel = "Frequency (kHz)"
362 self.xlabel = "Frequency (kHz)"
306 elif self.xaxis == "time":
363 elif self.xaxis == "time":
307 x = self.dataOut.getAcfRange(1)
364 x = self.data.xrange[1]
308 xlabel = "Time (ms)"
365 self.xlabel = "Time (ms)"
309 else:
366 else:
310 x = self.dataOut.getVelRange(1)
367 x = self.data.xrange[2]
311 xlabel = "Velocity (m/s)"
368 self.xlabel = "Velocity (m/s)"
312
369
313 y = self.dataOut.getHeiRange()
370 if self.CODE == 'spc_mean':
314 z = self.data[self.CODE]
371 x = self.data.xrange[2]
372 self.xlabel = "Velocity (m/s)"
373
374 self.titles = []
375
376 y = self.data.heights
377 self.y = y
378 z = self.data['spc']
315
379
316 for n, ax in enumerate(self.axes):
380 for n, ax in enumerate(self.axes):
381 noise = self.data['noise'][n][-1]
382 if self.CODE == 'spc_mean':
383 mean = self.data['mean'][n][-1]
317 if ax.firsttime:
384 if ax.firsttime:
318 self.xmax = self.xmax if self.xmax else np.nanmax(x)
385 self.xmax = self.xmax if self.xmax else numpy.nanmax(x)
319 self.xmin = self.xmin if self.xmin else -self.xmax
386 self.xmin = self.xmin if self.xmin else -self.xmax
320 self.ymin = self.ymin if self.ymin else np.nanmin(y)
387 self.zmin = self.zmin if self.zmin else numpy.nanmin(z)
321 self.ymax = self.ymax if self.ymax else np.nanmax(y)
388 self.zmax = self.zmax if self.zmax else numpy.nanmax(z)
322 self.zmin = self.zmin if self.zmin else np.nanmin(z)
389 ax.plt = ax.pcolormesh(x, y, z[n].T,
323 self.zmax = self.zmax if self.zmax else np.nanmax(z)
324 ax.plot = ax.pcolormesh(x, y, z[n].T,
325 vmin=self.zmin,
390 vmin=self.zmin,
326 vmax=self.zmax,
391 vmax=self.zmax,
327 cmap=plt.get_cmap(self.colormap)
392 cmap=plt.get_cmap(self.colormap)
328 )
393 )
329 divider = make_axes_locatable(ax)
330 cax = divider.new_horizontal(size='3%', pad=0.05)
331 self.figure.add_axes(cax)
332 plt.colorbar(ax.plot, cax)
333
334 ax.set_xlim(self.xmin, self.xmax)
335 ax.set_ylim(self.ymin, self.ymax)
336
337 ax.set_ylabel(self.ylabel)
338 ax.set_xlabel(xlabel)
339
340 ax.firsttime = False
341
394
342 if self.showprofile:
395 if self.showprofile:
343 ax.plot_profile= ax.ax_profile.plot(self.data['rti'][self.max_time][n], y)[0]
396 ax.plt_profile= self.pf_axes[n].plot(self.data['rti'][n][-1], y)[0]
344 ax.ax_profile.set_xlim(self.zmin, self.zmax)
397 ax.plt_noise = self.pf_axes[n].plot(numpy.repeat(noise, len(y)), y,
345 ax.ax_profile.set_ylim(self.ymin, self.ymax)
398 color="k", linestyle="dashed", lw=1)[0]
346 ax.ax_profile.set_xlabel('dB')
399 if self.CODE == 'spc_mean':
347 ax.ax_profile.grid(b=True, axis='x')
400 ax.plt_mean = ax.plot(mean, y, color='k')[0]
348 ax.plot_noise = ax.ax_profile.plot(numpy.repeat(self.data['noise'][self.max_time][n], len(y)), y,
349 color="k", linestyle="dashed", lw=2)[0]
350 [tick.set_visible(False) for tick in ax.ax_profile.get_yticklabels()]
351 else:
401 else:
352 ax.plot.set_array(z[n].T.ravel())
402 ax.plt.set_array(z[n].T.ravel())
353 if self.showprofile:
403 if self.showprofile:
354 ax.plot_profile.set_data(self.data['rti'][self.max_time][n], y)
404 ax.plt_profile.set_data(self.data['rti'][n][-1], y)
355 ax.plot_noise.set_data(numpy.repeat(self.data['noise'][self.max_time][n], len(y)), y)
405 ax.plt_noise.set_data(numpy.repeat(noise, len(y)), y)
406 if self.CODE == 'spc_mean':
407 ax.plt_mean.set_data(mean, y)
356
408
357 ax.set_title('{} - Noise: {:.2f} dB'.format(self.titles[n], self.data['noise'][self.max_time][n]),
409 self.titles.append('CH {}: {:3.2f}dB'.format(n, noise))
358 size=8)
359 self.saveTime = self.max_time
410 self.saveTime = self.max_time
360
411
361
412
@@ -366,537 +417,244 class PlotCrossSpectraData(PlotData):
366 zmax_coh = None
417 zmax_coh = None
367 zmin_phase = None
418 zmin_phase = None
368 zmax_phase = None
419 zmax_phase = None
369 CONFLATE = False
370
420
371 def setup(self):
421 def setup(self):
372
422
373 ncolspan = 1
423 self.ncols = 4
374 colspan = 1
424 self.nrows = len(self.data.pairs)
375 self.ncols = 2
425 self.nplots = self.nrows*4
376 self.nrows = self.dataOut.nPairs
426 self.width = 3.4*self.ncols
377 self.width = 3.6*self.ncols
427 self.height = 3*self.nrows
378 self.height = 3.2*self.nrows
379
380 self.ylabel = 'Range [Km]'
428 self.ylabel = 'Range [Km]'
381 self.titles = ['Channel {}'.format(x) for x in self.dataOut.channelList]
429 self.showprofile = False
382
383 if self.figure is None:
384 self.figure = plt.figure(figsize=(self.width, self.height),
385 edgecolor='k',
386 facecolor='w')
387 else:
388 self.figure.clf()
389
390 for y in range(self.nrows):
391 for x in range(self.ncols):
392 ax = plt.subplot2grid((self.nrows, self.ncols), (y, x), 1, 1)
393 ax.firsttime = True
394 self.axes.append(ax)
395
430
396 def plot(self):
431 def plot(self):
397
432
398 if self.xaxis == "frequency":
433 if self.xaxis == "frequency":
399 x = self.dataOut.getFreqRange(1)/1000.
434 x = self.data.xrange[0]
400 xlabel = "Frequency (kHz)"
435 self.xlabel = "Frequency (kHz)"
401 elif self.xaxis == "time":
436 elif self.xaxis == "time":
402 x = self.dataOut.getAcfRange(1)
437 x = self.data.xrange[1]
403 xlabel = "Time (ms)"
438 self.xlabel = "Time (ms)"
404 else:
439 else:
405 x = self.dataOut.getVelRange(1)
440 x = self.data.xrange[2]
406 xlabel = "Velocity (m/s)"
441 self.xlabel = "Velocity (m/s)"
442
443 self.titles = []
407
444
408 y = self.dataOut.getHeiRange()
445 y = self.data.heights
409 z_coh = self.data['cspc_coh']
446 self.y = y
410 z_phase = self.data['cspc_phase']
447 spc = self.data['spc']
448 cspc = self.data['cspc']
411
449
412 for n in range(self.nrows):
450 for n in range(self.nrows):
413 ax = self.axes[2*n]
451 noise = self.data['noise'][n][-1]
414 ax1 = self.axes[2*n+1]
452 pair = self.data.pairs[n]
453 ax = self.axes[4*n]
454 ax3 = self.axes[4*n+3]
415 if ax.firsttime:
455 if ax.firsttime:
416 self.xmax = self.xmax if self.xmax else np.nanmax(x)
456 self.xmax = self.xmax if self.xmax else numpy.nanmax(x)
417 self.xmin = self.xmin if self.xmin else -self.xmax
457 self.xmin = self.xmin if self.xmin else -self.xmax
418 self.ymin = self.ymin if self.ymin else np.nanmin(y)
458 self.zmin = self.zmin if self.zmin else numpy.nanmin(spc)
419 self.ymax = self.ymax if self.ymax else np.nanmax(y)
459 self.zmax = self.zmax if self.zmax else numpy.nanmax(spc)
420 self.zmin_coh = self.zmin_coh if self.zmin_coh else 0.0
460 ax.plt = ax.pcolormesh(x, y, spc[pair[0]].T,
421 self.zmax_coh = self.zmax_coh if self.zmax_coh else 1.0
461 vmin=self.zmin,
422 self.zmin_phase = self.zmin_phase if self.zmin_phase else -180
462 vmax=self.zmax,
423 self.zmax_phase = self.zmax_phase if self.zmax_phase else 180
463 cmap=plt.get_cmap(self.colormap)
424
425 ax.plot = ax.pcolormesh(x, y, z_coh[n].T,
426 vmin=self.zmin_coh,
427 vmax=self.zmax_coh,
428 cmap=plt.get_cmap(self.colormap_coh)
429 )
430 divider = make_axes_locatable(ax)
431 cax = divider.new_horizontal(size='3%', pad=0.05)
432 self.figure.add_axes(cax)
433 plt.colorbar(ax.plot, cax)
434
435 ax.set_xlim(self.xmin, self.xmax)
436 ax.set_ylim(self.ymin, self.ymax)
437
438 ax.set_ylabel(self.ylabel)
439 ax.set_xlabel(xlabel)
440 ax.firsttime = False
441
442 ax1.plot = ax1.pcolormesh(x, y, z_phase[n].T,
443 vmin=self.zmin_phase,
444 vmax=self.zmax_phase,
445 cmap=plt.get_cmap(self.colormap_phase)
446 )
464 )
447 divider = make_axes_locatable(ax1)
448 cax = divider.new_horizontal(size='3%', pad=0.05)
449 self.figure.add_axes(cax)
450 plt.colorbar(ax1.plot, cax)
451
452 ax1.set_xlim(self.xmin, self.xmax)
453 ax1.set_ylim(self.ymin, self.ymax)
454
455 ax1.set_ylabel(self.ylabel)
456 ax1.set_xlabel(xlabel)
457 ax1.firsttime = False
458 else:
459 ax.plot.set_array(z_coh[n].T.ravel())
460 ax1.plot.set_array(z_phase[n].T.ravel())
461
462 ax.set_title('Coherence Ch{} * Ch{}'.format(self.dataOut.pairsList[n][0], self.dataOut.pairsList[n][1]), size=8)
463 ax1.set_title('Phase Ch{} * Ch{}'.format(self.dataOut.pairsList[n][0], self.dataOut.pairsList[n][1]), size=8)
464 self.saveTime = self.max_time
465
466
467 class PlotSpectraMeanData(PlotSpectraData):
468
469 CODE = 'spc_mean'
470 colormap = 'jet'
471
472 def plot(self):
473
474 if self.xaxis == "frequency":
475 x = self.dataOut.getFreqRange(1)/1000.
476 xlabel = "Frequency (kHz)"
477 elif self.xaxis == "time":
478 x = self.dataOut.getAcfRange(1)
479 xlabel = "Time (ms)"
480 else:
465 else:
481 x = self.dataOut.getVelRange(1)
466 ax.plt.set_array(spc[pair[0]].T.ravel())
482 xlabel = "Velocity (m/s)"
467 self.titles.append('CH {}: {:3.2f}dB'.format(n, noise))
483
484 y = self.dataOut.getHeiRange()
485 z = self.data['spc']
486 mean = self.data['mean'][self.max_time]
487
488 for n, ax in enumerate(self.axes):
489
468
469 ax = self.axes[4*n+1]
490 if ax.firsttime:
470 if ax.firsttime:
491 self.xmax = self.xmax if self.xmax else np.nanmax(x)
471 ax.plt = ax.pcolormesh(x, y, spc[pair[1]].T,
492 self.xmin = self.xmin if self.xmin else -self.xmax
493 self.ymin = self.ymin if self.ymin else np.nanmin(y)
494 self.ymax = self.ymax if self.ymax else np.nanmax(y)
495 self.zmin = self.zmin if self.zmin else np.nanmin(z)
496 self.zmax = self.zmax if self.zmax else np.nanmax(z)
497 ax.plt = ax.pcolormesh(x, y, z[n].T,
498 vmin=self.zmin,
472 vmin=self.zmin,
499 vmax=self.zmax,
473 vmax=self.zmax,
500 cmap=plt.get_cmap(self.colormap)
474 cmap=plt.get_cmap(self.colormap)
501 )
475 )
502 ax.plt_dop = ax.plot(mean[n], y,
476 else:
503 color='k')[0]
477 ax.plt.set_array(spc[pair[1]].T.ravel())
504
478 self.titles.append('CH {}: {:3.2f}dB'.format(n, noise))
505 divider = make_axes_locatable(ax)
506 cax = divider.new_horizontal(size='3%', pad=0.05)
507 self.figure.add_axes(cax)
508 plt.colorbar(ax.plt, cax)
509
510 ax.set_xlim(self.xmin, self.xmax)
511 ax.set_ylim(self.ymin, self.ymax)
512
479
513 ax.set_ylabel(self.ylabel)
480 out = cspc[n]/numpy.sqrt(spc[pair[0]]*spc[pair[1]])
514 ax.set_xlabel(xlabel)
481 coh = numpy.abs(out)
482 phase = numpy.arctan2(out.imag, out.real)*180/numpy.pi
515
483
516 ax.firsttime = False
484 ax = self.axes[4*n+2]
485 if ax.firsttime:
486 ax.plt = ax.pcolormesh(x, y, coh.T,
487 vmin=0,
488 vmax=1,
489 cmap=plt.get_cmap(self.colormap_coh)
490 )
491 else:
492 ax.plt.set_array(coh.T.ravel())
493 self.titles.append('Coherence Ch{} * Ch{}'.format(pair[0], pair[1]))
517
494
518 if self.showprofile:
495 ax = self.axes[4*n+3]
519 ax.plt_profile= ax.ax_profile.plot(self.data['rti'][self.max_time][n], y)[0]
496 if ax.firsttime:
520 ax.ax_profile.set_xlim(self.zmin, self.zmax)
497 ax.plt = ax.pcolormesh(x, y, phase.T,
521 ax.ax_profile.set_ylim(self.ymin, self.ymax)
498 vmin=-180,
522 ax.ax_profile.set_xlabel('dB')
499 vmax=180,
523 ax.ax_profile.grid(b=True, axis='x')
500 cmap=plt.get_cmap(self.colormap_phase)
524 ax.plt_noise = ax.ax_profile.plot(numpy.repeat(self.data['noise'][self.max_time][n], len(y)), y,
501 )
525 color="k", linestyle="dashed", lw=2)[0]
526 [tick.set_visible(False) for tick in ax.ax_profile.get_yticklabels()]
527 else:
502 else:
528 ax.plt.set_array(z[n].T.ravel())
503 ax.plt.set_array(phase.T.ravel())
529 ax.plt_dop.set_data(mean[n], y)
504 self.titles.append('Phase CH{} * CH{}'.format(pair[0], pair[1]))
530 if self.showprofile:
531 ax.plt_profile.set_data(self.data['rti'][self.max_time][n], y)
532 ax.plt_noise.set_data(numpy.repeat(self.data['noise'][self.max_time][n], len(y)), y)
533
505
534 ax.set_title('{} - Noise: {:.2f} dB'.format(self.titles[n], self.data['noise'][self.max_time][n]),
535 size=8)
536 self.saveTime = self.max_time
506 self.saveTime = self.max_time
537
507
538
508
509 class PlotSpectraMeanData(PlotSpectraData):
510 '''
511 Plot for Spectra and Mean
512 '''
513 CODE = 'spc_mean'
514 colormap = 'jro'
515
516
539 class PlotRTIData(PlotData):
517 class PlotRTIData(PlotData):
518 '''
519 Plot for RTI data
520 '''
540
521
541 CODE = 'rti'
522 CODE = 'rti'
542 colormap = 'jro'
523 colormap = 'jro'
543
524
544 def setup(self):
525 def setup(self):
526 self.xaxis = 'time'
545 self.ncols = 1
527 self.ncols = 1
546 self.nrows = self.dataOut.nChannels
528 self.nrows = len(self.data.channels)
547 self.width = 10
529 self.nplots = len(self.data.channels)
548 #TODO : arreglar la altura de la figura, esta hardcodeada.
549 #Se arreglo, testear!
550 if self.ind_plt_ch:
551 self.height = 3.2#*self.nrows if self.nrows<6 else 12
552 else:
553 self.height = 2.2*self.nrows if self.nrows<6 else 12
554
555 '''
556 if self.nrows==1:
557 self.height += 1
558 '''
559 self.ylabel = 'Range [Km]'
530 self.ylabel = 'Range [Km]'
560 self.titles = ['Channel {}'.format(x) for x in self.dataOut.channelList]
531 self.cb_label = 'dB'
561
532 self.titles = ['{} Channel {}'.format(self.CODE.upper(), x) for x in range(self.nrows)]
562 '''
563 Logica:
564 1) Si la variable ind_plt_ch es True, va a crear mas de 1 figura
565 2) guardamos "Figures" en una lista y "axes" en otra, quizas se deberia guardar el
566 axis dentro de "Figures" como un diccionario.
567 '''
568 if self.ind_plt_ch is False: #standard mode
569
570 if self.figure is None: #solo para la priemra vez
571 self.figure = plt.figure(figsize=(self.width, self.height),
572 edgecolor='k',
573 facecolor='w')
574 else:
575 self.figure.clf()
576 self.axes = []
577
578
579 for n in range(self.nrows):
580 ax = self.figure.add_subplot(self.nrows, self.ncols, n+1)
581 #ax = self.figure(n+1)
582 ax.firsttime = True
583 self.axes.append(ax)
584
585 else : #append one figure foreach channel in channelList
586 if self.figurelist == None:
587 self.figurelist = []
588 for n in range(self.nrows):
589 self.figure = plt.figure(figsize=(self.width, self.height),
590 edgecolor='k',
591 facecolor='w')
592 #add always one subplot
593 self.figurelist.append(self.figure)
594
595 else : # cada dia nuevo limpia el axes, pero mantiene el figure
596 for eachfigure in self.figurelist:
597 eachfigure.clf() # eliminaria todas las figuras de la lista?
598 self.axes = []
599
600 for eachfigure in self.figurelist:
601 ax = eachfigure.add_subplot(1,1,1) #solo 1 axis por figura
602 #ax = self.figure(n+1)
603 ax.firsttime = True
604 #Cada figura tiene un distinto puntero
605 self.axes.append(ax)
606 #plt.close(eachfigure)
607
608
533
609 def plot(self):
534 def plot(self):
535 self.x = self.data.times
536 self.y = self.data.heights
537 self.z = self.data[self.CODE]
538 self.z = numpy.ma.masked_invalid(self.z)
610
539
611 if self.ind_plt_ch is False: #standard mode
612 self.x = np.array(self.times)
613 self.y = self.dataOut.getHeiRange()
614 self.z = []
615
616 for ch in range(self.nrows):
617 self.z.append([self.data[self.CODE][t][ch] for t in self.times])
618
619 self.z = np.array(self.z)
620 for n, ax in enumerate(self.axes):
540 for n, ax in enumerate(self.axes):
621 x, y, z = self.fill_gaps(*self.decimate())
541 x, y, z = self.fill_gaps(*self.decimate())
622 xmin = self.min_time
542 self.zmin = self.zmin if self.zmin else numpy.min(self.z)
623 xmax = xmin+self.xrange*60*60
543 self.zmax = self.zmax if self.zmax else numpy.max(self.z)
624 self.zmin = self.zmin if self.zmin else np.min(self.z)
625 self.zmax = self.zmax if self.zmax else np.max(self.z)
626 if ax.firsttime:
544 if ax.firsttime:
627 self.ymin = self.ymin if self.ymin else np.nanmin(self.y)
545 ax.plt = ax.pcolormesh(x, y, z[n].T,
628 self.ymax = self.ymax if self.ymax else np.nanmax(self.y)
629 plot = ax.pcolormesh(x, y, z[n].T,
630 vmin=self.zmin,
546 vmin=self.zmin,
631 vmax=self.zmax,
547 vmax=self.zmax,
632 cmap=plt.get_cmap(self.colormap)
548 cmap=plt.get_cmap(self.colormap)
633 )
549 )
634 divider = make_axes_locatable(ax)
550 if self.showprofile:
635 cax = divider.new_horizontal(size='2%', pad=0.05)
551 ax.plot_profile= self.pf_axes[n].plot(self.data['rti'][n][-1], self.y)[0]
636 self.figure.add_axes(cax)
552 ax.plot_noise = self.pf_axes[n].plot(numpy.repeat(self.data['noise'][n][-1], len(self.y)), self.y,
637 plt.colorbar(plot, cax)
553 color="k", linestyle="dashed", lw=1)[0]
638 ax.set_ylim(self.ymin, self.ymax)
639 ax.xaxis.set_major_formatter(FuncFormatter(func))
640 ax.xaxis.set_major_locator(LinearLocator(6))
641 ax.set_ylabel(self.ylabel)
642 if self.xmin is None:
643 xmin = self.min_time
644 else:
645 xmin = (datetime.datetime.combine(self.dataOut.datatime.date(),
646 datetime.time(self.xmin, 0, 0))-d1970).total_seconds()
647 ax.set_xlim(xmin, xmax)
648 ax.firsttime = False
649 else:
554 else:
650 ax.collections.remove(ax.collections[0])
555 ax.collections.remove(ax.collections[0])
651 ax.set_xlim(xmin, xmax)
556 ax.plt = ax.pcolormesh(x, y, z[n].T,
652 plot = ax.pcolormesh(x, y, z[n].T,
653 vmin=self.zmin,
654 vmax=self.zmax,
655 cmap=plt.get_cmap(self.colormap)
656 )
657 ax.set_title('{} {}'.format(self.titles[n],
658 datetime.datetime.fromtimestamp(self.max_time).strftime('%y/%m/%d %H:%M:%S')),
659 size=8)
660
661 self.saveTime = self.min_time
662 else :
663 self.x = np.array(self.times)
664 self.y = self.dataOut.getHeiRange()
665 self.z = []
666
667 for ch in range(self.nrows):
668 self.z.append([self.data[self.CODE][t][ch] for t in self.times])
669
670 self.z = np.array(self.z)
671 for n, eachfigure in enumerate(self.figurelist): #estaba ax in axes
672
673 x, y, z = self.fill_gaps(*self.decimate())
674 xmin = self.min_time
675 xmax = xmin+self.xrange*60*60
676 self.zmin = self.zmin if self.zmin else np.min(self.z)
677 self.zmax = self.zmax if self.zmax else np.max(self.z)
678 if self.axes[n].firsttime:
679 self.ymin = self.ymin if self.ymin else np.nanmin(self.y)
680 self.ymax = self.ymax if self.ymax else np.nanmax(self.y)
681 plot = self.axes[n].pcolormesh(x, y, z[n].T,
682 vmin=self.zmin,
683 vmax=self.zmax,
684 cmap=plt.get_cmap(self.colormap)
685 )
686 divider = make_axes_locatable(self.axes[n])
687 cax = divider.new_horizontal(size='2%', pad=0.05)
688 eachfigure.add_axes(cax)
689 #self.figure2.add_axes(cax)
690 plt.colorbar(plot, cax)
691 self.axes[n].set_ylim(self.ymin, self.ymax)
692
693 self.axes[n].xaxis.set_major_formatter(FuncFormatter(func))
694 self.axes[n].xaxis.set_major_locator(LinearLocator(6))
695
696 self.axes[n].set_ylabel(self.ylabel)
697
698 if self.xmin is None:
699 xmin = self.min_time
700 else:
701 xmin = (datetime.datetime.combine(self.dataOut.datatime.date(),
702 datetime.time(self.xmin, 0, 0))-d1970).total_seconds()
703
704 self.axes[n].set_xlim(xmin, xmax)
705 self.axes[n].firsttime = False
706 else:
707 self.axes[n].collections.remove(self.axes[n].collections[0])
708 self.axes[n].set_xlim(xmin, xmax)
709 plot = self.axes[n].pcolormesh(x, y, z[n].T,
710 vmin=self.zmin,
557 vmin=self.zmin,
711 vmax=self.zmax,
558 vmax=self.zmax,
712 cmap=plt.get_cmap(self.colormap)
559 cmap=plt.get_cmap(self.colormap)
713 )
560 )
714 self.axes[n].set_title('{} {}'.format(self.titles[n],
561 if self.showprofile:
715 datetime.datetime.fromtimestamp(self.max_time).strftime('%y/%m/%d %H:%M:%S')),
562 ax.plot_profile.set_data(self.data['rti'][n][-1], self.y)
716 size=8)
563 ax.plot_noise.set_data(numpy.repeat(self.data['noise'][n][-1], len(self.y)), self.y)
717
564
718 self.saveTime = self.min_time
565 self.saveTime = self.min_time
719
566
720
567
721 class PlotCOHData(PlotRTIData):
568 class PlotCOHData(PlotRTIData):
569 '''
570 Plot for Coherence data
571 '''
722
572
723 CODE = 'coh'
573 CODE = 'coh'
724
574
725 def setup(self):
575 def setup(self):
726
576 self.xaxis = 'time'
727 self.ncols = 1
577 self.ncols = 1
728 self.nrows = self.dataOut.nPairs
578 self.nrows = len(self.data.pairs)
729 self.width = 10
579 self.nplots = len(self.data.pairs)
730 self.height = 2.2*self.nrows if self.nrows<6 else 12
731 self.ind_plt_ch = False #just for coherence and phase
732 if self.nrows==1:
733 self.height += 1
734 self.ylabel = 'Range [Km]'
580 self.ylabel = 'Range [Km]'
735 self.titles = ['{} Ch{} * Ch{}'.format(self.CODE.upper(), x[0], x[1]) for x in self.dataOut.pairsList]
581 if self.CODE == 'coh':
736
582 self.cb_label = ''
737 if self.figure is None:
583 self.titles = ['Coherence Map Ch{} * Ch{}'.format(x[0], x[1]) for x in self.data.pairs]
738 self.figure = plt.figure(figsize=(self.width, self.height),
739 edgecolor='k',
740 facecolor='w')
741 else:
584 else:
742 self.figure.clf()
585 self.cb_label = 'Degrees'
743 self.axes = []
586 self.titles = ['Phase Map Ch{} * Ch{}'.format(x[0], x[1]) for x in self.data.pairs]
744
587
745 for n in range(self.nrows):
588
746 ax = self.figure.add_subplot(self.nrows, self.ncols, n+1)
589 class PlotPHASEData(PlotCOHData):
747 ax.firsttime = True
590 '''
748 self.axes.append(ax)
591 Plot for Phase map data
592 '''
593
594 CODE = 'phase'
595 colormap = 'seismic'
749
596
750
597
751 class PlotNoiseData(PlotData):
598 class PlotNoiseData(PlotData):
599 '''
600 Plot for noise
601 '''
602
752 CODE = 'noise'
603 CODE = 'noise'
753
604
754 def setup(self):
605 def setup(self):
755
606 self.xaxis = 'time'
756 self.ncols = 1
607 self.ncols = 1
757 self.nrows = 1
608 self.nrows = 1
758 self.width = 10
609 self.nplots = 1
759 self.height = 3.2
760 self.ylabel = 'Intensity [dB]'
610 self.ylabel = 'Intensity [dB]'
761 self.titles = ['Noise']
611 self.titles = ['Noise']
762
612 self.colorbar = False
763 if self.figure is None:
764 self.figure = plt.figure(figsize=(self.width, self.height),
765 edgecolor='k',
766 facecolor='w')
767 else:
768 self.figure.clf()
769 self.axes = []
770
771 self.ax = self.figure.add_subplot(self.nrows, self.ncols, 1)
772 self.ax.firsttime = True
773
774 def plot(self):
775
776 x = self.times
777 xmin = self.min_time
778 xmax = xmin+self.xrange*60*60
779 if self.ax.firsttime:
780 for ch in self.dataOut.channelList:
781 y = [self.data[self.CODE][t][ch] for t in self.times]
782 self.ax.plot(x, y, lw=1, label='Ch{}'.format(ch))
783 self.ax.firsttime = False
784 self.ax.xaxis.set_major_formatter(FuncFormatter(func))
785 self.ax.xaxis.set_major_locator(LinearLocator(6))
786 self.ax.set_ylabel(self.ylabel)
787 plt.legend()
788 else:
789 for ch in self.dataOut.channelList:
790 y = [self.data[self.CODE][t][ch] for t in self.times]
791 self.ax.lines[ch].set_data(x, y)
792
793 self.ax.set_xlim(xmin, xmax)
794 self.ax.set_ylim(min(y)-5, max(y)+5)
795 self.saveTime = self.min_time
796
797
798 class PlotWindProfilerData(PlotRTIData):
799
800 CODE = 'wind'
801 colormap = 'seismic'
802
803 def setup(self):
804 self.ncols = 1
805 self.nrows = self.dataOut.data_output.shape[0]
806 self.width = 10
807 self.height = 2.2*self.nrows
808 self.ylabel = 'Height [Km]'
809 self.titles = ['Zonal Wind' ,'Meridional Wind', 'Vertical Wind']
810 self.clabels = ['Velocity (m/s)','Velocity (m/s)','Velocity (cm/s)']
811 self.windFactor = [1, 1, 100]
812
813 if self.figure is None:
814 self.figure = plt.figure(figsize=(self.width, self.height),
815 edgecolor='k',
816 facecolor='w')
817 else:
818 self.figure.clf()
819 self.axes = []
820
821 for n in range(self.nrows):
822 ax = self.figure.add_subplot(self.nrows, self.ncols, n+1)
823 ax.firsttime = True
824 self.axes.append(ax)
825
613
826 def plot(self):
614 def plot(self):
827
615
828 self.x = np.array(self.times)
616 x = self.data.times
829 self.y = self.dataOut.heightList
830 self.z = []
831
832 for ch in range(self.nrows):
833 self.z.append([self.data['output'][t][ch] for t in self.times])
834
835 self.z = np.array(self.z)
836 self.z = numpy.ma.masked_invalid(self.z)
837
838 cmap=plt.get_cmap(self.colormap)
839 cmap.set_bad('black', 1.)
840
841 for n, ax in enumerate(self.axes):
842 x, y, z = self.fill_gaps(*self.decimate())
843 xmin = self.min_time
617 xmin = self.min_time
844 xmax = xmin+self.xrange*60*60
618 xmax = xmin+self.xrange*60*60
845 if ax.firsttime:
619 Y = self.data[self.CODE]
846 self.ymin = self.ymin if self.ymin else np.nanmin(self.y)
847 self.ymax = self.ymax if self.ymax else np.nanmax(self.y)
848 self.zmax = self.zmax if self.zmax else numpy.nanmax(abs(self.z[:-1, :]))
849 self.zmin = self.zmin if self.zmin else -self.zmax
850
851 plot = ax.pcolormesh(x, y, z[n].T*self.windFactor[n],
852 vmin=self.zmin,
853 vmax=self.zmax,
854 cmap=cmap
855 )
856 divider = make_axes_locatable(ax)
857 cax = divider.new_horizontal(size='2%', pad=0.05)
858 self.figure.add_axes(cax)
859 cb = plt.colorbar(plot, cax)
860 cb.set_label(self.clabels[n])
861 ax.set_ylim(self.ymin, self.ymax)
862
620
863 ax.xaxis.set_major_formatter(FuncFormatter(func))
621 if self.axes[0].firsttime:
864 ax.xaxis.set_major_locator(LinearLocator(6))
622 for ch in self.data.channels:
865
623 y = Y[ch]
866 ax.set_ylabel(self.ylabel)
624 self.axes[0].plot(x, y, lw=1, label='Ch{}'.format(ch))
867
625 plt.legend()
868 ax.set_xlim(xmin, xmax)
869 ax.firsttime = False
870 else:
626 else:
871 ax.collections.remove(ax.collections[0])
627 for ch in self.data.channels:
872 ax.set_xlim(xmin, xmax)
628 y = Y[ch]
873 plot = ax.pcolormesh(x, y, z[n].T*self.windFactor[n],
629 self.axes[0].lines[ch].set_data(x, y)
874 vmin=self.zmin,
875 vmax=self.zmax,
876 cmap=plt.get_cmap(self.colormap)
877 )
878 ax.set_title('{} {}'.format(self.titles[n],
879 datetime.datetime.fromtimestamp(self.max_time).strftime('%y/%m/%d %H:%M:%S')),
880 size=8)
881
630
631 self.ymin = numpy.nanmin(Y) - 5
632 self.ymax = numpy.nanmax(Y) + 5
882 self.saveTime = self.min_time
633 self.saveTime = self.min_time
883
634
884
635
885 class PlotSNRData(PlotRTIData):
636 class PlotSNRData(PlotRTIData):
637 '''
638 Plot for SNR Data
639 '''
640
886 CODE = 'snr'
641 CODE = 'snr'
887 colormap = 'jet'
642 colormap = 'jet'
888
643
644
889 class PlotDOPData(PlotRTIData):
645 class PlotDOPData(PlotRTIData):
646 '''
647 Plot for DOPPLER Data
648 '''
649
890 CODE = 'dop'
650 CODE = 'dop'
891 colormap = 'jet'
651 colormap = 'jet'
892
652
893
653
894 class PlotPHASEData(PlotCOHData):
895 CODE = 'phase'
896 colormap = 'seismic'
897
898
899 class PlotSkyMapData(PlotData):
654 class PlotSkyMapData(PlotData):
655 '''
656 Plot for meteors detection data
657 '''
900
658
901 CODE = 'met'
659 CODE = 'met'
902
660
@@ -923,7 +681,7 class PlotSkyMapData(PlotData):
923
681
924 def plot(self):
682 def plot(self):
925
683
926 arrayParameters = np.concatenate([self.data['param'][t] for t in self.times])
684 arrayParameters = numpy.concatenate([self.data['param'][t] for t in self.data.times])
927 error = arrayParameters[:,-1]
685 error = arrayParameters[:,-1]
928 indValid = numpy.where(error == 0)[0]
686 indValid = numpy.where(error == 0)[0]
929 finalMeteor = arrayParameters[indValid,:]
687 finalMeteor = arrayParameters[indValid,:]
@@ -953,3 +711,72 class PlotSkyMapData(PlotData):
953 self.ax.set_title(title, size=8)
711 self.ax.set_title(title, size=8)
954
712
955 self.saveTime = self.max_time
713 self.saveTime = self.max_time
714
715 class PlotParamData(PlotRTIData):
716 '''
717 Plot for data_param object
718 '''
719
720 CODE = 'param'
721 colormap = 'seismic'
722
723 def setup(self):
724 self.xaxis = 'time'
725 self.ncols = 1
726 self.nrows = self.data.shape(self.CODE)[0]
727 self.nplots = self.nrows
728 if self.showSNR:
729 self.nrows += 1
730
731 self.ylabel = 'Height [Km]'
732 self.titles = self.data.parameters \
733 if self.data.parameters else ['Param {}'.format(x) for x in xrange(self.nrows)]
734 if self.showSNR:
735 self.titles.append('SNR')
736
737 def plot(self):
738 self.data.normalize_heights()
739 self.x = self.data.times
740 self.y = self.data.heights
741 if self.showSNR:
742 self.z = numpy.concatenate(
743 (self.data[self.CODE], self.data['snr'])
744 )
745 else:
746 self.z = self.data[self.CODE]
747
748 self.z = numpy.ma.masked_invalid(self.z)
749
750 for n, ax in enumerate(self.axes):
751
752 x, y, z = self.fill_gaps(*self.decimate())
753
754 if ax.firsttime:
755 if self.zlimits is not None:
756 self.zmin, self.zmax = self.zlimits[n]
757 self.zmax = self.zmax if self.zmax is not None else numpy.nanmax(abs(self.z[:-1, :]))
758 self.zmin = self.zmin if self.zmin is not None else -self.zmax
759 ax.plt = ax.pcolormesh(x, y, z[n, :, :].T*self.factors[n],
760 vmin=self.zmin,
761 vmax=self.zmax,
762 cmap=self.cmaps[n]
763 )
764 else:
765 if self.zlimits is not None:
766 self.zmin, self.zmax = self.zlimits[n]
767 ax.collections.remove(ax.collections[0])
768 ax.plt = ax.pcolormesh(x, y, z[n, :, :].T*self.factors[n],
769 vmin=self.zmin,
770 vmax=self.zmax,
771 cmap=self.cmaps[n]
772 )
773
774 self.saveTime = self.min_time
775
776 class PlotOuputData(PlotParamData):
777 '''
778 Plot data_output object
779 '''
780
781 CODE = 'output'
782 colormap = 'seismic' No newline at end of file
Show file before | Show file after | Hide comments
@@ -225,7 +225,6 class MomentsPlot(Figure):
225 WIDTHPROF = None
225 WIDTHPROF = None
226 HEIGHTPROF = None
226 HEIGHTPROF = None
227 PREFIX = 'prm'
227 PREFIX = 'prm'
228
229 def __init__(self, **kwargs):
228 def __init__(self, **kwargs):
230 Figure.__init__(self, **kwargs)
229 Figure.__init__(self, **kwargs)
231 self.isConfig = False
230 self.isConfig = False
@@ -1626,6 +1625,7 class PhasePlot(Figure):
1626
1625
1627 PREFIX = 'mphase'
1626 PREFIX = 'mphase'
1628
1627
1628
1629 def __init__(self, **kwargs):
1629 def __init__(self, **kwargs):
1630 Figure.__init__(self, **kwargs)
1630 Figure.__init__(self, **kwargs)
1631 self.timerange = 24*60*60
1631 self.timerange = 24*60*60
Show file before | Show file after | Hide comments
@@ -10,6 +10,7 import numpy
10 from figure import Figure, isRealtime, isTimeInHourRange
10 from figure import Figure, isRealtime, isTimeInHourRange
11 from plotting_codes import *
11 from plotting_codes import *
12
12
13
13 class SpectraPlot(Figure):
14 class SpectraPlot(Figure):
14
15
15 isConfig = None
16 isConfig = None
Show file before | Show file after | Hide comments
1 NO CONTENT: modified file
NO CONTENT: modified file
Show file before | Show file after | Hide comments
@@ -267,7 +267,7 class AMISRReader(ProcessingUnit):
267 self.dirnameList = new_dirnameList
267 self.dirnameList = new_dirnameList
268 return 1
268 return 1
269
269
270 def __searchFilesOnline(self,
270 def searchFilesOnLine(self,
271 path,
271 path,
272 walk=True):
272 walk=True):
273
273
@@ -287,7 +287,7 class AMISRReader(ProcessingUnit):
287 return
287 return
288
288
289
289
290 def __searchFilesOffline(self,
290 def searchFilesOffLine(self,
291 path,
291 path,
292 startDate,
292 startDate,
293 endDate,
293 endDate,
@@ -494,9 +494,9 class AMISRReader(ProcessingUnit):
494 self.online = online
494 self.online = online
495 if not(online):
495 if not(online):
496 #Busqueda de archivos offline
496 #Busqueda de archivos offline
497 self.__searchFilesOffline(path, startDate, endDate, startTime, endTime, walk)
497 self.searchFilesOffLine(path, startDate, endDate, startTime, endTime, walk)
498 else:
498 else:
499 self.__searchFilesOnline(path, walk)
499 self.searchFilesOnLine(path, walk)
500
500
501 if not(self.filenameList):
501 if not(self.filenameList):
502 print "There is no files into the folder: %s"%(path)
502 print "There is no files into the folder: %s"%(path)
Show file before | Show file after | Hide comments
@@ -542,7 +542,6 class JRODataIO:
542
542
543 class JRODataReader(JRODataIO):
543 class JRODataReader(JRODataIO):
544
544
545
546 online = 0
545 online = 0
547
546
548 realtime = 0
547 realtime = 0
@@ -579,7 +578,6 class JRODataReader(JRODataIO):
579
578
580 selBlocktime = None
579 selBlocktime = None
581
580
582
583 def __init__(self):
581 def __init__(self):
584
582
585 """
583 """
@@ -603,7 +601,7 class JRODataReader(JRODataIO):
603
601
604 raise NotImplementedError
602 raise NotImplementedError
605
603
606 def __searchFilesOffLine(self,
604 def searchFilesOffLine(self,
607 path,
605 path,
608 startDate=None,
606 startDate=None,
609 endDate=None,
607 endDate=None,
@@ -612,7 +610,6 class JRODataReader(JRODataIO):
612 set=None,
610 set=None,
613 expLabel='',
611 expLabel='',
614 ext='.r',
612 ext='.r',
615 queue=None,
616 cursor=None,
613 cursor=None,
617 skip=None,
614 skip=None,
618 walk=True):
615 walk=True):
@@ -625,8 +622,7 class JRODataReader(JRODataIO):
625 dateList, pathList = self.findDatafiles(path, startDate, endDate, expLabel, ext, walk, include_path=True)
622 dateList, pathList = self.findDatafiles(path, startDate, endDate, expLabel, ext, walk, include_path=True)
626
623
627 if dateList == []:
624 if dateList == []:
628 # print "[Reading] Date range selected invalid [%s - %s]: No *%s files in %s)" %(startDate, endDate, ext, path)
625 return [], []
629 return None, None
630
626
631 if len(dateList) > 1:
627 if len(dateList) > 1:
632 print "[Reading] Data found for date range [%s - %s]: total days = %d" %(startDate, endDate, len(dateList))
628 print "[Reading] Data found for date range [%s - %s]: total days = %d" %(startDate, endDate, len(dateList))
@@ -637,7 +633,6 class JRODataReader(JRODataIO):
637 datetimeList = []
633 datetimeList = []
638
634
639 for thisPath in pathList:
635 for thisPath in pathList:
640 # thisPath = pathList[pathDict[file]]
641
636
642 fileList = glob.glob1(thisPath, "*%s" %ext)
637 fileList = glob.glob1(thisPath, "*%s" %ext)
643 fileList.sort()
638 fileList.sort()
@@ -645,10 +640,8 class JRODataReader(JRODataIO):
645 skippedFileList = []
640 skippedFileList = []
646
641
647 if cursor is not None and skip is not None:
642 if cursor is not None and skip is not None:
648 # if cursor*skip > len(fileList):
643
649 if skip == 0:
644 if skip == 0:
650 if queue is not None:
651 queue.put(len(fileList))
652 skippedFileList = []
645 skippedFileList = []
653 else:
646 else:
654 skippedFileList = fileList[cursor*skip: cursor*skip + skip]
647 skippedFileList = fileList[cursor*skip: cursor*skip + skip]
@@ -673,13 +666,13 class JRODataReader(JRODataIO):
673
666
674 if not(filenameList):
667 if not(filenameList):
675 print "[Reading] Time range selected invalid [%s - %s]: No *%s files in %s)" %(startTime, endTime, ext, path)
668 print "[Reading] Time range selected invalid [%s - %s]: No *%s files in %s)" %(startTime, endTime, ext, path)
676 return None, None
669 return [], []
677
670
678 print "[Reading] %d file(s) was(were) found in time range: %s - %s" %(len(filenameList), startTime, endTime)
671 print "[Reading] %d file(s) was(were) found in time range: %s - %s" %(len(filenameList), startTime, endTime)
679 print
672 print
680
673
681 for i in range(len(filenameList)):
674 # for i in range(len(filenameList)):
682 print "[Reading] %s -> [%s]" %(filenameList[i], datetimeList[i].ctime())
675 # print "[Reading] %s -> [%s]" %(filenameList[i], datetimeList[i].ctime())
683
676
684 self.filenameList = filenameList
677 self.filenameList = filenameList
685 self.datetimeList = datetimeList
678 self.datetimeList = datetimeList
@@ -1044,7 +1037,6 class JRODataReader(JRODataIO):
1044 #Skip block out of startTime and endTime
1037 #Skip block out of startTime and endTime
1045 while True:
1038 while True:
1046 if not(self.__setNewBlock()):
1039 if not(self.__setNewBlock()):
1047 print 'returning'
1048 return 0
1040 return 0
1049
1041
1050 if not(self.readBlock()):
1042 if not(self.readBlock()):
@@ -1273,7 +1265,6 class JRODataReader(JRODataIO):
1273 realtime=False,
1265 realtime=False,
1274 blocksize=None,
1266 blocksize=None,
1275 blocktime=None,
1267 blocktime=None,
1276 queue=None,
1277 skip=None,
1268 skip=None,
1278 cursor=None,
1269 cursor=None,
1279 warnings=True,
1270 warnings=True,
@@ -1322,19 +1313,13 class JRODataReader(JRODataIO):
1322 last_set = None
1313 last_set = None
1323 else:
1314 else:
1324 print "[Reading] Searching files in offline mode ..."
1315 print "[Reading] Searching files in offline mode ..."
1325 pathList, filenameList = self.__searchFilesOffLine(path, startDate=startDate, endDate=endDate,
1316 pathList, filenameList = self.searchFilesOffLine(path, startDate=startDate, endDate=endDate,
1326 startTime=startTime, endTime=endTime,
1317 startTime=startTime, endTime=endTime,
1327 set=set, expLabel=expLabel, ext=ext,
1318 set=set, expLabel=expLabel, ext=ext,
1328 walk=walk, cursor=cursor,
1319 walk=walk, cursor=cursor,
1329 skip=skip, queue=queue)
1320 skip=skip)
1330
1321
1331 if not(pathList):
1322 if not(pathList):
1332 # print "[Reading] No *%s files in %s (%s - %s)"%(ext, path,
1333 # datetime.datetime.combine(startDate,startTime).ctime(),
1334 # datetime.datetime.combine(endDate,endTime).ctime())
1335
1336 # sys.exit(-1)
1337
1338 self.fileIndex = -1
1323 self.fileIndex = -1
1339 self.pathList = []
1324 self.pathList = []
1340 self.filenameList = []
1325 self.filenameList = []
@@ -1451,7 +1436,6 class JRODataReader(JRODataIO):
1451
1436
1452 self.__printInfo = False
1437 self.__printInfo = False
1453
1438
1454
1455 def run(self,
1439 def run(self,
1456 path=None,
1440 path=None,
1457 startDate=None,
1441 startDate=None,
@@ -1469,7 +1453,6 class JRODataReader(JRODataIO):
1469 realtime=False,
1453 realtime=False,
1470 blocksize=None,
1454 blocksize=None,
1471 blocktime=None,
1455 blocktime=None,
1472 queue=None,
1473 skip=None,
1456 skip=None,
1474 cursor=None,
1457 cursor=None,
1475 warnings=True,
1458 warnings=True,
@@ -1477,7 +1460,6 class JRODataReader(JRODataIO):
1477 verbose=True, **kwargs):
1460 verbose=True, **kwargs):
1478
1461
1479 if not(self.isConfig):
1462 if not(self.isConfig):
1480 # self.dataOut = dataOut
1481 self.setup( path=path,
1463 self.setup(path=path,
1482 startDate=startDate,
1464 startDate=startDate,
1483 endDate=endDate,
1465 endDate=endDate,
@@ -1494,7 +1476,6 class JRODataReader(JRODataIO):
1494 realtime=realtime,
1476 realtime=realtime,
1495 blocksize=blocksize,
1477 blocksize=blocksize,
1496 blocktime=blocktime,
1478 blocktime=blocktime,
1497 queue=queue,
1498 skip=skip,
1479 skip=skip,
1499 cursor=cursor,
1480 cursor=cursor,
1500 warnings=warnings,
1481 warnings=warnings,
Show file before | Show file after | Hide comments
@@ -119,7 +119,6 class Metadata(object):
119 self.parmConfObjList.append(parmConfObj)
119 self.parmConfObjList.append(parmConfObj)
120
120
121 class FitsWriter(Operation):
121 class FitsWriter(Operation):
122
123 def __init__(self, **kwargs):
122 def __init__(self, **kwargs):
124 Operation.__init__(self, **kwargs)
123 Operation.__init__(self, **kwargs)
125 self.isConfig = False
124 self.isConfig = False
@@ -276,9 +275,9 class FitsWriter(Operation):
276 self.setNextFile()
275 self.setNextFile()
277 self.writeNextBlock()
276 self.writeNextBlock()
278
277
279 def run(self, dataOut, **kwargs):
278 def run(self, dataOut, path, dataBlocksPerFile=100, metadatafile=None, **kwargs):
280 if not(self.isConfig):
279 if not(self.isConfig):
281 self.setup(dataOut, **kwargs)
280 self.setup(dataOut, path, dataBlocksPerFile=dataBlocksPerFile, metadatafile=metadatafile, **kwargs)
282 self.isConfig = True
281 self.isConfig = True
283 self.putData()
282 self.putData()
284
283
@@ -454,7 +453,7 class FitsReader(ProcessingUnit):
454 # self.blockIndex = 1
453 # self.blockIndex = 1
455 return 1
454 return 1
456
455
457 def __searchFilesOffLine(self,
456 def searchFilesOffLine(self,
458 path,
457 path,
459 startDate,
458 startDate,
460 endDate,
459 endDate,
@@ -560,7 +559,7 class FitsReader(ProcessingUnit):
560
559
561 if not(online):
560 if not(online):
562 print "Searching files in offline mode ..."
561 print "Searching files in offline mode ..."
563 pathList, filenameList = self.__searchFilesOffLine(path, startDate=startDate, endDate=endDate,
562 pathList, filenameList = self.searchFilesOffLine(path, startDate=startDate, endDate=endDate,
564 startTime=startTime, endTime=endTime,
563 startTime=startTime, endTime=endTime,
565 set=set, expLabel=expLabel, ext=ext,
564 set=set, expLabel=expLabel, ext=ext,
566 walk=walk)
565 walk=walk)
Show file before | Show file after | Hide comments
@@ -415,7 +415,7 class HFReader(ProcessingUnit):
415
415
416
416
417
417
418 def __searchFilesOffline(self,
418 def searchFilesOffLine(self,
419 path,
419 path,
420 startDate,
420 startDate,
421 endDate,
421 endDate,
@@ -438,7 +438,7 class HFReader(ProcessingUnit):
438
438
439 return
439 return
440
440
441 def __searchFilesOnline(self,
441 def searchFilesOnLine(self,
442 path,
442 path,
443 expLabel= "",
443 expLabel= "",
444 ext=None,
444 ext=None,
@@ -636,10 +636,10 class HFReader(ProcessingUnit):
636 if not(online):
636 if not(online):
637 print "Searching files in offline mode..."
637 print "Searching files in offline mode..."
638
638
639 self.__searchFilesOffline(path, startDate, endDate, ext, startTime, endTime, walk)
639 self.searchFilesOffLine(path, startDate, endDate, ext, startTime, endTime, walk)
640 else:
640 else:
641 print "Searching files in online mode..."
641 print "Searching files in online mode..."
642 self.__searchFilesOnline(path, walk,ext,set=set)
642 self.searchFilesOnLine(path, walk,ext,set=set)
643 if set==None:
643 if set==None:
644 pass
644 pass
645 else:
645 else:
@@ -647,7 +647,7 class HFReader(ProcessingUnit):
647
647
648 # for nTries in range(self.nTries):
648 # for nTries in range(self.nTries):
649 #
649 #
650 # fullpath,file,year,month,day,set = self.__searchFilesOnline(path=path,expLabel=expLabel,ext=ext, walk=walk,set=set)
650 # fullpath,file,year,month,day,set = self.searchFilesOnLine(path=path,expLabel=expLabel,ext=ext, walk=walk,set=set)
651 #
651 #
652 # if fullpath:
652 # if fullpath:
653 # break
653 # break
Show file before | Show file after | Hide comments
@@ -106,9 +106,9 class AMISRReader(ProcessingUnit):
106 #self.findFiles()
106 #self.findFiles()
107 if not(online):
107 if not(online):
108 #Busqueda de archivos offline
108 #Busqueda de archivos offline
109 self.__searchFilesOffline(path, startDate, endDate, startTime, endTime, walk)
109 self.searchFilesOffLine(path, startDate, endDate, startTime, endTime, walk)
110 else:
110 else:
111 self.__searchFilesOnline(path, startDate, endDate, startTime,endTime,walk)
111 self.searchFilesOnLine(path, startDate, endDate, startTime,endTime,walk)
112
112
113 if not(self.filenameList):
113 if not(self.filenameList):
114 print "There is no files into the folder: %s"%(path)
114 print "There is no files into the folder: %s"%(path)
@@ -329,7 +329,7 class AMISRReader(ProcessingUnit):
329 self.dirnameList = new_dirnameList
329 self.dirnameList = new_dirnameList
330 return 1
330 return 1
331
331
332 def __searchFilesOnline(self, path, startDate, endDate, startTime=datetime.time(0,0,0),
332 def searchFilesOnLine(self, path, startDate, endDate, startTime=datetime.time(0,0,0),
333 endTime=datetime.time(23,59,59),walk=True):
333 endTime=datetime.time(23,59,59),walk=True):
334
334
335 if endDate ==None:
335 if endDate ==None:
@@ -349,7 +349,7 class AMISRReader(ProcessingUnit):
349 return
349 return
350
350
351
351
352 def __searchFilesOffline(self,
352 def searchFilesOffLine(self,
353 path,
353 path,
354 startDate,
354 startDate,
355 endDate,
355 endDate,
Show file before | Show file after | Hide comments
@@ -97,7 +97,7 class ParamReader(ProcessingUnit):
97 self.timezone = 'lt'
97 self.timezone = 'lt'
98
98
99 print "[Reading] Searching files in offline mode ..."
99 print "[Reading] Searching files in offline mode ..."
100 pathList, filenameList = self.__searchFilesOffLine(path, startDate=startDate, endDate=endDate,
100 pathList, filenameList = self.searchFilesOffLine(path, startDate=startDate, endDate=endDate,
101 startTime=startTime, endTime=endTime,
101 startTime=startTime, endTime=endTime,
102 ext=ext, walk=walk)
102 ext=ext, walk=walk)
103
103
@@ -115,7 +115,7 class ParamReader(ProcessingUnit):
115
115
116 return
116 return
117
117
118 def __searchFilesOffLine(self,
118 def searchFilesOffLine(self,
119 path,
119 path,
120 startDate=None,
120 startDate=None,
121 endDate=None,
121 endDate=None,
@@ -607,29 +607,18 class ParamWriter(Operation):
607 self.isConfig = False
607 self.isConfig = False
608 return
608 return
609
609
610 def setup(self, dataOut, **kwargs):
610 def setup(self, dataOut, path=None, blocksPerFile=10, metadataList=None, dataList=None, mode=None, **kwargs):
611
611 self.path = path
612 self.path = kwargs['path']
612 self.blocksPerFile = blocksPerFile
613 self.setType = kwargs.get('setType', None)
613 self.metadataList = metadataList
614
614 self.dataList = dataList
615 if kwargs.has_key('blocksPerFile'):
616 self.blocksPerFile = kwargs['blocksPerFile']
617 else:
618 self.blocksPerFile = 10
619
620 self.metadataList = kwargs['metadataList']
621 self.dataList = kwargs['dataList']
622 self.dataOut = dataOut
615 self.dataOut = dataOut
616 self.mode = mode
623
617
624 if kwargs.has_key('mode'):
618 if self.mode is not None:
625 mode = kwargs['mode']
619 self.mode = numpy.zeros(len(self.dataList)) + mode
626
627 if type(mode) == int:
628 mode = numpy.zeros(len(self.dataList)) + mode
629 else:
620 else:
630 mode = numpy.ones(len(self.dataList))
621 self.mode = numpy.ones(len(self.dataList))
631
632 self.mode = mode
633
622
634 arrayDim = numpy.zeros((len(self.dataList),5))
623 arrayDim = numpy.zeros((len(self.dataList),5))
635
624
@@ -1089,10 +1078,11 class ParamWriter(Operation):
1089 self.fp.close()
1078 self.fp.close()
1090 return
1079 return
1091
1080
1092 def run(self, dataOut, **kwargs):
1081 def run(self, dataOut, path=None, blocksPerFile=10, metadataList=None, dataList=None, mode=None, **kwargs):
1093
1082
1094 if not(self.isConfig):
1083 if not(self.isConfig):
1095 flagdata = self.setup(dataOut, **kwargs)
1084 flagdata = self.setup(dataOut, path=path, blocksPerFile=blocksPerFile,
1085 metadataList=metadataList, dataList=dataList, mode=mode, **kwargs)
1096
1086
1097 if not(flagdata):
1087 if not(flagdata):
1098 return
1088 return
Show file before | Show file after | Hide comments
1 NO CONTENT: modified file
NO CONTENT: modified file
Show file before | Show file after | Hide comments
@@ -1,11 +1,21
1 #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
2 #define NUM_CPY_THREADS 8
1 #include <Python.h>
3 #include <Python.h>
2 #include <numpy/arrayobject.h>
4 #include <numpy/arrayobject.h>
3 #include <math.h>
5 #include <math.h>
6 #include <complex.h>
7 #include <time.h>
4
8
9 // void printArr(int *array);
5 static PyObject *hildebrand_sekhon(PyObject *self, PyObject *args);
10 static PyObject *hildebrand_sekhon(PyObject *self, PyObject *args);
11 static PyObject *correlateByBlock(PyObject *self, PyObject *args);
12 #ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
13 #define PyMODINIT_FUNC void
14 #endif
6
15
7 static PyMethodDef extensionsMethods[] = {
16 static PyMethodDef extensionsMethods[] = {
8 { "hildebrand_sekhon", (PyCFunction)hildebrand_sekhon, METH_VARARGS, "get noise with" },
17 { "correlateByBlock", (PyCFunction)correlateByBlock, METH_VARARGS, "get correlation by block" },
18 { "hildebrand_sekhon", (PyCFunction)hildebrand_sekhon, METH_VARARGS, "get noise with hildebrand_sekhon" },
9 { NULL, NULL, 0, NULL }
19 { NULL, NULL, 0, NULL }
10 };
20 };
11
21
@@ -14,13 +24,83 PyMODINIT_FUNC initcSchain() {
14 import_array();
24 import_array();
15 }
25 }
16
26
27 static PyObject *correlateByBlock(PyObject *self, PyObject *args) {
28
29 // int *x = (int*) malloc(4000000 * 216 * sizeof(int));;
30 // int a = 5;
31 // x = &a;
32 // int b = 6;
33 // x = &b;
34 // printf("Antes de imprimir x \n");
35 // printf("%d \n", x[0]);
36
37 PyObject *data_obj1, *data_obj2;
38 PyArrayObject *data_array1, *data_array2, *correlateRow, *out, *dataRow, *codeRow; //, ,
39 int mode;
40
41 if (!PyArg_ParseTuple(args, "OOi", &data_obj1, &data_obj2, &mode)) return NULL;
42
43 data_array1 = (PyArrayObject *) PyArray_FROM_OTF(data_obj1, NPY_COMPLEX128, NPY_ARRAY_DEFAULT);
44 data_array2 = (PyArrayObject *) PyArray_FROM_OTF(data_obj2, NPY_FLOAT64, NPY_ARRAY_DEFAULT);
45
46 npy_intp dims[1];
47 dims[0] = 200;
48 npy_intp dims_code[1];
49 dims_code[0] = 16;
50
51 double complex * dataRaw;
52 double * codeRaw;
53 dataRaw = (double complex*) PyArray_DATA(data_array1);
54 codeRaw = (double *) PyArray_DATA(data_array2);
55 double complex ** outC = malloc(40000*200*sizeof(double complex));
56 int i;
57
58 clock_t start = clock();
59 for(i=0; i<40000; i++){
60 // codeRow = PyArray_SimpleNewFromData(1, dims_code, NPY_FLOAT64, codeRaw + 16 * i);
61 // dataRow = PyArray_SimpleNewFromData(1, dims, NPY_COMPLEX128, dataRaw + 200 * i);
62 // Py_INCREF(codeRow);
63 // Py_INCREF(dataRow);
64 // PyArray_ENABLEFLAGS(codeRow, NPY_ARRAY_OWNDATA);
65 // PyArray_ENABLEFLAGS(dataRow, NPY_ARRAY_OWNDATA);
66 correlateRow = (PyArrayObject *) PyArray_Correlate2(PyArray_SimpleNewFromData(1, dims_code, NPY_FLOAT64, codeRaw + 16 * i), PyArray_SimpleNewFromData(1, dims, NPY_COMPLEX128, dataRaw + 200 * i), (npy_intp) 2);
67 //Py_INCREF(correlateRow);
68 // PyArray_ENABLEFLAGS(correlateRow, NPY_ARRAY_OWNDATA);
69 memcpy(outC + 200*i, (double complex*) PyArray_DATA(correlateRow), 200 * sizeof(double complex));
70
71 Py_DECREF(correlateRow);
72 // Py_DECREF(codeRow);
73 // Py_DECREF(dataRow);
74 }
75 clock_t end = clock();
76 float seconds = (float)(end - start) / CLOCKS_PER_SEC;
77 printf("%f", seconds);
78 //
79 npy_intp dimsret[2];
80 dimsret[0] = 40000;
81 dimsret[1] = 200;
82 out = PyArray_SimpleNewFromData(2, dimsret, NPY_COMPLEX128, outC);
83 PyArray_ENABLEFLAGS(out, NPY_ARRAY_OWNDATA);
84 //Py_INCREF(out);
85 Py_DECREF(data_array1);
86 Py_DECREF(data_array2);
87 // PyArray_DebugPrint(out);
88 // Py_DECREF(data_obj2);
89 // Py_DECREF(data_obj1);
90 // Py_DECREF(codeRow);
91 // Py_DECREF(dataRow);
92 // free(dataRaw);
93 // free(codeRaw);
94
95 return PyArray_Return(out);
96 }
97
17 static PyObject *hildebrand_sekhon(PyObject *self, PyObject *args) {
98 static PyObject *hildebrand_sekhon(PyObject *self, PyObject *args) {
18 /* Do your stuff here. */
19 double navg;
99 double navg;
20 PyObject *data_obj, *data_array;
100 PyObject *data_obj, *data_array;
21
101
22 if (!PyArg_ParseTuple(args, "Od", &data_obj, &navg)) return NULL;
102 if (!PyArg_ParseTuple(args, "Od", &data_obj, &navg)) return NULL;
23 data_array = PyArray_FROM_OTF(data_obj, NPY_FLOAT64, NPY_IN_ARRAY);
103 data_array = PyArray_FROM_OTF(data_obj, NPY_FLOAT64, NPY_ARRAY_DEFAULT);
24 if (data_array == NULL) {
104 if (data_array == NULL) {
25 Py_XDECREF(data_array);
105 Py_XDECREF(data_array);
26 Py_XDECREF(data_obj);
106 Py_XDECREF(data_obj);
@@ -56,3 +136,4 static PyObject *hildebrand_sekhon(PyObject *self, PyObject *args) {
56
136
57 return Py_BuildValue("d", lnoise);
137 return Py_BuildValue("d", lnoise);
58 }
138 }
139
Show file before | Show file after | Hide comments
@@ -200,7 +200,6 class ProcessingUnit(object):
200 return True
200 return True
201
201
202 def call(self, opType, opName=None, opId=None):
202 def call(self, opType, opName=None, opId=None):
203
204 """
203 """
205 Return True si ejecuta la operacion interna nombrada "opName" o la operacion externa
204 Return True si ejecuta la operacion interna nombrada "opName" o la operacion externa
206 identificada con el id "opId"; con los argumentos "**kwargs".
205 identificada con el id "opId"; con los argumentos "**kwargs".
Show file before | Show file after | Hide comments
@@ -163,7 +163,6 class IncohInt4SpectraHeis(Operation):
163
163
164 n = None
164 n = None
165
165
166
167 def __init__(self, **kwargs):
166 def __init__(self, **kwargs):
168
167
169 Operation.__init__(self, **kwargs)
168 Operation.__init__(self, **kwargs)
@@ -324,10 +323,10 class IncohInt4SpectraHeis(Operation):
324
323
325 return avgdata, avgdatatime
324 return avgdata, avgdatatime
326
325
327 def run(self, dataOut, **kwargs):
326 def run(self, dataOut, n=None, timeInterval=None, overlapping=False, **kwargs):
328
327
329 if not self.isConfig:
328 if not self.isConfig:
330 self.setup(**kwargs)
329 self.setup(n=n, timeInterval=timeInterval, overlapping=overlapping)
331 self.isConfig = True
330 self.isConfig = True
332
331
333 avgdata, avgdatatime = self.integrate(dataOut.data_spc, dataOut.utctime)
332 avgdata, avgdatatime = self.integrate(dataOut.data_spc, dataOut.utctime)
Show file before | Show file after | Hide comments
@@ -52,13 +52,6 def _unpickle_method(func_name, obj, cls):
52 break
52 break
53 return func.__get__(obj, cls)
53 return func.__get__(obj, cls)
54
54
55
56
57
58
59
60
61
62 class ParametersProc(ProcessingUnit):
55 class ParametersProc(ProcessingUnit):
63
56
64 nSeconds = None
57 nSeconds = None
@@ -104,9 +97,7 class ParametersProc(ProcessingUnit):
104 self.dataOut.timeInterval1 = self.dataIn.timeInterval
97 self.dataOut.timeInterval1 = self.dataIn.timeInterval
105 self.dataOut.heightList = self.dataIn.getHeiRange()
98 self.dataOut.heightList = self.dataIn.getHeiRange()
106 self.dataOut.frequency = self.dataIn.frequency
99 self.dataOut.frequency = self.dataIn.frequency
107 self.dataOut.noise = self.dataIn.noise
100 # self.dataOut.noise = self.dataIn.noise
108
109
110
101
111 def run(self):
102 def run(self):
112
103
@@ -126,22 +117,23 class ParametersProc(ProcessingUnit):
126 if self.dataIn.type == "Spectra":
117 if self.dataIn.type == "Spectra":
127
118
128 self.dataOut.data_pre = (self.dataIn.data_spc, self.dataIn.data_cspc)
119 self.dataOut.data_pre = (self.dataIn.data_spc, self.dataIn.data_cspc)
129 print 'self.dataIn.data_spc', self.dataIn.data_spc.shape
120 self.dataOut.data_spc = self.dataIn.data_spc
121 self.dataOut.data_cspc = self.dataIn.data_cspc
122 self.dataOut.nProfiles = self.dataIn.nProfiles
123 self.dataOut.nIncohInt = self.dataIn.nIncohInt
124 self.dataOut.nFFTPoints = self.dataIn.nFFTPoints
125 self.dataOut.ippFactor = self.dataIn.ippFactor
130 self.dataOut.abscissaList = self.dataIn.getVelRange(1)
126 self.dataOut.abscissaList = self.dataIn.getVelRange(1)
131 self.dataOut.spc_noise = self.dataIn.getNoise()
127 self.dataOut.spc_noise = self.dataIn.getNoise()
132 self.dataOut.spc_range = (self.dataIn.getFreqRange(1)/1000. , self.dataIn.getAcfRange(1) , self.dataIn.getVelRange(1))
128 self.dataOut.spc_range = (self.dataIn.getFreqRange(1)/1000. , self.dataIn.getAcfRange(1) , self.dataIn.getVelRange(1))
133
129 self.dataOut.pairsList = self.dataIn.pairsList
134 self.dataOut.normFactor = self.dataIn.normFactor
135 #self.dataOut.outputInterval = self.dataIn.outputInterval
136 self.dataOut.groupList = self.dataIn.pairsList
130 self.dataOut.groupList = self.dataIn.pairsList
137 self.dataOut.flagNoData = False
131 self.dataOut.flagNoData = False
138 #print 'datain chandist ',self.dataIn.ChanDist
132
139 if hasattr(self.dataIn, 'ChanDist'): #Distances of receiver channels
133 if hasattr(self.dataIn, 'ChanDist'): #Distances of receiver channels
140 self.dataOut.ChanDist = self.dataIn.ChanDist
134 self.dataOut.ChanDist = self.dataIn.ChanDist
141 else: self.dataOut.ChanDist = None
135 else: self.dataOut.ChanDist = None
142
136
143 print 'datain chandist ',self.dataOut.ChanDist
144
145 if hasattr(self.dataIn, 'VelRange'): #Velocities range
137 if hasattr(self.dataIn, 'VelRange'): #Velocities range
146 self.dataOut.VelRange = self.dataIn.VelRange
138 self.dataOut.VelRange = self.dataIn.VelRange
147 else: self.dataOut.VelRange = None
139 else: self.dataOut.VelRange = None
Show file before | Show file after | Hide comments
@@ -1,3 +1,5
1 import itertools
2
1 import numpy
3 import numpy
2
4
3 from jroproc_base import ProcessingUnit, Operation
5 from jroproc_base import ProcessingUnit, Operation
@@ -109,7 +111,10 class SpectraProc(ProcessingUnit):
109
111
110 if self.dataIn.type == "Spectra":
112 if self.dataIn.type == "Spectra":
111 self.dataOut.copy(self.dataIn)
113 self.dataOut.copy(self.dataIn)
112 # self.__selectPairs(pairsList)
114 if not pairsList:
115 pairsList = itertools.combinations(self.dataOut.channelList, 2)
116 if self.dataOut.data_cspc is not None:
117 self.__selectPairs(pairsList)
113 return True
118 return True
114
119
115 if self.dataIn.type == "Voltage":
120 if self.dataIn.type == "Voltage":
@@ -178,27 +183,21 class SpectraProc(ProcessingUnit):
178
183
179 def __selectPairs(self, pairsList):
184 def __selectPairs(self, pairsList):
180
185
181 if channelList == None:
186 if not pairsList:
182 return
187 return
183
188
184 pairsIndexListSelected = []
189 pairs = []
185
190 pairsIndex = []
186 for thisPair in pairsList:
187
191
188 if thisPair not in self.dataOut.pairsList:
192 for pair in pairsList:
193 if pair[0] not in self.dataOut.channelList or pair[1] not in self.dataOut.channelList:
189 continue
194 continue
195 pairs.append(pair)
196 pairsIndex.append(pairs.index(pair))
190
197
191 pairIndex = self.dataOut.pairsList.index(thisPair)
198 self.dataOut.data_cspc = self.dataOut.data_cspc[pairsIndex]
192
199 self.dataOut.pairsList = pairs
193 pairsIndexListSelected.append(pairIndex)
200 self.dataOut.pairsIndexList = pairsIndex
194
195 if not pairsIndexListSelected:
196 self.dataOut.data_cspc = None
197 self.dataOut.pairsList = []
198 return
199
200 self.dataOut.data_cspc = self.dataOut.data_cspc[pairsIndexListSelected]
201 self.dataOut.pairsList = [self.dataOut.pairsList[i] for i in pairsIndexListSelected]
202
201
203 return
202 return
204
203
Show file before | Show file after | Hide comments
@@ -1,9 +1,10
1 import sys
1 import sys
2 import numpy
2 import numpy
3 from scipy import interpolate
3 from scipy import interpolate
4
4 from schainpy import cSchain
5 from jroproc_base import ProcessingUnit, Operation
5 from jroproc_base import ProcessingUnit, Operation
6 from schainpy.model.data.jrodata import Voltage
6 from schainpy.model.data.jrodata import Voltage
7 from time import time
7
8
8 class VoltageProc(ProcessingUnit):
9 class VoltageProc(ProcessingUnit):
9
10
@@ -332,7 +333,6 class CohInt(Operation):
332
333
333 n = None
334 n = None
334
335
335
336 def __init__(self, **kwargs):
336 def __init__(self, **kwargs):
337
337
338 Operation.__init__(self, **kwargs)
338 Operation.__init__(self, **kwargs)
@@ -348,7 +348,6 class CohInt(Operation):
348 n : Number of coherent integrations
348 n : Number of coherent integrations
349 timeInterval : Time of integration. If the parameter "n" is selected this one does not work
349 timeInterval : Time of integration. If the parameter "n" is selected this one does not work
350 overlapping :
350 overlapping :
351
352 """
351 """
353
352
354 self.__initime = None
353 self.__initime = None
@@ -548,7 +547,6 class Decoder(Operation):
548 nCode = None
547 nCode = None
549 nBaud = None
548 nBaud = None
550
549
551
552 def __init__(self, **kwargs):
550 def __init__(self, **kwargs):
553
551
554 Operation.__init__(self, **kwargs)
552 Operation.__init__(self, **kwargs)
@@ -631,11 +629,11 class Decoder(Operation):
631 junk = numpy.lib.stride_tricks.as_strided(self.code, (repetitions, self.code.size), (0, self.code.itemsize))
629 junk = numpy.lib.stride_tricks.as_strided(self.code, (repetitions, self.code.size), (0, self.code.itemsize))
632 junk = junk.flatten()
630 junk = junk.flatten()
633 code_block = numpy.reshape(junk, (self.nCode*repetitions, self.nBaud))
631 code_block = numpy.reshape(junk, (self.nCode*repetitions, self.nBaud))
632 profilesList = xrange(self.__nProfiles)
634
633
635 for i in range(self.__nChannels):
634 for i in range(self.__nChannels):
636 for j in range(self.__nProfiles):
635 for j in profilesList:
637 self.datadecTime[i,j,:] = numpy.correlate(data[i,j,:], code_block[j,:], mode='full')[self.nBaud-1:]
636 self.datadecTime[i,j,:] = numpy.correlate(data[i,j,:], code_block[j,:], mode='full')[self.nBaud-1:]
638
639 return self.datadecTime
637 return self.datadecTime
640
638
641 def __convolutionByBlockInFreq(self, data):
639 def __convolutionByBlockInFreq(self, data):
@@ -653,6 +651,7 class Decoder(Operation):
653
651
654 return data
652 return data
655
653
654
656 def run(self, dataOut, code=None, nCode=None, nBaud=None, mode = 0, osamp=None, times=None):
655 def run(self, dataOut, code=None, nCode=None, nBaud=None, mode = 0, osamp=None, times=None):
657
656
658 if dataOut.flagDecodeData:
657 if dataOut.flagDecodeData:
@@ -1087,7 +1086,6 class SplitProfiles(Operation):
1087 dataOut.ippSeconds /= n
1086 dataOut.ippSeconds /= n
1088
1087
1089 class CombineProfiles(Operation):
1088 class CombineProfiles(Operation):
1090
1091 def __init__(self, **kwargs):
1089 def __init__(self, **kwargs):
1092
1090
1093 Operation.__init__(self, **kwargs)
1091 Operation.__init__(self, **kwargs)
Show file before | Show file after | Hide comments
@@ -32,9 +32,7 class Remote(Thread):
32 None
32 None
33
33
34 Written by:
34 Written by:
35
36 "Miguel Urco":mailto:miguel.urco@jro.igp.gob.pe Jun. 03, 2015
35 "Miguel Urco":mailto:miguel.urco@jro.igp.gob.pe Jun. 03, 2015
37
38 """
36 """
39
37
40 server = None
38 server = None
@@ -621,6 +619,7 class SendToServer(ProcessingUnit):
621 filenameList = glob.glob1(thisFolder, '*%s' %self.ext)
619 filenameList = glob.glob1(thisFolder, '*%s' %self.ext)
622
620
623 if len(filenameList) < 1:
621 if len(filenameList) < 1:
622
624 continue
623 continue
625
624
626 for thisFile in filenameList:
625 for thisFile in filenameList:
Show file before | Show file after | Hide comments
@@ -7,7 +7,6 import json
7 import numpy
7 import numpy
8 import paho.mqtt.client as mqtt
8 import paho.mqtt.client as mqtt
9 import zmq
9 import zmq
10 from profilehooks import profile
11 import datetime
10 import datetime
12 from zmq.utils.monitor import recv_monitor_message
11 from zmq.utils.monitor import recv_monitor_message
13 from functools import wraps
12 from functools import wraps
@@ -16,6 +15,7 from multiprocessing import Process
16
15
17 from schainpy.model.proc.jroproc_base import Operation, ProcessingUnit
16 from schainpy.model.proc.jroproc_base import Operation, ProcessingUnit
18 from schainpy.model.data.jrodata import JROData
17 from schainpy.model.data.jrodata import JROData
18 from schainpy.utils import log
19
19
20 MAXNUMX = 100
20 MAXNUMX = 100
21 MAXNUMY = 100
21 MAXNUMY = 100
@@ -31,14 +31,13 def roundFloats(obj):
31 return round(obj, 2)
31 return round(obj, 2)
32
32
33 def decimate(z, MAXNUMY):
33 def decimate(z, MAXNUMY):
34 # dx = int(len(self.x)/self.__MAXNUMX) + 1
35
36 dy = int(len(z[0])/MAXNUMY) + 1
34 dy = int(len(z[0])/MAXNUMY) + 1
37
35
38 return z[::, ::dy]
36 return z[::, ::dy]
39
37
40 class throttle(object):
38 class throttle(object):
41 """Decorator that prevents a function from being called more than once every
39 '''
40 Decorator that prevents a function from being called more than once every
42 time period.
41 time period.
43 To create a function that cannot be called more than once a minute, but
42 To create a function that cannot be called more than once a minute, but
44 will sleep until it can be called:
43 will sleep until it can be called:
@@ -49,7 +48,7 class throttle(object):
49 for i in range(10):
48 for i in range(10):
50 foo()
49 foo()
51 print "This function has run %s times." % i
50 print "This function has run %s times." % i
52 """
51 '''
53
52
54 def __init__(self, seconds=0, minutes=0, hours=0):
53 def __init__(self, seconds=0, minutes=0, hours=0):
55 self.throttle_period = datetime.timedelta(
54 self.throttle_period = datetime.timedelta(
@@ -73,9 +72,169 class throttle(object):
73
72
74 return wrapper
73 return wrapper
75
74
75 class Data(object):
76 '''
77 Object to hold data to be plotted
78 '''
79
80 def __init__(self, plottypes, throttle_value):
81 self.plottypes = plottypes
82 self.throttle = throttle_value
83 self.ended = False
84 self.__times = []
85
86 def __str__(self):
87 dum = ['{}{}'.format(key, self.shape(key)) for key in self.data]
88 return 'Data[{}][{}]'.format(';'.join(dum), len(self.__times))
89
90 def __len__(self):
91 return len(self.__times)
92
93 def __getitem__(self, key):
94 if key not in self.data:
95 raise KeyError(log.error('Missing key: {}'.format(key)))
96
97 if 'spc' in key:
98 ret = self.data[key]
99 else:
100 ret = numpy.array([self.data[key][x] for x in self.times])
101 if ret.ndim > 1:
102 ret = numpy.swapaxes(ret, 0, 1)
103 return ret
104
105 def setup(self):
106 '''
107 Configure object
108 '''
109
110 self.ended = False
111 self.data = {}
112 self.__times = []
113 self.__heights = []
114 self.__all_heights = set()
115 for plot in self.plottypes:
116 self.data[plot] = {}
117
118 def shape(self, key):
119 '''
120 Get the shape of the one-element data for the given key
121 '''
122
123 if len(self.data[key]):
124 if 'spc' in key:
125 return self.data[key].shape
126 return self.data[key][self.__times[0]].shape
127 return (0,)
128
129 def update(self, dataOut):
130 '''
131 Update data object with new dataOut
132 '''
133
134 tm = dataOut.utctime
135 if tm in self.__times:
136 return
137
138 self.parameters = getattr(dataOut, 'parameters', [])
139 self.pairs = dataOut.pairsList
140 self.channels = dataOut.channelList
141 self.xrange = (dataOut.getFreqRange(1)/1000. , dataOut.getAcfRange(1) , dataOut.getVelRange(1))
142 self.interval = dataOut.getTimeInterval()
143 self.__heights.append(dataOut.heightList)
144 self.__all_heights.update(dataOut.heightList)
145 self.__times.append(tm)
146
147 for plot in self.plottypes:
148 if plot == 'spc':
149 z = dataOut.data_spc/dataOut.normFactor
150 self.data[plot] = 10*numpy.log10(z)
151 if plot == 'cspc':
152 self.data[plot] = dataOut.data_cspc
153 if plot == 'noise':
154 self.data[plot][tm] = 10*numpy.log10(dataOut.getNoise()/dataOut.normFactor)
155 if plot == 'rti':
156 self.data[plot][tm] = dataOut.getPower()
157 if plot == 'snr_db':
158 self.data['snr'][tm] = dataOut.data_SNR
159 if plot == 'snr':
160 self.data[plot][tm] = 10*numpy.log10(dataOut.data_SNR)
161 if plot == 'dop':
162 self.data[plot][tm] = 10*numpy.log10(dataOut.data_DOP)
163 if plot == 'mean':
164 self.data[plot][tm] = dataOut.data_MEAN
165 if plot == 'std':
166 self.data[plot][tm] = dataOut.data_STD
167 if plot == 'coh':
168 self.data[plot][tm] = dataOut.getCoherence()
169 if plot == 'phase':
170 self.data[plot][tm] = dataOut.getCoherence(phase=True)
171 if plot == 'output':
172 self.data[plot][tm] = dataOut.data_output
173 if plot == 'param':
174 self.data[plot][tm] = dataOut.data_param
175
176 def normalize_heights(self):
177 '''
178 Ensure same-dimension of the data for different heighList
179 '''
180
181 H = numpy.array(list(self.__all_heights))
182 H.sort()
183 for key in self.data:
184 shape = self.shape(key)[:-1] + H.shape
185 for tm, obj in self.data[key].items():
186 h = self.__heights[self.__times.index(tm)]
187 if H.size == h.size:
188 continue
189 index = numpy.where(numpy.in1d(H, h))[0]
190 dummy = numpy.zeros(shape) + numpy.nan
191 if len(shape) == 2:
192 dummy[:, index] = obj
193 else:
194 dummy[index] = obj
195 self.data[key][tm] = dummy
196
197 self.__heights = [H for tm in self.__times]
198
199 def jsonify(self, decimate=False):
200 '''
201 Convert data to json
202 '''
203
204 ret = {}
205 tm = self.times[-1]
206
207 for key, value in self.data:
208 if key in ('spc', 'cspc'):
209 ret[key] = roundFloats(self.data[key].to_list())
210 else:
211 ret[key] = roundFloats(self.data[key][tm].to_list())
212
213 ret['timestamp'] = tm
214 ret['interval'] = self.interval
215
216 @property
217 def times(self):
218 '''
219 Return the list of times of the current data
220 '''
221
222 ret = numpy.array(self.__times)
223 ret.sort()
224 return ret
225
226 @property
227 def heights(self):
228 '''
229 Return the list of heights of the current data
230 '''
231
232 return numpy.array(self.__heights[-1])
76
233
77 class PublishData(Operation):
234 class PublishData(Operation):
78 """Clase publish."""
235 '''
236 Operation to send data over zmq.
237 '''
79
238
80 def __init__(self, **kwargs):
239 def __init__(self, **kwargs):
81 """Inicio."""
240 """Inicio."""
@@ -87,11 +246,11 class PublishData(Operation):
87
246
88 def on_disconnect(self, client, userdata, rc):
247 def on_disconnect(self, client, userdata, rc):
89 if rc != 0:
248 if rc != 0:
90 print("Unexpected disconnection.")
249 log.warning('Unexpected disconnection.')
91 self.connect()
250 self.connect()
92
251
93 def connect(self):
252 def connect(self):
94 print 'trying to connect'
253 log.warning('trying to connect')
95 try:
254 try:
96 self.client.connect(
255 self.client.connect(
97 host=self.host,
256 host=self.host,
@@ -105,7 +264,7 class PublishData(Operation):
105 # retain=True
264 # retain=True
106 # )
265 # )
107 except:
266 except:
108 print "MQTT Conection error."
267 log.error('MQTT Conection error.')
109 self.client = False
268 self.client = False
110
269
111 def setup(self, port=1883, username=None, password=None, clientId="user", zeromq=1, verbose=True, **kwargs):
270 def setup(self, port=1883, username=None, password=None, clientId="user", zeromq=1, verbose=True, **kwargs):
@@ -121,7 +280,6 class PublishData(Operation):
121 self.mqtt = kwargs.get('plottype', 0)
280 self.mqtt = kwargs.get('plottype', 0)
122 self.client = None
281 self.client = None
123 self.verbose = verbose
282 self.verbose = verbose
124 self.dataOut.firstdata = True
125 setup = []
283 setup = []
126 if mqtt is 1:
284 if mqtt is 1:
127 self.client = mqtt.Client(
285 self.client = mqtt.Client(
@@ -176,7 +334,6 class PublishData(Operation):
176 'type': self.plottype,
334 'type': self.plottype,
177 'yData': yData
335 'yData': yData
178 }
336 }
179 # print payload
180
337
181 elif self.plottype in ('rti', 'power'):
338 elif self.plottype in ('rti', 'power'):
182 data = getattr(self.dataOut, 'data_spc')
339 data = getattr(self.dataOut, 'data_spc')
@@ -230,15 +387,16 class PublishData(Operation):
230 'timestamp': 'None',
387 'timestamp': 'None',
231 'type': None
388 'type': None
232 }
389 }
233 # print 'Publishing data to {}'.format(self.host)
390
234 self.client.publish(self.topic + self.plottype, json.dumps(payload), qos=0)
391 self.client.publish(self.topic + self.plottype, json.dumps(payload), qos=0)
235
392
236 if self.zeromq is 1:
393 if self.zeromq is 1:
237 if self.verbose:
394 if self.verbose:
238 print '[Sending] {} - {}'.format(self.dataOut.type, self.dataOut.datatime)
395 log.log(
396 '{} - {}'.format(self.dataOut.type, self.dataOut.datatime),
397 'Sending'
398 )
239 self.zmq_socket.send_pyobj(self.dataOut)
399 self.zmq_socket.send_pyobj(self.dataOut)
240 self.dataOut.firstdata = False
241
242
400
243 def run(self, dataOut, **kwargs):
401 def run(self, dataOut, **kwargs):
244 self.dataOut = dataOut
402 self.dataOut = dataOut
@@ -253,6 +411,7 class PublishData(Operation):
253 if self.zeromq is 1:
411 if self.zeromq is 1:
254 self.dataOut.finished = True
412 self.dataOut.finished = True
255 self.zmq_socket.send_pyobj(self.dataOut)
413 self.zmq_socket.send_pyobj(self.dataOut)
414 time.sleep(0.1)
256 self.zmq_socket.close()
415 self.zmq_socket.close()
257 if self.client:
416 if self.client:
258 self.client.loop_stop()
417 self.client.loop_stop()
@@ -281,7 +440,7 class ReceiverData(ProcessingUnit):
281 self.receiver = self.context.socket(zmq.PULL)
440 self.receiver = self.context.socket(zmq.PULL)
282 self.receiver.bind(self.address)
441 self.receiver.bind(self.address)
283 time.sleep(0.5)
442 time.sleep(0.5)
284 print '[Starting] ReceiverData from {}'.format(self.address)
443 log.success('ReceiverData from {}'.format(self.address))
285
444
286
445
287 def run(self):
446 def run(self):
@@ -291,8 +450,9 class ReceiverData(ProcessingUnit):
291 self.isConfig = True
450 self.isConfig = True
292
451
293 self.dataOut = self.receiver.recv_pyobj()
452 self.dataOut = self.receiver.recv_pyobj()
294 print '[Receiving] {} - {}'.format(self.dataOut.type,
453 log.log('{} - {}'.format(self.dataOut.type,
295 self.dataOut.datatime.ctime())
454 self.dataOut.datatime.ctime(),),
455 'Receiving')
296
456
297
457
298 class PlotterReceiver(ProcessingUnit, Process):
458 class PlotterReceiver(ProcessingUnit, Process):
@@ -306,7 +466,6 class PlotterReceiver(ProcessingUnit, Process):
306 self.mp = False
466 self.mp = False
307 self.isConfig = False
467 self.isConfig = False
308 self.isWebConfig = False
468 self.isWebConfig = False
309 self.plottypes = []
310 self.connections = 0
469 self.connections = 0
311 server = kwargs.get('server', 'zmq.pipe')
470 server = kwargs.get('server', 'zmq.pipe')
312 plot_server = kwargs.get('plot_server', 'zmq.web')
471 plot_server = kwargs.get('plot_server', 'zmq.web')
@@ -326,19 +485,13 class PlotterReceiver(ProcessingUnit, Process):
326 self.realtime = kwargs.get('realtime', False)
485 self.realtime = kwargs.get('realtime', False)
327 self.throttle_value = kwargs.get('throttle', 5)
486 self.throttle_value = kwargs.get('throttle', 5)
328 self.sendData = self.initThrottle(self.throttle_value)
487 self.sendData = self.initThrottle(self.throttle_value)
488 self.dates = []
329 self.setup()
489 self.setup()
330
490
331 def setup(self):
491 def setup(self):
332
492
333 self.data = {}
493 self.data = Data(self.plottypes, self.throttle_value)
334 self.data['times'] = []
335 for plottype in self.plottypes:
336 self.data[plottype] = {}
337 self.data['noise'] = {}
338 self.data['throttle'] = self.throttle_value
339 self.data['ENDED'] = False
340 self.isConfig = True
494 self.isConfig = True
341 self.data_web = {}
342
495
343 def event_monitor(self, monitor):
496 def event_monitor(self, monitor):
344
497
@@ -355,15 +508,13 class PlotterReceiver(ProcessingUnit, Process):
355 self.connections += 1
508 self.connections += 1
356 if evt['event'] == 512:
509 if evt['event'] == 512:
357 pass
510 pass
358 if self.connections == 0 and self.started is True:
359 self.ended = True
360
511
361 evt.update({'description': events[evt['event']]})
512 evt.update({'description': events[evt['event']]})
362
513
363 if evt['event'] == zmq.EVENT_MONITOR_STOPPED:
514 if evt['event'] == zmq.EVENT_MONITOR_STOPPED:
364 break
515 break
365 monitor.close()
516 monitor.close()
366 print("event monitor thread done!")
517 print('event monitor thread done!')
367
518
368 def initThrottle(self, throttle_value):
519 def initThrottle(self, throttle_value):
369
520
@@ -373,65 +524,16 class PlotterReceiver(ProcessingUnit, Process):
373
524
374 return sendDataThrottled
525 return sendDataThrottled
375
526
376
377 def send(self, data):
527 def send(self, data):
378 # print '[sending] data=%s size=%s' % (data.keys(), len(data['times']))
528 log.success('Sending {}'.format(data), self.name)
379 self.sender.send_pyobj(data)
529 self.sender.send_pyobj(data)
380
530
381
382 def update(self):
383 t = self.dataOut.utctime
384
385 if t in self.data['times']:
386 return
387
388 self.data['times'].append(t)
389 self.data['dataOut'] = self.dataOut
390
391 for plottype in self.plottypes:
392 if plottype == 'spc':
393 z = self.dataOut.data_spc/self.dataOut.normFactor
394 self.data[plottype] = 10*numpy.log10(z)
395 self.data['noise'][t] = 10*numpy.log10(self.dataOut.getNoise()/self.dataOut.normFactor)
396 if plottype == 'cspc':
397 jcoherence = self.dataOut.data_cspc/numpy.sqrt(self.dataOut.data_spc*self.dataOut.data_spc)
398 self.data['cspc_coh'] = numpy.abs(jcoherence)
399 self.data['cspc_phase'] = numpy.arctan2(jcoherence.imag, jcoherence.real)*180/numpy.pi
400 if plottype == 'rti':
401 self.data[plottype][t] = self.dataOut.getPower()
402 if plottype == 'snr':
403 self.data[plottype][t] = 10*numpy.log10(self.dataOut.data_SNR)
404 if plottype == 'dop':
405 self.data[plottype][t] = 10*numpy.log10(self.dataOut.data_DOP)
406 if plottype == 'mean':
407 self.data[plottype][t] = self.dataOut.data_MEAN
408 if plottype == 'std':
409 self.data[plottype][t] = self.dataOut.data_STD
410 if plottype == 'coh':
411 self.data[plottype][t] = self.dataOut.getCoherence()
412 if plottype == 'phase':
413 self.data[plottype][t] = self.dataOut.getCoherence(phase=True)
414 if plottype == 'output':
415 self.data[plottype][t] = self.dataOut.data_output
416 if plottype == 'param':
417 self.data[plottype][t] = self.dataOut.data_param
418 if self.realtime:
419 self.data_web['timestamp'] = t
420 if plottype == 'spc':
421 self.data_web[plottype] = roundFloats(decimate(self.data[plottype]).tolist())
422 elif plottype == 'cspc':
423 self.data_web['cspc_coh'] = roundFloats(decimate(self.data['cspc_coh']).tolist())
424 self.data_web['cspc_phase'] = roundFloats(decimate(self.data['cspc_phase']).tolist())
425 elif plottype == 'noise':
426 self.data_web['noise'] = roundFloats(self.data['noise'][t].tolist())
427 else:
428 self.data_web[plottype] = roundFloats(decimate(self.data[plottype][t]).tolist())
429 self.data_web['interval'] = self.dataOut.getTimeInterval()
430 self.data_web['type'] = plottype
431
432 def run(self):
531 def run(self):
433
532
434 print '[Starting] {} from {}'.format(self.name, self.address)
533 log.success(
534 'Starting from {}'.format(self.address),
535 self.name
536 )
435
537
436 self.context = zmq.Context()
538 self.context = zmq.Context()
437 self.receiver = self.context.socket(zmq.PULL)
539 self.receiver = self.context.socket(zmq.PULL)
@@ -448,39 +550,39 class PlotterReceiver(ProcessingUnit, Process):
448 else:
550 else:
449 self.sender.bind("ipc:///tmp/zmq.plots")
551 self.sender.bind("ipc:///tmp/zmq.plots")
450
552
451 time.sleep(3)
553 time.sleep(2)
452
554
453 t = Thread(target=self.event_monitor, args=(monitor,))
555 t = Thread(target=self.event_monitor, args=(monitor,))
454 t.start()
556 t.start()
455
557
456 while True:
558 while True:
457 self.dataOut = self.receiver.recv_pyobj()
559 dataOut = self.receiver.recv_pyobj()
458 # print '[Receiving] {} - {}'.format(self.dataOut.type,
560 dt = datetime.datetime.fromtimestamp(dataOut.utctime).date()
459 # self.dataOut.datatime.ctime())
561 sended = False
460
562 if dt not in self.dates:
461 self.update()
563 if self.data:
564 self.data.ended = True
565 self.send(self.data)
566 sended = True
567 self.data.setup()
568 self.dates.append(dt)
462
569
463 if self.dataOut.firstdata is True:
570 self.data.update(dataOut)
464 self.data['STARTED'] = True
465
571
466 if self.dataOut.finished is True:
572 if dataOut.finished is True:
467 self.send(self.data)
468 self.connections -= 1
573 self.connections -= 1
469 if self.connections == 0 and self.started:
574 if self.connections == 0 and dt in self.dates:
470 self.ended = True
575 self.data.ended = True
471 self.data['ENDED'] = True
472 self.send(self.data)
576 self.send(self.data)
473 self.setup()
577 self.data.setup()
474 self.started = False
475 else:
578 else:
476 if self.realtime:
579 if self.realtime:
477 self.send(self.data)
580 self.send(self.data)
478 self.sender_web.send_string(json.dumps(self.data_web))
581 # self.sender_web.send_string(self.data.jsonify())
479 else:
582 else:
583 if not sended:
480 self.sendData(self.send, self.data)
584 self.sendData(self.send, self.data)
481 self.started = True
482
585
483 self.data['STARTED'] = False
484 return
586 return
485
587
486 def sendToWeb(self):
588 def sendToWeb(self):
@@ -497,6 +599,6 class PlotterReceiver(ProcessingUnit, Process):
497 time.sleep(1)
599 time.sleep(1)
498 for kwargs in self.operationKwargs.values():
600 for kwargs in self.operationKwargs.values():
499 if 'plot' in kwargs:
601 if 'plot' in kwargs:
500 print '[Sending] Config data to web for {}'.format(kwargs['code'].upper())
602 log.success('[Sending] Config data to web for {}'.format(kwargs['code'].upper()))
501 sender_web_config.send_string(json.dumps(kwargs))
603 sender_web_config.send_string(json.dumps(kwargs))
502 self.isWebConfig = True No newline at end of file
604 self.isWebConfig = True
Show file before | Show file after | Hide comments
@@ -93,7 +93,11 def filterOffsets(offsets0, stdvLimit):
93
93
94 #---------------------- Setup ---------------------------
94 #---------------------- Setup ---------------------------
95
95
96 <<<<<<< HEAD
97 path = '/home/nanosat/Pictures/JASMET30_mp/201608/phase'
98 =======
96 path = '/home/jespinoza/Pictures/JASMET30/201608/phase'
99 path = '/home/jespinoza/Pictures/JASMET30/201608/phase'
100 >>>>>>> master
97 stdvLimit = 0.5
101 stdvLimit = 0.5
98
102
99 #---------------------- Script ---------------------------
103 #---------------------- Script ---------------------------
Show file before | Show file after | Hide comments
@@ -1,4 +1,4
1 """.
1 '''
2 SCHAINPY - LOG
2 SCHAINPY - LOG
3 Simple helper for log standarization
3 Simple helper for log standarization
4 Usage:
4 Usage:
@@ -13,33 +13,32 SCHAINPY - LOG
13 which will look like this:
13 which will look like this:
14 [NEVER GONNA] - give you up
14 [NEVER GONNA] - give you up
15 with color red as background and white as foreground.
15 with color red as background and white as foreground.
16 """
16 '''
17
17
18 import click
18 import click
19
19
20
20 def warning(message, tag='Warning'):
21 def warning(message):
21 click.echo(click.style('[{}] {}'.format(tag, message), fg='yellow'))
22 click.echo(click.style('[WARNING] - ' + message, fg='yellow'))
23 pass
22 pass
24
23
25
24
26 def error(message):
25 def error(message, tag='Error'):
27 click.echo(click.style('[ERROR] - ' + message, fg='red'))
26 click.echo(click.style('[{}] {}'.format(tag, message), fg='red'))
28 pass
27 pass
29
28
30
29
31 def success(message):
30 def success(message, tag='Info'):
32 click.echo(click.style(message, fg='green'))
31 click.echo(click.style('[{}] {}'.format(tag, message), fg='green'))
33 pass
32 pass
34
33
35
34
36 def log(message):
35 def log(message, tag='Info'):
37 click.echo('[LOG] - ' + message)
36 click.echo('[{}] {}'.format(tag, message))
38 pass
37 pass
39
38
40
39
41 def makelogger(topic, bg='reset', fg='reset'):
40 def makelogger(tag, bg='reset', fg='reset'):
42 def func(message):
41 def func(message):
43 click.echo(click.style('[{}] - '.format(topic.upper()) + message,
42 click.echo(click.style('[{}] {}'.format(tag.upper(), message),
44 bg=bg, fg=fg))
43 bg=bg, fg=fg))
45 return func
44 return func
Show file before | Show file after | Hide comments
@@ -1,14 +1,22
1 """.
1 '''
2
3 Created on Jul 16, 2014
2 Created on Jul 16, 2014
4
3
5 @author: Miguel Urco
4 @author: Miguel Urco
6 """
5 '''
7
6
8 import numpy
9 from setuptools import setup, Extension
7 from setuptools import setup, Extension
8 from setuptools.command.build_ext import build_ext as _build_ext
10 from schainpy import __version__
9 from schainpy import __version__
11
10
11 class build_ext(_build_ext):
12 def finalize_options(self):
13 _build_ext.finalize_options(self)
14 # Prevent numpy from thinking it is still in its setup process:
15 __builtins__.__NUMPY_SETUP__ = False
16 import numpy
17 self.include_dirs.append(numpy.get_include())
18
19
12 setup(name="schainpy",
20 setup(name="schainpy",
13 version=__version__,
21 version=__version__,
14 description="Python tools to read, write and process Jicamarca data",
22 description="Python tools to read, write and process Jicamarca data",
@@ -34,15 +42,19 setup(name="schainpy",
34 'schainpy.gui.figures': ['*.png', '*.jpg'],
42 'schainpy.gui.figures': ['*.png','*.jpg'],
35 },
43 },
36 include_package_data=False,
44 include_package_data=False,
45 scripts =['schainpy/gui/schainGUI'],
46 ext_modules=[
47 Extension("cSchain", ["schainpy/model/proc/extensions.c"]
48 )],
37 entry_points={
49 entry_points={
38 'console_scripts': [
50 'console_scripts': [
39 'schain = schaincli.cli:main',
51 'schain = schaincli.cli:main',
40 ],
52 ],
41 },
53 },
42 scripts=['schainpy/gui/schainGUI'],
54 cmdclass={'build_ext':build_ext},
43 ext_modules=[Extension("cSchain", ["schainpy/model/proc/extensions.c"], include_dirs=[numpy.get_include()])],
55 setup_requires=["numpy >= 1.11.2"],
44 install_requires=[
56 install_requires=[
45 "scipy >= 0.13.0",
57 "scipy >= 0.14.0",
46 "h5py >= 2.2.1",
58 "h5py >= 2.2.1",
47 "matplotlib >= 1.4.2",
59 "matplotlib >= 1.4.2",
48 "pyfits >= 3.4",
60 "pyfits >= 3.4",
@@ -55,4 +67,3 setup(name="schainpy",
55 "python-Levenshtein"
67 "python-Levenshtein"
56 ],
68 ],
57 )
69 )
58
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
NO CONTENT: file was removed
General Comments 0
You need to be logged in to leave comments. Login now