##// END OF EJS Templates
schain-jc
RSS
merge with v2.3
jespinoza -
r1063:ecbe1f26c35b merge
parent child
Show More
Side by Side Unified
Context file:
r1063:ecbe1f26c35b Loading diff...:
Show file before | Show file after | Hide comments
@@ -0,0 +1,109
1 ## CHANGELOG:
2
3 ### 2.3
4 * Added high order function `multiSchain` for multiprocessing scripts.
5 * Added two new Processing Units `PublishData` and `ReceiverData` for receiving and sending dataOut through multiple ways (tcp, ipc, inproc).
6 * Added a new graphics Processing Unit `PlotterReceiver`. It is decoupled from normal processing sequence with support for data generated by multiprocessing scripts.
7 * Added support for sending realtime graphic to web server.
8 * GUI command `schain` is now `schainGUI`.
9 * Added a CLI tool named `schain`.
10 * Scripts templates can be now generated with `schain generate`.
11 * Now it is possible to search Processing Units and Operations with `schain search [module]` to get the right name and its allowed parameters.
12 * `schain xml` to run xml scripts.
13 * Added suggestions when parameters are poorly written.
14 * `Controller.start()` now runs in a different process than the process calling it.
15 * Added `schainpy.utils.log` for log standarization.
16 * Running script on online mode no longer ignores date and hour. Issue #1109.
17 * Added support for receving voltage data directly from JARS (tcp, ipc).
18 * Updated README for MAC OS GUI installation.
19 * Setup now installs numpy.
20
21 ### 2.2.6
22 * Graphics generated by the GUI are now the same as generated by scripts. Issue #1074.
23 * Added support for C extensions.
24 * function `hildebrand_sehkon` optimized with a C wrapper.
25 * Numpy version updated.
26 * Migration to GIT.
27
28 ### 2.2.5:
29 * splitProfiles and combineProfiles modules were added to VoltageProc and Signal Chain GUI.
30 * nProfiles of USRP data (hdf5) is the number of profiles thera are in one second.
31 * jroPlotter works directly with data objects instead of dictionaries
32 * script "schain" was added to Signal Chain installer
33
34 ### 2.2.4.1:
35 * jroIO_usrp.py is update to read Sandra's data
36 * decimation in Spectra and RTI plots is always enabled.
37 * time* window option added to GUI
38
39 ### 2.2.4:
40 * jroproc_spectra_lags.py added to schainpy
41 * Bug fixed in schainGUI: ProcUnit was created with the same id in some cases.
42 * Bug fixed in jroHeaderIO: Header size validation.
43
44 ### 2.2.3.1:
45 * Filtering block by time has been added.
46 * Bug fixed plotting RTI, CoherenceMap and others using xmin and xmax parameters. The first day worked
47 properly but the next days did not.
48
49 ### 2.2.3:
50 * Bug fixed in GUI: Error getting(reading) Code value
51 * Bug fixed in GUI: Flip option always needs channelList field
52 * Bug fixed in jrodata: when one branch modified a value in "dataOut" (example: dataOut.code) this value
53 was modified for every branch (because this was a reference). It was modified in data.copy()
54 * Bug fixed in jroproc_voltage.profileSelector(): rangeList replaces to profileRangeList.
55
56 ### 2.2.2:
57 * VoltageProc: ProfileSelector, Reshape, Decoder with nTxs!=1 and getblock=True was tested
58 * Rawdata and testRawdata.py added to Signal Chain project
59
60 ### 2.2.1:
61 * Bugs fixed in GUI
62 * Views were improved in GUI
63 * Support to MST* ISR experiments
64 * Bug fixed getting noise using hyldebrant. (minimum number of points > 20%)
65 * handleError added to jroplotter.py
66
67 ### 2.2.0:
68 * GUI: use of external plotter
69 * Compatible with matplotlib 1.5.0
70
71 ### 2.1.5:
72 * serializer module added to Signal Chain
73 * jroplotter.py added to Signal Chain
74
75 ### 2.1.4.2:
76 * A new Plotter Class was added
77 * Project.start() does not accept filename as a parameter anymore
78
79 ### 2.1.4.1:
80 * Send notifications when an error different to ValueError is detected
81
82 ### 2.1.4:
83 * Sending error notifications to signal chain administrator
84 * Login to email server added
85
86 ### 2.1.3.3:
87 * Colored Button Icons were added to GUI
88
89 ### 2.1.3.2:
90 * GUI: user interaction enhanced
91 * controller_api.py: Safe access to ControllerThead
92
93 ### 2.1.3.1:
94 * GUI: every icon were resized
95 * jroproc_voltage.py: Print a message when "Read from code" option is selected and the code is not defined inside data file
96
97 ### 2.1.3:
98 * jroplot_heispectra.py: SpectraHeisScope was not showing the right channels
99 * jroproc_voltage.py: Bug fixed selecting profiles (self.nProfiles took a wrong value),
100 Bug fixed selecting heights by block (selecting profiles instead heights)
101 * jroproc_voltage.py: New feature added: decoding data by block using FFT.
102 * jroIO_heispectra.py: Bug fixed in FitsReader. Using local Fits instance instead schainpy.mode.data.jrodata.Fits.
103 * jroIO_heispectra.py: Channel index list does not exist.
104
105 ### 2.1.2:
106 * jroutils_ftp.py: Bug fixed, Any error sending file stopped the Server Thread
107 Server thread opens and closes remote server each time file list is sent
108 * jroplot_spectra.py: Noise path was not being created when noise data is saved.
109 * jroIO_base.py: startTime can be greater than endTime. Example: SpreadF [18:00 * 07:00] No newline at end of file
Show file before | Show file after | Hide comments
@@ -0,0 +1,95
1
2 import os, sys
3
4 from schainpy.controller import Project
5
6 controllerObj = Project()
7 controllerObj.setup(id = '002', name='script02', description="JASMET Meteor Detection")
8
9 #-------------------------------------- Setup -----------------------------------------
10 #Verificar estas variables
11
12 #Path para los archivos
13 # path = '/mnt/jars/2016_08/NOCHE'
14 # path = '/media/joscanoa/DATA_JASMET/JASMET/2016_08/DIA'
15 # path = '/media/joscanoa/DATA_JASMET/JASMET/2016_08/NOCHE'
16 path = '/home/nanosat/data/jasmet'
17
18 #Path para los graficos
19 pathfig = os.path.join(os.environ['HOME'],'Pictures/JASMET30/201608/graphics')
20
21 #Path para los archivos HDF5 de meteoros
22 pathfile = os.path.join(os.environ['HOME'],'Pictures/JASMET30/201608/meteor')
23
24 #Fechas para busqueda de archivos
25 startDate = '2010/08/29'
26 endDate = '2017/09/11'
27 #Horas para busqueda de archivos
28 startTime = '00:00:00'
29 endTime = '23:59:59'
30
31
32 #------------------------------ Voltage Reading Unit ----------------------------------
33
34 readUnitConfObj = controllerObj.addReadUnit(datatype='VoltageReader',
35 path=path,
36 startDate=startDate,
37 endDate=endDate,
38 startTime=startTime,
39 endTime=endTime,
40 online=0,
41 delay=30,
42 walk=1,
43 getblock=1,
44 blocktime=100)
45
46 opObj11 = readUnitConfObj.addOperation(name='printNumberOfBlock')
47
48 #-------------------------- Voltage Processing Unit ------------------------------------
49
50 procUnitConfObj0 = controllerObj.addProcUnit(datatype='VoltageProc', inputId=readUnitConfObj.getId())
51
52 opObj00 = procUnitConfObj0.addOperation(name='selectChannels')
53 opObj00.addParameter(name='channelList', value='0,1,2,3,4', format='intlist')
54
55 opObj01 = procUnitConfObj0.addOperation(name='setRadarFrequency')
56 opObj01.addParameter(name='frequency', value='30.e6', format='float')
57
58 # opObj01 = procUnitConfObj0.addOperation(name='interpolateHeights')
59 # opObj01.addParameter(name='topLim', value='73', format='int')
60 # opObj01.addParameter(name='botLim', value='71', format='int')
61
62 opObj02 = procUnitConfObj0.addOperation(name='Decoder', optype='other')
63
64 opObj03 = procUnitConfObj0.addOperation(name='CohInt', optype='other')
65 opObj03.addParameter(name='n', value='2', format='int')
66
67 procUnitConfObj1 = controllerObj.addProcUnit(datatype='SpectraProc', inputId=procUnitConfObj0.getId())
68 opObj11 = procUnitConfObj1.addOperation(name='RTIPlot', optype='other')
69 opObj11.addParameter(name='id', value='237', format='int')
70 opObj11.addParameter(name='xmin', value='9.0', format='float')
71 opObj11.addParameter(name='xmax', value='16.0', format='float')
72 opObj11.addParameter(name='zmin', value='15.0', format='float')
73 opObj11.addParameter(name='zmax', value='50.0', format='float')
74
75 #--------------------------- Parameters Processing Unit ------------------------------------
76
77 procUnitConfObj1 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=procUnitConfObj0.getId())
78 #
79 opObj10 = procUnitConfObj1.addOperation(name='SMDetection', optype='other')
80 opObj10.addParameter(name='azimuth', value='45', format='float')
81 opObj10.addParameter(name='hmin', value='60', format='float')
82 opObj10.addParameter(name='hmax', value='120', format='float')
83
84 opObj12 = procUnitConfObj1.addOperation(name='ParamWriter', optype='other')
85 opObj12.addParameter(name='path', value=pathfile)
86 opObj12.addParameter(name='blocksPerFile', value='1000', format='int')
87 opObj12.addParameter(name='metadataList',value='type,heightList,paramInterval,timeZone',format='list')
88 opObj12.addParameter(name='dataList',value='data_param,utctime',format='list')
89 opObj12.addParameter(name='mode',value='2',format='int')
90
91 #--------------------------------------------------------------------------------------------------
92
93 controllerObj.start()
94
95
Show file before | Show file after | Hide comments
@@ -0,0 +1,68
1 import os, sys
2
3 path = os.path.split(os.getcwd())[0]
4 path = os.path.split(path)[0]
5
6 sys.path.insert(0, path)
7
8 from schainpy.controller import Project
9
10 controllerObj = Project()
11 controllerObj.setup(id = '004', name='script04', description="JASMET Phase Calibration")
12
13 #-------------------------------------- Setup -----------------------------------------
14 #Verificar estas variables
15
16 #Path donde estan los archivos HDF5 de meteoros
17 path = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/meteor')
18
19 #Path para los graficos
20 pathfig = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/graphics')
21
22 #Path donde se almacenaran las fases calculadas
23 pathfile = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/phase')
24
25 #Fechas para busqueda de archivos
26 startDate = '2016/08/29'
27 endDate = '2016/09/11'
28 #Horas para busqueda de archivos
29 startTime = '00:00:00'
30 endTime = '23:59:59'
31
32 #------------------------------------------------------------------------------------------------
33 readUnitConfObj = controllerObj.addReadUnit(datatype='ParamReader',
34 path=path,
35 startDate=startDate,
36 endDate=endDate,
37 startTime=startTime,
38 endTime=endTime,
39 walk=1)
40
41 #--------------------------------------------------------------------------------------------------
42
43 procUnitConfObj1 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=readUnitConfObj.getId())
44 # #
45 opObj31 = procUnitConfObj1.addOperation(name='SMPhaseCalibration', optype='other')
46 opObj31.addParameter(name='nHours', value='1', format='float')
47 opObj31.addParameter(name='hmin', value='60', format='float')
48 opObj31.addParameter(name='hmax', value='120', format='float')
49 # opObj31.addParameter(name='channelPositions', value='(2.5,0),(0,2.5),(0,0),(0,4.5),(-2,0)', format='pairslist')
50
51 opObj32 = procUnitConfObj1.addOperation(name='PhasePlot', optype='other')
52 opObj32.addParameter(name='id', value='201', format='int')
53 opObj32.addParameter(name='wintitle', value='PhaseCalibration', format='str')
54 opObj32.addParameter(name='save', value='1', format='bool')
55 opObj32.addParameter(name='xmin', value='0', format='float')
56 opObj32.addParameter(name='xmax', value='24', format='float')
57 opObj32.addParameter(name='ymin', value='-180', format='float')
58 opObj32.addParameter(name='ymax', value='180', format='float')
59 opObj32.addParameter(name='figpath', value=pathfig, format='str')
60 # #
61 opObj33 = procUnitConfObj1.addOperation(name='ParamWriter', optype='other')
62 opObj33.addParameter(name='path', value=pathfile)
63 opObj33.addParameter(name='blocksPerFile', value='1000', format='int')
64 opObj33.addParameter(name='metadataList',value='type,outputInterval,timeZone',format='list')
65 opObj33.addParameter(name='dataList',value='data_output,utctime',format='list')
66 # # opObj25.addParameter(name='mode',value='1,0,0',format='intlist')
67
68 controllerObj.start() No newline at end of file
Show file before | Show file after | Hide comments
@@ -0,0 +1,85
1 import os, sys
2
3 path = os.path.split(os.getcwd())[0]
4 path = os.path.split(path)[0]
5
6 sys.path.insert(0, path)
7
8 from schainpy.controller import Project
9
10 controllerObj = Project()
11 controllerObj.setup(id = '005', name='script05', description="JASMET Wind Estimation")
12
13 #-------------------------------------- Setup -----------------------------------------
14 #Verificar estas variables
15
16 #Path donde estan los archivos HDF5 de meteoros
17 path = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/meteor')
18
19 #Path para los graficos
20 pathfig = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/graphics')
21
22 #Path donde se almacenaran las estimaciones de vientos
23 pathfile = os.path.join(os.environ['HOME'],'Pictures/JASMET30_mp/201608/phase')
24
25 #Fechas para busqueda de archivos
26 startDate = '2016/08/29'
27 endDate = '2016/09/11'
28 #Horas para busqueda de archivos
29 startTime = '00:00:00'
30 endTime = '23:59:59'
31
32 #Offsets optimos obtenidos con OptimumOffset.py
33 phaseOffsets = '-2.84, -1.77, 11.94, 9.71'
34 phaseOffsets = '-5.86, -0.93, -7.29, 23.35'
35 #------------------------------------------------------------------------------------------------
36 readUnitConfObj = controllerObj.addReadUnit(datatype='ParamReader',
37 path=path,
38 startDate=startDate,
39 endDate=endDate,
40 startTime=startTime,
41 endTime=endTime,
42 walk=1)
43 #--------------------------------------------------------------------------------------------------
44
45 procUnitConfObj1 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=readUnitConfObj.getId())
46 opObj10 = procUnitConfObj1.addOperation(name='CorrectSMPhases',optype='other')
47 opObj10.addParameter(name='phaseOffsets', value=phaseOffsets, format='floatlist')
48
49 opObj13 = procUnitConfObj1.addOperation(name='SkyMapPlot', optype='other')
50 opObj13.addParameter(name='id', value='1', format='int')
51 opObj13.addParameter(name='wintitle', value='Sky Map', format='str')
52 opObj13.addParameter(name='save', value='1', format='bool')
53 opObj13.addParameter(name='figpath', value=pathfig, format='str')
54 opObj13.addParameter(name='ftp', value='1', format='int')
55 opObj13.addParameter(name='exp_code', value='15', format='int')
56 opObj13.addParameter(name='sub_exp_code', value='1', format='int')
57 opObj13.addParameter(name='tmin', value='0', format='int')
58 opObj13.addParameter(name='tmax', value='24', format='int')
59
60 opObj22 = procUnitConfObj1.addOperation(name='WindProfiler', optype='other')
61 opObj22.addParameter(name='technique', value='Meteors', format='str')
62 opObj22.addParameter(name='nHours', value='1', format='float')
63 opObj22.addParameter(name='hmin', value='70', format='float')
64 opObj22.addParameter(name='hmax', value='120', format='float')
65
66 opObj23 = procUnitConfObj1.addOperation(name='WindProfilerPlot', optype='other')
67 opObj23.addParameter(name='id', value='2', format='int')
68 opObj23.addParameter(name='wintitle', value='Wind Profiler', format='str')
69 opObj23.addParameter(name='save', value='1', format='bool')
70 opObj23.addParameter(name='figpath', value = pathfig, format='str')
71 opObj23.addParameter(name='zmin', value='-140', format='int')
72 opObj23.addParameter(name='zmax', value='140', format='int')
73 opObj23.addParameter(name='xmin', value='0', format='float')
74 opObj23.addParameter(name='xmax', value='24', format='float')
75 opObj23.addParameter(name='ymin', value='70', format='float')
76 opObj23.addParameter(name='ymax', value='110', format='float')
77
78 opObj33 = procUnitConfObj1.addOperation(name='ParamWriter', optype='other')
79 opObj33.addParameter(name='path', value=pathfile)
80 opObj33.addParameter(name='blocksPerFile', value='1000', format='int')
81 opObj33.addParameter(name='metadataList',value='type,outputInterval,timeZone',format='list')
82 opObj33.addParameter(name='dataList',value='data_output,utctime',format='list')
83 #--------------------------------------------------------------------------------------------------
84
85 controllerObj.start() No newline at end of file
Show file before | Show file after | Hide comments
@@ -0,0 +1,97
1 import argparse
2
3 from schainpy.controller import Project, multiSchain
4
5 desc = "HF_EXAMPLE"
6
7 def fiber(cursor, skip, q, dt):
8
9 controllerObj = Project()
10
11 controllerObj.setup(id='191', name='test01', description=desc)
12
13 readUnitConfObj = controllerObj.addReadUnit(datatype='SpectraReader',
14 path='/home/nanosat/data/sp1_f0',
15 startDate=dt,
16 endDate=dt,
17 startTime="00:00:00",
18 endTime="23:59:59",
19 online=0,
20 #set=1426485881,
21 walk=1,
22 queue=q,
23 cursor=cursor,
24 skip=skip,
25 verbose=1
26 #timezone=-5*3600
27 )
28
29 # #opObj11 = readUnitConfObj.addOperation(name='printNumberOfBlock')
30 #
31 procUnitConfObj2 = controllerObj.addProcUnit(datatype='Spectra', inputId=readUnitConfObj.getId())
32 # procUnitConfObj2.addParameter(name='nipp', value='5', format='int')
33
34 procUnitConfObj3 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=readUnitConfObj.getId())
35 opObj11 = procUnitConfObj3.addOperation(name='SpectralMoments', optype='other')
36
37 #
38 # opObj11 = procUnitConfObj1.addOperation(name='SpectraPlot', optype='other')
39 # opObj11.addParameter(name='id', value='1000', format='int')
40 # opObj11.addParameter(name='wintitle', value='HF_Jicamarca_Spc', format='str')
41 # opObj11.addParameter(name='channelList', value='0', format='intlist')
42 # opObj11.addParameter(name='zmin', value='-120', format='float')
43 # opObj11.addParameter(name='zmax', value='-70', format='float')
44 # opObj11.addParameter(name='save', value='1', format='int')
45 # opObj11.addParameter(name='figpath', value=figpath, format='str')
46
47 # opObj11 = procUnitConfObj3.addOperation(name='Parameters1Plot', optype='other')
48 # opObj11.addParameter(name='channelList', value='0', format='intList')
49
50 # opObj11.addParameter(name='id', value='2000', format='int')
51 # # opObj11.addParameter(name='colormap', value='0', format='bool')
52 # opObj11.addParameter(name='onlySNR', value='1', format='bool')
53 # opObj11.addParameter(name='DOP', value='0', format='bool')
54 # # opObj11.addParameter(name='showSNR', value='1', format='bool')
55 # # opObj11.addParameter(name='SNRthresh', value='0', format='int')
56 # opObj11.addParameter(name='SNRmin', value='-10', format='int')
57 # opObj11.addParameter(name='SNRmax', value='30', format='int')
58
59 # opObj11.addParameter(name='showSNR', value='1', format='int')
60 # # opObj11.addParameter(name='channelList', value='0', format='intlist')
61 # # opObj11.addParameter(name='xmin', value='0', format='float')
62 # opObj11.addParameter(name='xmin', value='0', format='float')
63 # opObj11.addParameter(name='xmax', value='24', format='float')
64
65 # opObj11.addParameter(name='zmin', value='-110', format='float')
66 # opObj11.addParameter(name='zmax', value='-70', format='float')
67 # opObj11.addParameter(name='save', value='0', format='int')
68 # # opObj11.addParameter(name='figpath', value='/tmp/', format='str')
69 #
70 opObj12 = procUnitConfObj3.addOperation(name='PublishData', optype='other')
71 opObj12.addParameter(name='zeromq', value=1, format='int')
72 opObj12.addParameter(name='verbose', value=0, format='bool')
73
74
75 # opObj13 = procUnitConfObj3.addOperation(name='PublishData', optype='other')
76 # opObj13.addParameter(name='zeromq', value=1, format='int')
77 # opObj13.addParameter(name='server', value="juanca", format='str')
78
79 opObj12.addParameter(name='delay', value=0, format='int')
80
81
82 # print "Escribiendo el archivo XML"
83 # controllerObj.writeXml(filename)
84 # print "Leyendo el archivo XML"
85 # controllerObj.readXml(filename)
86
87
88 # timeit.timeit('controllerObj.run()', number=2)
89
90 controllerObj.start()
91
92
93 if __name__ == '__main__':
94 parser = argparse.ArgumentParser(description='Set number of parallel processes')
95 parser.add_argument('--nProcess', default=1, type=int)
96 args = parser.parse_args()
97 multiSchain(fiber, nProcess=args.nProcess, startDate='2017/01/26', endDate='2017/01/26')
Show file before | Show file after | Hide comments
@@ -0,0 +1,94
1 import argparse
2
3 from schainpy.controller import Project, multiSchain
4
5 desc = "HF_EXAMPLE"
6
7 def fiber(cursor, skip, q, dt):
8
9 controllerObj = Project()
10
11 controllerObj.setup(id='191', name='test01', description=desc)
12
13 readUnitConfObj = controllerObj.addReadUnit(datatype='SpectraReader',
14 path='/home/nanosat/data/julia',
15 startDate=dt,
16 endDate=dt,
17 startTime="00:00:00",
18 endTime="23:59:59",
19 online=0,
20 #set=1426485881,
21 delay=10,
22 walk=1,
23 queue=q,
24 cursor=cursor,
25 skip=skip,
26 #timezone=-5*3600
27 )
28
29 # #opObj11 = readUnitConfObj.addOperation(name='printNumberOfBlock')
30 #
31 procUnitConfObj2 = controllerObj.addProcUnit(datatype='Spectra', inputId=readUnitConfObj.getId())
32 # procUnitConfObj2.addParameter(name='nipp', value='5', format='int')
33
34 # procUnitConfObj3 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=readUnitConfObj.getId())
35 # opObj11 = procUnitConfObj3.addOperation(name='SpectralMoments', optype='other')
36
37 #
38 opObj11 = procUnitConfObj2.addOperation(name='RTIPlot', optype='other')
39 opObj11.addParameter(name='id', value='1000', format='int')
40 opObj11.addParameter(name='wintitle', value='HF_Jicamarca_Spc', format='str')
41 opObj11.addParameter(name='xmin', value='0', format='int')
42 opObj11.addParameter(name='xmax', value='24', format='int')
43
44 # opObj11 = procUnitConfObj3.addOperation(name='Parameters1Plot', optype='other')
45 # opObj11.addParameter(name='channelList', value='0', format='intList')
46 #
47 # opObj11.addParameter(name='id', value='2000', format='int')
48 # # opObj11.addParameter(name='colormap', value='0', format='bool')
49 # opObj11.addParameter(name='onlySNR', value='1', format='bool')
50 # opObj11.addParameter(name='DOP', value='0', format='bool')
51 # opObj11.addParameter(name='showSNR', value='1', format='bool')
52 # opObj11.addParameter(name='SNRthresh', value='0', format='int')
53 # opObj11.addParameter(name='SNRmin', value='-10', format='int')
54 # opObj11.addParameter(name='SNRmax', value='30', format='int')
55
56 # opObj11.addParameter(name='showSNR', value='1', format='int')
57 # # opObj11.addParameter(name='channelList', value='0', format='intlist')
58 # # opObj11.addParameter(name='xmin', value='0', format='float')
59 # opObj11.addParameter(name='xmin', value='0', format='float')
60 # opObj11.addParameter(name='xmax', value='24', format='float')
61
62 # opObj11.addParameter(name='zmin', value='-110', format='float')
63 # opObj11.addParameter(name='zmax', value='-70', format='float')
64 # opObj11.addParameter(name='save', value='0', format='int')
65 # # opObj11.addParameter(name='figpath', value='/tmp/', format='str')
66 #
67 # opObj12 = procUnitConfObj2.addOperation(name='PublishData', optype='other')
68 # opObj12.addParameter(name='zeromq', value=1, format='int')
69 # opObj12.addParameter(name='server', value='tcp://10.10.10.82:7000', format='str')
70
71
72 # opObj13 = procUnitConfObj3.addOperation(name='PublishData', optype='other')
73 # opObj13.addParameter(name='zeromq', value=1, format='int')
74 # opObj13.addParameter(name='server', value="juanca", format='str')
75
76 # opObj12.addParameter(name='delay', value=1, format='int')
77
78
79 # print "Escribiendo el archivo XML"
80 # controllerObj.writeXml(filename)
81 # print "Leyendo el archivo XML"
82 # controllerObj.readXml(filename)
83
84
85 # timeit.timeit('controllerObj.run()', number=2)
86
87 controllerObj.start()
88
89
90 if __name__ == '__main__':
91 parser = argparse.ArgumentParser(description='Set number of parallel processes')
92 parser.add_argument('--nProcess', default=1, type=int)
93 args = parser.parse_args()
94 multiSchain(fiber, nProcess=args.nProcess, startDate='2016/08/19', endDate='2016/08/19')
Show file before | Show file after | Hide comments
@@ -0,0 +1,1
1 <Project description="Claire" id="002" name="script02"><ReadUnit datatype="VoltageReader" id="21" inputId="0" name="VoltageReader"><Operation id="211" name="run" priority="1" type="self"><Parameter format="str" id="2111" name="datatype" value="VoltageReader" /><Parameter format="str" id="2112" name="path" value="/media/nanosat/0BDE10E00BDE10E0/CLAIRE" /><Parameter format="date" id="2113" name="startDate" value="2017/07/26" /><Parameter format="date" id="2114" name="endDate" value="2017/07/26" /><Parameter format="time" id="2115" name="startTime" value="15:00:00" /><Parameter format="time" id="2116" name="endTime" value="16:00:00" /><Parameter format="int" id="2118" name="delay" value="30" /><Parameter format="int" id="2119" name="walk" value="1" /><Parameter format="int" id="2120" name="online" value="0" /></Operation><Operation id="212" name="printNumberOfBlock" priority="2" type="self" /></ReadUnit><ProcUnit datatype="VoltageProc" id="22" inputId="21" name="VoltageProc"><Operation id="221" name="run" priority="1" type="self" /></ProcUnit><ProcUnit datatype="SpectraProc" id="23" inputId="22" name="SpectraProc"><Operation id="231" name="run" priority="1" type="self"><Parameter format="int" id="2311" name="nFFTPoints" value="128" /><Parameter format="int" id="2312" name="nProfiles" value="128" /><Parameter format="pairslist" id="2313" name="pairsList" value="(0,1),(0,2),(1,2)" /></Operation><Operation id="232" name="setRadarFrequency" priority="2" type="self"><Parameter format="float" id="2321" name="frequency" value="445000000.0" /></Operation><Operation id="233" name="IncohInt" priority="3" type="other"><Parameter format="float" id="2331" name="timeInterval" value="128" /></Operation><Operation id="234" name="removeDC" priority="4" type="self"><Parameter format="int" id="2341" name="mode" value="2" /></Operation><Operation id="235" name="RTIPlot" priority="5" type="other"><Parameter format="int" id="2351" name="id" value="237" /><Parameter format="float" id="2352" name="xmin" value="9.0" /><Parameter format="float" id="2353" name="xmax" value="16.0" /><Parameter format="float" id="2354" name="zmin" value="15.0" /><Parameter format="float" id="2355" name="zmax" value="50.0" /></Operation></ProcUnit></Project> No newline at end of file
Show file before | Show file after | Hide comments
@@ -0,0 +1,81
1 import schainpy
2 from schainpy.model import Operation, ProcessingUnit
3 from importlib import import_module
4 from pydoc import locate
5
6 def clean_modules(module):
7 noEndsUnder = [x for x in module if not x.endswith('__')]
8 noStartUnder = [x for x in noEndsUnder if not x.startswith('__')]
9 noFullUpper = [x for x in noStartUnder if not x.isupper()]
10 return noFullUpper
11
12 def check_module(possible, instance):
13 def check(x):
14 try:
15 instancia = locate('schainpy.model.{}'.format(x))
16 return isinstance(instancia(), instance)
17 except Exception as e:
18 return False
19 clean = clean_modules(possible)
20 return [x for x in clean if check(x)]
21
22
23 def getProcs():
24 module = dir(import_module('schainpy.model'))
25 procs = check_module(module, ProcessingUnit)
26 try:
27 procs.remove('ProcessingUnit')
28 except Exception as e:
29 pass
30 return procs
31
32 def getOperations():
33 module = dir(import_module('schainpy.model'))
34 noProcs = [x for x in module if not x.endswith('Proc')]
35 operations = check_module(noProcs, Operation)
36 try:
37 operations.remove('Operation')
38 except Exception as e:
39 pass
40 return operations
41
42 def getArgs(op):
43 module = locate('schainpy.model.{}'.format(op))
44 args = module().getAllowedArgs()
45 try:
46 args.remove('self')
47 except Exception as e:
48 pass
49 try:
50 args.remove('dataOut')
51 except Exception as e:
52 pass
53 return args
54
55 def getAll():
56 allModules = dir(import_module('schainpy.model'))
57 modules = check_module(allModules, Operation)
58 modules.extend(check_module(allModules, ProcessingUnit))
59 return modules
60
61 def formatArgs(op):
62 args = getArgs(op)
63
64 argsAsKey = ["\t'{}'".format(x) for x in args]
65 argsFormatted = ": 'string',\n".join(argsAsKey)
66
67 print op
68 print "parameters = { \n" + argsFormatted + ": 'string',\n }"
69 print '\n'
70
71
72 if __name__ == "__main__":
73 getAll()
74 [formatArgs(x) for x in getAll()]
75
76 '''
77 parameters = {
78 'id': ,
79 'wintitle': ,
80 }
81 ''' No newline at end of file
Show file before | Show file after | Hide comments
@@ -105,9 +105,11 ENV/
105 105
106 106 .vscode
107 107
108 schainpy/scripts/
109 schaingui/node_modules/
108 110 .svn/
109 111 *.png
110 112 *.pyc
111 schainpy/scripts
112
113 schaingui/node_modules
113 *.xml
114 *.log
115 trash No newline at end of file
Show file before | Show file after | Hide comments
@@ -8,24 +8,39 Signal Chain (SCh) is a radar data processing library developed using [Python](w
8 8
9 9 Install system dependencies, clone the latest version from [git](http://jro-dev.igp.gob.pe/rhodecode/schain/) and install it as a normal python package.
10 10
11 ### Linux based system
11 12 ```
12 13 $ sudo apt-get install python-pip python-dev gfortran libpng-dev freetype* libblas-dev liblapack-dev libatlas-base-dev python-qt4 python-tk libssl-dev libhdf5-dev
13 14 $ sudo pip install numpy
14 15 $ git clone http://jro-dev.igp.gob.pe/rhodecode/schain/
15 16 $ cd schain
16 17 $ sudo pip install ./
17 ```
18
19 **Its recommended to install schain in a virtual environment**
20 18
21 19 ```
20 **It is recommended to install schain in a virtual environment**
21 ```
22 22 $ sudo pip install virtualenv
23 23 $ virtualenv /path/to/virtual --system-site-packages
24 24 $ source /path/to/virtual/bin/activate
25 25 (virtual) $ cd schain
26 26 (virtual) $ pip install ./
27
27 28 ```
28 29
30 ### MAC Os
31 ```
32 $ brew install cartr/qt4/pyqt
33 $ git clone http://jro-dev.igp.gob.pe/rhodecode/schain/
34 $ cd schain
35 $ pip install ./
36 ```
37
38 if ```pip install ./``` does not work, install a proper python enviroment, and repeat the steps.
39 ```
40 $ brew install python
41 ```
42
43
29 44 ## First Script
30 45
31 46 Read Spectra data (.pdata) - remove dc - plot spectra & RTI
Show file before | Show file after | Hide comments
@@ -8,12 +8,13 save_stdout = sys.stdout
8 8 sys.stdout = open('trash', 'w')
9 9 from multiprocessing import cpu_count
10 10 from schaincli import templates
11 from schainpy import controller_api
11 from schainpy.controller import Project
12 12 from schainpy.model import Operation, ProcessingUnit
13 13 from schainpy.utils import log
14 14 from importlib import import_module
15 15 from pydoc import locate
16 16 from fuzzywuzzy import process
17 from schainpy.utils import paramsFinder
17 18 sys.stdout = save_stdout
18 19
19 20
@@ -75,47 +76,25 def search(nextcommand):
75 76 if nextcommand is None:
76 77 log.error('There is no Operation/ProcessingUnit to search')
77 78 elif nextcommand == 'procs':
78 module = dir(import_module('schainpy.model'))
79 procs = check_module(module, ProcessingUnit)
80 try:
81 procs.remove('ProcessingUnit')
82 except Exception as e:
83 pass
79 procs = paramsFinder.getProcs()
84 80 log.success('Current ProcessingUnits are:\n\033[1m{}\033[0m'.format('\n'.join(procs)))
85 81
86 82 elif nextcommand == 'operations':
87 module = dir(import_module('schainpy.model'))
88 noProcs = [x for x in module if not x.endswith('Proc')]
89 operations = check_module(noProcs, Operation)
90 try:
91 operations.remove('Operation')
92 except Exception as e:
93 pass
83 operations = paramsFinder.getOperations()
94 84 log.success('Current Operations are:\n\033[1m{}\033[0m'.format('\n'.join(operations)))
95 85 else:
96 86 try:
97 module = locate('schainpy.model.{}'.format(nextcommand))
98 args = module().getAllowedArgs()
87 args = paramsFinder.getArgs(nextcommand)
99 88 log.warning('Use this feature with caution. It may not return all the allowed arguments')
100 try:
101 args.remove('self')
102 except Exception as e:
103 pass
104 try:
105 args.remove('dataOut')
106 except Exception as e:
107 pass
108 89 if len(args) == 0:
109 90 log.success('{} has no arguments'.format(nextcommand))
110 91 else:
111 92 log.success('Showing arguments of {} are:\n\033[1m{}\033[0m'.format(nextcommand, '\n'.join(args)))
112 93 except Exception as e:
113 94 log.error('Module {} does not exists'.format(nextcommand))
114 allModules = dir(import_module('schainpy.model'))
115 module = check_module(allModules, Operation)
116 module.extend(check_module(allModules, ProcessingUnit))
117 similar = process.extractOne(nextcommand, module)[0]
118 log.success('Searching {} instead'.format(similar))
95 allModules = paramsFinder.getAll()
96 similar = process.extractOne(nextcommand, allModules)[0]
97 log.success('Showing {} instead'.format(similar))
119 98 search(similar)
120 99
121 100
@@ -171,18 +150,8 def test():
171 150
172 151
173 152 def runFromXML(filename):
174 controller = controller_api.ControllerThread()
153 controller = Project()
175 154 if not controller.readXml(filename):
176 155 return
177
178 plotterObj = controller.useExternalPlotter()
179
180 156 controller.start()
181 plotterObj.start()
182
183 cliLogger("Finishing all processes")
184
185 controller.join(5)
186
187 cliLogger("End of script")
188 157 return
Show file before | Show file after | Hide comments
@@ -4,4 +4,4 Created on Feb 7, 2012
4 4 @author $Author$
5 5 @version $Id$
6 6 '''
7 __version__ = "2.3" No newline at end of file
7 __version__ = "2.3"
Show file before | Show file after | Hide comments
@@ -9,53 +9,53 import datetime
9 9 import traceback
10 10 import math
11 11 import time
12 from multiprocessing import Process, Queue, cpu_count
13
14 import schainpy
15 import schainpy.admin
12 from multiprocessing import Process, cpu_count
16 13
17 14 from xml.etree.ElementTree import ElementTree, Element, SubElement, tostring
18 15 from xml.dom import minidom
19 16
17 import schainpy
18 import schainpy.admin
20 19 from schainpy.model import *
21 from time import sleep
22
23 def prettify(elem):
24 """Return a pretty-printed XML string for the Element.
25 """
26 rough_string = tostring(elem, 'utf-8')
27 reparsed = minidom.parseString(rough_string)
28 return reparsed.toprettyxml(indent=" ")
29
30 def multiSchain(child, nProcess=cpu_count(), startDate=None, endDate=None, by_day=False):
31 skip = 0
32 cursor = 0
33 nFiles = None
34 processes = []
35 dt1 = datetime.datetime.strptime(startDate, '%Y/%m/%d')
36 dt2 = datetime.datetime.strptime(endDate, '%Y/%m/%d')
20 from schainpy.utils import log
21
22 DTYPES = {
23 'Voltage': '.r',
24 'Spectra': '.pdata'
25 }
26
27 def MPProject(project, n=cpu_count()):
28 '''
29 Project wrapper to run schain in n processes
30 '''
31
32 rconf = project.getReadUnitObj()
33 op = rconf.getOperationObj('run')
34 dt1 = op.getParameterValue('startDate')
35 dt2 = op.getParameterValue('endDate')
37 36 days = (dt2 - dt1).days
38
37
39 38 for day in range(days+1):
40 39 skip = 0
41 40 cursor = 0
42 q = Queue()
43 41 processes = []
44 dt = (dt1 + datetime.timedelta(day)).strftime('%Y/%m/%d')
45 firstProcess = Process(target=child, args=(cursor, skip, q, dt))
46 firstProcess.start()
47 if by_day:
48 continue
49 nFiles = q.get()
50 if nFiles==0:
42 dt = dt1 + datetime.timedelta(day)
43 dt_str = dt.strftime('%Y/%m/%d')
44 reader = JRODataReader()
45 paths, files = reader.searchFilesOffLine(path=rconf.path,
46 startDate=dt,
47 endDate=dt,
48 ext=DTYPES[rconf.datatype])
49 nFiles = len(files)
50 if nFiles == 0:
51 51 continue
52 firstProcess.terminate()
53 skip = int(math.ceil(nFiles/nProcess))
54 while True:
55 processes.append(Process(target=child, args=(cursor, skip, q, dt)))
56 processes[cursor].start()
57 if nFiles < cursor*skip:
58 break
52 skip = int(math.ceil(nFiles/n))
53 while nFiles > cursor*skip:
54 rconf.update(startDate=dt_str, endDate=dt_str, cursor=cursor,
55 skip=skip)
56 p = project.clone()
57 p.start()
58 processes.append(p)
59 59 cursor += 1
60 60
61 61 def beforeExit(exctype, value, trace):
@@ -72,7 +72,6 def multiSchain(child, nProcess=cpu_count(), startDate=None, endDate=None, by_da
72 72
73 73 time.sleep(3)
74 74
75
76 75 class ParameterConf():
77 76
78 77 id = None
@@ -109,7 +108,7 class ParameterConf():
109 108 return self.__formated_value
110 109
111 110 if value == '':
112 raise ValueError, "%s: This parameter value is empty" %self.name
111 raise ValueError, '%s: This parameter value is empty' %self.name
113 112
114 113 if format == 'list':
115 114 strList = value.split(',')
@@ -119,10 +118,10 class ParameterConf():
119 118 return self.__formated_value
120 119
121 120 if format == 'intlist':
122 """
121 '''
123 122 Example:
124 123 value = (0,1,2)
125 """
124 '''
126 125
127 126 new_value = ast.literal_eval(value)
128 127
@@ -134,10 +133,10 class ParameterConf():
134 133 return self.__formated_value
135 134
136 135 if format == 'floatlist':
137 """
136 '''
138 137 Example:
139 138 value = (0.5, 1.4, 2.7)
140 """
139 '''
141 140
142 141 new_value = ast.literal_eval(value)
143 142
@@ -167,38 +166,38 class ParameterConf():
167 166 return self.__formated_value
168 167
169 168 if format == 'pairslist':
170 """
169 '''
171 170 Example:
172 171 value = (0,1),(1,2)
173 """
172 '''
174 173
175 174 new_value = ast.literal_eval(value)
176 175
177 176 if type(new_value) not in (tuple, list):
178 raise ValueError, "%s has to be a tuple or list of pairs" %value
177 raise ValueError, '%s has to be a tuple or list of pairs' %value
179 178
180 179 if type(new_value[0]) not in (tuple, list):
181 180 if len(new_value) != 2:
182 raise ValueError, "%s has to be a tuple or list of pairs" %value
181 raise ValueError, '%s has to be a tuple or list of pairs' %value
183 182 new_value = [new_value]
184 183
185 184 for thisPair in new_value:
186 185 if len(thisPair) != 2:
187 raise ValueError, "%s has to be a tuple or list of pairs" %value
186 raise ValueError, '%s has to be a tuple or list of pairs' %value
188 187
189 188 self.__formated_value = new_value
190 189
191 190 return self.__formated_value
192 191
193 192 if format == 'multilist':
194 """
193 '''
195 194 Example:
196 195 value = (0,1,2),(3,4,5)
197 """
196 '''
198 197 multiList = ast.literal_eval(value)
199 198
200 199 if type(multiList[0]) == int:
201 multiList = ast.literal_eval("(" + value + ")")
200 multiList = ast.literal_eval('(' + value + ')')
202 201
203 202 self.__formated_value = multiList
204 203
@@ -260,7 +259,7 class ParameterConf():
260 259
261 260 def printattr(self):
262 261
263 print "Parameter[%s]: name = %s, value = %s, format = %s" %(self.id, self.name, self.value, self.format)
262 print 'Parameter[%s]: name = %s, value = %s, format = %s' %(self.id, self.name, self.value, self.format)
264 263
265 264 class OperationConf():
266 265
@@ -368,7 +367,9 class OperationConf():
368 367 self.parmConfObjList = []
369 368
370 369 def addParameter(self, name, value, format='str'):
371
370
371 if value is None:
372 return None
372 373 id = self.__getNewId()
373 374
374 375 parmConfObj = ParameterConf()
@@ -428,7 +429,7 class OperationConf():
428 429
429 430 def printattr(self):
430 431
431 print "%s[%s]: name = %s, type = %s, priority = %s" %(self.ELEMENTNAME,
432 print '%s[%s]: name = %s, type = %s, priority = %s' %(self.ELEMENTNAME,
432 433 self.id,
433 434 self.name,
434 435 self.type,
@@ -441,12 +442,11 class OperationConf():
441 442
442 443
443 444 if self.type == 'self':
444 raise ValueError, "This operation type cannot be created"
445 raise ValueError, 'This operation type cannot be created'
445 446
446 if self.type == 'plotter':
447 #Plotter(plotter_name)
447 if self.type == 'plotter':
448 448 if not plotter_queue:
449 raise ValueError, "plotter_queue is not defined. Use:\nmyProject = Project()\nmyProject.setPlotterQueue(plotter_queue)"
449 raise ValueError, 'plotter_queue is not defined. Use:\nmyProject = Project()\nmyProject.setPlotterQueue(plotter_queue)'
450 450
451 451 opObj = Plotter(self.name, plotter_queue)
452 452
@@ -561,7 +561,7 class ProcUnitConf():
561 561
562 562 #Compatible with old signal chain version
563 563 if datatype==None and name==None:
564 raise ValueError, "datatype or name should be defined"
564 raise ValueError, 'datatype or name should be defined'
565 565
566 566 if name==None:
567 567 if 'Proc' in datatype:
@@ -592,7 +592,7 class ProcUnitConf():
592 592
593 593 def addParameter(self, **kwargs):
594 594 '''
595 Add parameters to "run" operation
595 Add parameters to 'run' operation
596 596 '''
597 597 opObj = self.opConfObjList[0]
598 598
@@ -630,11 +630,11 class ProcUnitConf():
630 630 self.datatype = upElement.get('datatype')
631 631 self.inputId = upElement.get('inputId')
632 632
633 if self.ELEMENTNAME == "ReadUnit":
634 self.datatype = self.datatype.replace("Reader", "")
633 if self.ELEMENTNAME == 'ReadUnit':
634 self.datatype = self.datatype.replace('Reader', '')
635 635
636 if self.ELEMENTNAME == "ProcUnit":
637 self.datatype = self.datatype.replace("Proc", "")
636 if self.ELEMENTNAME == 'ProcUnit':
637 self.datatype = self.datatype.replace('Proc', '')
638 638
639 639 if self.inputId == 'None':
640 640 self.inputId = '0'
@@ -650,7 +650,7 class ProcUnitConf():
650 650
651 651 def printattr(self):
652 652
653 print "%s[%s]: name = %s, datatype = %s, inputId = %s" %(self.ELEMENTNAME,
653 print '%s[%s]: name = %s, datatype = %s, inputId = %s' %(self.ELEMENTNAME,
654 654 self.id,
655 655 self.name,
656 656 self.datatype,
@@ -751,12 +751,12 class ReadUnitConf(ProcUnitConf):
751 751
752 752 return self.ELEMENTNAME
753 753
754 def setup(self, id, name, datatype, path='', startDate="", endDate="", startTime="",
755 endTime="", parentId=None, queue=None, server=None, **kwargs):
754 def setup(self, id, name, datatype, path='', startDate='', endDate='',
755 startTime='', endTime='', parentId=None, server=None, **kwargs):
756 756
757 757 #Compatible with old signal chain version
758 758 if datatype==None and name==None:
759 raise ValueError, "datatype or name should be defined"
759 raise ValueError, 'datatype or name should be defined'
760 760
761 761 if name==None:
762 762 if 'Reader' in datatype:
@@ -775,39 +775,28 class ReadUnitConf(ProcUnitConf):
775 775 self.endDate = endDate
776 776 self.startTime = startTime
777 777 self.endTime = endTime
778
779 778 self.inputId = '0'
780 779 self.parentId = parentId
781 self.queue = queue
782 780 self.server = server
783 781 self.addRunOperation(**kwargs)
784 782
785 def update(self, datatype, path, startDate, endDate, startTime, endTime, parentId=None, name=None, **kwargs):
783 def update(self, **kwargs):
786 784
787 #Compatible with old signal chain version
788 if datatype==None and name==None:
789 raise ValueError, "datatype or name should be defined"
790
791 if name==None:
785 if 'datatype' in kwargs:
786 datatype = kwargs.pop('datatype')
792 787 if 'Reader' in datatype:
793 name = datatype
788 self.name = datatype
794 789 else:
795 name = '%sReader' %(datatype)
796
797 if datatype==None:
798 datatype = name.replace('Reader','')
799
800 self.datatype = datatype
801 self.name = name
802 self.path = path
803 self.startDate = startDate
804 self.endDate = endDate
805 self.startTime = startTime
806 self.endTime = endTime
790 self.name = '%sReader' %(datatype)
791 self.datatype = self.name.replace('Reader', '')
807 792
793 attrs = ('path', 'startDate', 'endDate', 'startTime', 'endTime', 'parentId')
794
795 for attr in attrs:
796 if attr in kwargs:
797 setattr(self, attr, kwargs.pop(attr))
798
808 799 self.inputId = '0'
809 self.parentId = parentId
810
811 800 self.updateRunOperation(**kwargs)
812 801
813 802 def removeOperations(self):
@@ -822,13 +811,13 class ReadUnitConf(ProcUnitConf):
822 811 opObj = self.addOperation(name = 'run', optype = 'self')
823 812
824 813 if self.server is None:
825 opObj.addParameter(name='datatype' , value=self.datatype, format='str')
826 opObj.addParameter(name='path' , value=self.path, format='str')
827 opObj.addParameter(name='startDate' , value=self.startDate, format='date')
828 opObj.addParameter(name='endDate' , value=self.endDate, format='date')
829 opObj.addParameter(name='startTime' , value=self.startTime, format='time')
830 opObj.addParameter(name='endTime' , value=self.endTime, format='time')
831 opObj.addParameter(name='queue' , value=self.queue, format='obj')
814 opObj.addParameter(name='datatype', value=self.datatype, format='str')
815 opObj.addParameter(name='path', value=self.path, format='str')
816 opObj.addParameter(name='startDate', value=self.startDate, format='date')
817 opObj.addParameter(name='endDate', value=self.endDate, format='date')
818 opObj.addParameter(name='startTime', value=self.startTime, format='time')
819 opObj.addParameter(name='endTime', value=self.endTime, format='time')
820
832 821 for key, value in kwargs.items():
833 822 opObj.addParameter(name=key, value=value, format=type(value).__name__)
834 823 else:
@@ -839,32 +828,21 class ReadUnitConf(ProcUnitConf):
839 828
840 829 def updateRunOperation(self, **kwargs):
841 830
842 opObj = self.getOperationObj(name = 'run')
831 opObj = self.getOperationObj(name='run')
843 832 opObj.removeParameters()
844 833
845 opObj.addParameter(name='datatype' , value=self.datatype, format='str')
846 opObj.addParameter(name='path' , value=self.path, format='str')
847 opObj.addParameter(name='startDate' , value=self.startDate, format='date')
848 opObj.addParameter(name='endDate' , value=self.endDate, format='date')
849 opObj.addParameter(name='startTime' , value=self.startTime, format='time')
850 opObj.addParameter(name='endTime' , value=self.endTime, format='time')
851
834 opObj.addParameter(name='datatype', value=self.datatype, format='str')
835 opObj.addParameter(name='path', value=self.path, format='str')
836 opObj.addParameter(name='startDate', value=self.startDate, format='date')
837 opObj.addParameter(name='endDate', value=self.endDate, format='date')
838 opObj.addParameter(name='startTime', value=self.startTime, format='time')
839 opObj.addParameter(name='endTime', value=self.endTime, format='time')
840
852 841 for key, value in kwargs.items():
853 842 opObj.addParameter(name=key, value=value, format=type(value).__name__)
854 843
855 844 return opObj
856 845
857 # def makeXml(self, projectElement):
858 #
859 # procUnitElement = SubElement(projectElement, self.ELEMENTNAME)
860 # procUnitElement.set('id', str(self.id))
861 # procUnitElement.set('name', self.name)
862 # procUnitElement.set('datatype', self.datatype)
863 # procUnitElement.set('inputId', str(self.inputId))
864 #
865 # for opConfObj in self.opConfObjList:
866 # opConfObj.makeXml(procUnitElement)
867
868 846 def readXml(self, upElement):
869 847
870 848 self.id = upElement.get('id')
@@ -872,8 +850,8 class ReadUnitConf(ProcUnitConf):
872 850 self.datatype = upElement.get('datatype')
873 851 self.inputId = upElement.get('inputId')
874 852
875 if self.ELEMENTNAME == "ReadUnit":
876 self.datatype = self.datatype.replace("Reader", "")
853 if self.ELEMENTNAME == 'ReadUnit':
854 self.datatype = self.datatype.replace('Reader', '')
877 855
878 856 if self.inputId == 'None':
879 857 self.inputId = '0'
@@ -894,10 +872,10 class ReadUnitConf(ProcUnitConf):
894 872 self.startTime = opConfObj.getParameterValue('startTime')
895 873 self.endTime = opConfObj.getParameterValue('endTime')
896 874
897 class Project():
875 class Project(Process):
898 876
899 877 id = None
900 name = None
878 # name = None
901 879 description = None
902 880 filename = None
903 881
@@ -909,8 +887,9 class Project():
909 887
910 888 def __init__(self, plotter_queue=None):
911 889
890 Process.__init__(self)
912 891 self.id = None
913 self.name = None
892 # self.name = None
914 893 self.description = None
915 894
916 895 self.plotterQueue = plotter_queue
@@ -962,17 +941,26 class Project():
962 941
963 942 self.procUnitConfObjDict = newProcUnitConfObjDict
964 943
965 def setup(self, id, name, description):
944 def setup(self, id, name='', description=''):
966 945
946 print
947 print '*'*60
948 print ' Starting SIGNAL CHAIN PROCESSING v%s ' % schainpy.__version__
949 print '*'*60
950 print
967 951 self.id = str(id)
968 self.name = name
969 952 self.description = description
970 953
971 954 def update(self, name, description):
972 955
973 self.name = name
974 956 self.description = description
975 957
958 def clone(self):
959
960 p = Project()
961 p.procUnitConfObjDict = self.procUnitConfObjDict
962 return p
963
976 964 def addReadUnit(self, id=None, datatype=None, name=None, **kwargs):
977 965
978 966 if id is None:
@@ -1012,7 +1000,7 class Project():
1012 1000 def getReadUnitObj(self):
1013 1001
1014 1002 for obj in self.procUnitConfObjDict.values():
1015 if obj.getElementName() == "ReadUnit":
1003 if obj.getElementName() == 'ReadUnit':
1016 1004 return obj
1017 1005
1018 1006 return None
@@ -1057,20 +1045,20 class Project():
1057 1045 if self.filename:
1058 1046 filename = self.filename
1059 1047 else:
1060 filename = "schain.xml"
1048 filename = 'schain.xml'
1061 1049
1062 1050 if not filename:
1063 print "filename has not been defined. Use setFilename(filename) for do it."
1051 print 'filename has not been defined. Use setFilename(filename) for do it.'
1064 1052 return 0
1065 1053
1066 1054 abs_file = os.path.abspath(filename)
1067 1055
1068 1056 if not os.access(os.path.dirname(abs_file), os.W_OK):
1069 print "No write permission on %s" %os.path.dirname(abs_file)
1057 print 'No write permission on %s' %os.path.dirname(abs_file)
1070 1058 return 0
1071 1059
1072 1060 if os.path.isfile(abs_file) and not(os.access(abs_file, os.W_OK)):
1073 print "File %s already exists and it could not be overwriten" %abs_file
1061 print 'File %s already exists and it could not be overwriten' %abs_file
1074 1062 return 0
1075 1063
1076 1064 self.makeXml()
@@ -1084,13 +1072,13 class Project():
1084 1072 def readXml(self, filename = None):
1085 1073
1086 1074 if not filename:
1087 print "filename is not defined"
1075 print 'filename is not defined'
1088 1076 return 0
1089 1077
1090 1078 abs_file = os.path.abspath(filename)
1091 1079
1092 1080 if not os.path.isfile(abs_file):
1093 print "%s file does not exist" %abs_file
1081 print '%s file does not exist' %abs_file
1094 1082 return 0
1095 1083
1096 1084 self.projectElement = None
@@ -1099,7 +1087,7 class Project():
1099 1087 try:
1100 1088 self.projectElement = ElementTree().parse(abs_file)
1101 1089 except:
1102 print "Error reading %s, verify file format" %filename
1090 print 'Error reading %s, verify file format' %filename
1103 1091 return 0
1104 1092
1105 1093 self.project = self.projectElement.tag
@@ -1136,7 +1124,7 class Project():
1136 1124
1137 1125 def printattr(self):
1138 1126
1139 print "Project[%s]: name = %s, description = %s" %(self.id,
1127 print 'Project[%s]: name = %s, description = %s' %(self.id,
1140 1128 self.name,
1141 1129 self.description)
1142 1130
@@ -1170,7 +1158,7 class Project():
1170 1158
1171 1159 self.__connect(puObjIN, thisPUObj)
1172 1160
1173 def __handleError(self, procUnitConfObj, send_email=True):
1161 def __handleError(self, procUnitConfObj, send_email=False):
1174 1162
1175 1163 import socket
1176 1164
@@ -1178,33 +1166,33 class Project():
1178 1166 sys.exc_info()[1],
1179 1167 sys.exc_info()[2])
1180 1168
1181 print "***** Error occurred in %s *****" %(procUnitConfObj.name)
1182 print "***** %s" %err[-1]
1169 print '***** Error occurred in %s *****' %(procUnitConfObj.name)
1170 print '***** %s' %err[-1]
1183 1171
1184 message = "".join(err)
1172 message = ''.join(err)
1185 1173
1186 1174 sys.stderr.write(message)
1187 1175
1188 1176 if not send_email:
1189 1177 return
1190 1178
1191 subject = "SChain v%s: Error running %s\n" %(schainpy.__version__, procUnitConfObj.name)
1179 subject = 'SChain v%s: Error running %s\n' %(schainpy.__version__, procUnitConfObj.name)
1192 1180
1193 subtitle = "%s: %s\n" %(procUnitConfObj.getElementName() ,procUnitConfObj.name)
1194 subtitle += "Hostname: %s\n" %socket.gethostbyname(socket.gethostname())
1195 subtitle += "Working directory: %s\n" %os.path.abspath("./")
1196 subtitle += "Configuration file: %s\n" %self.filename
1197 subtitle += "Time: %s\n" %str(datetime.datetime.now())
1181 subtitle = '%s: %s\n' %(procUnitConfObj.getElementName() ,procUnitConfObj.name)
1182 subtitle += 'Hostname: %s\n' %socket.gethostbyname(socket.gethostname())
1183 subtitle += 'Working directory: %s\n' %os.path.abspath('./')
1184 subtitle += 'Configuration file: %s\n' %self.filename
1185 subtitle += 'Time: %s\n' %str(datetime.datetime.now())
1198 1186
1199 1187 readUnitConfObj = self.getReadUnitObj()
1200 1188 if readUnitConfObj:
1201 subtitle += "\nInput parameters:\n"
1202 subtitle += "[Data path = %s]\n" %readUnitConfObj.path
1203 subtitle += "[Data type = %s]\n" %readUnitConfObj.datatype
1204 subtitle += "[Start date = %s]\n" %readUnitConfObj.startDate
1205 subtitle += "[End date = %s]\n" %readUnitConfObj.endDate
1206 subtitle += "[Start time = %s]\n" %readUnitConfObj.startTime
1207 subtitle += "[End time = %s]\n" %readUnitConfObj.endTime
1189 subtitle += '\nInput parameters:\n'
1190 subtitle += '[Data path = %s]\n' %readUnitConfObj.path
1191 subtitle += '[Data type = %s]\n' %readUnitConfObj.datatype
1192 subtitle += '[Start date = %s]\n' %readUnitConfObj.startDate
1193 subtitle += '[End date = %s]\n' %readUnitConfObj.endDate
1194 subtitle += '[Start time = %s]\n' %readUnitConfObj.startTime
1195 subtitle += '[End time = %s]\n' %readUnitConfObj.endTime
1208 1196
1209 1197 adminObj = schainpy.admin.SchainNotify()
1210 1198 adminObj.sendAlert(message=message,
@@ -1219,15 +1207,15 class Project():
1219 1207 return 0
1220 1208
1221 1209 def runController(self):
1222 """
1210 '''
1223 1211 returns 0 when this process has been stopped, 1 otherwise
1224 """
1212 '''
1225 1213
1226 1214 if self.isPaused():
1227 print "Process suspended"
1215 print 'Process suspended'
1228 1216
1229 1217 while True:
1230 sleep(0.1)
1218 time.sleep(0.1)
1231 1219
1232 1220 if not self.isPaused():
1233 1221 break
@@ -1235,10 +1223,10 class Project():
1235 1223 if self.isStopped():
1236 1224 break
1237 1225
1238 print "Process reinitialized"
1226 print 'Process reinitialized'
1239 1227
1240 1228 if self.isStopped():
1241 print "Process stopped"
1229 print 'Process stopped'
1242 1230 return 0
1243 1231
1244 1232 return 1
@@ -1249,23 +1237,22 class Project():
1249 1237
1250 1238 def setPlotterQueue(self, plotter_queue):
1251 1239
1252 raise NotImplementedError, "Use schainpy.controller_api.ControllerThread instead Project class"
1240 raise NotImplementedError, 'Use schainpy.controller_api.ControllerThread instead Project class'
1253 1241
1254 1242 def getPlotterQueue(self):
1255 1243
1256 raise NotImplementedError, "Use schainpy.controller_api.ControllerThread instead Project class"
1244 raise NotImplementedError, 'Use schainpy.controller_api.ControllerThread instead Project class'
1257 1245
1258 1246 def useExternalPlotter(self):
1259 1247
1260 raise NotImplementedError, "Use schainpy.controller_api.ControllerThread instead Project class"
1248 raise NotImplementedError, 'Use schainpy.controller_api.ControllerThread instead Project class'
1261 1249
1262 1250 def run(self):
1263 1251
1264 print
1265 print "*"*60
1266 print " Starting SIGNAL CHAIN PROCESSING v%s " %schainpy.__version__
1267 print "*"*60
1268 print
1252 log.success('Starting {}'.format(self.name))
1253
1254 self.createObjects()
1255 self.connectObjects()
1269 1256
1270 1257 keyList = self.procUnitConfObjDict.keys()
1271 1258 keyList.sort()
@@ -1275,7 +1262,6 class Project():
1275 1262 is_ok = False
1276 1263
1277 1264 for procKey in keyList:
1278 # print "Running the '%s' process with %s" %(procUnitConfObj.name, procUnitConfObj.id)
1279 1265
1280 1266 procUnitConfObj = self.procUnitConfObjDict[procKey]
1281 1267
@@ -1286,19 +1272,18 class Project():
1286 1272 is_ok = False
1287 1273 break
1288 1274 except ValueError, e:
1289 sleep(0.5)
1275 time.sleep(0.5)
1290 1276 self.__handleError(procUnitConfObj, send_email=True)
1291 1277 is_ok = False
1292 1278 break
1293 1279 except:
1294 sleep(0.5)
1280 time.sleep(0.5)
1295 1281 self.__handleError(procUnitConfObj)
1296 1282 is_ok = False
1297 1283 break
1298 1284
1299 1285 #If every process unit finished so end process
1300 1286 if not(is_ok):
1301 # print "Every process unit have finished"
1302 1287 break
1303 1288
1304 1289 if not self.runController():
@@ -1309,11 +1294,4 class Project():
1309 1294 procUnitConfObj = self.procUnitConfObjDict[procKey]
1310 1295 procUnitConfObj.close()
1311 1296
1312 print "Process finished"
1313
1314 def start(self, filename=None):
1315
1316 self.writeXml(filename)
1317 self.createObjects()
1318 self.connectObjects()
1319 self.run()
1297 log.success('{} finished'.format(self.name))
Show file before | Show file after | Hide comments
@@ -14,8 +14,8 class ControllerThread(threading.Thread, Project):
14 14 self.setDaemon(True)
15 15
16 16 self.lock = threading.Lock()
17 self.control = {'stop':False, 'pause':False}
18
17 self.control = { 'stop':False, 'pause':False }
18
19 19 def __del__(self):
20 20
21 21 self.control['stop'] = True
Show file before | Show file after | Hide comments
@@ -6218,3 +6218,6 class ShowMeConsole(QtCore.QObject):
6218 6218 text = text[:-1]
6219 6219
6220 6220 self.textWritten.emit(str(text))
6221
6222 def flush(self):
6223 pass
Show file before | Show file after | Hide comments
@@ -9,4 +9,4 from data import *
9 9 from io import *
10 10 from proc import *
11 11 from graphics import *
12 from utils import *
12 from utils import * No newline at end of file
Show file before | Show file after | Hide comments
@@ -700,7 +700,7 class Spectra(JROData):
700 700 for pair in pairsList:
701 701 if pair not in self.pairsList:
702 702 raise ValueError, "Pair %s is not in dataOut.pairsList" %(pair)
703 pairsIndexList.append(self.pairsList.index(pair))
703 pairsIndexList.append(self.pairsList.index(pair))
704 704 for i in range(len(pairsIndexList)):
705 705 pair = self.pairsList[pairsIndexList[i]]
706 706 ccf = numpy.average(self.data_cspc[pairsIndexList[i], :, :], axis=0)
Show file before | Show file after | Hide comments
@@ -61,7 +61,7 class Figure(Operation):
61 61 figfile = None
62 62
63 63 created = False
64
64 parameters = {}
65 65 def __init__(self, **kwargs):
66 66
67 67 Operation.__init__(self, **kwargs)
Show file before | Show file after | Hide comments
@@ -2,11 +2,10 import os
2 2 import datetime
3 3 import numpy
4 4 import copy
5
5 from schainpy.model import *
6 6 from figure import Figure, isRealtime
7 7
8 8 class CorrelationPlot(Figure):
9
10 9 isConfig = None
11 10 __nsubplots = None
12 11
Show file before | Show file after | Hide comments
This diff has been collapsed as it changes many lines, (1201 lines changed) Show them Hide them
@@ -1,30 +1,33
1 1
2 2 import os
3 import zmq
4 3 import time
5 import numpy
4 import glob
6 5 import datetime
7 import numpy as np
6 from multiprocessing import Process
7
8 import zmq
9 import numpy
8 10 import matplotlib
9 import glob
10 matplotlib.use('TkAgg')
11 11 import matplotlib.pyplot as plt
12 12 from mpl_toolkits.axes_grid1 import make_axes_locatable
13 from matplotlib.ticker import FuncFormatter, LinearLocator
14 from multiprocessing import Process
13 from matplotlib.ticker import FuncFormatter, LinearLocator, MultipleLocator
15 14
16 15 from schainpy.model.proc.jroproc_base import Operation
17
18 plt.ion()
16 from schainpy.utils import log
19 17
20 18 func = lambda x, pos: ('%s') %(datetime.datetime.fromtimestamp(x).strftime('%H:%M'))
21 19
22 d1970 = datetime.datetime(1970,1,1)
20 d1970 = datetime.datetime(1970, 1, 1)
21
23 22
24 23 class PlotData(Operation, Process):
24 '''
25 Base class for Schain plotting operations
26 '''
25 27
26 28 CODE = 'Figure'
27 29 colormap = 'jro'
30 bgcolor = 'white'
28 31 CONFLATE = False
29 32 __MAXNUMX = 80
30 33 __missing = 1E30
@@ -35,54 +38,143 class PlotData(Operation, Process):
35 38 Process.__init__(self)
36 39 self.kwargs['code'] = self.CODE
37 40 self.mp = False
38 self.dataOut = None
39 self.isConfig = False
40 self.figure = None
41 self.data = None
42 self.isConfig = False
43 self.figures = []
41 44 self.axes = []
45 self.cb_axes = []
42 46 self.localtime = kwargs.pop('localtime', True)
43 47 self.show = kwargs.get('show', True)
44 48 self.save = kwargs.get('save', False)
45 49 self.colormap = kwargs.get('colormap', self.colormap)
46 50 self.colormap_coh = kwargs.get('colormap_coh', 'jet')
47 51 self.colormap_phase = kwargs.get('colormap_phase', 'RdBu_r')
48 self.showprofile = kwargs.get('showprofile', True)
49 self.title = kwargs.get('wintitle', '')
52 self.colormaps = kwargs.get('colormaps', None)
53 self.bgcolor = kwargs.get('bgcolor', self.bgcolor)
54 self.showprofile = kwargs.get('showprofile', False)
55 self.title = kwargs.get('wintitle', self.CODE.upper())
56 self.cb_label = kwargs.get('cb_label', None)
57 self.cb_labels = kwargs.get('cb_labels', None)
50 58 self.xaxis = kwargs.get('xaxis', 'frequency')
51 59 self.zmin = kwargs.get('zmin', None)
52 60 self.zmax = kwargs.get('zmax', None)
61 self.zlimits = kwargs.get('zlimits', None)
53 62 self.xmin = kwargs.get('xmin', None)
63 if self.xmin is not None:
64 self.xmin += 5
54 65 self.xmax = kwargs.get('xmax', None)
55 66 self.xrange = kwargs.get('xrange', 24)
56 67 self.ymin = kwargs.get('ymin', None)
57 68 self.ymax = kwargs.get('ymax', None)
58 self.__MAXNUMY = kwargs.get('decimation', 80)
59 self.throttle_value = 5
60 self.times = []
61 #self.interactive = self.kwargs['parent']
69 self.xlabel = kwargs.get('xlabel', None)
70 self.__MAXNUMY = kwargs.get('decimation', 100)
71 self.showSNR = kwargs.get('showSNR', False)
72 self.oneFigure = kwargs.get('oneFigure', True)
73 self.width = kwargs.get('width', None)
74 self.height = kwargs.get('height', None)
75 self.colorbar = kwargs.get('colorbar', True)
76 self.factors = kwargs.get('factors', [1, 1, 1, 1, 1, 1, 1, 1])
77 self.titles = ['' for __ in range(16)]
78
79 def __setup(self):
80 '''
81 Common setup for all figures, here figures and axes are created
82 '''
83
84 self.setup()
62 85
86 if self.width is None:
87 self.width = 8
88
89 self.figures = []
90 self.axes = []
91 self.cb_axes = []
92 self.pf_axes = []
93 self.cmaps = []
94
95 size = '15%' if self.ncols==1 else '30%'
96 pad = '4%' if self.ncols==1 else '8%'
97
98 if self.oneFigure:
99 if self.height is None:
100 self.height = 1.4*self.nrows + 1
101 fig = plt.figure(figsize=(self.width, self.height),
102 edgecolor='k',
103 facecolor='w')
104 self.figures.append(fig)
105 for n in range(self.nplots):
106 ax = fig.add_subplot(self.nrows, self.ncols, n+1)
107 ax.tick_params(labelsize=8)
108 ax.firsttime = True
109 self.axes.append(ax)
110 if self.showprofile:
111 cax = self.__add_axes(ax, size=size, pad=pad)
112 cax.tick_params(labelsize=8)
113 self.pf_axes.append(cax)
114 else:
115 if self.height is None:
116 self.height = 3
117 for n in range(self.nplots):
118 fig = plt.figure(figsize=(self.width, self.height),
119 edgecolor='k',
120 facecolor='w')
121 ax = fig.add_subplot(1, 1, 1)
122 ax.tick_params(labelsize=8)
123 ax.firsttime = True
124 self.figures.append(fig)
125 self.axes.append(ax)
126 if self.showprofile:
127 cax = self.__add_axes(ax, size=size, pad=pad)
128 cax.tick_params(labelsize=8)
129 self.pf_axes.append(cax)
130
131 for n in range(self.nrows):
132 if self.colormaps is not None:
133 cmap = plt.get_cmap(self.colormaps[n])
134 else:
135 cmap = plt.get_cmap(self.colormap)
136 cmap.set_bad(self.bgcolor, 1.)
137 self.cmaps.append(cmap)
138
139 def __add_axes(self, ax, size='30%', pad='8%'):
63 140 '''
64 this new parameter is created to plot data from varius channels at different figures
65 1. crear una lista de figuras donde se puedan plotear las figuras,
66 2. dar las opciones de configuracion a cada figura, estas opciones son iguales para ambas figuras
67 3. probar?
141 Add new axes to the given figure
68 142 '''
69 self.ind_plt_ch = kwargs.get('ind_plt_ch', False)
70 self.figurelist = None
143 divider = make_axes_locatable(ax)
144 nax = divider.new_horizontal(size=size, pad=pad)
145 ax.figure.add_axes(nax)
146 return nax
71 147
72 148
73 def fill_gaps(self, x_buffer, y_buffer, z_buffer):
149 def setup(self):
150 '''
151 This method should be implemented in the child class, the following
152 attributes should be set:
153
154 self.nrows: number of rows
155 self.ncols: number of cols
156 self.nplots: number of plots (channels or pairs)
157 self.ylabel: label for Y axes
158 self.titles: list of axes title
159
160 '''
161 raise(NotImplementedError, 'Implement this method in child class')
74 162
163 def fill_gaps(self, x_buffer, y_buffer, z_buffer):
164 '''
165 Create a masked array for missing data
166 '''
75 167 if x_buffer.shape[0] < 2:
76 168 return x_buffer, y_buffer, z_buffer
77 169
78 170 deltas = x_buffer[1:] - x_buffer[0:-1]
79 x_median = np.median(deltas)
171 x_median = numpy.median(deltas)
80 172
81 index = np.where(deltas > 5*x_median)
173 index = numpy.where(deltas > 5*x_median)
82 174
83 175 if len(index[0]) != 0:
84 176 z_buffer[::, index[0], ::] = self.__missing
85 z_buffer = np.ma.masked_inside(z_buffer,
177 z_buffer = numpy.ma.masked_inside(z_buffer,
86 178 0.99*self.__missing,
87 179 1.01*self.__missing)
88 180
@@ -97,110 +189,117 class PlotData(Operation, Process):
97 189 x = self.x
98 190 y = self.y[::dy]
99 191 z = self.z[::, ::, ::dy]
100
192
101 193 return x, y, z
102 194
103 '''
104 JM:
105 elimana las otras imagenes generadas debido a que lso workers no llegan en orden y le pueden
106 poner otro tiempo a la figura q no necesariamente es el ultimo.
107 Solo se realiza cuando termina la imagen.
108 Problemas:
195 def format(self):
196 '''
197 Set min and max values, labels, ticks and titles
198 '''
109 199
110 File "/home/ci-81/workspace/schainv2.3/schainpy/model/graphics/jroplot_data.py", line 145, in __plot
111 for n, eachfigure in enumerate(self.figurelist):
112 TypeError: 'NoneType' object is not iterable
200 if self.xmin is None:
201 xmin = self.min_time
202 else:
203 if self.xaxis is 'time':
204 dt = datetime.datetime.fromtimestamp(self.min_time)
205 xmin = (datetime.datetime.combine(dt.date(),
206 datetime.time(int(self.xmin), 0, 0))-d1970).total_seconds()
207 else:
208 xmin = self.xmin
113 209
114 '''
115 def deleteanotherfiles(self):
116 figurenames=[]
117 if self.figurelist != None:
118 for n, eachfigure in enumerate(self.figurelist):
119 #add specific name for each channel in channelList
120 ghostfigname = os.path.join(self.save, '{}_{}_{}'.format(self.titles[n].replace(' ',''),self.CODE,
121 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d')))
122 figname = os.path.join(self.save, '{}_{}_{}.png'.format(self.titles[n].replace(' ',''),self.CODE,
123 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
124
125 for ghostfigure in glob.glob(ghostfigname+'*'): #ghostfigure will adopt all posible names of figures
126 if ghostfigure != figname:
127 os.remove(ghostfigure)
128 print 'Removing GhostFigures:' , figname
129 else :
130 '''Erasing ghost images for just on******************'''
131 ghostfigname = os.path.join(self.save, '{}_{}'.format(self.CODE,datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d')))
132 figname = os.path.join(self.save, '{}_{}.png'.format(self.CODE,datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
133 for ghostfigure in glob.glob(ghostfigname+'*'): #ghostfigure will adopt all posible names of figures
134 if ghostfigure != figname:
135 os.remove(ghostfigure)
136 print 'Removing GhostFigures:' , figname
210 if self.xmax is None:
211 xmax = xmin+self.xrange*60*60
212 else:
213 if self.xaxis is 'time':
214 dt = datetime.datetime.fromtimestamp(self.min_time)
215 xmax = (datetime.datetime.combine(dt.date(),
216 datetime.time(int(self.xmax), 0, 0))-d1970).total_seconds()
217 else:
218 xmax = self.xmax
219
220 ymin = self.ymin if self.ymin else numpy.nanmin(self.y)
221 ymax = self.ymax if self.ymax else numpy.nanmax(self.y)
222
223 ystep = 200 if ymax>= 800 else 100 if ymax>=400 else 50 if ymax>=200 else 20
224
225 for n, ax in enumerate(self.axes):
226 if ax.firsttime:
227 ax.set_facecolor(self.bgcolor)
228 ax.yaxis.set_major_locator(MultipleLocator(ystep))
229 if self.xaxis is 'time':
230 ax.xaxis.set_major_formatter(FuncFormatter(func))
231 ax.xaxis.set_major_locator(LinearLocator(9))
232 if self.xlabel is not None:
233 ax.set_xlabel(self.xlabel)
234 ax.set_ylabel(self.ylabel)
235 ax.firsttime = False
236 if self.showprofile:
237 self.pf_axes[n].set_ylim(ymin, ymax)
238 self.pf_axes[n].set_xlim(self.zmin, self.zmax)
239 self.pf_axes[n].set_xlabel('dB')
240 self.pf_axes[n].grid(b=True, axis='x')
241 [tick.set_visible(False) for tick in self.pf_axes[n].get_yticklabels()]
242 if self.colorbar:
243 cb = plt.colorbar(ax.plt, ax=ax, pad=0.02)
244 cb.ax.tick_params(labelsize=8)
245 if self.cb_label:
246 cb.set_label(self.cb_label, size=8)
247 elif self.cb_labels:
248 cb.set_label(self.cb_labels[n], size=8)
249
250 ax.set_title('{} - {} UTC'.format(
251 self.titles[n],
252 datetime.datetime.fromtimestamp(self.max_time).strftime('%H:%M:%S')),
253 size=8)
254 ax.set_xlim(xmin, xmax)
255 ax.set_ylim(ymin, ymax)
256
137 257
138 258 def __plot(self):
139
140 print 'plotting...{}'.format(self.CODE)
141 if self.ind_plt_ch is False : #standard
259 '''
260 '''
261 log.success('Plotting', self.name)
262
263 self.plot()
264 self.format()
265
266 for n, fig in enumerate(self.figures):
267 if self.nrows == 0 or self.nplots == 0:
268 log.warning('No data', self.name)
269 continue
142 270 if self.show:
143 self.figure.show()
144 self.plot()
145 plt.tight_layout()
146 self.figure.canvas.manager.set_window_title('{} {} - {}'.format(self.title, self.CODE.upper(),
147 datetime.datetime.fromtimestamp(self.max_time).strftime('%Y/%m/%d')))
148 else :
149 print 'len(self.figurelist): ',len(self.figurelist)
150 for n, eachfigure in enumerate(self.figurelist):
151 if self.show:
152 eachfigure.show()
153
154 self.plot()
155 eachfigure.tight_layout() # ajuste de cada subplot
156 eachfigure.canvas.manager.set_window_title('{} {} - {}'.format(self.title[n], self.CODE.upper(),
157 datetime.datetime.fromtimestamp(self.max_time).strftime('%Y/%m/%d')))
158
159 # if self.save:
160 # if self.ind_plt_ch is False : #standard
161 # figname = os.path.join(self.save, '{}_{}.png'.format(self.CODE,
162 # datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
163 # print 'Saving figure: {}'.format(figname)
164 # self.figure.savefig(figname)
165 # else :
166 # for n, eachfigure in enumerate(self.figurelist):
167 # #add specific name for each channel in channelList
168 # figname = os.path.join(self.save, '{}_{}_{}.png'.format(self.titles[n],self.CODE,
169 # datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
170 #
171 # print 'Saving figure: {}'.format(figname)
172 # eachfigure.savefig(figname)
173
174 if self.ind_plt_ch is False :
175 self.figure.canvas.draw()
176 else :
177 for eachfigure in self.figurelist:
178 eachfigure.canvas.draw()
179
180 if self.save:
181 if self.ind_plt_ch is False : #standard
182 figname = os.path.join(self.save, '{}_{}.png'.format(self.CODE,
183 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
271 fig.show()
272
273 fig.tight_layout()
274 fig.canvas.manager.set_window_title('{} - {}'.format(self.title,
275 datetime.datetime.fromtimestamp(self.max_time).strftime('%Y/%m/%d')))
276 # fig.canvas.draw()
277
278 if self.save and self.data.ended:
279 channels = range(self.nrows)
280 if self.oneFigure:
281 label = ''
282 else:
283 label = '_{}'.format(channels[n])
284 figname = os.path.join(
285 self.save,
286 '{}{}_{}.png'.format(
287 self.CODE,
288 label,
289 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')
290 )
291 )
184 292 print 'Saving figure: {}'.format(figname)
185 self.figure.savefig(figname)
186 else :
187 for n, eachfigure in enumerate(self.figurelist):
188 #add specific name for each channel in channelList
189 figname = os.path.join(self.save, '{}_{}_{}.png'.format(self.titles[n].replace(' ',''),self.CODE,
190 datetime.datetime.fromtimestamp(self.saveTime).strftime('%y%m%d_%H%M%S')))
191
192 print 'Saving figure: {}'.format(figname)
193 eachfigure.savefig(figname)
194
293 fig.savefig(figname)
195 294
196 295 def plot(self):
197
198 print 'plotting...{}'.format(self.CODE.upper())
199 return
296 '''
297 '''
298 raise(NotImplementedError, 'Implement this method in child class')
200 299
201 300 def run(self):
202 301
203 print '[Starting] {}'.format(self.name)
302 log.success('Starting', self.name)
204 303
205 304 context = zmq.Context()
206 305 receiver = context.socket(zmq.SUB)
@@ -210,152 +309,104 class PlotData(Operation, Process):
210 309 if 'server' in self.kwargs['parent']:
211 310 receiver.connect('ipc:///tmp/{}.plots'.format(self.kwargs['parent']['server']))
212 311 else:
213 receiver.connect("ipc:///tmp/zmq.plots")
214
215 seconds_passed = 0
312 receiver.connect("ipc:///tmp/zmq.plots")
216 313
217 314 while True:
218 315 try:
219 self.data = receiver.recv_pyobj(flags=zmq.NOBLOCK)#flags=zmq.NOBLOCK
220 self.started = self.data['STARTED']
221 self.dataOut = self.data['dataOut']
222
223 if (len(self.times) < len(self.data['times']) and not self.started and self.data['ENDED']):
224 continue
225
226 self.times = self.data['times']
227 self.times.sort()
228 self.throttle_value = self.data['throttle']
229 self.min_time = self.times[0]
230 self.max_time = self.times[-1]
316 self.data = receiver.recv_pyobj(flags=zmq.NOBLOCK)
317
318 self.min_time = self.data.times[0]
319 self.max_time = self.data.times[-1]
231 320
232 321 if self.isConfig is False:
233 print 'setting up'
234 self.setup()
322 self.__setup()
235 323 self.isConfig = True
236 self.__plot()
237
238 if self.data['ENDED'] is True:
239 print '********GRAPHIC ENDED********'
240 self.ended = True
241 self.isConfig = False
242 self.__plot()
243 self.deleteanotherfiles() #CLPDG
244 elif seconds_passed >= self.data['throttle']:
245 print 'passed', seconds_passed
246 self.__plot()
247 seconds_passed = 0
324
325 self.__plot()
248 326
249 327 except zmq.Again as e:
250 print 'Waiting for data...'
251 plt.pause(2)
252 seconds_passed += 2
328 log.log('Waiting for data...')
329 if self.data:
330 plt.pause(self.data.throttle)
331 else:
332 time.sleep(2)
253 333
254 334 def close(self):
255 if self.dataOut:
335 if self.data:
256 336 self.__plot()
257 337
258 338
259 339 class PlotSpectraData(PlotData):
340 '''
341 Plot for Spectra data
342 '''
260 343
261 344 CODE = 'spc'
262 colormap = 'jro'
263 CONFLATE = False
345 colormap = 'jro'
264 346
265 347 def setup(self):
266
267 ncolspan = 1
268 colspan = 1
269 self.ncols = int(numpy.sqrt(self.dataOut.nChannels)+0.9)
270 self.nrows = int(self.dataOut.nChannels*1./self.ncols + 0.9)
271 self.width = 3.6*self.ncols
272 self.height = 3.2*self.nrows
273 if self.showprofile:
274 ncolspan = 3
275 colspan = 2
276 self.width += 1.2*self.ncols
348 self.nplots = len(self.data.channels)
349 self.ncols = int(numpy.sqrt(self.nplots)+ 0.9)
350 self.nrows = int((1.0*self.nplots/self.ncols) + 0.9)
351 self.width = 3.4*self.ncols
352 self.height = 3*self.nrows
353 self.cb_label = 'dB'
354 if self.showprofile:
355 self.width += 0.8*self.ncols
277 356
278 357 self.ylabel = 'Range [Km]'
279 self.titles = ['Channel {}'.format(x) for x in self.dataOut.channelList]
280
281 if self.figure is None:
282 self.figure = plt.figure(figsize=(self.width, self.height),
283 edgecolor='k',
284 facecolor='w')
285 else:
286 self.figure.clf()
287
288 n = 0
289 for y in range(self.nrows):
290 for x in range(self.ncols):
291 if n >= self.dataOut.nChannels:
292 break
293 ax = plt.subplot2grid((self.nrows, self.ncols*ncolspan), (y, x*ncolspan), 1, colspan)
294 if self.showprofile:
295 ax.ax_profile = plt.subplot2grid((self.nrows, self.ncols*ncolspan), (y, x*ncolspan+colspan), 1, 1)
296
297 ax.firsttime = True
298 self.axes.append(ax)
299 n += 1
300 358
301 359 def plot(self):
302
303 360 if self.xaxis == "frequency":
304 x = self.dataOut.getFreqRange(1)/1000.
305 xlabel = "Frequency (kHz)"
361 x = self.data.xrange[0]
362 self.xlabel = "Frequency (kHz)"
306 363 elif self.xaxis == "time":
307 x = self.dataOut.getAcfRange(1)
308 xlabel = "Time (ms)"
364 x = self.data.xrange[1]
365 self.xlabel = "Time (ms)"
309 366 else:
310 x = self.dataOut.getVelRange(1)
311 xlabel = "Velocity (m/s)"
367 x = self.data.xrange[2]
368 self.xlabel = "Velocity (m/s)"
312 369
313 y = self.dataOut.getHeiRange()
314 z = self.data[self.CODE]
370 if self.CODE == 'spc_mean':
371 x = self.data.xrange[2]
372 self.xlabel = "Velocity (m/s)"
315 373
374 self.titles = []
375
376 y = self.data.heights
377 self.y = y
378 z = self.data['spc']
379
316 380 for n, ax in enumerate(self.axes):
381 noise = self.data['noise'][n][-1]
382 if self.CODE == 'spc_mean':
383 mean = self.data['mean'][n][-1]
317 384 if ax.firsttime:
318 self.xmax = self.xmax if self.xmax else np.nanmax(x)
385 self.xmax = self.xmax if self.xmax else numpy.nanmax(x)
319 386 self.xmin = self.xmin if self.xmin else -self.xmax
320 self.ymin = self.ymin if self.ymin else np.nanmin(y)
321 self.ymax = self.ymax if self.ymax else np.nanmax(y)
322 self.zmin = self.zmin if self.zmin else np.nanmin(z)
323 self.zmax = self.zmax if self.zmax else np.nanmax(z)
324 ax.plot = ax.pcolormesh(x, y, z[n].T,
325 vmin=self.zmin,
326 vmax=self.zmax,
327 cmap=plt.get_cmap(self.colormap)
328 )
329 divider = make_axes_locatable(ax)
330 cax = divider.new_horizontal(size='3%', pad=0.05)
331 self.figure.add_axes(cax)
332 plt.colorbar(ax.plot, cax)
333
334 ax.set_xlim(self.xmin, self.xmax)
335 ax.set_ylim(self.ymin, self.ymax)
336
337 ax.set_ylabel(self.ylabel)
338 ax.set_xlabel(xlabel)
339
340 ax.firsttime = False
387 self.zmin = self.zmin if self.zmin else numpy.nanmin(z)
388 self.zmax = self.zmax if self.zmax else numpy.nanmax(z)
389 ax.plt = ax.pcolormesh(x, y, z[n].T,
390 vmin=self.zmin,
391 vmax=self.zmax,
392 cmap=plt.get_cmap(self.colormap)
393 )
341 394
342 395 if self.showprofile:
343 ax.plot_profile= ax.ax_profile.plot(self.data['rti'][self.max_time][n], y)[0]
344 ax.ax_profile.set_xlim(self.zmin, self.zmax)
345 ax.ax_profile.set_ylim(self.ymin, self.ymax)
346 ax.ax_profile.set_xlabel('dB')
347 ax.ax_profile.grid(b=True, axis='x')
348 ax.plot_noise = ax.ax_profile.plot(numpy.repeat(self.data['noise'][self.max_time][n], len(y)), y,
349 color="k", linestyle="dashed", lw=2)[0]
350 [tick.set_visible(False) for tick in ax.ax_profile.get_yticklabels()]
396 ax.plt_profile= self.pf_axes[n].plot(self.data['rti'][n][-1], y)[0]
397 ax.plt_noise = self.pf_axes[n].plot(numpy.repeat(noise, len(y)), y,
398 color="k", linestyle="dashed", lw=1)[0]
399 if self.CODE == 'spc_mean':
400 ax.plt_mean = ax.plot(mean, y, color='k')[0]
351 401 else:
352 ax.plot.set_array(z[n].T.ravel())
402 ax.plt.set_array(z[n].T.ravel())
353 403 if self.showprofile:
354 ax.plot_profile.set_data(self.data['rti'][self.max_time][n], y)
355 ax.plot_noise.set_data(numpy.repeat(self.data['noise'][self.max_time][n], len(y)), y)
404 ax.plt_profile.set_data(self.data['rti'][n][-1], y)
405 ax.plt_noise.set_data(numpy.repeat(noise, len(y)), y)
406 if self.CODE == 'spc_mean':
407 ax.plt_mean.set_data(mean, y)
356 408
357 ax.set_title('{} - Noise: {:.2f} dB'.format(self.titles[n], self.data['noise'][self.max_time][n]),
358 size=8)
409 self.titles.append('CH {}: {:3.2f}dB'.format(n, noise))
359 410 self.saveTime = self.max_time
360 411
361 412
@@ -365,538 +416,245 class PlotCrossSpectraData(PlotData):
365 416 zmin_coh = None
366 417 zmax_coh = None
367 418 zmin_phase = None
368 zmax_phase = None
369 CONFLATE = False
419 zmax_phase = None
370 420
371 421 def setup(self):
372 422
373 ncolspan = 1
374 colspan = 1
375 self.ncols = 2
376 self.nrows = self.dataOut.nPairs
377 self.width = 3.6*self.ncols
378 self.height = 3.2*self.nrows
379
423 self.ncols = 4
424 self.nrows = len(self.data.pairs)
425 self.nplots = self.nrows*4
426 self.width = 3.4*self.ncols
427 self.height = 3*self.nrows
380 428 self.ylabel = 'Range [Km]'
381 self.titles = ['Channel {}'.format(x) for x in self.dataOut.channelList]
382
383 if self.figure is None:
384 self.figure = plt.figure(figsize=(self.width, self.height),
385 edgecolor='k',
386 facecolor='w')
387 else:
388 self.figure.clf()
389
390 for y in range(self.nrows):
391 for x in range(self.ncols):
392 ax = plt.subplot2grid((self.nrows, self.ncols), (y, x), 1, 1)
393 ax.firsttime = True
394 self.axes.append(ax)
429 self.showprofile = False
395 430
396 431 def plot(self):
397 432
398 433 if self.xaxis == "frequency":
399 x = self.dataOut.getFreqRange(1)/1000.
400 xlabel = "Frequency (kHz)"
434 x = self.data.xrange[0]
435 self.xlabel = "Frequency (kHz)"
401 436 elif self.xaxis == "time":
402 x = self.dataOut.getAcfRange(1)
403 xlabel = "Time (ms)"
437 x = self.data.xrange[1]
438 self.xlabel = "Time (ms)"
404 439 else:
405 x = self.dataOut.getVelRange(1)
406 xlabel = "Velocity (m/s)"
440 x = self.data.xrange[2]
441 self.xlabel = "Velocity (m/s)"
407 442
408 y = self.dataOut.getHeiRange()
409 z_coh = self.data['cspc_coh']
410 z_phase = self.data['cspc_phase']
443 self.titles = []
444
445 y = self.data.heights
446 self.y = y
447 spc = self.data['spc']
448 cspc = self.data['cspc']
411 449
412 450 for n in range(self.nrows):
413 ax = self.axes[2*n]
414 ax1 = self.axes[2*n+1]
451 noise = self.data['noise'][n][-1]
452 pair = self.data.pairs[n]
453 ax = self.axes[4*n]
454 ax3 = self.axes[4*n+3]
415 455 if ax.firsttime:
416 self.xmax = self.xmax if self.xmax else np.nanmax(x)
456 self.xmax = self.xmax if self.xmax else numpy.nanmax(x)
417 457 self.xmin = self.xmin if self.xmin else -self.xmax
418 self.ymin = self.ymin if self.ymin else np.nanmin(y)
419 self.ymax = self.ymax if self.ymax else np.nanmax(y)
420 self.zmin_coh = self.zmin_coh if self.zmin_coh else 0.0
421 self.zmax_coh = self.zmax_coh if self.zmax_coh else 1.0
422 self.zmin_phase = self.zmin_phase if self.zmin_phase else -180
423 self.zmax_phase = self.zmax_phase if self.zmax_phase else 180
424
425 ax.plot = ax.pcolormesh(x, y, z_coh[n].T,
426 vmin=self.zmin_coh,
427 vmax=self.zmax_coh,
428 cmap=plt.get_cmap(self.colormap_coh)
429 )
430 divider = make_axes_locatable(ax)
431 cax = divider.new_horizontal(size='3%', pad=0.05)
432 self.figure.add_axes(cax)
433 plt.colorbar(ax.plot, cax)
434
435 ax.set_xlim(self.xmin, self.xmax)
436 ax.set_ylim(self.ymin, self.ymax)
437
438 ax.set_ylabel(self.ylabel)
439 ax.set_xlabel(xlabel)
440 ax.firsttime = False
441
442 ax1.plot = ax1.pcolormesh(x, y, z_phase[n].T,
443 vmin=self.zmin_phase,
444 vmax=self.zmax_phase,
445 cmap=plt.get_cmap(self.colormap_phase)
446 )
447 divider = make_axes_locatable(ax1)
448 cax = divider.new_horizontal(size='3%', pad=0.05)
449 self.figure.add_axes(cax)
450 plt.colorbar(ax1.plot, cax)
451
452 ax1.set_xlim(self.xmin, self.xmax)
453 ax1.set_ylim(self.ymin, self.ymax)
454
455 ax1.set_ylabel(self.ylabel)
456 ax1.set_xlabel(xlabel)
457 ax1.firsttime = False
458 self.zmin = self.zmin if self.zmin else numpy.nanmin(spc)
459 self.zmax = self.zmax if self.zmax else numpy.nanmax(spc)
460 ax.plt = ax.pcolormesh(x, y, spc[pair[0]].T,
461 vmin=self.zmin,
462 vmax=self.zmax,
463 cmap=plt.get_cmap(self.colormap)
464 )
458 465 else:
459 ax.plot.set_array(z_coh[n].T.ravel())
460 ax1.plot.set_array(z_phase[n].T.ravel())
461
462 ax.set_title('Coherence Ch{} * Ch{}'.format(self.dataOut.pairsList[n][0], self.dataOut.pairsList[n][1]), size=8)
463 ax1.set_title('Phase Ch{} * Ch{}'.format(self.dataOut.pairsList[n][0], self.dataOut.pairsList[n][1]), size=8)
464 self.saveTime = self.max_time
465
466
467 class PlotSpectraMeanData(PlotSpectraData):
466 ax.plt.set_array(spc[pair[0]].T.ravel())
467 self.titles.append('CH {}: {:3.2f}dB'.format(n, noise))
468 468
469 CODE = 'spc_mean'
470 colormap = 'jet'
471
472 def plot(self):
473
474 if self.xaxis == "frequency":
475 x = self.dataOut.getFreqRange(1)/1000.
476 xlabel = "Frequency (kHz)"
477 elif self.xaxis == "time":
478 x = self.dataOut.getAcfRange(1)
479 xlabel = "Time (ms)"
480 else:
481 x = self.dataOut.getVelRange(1)
482 xlabel = "Velocity (m/s)"
483
484 y = self.dataOut.getHeiRange()
485 z = self.data['spc']
486 mean = self.data['mean'][self.max_time]
487
488 for n, ax in enumerate(self.axes):
489
490 if ax.firsttime:
491 self.xmax = self.xmax if self.xmax else np.nanmax(x)
492 self.xmin = self.xmin if self.xmin else -self.xmax
493 self.ymin = self.ymin if self.ymin else np.nanmin(y)
494 self.ymax = self.ymax if self.ymax else np.nanmax(y)
495 self.zmin = self.zmin if self.zmin else np.nanmin(z)
496 self.zmax = self.zmax if self.zmax else np.nanmax(z)
497 ax.plt = ax.pcolormesh(x, y, z[n].T,
469 ax = self.axes[4*n+1]
470 if ax.firsttime:
471 ax.plt = ax.pcolormesh(x, y, spc[pair[1]].T,
498 472 vmin=self.zmin,
499 473 vmax=self.zmax,
500 474 cmap=plt.get_cmap(self.colormap)
501 475 )
502 ax.plt_dop = ax.plot(mean[n], y,
503 color='k')[0]
504
505 divider = make_axes_locatable(ax)
506 cax = divider.new_horizontal(size='3%', pad=0.05)
507 self.figure.add_axes(cax)
508 plt.colorbar(ax.plt, cax)
509
510 ax.set_xlim(self.xmin, self.xmax)
511 ax.set_ylim(self.ymin, self.ymax)
512
513 ax.set_ylabel(self.ylabel)
514 ax.set_xlabel(xlabel)
515
516 ax.firsttime = False
517
518 if self.showprofile:
519 ax.plt_profile= ax.ax_profile.plot(self.data['rti'][self.max_time][n], y)[0]
520 ax.ax_profile.set_xlim(self.zmin, self.zmax)
521 ax.ax_profile.set_ylim(self.ymin, self.ymax)
522 ax.ax_profile.set_xlabel('dB')
523 ax.ax_profile.grid(b=True, axis='x')
524 ax.plt_noise = ax.ax_profile.plot(numpy.repeat(self.data['noise'][self.max_time][n], len(y)), y,
525 color="k", linestyle="dashed", lw=2)[0]
526 [tick.set_visible(False) for tick in ax.ax_profile.get_yticklabels()]
527 476 else:
528 ax.plt.set_array(z[n].T.ravel())
529 ax.plt_dop.set_data(mean[n], y)
530 if self.showprofile:
531 ax.plt_profile.set_data(self.data['rti'][self.max_time][n], y)
532 ax.plt_noise.set_data(numpy.repeat(self.data['noise'][self.max_time][n], len(y)), y)
477 ax.plt.set_array(spc[pair[1]].T.ravel())
478 self.titles.append('CH {}: {:3.2f}dB'.format(n, noise))
479
480 out = cspc[n]/numpy.sqrt(spc[pair[0]]*spc[pair[1]])
481 coh = numpy.abs(out)
482 phase = numpy.arctan2(out.imag, out.real)*180/numpy.pi
483
484 ax = self.axes[4*n+2]
485 if ax.firsttime:
486 ax.plt = ax.pcolormesh(x, y, coh.T,
487 vmin=0,
488 vmax=1,
489 cmap=plt.get_cmap(self.colormap_coh)
490 )
491 else:
492 ax.plt.set_array(coh.T.ravel())
493 self.titles.append('Coherence Ch{} * Ch{}'.format(pair[0], pair[1]))
533 494
534 ax.set_title('{} - Noise: {:.2f} dB'.format(self.titles[n], self.data['noise'][self.max_time][n]),
535 size=8)
495 ax = self.axes[4*n+3]
496 if ax.firsttime:
497 ax.plt = ax.pcolormesh(x, y, phase.T,
498 vmin=-180,
499 vmax=180,
500 cmap=plt.get_cmap(self.colormap_phase)
501 )
502 else:
503 ax.plt.set_array(phase.T.ravel())
504 self.titles.append('Phase CH{} * CH{}'.format(pair[0], pair[1]))
505
536 506 self.saveTime = self.max_time
537 507
538 508
509 class PlotSpectraMeanData(PlotSpectraData):
510 '''
511 Plot for Spectra and Mean
512 '''
513 CODE = 'spc_mean'
514 colormap = 'jro'
515
516
539 517 class PlotRTIData(PlotData):
518 '''
519 Plot for RTI data
520 '''
540 521
541 522 CODE = 'rti'
542 523 colormap = 'jro'
543 524
544 525 def setup(self):
545 self.ncols = 1
546 self.nrows = self.dataOut.nChannels
547 self.width = 10
548 #TODO : arreglar la altura de la figura, esta hardcodeada.
549 #Se arreglo, testear!
550 if self.ind_plt_ch:
551 self.height = 3.2#*self.nrows if self.nrows<6 else 12
552 else:
553 self.height = 2.2*self.nrows if self.nrows<6 else 12
554
555 '''
556 if self.nrows==1:
557 self.height += 1
558 '''
526 self.xaxis = 'time'
527 self.ncols = 1
528 self.nrows = len(self.data.channels)
529 self.nplots = len(self.data.channels)
559 530 self.ylabel = 'Range [Km]'
560 self.titles = ['Channel {}'.format(x) for x in self.dataOut.channelList]
561
562 '''
563 Logica:
564 1) Si la variable ind_plt_ch es True, va a crear mas de 1 figura
565 2) guardamos "Figures" en una lista y "axes" en otra, quizas se deberia guardar el
566 axis dentro de "Figures" como un diccionario.
567 '''
568 if self.ind_plt_ch is False: #standard mode
569
570 if self.figure is None: #solo para la priemra vez
571 self.figure = plt.figure(figsize=(self.width, self.height),
572 edgecolor='k',
573 facecolor='w')
574 else:
575 self.figure.clf()
576 self.axes = []
577
578
579 for n in range(self.nrows):
580 ax = self.figure.add_subplot(self.nrows, self.ncols, n+1)
581 #ax = self.figure(n+1)
582 ax.firsttime = True
583 self.axes.append(ax)
584
585 else : #append one figure foreach channel in channelList
586 if self.figurelist == None:
587 self.figurelist = []
588 for n in range(self.nrows):
589 self.figure = plt.figure(figsize=(self.width, self.height),
590 edgecolor='k',
591 facecolor='w')
592 #add always one subplot
593 self.figurelist.append(self.figure)
594
595 else : # cada dia nuevo limpia el axes, pero mantiene el figure
596 for eachfigure in self.figurelist:
597 eachfigure.clf() # eliminaria todas las figuras de la lista?
598 self.axes = []
599
600 for eachfigure in self.figurelist:
601 ax = eachfigure.add_subplot(1,1,1) #solo 1 axis por figura
602 #ax = self.figure(n+1)
603 ax.firsttime = True
604 #Cada figura tiene un distinto puntero
605 self.axes.append(ax)
606 #plt.close(eachfigure)
607
531 self.cb_label = 'dB'
532 self.titles = ['{} Channel {}'.format(self.CODE.upper(), x) for x in range(self.nrows)]
608 533
609 534 def plot(self):
535 self.x = self.data.times
536 self.y = self.data.heights
537 self.z = self.data[self.CODE]
538 self.z = numpy.ma.masked_invalid(self.z)
610 539
611 if self.ind_plt_ch is False: #standard mode
612 self.x = np.array(self.times)
613 self.y = self.dataOut.getHeiRange()
614 self.z = []
615
616 for ch in range(self.nrows):
617 self.z.append([self.data[self.CODE][t][ch] for t in self.times])
618
619 self.z = np.array(self.z)
620 for n, ax in enumerate(self.axes):
621 x, y, z = self.fill_gaps(*self.decimate())
622 xmin = self.min_time
623 xmax = xmin+self.xrange*60*60
624 self.zmin = self.zmin if self.zmin else np.min(self.z)
625 self.zmax = self.zmax if self.zmax else np.max(self.z)
626 if ax.firsttime:
627 self.ymin = self.ymin if self.ymin else np.nanmin(self.y)
628 self.ymax = self.ymax if self.ymax else np.nanmax(self.y)
629 plot = ax.pcolormesh(x, y, z[n].T,
630 vmin=self.zmin,
631 vmax=self.zmax,
632 cmap=plt.get_cmap(self.colormap)
633 )
634 divider = make_axes_locatable(ax)
635 cax = divider.new_horizontal(size='2%', pad=0.05)
636 self.figure.add_axes(cax)
637 plt.colorbar(plot, cax)
638 ax.set_ylim(self.ymin, self.ymax)
639 ax.xaxis.set_major_formatter(FuncFormatter(func))
640 ax.xaxis.set_major_locator(LinearLocator(6))
641 ax.set_ylabel(self.ylabel)
642 if self.xmin is None:
643 xmin = self.min_time
644 else:
645 xmin = (datetime.datetime.combine(self.dataOut.datatime.date(),
646 datetime.time(self.xmin, 0, 0))-d1970).total_seconds()
647 ax.set_xlim(xmin, xmax)
648 ax.firsttime = False
649 else:
650 ax.collections.remove(ax.collections[0])
651 ax.set_xlim(xmin, xmax)
652 plot = ax.pcolormesh(x, y, z[n].T,
653 vmin=self.zmin,
654 vmax=self.zmax,
655 cmap=plt.get_cmap(self.colormap)
656 )
657 ax.set_title('{} {}'.format(self.titles[n],
658 datetime.datetime.fromtimestamp(self.max_time).strftime('%y/%m/%d %H:%M:%S')),
659 size=8)
660
661 self.saveTime = self.min_time
662 else :
663 self.x = np.array(self.times)
664 self.y = self.dataOut.getHeiRange()
665 self.z = []
666
667 for ch in range(self.nrows):
668 self.z.append([self.data[self.CODE][t][ch] for t in self.times])
669
670 self.z = np.array(self.z)
671 for n, eachfigure in enumerate(self.figurelist): #estaba ax in axes
672
673 x, y, z = self.fill_gaps(*self.decimate())
674 xmin = self.min_time
675 xmax = xmin+self.xrange*60*60
676 self.zmin = self.zmin if self.zmin else np.min(self.z)
677 self.zmax = self.zmax if self.zmax else np.max(self.z)
678 if self.axes[n].firsttime:
679 self.ymin = self.ymin if self.ymin else np.nanmin(self.y)
680 self.ymax = self.ymax if self.ymax else np.nanmax(self.y)
681 plot = self.axes[n].pcolormesh(x, y, z[n].T,
682 vmin=self.zmin,
683 vmax=self.zmax,
684 cmap=plt.get_cmap(self.colormap)
685 )
686 divider = make_axes_locatable(self.axes[n])
687 cax = divider.new_horizontal(size='2%', pad=0.05)
688 eachfigure.add_axes(cax)
689 #self.figure2.add_axes(cax)
690 plt.colorbar(plot, cax)
691 self.axes[n].set_ylim(self.ymin, self.ymax)
692
693 self.axes[n].xaxis.set_major_formatter(FuncFormatter(func))
694 self.axes[n].xaxis.set_major_locator(LinearLocator(6))
695
696 self.axes[n].set_ylabel(self.ylabel)
697
698 if self.xmin is None:
699 xmin = self.min_time
700 else:
701 xmin = (datetime.datetime.combine(self.dataOut.datatime.date(),
702 datetime.time(self.xmin, 0, 0))-d1970).total_seconds()
703
704 self.axes[n].set_xlim(xmin, xmax)
705 self.axes[n].firsttime = False
706 else:
707 self.axes[n].collections.remove(self.axes[n].collections[0])
708 self.axes[n].set_xlim(xmin, xmax)
709 plot = self.axes[n].pcolormesh(x, y, z[n].T,
710 vmin=self.zmin,
711 vmax=self.zmax,
712 cmap=plt.get_cmap(self.colormap)
713 )
714 self.axes[n].set_title('{} {}'.format(self.titles[n],
715 datetime.datetime.fromtimestamp(self.max_time).strftime('%y/%m/%d %H:%M:%S')),
716 size=8)
540 for n, ax in enumerate(self.axes):
541 x, y, z = self.fill_gaps(*self.decimate())
542 self.zmin = self.zmin if self.zmin else numpy.min(self.z)
543 self.zmax = self.zmax if self.zmax else numpy.max(self.z)
544 if ax.firsttime:
545 ax.plt = ax.pcolormesh(x, y, z[n].T,
546 vmin=self.zmin,
547 vmax=self.zmax,
548 cmap=plt.get_cmap(self.colormap)
549 )
550 if self.showprofile:
551 ax.plot_profile= self.pf_axes[n].plot(self.data['rti'][n][-1], self.y)[0]
552 ax.plot_noise = self.pf_axes[n].plot(numpy.repeat(self.data['noise'][n][-1], len(self.y)), self.y,
553 color="k", linestyle="dashed", lw=1)[0]
554 else:
555 ax.collections.remove(ax.collections[0])
556 ax.plt = ax.pcolormesh(x, y, z[n].T,
557 vmin=self.zmin,
558 vmax=self.zmax,
559 cmap=plt.get_cmap(self.colormap)
560 )
561 if self.showprofile:
562 ax.plot_profile.set_data(self.data['rti'][n][-1], self.y)
563 ax.plot_noise.set_data(numpy.repeat(self.data['noise'][n][-1], len(self.y)), self.y)
717 564
718 self.saveTime = self.min_time
565 self.saveTime = self.min_time
719 566
720 567
721 568 class PlotCOHData(PlotRTIData):
569 '''
570 Plot for Coherence data
571 '''
722 572
723 573 CODE = 'coh'
724 574
725 575 def setup(self):
726
576 self.xaxis = 'time'
727 577 self.ncols = 1
728 self.nrows = self.dataOut.nPairs
729 self.width = 10
730 self.height = 2.2*self.nrows if self.nrows<6 else 12
731 self.ind_plt_ch = False #just for coherence and phase
732 if self.nrows==1:
733 self.height += 1
734 self.ylabel = 'Range [Km]'
735 self.titles = ['{} Ch{} * Ch{}'.format(self.CODE.upper(), x[0], x[1]) for x in self.dataOut.pairsList]
736
737 if self.figure is None:
738 self.figure = plt.figure(figsize=(self.width, self.height),
739 edgecolor='k',
740 facecolor='w')
578 self.nrows = len(self.data.pairs)
579 self.nplots = len(self.data.pairs)
580 self.ylabel = 'Range [Km]'
581 if self.CODE == 'coh':
582 self.cb_label = ''
583 self.titles = ['Coherence Map Ch{} * Ch{}'.format(x[0], x[1]) for x in self.data.pairs]
741 584 else:
742 self.figure.clf()
743 self.axes = []
585 self.cb_label = 'Degrees'
586 self.titles = ['Phase Map Ch{} * Ch{}'.format(x[0], x[1]) for x in self.data.pairs]
744 587
745 for n in range(self.nrows):
746 ax = self.figure.add_subplot(self.nrows, self.ncols, n+1)
747 ax.firsttime = True
748 self.axes.append(ax)
588
589 class PlotPHASEData(PlotCOHData):
590 '''
591 Plot for Phase map data
592 '''
593
594 CODE = 'phase'
595 colormap = 'seismic'
749 596
750 597
751 598 class PlotNoiseData(PlotData):
599 '''
600 Plot for noise
601 '''
602
752 603 CODE = 'noise'
753 604
754 605 def setup(self):
755
606 self.xaxis = 'time'
756 607 self.ncols = 1
757 608 self.nrows = 1
758 self.width = 10
759 self.height = 3.2
609 self.nplots = 1
760 610 self.ylabel = 'Intensity [dB]'
761 611 self.titles = ['Noise']
762
763 if self.figure is None:
764 self.figure = plt.figure(figsize=(self.width, self.height),
765 edgecolor='k',
766 facecolor='w')
767 else:
768 self.figure.clf()
769 self.axes = []
770
771 self.ax = self.figure.add_subplot(self.nrows, self.ncols, 1)
772 self.ax.firsttime = True
612 self.colorbar = False
773 613
774 614 def plot(self):
775 615
776 x = self.times
616 x = self.data.times
777 617 xmin = self.min_time
778 618 xmax = xmin+self.xrange*60*60
779 if self.ax.firsttime:
780 for ch in self.dataOut.channelList:
781 y = [self.data[self.CODE][t][ch] for t in self.times]
782 self.ax.plot(x, y, lw=1, label='Ch{}'.format(ch))
783 self.ax.firsttime = False
784 self.ax.xaxis.set_major_formatter(FuncFormatter(func))
785 self.ax.xaxis.set_major_locator(LinearLocator(6))
786 self.ax.set_ylabel(self.ylabel)
619 Y = self.data[self.CODE]
620
621 if self.axes[0].firsttime:
622 for ch in self.data.channels:
623 y = Y[ch]
624 self.axes[0].plot(x, y, lw=1, label='Ch{}'.format(ch))
787 625 plt.legend()
788 626 else:
789 for ch in self.dataOut.channelList:
790 y = [self.data[self.CODE][t][ch] for t in self.times]
791 self.ax.lines[ch].set_data(x, y)
792
793 self.ax.set_xlim(xmin, xmax)
794 self.ax.set_ylim(min(y)-5, max(y)+5)
627 for ch in self.data.channels:
628 y = Y[ch]
629 self.axes[0].lines[ch].set_data(x, y)
630
631 self.ymin = numpy.nanmin(Y) - 5
632 self.ymax = numpy.nanmax(Y) + 5
795 633 self.saveTime = self.min_time
796 634
797 635
798 class PlotWindProfilerData(PlotRTIData):
799
800 CODE = 'wind'
801 colormap = 'seismic'
802
803 def setup(self):
804 self.ncols = 1
805 self.nrows = self.dataOut.data_output.shape[0]
806 self.width = 10
807 self.height = 2.2*self.nrows
808 self.ylabel = 'Height [Km]'
809 self.titles = ['Zonal Wind' ,'Meridional Wind', 'Vertical Wind']
810 self.clabels = ['Velocity (m/s)','Velocity (m/s)','Velocity (cm/s)']
811 self.windFactor = [1, 1, 100]
812
813 if self.figure is None:
814 self.figure = plt.figure(figsize=(self.width, self.height),
815 edgecolor='k',
816 facecolor='w')
817 else:
818 self.figure.clf()
819 self.axes = []
820
821 for n in range(self.nrows):
822 ax = self.figure.add_subplot(self.nrows, self.ncols, n+1)
823 ax.firsttime = True
824 self.axes.append(ax)
825
826 def plot(self):
827
828 self.x = np.array(self.times)
829 self.y = self.dataOut.heightList
830 self.z = []
831
832 for ch in range(self.nrows):
833 self.z.append([self.data['output'][t][ch] for t in self.times])
834
835 self.z = np.array(self.z)
836 self.z = numpy.ma.masked_invalid(self.z)
837
838 cmap=plt.get_cmap(self.colormap)
839 cmap.set_bad('black', 1.)
840
841 for n, ax in enumerate(self.axes):
842 x, y, z = self.fill_gaps(*self.decimate())
843 xmin = self.min_time
844 xmax = xmin+self.xrange*60*60
845 if ax.firsttime:
846 self.ymin = self.ymin if self.ymin else np.nanmin(self.y)
847 self.ymax = self.ymax if self.ymax else np.nanmax(self.y)
848 self.zmax = self.zmax if self.zmax else numpy.nanmax(abs(self.z[:-1, :]))
849 self.zmin = self.zmin if self.zmin else -self.zmax
850
851 plot = ax.pcolormesh(x, y, z[n].T*self.windFactor[n],
852 vmin=self.zmin,
853 vmax=self.zmax,
854 cmap=cmap
855 )
856 divider = make_axes_locatable(ax)
857 cax = divider.new_horizontal(size='2%', pad=0.05)
858 self.figure.add_axes(cax)
859 cb = plt.colorbar(plot, cax)
860 cb.set_label(self.clabels[n])
861 ax.set_ylim(self.ymin, self.ymax)
862
863 ax.xaxis.set_major_formatter(FuncFormatter(func))
864 ax.xaxis.set_major_locator(LinearLocator(6))
865
866 ax.set_ylabel(self.ylabel)
867
868 ax.set_xlim(xmin, xmax)
869 ax.firsttime = False
870 else:
871 ax.collections.remove(ax.collections[0])
872 ax.set_xlim(xmin, xmax)
873 plot = ax.pcolormesh(x, y, z[n].T*self.windFactor[n],
874 vmin=self.zmin,
875 vmax=self.zmax,
876 cmap=plt.get_cmap(self.colormap)
877 )
878 ax.set_title('{} {}'.format(self.titles[n],
879 datetime.datetime.fromtimestamp(self.max_time).strftime('%y/%m/%d %H:%M:%S')),
880 size=8)
881
882 self.saveTime = self.min_time
883
884
885 636 class PlotSNRData(PlotRTIData):
637 '''
638 Plot for SNR Data
639 '''
640
886 641 CODE = 'snr'
887 642 colormap = 'jet'
888 643
644
889 645 class PlotDOPData(PlotRTIData):
646 '''
647 Plot for DOPPLER Data
648 '''
649
890 650 CODE = 'dop'
891 651 colormap = 'jet'
892 652
893 653
894 class PlotPHASEData(PlotCOHData):
895 CODE = 'phase'
896 colormap = 'seismic'
897
898
899 654 class PlotSkyMapData(PlotData):
655 '''
656 Plot for meteors detection data
657 '''
900 658
901 659 CODE = 'met'
902 660
@@ -923,7 +681,7 class PlotSkyMapData(PlotData):
923 681
924 682 def plot(self):
925 683
926 arrayParameters = np.concatenate([self.data['param'][t] for t in self.times])
684 arrayParameters = numpy.concatenate([self.data['param'][t] for t in self.data.times])
927 685 error = arrayParameters[:,-1]
928 686 indValid = numpy.where(error == 0)[0]
929 687 finalMeteor = arrayParameters[indValid,:]
@@ -953,3 +711,72 class PlotSkyMapData(PlotData):
953 711 self.ax.set_title(title, size=8)
954 712
955 713 self.saveTime = self.max_time
714
715 class PlotParamData(PlotRTIData):
716 '''
717 Plot for data_param object
718 '''
719
720 CODE = 'param'
721 colormap = 'seismic'
722
723 def setup(self):
724 self.xaxis = 'time'
725 self.ncols = 1
726 self.nrows = self.data.shape(self.CODE)[0]
727 self.nplots = self.nrows
728 if self.showSNR:
729 self.nrows += 1
730
731 self.ylabel = 'Height [Km]'
732 self.titles = self.data.parameters \
733 if self.data.parameters else ['Param {}'.format(x) for x in xrange(self.nrows)]
734 if self.showSNR:
735 self.titles.append('SNR')
736
737 def plot(self):
738 self.data.normalize_heights()
739 self.x = self.data.times
740 self.y = self.data.heights
741 if self.showSNR:
742 self.z = numpy.concatenate(
743 (self.data[self.CODE], self.data['snr'])
744 )
745 else:
746 self.z = self.data[self.CODE]
747
748 self.z = numpy.ma.masked_invalid(self.z)
749
750 for n, ax in enumerate(self.axes):
751
752 x, y, z = self.fill_gaps(*self.decimate())
753
754 if ax.firsttime:
755 if self.zlimits is not None:
756 self.zmin, self.zmax = self.zlimits[n]
757 self.zmax = self.zmax if self.zmax is not None else numpy.nanmax(abs(self.z[:-1, :]))
758 self.zmin = self.zmin if self.zmin is not None else -self.zmax
759 ax.plt = ax.pcolormesh(x, y, z[n, :, :].T*self.factors[n],
760 vmin=self.zmin,
761 vmax=self.zmax,
762 cmap=self.cmaps[n]
763 )
764 else:
765 if self.zlimits is not None:
766 self.zmin, self.zmax = self.zlimits[n]
767 ax.collections.remove(ax.collections[0])
768 ax.plt = ax.pcolormesh(x, y, z[n, :, :].T*self.factors[n],
769 vmin=self.zmin,
770 vmax=self.zmax,
771 cmap=self.cmaps[n]
772 )
773
774 self.saveTime = self.min_time
775
776 class PlotOuputData(PlotParamData):
777 '''
778 Plot data_output object
779 '''
780
781 CODE = 'output'
782 colormap = 'seismic' No newline at end of file
Show file before | Show file after | Hide comments
@@ -225,7 +225,6 class MomentsPlot(Figure):
225 225 WIDTHPROF = None
226 226 HEIGHTPROF = None
227 227 PREFIX = 'prm'
228
229 228 def __init__(self, **kwargs):
230 229 Figure.__init__(self, **kwargs)
231 230 self.isConfig = False
@@ -1626,6 +1625,7 class PhasePlot(Figure):
1626 1625
1627 1626 PREFIX = 'mphase'
1628 1627
1628
1629 1629 def __init__(self, **kwargs):
1630 1630 Figure.__init__(self, **kwargs)
1631 1631 self.timerange = 24*60*60
Show file before | Show file after | Hide comments
@@ -10,6 +10,7 import numpy
10 10 from figure import Figure, isRealtime, isTimeInHourRange
11 11 from plotting_codes import *
12 12
13
13 14 class SpectraPlot(Figure):
14 15
15 16 isConfig = None
Show file before | Show file after | Hide comments
@@ -124,7 +124,7 class PlotManager():
124 124 subtitle = "%s:\n" %(name)
125 125 subtitle += "Hostname: %s\n" %socket.gethostbyname(socket.gethostname())
126 126 subtitle += "Working directory: %s\n" %os.path.abspath("./")
127 # subtitle += "Configuration file: %s\n" %self.filename
127 # subtitle += "Configuration file: %s\n" %self.filename
128 128 subtitle += "Time: %s\n" %str(datetime.datetime.now())
129 129
130 130 adminObj = schainpy.admin.SchainNotify()
Show file before | Show file after | Hide comments
@@ -267,7 +267,7 class AMISRReader(ProcessingUnit):
267 267 self.dirnameList = new_dirnameList
268 268 return 1
269 269
270 def __searchFilesOnline(self,
270 def searchFilesOnLine(self,
271 271 path,
272 272 walk=True):
273 273
@@ -287,7 +287,7 class AMISRReader(ProcessingUnit):
287 287 return
288 288
289 289
290 def __searchFilesOffline(self,
290 def searchFilesOffLine(self,
291 291 path,
292 292 startDate,
293 293 endDate,
@@ -494,9 +494,9 class AMISRReader(ProcessingUnit):
494 494 self.online = online
495 495 if not(online):
496 496 #Busqueda de archivos offline
497 self.__searchFilesOffline(path, startDate, endDate, startTime, endTime, walk)
497 self.searchFilesOffLine(path, startDate, endDate, startTime, endTime, walk)
498 498 else:
499 self.__searchFilesOnline(path, walk)
499 self.searchFilesOnLine(path, walk)
500 500
501 501 if not(self.filenameList):
502 502 print "There is no files into the folder: %s"%(path)
Show file before | Show file after | Hide comments
@@ -542,7 +542,6 class JRODataIO:
542 542
543 543 class JRODataReader(JRODataIO):
544 544
545
546 545 online = 0
547 546
548 547 realtime = 0
@@ -579,7 +578,6 class JRODataReader(JRODataIO):
579 578
580 579 selBlocktime = None
581 580
582
583 581 def __init__(self):
584 582
585 583 """
@@ -603,19 +601,18 class JRODataReader(JRODataIO):
603 601
604 602 raise NotImplementedError
605 603
606 def __searchFilesOffLine(self,
607 path,
608 startDate=None,
609 endDate=None,
610 startTime=datetime.time(0,0,0),
611 endTime=datetime.time(23,59,59),
612 set=None,
613 expLabel='',
614 ext='.r',
615 queue=None,
616 cursor=None,
617 skip=None,
618 walk=True):
604 def searchFilesOffLine(self,
605 path,
606 startDate=None,
607 endDate=None,
608 startTime=datetime.time(0,0,0),
609 endTime=datetime.time(23,59,59),
610 set=None,
611 expLabel='',
612 ext='.r',
613 cursor=None,
614 skip=None,
615 walk=True):
619 616
620 617 self.filenameList = []
621 618 self.datetimeList = []
@@ -625,8 +622,7 class JRODataReader(JRODataIO):
625 622 dateList, pathList = self.findDatafiles(path, startDate, endDate, expLabel, ext, walk, include_path=True)
626 623
627 624 if dateList == []:
628 # print "[Reading] Date range selected invalid [%s - %s]: No *%s files in %s)" %(startDate, endDate, ext, path)
629 return None, None
625 return [], []
630 626
631 627 if len(dateList) > 1:
632 628 print "[Reading] Data found for date range [%s - %s]: total days = %d" %(startDate, endDate, len(dateList))
@@ -637,7 +633,6 class JRODataReader(JRODataIO):
637 633 datetimeList = []
638 634
639 635 for thisPath in pathList:
640 # thisPath = pathList[pathDict[file]]
641 636
642 637 fileList = glob.glob1(thisPath, "*%s" %ext)
643 638 fileList.sort()
@@ -645,10 +640,8 class JRODataReader(JRODataIO):
645 640 skippedFileList = []
646 641
647 642 if cursor is not None and skip is not None:
648 # if cursor*skip > len(fileList):
643
649 644 if skip == 0:
650 if queue is not None:
651 queue.put(len(fileList))
652 645 skippedFileList = []
653 646 else:
654 647 skippedFileList = fileList[cursor*skip: cursor*skip + skip]
@@ -673,13 +666,13 class JRODataReader(JRODataIO):
673 666
674 667 if not(filenameList):
675 668 print "[Reading] Time range selected invalid [%s - %s]: No *%s files in %s)" %(startTime, endTime, ext, path)
676 return None, None
669 return [], []
677 670
678 671 print "[Reading] %d file(s) was(were) found in time range: %s - %s" %(len(filenameList), startTime, endTime)
679 672 print
680 673
681 for i in range(len(filenameList)):
682 print "[Reading] %s -> [%s]" %(filenameList[i], datetimeList[i].ctime())
674 # for i in range(len(filenameList)):
675 # print "[Reading] %s -> [%s]" %(filenameList[i], datetimeList[i].ctime())
683 676
684 677 self.filenameList = filenameList
685 678 self.datetimeList = datetimeList
@@ -1043,8 +1036,7 class JRODataReader(JRODataIO):
1043 1036
1044 1037 #Skip block out of startTime and endTime
1045 1038 while True:
1046 if not(self.__setNewBlock()):
1047 print 'returning'
1039 if not(self.__setNewBlock()):
1048 1040 return 0
1049 1041
1050 1042 if not(self.readBlock()):
@@ -1273,7 +1265,6 class JRODataReader(JRODataIO):
1273 1265 realtime=False,
1274 1266 blocksize=None,
1275 1267 blocktime=None,
1276 queue=None,
1277 1268 skip=None,
1278 1269 cursor=None,
1279 1270 warnings=True,
@@ -1322,19 +1313,13 class JRODataReader(JRODataIO):
1322 1313 last_set = None
1323 1314 else:
1324 1315 print "[Reading] Searching files in offline mode ..."
1325 pathList, filenameList = self.__searchFilesOffLine(path, startDate=startDate, endDate=endDate,
1316 pathList, filenameList = self.searchFilesOffLine(path, startDate=startDate, endDate=endDate,
1326 1317 startTime=startTime, endTime=endTime,
1327 1318 set=set, expLabel=expLabel, ext=ext,
1328 1319 walk=walk, cursor=cursor,
1329 skip=skip, queue=queue)
1320 skip=skip)
1330 1321
1331 1322 if not(pathList):
1332 # print "[Reading] No *%s files in %s (%s - %s)"%(ext, path,
1333 # datetime.datetime.combine(startDate,startTime).ctime(),
1334 # datetime.datetime.combine(endDate,endTime).ctime())
1335
1336 # sys.exit(-1)
1337
1338 1323 self.fileIndex = -1
1339 1324 self.pathList = []
1340 1325 self.filenameList = []
@@ -1451,7 +1436,6 class JRODataReader(JRODataIO):
1451 1436
1452 1437 self.__printInfo = False
1453 1438
1454
1455 1439 def run(self,
1456 1440 path=None,
1457 1441 startDate=None,
@@ -1469,7 +1453,6 class JRODataReader(JRODataIO):
1469 1453 realtime=False,
1470 1454 blocksize=None,
1471 1455 blocktime=None,
1472 queue=None,
1473 1456 skip=None,
1474 1457 cursor=None,
1475 1458 warnings=True,
@@ -1477,29 +1460,27 class JRODataReader(JRODataIO):
1477 1460 verbose=True, **kwargs):
1478 1461
1479 1462 if not(self.isConfig):
1480 # self.dataOut = dataOut
1481 self.setup( path=path,
1482 startDate=startDate,
1483 endDate=endDate,
1484 startTime=startTime,
1485 endTime=endTime,
1486 set=set,
1487 expLabel=expLabel,
1488 ext=ext,
1489 online=online,
1490 delay=delay,
1491 walk=walk,
1492 getblock=getblock,
1493 nTxs=nTxs,
1494 realtime=realtime,
1495 blocksize=blocksize,
1496 blocktime=blocktime,
1497 queue=queue,
1498 skip=skip,
1499 cursor=cursor,
1500 warnings=warnings,
1501 server=server,
1502 verbose=verbose)
1463 self.setup(path=path,
1464 startDate=startDate,
1465 endDate=endDate,
1466 startTime=startTime,
1467 endTime=endTime,
1468 set=set,
1469 expLabel=expLabel,
1470 ext=ext,
1471 online=online,
1472 delay=delay,
1473 walk=walk,
1474 getblock=getblock,
1475 nTxs=nTxs,
1476 realtime=realtime,
1477 blocksize=blocksize,
1478 blocktime=blocktime,
1479 skip=skip,
1480 cursor=cursor,
1481 warnings=warnings,
1482 server=server,
1483 verbose=verbose)
1503 1484 self.isConfig = True
1504 1485 if server is None:
1505 1486 self.getData()
Show file before | Show file after | Hide comments
@@ -119,7 +119,6 class Metadata(object):
119 119 self.parmConfObjList.append(parmConfObj)
120 120
121 121 class FitsWriter(Operation):
122
123 122 def __init__(self, **kwargs):
124 123 Operation.__init__(self, **kwargs)
125 124 self.isConfig = False
@@ -276,9 +275,9 class FitsWriter(Operation):
276 275 self.setNextFile()
277 276 self.writeNextBlock()
278 277
279 def run(self, dataOut, **kwargs):
278 def run(self, dataOut, path, dataBlocksPerFile=100, metadatafile=None, **kwargs):
280 279 if not(self.isConfig):
281 self.setup(dataOut, **kwargs)
280 self.setup(dataOut, path, dataBlocksPerFile=dataBlocksPerFile, metadatafile=metadatafile, **kwargs)
282 281 self.isConfig = True
283 282 self.putData()
284 283
@@ -454,7 +453,7 class FitsReader(ProcessingUnit):
454 453 # self.blockIndex = 1
455 454 return 1
456 455
457 def __searchFilesOffLine(self,
456 def searchFilesOffLine(self,
458 457 path,
459 458 startDate,
460 459 endDate,
@@ -560,7 +559,7 class FitsReader(ProcessingUnit):
560 559
561 560 if not(online):
562 561 print "Searching files in offline mode ..."
563 pathList, filenameList = self.__searchFilesOffLine(path, startDate=startDate, endDate=endDate,
562 pathList, filenameList = self.searchFilesOffLine(path, startDate=startDate, endDate=endDate,
564 563 startTime=startTime, endTime=endTime,
565 564 set=set, expLabel=expLabel, ext=ext,
566 565 walk=walk)
Show file before | Show file after | Hide comments
@@ -415,7 +415,7 class HFReader(ProcessingUnit):
415 415
416 416
417 417
418 def __searchFilesOffline(self,
418 def searchFilesOffLine(self,
419 419 path,
420 420 startDate,
421 421 endDate,
@@ -438,7 +438,7 class HFReader(ProcessingUnit):
438 438
439 439 return
440 440
441 def __searchFilesOnline(self,
441 def searchFilesOnLine(self,
442 442 path,
443 443 expLabel= "",
444 444 ext=None,
@@ -636,10 +636,10 class HFReader(ProcessingUnit):
636 636 if not(online):
637 637 print "Searching files in offline mode..."
638 638
639 self.__searchFilesOffline(path, startDate, endDate, ext, startTime, endTime, walk)
639 self.searchFilesOffLine(path, startDate, endDate, ext, startTime, endTime, walk)
640 640 else:
641 641 print "Searching files in online mode..."
642 self.__searchFilesOnline(path, walk,ext,set=set)
642 self.searchFilesOnLine(path, walk,ext,set=set)
643 643 if set==None:
644 644 pass
645 645 else:
@@ -647,7 +647,7 class HFReader(ProcessingUnit):
647 647
648 648 # for nTries in range(self.nTries):
649 649 #
650 # fullpath,file,year,month,day,set = self.__searchFilesOnline(path=path,expLabel=expLabel,ext=ext, walk=walk,set=set)
650 # fullpath,file,year,month,day,set = self.searchFilesOnLine(path=path,expLabel=expLabel,ext=ext, walk=walk,set=set)
651 651 #
652 652 # if fullpath:
653 653 # break
Show file before | Show file after | Hide comments
@@ -106,9 +106,9 class AMISRReader(ProcessingUnit):
106 106 #self.findFiles()
107 107 if not(online):
108 108 #Busqueda de archivos offline
109 self.__searchFilesOffline(path, startDate, endDate, startTime, endTime, walk)
109 self.searchFilesOffLine(path, startDate, endDate, startTime, endTime, walk)
110 110 else:
111 self.__searchFilesOnline(path, startDate, endDate, startTime,endTime,walk)
111 self.searchFilesOnLine(path, startDate, endDate, startTime,endTime,walk)
112 112
113 113 if not(self.filenameList):
114 114 print "There is no files into the folder: %s"%(path)
@@ -329,7 +329,7 class AMISRReader(ProcessingUnit):
329 329 self.dirnameList = new_dirnameList
330 330 return 1
331 331
332 def __searchFilesOnline(self, path, startDate, endDate, startTime=datetime.time(0,0,0),
332 def searchFilesOnLine(self, path, startDate, endDate, startTime=datetime.time(0,0,0),
333 333 endTime=datetime.time(23,59,59),walk=True):
334 334
335 335 if endDate ==None:
@@ -349,7 +349,7 class AMISRReader(ProcessingUnit):
349 349 return
350 350
351 351
352 def __searchFilesOffline(self,
352 def searchFilesOffLine(self,
353 353 path,
354 354 startDate,
355 355 endDate,
Show file before | Show file after | Hide comments
@@ -97,7 +97,7 class ParamReader(ProcessingUnit):
97 97 self.timezone = 'lt'
98 98
99 99 print "[Reading] Searching files in offline mode ..."
100 pathList, filenameList = self.__searchFilesOffLine(path, startDate=startDate, endDate=endDate,
100 pathList, filenameList = self.searchFilesOffLine(path, startDate=startDate, endDate=endDate,
101 101 startTime=startTime, endTime=endTime,
102 102 ext=ext, walk=walk)
103 103
@@ -115,7 +115,7 class ParamReader(ProcessingUnit):
115 115
116 116 return
117 117
118 def __searchFilesOffLine(self,
118 def searchFilesOffLine(self,
119 119 path,
120 120 startDate=None,
121 121 endDate=None,
@@ -607,29 +607,18 class ParamWriter(Operation):
607 607 self.isConfig = False
608 608 return
609 609
610 def setup(self, dataOut, **kwargs):
611
612 self.path = kwargs['path']
613 self.setType = kwargs.get('setType', None)
614
615 if kwargs.has_key('blocksPerFile'):
616 self.blocksPerFile = kwargs['blocksPerFile']
617 else:
618 self.blocksPerFile = 10
619
620 self.metadataList = kwargs['metadataList']
621 self.dataList = kwargs['dataList']
610 def setup(self, dataOut, path=None, blocksPerFile=10, metadataList=None, dataList=None, mode=None, **kwargs):
611 self.path = path
612 self.blocksPerFile = blocksPerFile
613 self.metadataList = metadataList
614 self.dataList = dataList
622 615 self.dataOut = dataOut
623
624 if kwargs.has_key('mode'):
625 mode = kwargs['mode']
626
627 if type(mode) == int:
628 mode = numpy.zeros(len(self.dataList)) + mode
629 else:
630 mode = numpy.ones(len(self.dataList))
631
632 616 self.mode = mode
617
618 if self.mode is not None:
619 self.mode = numpy.zeros(len(self.dataList)) + mode
620 else:
621 self.mode = numpy.ones(len(self.dataList))
633 622
634 623 arrayDim = numpy.zeros((len(self.dataList),5))
635 624
@@ -1089,10 +1078,11 class ParamWriter(Operation):
1089 1078 self.fp.close()
1090 1079 return
1091 1080
1092 def run(self, dataOut, **kwargs):
1081 def run(self, dataOut, path=None, blocksPerFile=10, metadataList=None, dataList=None, mode=None, **kwargs):
1093 1082
1094 1083 if not(self.isConfig):
1095 flagdata = self.setup(dataOut, **kwargs)
1084 flagdata = self.setup(dataOut, path=path, blocksPerFile=blocksPerFile,
1085 metadataList=metadataList, dataList=dataList, mode=mode, **kwargs)
1096 1086
1097 1087 if not(flagdata):
1098 1088 return
Show file before | Show file after | Hide comments
@@ -180,13 +180,13 class VoltageReader(JRODataReader, ProcessingUnit):
180 180
181 181 def getBlockDimension(self):
182 182 """
183 Obtiene la cantidad de puntos a leer por cada bloque de datos
183 Obtiene la cantidad de puntos a leer por cada bloque de datos
184
185 Affected:
186 self.blocksize
184 187
185 Affected:
186 self.blocksize
187
188 Return:
189 None
188 Return:
189 None
190 190 """
191 191 pts2read = self.processingHeaderObj.profilesPerBlock * self.processingHeaderObj.nHeights * self.systemHeaderObj.nChannels
192 192 self.blocksize = pts2read
@@ -195,26 +195,26 class VoltageReader(JRODataReader, ProcessingUnit):
195 195
196 196 def readBlock(self):
197 197 """
198 readBlock lee el bloque de datos desde la posicion actual del puntero del archivo
199 (self.fp) y actualiza todos los parametros relacionados al bloque de datos
200 (metadata + data). La data leida es almacenada en el buffer y el contador del buffer
201 es seteado a 0
202
203 Inputs:
204 None
205
206 Return:
207 None
208
209 Affected:
210 self.profileIndex
211 self.datablock
212 self.flagIsNewFile
213 self.flagIsNewBlock
214 self.nTotalBlocks
215
216 Exceptions:
217 Si un bloque leido no es un bloque valido
198 readBlock lee el bloque de datos desde la posicion actual del puntero del archivo
199 (self.fp) y actualiza todos los parametros relacionados al bloque de datos
200 (metadata + data). La data leida es almacenada en el buffer y el contador del buffer
201 es seteado a 0
202
203 Inputs:
204 None
205
206 Return:
207 None
208
209 Affected:
210 self.profileIndex
211 self.datablock
212 self.flagIsNewFile
213 self.flagIsNewBlock
214 self.nTotalBlocks
215
216 Exceptions:
217 Si un bloque leido no es un bloque valido
218 218 """
219 219
220 220 # if self.server is not None:
@@ -237,7 +237,7 class VoltageReader(JRODataReader, ProcessingUnit):
237 237 if self.waitDataBlock(pointer_location=current_pointer_location):
238 238 junk = numpy.fromfile( self.fp, self.dtype, self.blocksize )
239 239 junk = junk.reshape( (self.processingHeaderObj.profilesPerBlock, self.processingHeaderObj.nHeights, self.systemHeaderObj.nChannels) )
240 # return 0
240 # return 0
241 241
242 242 #Dimensions : nChannels, nProfiles, nSamples
243 243
@@ -267,15 +267,15 class VoltageReader(JRODataReader, ProcessingUnit):
267 267
268 268 #Time interval and code are propierties of dataOut. Its value depends of radarControllerHeaderObj.
269 269
270 # self.dataOut.timeInterval = self.radarControllerHeaderObj.ippSeconds * self.processingHeaderObj.nCohInt
271 #
272 # if self.radarControllerHeaderObj.code is not None:
273 #
274 # self.dataOut.nCode = self.radarControllerHeaderObj.nCode
275 #
276 # self.dataOut.nBaud = self.radarControllerHeaderObj.nBaud
277 #
278 # self.dataOut.code = self.radarControllerHeaderObj.code
270 # self.dataOut.timeInterval = self.radarControllerHeaderObj.ippSeconds * self.processingHeaderObj.nCohInt
271 #
272 # if self.radarControllerHeaderObj.code is not None:
273 #
274 # self.dataOut.nCode = self.radarControllerHeaderObj.nCode
275 #
276 # self.dataOut.nBaud = self.radarControllerHeaderObj.nBaud
277 #
278 # self.dataOut.code = self.radarControllerHeaderObj.code
279 279
280 280 self.dataOut.dtype = self.dtype
281 281
@@ -340,8 +340,8 class VoltageReader(JRODataReader, ProcessingUnit):
340 340 self.dtype = datatype_str
341 341 #self.ippSeconds = 2 * 1000 * self.radarControllerHeaderObj.ipp / self.c
342 342 self.fileSizeByHeader = self.processingHeaderObj.dataBlocksPerFile * self.processingHeaderObj.blockSize + self.firstHeaderSize + self.basicHeaderSize*(self.processingHeaderObj.dataBlocksPerFile - 1)
343 # self.dataOut.channelList = numpy.arange(self.systemHeaderObj.numChannels)
344 # self.dataOut.channelIndexList = numpy.arange(self.systemHeaderObj.numChannels)
343 # self.dataOut.channelList = numpy.arange(self.systemHeaderObj.numChannels)
344 # self.dataOut.channelIndexList = numpy.arange(self.systemHeaderObj.numChannels)
345 345 self.getBlockDimension()
346 346
347 347
@@ -355,7 +355,7 class VoltageReader(JRODataReader, ProcessingUnit):
355 355 self.blockPointer = 0
356 356
357 357 block = self.receiver.recv()
358
358
359 359 self.basicHeaderObj.read(block[self.blockPointer:])
360 360 self.blockPointer += self.basicHeaderObj.length
361 361 self.systemHeaderObj.read(block[self.blockPointer:])
@@ -378,7 +378,7 class VoltageReader(JRODataReader, ProcessingUnit):
378 378 if self.waitDataBlock(pointer_location=current_pointer_location):
379 379 junk = numpy.fromstring( block[self.blockPointer:], self.dtype, self.blocksize )
380 380 junk = junk.reshape( (self.processingHeaderObj.profilesPerBlock, self.processingHeaderObj.nHeights, self.systemHeaderObj.nChannels) )
381 # return 0
381 # return 0
382 382
383 383 #Dimensions : nChannels, nProfiles, nSamples
384 384
@@ -408,39 +408,39 class VoltageReader(JRODataReader, ProcessingUnit):
408 408
409 409 def getData(self):
410 410 """
411 getData obtiene una unidad de datos del buffer de lectura, un perfil, y la copia al objeto self.dataOut
412 del tipo "Voltage" con todos los parametros asociados a este (metadata). cuando no hay datos
413 en el buffer de lectura es necesario hacer una nueva lectura de los bloques de datos usando
414 "readNextBlock"
415
416 Ademas incrementa el contador del buffer "self.profileIndex" en 1.
417
418 Return:
419
420 Si el flag self.getByBlock ha sido seteado el bloque completo es copiado a self.dataOut y el self.profileIndex
421 es igual al total de perfiles leidos desde el archivo.
422
423 Si self.getByBlock == False:
424
425 self.dataOut.data = buffer[:, thisProfile, :]
426
427 shape = [nChannels, nHeis]
428
429 Si self.getByBlock == True:
430
431 self.dataOut.data = buffer[:, :, :]
432
433 shape = [nChannels, nProfiles, nHeis]
434
435 Variables afectadas:
436 self.dataOut
437 self.profileIndex
438
439 Affected:
440 self.dataOut
441 self.profileIndex
442 self.flagDiscontinuousBlock
443 self.flagIsNewBlock
411 getData obtiene una unidad de datos del buffer de lectura, un perfil, y la copia al objeto self.dataOut
412 del tipo "Voltage" con todos los parametros asociados a este (metadata). cuando no hay datos
413 en el buffer de lectura es necesario hacer una nueva lectura de los bloques de datos usando
414 "readNextBlock"
415
416 Ademas incrementa el contador del buffer "self.profileIndex" en 1.
417
418 Return:
419
420 Si el flag self.getByBlock ha sido seteado el bloque completo es copiado a self.dataOut y el self.profileIndex
421 es igual al total de perfiles leidos desde el archivo.
422
423 Si self.getByBlock == False:
424
425 self.dataOut.data = buffer[:, thisProfile, :]
426
427 shape = [nChannels, nHeis]
428
429 Si self.getByBlock == True:
430
431 self.dataOut.data = buffer[:, :, :]
432
433 shape = [nChannels, nProfiles, nHeis]
434
435 Variables afectadas:
436 self.dataOut
437 self.profileIndex
438
439 Affected:
440 self.dataOut
441 self.profileIndex
442 self.flagDiscontinuousBlock
443 self.flagIsNewBlock
444 444 """
445 445 if self.flagNoMoreFiles:
446 446 self.dataOut.flagNoData = True
@@ -462,30 +462,30 class VoltageReader(JRODataReader, ProcessingUnit):
462 462 if not self.getByBlock:
463 463
464 464 """
465 Return profile by profile
465 Return profile by profile
466 466
467 If nTxs > 1 then one profile is divided by nTxs and number of total
468 blocks is increased by nTxs (nProfiles *= nTxs)
467 If nTxs > 1 then one profile is divided by nTxs and number of total
468 blocks is increased by nTxs (nProfiles *= nTxs)
469 469 """
470 470 self.dataOut.flagDataAsBlock = False
471 471 self.dataOut.data = self.datablock[:,self.profileIndex,:]
472 472 self.dataOut.profileIndex = self.profileIndex
473 473
474 474 self.profileIndex += 1
475
476 # elif self.selBlocksize==None or self.selBlocksize==self.dataOut.nProfiles:
477 # """
478 # Return all block
479 # """
480 # self.dataOut.flagDataAsBlock = True
481 # self.dataOut.data = self.datablock
482 # self.dataOut.profileIndex = self.dataOut.nProfiles - 1
483 #
484 # self.profileIndex = self.dataOut.nProfiles
485
475
476 # elif self.selBlocksize==None or self.selBlocksize==self.dataOut.nProfiles:
477 # """
478 # Return all block
479 # """
480 # self.dataOut.flagDataAsBlock = True
481 # self.dataOut.data = self.datablock
482 # self.dataOut.profileIndex = self.dataOut.nProfiles - 1
483 #
484 # self.profileIndex = self.dataOut.nProfiles
485
486 486 else:
487 487 """
488 Return a block
488 Return a block
489 489 """
490 490 if self.selBlocksize == None: self.selBlocksize = self.dataOut.nProfiles
491 491 if self.selBlocktime != None:
Show file before | Show file after | Hide comments
@@ -1,12 +1,22
1 #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
2 #define NUM_CPY_THREADS 8
1 3 #include <Python.h>
2 4 #include <numpy/arrayobject.h>
3 5 #include <math.h>
6 #include <complex.h>
7 #include <time.h>
4 8
9 // void printArr(int *array);
5 10 static PyObject *hildebrand_sekhon(PyObject *self, PyObject *args);
11 static PyObject *correlateByBlock(PyObject *self, PyObject *args);
12 #ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
13 #define PyMODINIT_FUNC void
14 #endif
6 15
7 16 static PyMethodDef extensionsMethods[] = {
8 { "hildebrand_sekhon", (PyCFunction)hildebrand_sekhon, METH_VARARGS, "get noise with" },
9 { NULL, NULL, 0, NULL }
17 { "correlateByBlock", (PyCFunction)correlateByBlock, METH_VARARGS, "get correlation by block" },
18 { "hildebrand_sekhon", (PyCFunction)hildebrand_sekhon, METH_VARARGS, "get noise with hildebrand_sekhon" },
19 { NULL, NULL, 0, NULL }
10 20 };
11 21
12 22 PyMODINIT_FUNC initcSchain() {
@@ -14,13 +24,83 PyMODINIT_FUNC initcSchain() {
14 24 import_array();
15 25 }
16 26
27 static PyObject *correlateByBlock(PyObject *self, PyObject *args) {
28
29 // int *x = (int*) malloc(4000000 * 216 * sizeof(int));;
30 // int a = 5;
31 // x = &a;
32 // int b = 6;
33 // x = &b;
34 // printf("Antes de imprimir x \n");
35 // printf("%d \n", x[0]);
36
37 PyObject *data_obj1, *data_obj2;
38 PyArrayObject *data_array1, *data_array2, *correlateRow, *out, *dataRow, *codeRow; //, ,
39 int mode;
40
41 if (!PyArg_ParseTuple(args, "OOi", &data_obj1, &data_obj2, &mode)) return NULL;
42
43 data_array1 = (PyArrayObject *) PyArray_FROM_OTF(data_obj1, NPY_COMPLEX128, NPY_ARRAY_DEFAULT);
44 data_array2 = (PyArrayObject *) PyArray_FROM_OTF(data_obj2, NPY_FLOAT64, NPY_ARRAY_DEFAULT);
45
46 npy_intp dims[1];
47 dims[0] = 200;
48 npy_intp dims_code[1];
49 dims_code[0] = 16;
50
51 double complex * dataRaw;
52 double * codeRaw;
53 dataRaw = (double complex*) PyArray_DATA(data_array1);
54 codeRaw = (double *) PyArray_DATA(data_array2);
55 double complex ** outC = malloc(40000*200*sizeof(double complex));
56 int i;
57
58 clock_t start = clock();
59 for(i=0; i<40000; i++){
60 // codeRow = PyArray_SimpleNewFromData(1, dims_code, NPY_FLOAT64, codeRaw + 16 * i);
61 // dataRow = PyArray_SimpleNewFromData(1, dims, NPY_COMPLEX128, dataRaw + 200 * i);
62 // Py_INCREF(codeRow);
63 // Py_INCREF(dataRow);
64 // PyArray_ENABLEFLAGS(codeRow, NPY_ARRAY_OWNDATA);
65 // PyArray_ENABLEFLAGS(dataRow, NPY_ARRAY_OWNDATA);
66 correlateRow = (PyArrayObject *) PyArray_Correlate2(PyArray_SimpleNewFromData(1, dims_code, NPY_FLOAT64, codeRaw + 16 * i), PyArray_SimpleNewFromData(1, dims, NPY_COMPLEX128, dataRaw + 200 * i), (npy_intp) 2);
67 //Py_INCREF(correlateRow);
68 // PyArray_ENABLEFLAGS(correlateRow, NPY_ARRAY_OWNDATA);
69 memcpy(outC + 200*i, (double complex*) PyArray_DATA(correlateRow), 200 * sizeof(double complex));
70
71 Py_DECREF(correlateRow);
72 // Py_DECREF(codeRow);
73 // Py_DECREF(dataRow);
74 }
75 clock_t end = clock();
76 float seconds = (float)(end - start) / CLOCKS_PER_SEC;
77 printf("%f", seconds);
78 //
79 npy_intp dimsret[2];
80 dimsret[0] = 40000;
81 dimsret[1] = 200;
82 out = PyArray_SimpleNewFromData(2, dimsret, NPY_COMPLEX128, outC);
83 PyArray_ENABLEFLAGS(out, NPY_ARRAY_OWNDATA);
84 //Py_INCREF(out);
85 Py_DECREF(data_array1);
86 Py_DECREF(data_array2);
87 // PyArray_DebugPrint(out);
88 // Py_DECREF(data_obj2);
89 // Py_DECREF(data_obj1);
90 // Py_DECREF(codeRow);
91 // Py_DECREF(dataRow);
92 // free(dataRaw);
93 // free(codeRaw);
94
95 return PyArray_Return(out);
96 }
97
17 98 static PyObject *hildebrand_sekhon(PyObject *self, PyObject *args) {
18 /* Do your stuff here. */
19 99 double navg;
20 100 PyObject *data_obj, *data_array;
21 101
22 102 if (!PyArg_ParseTuple(args, "Od", &data_obj, &navg)) return NULL;
23 data_array = PyArray_FROM_OTF(data_obj, NPY_FLOAT64, NPY_IN_ARRAY);
103 data_array = PyArray_FROM_OTF(data_obj, NPY_FLOAT64, NPY_ARRAY_DEFAULT);
24 104 if (data_array == NULL) {
25 105 Py_XDECREF(data_array);
26 106 Py_XDECREF(data_obj);
@@ -56,3 +136,4 static PyObject *hildebrand_sekhon(PyObject *self, PyObject *args) {
56 136
57 137 return Py_BuildValue("d", lnoise);
58 138 }
139
Show file before | Show file after | Hide comments
@@ -200,7 +200,6 class ProcessingUnit(object):
200 200 return True
201 201
202 202 def call(self, opType, opName=None, opId=None):
203
204 203 """
205 204 Return True si ejecuta la operacion interna nombrada "opName" o la operacion externa
206 205 identificada con el id "opId"; con los argumentos "**kwargs".
Show file before | Show file after | Hide comments
@@ -163,7 +163,6 class IncohInt4SpectraHeis(Operation):
163 163
164 164 n = None
165 165
166
167 166 def __init__(self, **kwargs):
168 167
169 168 Operation.__init__(self, **kwargs)
@@ -324,10 +323,10 class IncohInt4SpectraHeis(Operation):
324 323
325 324 return avgdata, avgdatatime
326 325
327 def run(self, dataOut, **kwargs):
326 def run(self, dataOut, n=None, timeInterval=None, overlapping=False, **kwargs):
328 327
329 328 if not self.isConfig:
330 self.setup(**kwargs)
329 self.setup(n=n, timeInterval=timeInterval, overlapping=overlapping)
331 330 self.isConfig = True
332 331
333 332 avgdata, avgdatatime = self.integrate(dataOut.data_spc, dataOut.utctime)
Show file before | Show file after | Hide comments
@@ -52,13 +52,6 def _unpickle_method(func_name, obj, cls):
52 52 break
53 53 return func.__get__(obj, cls)
54 54
55
56
57
58
59
60
61
62 55 class ParametersProc(ProcessingUnit):
63 56
64 57 nSeconds = None
@@ -104,9 +97,7 class ParametersProc(ProcessingUnit):
104 97 self.dataOut.timeInterval1 = self.dataIn.timeInterval
105 98 self.dataOut.heightList = self.dataIn.getHeiRange()
106 99 self.dataOut.frequency = self.dataIn.frequency
107 self.dataOut.noise = self.dataIn.noise
108
109
100 # self.dataOut.noise = self.dataIn.noise
110 101
111 102 def run(self):
112 103
@@ -125,22 +116,23 class ParametersProc(ProcessingUnit):
125 116
126 117 if self.dataIn.type == "Spectra":
127 118
128 self.dataOut.data_pre = (self.dataIn.data_spc,self.dataIn.data_cspc)
129 print 'self.dataIn.data_spc', self.dataIn.data_spc.shape
119 self.dataOut.data_pre = (self.dataIn.data_spc, self.dataIn.data_cspc)
120 self.dataOut.data_spc = self.dataIn.data_spc
121 self.dataOut.data_cspc = self.dataIn.data_cspc
122 self.dataOut.nProfiles = self.dataIn.nProfiles
123 self.dataOut.nIncohInt = self.dataIn.nIncohInt
124 self.dataOut.nFFTPoints = self.dataIn.nFFTPoints
125 self.dataOut.ippFactor = self.dataIn.ippFactor
130 126 self.dataOut.abscissaList = self.dataIn.getVelRange(1)
131 127 self.dataOut.spc_noise = self.dataIn.getNoise()
132 self.dataOut.spc_range = (self.dataIn.getFreqRange(1)/1000. , self.dataIn.getAcfRange(1) , self.dataIn.getVelRange(1) )
133
134 self.dataOut.normFactor = self.dataIn.normFactor
135 #self.dataOut.outputInterval = self.dataIn.outputInterval
128 self.dataOut.spc_range = (self.dataIn.getFreqRange(1)/1000. , self.dataIn.getAcfRange(1) , self.dataIn.getVelRange(1))
129 self.dataOut.pairsList = self.dataIn.pairsList
136 130 self.dataOut.groupList = self.dataIn.pairsList
137 131 self.dataOut.flagNoData = False
138 #print 'datain chandist ',self.dataIn.ChanDist
132
139 133 if hasattr(self.dataIn, 'ChanDist'): #Distances of receiver channels
140 134 self.dataOut.ChanDist = self.dataIn.ChanDist
141 else: self.dataOut.ChanDist = None
142
143 print 'datain chandist ',self.dataOut.ChanDist
135 else: self.dataOut.ChanDist = None
144 136
145 137 if hasattr(self.dataIn, 'VelRange'): #Velocities range
146 138 self.dataOut.VelRange = self.dataIn.VelRange
Show file before | Show file after | Hide comments
@@ -1,3 +1,5
1 import itertools
2
1 3 import numpy
2 4
3 5 from jroproc_base import ProcessingUnit, Operation
@@ -109,7 +111,10 class SpectraProc(ProcessingUnit):
109 111
110 112 if self.dataIn.type == "Spectra":
111 113 self.dataOut.copy(self.dataIn)
112 # self.__selectPairs(pairsList)
114 if not pairsList:
115 pairsList = itertools.combinations(self.dataOut.channelList, 2)
116 if self.dataOut.data_cspc is not None:
117 self.__selectPairs(pairsList)
113 118 return True
114 119
115 120 if self.dataIn.type == "Voltage":
@@ -178,27 +183,21 class SpectraProc(ProcessingUnit):
178 183
179 184 def __selectPairs(self, pairsList):
180 185
181 if channelList == None:
186 if not pairsList:
182 187 return
183 188
184 pairsIndexListSelected = []
185
186 for thisPair in pairsList:
189 pairs = []
190 pairsIndex = []
187 191
188 if thisPair not in self.dataOut.pairsList:
192 for pair in pairsList:
193 if pair[0] not in self.dataOut.channelList or pair[1] not in self.dataOut.channelList:
189 194 continue
190
191 pairIndex = self.dataOut.pairsList.index(thisPair)
192
193 pairsIndexListSelected.append(pairIndex)
194
195 if not pairsIndexListSelected:
196 self.dataOut.data_cspc = None
197 self.dataOut.pairsList = []
198 return
199
200 self.dataOut.data_cspc = self.dataOut.data_cspc[pairsIndexListSelected]
201 self.dataOut.pairsList = [self.dataOut.pairsList[i] for i in pairsIndexListSelected]
195 pairs.append(pair)
196 pairsIndex.append(pairs.index(pair))
197
198 self.dataOut.data_cspc = self.dataOut.data_cspc[pairsIndex]
199 self.dataOut.pairsList = pairs
200 self.dataOut.pairsIndexList = pairsIndex
202 201
203 202 return
204 203
Show file before | Show file after | Hide comments
@@ -1,9 +1,10
1 1 import sys
2 2 import numpy
3 3 from scipy import interpolate
4
4 from schainpy import cSchain
5 5 from jroproc_base import ProcessingUnit, Operation
6 6 from schainpy.model.data.jrodata import Voltage
7 from time import time
7 8
8 9 class VoltageProc(ProcessingUnit):
9 10
@@ -332,7 +333,6 class CohInt(Operation):
332 333
333 334 n = None
334 335
335
336 336 def __init__(self, **kwargs):
337 337
338 338 Operation.__init__(self, **kwargs)
@@ -345,10 +345,9 class CohInt(Operation):
345 345
346 346 Inputs:
347 347
348 n : Number of coherent integrations
349 timeInterval : Time of integration. If the parameter "n" is selected this one does not work
350 overlapping :
351
348 n : Number of coherent integrations
349 timeInterval : Time of integration. If the parameter "n" is selected this one does not work
350 overlapping :
352 351 """
353 352
354 353 self.__initime = None
@@ -548,14 +547,13 class Decoder(Operation):
548 547 nCode = None
549 548 nBaud = None
550 549
551
552 550 def __init__(self, **kwargs):
553 551
554 552 Operation.__init__(self, **kwargs)
555 553
556 554 self.times = None
557 555 self.osamp = None
558 # self.__setValues = False
556 # self.__setValues = False
559 557 self.isConfig = False
560 558
561 559 def setup(self, code, osamp, dataOut):
@@ -631,12 +629,12 class Decoder(Operation):
631 629 junk = numpy.lib.stride_tricks.as_strided(self.code, (repetitions, self.code.size), (0, self.code.itemsize))
632 630 junk = junk.flatten()
633 631 code_block = numpy.reshape(junk, (self.nCode*repetitions, self.nBaud))
634
635 for i in range(self.__nChannels):
636 for j in range(self.__nProfiles):
637 self.datadecTime[i,j,:] = numpy.correlate(data[i,j,:], code_block[j,:], mode='full')[self.nBaud-1:]
638
639 return self.datadecTime
632 profilesList = xrange(self.__nProfiles)
633
634 for i in range(self.__nChannels):
635 for j in profilesList:
636 self.datadecTime[i,j,:] = numpy.correlate(data[i,j,:], code_block[j,:], mode='full')[self.nBaud-1:]
637 return self.datadecTime
640 638
641 639 def __convolutionByBlockInFreq(self, data):
642 640
@@ -653,6 +651,7 class Decoder(Operation):
653 651
654 652 return data
655 653
654
656 655 def run(self, dataOut, code=None, nCode=None, nBaud=None, mode = 0, osamp=None, times=None):
657 656
658 657 if dataOut.flagDecodeData:
@@ -1087,7 +1086,6 class SplitProfiles(Operation):
1087 1086 dataOut.ippSeconds /= n
1088 1087
1089 1088 class CombineProfiles(Operation):
1090
1091 1089 def __init__(self, **kwargs):
1092 1090
1093 1091 Operation.__init__(self, **kwargs)
Show file before | Show file after | Hide comments
@@ -32,9 +32,7 class Remote(Thread):
32 32 None
33 33
34 34 Written by:
35
36 "Miguel Urco":mailto:miguel.urco@jro.igp.gob.pe Jun. 03, 2015
37
35 "Miguel Urco":mailto:miguel.urco@jro.igp.gob.pe Jun. 03, 2015
38 36 """
39 37
40 38 server = None
@@ -168,13 +166,13 class Remote(Thread):
168 166 return 0
169 167
170 168 self.mutex.acquire()
171 # init = time.time()
172 #
173 # while(self.bussy):
174 # sleep(0.1)
175 # if time.time() - init > 2*self.period:
176 # return 0
177
169 # init = time.time()
170 #
171 # while(self.bussy):
172 # sleep(0.1)
173 # if time.time() - init > 2*self.period:
174 # return 0
175
178 176 self.fileList = fileList
179 177 self.mutex.release()
180 178 return 1
@@ -197,8 +195,8 class Remote(Thread):
197 195
198 196 if self.stopFlag:
199 197 break
200
201 # self.bussy = True
198
199 # self.bussy = True
202 200 self.mutex.acquire()
203 201
204 202 print "[Remote Server] Opening %s" %self.__server
@@ -213,7 +211,7 class Remote(Thread):
213 211 self.close()
214 212
215 213 self.mutex.release()
216 # self.bussy = False
214 # self.bussy = False
217 215
218 216 print "[Remote Server] Thread stopped successfully"
219 217
@@ -400,21 +398,21 class SSHClient(Remote):
400 398 def open(self, server, username, password, remotefolder, port=22):
401 399
402 400 """
403 This method is used to set SSH parameters and establish a connection to a remote server
404
405 Inputs:
406 server - remote server IP Address
407
408 username - remote server Username
409
410 password - remote server password
411
412 remotefolder - remote server current working directory
413
414 Return: void
415
416 Affects:
417 self.status - in case of error or fail connection this parameter is set to 0 else 1
401 This method is used to set SSH parameters and establish a connection to a remote server
402
403 Inputs:
404 server - remote server IP Address
405
406 username - remote server Username
407
408 password - remote server password
409
410 remotefolder - remote server current working directory
411
412 Return: void
413
414 Affects:
415 self.status - in case of error or fail connection this parameter is set to 0 else 1
418 416
419 417 """
420 418 import socket
@@ -440,7 +438,7 class SSHClient(Remote):
440 438 try:
441 439 sshClientObj.connect(server, username=username, password=password, port=port)
442 440 except paramiko.AuthenticationException, e:
443 # print "SSH username or password are incorrect: %s"
441 # print "SSH username or password are incorrect: %s"
444 442 print "[SSH Server]:", e
445 443 return 0
446 444 except SSHException, e:
@@ -474,7 +472,7 class SSHClient(Remote):
474 472
475 473 def close(self):
476 474 """
477 Close connection to remote server
475 Close connection to remote server
478 476 """
479 477 if not self.status:
480 478 return 0
@@ -484,13 +482,13 class SSHClient(Remote):
484 482
485 483 def __execute(self, command):
486 484 """
487 __execute a command on remote server
488
489 Input:
490 command - Exmaple 'ls -l'
491
492 Return:
493 0 in error case else 1
485 __execute a command on remote server
486
487 Input:
488 command - Exmaple 'ls -l'
489
490 Return:
491 0 in error case else 1
494 492 """
495 493 if not self.status:
496 494 return 0
@@ -509,13 +507,13 class SSHClient(Remote):
509 507
510 508 def mkdir(self, remotefolder):
511 509 """
512 mkdir is used to make a new directory in remote server
513
514 Input:
515 remotefolder - directory name
516
517 Return:
518 0 in error case else 1
510 mkdir is used to make a new directory in remote server
511
512 Input:
513 remotefolder - directory name
514
515 Return:
516 0 in error case else 1
519 517 """
520 518
521 519 command = 'mkdir %s' %remotefolder
@@ -530,16 +528,16 class SSHClient(Remote):
530 528
531 529 def cd(self, remotefolder):
532 530 """
533 cd is used to change remote working directory on server
534
535 Input:
536 remotefolder - current working directory
537
538 Affects:
539 self.remotefolder
540
541 Return:
542 0 in case of error else 1
531 cd is used to change remote working directory on server
532
533 Input:
534 remotefolder - current working directory
535
536 Affects:
537 self.remotefolder
538
539 Return:
540 0 in case of error else 1
543 541 """
544 542 if not self.status:
545 543 return 0
@@ -582,8 +580,8 class SendToServer(ProcessingUnit):
582 580 ProcessingUnit.__init__(self, **kwargs)
583 581
584 582 self.isConfig = False
585 self.clientObj = None
586
583 self.clientObj = None
584
587 585 def setup(self, server, username, password, remotefolder, localfolder, ext='.png', period=60, protocol='ftp', **kwargs):
588 586
589 587 self.clientObj = None
@@ -621,6 +619,7 class SendToServer(ProcessingUnit):
621 619 filenameList = glob.glob1(thisFolder, '*%s' %self.ext)
622 620
623 621 if len(filenameList) < 1:
622
624 623 continue
625 624
626 625 for thisFile in filenameList:
@@ -707,9 +706,9 class FTP(object):
707 706 try:
708 707 self.ftp = ftplib.FTP(self.server)
709 708 self.ftp.login(self.username,self.password)
710 self.ftp.cwd(self.remotefolder)
711 # print 'Connect to FTP Server: Successfully'
712
709 self.ftp.cwd(self.remotefolder)
710 # print 'Connect to FTP Server: Successfully'
711
713 712 except ftplib.all_errors:
714 713 print 'Error FTP Service'
715 714 self.status = 1
@@ -740,7 +739,7 class FTP(object):
740 739 name, ext = os.path.splitext(f)
741 740 if ext != '':
742 741 self.fileList.append(f)
743 # print 'filename: %s - size: %d'%(f,self.ftp.size(f))
742 # print 'filename: %s - size: %d'%(f,self.ftp.size(f))
744 743
745 744 def parmsByDefault(self):
746 745 server = 'jro-app.igp.gob.pe'
Show file before | Show file after | Hide comments
@@ -7,7 +7,6 import json
7 7 import numpy
8 8 import paho.mqtt.client as mqtt
9 9 import zmq
10 from profilehooks import profile
11 10 import datetime
12 11 from zmq.utils.monitor import recv_monitor_message
13 12 from functools import wraps
@@ -16,6 +15,7 from multiprocessing import Process
16 15
17 16 from schainpy.model.proc.jroproc_base import Operation, ProcessingUnit
18 17 from schainpy.model.data.jrodata import JROData
18 from schainpy.utils import log
19 19
20 20 MAXNUMX = 100
21 21 MAXNUMY = 100
@@ -31,14 +31,13 def roundFloats(obj):
31 31 return round(obj, 2)
32 32
33 33 def decimate(z, MAXNUMY):
34 # dx = int(len(self.x)/self.__MAXNUMX) + 1
35
36 34 dy = int(len(z[0])/MAXNUMY) + 1
37 35
38 36 return z[::, ::dy]
39 37
40 38 class throttle(object):
41 """Decorator that prevents a function from being called more than once every
39 '''
40 Decorator that prevents a function from being called more than once every
42 41 time period.
43 42 To create a function that cannot be called more than once a minute, but
44 43 will sleep until it can be called:
@@ -49,7 +48,7 class throttle(object):
49 48 for i in range(10):
50 49 foo()
51 50 print "This function has run %s times." % i
52 """
51 '''
53 52
54 53 def __init__(self, seconds=0, minutes=0, hours=0):
55 54 self.throttle_period = datetime.timedelta(
@@ -73,9 +72,169 class throttle(object):
73 72
74 73 return wrapper
75 74
75 class Data(object):
76 '''
77 Object to hold data to be plotted
78 '''
79
80 def __init__(self, plottypes, throttle_value):
81 self.plottypes = plottypes
82 self.throttle = throttle_value
83 self.ended = False
84 self.__times = []
85
86 def __str__(self):
87 dum = ['{}{}'.format(key, self.shape(key)) for key in self.data]
88 return 'Data[{}][{}]'.format(';'.join(dum), len(self.__times))
89
90 def __len__(self):
91 return len(self.__times)
92
93 def __getitem__(self, key):
94 if key not in self.data:
95 raise KeyError(log.error('Missing key: {}'.format(key)))
96
97 if 'spc' in key:
98 ret = self.data[key]
99 else:
100 ret = numpy.array([self.data[key][x] for x in self.times])
101 if ret.ndim > 1:
102 ret = numpy.swapaxes(ret, 0, 1)
103 return ret
104
105 def setup(self):
106 '''
107 Configure object
108 '''
109
110 self.ended = False
111 self.data = {}
112 self.__times = []
113 self.__heights = []
114 self.__all_heights = set()
115 for plot in self.plottypes:
116 self.data[plot] = {}
117
118 def shape(self, key):
119 '''
120 Get the shape of the one-element data for the given key
121 '''
122
123 if len(self.data[key]):
124 if 'spc' in key:
125 return self.data[key].shape
126 return self.data[key][self.__times[0]].shape
127 return (0,)
128
129 def update(self, dataOut):
130 '''
131 Update data object with new dataOut
132 '''
133
134 tm = dataOut.utctime
135 if tm in self.__times:
136 return
137
138 self.parameters = getattr(dataOut, 'parameters', [])
139 self.pairs = dataOut.pairsList
140 self.channels = dataOut.channelList
141 self.xrange = (dataOut.getFreqRange(1)/1000. , dataOut.getAcfRange(1) , dataOut.getVelRange(1))
142 self.interval = dataOut.getTimeInterval()
143 self.__heights.append(dataOut.heightList)
144 self.__all_heights.update(dataOut.heightList)
145 self.__times.append(tm)
146
147 for plot in self.plottypes:
148 if plot == 'spc':
149 z = dataOut.data_spc/dataOut.normFactor
150 self.data[plot] = 10*numpy.log10(z)
151 if plot == 'cspc':
152 self.data[plot] = dataOut.data_cspc
153 if plot == 'noise':
154 self.data[plot][tm] = 10*numpy.log10(dataOut.getNoise()/dataOut.normFactor)
155 if plot == 'rti':
156 self.data[plot][tm] = dataOut.getPower()
157 if plot == 'snr_db':
158 self.data['snr'][tm] = dataOut.data_SNR
159 if plot == 'snr':
160 self.data[plot][tm] = 10*numpy.log10(dataOut.data_SNR)
161 if plot == 'dop':
162 self.data[plot][tm] = 10*numpy.log10(dataOut.data_DOP)
163 if plot == 'mean':
164 self.data[plot][tm] = dataOut.data_MEAN
165 if plot == 'std':
166 self.data[plot][tm] = dataOut.data_STD
167 if plot == 'coh':
168 self.data[plot][tm] = dataOut.getCoherence()
169 if plot == 'phase':
170 self.data[plot][tm] = dataOut.getCoherence(phase=True)
171 if plot == 'output':
172 self.data[plot][tm] = dataOut.data_output
173 if plot == 'param':
174 self.data[plot][tm] = dataOut.data_param
175
176 def normalize_heights(self):
177 '''
178 Ensure same-dimension of the data for different heighList
179 '''
180
181 H = numpy.array(list(self.__all_heights))
182 H.sort()
183 for key in self.data:
184 shape = self.shape(key)[:-1] + H.shape
185 for tm, obj in self.data[key].items():
186 h = self.__heights[self.__times.index(tm)]
187 if H.size == h.size:
188 continue
189 index = numpy.where(numpy.in1d(H, h))[0]
190 dummy = numpy.zeros(shape) + numpy.nan
191 if len(shape) == 2:
192 dummy[:, index] = obj
193 else:
194 dummy[index] = obj
195 self.data[key][tm] = dummy
196
197 self.__heights = [H for tm in self.__times]
198
199 def jsonify(self, decimate=False):
200 '''
201 Convert data to json
202 '''
203
204 ret = {}
205 tm = self.times[-1]
206
207 for key, value in self.data:
208 if key in ('spc', 'cspc'):
209 ret[key] = roundFloats(self.data[key].to_list())
210 else:
211 ret[key] = roundFloats(self.data[key][tm].to_list())
212
213 ret['timestamp'] = tm
214 ret['interval'] = self.interval
215
216 @property
217 def times(self):
218 '''
219 Return the list of times of the current data
220 '''
221
222 ret = numpy.array(self.__times)
223 ret.sort()
224 return ret
225
226 @property
227 def heights(self):
228 '''
229 Return the list of heights of the current data
230 '''
231
232 return numpy.array(self.__heights[-1])
76 233
77 234 class PublishData(Operation):
78 """Clase publish."""
235 '''
236 Operation to send data over zmq.
237 '''
79 238
80 239 def __init__(self, **kwargs):
81 240 """Inicio."""
@@ -87,11 +246,11 class PublishData(Operation):
87 246
88 247 def on_disconnect(self, client, userdata, rc):
89 248 if rc != 0:
90 print("Unexpected disconnection.")
249 log.warning('Unexpected disconnection.')
91 250 self.connect()
92 251
93 252 def connect(self):
94 print 'trying to connect'
253 log.warning('trying to connect')
95 254 try:
96 255 self.client.connect(
97 256 host=self.host,
@@ -105,7 +264,7 class PublishData(Operation):
105 264 # retain=True
106 265 # )
107 266 except:
108 print "MQTT Conection error."
267 log.error('MQTT Conection error.')
109 268 self.client = False
110 269
111 270 def setup(self, port=1883, username=None, password=None, clientId="user", zeromq=1, verbose=True, **kwargs):
@@ -120,8 +279,7 class PublishData(Operation):
120 279 self.zeromq = zeromq
121 280 self.mqtt = kwargs.get('plottype', 0)
122 281 self.client = None
123 self.verbose = verbose
124 self.dataOut.firstdata = True
282 self.verbose = verbose
125 283 setup = []
126 284 if mqtt is 1:
127 285 self.client = mqtt.Client(
@@ -176,7 +334,6 class PublishData(Operation):
176 334 'type': self.plottype,
177 335 'yData': yData
178 336 }
179 # print payload
180 337
181 338 elif self.plottype in ('rti', 'power'):
182 339 data = getattr(self.dataOut, 'data_spc')
@@ -230,15 +387,16 class PublishData(Operation):
230 387 'timestamp': 'None',
231 388 'type': None
232 389 }
233 # print 'Publishing data to {}'.format(self.host)
390
234 391 self.client.publish(self.topic + self.plottype, json.dumps(payload), qos=0)
235 392
236 393 if self.zeromq is 1:
237 394 if self.verbose:
238 print '[Sending] {} - {}'.format(self.dataOut.type, self.dataOut.datatime)
395 log.log(
396 '{} - {}'.format(self.dataOut.type, self.dataOut.datatime),
397 'Sending'
398 )
239 399 self.zmq_socket.send_pyobj(self.dataOut)
240 self.dataOut.firstdata = False
241
242 400
243 401 def run(self, dataOut, **kwargs):
244 402 self.dataOut = dataOut
@@ -253,6 +411,7 class PublishData(Operation):
253 411 if self.zeromq is 1:
254 412 self.dataOut.finished = True
255 413 self.zmq_socket.send_pyobj(self.dataOut)
414 time.sleep(0.1)
256 415 self.zmq_socket.close()
257 416 if self.client:
258 417 self.client.loop_stop()
@@ -281,7 +440,7 class ReceiverData(ProcessingUnit):
281 440 self.receiver = self.context.socket(zmq.PULL)
282 441 self.receiver.bind(self.address)
283 442 time.sleep(0.5)
284 print '[Starting] ReceiverData from {}'.format(self.address)
443 log.success('ReceiverData from {}'.format(self.address))
285 444
286 445
287 446 def run(self):
@@ -291,8 +450,9 class ReceiverData(ProcessingUnit):
291 450 self.isConfig = True
292 451
293 452 self.dataOut = self.receiver.recv_pyobj()
294 print '[Receiving] {} - {}'.format(self.dataOut.type,
295 self.dataOut.datatime.ctime())
453 log.log('{} - {}'.format(self.dataOut.type,
454 self.dataOut.datatime.ctime(),),
455 'Receiving')
296 456
297 457
298 458 class PlotterReceiver(ProcessingUnit, Process):
@@ -306,7 +466,6 class PlotterReceiver(ProcessingUnit, Process):
306 466 self.mp = False
307 467 self.isConfig = False
308 468 self.isWebConfig = False
309 self.plottypes = []
310 469 self.connections = 0
311 470 server = kwargs.get('server', 'zmq.pipe')
312 471 plot_server = kwargs.get('plot_server', 'zmq.web')
@@ -326,19 +485,13 class PlotterReceiver(ProcessingUnit, Process):
326 485 self.realtime = kwargs.get('realtime', False)
327 486 self.throttle_value = kwargs.get('throttle', 5)
328 487 self.sendData = self.initThrottle(self.throttle_value)
488 self.dates = []
329 489 self.setup()
330 490
331 491 def setup(self):
332 492
333 self.data = {}
334 self.data['times'] = []
335 for plottype in self.plottypes:
336 self.data[plottype] = {}
337 self.data['noise'] = {}
338 self.data['throttle'] = self.throttle_value
339 self.data['ENDED'] = False
340 self.isConfig = True
341 self.data_web = {}
493 self.data = Data(self.plottypes, self.throttle_value)
494 self.isConfig = True
342 495
343 496 def event_monitor(self, monitor):
344 497
@@ -355,15 +508,13 class PlotterReceiver(ProcessingUnit, Process):
355 508 self.connections += 1
356 509 if evt['event'] == 512:
357 510 pass
358 if self.connections == 0 and self.started is True:
359 self.ended = True
360 511
361 512 evt.update({'description': events[evt['event']]})
362 513
363 514 if evt['event'] == zmq.EVENT_MONITOR_STOPPED:
364 515 break
365 516 monitor.close()
366 print("event monitor thread done!")
517 print('event monitor thread done!')
367 518
368 519 def initThrottle(self, throttle_value):
369 520
@@ -373,65 +524,16 class PlotterReceiver(ProcessingUnit, Process):
373 524
374 525 return sendDataThrottled
375 526
376
377 527 def send(self, data):
378 # print '[sending] data=%s size=%s' % (data.keys(), len(data['times']))
528 log.success('Sending {}'.format(data), self.name)
379 529 self.sender.send_pyobj(data)
380 530
381
382 def update(self):
383 t = self.dataOut.utctime
384
385 if t in self.data['times']:
386 return
387
388 self.data['times'].append(t)
389 self.data['dataOut'] = self.dataOut
390
391 for plottype in self.plottypes:
392 if plottype == 'spc':
393 z = self.dataOut.data_spc/self.dataOut.normFactor
394 self.data[plottype] = 10*numpy.log10(z)
395 self.data['noise'][t] = 10*numpy.log10(self.dataOut.getNoise()/self.dataOut.normFactor)
396 if plottype == 'cspc':
397 jcoherence = self.dataOut.data_cspc/numpy.sqrt(self.dataOut.data_spc*self.dataOut.data_spc)
398 self.data['cspc_coh'] = numpy.abs(jcoherence)
399 self.data['cspc_phase'] = numpy.arctan2(jcoherence.imag, jcoherence.real)*180/numpy.pi
400 if plottype == 'rti':
401 self.data[plottype][t] = self.dataOut.getPower()
402 if plottype == 'snr':
403 self.data[plottype][t] = 10*numpy.log10(self.dataOut.data_SNR)
404 if plottype == 'dop':
405 self.data[plottype][t] = 10*numpy.log10(self.dataOut.data_DOP)
406 if plottype == 'mean':
407 self.data[plottype][t] = self.dataOut.data_MEAN
408 if plottype == 'std':
409 self.data[plottype][t] = self.dataOut.data_STD
410 if plottype == 'coh':
411 self.data[plottype][t] = self.dataOut.getCoherence()
412 if plottype == 'phase':
413 self.data[plottype][t] = self.dataOut.getCoherence(phase=True)
414 if plottype == 'output':
415 self.data[plottype][t] = self.dataOut.data_output
416 if plottype == 'param':
417 self.data[plottype][t] = self.dataOut.data_param
418 if self.realtime:
419 self.data_web['timestamp'] = t
420 if plottype == 'spc':
421 self.data_web[plottype] = roundFloats(decimate(self.data[plottype]).tolist())
422 elif plottype == 'cspc':
423 self.data_web['cspc_coh'] = roundFloats(decimate(self.data['cspc_coh']).tolist())
424 self.data_web['cspc_phase'] = roundFloats(decimate(self.data['cspc_phase']).tolist())
425 elif plottype == 'noise':
426 self.data_web['noise'] = roundFloats(self.data['noise'][t].tolist())
427 else:
428 self.data_web[plottype] = roundFloats(decimate(self.data[plottype][t]).tolist())
429 self.data_web['interval'] = self.dataOut.getTimeInterval()
430 self.data_web['type'] = plottype
431
432 531 def run(self):
433 532
434 print '[Starting] {} from {}'.format(self.name, self.address)
533 log.success(
534 'Starting from {}'.format(self.address),
535 self.name
536 )
435 537
436 538 self.context = zmq.Context()
437 539 self.receiver = self.context.socket(zmq.PULL)
@@ -448,39 +550,39 class PlotterReceiver(ProcessingUnit, Process):
448 550 else:
449 551 self.sender.bind("ipc:///tmp/zmq.plots")
450 552
451 time.sleep(3)
553 time.sleep(2)
452 554
453 555 t = Thread(target=self.event_monitor, args=(monitor,))
454 556 t.start()
455 557
456 558 while True:
457 self.dataOut = self.receiver.recv_pyobj()
458 # print '[Receiving] {} - {}'.format(self.dataOut.type,
459 # self.dataOut.datatime.ctime())
460
461 self.update()
559 dataOut = self.receiver.recv_pyobj()
560 dt = datetime.datetime.fromtimestamp(dataOut.utctime).date()
561 sended = False
562 if dt not in self.dates:
563 if self.data:
564 self.data.ended = True
565 self.send(self.data)
566 sended = True
567 self.data.setup()
568 self.dates.append(dt)
462 569
463 if self.dataOut.firstdata is True:
464 self.data['STARTED'] = True
570 self.data.update(dataOut)
465 571
466 if self.dataOut.finished is True:
467 self.send(self.data)
572 if dataOut.finished is True:
468 573 self.connections -= 1
469 if self.connections == 0 and self.started:
470 self.ended = True
471 self.data['ENDED'] = True
574 if self.connections == 0 and dt in self.dates:
575 self.data.ended = True
472 576 self.send(self.data)
473 self.setup()
474 self.started = False
577 self.data.setup()
475 578 else:
476 579 if self.realtime:
477 580 self.send(self.data)
478 self.sender_web.send_string(json.dumps(self.data_web))
581 # self.sender_web.send_string(self.data.jsonify())
479 582 else:
480 self.sendData(self.send, self.data)
481 self.started = True
583 if not sended:
584 self.sendData(self.send, self.data)
482 585
483 self.data['STARTED'] = False
484 586 return
485 587
486 588 def sendToWeb(self):
@@ -497,6 +599,6 class PlotterReceiver(ProcessingUnit, Process):
497 599 time.sleep(1)
498 600 for kwargs in self.operationKwargs.values():
499 601 if 'plot' in kwargs:
500 print '[Sending] Config data to web for {}'.format(kwargs['code'].upper())
602 log.success('[Sending] Config data to web for {}'.format(kwargs['code'].upper()))
501 603 sender_web_config.send_string(json.dumps(kwargs))
502 self.isWebConfig = True
604 self.isWebConfig = True No newline at end of file
Show file before | Show file after | Hide comments
@@ -93,7 +93,11 def filterOffsets(offsets0, stdvLimit):
93 93
94 94 #---------------------- Setup ---------------------------
95 95
96 <<<<<<< HEAD
97 path = '/home/nanosat/Pictures/JASMET30_mp/201608/phase'
98 =======
96 99 path = '/home/jespinoza/Pictures/JASMET30/201608/phase'
100 >>>>>>> master
97 101 stdvLimit = 0.5
98 102
99 103 #---------------------- Script ---------------------------
Show file before | Show file after | Hide comments
@@ -1,4 +1,4
1 """.
1 '''
2 2 SCHAINPY - LOG
3 3 Simple helper for log standarization
4 4 Usage:
@@ -13,33 +13,32 SCHAINPY - LOG
13 13 which will look like this:
14 14 [NEVER GONNA] - give you up
15 15 with color red as background and white as foreground.
16 """
16 '''
17 17
18 18 import click
19 19
20
21 def warning(message):
22 click.echo(click.style('[WARNING] - ' + message, fg='yellow'))
20 def warning(message, tag='Warning'):
21 click.echo(click.style('[{}] {}'.format(tag, message), fg='yellow'))
23 22 pass
24 23
25 24
26 def error(message):
27 click.echo(click.style('[ERROR] - ' + message, fg='red'))
25 def error(message, tag='Error'):
26 click.echo(click.style('[{}] {}'.format(tag, message), fg='red'))
28 27 pass
29 28
30 29
31 def success(message):
32 click.echo(click.style(message, fg='green'))
30 def success(message, tag='Info'):
31 click.echo(click.style('[{}] {}'.format(tag, message), fg='green'))
33 32 pass
34 33
35 34
36 def log(message):
37 click.echo('[LOG] - ' + message)
35 def log(message, tag='Info'):
36 click.echo('[{}] {}'.format(tag, message))
38 37 pass
39 38
40 39
41 def makelogger(topic, bg='reset', fg='reset'):
40 def makelogger(tag, bg='reset', fg='reset'):
42 41 def func(message):
43 click.echo(click.style('[{}] - '.format(topic.upper()) + message,
42 click.echo(click.style('[{}] {}'.format(tag.upper(), message),
44 43 bg=bg, fg=fg))
45 44 return func
Show file before | Show file after | Hide comments
@@ -1,58 +1,69
1 """.
2
1 '''
3 2 Created on Jul 16, 2014
4 3
5 4 @author: Miguel Urco
6 """
5 '''
7 6
8 import numpy
9 7 from setuptools import setup, Extension
8 from setuptools.command.build_ext import build_ext as _build_ext
10 9 from schainpy import __version__
11 10
11 class build_ext(_build_ext):
12 def finalize_options(self):
13 _build_ext.finalize_options(self)
14 # Prevent numpy from thinking it is still in its setup process:
15 __builtins__.__NUMPY_SETUP__ = False
16 import numpy
17 self.include_dirs.append(numpy.get_include())
18
19
12 20 setup(name="schainpy",
13 21 version=__version__,
14 22 description="Python tools to read, write and process Jicamarca data",
15 23 author="Miguel Urco",
16 24 author_email="miguel.urco@jro.igp.gob.pe",
17 25 url="http://jro.igp.gob.pe",
18 packages={'schainpy',
19 'schainpy.model',
20 'schainpy.model.data',
21 'schainpy.model.graphics',
22 'schainpy.model.io',
23 'schainpy.model.proc',
24 'schainpy.model.serializer',
25 'schainpy.model.utils',
26 'schainpy.gui',
27 'schainpy.gui.figures',
28 'schainpy.gui.viewcontroller',
29 'schainpy.gui.viewer',
30 'schainpy.gui.viewer.windows'},
26 packages = {'schainpy',
27 'schainpy.model',
28 'schainpy.model.data',
29 'schainpy.model.graphics',
30 'schainpy.model.io',
31 'schainpy.model.proc',
32 'schainpy.model.serializer',
33 'schainpy.model.utils',
34 'schainpy.gui',
35 'schainpy.gui.figures',
36 'schainpy.gui.viewcontroller',
37 'schainpy.gui.viewer',
38 'schainpy.gui.viewer.windows'},
31 39 ext_package='schainpy',
32 40 py_modules=[''],
33 41 package_data={'': ['schain.conf.template'],
34 'schainpy.gui.figures': ['*.png', '*.jpg'],
42 'schainpy.gui.figures': ['*.png','*.jpg'],
35 43 },
36 44 include_package_data=False,
45 scripts =['schainpy/gui/schainGUI'],
46 ext_modules=[
47 Extension("cSchain", ["schainpy/model/proc/extensions.c"]
48 )],
37 49 entry_points={
38 50 'console_scripts': [
39 51 'schain = schaincli.cli:main',
40 52 ],
41 53 },
42 scripts=['schainpy/gui/schainGUI'],
43 ext_modules=[Extension("cSchain", ["schainpy/model/proc/extensions.c"], include_dirs=[numpy.get_include()])],
54 cmdclass={'build_ext':build_ext},
55 setup_requires=["numpy >= 1.11.2"],
44 56 install_requires=[
45 "scipy >= 0.13.0",
46 "h5py >= 2.2.1",
47 "matplotlib >= 1.4.2",
48 "pyfits >= 3.4",
49 "paramiko >= 2.1.2",
50 "paho-mqtt >= 1.2",
51 "zmq",
52 "fuzzywuzzy",
53 "click",
54 "colorama",
55 "python-Levenshtein"
56 ],
57 )
58
57 "scipy >= 0.14.0",
58 "h5py >= 2.2.1",
59 "matplotlib >= 1.4.2",
60 "pyfits >= 3.4",
61 "paramiko >= 2.1.2",
62 "paho-mqtt >= 1.2",
63 "zmq",
64 "fuzzywuzzy",
65 "click",
66 "colorama",
67 "python-Levenshtein"
68 ],
69 )
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
General Comments 0
You need to be logged in to leave comments. Login now