##// END OF EJS Templates
schain-jc
RSS
merge from master
José Chávez -
r1054:10ce564971bb merge
parent child
Show More
Side by Side Unified
Context file:
r1054:10ce564971bb Loading diff...:

The requested changes are too big and content was truncated. Show full diff

Show file before | Show file after | Hide comments
@@ -0,0 +1,108
1 ## CHANGELOG:
2
3 ### 2.3
4 * Added high order function `multiSchain` for multiprocessing scripts.
5 * Added two new Processing Units `PublishData` and `ReceiverData` for receiving and sending dataOut through multiple ways (tcp, ipc, inproc).
6 * Added a new graphics Processing Unit `PlotterReceiver`. It is decoupled from normal processing sequence with support for data generated by multiprocessing scripts.
7 * Added support for sending realtime graphic to web server.
8 * XML command `schain` is now `schain --xml`.
9 * Added a CLI tool named `schain`.
10 * Scripts templates can be now generated with `schain generate`.
11 * Now it is possible to search Processing Units and Operations with `schain search [module]` to get the right name and its allowed parameters.
12 * Added suggestions when parameters are poorly written.
13 * `Controller.start()` now runs in a different process than the process calling it.
14 * Added `schainpy.utils.log` for log standarization.
15 * Running script on online mode no longer ignores date and hour. Issue #1109.
16 * Added support for receving voltage data directly from JARS (tcp, ipc).
17 * Updated README for compatibility with MAC OS GUI installation.
18 * Setup now installs numpy.
19
20 ### 2.2.6
21 * Graphics generated by the GUI are now the same as generated by scripts. Issue #1074.
22 * Added support for C extensions.
23 * Function `hildebrand_sehkon` optimized with a C wrapper.
24 * Numpy version updated.
25 * Migration to GIT.
26
27 ### 2.2.5:
28 * splitProfiles and combineProfiles modules were added to VoltageProc and Signal Chain GUI.
29 * nProfiles of USRP data (hdf5) is the number of profiles thera are in one second.
30 * jroPlotter works directly with data objects instead of dictionaries
31 * script "schain" was added to Signal Chain installer
32
33 ### 2.2.4.1:
34 * jroIO_usrp.py is update to read Sandra's data
35 * decimation in Spectra and RTI plots is always enabled.
36 * time* window option added to GUI
37
38 ### 2.2.4:
39 * jroproc_spectra_lags.py added to schainpy
40 * Bug fixed in schainGUI: ProcUnit was created with the same id in some cases.
41 * Bug fixed in jroHeaderIO: Header size validation.
42
43 ### 2.2.3.1:
44 * Filtering block by time has been added.
45 * Bug fixed plotting RTI, CoherenceMap and others using xmin and xmax parameters. The first day worked
46 properly but the next days did not.
47
48 ### 2.2.3:
49 * Bug fixed in GUI: Error getting(reading) Code value
50 * Bug fixed in GUI: Flip option always needs channelList field
51 * Bug fixed in jrodata: when one branch modified a value in "dataOut" (example: dataOut.code) this value
52 was modified for every branch (because this was a reference). It was modified in data.copy()
53 * Bug fixed in jroproc_voltage.profileSelector(): rangeList replaces to profileRangeList.
54
55 ### 2.2.2:
56 * VoltageProc: ProfileSelector, Reshape, Decoder with nTxs!=1 and getblock=True was tested
57 * Rawdata and testRawdata.py added to Signal Chain project
58
59 ### 2.2.1:
60 * Bugs fixed in GUI
61 * Views were improved in GUI
62 * Support to MST* ISR experiments
63 * Bug fixed getting noise using hyldebrant. (minimum number of points > 20%)
64 * handleError added to jroplotter.py
65
66 ### 2.2.0:
67 * GUI: use of external plotter
68 * Compatible with matplotlib 1.5.0
69
70 ### 2.1.5:
71 * serializer module added to Signal Chain
72 * jroplotter.py added to Signal Chain
73
74 ### 2.1.4.2:
75 * A new Plotter Class was added
76 * Project.start() does not accept filename as a parameter anymore
77
78 ### 2.1.4.1:
79 * Send notifications when an error different to ValueError is detected
80
81 ### 2.1.4:
82 * Sending error notifications to signal chain administrator
83 * Login to email server added
84
85 ### 2.1.3.3:
86 * Colored Button Icons were added to GUI
87
88 ### 2.1.3.2:
89 * GUI: user interaction enhanced
90 * controller_api.py: Safe access to ControllerThead
91
92 ### 2.1.3.1:
93 * GUI: every icon were resized
94 * jroproc_voltage.py: Print a message when "Read from code" option is selected and the code is not defined inside data file
95
96 ### 2.1.3:
97 * jroplot_heispectra.py: SpectraHeisScope was not showing the right channels
98 * jroproc_voltage.py: Bug fixed selecting profiles (self.nProfiles took a wrong value),
99 Bug fixed selecting heights by block (selecting profiles instead heights)
100 * jroproc_voltage.py: New feature added: decoding data by block using FFT.
101 * jroIO_heispectra.py: Bug fixed in FitsReader. Using local Fits instance instead schainpy.mode.data.jrodata.Fits.
102 * jroIO_heispectra.py: Channel index list does not exist.
103
104 ### 2.1.2:
105 * jroutils_ftp.py: Bug fixed, Any error sending file stopped the Server Thread
106 Server thread opens and closes remote server each time file list is sent
107 * jroplot_spectra.py: Noise path was not being created when noise data is saved.
108 * jroIO_base.py: startTime can be greater than endTime. Example: SpreadF [18:00 * 07:00] No newline at end of file
Show file before | Show file after | Hide comments
@@ -1,109 +1,119
1 1 # Byte-compiled / optimized / DLL files
2 2 __pycache__/
3 3 *.py[cod]
4 4 *$py.class
5 5
6 6 # C extensions
7 7 *.so
8 8
9 9 # Distribution / packaging
10 10 .Python
11 11 env/
12 12 build/
13 13 develop-eggs/
14 14 dist/
15 15 downloads/
16 16 eggs/
17 17 .eggs/
18 18 lib/
19 19 lib64/
20 20 parts/
21 21 sdist/
22 22 var/
23 23 wheels/
24 24 *.egg-info/
25 25 .installed.cfg
26 26 *.egg
27 27
28 28 # PyInstaller
29 29 # Usually these files are written by a python script from a template
30 30 # before PyInstaller builds the exe, so as to inject date/other infos into it.
31 31 *.manifest
32 32 *.spec
33 33
34 34 # Installer logs
35 35 pip-log.txt
36 36 pip-delete-this-directory.txt
37 37
38 38 # Unit test / coverage reports
39 39 htmlcov/
40 40 .tox/
41 41 .coverage
42 42 .coverage.*
43 43 .cache
44 44 nosetests.xml
45 45 coverage.xml
46 46 *,cover
47 47 .hypothesis/
48 48
49 49 # Translations
50 50 *.mo
51 51 *.pot
52 52
53 53 # Django stuff:
54 54 *.log
55 55 local_settings.py
56 56
57 57 # Flask stuff:
58 58 instance/
59 59 .webassets-cache
60 60
61 61 # Scrapy stuff:
62 62 .scrapy
63 63
64 64 # Sphinx documentation
65 65 docs/_build/
66 66
67 67 # PyBuilder
68 68 target/
69 69
70 70 # Jupyter Notebook
71 71 .ipynb_checkpoints
72 72
73 73 # pyenv
74 74 .python-version
75 75
76 76 # celery beat schedule file
77 77 celerybeat-schedule
78 78
79 79 # SageMath parsed files
80 80 *.sage.py
81 81
82 82 # dotenv
83 83 .env
84 84
85 85 # virtualenv
86 86 .venv
87 87 venv/
88 88 ENV/
89 89
90 90 # Spyder project settings
91 91 .spyderproject
92 92 .spyproject
93 93
94 94 # Rope project settings
95 95 .ropeproject
96 96
97 97 # mkdocs documentation
98 98 /site
99 99
100 100 # eclipse
101 101 .project
102 102 .pydevproject
103 <<<<<<< HEAD
103 104
104 105 # vscode
105 106
106 107 .vscode
107 108
108 109 schaingui/node_modules/
109 schainpy/scripts/ No newline at end of file
110 schainpy/scripts/
111 =======
112 .svn/
113 *.png
114 *.pyc
115 schainpy/scripts
116 .vscode
117 schaingui/node_modules
118 trash
119 >>>>>>> master
Show file before | Show file after | Hide comments
@@ -1,131 +1,147
1 1 # Signal Chain
2 2
3 3 ## Introduction
4 4
5 5 Signal Chain (SCh) is a radar data processing library developed using [Python](www.python.org) at JRO. SCh provides modules to read, write, process and plot data.
6 6
7 7 ## Installation
8 8
9 Install system dependencies, download the latest stable release from [svn](http://jro-dev.igp.gob.pe/svn/jro_soft/schain/Releases/) e.g. schainpy-2.2.5.tar.gz. and install it as a normal python package.
9 Install system dependencies, clone the latest version from [git](http://jro-dev.igp.gob.pe/rhodecode/schain/) and install it as a normal python package.
10 10
11 ### Linux based system
11 12 ```
12 13 $ sudo apt-get install python-pip python-dev gfortran libpng-dev freetype* libblas-dev liblapack-dev libatlas-base-dev python-qt4 python-tk libssl-dev libhdf5-dev
13 $ tar xvzf schainpy-2.2.5.tar.gz
14 $ cd schainpy-2.2.5
14 $ sudo pip install numpy
15 $ git clone http://jro-dev.igp.gob.pe/rhodecode/schain/
16 $ cd schain
15 17 $ sudo pip install ./
16 ```
17
18 **Its recommended to install schain in a virtual environment**
19 18
20 19 ```
20 **It is recommended to install schain in a virtual environment**
21 ```
21 22 $ sudo pip install virtualenv
22 23 $ virtualenv /path/to/virtual --system-site-packages
23 24 $ source /path/to/virtual/bin/activate
24 (virtual) $ cd schainpy-2.2.5
25 (virtual) $ cd schain
25 26 (virtual) $ pip install ./
27
28 ```
29
30 ### MAC Os
26 31 ```
32 $ brew install cartr/qt4/pyqt
33 $ git clone http://jro-dev.igp.gob.pe/rhodecode/schain/
34 $ cd schain
35 $ pip install ./
36 ```
37
38 if ```pip install ./``` does not work, install a proper python enviroment, and repeat the steps.
39 ```
40 $ brew install python
41 ```
42
27 43
28 44 ## First Script
29 45
30 46 Read Spectra data (.pdata) - remove dc - plot spectra & RTI
31 47
32 48 Import SCh and creating a project
33 49
34 50 ```python
35 51 #!/usr/bin/python
36 52
37 53 from schainpy.controller import Project
38 54
39 55 controller = Project()
40 56 controller.setup(id = '100',
41 57 name='test',
42 58 description='Basic experiment')
43 59
44 60
45 61 ```
46 62
47 63 Adding read unit and operations
48 64
49 65 ```python
50 66 read_unit = controller.addReadUnit(datatype='Spectra',
51 67 path='/path/to/pdata/',
52 68 startDate='2014/01/31',
53 69 endDate='2014/03/31',
54 70 startTime='00:00:00',
55 71 endTime='23:59:59',
56 72 online=0,
57 73 walk=0)
58 74
59 75 proc_unit = controller.addProcUnit(datatype='Spectra',
60 76 inputId=read_unit.getId())
61 77
62 78 op = proc_unit.addOperation(name='selectChannels')
63 79 op.addParameter(name='channelList', value='0,1', format='intlist')
64 80
65 81 op = proc_unit.addOperation(name='selectHeights')
66 82 op.addParameter(name='minHei', value='80', format='float')
67 83 op.addParameter(name='maxHei', value='200', format='float')
68 84
69 85 op = proc_unit.addOperation(name='removeDC')
70 86
71 87 ```
72 88
73 89 Plotting data & start project
74 90
75 91 ```python
76 92 op = proc_unit.addOperation(name='SpectraPlot', optype='other')
77 93 op.addParameter(name='id', value='1', format='int')
78 94 op.addParameter(name='wintitle', value='Spectra', format='str')
79 95
80 96 op = procUnitConfObj1.addOperation(name='RTIPlot', optype='other')
81 97 op.addParameter(name='id', value='2', format='int')
82 98 op.addParameter(name='wintitle', value='RTI', format='str')
83 99
84 100 controller.start()
85 101
86 102 ```
87 103
88 104 Full script
89 105
90 106
91 107 ```python
92 108 #!/usr/bin/python
93 109
94 110 from schainpy.controller import Project
95 111
96 112 controller = Project()
97 113 controller.setup(id = '100',
98 114 name='test',
99 115 description='Basic experiment')
100 116 read_unit = controller.addReadUnit(datatype='Spectra',
101 117 path='/path/to/pdata/',
102 118 startDate='2014/01/31',
103 119 endDate='2014/03/31',
104 120 startTime='00:00:00',
105 121 endTime='23:59:59',
106 122 online=0,
107 123 walk=0)
108 124
109 125 proc_unit = controller.addProcUnit(datatype='Spectra',
110 126 inputId=read_unit.getId())
111 127
112 128 op = proc_unit.addOperation(name='selectChannels')
113 129 op.addParameter(name='channelList', value='0,1', format='intlist')
114 130
115 131 op = proc_unit.addOperation(name='selectHeights')
116 132 op.addParameter(name='minHei', value='80', format='float')
117 133 op.addParameter(name='maxHei', value='200', format='float')
118 134
119 135 op = proc_unit.addOperation(name='removeDC')
120 136
121 137 op = proc_unit.addOperation(name='SpectraPlot', optype='other')
122 138 op.addParameter(name='id', value='6', format='int')
123 139 op.addParameter(name='wintitle', value='Spectra', format='str')
124 140
125 141 op = procUnitConfObj1.addOperation(name='RTIPlot', optype='other')
126 142 op.addParameter(name='id', value='2', format='int')
127 143 op.addParameter(name='wintitle', value='RTI', format='str')
128 144
129 145 controller.start()
130 146
131 ```
147 ``` No newline at end of file
Show file before | Show file after | Hide comments
@@ -1,7 +1,8
1 1 '''
2 2 Created on Feb 7, 2012
3 3
4 4 @author $Author$
5 5 @version $Id$
6 6 '''
7 __version__ = "2.3" No newline at end of file
7
8 __version__ = "2.3"
Show file before | Show file after | Hide comments
@@ -1,1324 +1,1324
1 1 '''
2 2 Created on September , 2012
3 3 @author:
4 4 '''
5 5
6 6 import sys
7 7 import ast
8 8 import datetime
9 9 import traceback
10 10 import math
11 11 import time
12 12 from multiprocessing import Process, Queue, cpu_count
13 13
14 14 import schainpy
15 15 import schainpy.admin
16 16
17 17 from xml.etree.ElementTree import ElementTree, Element, SubElement, tostring
18 18 from xml.dom import minidom
19 19
20 20 from schainpy.model import *
21 21 from time import sleep
22 22
23 23 def prettify(elem):
24 24 """Return a pretty-printed XML string for the Element.
25 25 """
26 26 rough_string = tostring(elem, 'utf-8')
27 27 reparsed = minidom.parseString(rough_string)
28 28 return reparsed.toprettyxml(indent=" ")
29 29
30 30 def multiSchain(child, nProcess=cpu_count(), startDate=None, endDate=None, by_day=False):
31 31 skip = 0
32 32 cursor = 0
33 33 nFiles = None
34 34 processes = []
35 35 dt1 = datetime.datetime.strptime(startDate, '%Y/%m/%d')
36 36 dt2 = datetime.datetime.strptime(endDate, '%Y/%m/%d')
37 37 days = (dt2 - dt1).days
38 38
39 39 for day in range(days+1):
40 40 skip = 0
41 41 cursor = 0
42 42 q = Queue()
43 43 processes = []
44 44 dt = (dt1 + datetime.timedelta(day)).strftime('%Y/%m/%d')
45 45 firstProcess = Process(target=child, args=(cursor, skip, q, dt))
46 46 firstProcess.start()
47 47 if by_day:
48 48 continue
49 49 nFiles = q.get()
50 50 firstProcess.terminate()
51 51 skip = int(math.ceil(nFiles/nProcess))
52 52 while True:
53 53 processes.append(Process(target=child, args=(cursor, skip, q, dt)))
54 54 processes[cursor].start()
55 55 if nFiles < cursor*skip:
56 56 break
57 57 cursor += 1
58 58
59 59 def beforeExit(exctype, value, trace):
60 60 for process in processes:
61 61 process.terminate()
62 62 process.join()
63 63 print traceback.print_tb(trace)
64 64
65 65 sys.excepthook = beforeExit
66 66
67 67 for process in processes:
68 68 process.join()
69 69 process.terminate()
70 70 time.sleep(3)
71 71
72 72 class ParameterConf():
73 73
74 74 id = None
75 75 name = None
76 76 value = None
77 77 format = None
78 78
79 79 __formated_value = None
80 80
81 81 ELEMENTNAME = 'Parameter'
82 82
83 83 def __init__(self):
84 84
85 85 self.format = 'str'
86 86
87 87 def getElementName(self):
88 88
89 89 return self.ELEMENTNAME
90 90
91 91 def getValue(self):
92 92
93 93 value = self.value
94 94 format = self.format
95 95
96 96 if self.__formated_value != None:
97 97
98 98 return self.__formated_value
99 99
100 100 if format == 'obj':
101 101 return value
102 102
103 103 if format == 'str':
104 104 self.__formated_value = str(value)
105 105 return self.__formated_value
106 106
107 107 if value == '':
108 108 raise ValueError, "%s: This parameter value is empty" %self.name
109 109
110 110 if format == 'list':
111 111 strList = value.split(',')
112 112
113 113 self.__formated_value = strList
114 114
115 115 return self.__formated_value
116 116
117 117 if format == 'intlist':
118 118 """
119 119 Example:
120 120 value = (0,1,2)
121 121 """
122 122
123 123 new_value = ast.literal_eval(value)
124 124
125 125 if type(new_value) not in (tuple, list):
126 126 new_value = [int(new_value)]
127 127
128 128 self.__formated_value = new_value
129 129
130 130 return self.__formated_value
131 131
132 132 if format == 'floatlist':
133 133 """
134 134 Example:
135 135 value = (0.5, 1.4, 2.7)
136 136 """
137 137
138 138 new_value = ast.literal_eval(value)
139 139
140 140 if type(new_value) not in (tuple, list):
141 141 new_value = [float(new_value)]
142 142
143 143 self.__formated_value = new_value
144 144
145 145 return self.__formated_value
146 146
147 147 if format == 'date':
148 148 strList = value.split('/')
149 149 intList = [int(x) for x in strList]
150 150 date = datetime.date(intList[0], intList[1], intList[2])
151 151
152 152 self.__formated_value = date
153 153
154 154 return self.__formated_value
155 155
156 156 if format == 'time':
157 157 strList = value.split(':')
158 158 intList = [int(x) for x in strList]
159 159 time = datetime.time(intList[0], intList[1], intList[2])
160 160
161 161 self.__formated_value = time
162 162
163 163 return self.__formated_value
164 164
165 165 if format == 'pairslist':
166 166 """
167 167 Example:
168 168 value = (0,1),(1,2)
169 169 """
170 170
171 171 new_value = ast.literal_eval(value)
172 172
173 173 if type(new_value) not in (tuple, list):
174 174 raise ValueError, "%s has to be a tuple or list of pairs" %value
175 175
176 176 if type(new_value[0]) not in (tuple, list):
177 177 if len(new_value) != 2:
178 178 raise ValueError, "%s has to be a tuple or list of pairs" %value
179 179 new_value = [new_value]
180 180
181 181 for thisPair in new_value:
182 182 if len(thisPair) != 2:
183 183 raise ValueError, "%s has to be a tuple or list of pairs" %value
184 184
185 185 self.__formated_value = new_value
186 186
187 187 return self.__formated_value
188 188
189 189 if format == 'multilist':
190 190 """
191 191 Example:
192 192 value = (0,1,2),(3,4,5)
193 193 """
194 194 multiList = ast.literal_eval(value)
195 195
196 196 if type(multiList[0]) == int:
197 197 multiList = ast.literal_eval("(" + value + ")")
198 198
199 199 self.__formated_value = multiList
200 200
201 201 return self.__formated_value
202 202
203 203 if format == 'bool':
204 204 value = int(value)
205 205
206 206 if format == 'int':
207 207 value = float(value)
208 208
209 209 format_func = eval(format)
210 210
211 211 self.__formated_value = format_func(value)
212 212
213 213 return self.__formated_value
214 214
215 215 def updateId(self, new_id):
216 216
217 217 self.id = str(new_id)
218 218
219 219 def setup(self, id, name, value, format='str'):
220 220 self.id = str(id)
221 221 self.name = name
222 222 if format == 'obj':
223 223 self.value = value
224 224 else:
225 225 self.value = str(value)
226 226 self.format = str.lower(format)
227 227
228 228 self.getValue()
229 229
230 230 return 1
231 231
232 232 def update(self, name, value, format='str'):
233 233
234 234 self.name = name
235 235 self.value = str(value)
236 236 self.format = format
237 237
238 238 def makeXml(self, opElement):
239 239 if self.name not in ('queue',):
240 240 parmElement = SubElement(opElement, self.ELEMENTNAME)
241 241 parmElement.set('id', str(self.id))
242 242 parmElement.set('name', self.name)
243 243 parmElement.set('value', self.value)
244 244 parmElement.set('format', self.format)
245 245
246 246 def readXml(self, parmElement):
247 247
248 248 self.id = parmElement.get('id')
249 249 self.name = parmElement.get('name')
250 250 self.value = parmElement.get('value')
251 251 self.format = str.lower(parmElement.get('format'))
252 252
253 253 #Compatible with old signal chain version
254 254 if self.format == 'int' and self.name == 'idfigure':
255 255 self.name = 'id'
256 256
257 257 def printattr(self):
258 258
259 259 print "Parameter[%s]: name = %s, value = %s, format = %s" %(self.id, self.name, self.value, self.format)
260 260
261 261 class OperationConf():
262 262
263 263 id = None
264 264 name = None
265 265 priority = None
266 266 type = None
267 267
268 268 parmConfObjList = []
269 269
270 270 ELEMENTNAME = 'Operation'
271 271
272 272 def __init__(self):
273 273
274 274 self.id = '0'
275 275 self.name = None
276 276 self.priority = None
277 277 self.type = 'self'
278 278
279 279
280 280 def __getNewId(self):
281 281
282 282 return int(self.id)*10 + len(self.parmConfObjList) + 1
283 283
284 284 def updateId(self, new_id):
285 285
286 286 self.id = str(new_id)
287 287
288 288 n = 1
289 289 for parmObj in self.parmConfObjList:
290 290
291 291 idParm = str(int(new_id)*10 + n)
292 292 parmObj.updateId(idParm)
293 293
294 294 n += 1
295 295
296 296 def getElementName(self):
297 297
298 298 return self.ELEMENTNAME
299 299
300 300 def getParameterObjList(self):
301 301
302 302 return self.parmConfObjList
303 303
304 304 def getParameterObj(self, parameterName):
305 305
306 306 for parmConfObj in self.parmConfObjList:
307 307
308 308 if parmConfObj.name != parameterName:
309 309 continue
310 310
311 311 return parmConfObj
312 312
313 313 return None
314 314
315 315 def getParameterObjfromValue(self, parameterValue):
316 316
317 317 for parmConfObj in self.parmConfObjList:
318 318
319 319 if parmConfObj.getValue() != parameterValue:
320 320 continue
321 321
322 322 return parmConfObj.getValue()
323 323
324 324 return None
325 325
326 326 def getParameterValue(self, parameterName):
327 327
328 328 parameterObj = self.getParameterObj(parameterName)
329
330 # if not parameterObj:
331 # return None
332
329
330 # if not parameterObj:
331 # return None
332
333 333 value = parameterObj.getValue()
334 334
335 335 return value
336 336
337 337
338 338 def getKwargs(self):
339 339
340 340 kwargs = {}
341 341
342 342 for parmConfObj in self.parmConfObjList:
343 343 if self.name == 'run' and parmConfObj.name == 'datatype':
344 344 continue
345 345
346 346 kwargs[parmConfObj.name] = parmConfObj.getValue()
347 347
348 348 return kwargs
349 349
350 350 def setup(self, id, name, priority, type):
351 351
352 352 self.id = str(id)
353 353 self.name = name
354 354 self.type = type
355 355 self.priority = priority
356 356
357 357 self.parmConfObjList = []
358 358
359 359 def removeParameters(self):
360 360
361 361 for obj in self.parmConfObjList:
362 362 del obj
363 363
364 364 self.parmConfObjList = []
365 365
366 366 def addParameter(self, name, value, format='str'):
367 367
368 368 id = self.__getNewId()
369 369
370 370 parmConfObj = ParameterConf()
371 371 if not parmConfObj.setup(id, name, value, format):
372 372 return None
373 373
374 374 self.parmConfObjList.append(parmConfObj)
375 375
376 376 return parmConfObj
377 377
378 378 def changeParameter(self, name, value, format='str'):
379 379
380 380 parmConfObj = self.getParameterObj(name)
381 381 parmConfObj.update(name, value, format)
382 382
383 383 return parmConfObj
384 384
385 385 def makeXml(self, procUnitElement):
386 386
387 387 opElement = SubElement(procUnitElement, self.ELEMENTNAME)
388 388 opElement.set('id', str(self.id))
389 389 opElement.set('name', self.name)
390 390 opElement.set('type', self.type)
391 391 opElement.set('priority', str(self.priority))
392 392
393 393 for parmConfObj in self.parmConfObjList:
394 394 parmConfObj.makeXml(opElement)
395 395
396 396 def readXml(self, opElement):
397 397
398 398 self.id = opElement.get('id')
399 399 self.name = opElement.get('name')
400 400 self.type = opElement.get('type')
401 401 self.priority = opElement.get('priority')
402 402
403 403 #Compatible with old signal chain version
404 404 #Use of 'run' method instead 'init'
405 405 if self.type == 'self' and self.name == 'init':
406 406 self.name = 'run'
407 407
408 408 self.parmConfObjList = []
409 409
410 410 parmElementList = opElement.iter(ParameterConf().getElementName())
411 411
412 412 for parmElement in parmElementList:
413 413 parmConfObj = ParameterConf()
414 414 parmConfObj.readXml(parmElement)
415 415
416 416 #Compatible with old signal chain version
417 417 #If an 'plot' OPERATION is found, changes name operation by the value of its type PARAMETER
418 418 if self.type != 'self' and self.name == 'Plot':
419 419 if parmConfObj.format == 'str' and parmConfObj.name == 'type':
420 420 self.name = parmConfObj.value
421 421 continue
422 422
423 423 self.parmConfObjList.append(parmConfObj)
424 424
425 425 def printattr(self):
426 426
427 427 print "%s[%s]: name = %s, type = %s, priority = %s" %(self.ELEMENTNAME,
428 428 self.id,
429 429 self.name,
430 430 self.type,
431 431 self.priority)
432 432
433 433 for parmConfObj in self.parmConfObjList:
434 434 parmConfObj.printattr()
435 435
436 436 def createObject(self, plotter_queue=None):
437 437
438 438
439 439 if self.type == 'self':
440 440 raise ValueError, "This operation type cannot be created"
441 441
442 442 if self.type == 'plotter':
443 443 #Plotter(plotter_name)
444 444 if not plotter_queue:
445 445 raise ValueError, "plotter_queue is not defined. Use:\nmyProject = Project()\nmyProject.setPlotterQueue(plotter_queue)"
446 446
447 447 opObj = Plotter(self.name, plotter_queue)
448 448
449 449 if self.type == 'external' or self.type == 'other':
450 450
451 451 className = eval(self.name)
452 452 kwargs = self.getKwargs()
453 453
454 454 opObj = className(**kwargs)
455 455
456 456 return opObj
457 457
458 458
459 459 class ProcUnitConf():
460 460
461 461 id = None
462 462 name = None
463 463 datatype = None
464 464 inputId = None
465 465 parentId = None
466 466
467 467 opConfObjList = []
468 468
469 469 procUnitObj = None
470 470 opObjList = []
471 471
472 472 ELEMENTNAME = 'ProcUnit'
473 473
474 474 def __init__(self):
475 475
476 476 self.id = None
477 477 self.datatype = None
478 478 self.name = None
479 479 self.inputId = None
480 480
481 481 self.opConfObjList = []
482 482
483 483 self.procUnitObj = None
484 484 self.opObjDict = {}
485 485
486 486 def __getPriority(self):
487 487
488 488 return len(self.opConfObjList)+1
489 489
490 490 def __getNewId(self):
491 491
492 492 return int(self.id)*10 + len(self.opConfObjList) + 1
493 493
494 494 def getElementName(self):
495 495
496 496 return self.ELEMENTNAME
497 497
498 498 def getId(self):
499 499
500 500 return self.id
501 501
502 502 def updateId(self, new_id, parentId=parentId):
503 503
504 504
505 505 new_id = int(parentId)*10 + (int(self.id) % 10)
506 506 new_inputId = int(parentId)*10 + (int(self.inputId) % 10)
507 507
508 508 #If this proc unit has not inputs
509 509 if self.inputId == '0':
510 510 new_inputId = 0
511 511
512 512 n = 1
513 513 for opConfObj in self.opConfObjList:
514 514
515 515 idOp = str(int(new_id)*10 + n)
516 516 opConfObj.updateId(idOp)
517 517
518 518 n += 1
519 519
520 520 self.parentId = str(parentId)
521 521 self.id = str(new_id)
522 522 self.inputId = str(new_inputId)
523 523
524 524
525 525 def getInputId(self):
526 526
527 527 return self.inputId
528 528
529 529 def getOperationObjList(self):
530 530
531 531 return self.opConfObjList
532 532
533 533 def getOperationObj(self, name=None):
534 534
535 535 for opConfObj in self.opConfObjList:
536 536
537 537 if opConfObj.name != name:
538 538 continue
539 539
540 540 return opConfObj
541 541
542 542 return None
543 543
544 544 def getOpObjfromParamValue(self, value=None):
545 545
546 546 for opConfObj in self.opConfObjList:
547 547 if opConfObj.getParameterObjfromValue(parameterValue=value) != value:
548 548 continue
549 549 return opConfObj
550 550 return None
551 551
552 552 def getProcUnitObj(self):
553 553
554 554 return self.procUnitObj
555 555
556 556 def setup(self, id, name, datatype, inputId, parentId=None):
557 557
558 558 #Compatible with old signal chain version
559 559 if datatype==None and name==None:
560 560 raise ValueError, "datatype or name should be defined"
561 561
562 562 if name==None:
563 563 if 'Proc' in datatype:
564 564 name = datatype
565 565 else:
566 566 name = '%sProc' %(datatype)
567 567
568 568 if datatype==None:
569 569 datatype = name.replace('Proc','')
570 570
571 571 self.id = str(id)
572 572 self.name = name
573 573 self.datatype = datatype
574 574 self.inputId = inputId
575 575 self.parentId = parentId
576 576
577 577 self.opConfObjList = []
578 578
579 579 self.addOperation(name='run', optype='self')
580 580
581 581 def removeOperations(self):
582 582
583 583 for obj in self.opConfObjList:
584 584 del obj
585 585
586 586 self.opConfObjList = []
587 587 self.addOperation(name='run')
588 588
589 589 def addParameter(self, **kwargs):
590 590 '''
591 591 Add parameters to "run" operation
592 592 '''
593 593 opObj = self.opConfObjList[0]
594 594
595 595 opObj.addParameter(**kwargs)
596 596
597 597 return opObj
598 598
599 599 def addOperation(self, name, optype='self'):
600 600
601 601 id = self.__getNewId()
602 602 priority = self.__getPriority()
603 603
604 604 opConfObj = OperationConf()
605 605 opConfObj.setup(id, name=name, priority=priority, type=optype)
606 606
607 607 self.opConfObjList.append(opConfObj)
608 608
609 609 return opConfObj
610 610
611 611 def makeXml(self, projectElement):
612 612
613 613 procUnitElement = SubElement(projectElement, self.ELEMENTNAME)
614 614 procUnitElement.set('id', str(self.id))
615 615 procUnitElement.set('name', self.name)
616 616 procUnitElement.set('datatype', self.datatype)
617 617 procUnitElement.set('inputId', str(self.inputId))
618 618
619 619 for opConfObj in self.opConfObjList:
620 620 opConfObj.makeXml(procUnitElement)
621 621
622 622 def readXml(self, upElement):
623 623
624 624 self.id = upElement.get('id')
625 625 self.name = upElement.get('name')
626 626 self.datatype = upElement.get('datatype')
627 627 self.inputId = upElement.get('inputId')
628 628
629 629 if self.ELEMENTNAME == "ReadUnit":
630 630 self.datatype = self.datatype.replace("Reader", "")
631 631
632 632 if self.ELEMENTNAME == "ProcUnit":
633 633 self.datatype = self.datatype.replace("Proc", "")
634 634
635 635 if self.inputId == 'None':
636 636 self.inputId = '0'
637 637
638 638 self.opConfObjList = []
639 639
640 640 opElementList = upElement.iter(OperationConf().getElementName())
641 641
642 642 for opElement in opElementList:
643 643 opConfObj = OperationConf()
644 644 opConfObj.readXml(opElement)
645 645 self.opConfObjList.append(opConfObj)
646 646
647 647 def printattr(self):
648 648
649 649 print "%s[%s]: name = %s, datatype = %s, inputId = %s" %(self.ELEMENTNAME,
650 self.id,
651 self.name,
652 self.datatype,
653 self.inputId)
654
650 self.id,
651 self.name,
652 self.datatype,
653 self.inputId)
654
655 655 for opConfObj in self.opConfObjList:
656 656 opConfObj.printattr()
657 657
658 658
659 659 def getKwargs(self):
660 660
661 661 opObj = self.opConfObjList[0]
662 662 kwargs = opObj.getKwargs()
663 663
664 664 return kwargs
665 665
666 666 def createObjects(self, plotter_queue=None):
667 667
668 668 className = eval(self.name)
669 669 kwargs = self.getKwargs()
670 670 procUnitObj = className(**kwargs)
671 671
672 672 for opConfObj in self.opConfObjList:
673 673
674 674 if opConfObj.type=='self' and self.name=='run':
675 675 continue
676 676 elif opConfObj.type=='self':
677 677 procUnitObj.addOperationKwargs(opConfObj.id, **opConfObj.getKwargs())
678 678 continue
679 679
680 680 opObj = opConfObj.createObject(plotter_queue)
681 681
682 682 self.opObjDict[opConfObj.id] = opObj
683 683
684 684 procUnitObj.addOperation(opObj, opConfObj.id)
685 685
686 686 self.procUnitObj = procUnitObj
687 687
688 688 return procUnitObj
689 689
690 690 def run(self):
691 691
692 692 is_ok = False
693 693
694 694 for opConfObj in self.opConfObjList:
695 695
696 696 kwargs = {}
697 697 for parmConfObj in opConfObj.getParameterObjList():
698 698 if opConfObj.name == 'run' and parmConfObj.name == 'datatype':
699 699 continue
700 700
701 701 kwargs[parmConfObj.name] = parmConfObj.getValue()
702 702
703 703 #ini = time.time()
704 704
705 705 #print "\tRunning the '%s' operation with %s" %(opConfObj.name, opConfObj.id)
706 706 sts = self.procUnitObj.call(opType = opConfObj.type,
707 707 opName = opConfObj.name,
708 708 opId = opConfObj.id,
709 )
710
711 # total_time = time.time() - ini
712 #
713 # if total_time > 0.002:
714 # print "%s::%s took %f seconds" %(self.name, opConfObj.name, total_time)
715
709 **kwargs)
710
711 # total_time = time.time() - ini
712 #
713 # if total_time > 0.002:
714 # print "%s::%s took %f seconds" %(self.name, opConfObj.name, total_time)
715
716 716 is_ok = is_ok or sts
717 717
718 718 return is_ok
719 719
720 720 def close(self):
721 721
722 722 for opConfObj in self.opConfObjList:
723 723 if opConfObj.type == 'self':
724 724 continue
725 725
726 726 opObj = self.procUnitObj.getOperationObj(opConfObj.id)
727 727 opObj.close()
728 728
729 729 self.procUnitObj.close()
730 730
731 731 return
732 732
733 733 class ReadUnitConf(ProcUnitConf):
734 734
735 735 path = None
736 736 startDate = None
737 737 endDate = None
738 738 startTime = None
739 739 endTime = None
740 740
741 741 ELEMENTNAME = 'ReadUnit'
742 742
743 743 def __init__(self):
744 744
745 745 self.id = None
746 746 self.datatype = None
747 747 self.name = None
748 748 self.inputId = None
749 749
750 750 self.parentId = None
751 751
752 752 self.opConfObjList = []
753 753 self.opObjList = []
754 754
755 755 def getElementName(self):
756 756
757 757 return self.ELEMENTNAME
758 758
759 759 def setup(self, id, name, datatype, path='', startDate="", endDate="", startTime="",
760 760 endTime="", parentId=None, queue=None, server=None, **kwargs):
761 761
762 762 #Compatible with old signal chain version
763 763 if datatype==None and name==None:
764 764 raise ValueError, "datatype or name should be defined"
765 765
766 766 if name==None:
767 767 if 'Reader' in datatype:
768 768 name = datatype
769 769 else:
770 770 name = '%sReader' %(datatype)
771 771 if datatype==None:
772 772 datatype = name.replace('Reader','')
773 773
774 774 self.id = id
775 775 self.name = name
776 776 self.datatype = datatype
777 777 if path != '':
778 778 self.path = os.path.abspath(path)
779 779 self.startDate = startDate
780 780 self.endDate = endDate
781 781 self.startTime = startTime
782 782 self.endTime = endTime
783 783
784 784 self.inputId = '0'
785 785 self.parentId = parentId
786 786 self.queue = queue
787 787 self.server = server
788 788 self.addRunOperation(**kwargs)
789 789
790 790 def update(self, datatype, path, startDate, endDate, startTime, endTime, parentId=None, name=None, **kwargs):
791 791
792 792 #Compatible with old signal chain version
793 793 if datatype==None and name==None:
794 794 raise ValueError, "datatype or name should be defined"
795 795
796 796 if name==None:
797 797 if 'Reader' in datatype:
798 798 name = datatype
799 799 else:
800 800 name = '%sReader' %(datatype)
801 801
802 802 if datatype==None:
803 803 datatype = name.replace('Reader','')
804 804
805 805 self.datatype = datatype
806 806 self.name = name
807 807 self.path = path
808 808 self.startDate = startDate
809 809 self.endDate = endDate
810 810 self.startTime = startTime
811 811 self.endTime = endTime
812 812
813 813 self.inputId = '0'
814 814 self.parentId = parentId
815 815
816 816 self.updateRunOperation(**kwargs)
817 817
818 818 def removeOperations(self):
819 819
820 820 for obj in self.opConfObjList:
821 821 del obj
822 822
823 823 self.opConfObjList = []
824 824
825 825 def addRunOperation(self, **kwargs):
826 826
827 827 opObj = self.addOperation(name = 'run', optype = 'self')
828 828
829 829 if self.server is None:
830 830 opObj.addParameter(name='datatype' , value=self.datatype, format='str')
831 831 opObj.addParameter(name='path' , value=self.path, format='str')
832 832 opObj.addParameter(name='startDate' , value=self.startDate, format='date')
833 833 opObj.addParameter(name='endDate' , value=self.endDate, format='date')
834 834 opObj.addParameter(name='startTime' , value=self.startTime, format='time')
835 835 opObj.addParameter(name='endTime' , value=self.endTime, format='time')
836 836 opObj.addParameter(name='queue' , value=self.queue, format='obj')
837 837 for key, value in kwargs.items():
838 838 opObj.addParameter(name=key, value=value, format=type(value).__name__)
839 839 else:
840 840 opObj.addParameter(name='server' , value=self.server, format='str')
841 841
842 842
843 843 return opObj
844 844
845 845 def updateRunOperation(self, **kwargs):
846 846
847 847 opObj = self.getOperationObj(name = 'run')
848 848 opObj.removeParameters()
849 849
850 850 opObj.addParameter(name='datatype' , value=self.datatype, format='str')
851 851 opObj.addParameter(name='path' , value=self.path, format='str')
852 852 opObj.addParameter(name='startDate' , value=self.startDate, format='date')
853 853 opObj.addParameter(name='endDate' , value=self.endDate, format='date')
854 854 opObj.addParameter(name='startTime' , value=self.startTime, format='time')
855 855 opObj.addParameter(name='endTime' , value=self.endTime, format='time')
856 856
857 857 for key, value in kwargs.items():
858 858 opObj.addParameter(name=key, value=value, format=type(value).__name__)
859 859
860 860 return opObj
861
862 # def makeXml(self, projectElement):
863 #
864 # procUnitElement = SubElement(projectElement, self.ELEMENTNAME)
865 # procUnitElement.set('id', str(self.id))
866 # procUnitElement.set('name', self.name)
867 # procUnitElement.set('datatype', self.datatype)
868 # procUnitElement.set('inputId', str(self.inputId))
869 #
870 # for opConfObj in self.opConfObjList:
871 # opConfObj.makeXml(procUnitElement)
872
861
862 # def makeXml(self, projectElement):
863 #
864 # procUnitElement = SubElement(projectElement, self.ELEMENTNAME)
865 # procUnitElement.set('id', str(self.id))
866 # procUnitElement.set('name', self.name)
867 # procUnitElement.set('datatype', self.datatype)
868 # procUnitElement.set('inputId', str(self.inputId))
869 #
870 # for opConfObj in self.opConfObjList:
871 # opConfObj.makeXml(procUnitElement)
872
873 873 def readXml(self, upElement):
874 874
875 875 self.id = upElement.get('id')
876 876 self.name = upElement.get('name')
877 877 self.datatype = upElement.get('datatype')
878 878 self.inputId = upElement.get('inputId')
879 879
880 880 if self.ELEMENTNAME == "ReadUnit":
881 881 self.datatype = self.datatype.replace("Reader", "")
882 882
883 883 if self.inputId == 'None':
884 884 self.inputId = '0'
885 885
886 886 self.opConfObjList = []
887 887
888 888 opElementList = upElement.iter(OperationConf().getElementName())
889 889
890 890 for opElement in opElementList:
891 891 opConfObj = OperationConf()
892 892 opConfObj.readXml(opElement)
893 893 self.opConfObjList.append(opConfObj)
894 894
895 895 if opConfObj.name == 'run':
896 896 self.path = opConfObj.getParameterValue('path')
897 897 self.startDate = opConfObj.getParameterValue('startDate')
898 898 self.endDate = opConfObj.getParameterValue('endDate')
899 899 self.startTime = opConfObj.getParameterValue('startTime')
900 900 self.endTime = opConfObj.getParameterValue('endTime')
901 901
902 902 class Project():
903 903
904 904 id = None
905 905 name = None
906 906 description = None
907 907 filename = None
908 908
909 909 procUnitConfObjDict = None
910 910
911 911 ELEMENTNAME = 'Project'
912 912
913 913 plotterQueue = None
914 914
915 915 def __init__(self, plotter_queue=None):
916 916
917 917 self.id = None
918 918 self.name = None
919 919 self.description = None
920 920
921 921 self.plotterQueue = plotter_queue
922 922
923 923 self.procUnitConfObjDict = {}
924 924
925 925 def __getNewId(self):
926 926
927 927 idList = self.procUnitConfObjDict.keys()
928 928
929 929 id = int(self.id)*10
930 930
931 931 while True:
932 932 id += 1
933 933
934 934 if str(id) in idList:
935 935 continue
936 936
937 937 break
938 938
939 939 return str(id)
940 940
941 941 def getElementName(self):
942 942
943 943 return self.ELEMENTNAME
944 944
945 945 def getId(self):
946 946
947 947 return self.id
948 948
949 949 def updateId(self, new_id):
950 950
951 951 self.id = str(new_id)
952 952
953 953 keyList = self.procUnitConfObjDict.keys()
954 954 keyList.sort()
955 955
956 956 n = 1
957 957 newProcUnitConfObjDict = {}
958 958
959 959 for procKey in keyList:
960 960
961 961 procUnitConfObj = self.procUnitConfObjDict[procKey]
962 962 idProcUnit = str(int(self.id)*10 + n)
963 963 procUnitConfObj.updateId(idProcUnit, parentId = self.id)
964 964
965 965 newProcUnitConfObjDict[idProcUnit] = procUnitConfObj
966 966 n += 1
967 967
968 968 self.procUnitConfObjDict = newProcUnitConfObjDict
969 969
970 970 def setup(self, id, name, description):
971 971
972 972 self.id = str(id)
973 973 self.name = name
974 974 self.description = description
975 975
976 976 def update(self, name, description):
977 977
978 978 self.name = name
979 979 self.description = description
980 980
981 981 def addReadUnit(self, id=None, datatype=None, name=None, **kwargs):
982 982
983 983 if id is None:
984 984 idReadUnit = self.__getNewId()
985 985 else:
986 986 idReadUnit = str(id)
987 987
988 988 readUnitConfObj = ReadUnitConf()
989 989 readUnitConfObj.setup(idReadUnit, name, datatype, parentId=self.id, **kwargs)
990 990
991 991 self.procUnitConfObjDict[readUnitConfObj.getId()] = readUnitConfObj
992 992
993 993 return readUnitConfObj
994 994
995 995 def addProcUnit(self, inputId='0', datatype=None, name=None):
996 996
997 997 idProcUnit = self.__getNewId()
998 998
999 999 procUnitConfObj = ProcUnitConf()
1000 1000 procUnitConfObj.setup(idProcUnit, name, datatype, inputId, parentId=self.id)
1001 1001
1002 1002 self.procUnitConfObjDict[procUnitConfObj.getId()] = procUnitConfObj
1003 1003
1004 1004 return procUnitConfObj
1005 1005
1006 1006 def removeProcUnit(self, id):
1007 1007
1008 1008 if id in self.procUnitConfObjDict.keys():
1009 1009 self.procUnitConfObjDict.pop(id)
1010 1010
1011 1011 def getReadUnitId(self):
1012 1012
1013 1013 readUnitConfObj = self.getReadUnitObj()
1014 1014
1015 1015 return readUnitConfObj.id
1016 1016
1017 1017 def getReadUnitObj(self):
1018 1018
1019 1019 for obj in self.procUnitConfObjDict.values():
1020 1020 if obj.getElementName() == "ReadUnit":
1021 1021 return obj
1022 1022
1023 1023 return None
1024 1024
1025 1025 def getProcUnitObj(self, id=None, name=None):
1026 1026
1027 1027 if id != None:
1028 1028 return self.procUnitConfObjDict[id]
1029 1029
1030 1030 if name != None:
1031 1031 return self.getProcUnitObjByName(name)
1032 1032
1033 1033 return None
1034 1034
1035 1035 def getProcUnitObjByName(self, name):
1036 1036
1037 1037 for obj in self.procUnitConfObjDict.values():
1038 1038 if obj.name == name:
1039 1039 return obj
1040 1040
1041 1041 return None
1042 1042
1043 1043 def procUnitItems(self):
1044 1044
1045 1045 return self.procUnitConfObjDict.items()
1046 1046
1047 1047 def makeXml(self):
1048 1048
1049 1049 projectElement = Element('Project')
1050 1050 projectElement.set('id', str(self.id))
1051 1051 projectElement.set('name', self.name)
1052 1052 projectElement.set('description', self.description)
1053 1053
1054 1054 for procUnitConfObj in self.procUnitConfObjDict.values():
1055 1055 procUnitConfObj.makeXml(projectElement)
1056 1056
1057 1057 self.projectElement = projectElement
1058 1058
1059 1059 def writeXml(self, filename=None):
1060 1060
1061 1061 if filename == None:
1062 1062 if self.filename:
1063 1063 filename = self.filename
1064 1064 else:
1065 1065 filename = "schain.xml"
1066 1066
1067 1067 if not filename:
1068 1068 print "filename has not been defined. Use setFilename(filename) for do it."
1069 1069 return 0
1070 1070
1071 1071 abs_file = os.path.abspath(filename)
1072 1072
1073 1073 if not os.access(os.path.dirname(abs_file), os.W_OK):
1074 1074 print "No write permission on %s" %os.path.dirname(abs_file)
1075 1075 return 0
1076 1076
1077 1077 if os.path.isfile(abs_file) and not(os.access(abs_file, os.W_OK)):
1078 1078 print "File %s already exists and it could not be overwriten" %abs_file
1079 1079 return 0
1080 1080
1081 1081 self.makeXml()
1082 1082
1083 1083 ElementTree(self.projectElement).write(abs_file, method='xml')
1084 1084
1085 1085 self.filename = abs_file
1086 1086
1087 1087 return 1
1088 1088
1089 1089 def readXml(self, filename = None):
1090 1090
1091 1091 if not filename:
1092 1092 print "filename is not defined"
1093 1093 return 0
1094 1094
1095 1095 abs_file = os.path.abspath(filename)
1096 1096
1097 1097 if not os.path.isfile(abs_file):
1098 1098 print "%s file does not exist" %abs_file
1099 1099 return 0
1100 1100
1101 1101 self.projectElement = None
1102 1102 self.procUnitConfObjDict = {}
1103 1103
1104 1104 try:
1105 1105 self.projectElement = ElementTree().parse(abs_file)
1106 1106 except:
1107 1107 print "Error reading %s, verify file format" %filename
1108 1108 return 0
1109 1109
1110 1110 self.project = self.projectElement.tag
1111 1111
1112 1112 self.id = self.projectElement.get('id')
1113 1113 self.name = self.projectElement.get('name')
1114 self.description = self.projectElement.get('description')
1115
1114 self.description = self.projectElement.get('description')
1115
1116 1116 readUnitElementList = self.projectElement.iter(ReadUnitConf().getElementName())
1117 1117
1118 1118 for readUnitElement in readUnitElementList:
1119 1119 readUnitConfObj = ReadUnitConf()
1120 1120 readUnitConfObj.readXml(readUnitElement)
1121 1121
1122 1122 if readUnitConfObj.parentId == None:
1123 1123 readUnitConfObj.parentId = self.id
1124 1124
1125 1125 self.procUnitConfObjDict[readUnitConfObj.getId()] = readUnitConfObj
1126 1126
1127 1127 procUnitElementList = self.projectElement.iter(ProcUnitConf().getElementName())
1128 1128
1129 1129 for procUnitElement in procUnitElementList:
1130 1130 procUnitConfObj = ProcUnitConf()
1131 1131 procUnitConfObj.readXml(procUnitElement)
1132 1132
1133 1133 if procUnitConfObj.parentId == None:
1134 1134 procUnitConfObj.parentId = self.id
1135 1135
1136 1136 self.procUnitConfObjDict[procUnitConfObj.getId()] = procUnitConfObj
1137 1137
1138 1138 self.filename = abs_file
1139 1139
1140 1140 return 1
1141 1141
1142 1142 def printattr(self):
1143 1143
1144 1144 print "Project[%s]: name = %s, description = %s" %(self.id,
1145 self.name,
1146 self.description)
1145 self.name,
1146 self.description)
1147 1147
1148 1148 for procUnitConfObj in self.procUnitConfObjDict.values():
1149 1149 procUnitConfObj.printattr()
1150 1150
1151 1151 def createObjects(self):
1152 1152
1153 1153 for procUnitConfObj in self.procUnitConfObjDict.values():
1154 1154 procUnitConfObj.createObjects(self.plotterQueue)
1155 1155
1156 1156 def __connect(self, objIN, thisObj):
1157 1157
1158 1158 thisObj.setInput(objIN.getOutputObj())
1159 1159
1160 1160 def connectObjects(self):
1161 1161
1162 1162 for thisPUConfObj in self.procUnitConfObjDict.values():
1163 1163
1164 1164 inputId = thisPUConfObj.getInputId()
1165 1165
1166 1166 if int(inputId) == 0:
1167 1167 continue
1168 1168
1169 1169 #Get input object
1170 1170 puConfINObj = self.procUnitConfObjDict[inputId]
1171 1171 puObjIN = puConfINObj.getProcUnitObj()
1172 1172
1173 1173 #Get current object
1174 1174 thisPUObj = thisPUConfObj.getProcUnitObj()
1175 1175
1176 1176 self.__connect(puObjIN, thisPUObj)
1177 1177
1178 1178 def __handleError(self, procUnitConfObj, send_email=True):
1179 1179
1180 1180 import socket
1181 1181
1182 1182 err = traceback.format_exception(sys.exc_info()[0],
1183 sys.exc_info()[1],
1184 sys.exc_info()[2])
1185
1183 sys.exc_info()[1],
1184 sys.exc_info()[2])
1185
1186 1186 print "***** Error occurred in %s *****" %(procUnitConfObj.name)
1187 1187 print "***** %s" %err[-1]
1188 1188
1189 1189 message = "".join(err)
1190 1190
1191 1191 sys.stderr.write(message)
1192 1192
1193 1193 if not send_email:
1194 1194 return
1195 1195
1196 1196 subject = "SChain v%s: Error running %s\n" %(schainpy.__version__, procUnitConfObj.name)
1197 1197
1198 1198 subtitle = "%s: %s\n" %(procUnitConfObj.getElementName() ,procUnitConfObj.name)
1199 1199 subtitle += "Hostname: %s\n" %socket.gethostbyname(socket.gethostname())
1200 1200 subtitle += "Working directory: %s\n" %os.path.abspath("./")
1201 1201 subtitle += "Configuration file: %s\n" %self.filename
1202 1202 subtitle += "Time: %s\n" %str(datetime.datetime.now())
1203 1203
1204 1204 readUnitConfObj = self.getReadUnitObj()
1205 1205 if readUnitConfObj:
1206 1206 subtitle += "\nInput parameters:\n"
1207 1207 subtitle += "[Data path = %s]\n" %readUnitConfObj.path
1208 1208 subtitle += "[Data type = %s]\n" %readUnitConfObj.datatype
1209 1209 subtitle += "[Start date = %s]\n" %readUnitConfObj.startDate
1210 1210 subtitle += "[End date = %s]\n" %readUnitConfObj.endDate
1211 1211 subtitle += "[Start time = %s]\n" %readUnitConfObj.startTime
1212 1212 subtitle += "[End time = %s]\n" %readUnitConfObj.endTime
1213 1213
1214 1214 adminObj = schainpy.admin.SchainNotify()
1215 1215 adminObj.sendAlert(message=message,
1216 subject=subject,
1217 subtitle=subtitle,
1218 filename=self.filename)
1219
1216 subject=subject,
1217 subtitle=subtitle,
1218 filename=self.filename)
1219
1220 1220 def isPaused(self):
1221 1221 return 0
1222 1222
1223 1223 def isStopped(self):
1224 1224 return 0
1225 1225
1226 1226 def runController(self):
1227 1227 """
1228 1228 returns 0 when this process has been stopped, 1 otherwise
1229 1229 """
1230 1230
1231 1231 if self.isPaused():
1232 1232 print "Process suspended"
1233 1233
1234 1234 while True:
1235 1235 sleep(0.1)
1236 1236
1237 1237 if not self.isPaused():
1238 1238 break
1239 1239
1240 1240 if self.isStopped():
1241 1241 break
1242 1242
1243 1243 print "Process reinitialized"
1244 1244
1245 1245 if self.isStopped():
1246 1246 print "Process stopped"
1247 1247 return 0
1248 1248
1249 1249 return 1
1250 1250
1251 1251 def setFilename(self, filename):
1252 1252
1253 1253 self.filename = filename
1254 1254
1255 1255 def setPlotterQueue(self, plotter_queue):
1256 1256
1257 1257 raise NotImplementedError, "Use schainpy.controller_api.ControllerThread instead Project class"
1258 1258
1259 1259 def getPlotterQueue(self):
1260 1260
1261 1261 raise NotImplementedError, "Use schainpy.controller_api.ControllerThread instead Project class"
1262 1262
1263 1263 def useExternalPlotter(self):
1264 1264
1265 1265 raise NotImplementedError, "Use schainpy.controller_api.ControllerThread instead Project class"
1266 1266
1267 1267 def run(self):
1268 1268
1269 1269 print
1270 1270 print "*"*60
1271 1271 print " Starting SIGNAL CHAIN PROCESSING v%s " %schainpy.__version__
1272 1272 print "*"*60
1273 1273 print
1274 1274
1275 1275 keyList = self.procUnitConfObjDict.keys()
1276 1276 keyList.sort()
1277 1277
1278 1278 while(True):
1279 1279
1280 1280 is_ok = False
1281 1281
1282 1282 for procKey in keyList:
1283 1283 # print "Running the '%s' process with %s" %(procUnitConfObj.name, procUnitConfObj.id)
1284 1284
1285 1285 procUnitConfObj = self.procUnitConfObjDict[procKey]
1286 1286
1287 1287 try:
1288 1288 sts = procUnitConfObj.run()
1289 1289 is_ok = is_ok or sts
1290 1290 except KeyboardInterrupt:
1291 1291 is_ok = False
1292 1292 break
1293 1293 except ValueError, e:
1294 1294 sleep(0.5)
1295 1295 self.__handleError(procUnitConfObj, send_email=True)
1296 1296 is_ok = False
1297 1297 break
1298 1298 except:
1299 1299 sleep(0.5)
1300 1300 self.__handleError(procUnitConfObj)
1301 1301 is_ok = False
1302 1302 break
1303 1303
1304 1304 #If every process unit finished so end process
1305 1305 if not(is_ok):
1306 1306 # print "Every process unit have finished"
1307 1307 break
1308 1308
1309 1309 if not self.runController():
1310 1310 break
1311 1311
1312 1312 #Closing every process
1313 1313 for procKey in keyList:
1314 1314 procUnitConfObj = self.procUnitConfObjDict[procKey]
1315 1315 procUnitConfObj.close()
1316 1316
1317 1317 print "Process finished"
1318 1318
1319 1319 def start(self, filename=None):
1320 1320
1321 1321 self.writeXml(filename)
1322 1322 self.createObjects()
1323 1323 self.connectObjects()
1324 1324 self.run()
Show file before | Show file after | Hide comments
@@ -1,179 +1,179
1 1 import threading
2 2 from Queue import Queue
3 3
4 4 from schainpy.controller import Project
5 5 from schainpy.model.graphics.jroplotter import PlotManager
6 6
7 7 class ControllerThread(threading.Thread, Project):
8 8
9 9 def __init__(self, plotter_queue=None):
10 10
11 11 threading.Thread.__init__(self)
12 12 Project.__init__(self, plotter_queue)
13 13
14 14 self.setDaemon(True)
15 15
16 16 self.lock = threading.Lock()
17 self.control = {'stop':False, 'pause':False}
18
17 self.control = { 'stop':False, 'pause':False }
18
19 19 def __del__(self):
20 20
21 21 self.control['stop'] = True
22 22
23 23 def stop(self):
24 24
25 25 self.lock.acquire()
26 26
27 27 self.control['stop'] = True
28 28
29 29 self.lock.release()
30 30
31 31 def pause(self):
32 32
33 33 self.lock.acquire()
34 34
35 35 self.control['pause'] = not(self.control['pause'])
36 36 paused = self.control['pause']
37 37
38 38 self.lock.release()
39 39
40 40 return paused
41 41
42 42 def isPaused(self):
43 43
44 44 self.lock.acquire()
45 45 paused = self.control['pause']
46 46 self.lock.release()
47 47
48 48 return paused
49 49
50 50 def isStopped(self):
51 51
52 52 self.lock.acquire()
53 53 stopped = self.control['stop']
54 54 self.lock.release()
55 55
56 56 return stopped
57 57
58 58 def run(self):
59 59 self.control['stop'] = False
60 60 self.control['pause'] = False
61 61
62 62 self.writeXml()
63 63
64 64 self.createObjects()
65 65 self.connectObjects()
66 66 Project.run(self)
67 67
68 68 def isRunning(self):
69 69
70 70 return self.is_alive()
71 71
72 72 def isFinished(self):
73 73
74 74 return not self.is_alive()
75 75
76 76 def setPlotters(self):
77 77
78 78 plotterList = PlotManager.plotterList
79 79
80 80 for thisPUConfObj in self.procUnitConfObjDict.values():
81 81
82 82 inputId = thisPUConfObj.getInputId()
83 83
84 84 if int(inputId) == 0:
85 85 continue
86 86
87 87 for thisOpObj in thisPUConfObj.getOperationObjList():
88 88
89 89 if thisOpObj.type == "self":
90 90 continue
91 91
92 92 if thisOpObj.name in plotterList:
93 thisOpObj.type = "plotter"
93 thisOpObj.type = "other"
94 94
95 95 def setPlotterQueue(self, plotter_queue):
96 96
97 97 self.plotterQueue = plotter_queue
98 98
99 99 def getPlotterQueue(self):
100 100
101 101 return self.plotterQueue
102 102
103 103 def useExternalPlotter(self):
104 104
105 105 self.plotterQueue = Queue(10)
106 106 self.setPlotters()
107 107
108 108 plotManagerObj = PlotManager(self.plotterQueue)
109 109 plotManagerObj.setController(self)
110 110
111 111 return plotManagerObj
112 112
113 113 # from PyQt4 import QtCore
114 114 # from PyQt4.QtCore import SIGNAL
115 115 #
116 116 # class ControllerQThread(QtCore.QThread, Project):
117 117 #
118 118 # def __init__(self, filename):
119 119 #
120 120 # QtCore.QThread.__init__(self)
121 121 # Project.__init__(self)
122 122 #
123 123 # self.filename = filename
124 124 #
125 125 # self.lock = threading.Lock()
126 126 # self.control = {'stop':False, 'pause':False}
127 127 #
128 128 # def __del__(self):
129 129 #
130 130 # self.control['stop'] = True
131 131 # self.wait()
132 132 #
133 133 # def stop(self):
134 134 #
135 135 # self.lock.acquire()
136 136 #
137 137 # self.control['stop'] = True
138 138 #
139 139 # self.lock.release()
140 140 #
141 141 # def pause(self):
142 142 #
143 143 # self.lock.acquire()
144 144 #
145 145 # self.control['pause'] = not(self.control['pause'])
146 146 # paused = self.control['pause']
147 147 #
148 148 # self.lock.release()
149 149 #
150 150 # return paused
151 151 #
152 152 # def isPaused(self):
153 153 #
154 154 # self.lock.acquire()
155 155 # paused = self.control['pause']
156 156 # self.lock.release()
157 157 #
158 158 # return paused
159 159 #
160 160 # def isStopped(self):
161 161 #
162 162 # self.lock.acquire()
163 163 # stopped = self.control['stop']
164 164 # self.lock.release()
165 165 #
166 166 # return stopped
167 167 #
168 168 # def run(self):
169 169 #
170 170 # self.control['stop'] = False
171 171 # self.control['pause'] = False
172 172 #
173 173 # self.readXml(self.filename)
174 174 # self.createObjects()
175 175 # self.connectObjects()
176 176 # self.emit( SIGNAL( "jobStarted( PyQt_PyObject )" ), 1)
177 177 # Project.run(self)
178 178 # self.emit( SIGNAL( "jobFinished( PyQt_PyObject )" ), 1)
179 179 #
Show file before | Show file after | Hide comments
@@ -1,1220 +1,1218
1 1 '''
2 2
3 3 $Author: murco $
4 4 $Id: JROData.py 173 2012-11-20 15:06:21Z murco $
5 5 '''
6 6
7 7 import copy
8 8 import numpy
9 9 import datetime
10 10
11 11 from jroheaderIO import SystemHeader, RadarControllerHeader
12 12 from schainpy import cSchain
13 13
14 14
15 15 def getNumpyDtype(dataTypeCode):
16 16
17 17 if dataTypeCode == 0:
18 18 numpyDtype = numpy.dtype([('real','<i1'),('imag','<i1')])
19 19 elif dataTypeCode == 1:
20 20 numpyDtype = numpy.dtype([('real','<i2'),('imag','<i2')])
21 21 elif dataTypeCode == 2:
22 22 numpyDtype = numpy.dtype([('real','<i4'),('imag','<i4')])
23 23 elif dataTypeCode == 3:
24 24 numpyDtype = numpy.dtype([('real','<i8'),('imag','<i8')])
25 25 elif dataTypeCode == 4:
26 26 numpyDtype = numpy.dtype([('real','<f4'),('imag','<f4')])
27 27 elif dataTypeCode == 5:
28 28 numpyDtype = numpy.dtype([('real','<f8'),('imag','<f8')])
29 29 else:
30 30 raise ValueError, 'dataTypeCode was not defined'
31 31
32 32 return numpyDtype
33 33
34 34 def getDataTypeCode(numpyDtype):
35 35
36 36 if numpyDtype == numpy.dtype([('real','<i1'),('imag','<i1')]):
37 37 datatype = 0
38 38 elif numpyDtype == numpy.dtype([('real','<i2'),('imag','<i2')]):
39 39 datatype = 1
40 40 elif numpyDtype == numpy.dtype([('real','<i4'),('imag','<i4')]):
41 41 datatype = 2
42 42 elif numpyDtype == numpy.dtype([('real','<i8'),('imag','<i8')]):
43 43 datatype = 3
44 44 elif numpyDtype == numpy.dtype([('real','<f4'),('imag','<f4')]):
45 45 datatype = 4
46 46 elif numpyDtype == numpy.dtype([('real','<f8'),('imag','<f8')]):
47 47 datatype = 5
48 48 else:
49 49 datatype = None
50 50
51 51 return datatype
52 52
53 53 def hildebrand_sekhon(data, navg):
54 54 """
55 55 This method is for the objective determination of the noise level in Doppler spectra. This
56 56 implementation technique is based on the fact that the standard deviation of the spectral
57 57 densities is equal to the mean spectral density for white Gaussian noise
58 58
59 59 Inputs:
60 60 Data : heights
61 61 navg : numbers of averages
62 62
63 63 Return:
64 64 -1 : any error
65 65 anoise : noise's level
66 66 """
67 67
68 68 sortdata = numpy.sort(data, axis=None)
69 69 # lenOfData = len(sortdata)
70 70 # nums_min = lenOfData*0.2
71 71 #
72 72 # if nums_min <= 5:
73 73 # nums_min = 5
74 74 #
75 75 # sump = 0.
76 76 #
77 77 # sumq = 0.
78 78 #
79 79 # j = 0
80 80 #
81 81 # cont = 1
82 82 #
83 83 # while((cont==1)and(j<lenOfData)):
84 84 #
85 85 # sump += sortdata[j]
86 86 #
87 87 # sumq += sortdata[j]**2
88 88 #
89 89 # if j > nums_min:
90 90 # rtest = float(j)/(j-1) + 1.0/navg
91 91 # if ((sumq*j) > (rtest*sump**2)):
92 92 # j = j - 1
93 93 # sump = sump - sortdata[j]
94 94 # sumq = sumq - sortdata[j]**2
95 95 # cont = 0
96 96 #
97 97 # j += 1
98 98 #
99 99 # lnoise = sump /j
100 100 #
101 101 # return lnoise
102 102
103 103 return cSchain.hildebrand_sekhon(sortdata, navg)
104 104
105 105
106 106 class Beam:
107 107
108 108 def __init__(self):
109 109 self.codeList = []
110 110 self.azimuthList = []
111 111 self.zenithList = []
112 112
113 113 class GenericData(object):
114 114
115 115 flagNoData = True
116 116
117 117 def copy(self, inputObj=None):
118 118
119 119 if inputObj == None:
120 120 return copy.deepcopy(self)
121 121
122 122 for key in inputObj.__dict__.keys():
123 123
124 124 attribute = inputObj.__dict__[key]
125 125
126 126 #If this attribute is a tuple or list
127 127 if type(inputObj.__dict__[key]) in (tuple, list):
128 128 self.__dict__[key] = attribute[:]
129 129 continue
130 130
131 131 #If this attribute is another object or instance
132 132 if hasattr(attribute, '__dict__'):
133 133 self.__dict__[key] = attribute.copy()
134 134 continue
135 135
136 136 self.__dict__[key] = inputObj.__dict__[key]
137 137
138 138 def deepcopy(self):
139 139
140 140 return copy.deepcopy(self)
141 141
142 142 def isEmpty(self):
143 143
144 144 return self.flagNoData
145 145
146 146 class JROData(GenericData):
147 147
148 148 # m_BasicHeader = BasicHeader()
149 149 # m_ProcessingHeader = ProcessingHeader()
150 150
151 151 systemHeaderObj = SystemHeader()
152 152
153 153 radarControllerHeaderObj = RadarControllerHeader()
154 154
155 155 # data = None
156 156
157 157 type = None
158 158
159 159 datatype = None #dtype but in string
160 160
161 161 # dtype = None
162 162
163 163 # nChannels = None
164 164
165 165 # nHeights = None
166 166
167 167 nProfiles = None
168 168
169 169 heightList = None
170 170
171 171 channelList = None
172 172
173 173 flagDiscontinuousBlock = False
174 174
175 175 useLocalTime = False
176 176
177 177 utctime = None
178 178
179 179 timeZone = None
180 180
181 181 dstFlag = None
182 182
183 183 errorCount = None
184 184
185 185 blocksize = None
186 186
187 187 # nCode = None
188 188 #
189 189 # nBaud = None
190 190 #
191 191 # code = None
192 192
193 193 flagDecodeData = False #asumo q la data no esta decodificada
194 194
195 195 flagDeflipData = False #asumo q la data no esta sin flip
196 196
197 197 flagShiftFFT = False
198 198
199 199 # ippSeconds = None
200 200
201 201 # timeInterval = None
202 202
203 203 nCohInt = None
204 204
205 205 # noise = None
206 206
207 207 windowOfFilter = 1
208 208
209 209 #Speed of ligth
210 210 C = 3e8
211 211
212 212 frequency = 49.92e6
213 213
214 214 realtime = False
215 215
216 216 beacon_heiIndexList = None
217 217
218 218 last_block = None
219 219
220 220 blocknow = None
221 221
222 222 azimuth = None
223 223
224 224 zenith = None
225 225
226 226 beam = Beam()
227 227
228 228 profileIndex = None
229 229
230 230 def getNoise(self):
231 231
232 232 raise NotImplementedError
233 233
234 234 def getNChannels(self):
235 235
236 236 return len(self.channelList)
237 237
238 238 def getChannelIndexList(self):
239 239
240 240 return range(self.nChannels)
241 241
242 242 def getNHeights(self):
243 243
244 244 return len(self.heightList)
245 245
246 246 def getHeiRange(self, extrapoints=0):
247 247
248 248 heis = self.heightList
249 249 # deltah = self.heightList[1] - self.heightList[0]
250 250 #
251 251 # heis.append(self.heightList[-1])
252 252
253 253 return heis
254 254
255 255 def getDeltaH(self):
256 256
257 257 delta = self.heightList[1] - self.heightList[0]
258 258
259 259 return delta
260 260
261 261 def getltctime(self):
262 262
263 263 if self.useLocalTime:
264 264 return self.utctime - self.timeZone*60
265 265
266 266 return self.utctime
267 267
268 268 def getDatatime(self):
269 269
270 270 datatimeValue = datetime.datetime.utcfromtimestamp(self.ltctime)
271 271 return datatimeValue
272 272
273 273 def getTimeRange(self):
274 274
275 275 datatime = []
276 276
277 277 datatime.append(self.ltctime)
278 278 datatime.append(self.ltctime + self.timeInterval+1)
279 279
280 280 datatime = numpy.array(datatime)
281 281
282 282 return datatime
283 283
284 284 def getFmaxTimeResponse(self):
285 285
286 286 period = (10**-6)*self.getDeltaH()/(0.15)
287 287
288 288 PRF = 1./(period * self.nCohInt)
289 289
290 290 fmax = PRF
291 291
292 292 return fmax
293 293
294 294 def getFmax(self):
295
296 295 PRF = 1./(self.ippSeconds * self.nCohInt)
297 296
298 297 fmax = PRF
299
300 298 return fmax
301 299
302 300 def getVmax(self):
303 301
304 302 _lambda = self.C/self.frequency
305 303
306 304 vmax = self.getFmax() * _lambda/2
307 305
308 306 return vmax
309 307
310 308 def get_ippSeconds(self):
311 309 '''
312 310 '''
313 311 return self.radarControllerHeaderObj.ippSeconds
314 312
315 313 def set_ippSeconds(self, ippSeconds):
316 314 '''
317 315 '''
318 316
319 317 self.radarControllerHeaderObj.ippSeconds = ippSeconds
320 318
321 319 return
322 320
323 321 def get_dtype(self):
324 322 '''
325 323 '''
326 324 return getNumpyDtype(self.datatype)
327 325
328 326 def set_dtype(self, numpyDtype):
329 327 '''
330 328 '''
331 329
332 330 self.datatype = getDataTypeCode(numpyDtype)
333 331
334 332 def get_code(self):
335 333 '''
336 334 '''
337 335 return self.radarControllerHeaderObj.code
338 336
339 337 def set_code(self, code):
340 338 '''
341 339 '''
342 340 self.radarControllerHeaderObj.code = code
343 341
344 342 return
345 343
346 344 def get_ncode(self):
347 345 '''
348 346 '''
349 347 return self.radarControllerHeaderObj.nCode
350 348
351 349 def set_ncode(self, nCode):
352 350 '''
353 351 '''
354 352 self.radarControllerHeaderObj.nCode = nCode
355 353
356 354 return
357 355
358 356 def get_nbaud(self):
359 357 '''
360 358 '''
361 359 return self.radarControllerHeaderObj.nBaud
362 360
363 361 def set_nbaud(self, nBaud):
364 362 '''
365 363 '''
366 364 self.radarControllerHeaderObj.nBaud = nBaud
367 365
368 366 return
369 367
370 368 nChannels = property(getNChannels, "I'm the 'nChannel' property.")
371 369 channelIndexList = property(getChannelIndexList, "I'm the 'channelIndexList' property.")
372 370 nHeights = property(getNHeights, "I'm the 'nHeights' property.")
373 371 #noise = property(getNoise, "I'm the 'nHeights' property.")
374 372 datatime = property(getDatatime, "I'm the 'datatime' property")
375 373 ltctime = property(getltctime, "I'm the 'ltctime' property")
376 374 ippSeconds = property(get_ippSeconds, set_ippSeconds)
377 375 dtype = property(get_dtype, set_dtype)
378 376 # timeInterval = property(getTimeInterval, "I'm the 'timeInterval' property")
379 377 code = property(get_code, set_code)
380 378 nCode = property(get_ncode, set_ncode)
381 379 nBaud = property(get_nbaud, set_nbaud)
382 380
383 381 class Voltage(JROData):
384 382
385 383 #data es un numpy array de 2 dmensiones (canales, alturas)
386 384 data = None
387 385
388 386 def __init__(self):
389 387 '''
390 388 Constructor
391 389 '''
392 390
393 391 self.useLocalTime = True
394 392
395 393 self.radarControllerHeaderObj = RadarControllerHeader()
396 394
397 395 self.systemHeaderObj = SystemHeader()
398 396
399 397 self.type = "Voltage"
400 398
401 399 self.data = None
402 400
403 401 # self.dtype = None
404 402
405 403 # self.nChannels = 0
406 404
407 405 # self.nHeights = 0
408 406
409 407 self.nProfiles = None
410 408
411 409 self.heightList = None
412 410
413 411 self.channelList = None
414 412
415 413 # self.channelIndexList = None
416 414
417 415 self.flagNoData = True
418 416
419 417 self.flagDiscontinuousBlock = False
420 418
421 419 self.utctime = None
422 420
423 421 self.timeZone = None
424 422
425 423 self.dstFlag = None
426 424
427 425 self.errorCount = None
428 426
429 427 self.nCohInt = None
430 428
431 429 self.blocksize = None
432 430
433 431 self.flagDecodeData = False #asumo q la data no esta decodificada
434 432
435 433 self.flagDeflipData = False #asumo q la data no esta sin flip
436 434
437 435 self.flagShiftFFT = False
438 436
439 437 self.flagDataAsBlock = False #Asumo que la data es leida perfil a perfil
440 438
441 439 self.profileIndex = 0
442 440
443 441 def getNoisebyHildebrand(self, channel = None):
444 442 """
445 443 Determino el nivel de ruido usando el metodo Hildebrand-Sekhon
446 444
447 445 Return:
448 446 noiselevel
449 447 """
450 448
451 449 if channel != None:
452 450 data = self.data[channel]
453 451 nChannels = 1
454 452 else:
455 453 data = self.data
456 454 nChannels = self.nChannels
457 455
458 456 noise = numpy.zeros(nChannels)
459 457 power = data * numpy.conjugate(data)
460 458
461 459 for thisChannel in range(nChannels):
462 460 if nChannels == 1:
463 461 daux = power[:].real
464 462 else:
465 463 daux = power[thisChannel,:].real
466 464 noise[thisChannel] = hildebrand_sekhon(daux, self.nCohInt)
467 465
468 466 return noise
469 467
470 468 def getNoise(self, type = 1, channel = None):
471 469
472 470 if type == 1:
473 471 noise = self.getNoisebyHildebrand(channel)
474 472
475 473 return noise
476 474
477 475 def getPower(self, channel = None):
478 476
479 477 if channel != None:
480 478 data = self.data[channel]
481 479 else:
482 480 data = self.data
483 481
484 482 power = data * numpy.conjugate(data)
485 483 powerdB = 10*numpy.log10(power.real)
486 484 powerdB = numpy.squeeze(powerdB)
487 485
488 486 return powerdB
489 487
490 488 def getTimeInterval(self):
491 489
492 490 timeInterval = self.ippSeconds * self.nCohInt
493 491
494 492 return timeInterval
495 493
496 494 noise = property(getNoise, "I'm the 'nHeights' property.")
497 495 timeInterval = property(getTimeInterval, "I'm the 'timeInterval' property")
498 496
499 497 class Spectra(JROData):
500 498
501 499 #data spc es un numpy array de 2 dmensiones (canales, perfiles, alturas)
502 500 data_spc = None
503 501
504 502 #data cspc es un numpy array de 2 dmensiones (canales, pares, alturas)
505 503 data_cspc = None
506 504
507 505 #data dc es un numpy array de 2 dmensiones (canales, alturas)
508 506 data_dc = None
509 507
510 508 #data power
511 509 data_pwr = None
512 510
513 511 nFFTPoints = None
514 512
515 513 # nPairs = None
516 514
517 515 pairsList = None
518 516
519 517 nIncohInt = None
520 518
521 519 wavelength = None #Necesario para cacular el rango de velocidad desde la frecuencia
522 520
523 521 nCohInt = None #se requiere para determinar el valor de timeInterval
524 522
525 523 ippFactor = None
526 524
527 525 profileIndex = 0
528 526
529 527 plotting = "spectra"
530 528
531 529 def __init__(self):
532 530 '''
533 531 Constructor
534 532 '''
535 533
536 534 self.useLocalTime = True
537 535
538 536 self.radarControllerHeaderObj = RadarControllerHeader()
539 537
540 538 self.systemHeaderObj = SystemHeader()
541 539
542 540 self.type = "Spectra"
543 541
544 542 # self.data = None
545 543
546 544 # self.dtype = None
547 545
548 546 # self.nChannels = 0
549 547
550 548 # self.nHeights = 0
551 549
552 550 self.nProfiles = None
553 551
554 552 self.heightList = None
555 553
556 554 self.channelList = None
557 555
558 556 # self.channelIndexList = None
559 557
560 558 self.pairsList = None
561 559
562 560 self.flagNoData = True
563 561
564 562 self.flagDiscontinuousBlock = False
565 563
566 564 self.utctime = None
567 565
568 566 self.nCohInt = None
569 567
570 568 self.nIncohInt = None
571 569
572 570 self.blocksize = None
573 571
574 572 self.nFFTPoints = None
575 573
576 574 self.wavelength = None
577 575
578 576 self.flagDecodeData = False #asumo q la data no esta decodificada
579 577
580 578 self.flagDeflipData = False #asumo q la data no esta sin flip
581 579
582 580 self.flagShiftFFT = False
583 581
584 582 self.ippFactor = 1
585 583
586 584 #self.noise = None
587 585
588 586 self.beacon_heiIndexList = []
589 587
590 588 self.noise_estimation = None
591 589
592 590
593 591 def getNoisebyHildebrand(self, xmin_index=None, xmax_index=None, ymin_index=None, ymax_index=None):
594 592 """
595 593 Determino el nivel de ruido usando el metodo Hildebrand-Sekhon
596 594
597 595 Return:
598 596 noiselevel
599 597 """
600 598
601 599 noise = numpy.zeros(self.nChannels)
602 600
603 601 for channel in range(self.nChannels):
604 602 daux = self.data_spc[channel,xmin_index:xmax_index,ymin_index:ymax_index]
605 603 noise[channel] = hildebrand_sekhon(daux, self.nIncohInt)
606 604
607 605 return noise
608 606
609 607 def getNoise(self, xmin_index=None, xmax_index=None, ymin_index=None, ymax_index=None):
610 608
611 609 if self.noise_estimation is not None:
612 610 return self.noise_estimation #this was estimated by getNoise Operation defined in jroproc_spectra.py
613 611 else:
614 612 noise = self.getNoisebyHildebrand(xmin_index, xmax_index, ymin_index, ymax_index)
615 613 return noise
616 614
617 615 def getFreqRangeTimeResponse(self, extrapoints=0):
618 616
619 617 deltafreq = self.getFmaxTimeResponse() / (self.nFFTPoints*self.ippFactor)
620 618 freqrange = deltafreq*(numpy.arange(self.nFFTPoints+extrapoints)-self.nFFTPoints/2.) - deltafreq/2
621 619
622 620 return freqrange
623 621
624 622 def getAcfRange(self, extrapoints=0):
625 623
626 624 deltafreq = 10./(self.getFmax() / (self.nFFTPoints*self.ippFactor))
627 625 freqrange = deltafreq*(numpy.arange(self.nFFTPoints+extrapoints)-self.nFFTPoints/2.) - deltafreq/2
628 626
629 627 return freqrange
630 628
631 629 def getFreqRange(self, extrapoints=0):
632 630
633 631 deltafreq = self.getFmax() / (self.nFFTPoints*self.ippFactor)
634 632 freqrange = deltafreq*(numpy.arange(self.nFFTPoints+extrapoints)-self.nFFTPoints/2.) - deltafreq/2
635 633
636 634 return freqrange
637 635
638 636 def getVelRange(self, extrapoints=0):
639 637
640 638 deltav = self.getVmax() / (self.nFFTPoints*self.ippFactor)
641 639 velrange = deltav*(numpy.arange(self.nFFTPoints+extrapoints)-self.nFFTPoints/2.) #- deltav/2
642 640
643 641 return velrange
644 642
645 643 def getNPairs(self):
646 644
647 645 return len(self.pairsList)
648 646
649 647 def getPairsIndexList(self):
650 648
651 649 return range(self.nPairs)
652 650
653 651 def getNormFactor(self):
654 652
655 653 pwcode = 1
656 654
657 655 if self.flagDecodeData:
658 656 pwcode = numpy.sum(self.code[0]**2)
659 657 #normFactor = min(self.nFFTPoints,self.nProfiles)*self.nIncohInt*self.nCohInt*pwcode*self.windowOfFilter
660 658 normFactor = self.nProfiles*self.nIncohInt*self.nCohInt*pwcode*self.windowOfFilter
661 659
662 660 return normFactor
663 661
664 662 def getFlagCspc(self):
665 663
666 664 if self.data_cspc is None:
667 665 return True
668 666
669 667 return False
670 668
671 669 def getFlagDc(self):
672 670
673 671 if self.data_dc is None:
674 672 return True
675 673
676 674 return False
677 675
678 676 def getTimeInterval(self):
679 677
680 678 timeInterval = self.ippSeconds * self.nCohInt * self.nIncohInt * self.nProfiles
681 679
682 680 return timeInterval
683 681
684 682 def getPower(self):
685 683
686 684 factor = self.normFactor
687 685 z = self.data_spc/factor
688 686 z = numpy.where(numpy.isfinite(z), z, numpy.NAN)
689 687 avg = numpy.average(z, axis=1)
690 688
691 689 return 10*numpy.log10(avg)
692 690
693 691 def getCoherence(self, pairsList=None, phase=False):
694 692
695 693 z = []
696 694 if pairsList is None:
697 695 pairsIndexList = self.pairsIndexList
698 696 else:
699 697 pairsIndexList = []
700 698 for pair in pairsList:
701 699 if pair not in self.pairsList:
702 700 raise ValueError, "Pair %s is not in dataOut.pairsList" %(pair)
703 701 pairsIndexList.append(self.pairsList.index(pair))
704 702 for i in range(len(pairsIndexList)):
705 703 pair = self.pairsList[pairsIndexList[i]]
706 704 ccf = numpy.average(self.data_cspc[pairsIndexList[i], :, :], axis=0)
707 705 powa = numpy.average(self.data_spc[pair[0], :, :], axis=0)
708 706 powb = numpy.average(self.data_spc[pair[1], :, :], axis=0)
709 707 avgcoherenceComplex = ccf/numpy.sqrt(powa*powb)
710 708 if phase:
711 709 data = numpy.arctan2(avgcoherenceComplex.imag,
712 710 avgcoherenceComplex.real)*180/numpy.pi
713 711 else:
714 712 data = numpy.abs(avgcoherenceComplex)
715 713
716 714 z.append(data)
717 715
718 716 return numpy.array(z)
719 717
720 718 def setValue(self, value):
721 719
722 720 print "This property should not be initialized"
723 721
724 722 return
725 723
726 724 nPairs = property(getNPairs, setValue, "I'm the 'nPairs' property.")
727 725 pairsIndexList = property(getPairsIndexList, setValue, "I'm the 'pairsIndexList' property.")
728 726 normFactor = property(getNormFactor, setValue, "I'm the 'getNormFactor' property.")
729 727 flag_cspc = property(getFlagCspc, setValue)
730 728 flag_dc = property(getFlagDc, setValue)
731 729 noise = property(getNoise, setValue, "I'm the 'nHeights' property.")
732 730 timeInterval = property(getTimeInterval, setValue, "I'm the 'timeInterval' property")
733 731
734 732 class SpectraHeis(Spectra):
735 733
736 734 data_spc = None
737 735
738 736 data_cspc = None
739 737
740 738 data_dc = None
741 739
742 740 nFFTPoints = None
743 741
744 742 # nPairs = None
745 743
746 744 pairsList = None
747 745
748 746 nCohInt = None
749 747
750 748 nIncohInt = None
751 749
752 750 def __init__(self):
753 751
754 752 self.radarControllerHeaderObj = RadarControllerHeader()
755 753
756 754 self.systemHeaderObj = SystemHeader()
757 755
758 756 self.type = "SpectraHeis"
759 757
760 758 # self.dtype = None
761 759
762 760 # self.nChannels = 0
763 761
764 762 # self.nHeights = 0
765 763
766 764 self.nProfiles = None
767 765
768 766 self.heightList = None
769 767
770 768 self.channelList = None
771 769
772 770 # self.channelIndexList = None
773 771
774 772 self.flagNoData = True
775 773
776 774 self.flagDiscontinuousBlock = False
777 775
778 776 # self.nPairs = 0
779 777
780 778 self.utctime = None
781 779
782 780 self.blocksize = None
783 781
784 782 self.profileIndex = 0
785 783
786 784 self.nCohInt = 1
787 785
788 786 self.nIncohInt = 1
789 787
790 788 def getNormFactor(self):
791 789 pwcode = 1
792 790 if self.flagDecodeData:
793 791 pwcode = numpy.sum(self.code[0]**2)
794 792
795 793 normFactor = self.nIncohInt*self.nCohInt*pwcode
796 794
797 795 return normFactor
798 796
799 797 def getTimeInterval(self):
800 798
801 799 timeInterval = self.ippSeconds * self.nCohInt * self.nIncohInt
802 800
803 801 return timeInterval
804 802
805 803 normFactor = property(getNormFactor, "I'm the 'getNormFactor' property.")
806 804 timeInterval = property(getTimeInterval, "I'm the 'timeInterval' property")
807 805
808 806 class Fits(JROData):
809 807
810 808 heightList = None
811 809
812 810 channelList = None
813 811
814 812 flagNoData = True
815 813
816 814 flagDiscontinuousBlock = False
817 815
818 816 useLocalTime = False
819 817
820 818 utctime = None
821 819
822 820 timeZone = None
823 821
824 822 # ippSeconds = None
825 823
826 824 # timeInterval = None
827 825
828 826 nCohInt = None
829 827
830 828 nIncohInt = None
831 829
832 830 noise = None
833 831
834 832 windowOfFilter = 1
835 833
836 834 #Speed of ligth
837 835 C = 3e8
838 836
839 837 frequency = 49.92e6
840 838
841 839 realtime = False
842 840
843 841
844 842 def __init__(self):
845 843
846 844 self.type = "Fits"
847 845
848 846 self.nProfiles = None
849 847
850 848 self.heightList = None
851 849
852 850 self.channelList = None
853 851
854 852 # self.channelIndexList = None
855 853
856 854 self.flagNoData = True
857 855
858 856 self.utctime = None
859 857
860 858 self.nCohInt = 1
861 859
862 860 self.nIncohInt = 1
863 861
864 862 self.useLocalTime = True
865 863
866 864 self.profileIndex = 0
867 865
868 866 # self.utctime = None
869 867 # self.timeZone = None
870 868 # self.ltctime = None
871 869 # self.timeInterval = None
872 870 # self.header = None
873 871 # self.data_header = None
874 872 # self.data = None
875 873 # self.datatime = None
876 874 # self.flagNoData = False
877 875 # self.expName = ''
878 876 # self.nChannels = None
879 877 # self.nSamples = None
880 878 # self.dataBlocksPerFile = None
881 879 # self.comments = ''
882 880 #
883 881
884 882
885 883 def getltctime(self):
886 884
887 885 if self.useLocalTime:
888 886 return self.utctime - self.timeZone*60
889 887
890 888 return self.utctime
891 889
892 890 def getDatatime(self):
893 891
894 892 datatime = datetime.datetime.utcfromtimestamp(self.ltctime)
895 893 return datatime
896 894
897 895 def getTimeRange(self):
898 896
899 897 datatime = []
900 898
901 899 datatime.append(self.ltctime)
902 900 datatime.append(self.ltctime + self.timeInterval)
903 901
904 902 datatime = numpy.array(datatime)
905 903
906 904 return datatime
907 905
908 906 def getHeiRange(self):
909 907
910 908 heis = self.heightList
911 909
912 910 return heis
913 911
914 912 def getNHeights(self):
915 913
916 914 return len(self.heightList)
917 915
918 916 def getNChannels(self):
919 917
920 918 return len(self.channelList)
921 919
922 920 def getChannelIndexList(self):
923 921
924 922 return range(self.nChannels)
925 923
926 924 def getNoise(self, type = 1):
927 925
928 926 #noise = numpy.zeros(self.nChannels)
929 927
930 928 if type == 1:
931 929 noise = self.getNoisebyHildebrand()
932 930
933 931 if type == 2:
934 932 noise = self.getNoisebySort()
935 933
936 934 if type == 3:
937 935 noise = self.getNoisebyWindow()
938 936
939 937 return noise
940 938
941 939 def getTimeInterval(self):
942 940
943 941 timeInterval = self.ippSeconds * self.nCohInt * self.nIncohInt
944 942
945 943 return timeInterval
946 944
947 945 datatime = property(getDatatime, "I'm the 'datatime' property")
948 946 nHeights = property(getNHeights, "I'm the 'nHeights' property.")
949 947 nChannels = property(getNChannels, "I'm the 'nChannel' property.")
950 948 channelIndexList = property(getChannelIndexList, "I'm the 'channelIndexList' property.")
951 949 noise = property(getNoise, "I'm the 'nHeights' property.")
952 950
953 951 ltctime = property(getltctime, "I'm the 'ltctime' property")
954 952 timeInterval = property(getTimeInterval, "I'm the 'timeInterval' property")
955 953
956 954
957 955 class Correlation(JROData):
958 956
959 957 noise = None
960 958
961 959 SNR = None
962 960
963 961 #--------------------------------------------------
964 962
965 963 mode = None
966 964
967 965 split = False
968 966
969 967 data_cf = None
970 968
971 969 lags = None
972 970
973 971 lagRange = None
974 972
975 973 pairsList = None
976 974
977 975 normFactor = None
978 976
979 977 #--------------------------------------------------
980 978
981 979 # calculateVelocity = None
982 980
983 981 nLags = None
984 982
985 983 nPairs = None
986 984
987 985 nAvg = None
988 986
989 987
990 988 def __init__(self):
991 989 '''
992 990 Constructor
993 991 '''
994 992 self.radarControllerHeaderObj = RadarControllerHeader()
995 993
996 994 self.systemHeaderObj = SystemHeader()
997 995
998 996 self.type = "Correlation"
999 997
1000 998 self.data = None
1001 999
1002 1000 self.dtype = None
1003 1001
1004 1002 self.nProfiles = None
1005 1003
1006 1004 self.heightList = None
1007 1005
1008 1006 self.channelList = None
1009 1007
1010 1008 self.flagNoData = True
1011 1009
1012 1010 self.flagDiscontinuousBlock = False
1013 1011
1014 1012 self.utctime = None
1015 1013
1016 1014 self.timeZone = None
1017 1015
1018 1016 self.dstFlag = None
1019 1017
1020 1018 self.errorCount = None
1021 1019
1022 1020 self.blocksize = None
1023 1021
1024 1022 self.flagDecodeData = False #asumo q la data no esta decodificada
1025 1023
1026 1024 self.flagDeflipData = False #asumo q la data no esta sin flip
1027 1025
1028 1026 self.pairsList = None
1029 1027
1030 1028 self.nPoints = None
1031 1029
1032 1030 def getPairsList(self):
1033 1031
1034 1032 return self.pairsList
1035 1033
1036 1034 def getNoise(self, mode = 2):
1037 1035
1038 1036 indR = numpy.where(self.lagR == 0)[0][0]
1039 1037 indT = numpy.where(self.lagT == 0)[0][0]
1040 1038
1041 1039 jspectra0 = self.data_corr[:,:,indR,:]
1042 1040 jspectra = copy.copy(jspectra0)
1043 1041
1044 1042 num_chan = jspectra.shape[0]
1045 1043 num_hei = jspectra.shape[2]
1046 1044
1047 1045 freq_dc = jspectra.shape[1]/2
1048 1046 ind_vel = numpy.array([-2,-1,1,2]) + freq_dc
1049 1047
1050 1048 if ind_vel[0]<0:
1051 1049 ind_vel[range(0,1)] = ind_vel[range(0,1)] + self.num_prof
1052 1050
1053 1051 if mode == 1:
1054 1052 jspectra[:,freq_dc,:] = (jspectra[:,ind_vel[1],:] + jspectra[:,ind_vel[2],:])/2 #CORRECCION
1055 1053
1056 1054 if mode == 2:
1057 1055
1058 1056 vel = numpy.array([-2,-1,1,2])
1059 1057 xx = numpy.zeros([4,4])
1060 1058
1061 1059 for fil in range(4):
1062 1060 xx[fil,:] = vel[fil]**numpy.asarray(range(4))
1063 1061
1064 1062 xx_inv = numpy.linalg.inv(xx)
1065 1063 xx_aux = xx_inv[0,:]
1066 1064
1067 1065 for ich in range(num_chan):
1068 1066 yy = jspectra[ich,ind_vel,:]
1069 1067 jspectra[ich,freq_dc,:] = numpy.dot(xx_aux,yy)
1070 1068
1071 1069 junkid = jspectra[ich,freq_dc,:]<=0
1072 1070 cjunkid = sum(junkid)
1073 1071
1074 1072 if cjunkid.any():
1075 1073 jspectra[ich,freq_dc,junkid.nonzero()] = (jspectra[ich,ind_vel[1],junkid] + jspectra[ich,ind_vel[2],junkid])/2
1076 1074
1077 1075 noise = jspectra0[:,freq_dc,:] - jspectra[:,freq_dc,:]
1078 1076
1079 1077 return noise
1080 1078
1081 1079 def getTimeInterval(self):
1082 1080
1083 1081 timeInterval = self.ippSeconds * self.nCohInt * self.nProfiles
1084 1082
1085 1083 return timeInterval
1086 1084
1087 1085 def splitFunctions(self):
1088 1086
1089 1087 pairsList = self.pairsList
1090 1088 ccf_pairs = []
1091 1089 acf_pairs = []
1092 1090 ccf_ind = []
1093 1091 acf_ind = []
1094 1092 for l in range(len(pairsList)):
1095 1093 chan0 = pairsList[l][0]
1096 1094 chan1 = pairsList[l][1]
1097 1095
1098 1096 #Obteniendo pares de Autocorrelacion
1099 1097 if chan0 == chan1:
1100 1098 acf_pairs.append(chan0)
1101 1099 acf_ind.append(l)
1102 1100 else:
1103 1101 ccf_pairs.append(pairsList[l])
1104 1102 ccf_ind.append(l)
1105 1103
1106 1104 data_acf = self.data_cf[acf_ind]
1107 1105 data_ccf = self.data_cf[ccf_ind]
1108 1106
1109 1107 return acf_ind, ccf_ind, acf_pairs, ccf_pairs, data_acf, data_ccf
1110 1108
1111 1109 def getNormFactor(self):
1112 1110 acf_ind, ccf_ind, acf_pairs, ccf_pairs, data_acf, data_ccf = self.splitFunctions()
1113 1111 acf_pairs = numpy.array(acf_pairs)
1114 1112 normFactor = numpy.zeros((self.nPairs,self.nHeights))
1115 1113
1116 1114 for p in range(self.nPairs):
1117 1115 pair = self.pairsList[p]
1118 1116
1119 1117 ch0 = pair[0]
1120 1118 ch1 = pair[1]
1121 1119
1122 1120 ch0_max = numpy.max(data_acf[acf_pairs==ch0,:,:], axis=1)
1123 1121 ch1_max = numpy.max(data_acf[acf_pairs==ch1,:,:], axis=1)
1124 1122 normFactor[p,:] = numpy.sqrt(ch0_max*ch1_max)
1125 1123
1126 1124 return normFactor
1127 1125
1128 1126 timeInterval = property(getTimeInterval, "I'm the 'timeInterval' property")
1129 1127 normFactor = property(getNormFactor, "I'm the 'normFactor property'")
1130 1128
1131 1129 class Parameters(Spectra):
1132 1130
1133 1131 experimentInfo = None #Information about the experiment
1134 1132
1135 1133 #Information from previous data
1136 1134
1137 1135 inputUnit = None #Type of data to be processed
1138 1136
1139 1137 operation = None #Type of operation to parametrize
1140 1138
1141 1139 #normFactor = None #Normalization Factor
1142 1140
1143 1141 groupList = None #List of Pairs, Groups, etc
1144 1142
1145 1143 #Parameters
1146 1144
1147 1145 data_param = None #Parameters obtained
1148 1146
1149 1147 data_pre = None #Data Pre Parametrization
1150 1148
1151 1149 data_SNR = None #Signal to Noise Ratio
1152 1150
1153 1151 # heightRange = None #Heights
1154 1152
1155 1153 abscissaList = None #Abscissa, can be velocities, lags or time
1156 1154
1157 1155 # noise = None #Noise Potency
1158 1156
1159 1157 utctimeInit = None #Initial UTC time
1160 1158
1161 1159 paramInterval = None #Time interval to calculate Parameters in seconds
1162 1160
1163 1161 useLocalTime = True
1164 1162
1165 1163 #Fitting
1166 1164
1167 1165 data_error = None #Error of the estimation
1168 1166
1169 1167 constants = None
1170 1168
1171 1169 library = None
1172 1170
1173 1171 #Output signal
1174 1172
1175 1173 outputInterval = None #Time interval to calculate output signal in seconds
1176 1174
1177 1175 data_output = None #Out signal
1178 1176
1179 1177 nAvg = None
1180 1178
1181 1179 noise_estimation = None
1182 1180
1183 1181
1184 1182 def __init__(self):
1185 1183 '''
1186 1184 Constructor
1187 1185 '''
1188 1186 self.radarControllerHeaderObj = RadarControllerHeader()
1189 1187
1190 1188 self.systemHeaderObj = SystemHeader()
1191 1189
1192 1190 self.type = "Parameters"
1193 1191
1194 1192 def getTimeRange1(self, interval):
1195 1193
1196 1194 datatime = []
1197 1195
1198 1196 if self.useLocalTime:
1199 1197 time1 = self.utctimeInit - self.timeZone*60
1200 1198 else:
1201 1199 time1 = self.utctimeInit
1202 1200
1203 1201 datatime.append(time1)
1204 1202 datatime.append(time1 + interval)
1205 1203 datatime = numpy.array(datatime)
1206 1204
1207 1205 return datatime
1208 1206
1209 1207 def getTimeInterval(self):
1210 1208
1211 1209 if hasattr(self, 'timeInterval1'):
1212 1210 return self.timeInterval1
1213 1211 else:
1214 1212 return self.paramInterval
1215 1213
1216 1214 def getNoise(self):
1217 1215
1218 1216 return self.spc_noise
1219 1217
1220 1218 timeInterval = property(getTimeInterval)
Show file before | Show file after | Hide comments
@@ -1,1534 +1,1534
1 1 '''
2 2 Created on Jul 9, 2014
3 3
4 4 @author: roj-idl71
5 5 '''
6 6 import os
7 7 import datetime
8 8 import numpy
9 9
10 10 from figure import Figure, isRealtime, isTimeInHourRange
11 11 from plotting_codes import *
12 12
13 13 class SpectraPlot(Figure):
14 14
15 15 isConfig = None
16 16 __nsubplots = None
17 17
18 18 WIDTHPROF = None
19 19 HEIGHTPROF = None
20 20 PREFIX = 'spc'
21 21
22 22 def __init__(self, **kwargs):
23 23 Figure.__init__(self, **kwargs)
24 24 self.isConfig = False
25 25 self.__nsubplots = 1
26 26
27 self.WIDTH = 250
28 self.HEIGHT = 250
27 self.WIDTH = 1000
28 self.HEIGHT = 1000
29 29 self.WIDTHPROF = 120
30 30 self.HEIGHTPROF = 0
31 31 self.counter_imagwr = 0
32 32
33 33 self.PLOT_CODE = SPEC_CODE
34 34
35 35 self.FTP_WEI = None
36 36 self.EXP_CODE = None
37 37 self.SUB_EXP_CODE = None
38 38 self.PLOT_POS = None
39 39
40 40 self.__xfilter_ena = False
41 41 self.__yfilter_ena = False
42 42
43 43 def getSubplots(self):
44 44
45 45 ncol = int(numpy.sqrt(self.nplots)+0.9)
46 46 nrow = int(self.nplots*1./ncol + 0.9)
47 47
48 48 return nrow, ncol
49 49
50 50 def setup(self, id, nplots, wintitle, showprofile=True, show=True):
51 51
52 52 self.__showprofile = showprofile
53 53 self.nplots = nplots
54 54
55 55 ncolspan = 1
56 56 colspan = 1
57 57 if showprofile:
58 58 ncolspan = 3
59 59 colspan = 2
60 60 self.__nsubplots = 2
61 61
62 62 self.createFigure(id = id,
63 63 wintitle = wintitle,
64 64 widthplot = self.WIDTH + self.WIDTHPROF,
65 65 heightplot = self.HEIGHT + self.HEIGHTPROF,
66 66 show=show)
67 67
68 68 nrow, ncol = self.getSubplots()
69 69
70 70 counter = 0
71 71 for y in range(nrow):
72 72 for x in range(ncol):
73 73
74 74 if counter >= self.nplots:
75 75 break
76 76
77 77 self.addAxes(nrow, ncol*ncolspan, y, x*ncolspan, colspan, 1)
78 78
79 79 if showprofile:
80 80 self.addAxes(nrow, ncol*ncolspan, y, x*ncolspan+colspan, 1, 1)
81 81
82 82 counter += 1
83 83
84 84 def run(self, dataOut, id, wintitle="", channelList=None, showprofile=True,
85 85 xmin=None, xmax=None, ymin=None, ymax=None, zmin=None, zmax=None,
86 86 save=False, figpath='./', figfile=None, show=True, ftp=False, wr_period=1,
87 87 server=None, folder=None, username=None, password=None,
88 88 ftp_wei=0, exp_code=0, sub_exp_code=0, plot_pos=0, realtime=False,
89 89 xaxis="velocity", **kwargs):
90 90
91 91 """
92 92
93 93 Input:
94 94 dataOut :
95 95 id :
96 96 wintitle :
97 97 channelList :
98 98 showProfile :
99 99 xmin : None,
100 100 xmax : None,
101 101 ymin : None,
102 102 ymax : None,
103 103 zmin : None,
104 104 zmax : None
105 105 """
106 106
107 107 colormap = kwargs.get('colormap','jet')
108 108
109 109 if realtime:
110 110 if not(isRealtime(utcdatatime = dataOut.utctime)):
111 111 print 'Skipping this plot function'
112 112 return
113 113
114 114 if channelList == None:
115 115 channelIndexList = dataOut.channelIndexList
116 116 else:
117 117 channelIndexList = []
118 118 for channel in channelList:
119 119 if channel not in dataOut.channelList:
120 120 raise ValueError, "Channel %d is not in dataOut.channelList" %channel
121 121 channelIndexList.append(dataOut.channelList.index(channel))
122 122
123 123 factor = dataOut.normFactor
124 124
125 125 if xaxis == "frequency":
126 126 x = dataOut.getFreqRange(1)/1000.
127 127 xlabel = "Frequency (kHz)"
128 128
129 129 elif xaxis == "time":
130 130 x = dataOut.getAcfRange(1)
131 131 xlabel = "Time (ms)"
132 132
133 133 else:
134 134 x = dataOut.getVelRange(1)
135 135 xlabel = "Velocity (m/s)"
136 136
137 137 ylabel = "Range (Km)"
138 138
139 139 y = dataOut.getHeiRange()
140 140
141 141 z = dataOut.data_spc/factor
142 142 z = numpy.where(numpy.isfinite(z), z, numpy.NAN)
143 143 zdB = 10*numpy.log10(z)
144 144
145 145 avg = numpy.average(z, axis=1)
146 146 avgdB = 10*numpy.log10(avg)
147 147
148 148 noise = dataOut.getNoise()/factor
149 149 noisedB = 10*numpy.log10(noise)
150 150
151 151 thisDatetime = datetime.datetime.utcfromtimestamp(dataOut.getTimeRange()[0])
152 152 title = wintitle + " Spectra"
153 153 if ((dataOut.azimuth!=None) and (dataOut.zenith!=None)):
154 154 title = title + '_' + 'azimuth,zenith=%2.2f,%2.2f'%(dataOut.azimuth, dataOut.zenith)
155 155
156 156 if not self.isConfig:
157 157
158 158 nplots = len(channelIndexList)
159 159
160 160 self.setup(id=id,
161 161 nplots=nplots,
162 162 wintitle=wintitle,
163 163 showprofile=showprofile,
164 164 show=show)
165 165
166 166 if xmin == None: xmin = numpy.nanmin(x)
167 167 if xmax == None: xmax = numpy.nanmax(x)
168 168 if ymin == None: ymin = numpy.nanmin(y)
169 169 if ymax == None: ymax = numpy.nanmax(y)
170 170 if zmin == None: zmin = numpy.floor(numpy.nanmin(noisedB)) - 3
171 171 if zmax == None: zmax = numpy.ceil(numpy.nanmax(avgdB)) + 3
172 172
173 173 self.FTP_WEI = ftp_wei
174 174 self.EXP_CODE = exp_code
175 175 self.SUB_EXP_CODE = sub_exp_code
176 176 self.PLOT_POS = plot_pos
177 177
178 178 self.isConfig = True
179 179
180 180 self.setWinTitle(title)
181 181
182 182 for i in range(self.nplots):
183 183 index = channelIndexList[i]
184 184 str_datetime = '%s %s'%(thisDatetime.strftime("%Y/%m/%d"),thisDatetime.strftime("%H:%M:%S"))
185 185 title = "Channel %d: %4.2fdB: %s" %(dataOut.channelList[index], noisedB[index], str_datetime)
186 186 if len(dataOut.beam.codeList) != 0:
187 187 title = "Ch%d:%4.2fdB,%2.2f,%2.2f:%s" %(dataOut.channelList[index], noisedB[index], dataOut.beam.azimuthList[index], dataOut.beam.zenithList[index], str_datetime)
188 188
189 189 axes = self.axesList[i*self.__nsubplots]
190 190 axes.pcolor(x, y, zdB[index,:,:],
191 191 xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax, zmin=zmin, zmax=zmax,
192 192 xlabel=xlabel, ylabel=ylabel, title=title, colormap=colormap,
193 193 ticksize=9, cblabel='')
194 194
195 195 if self.__showprofile:
196 196 axes = self.axesList[i*self.__nsubplots +1]
197 197 axes.pline(avgdB[index,:], y,
198 198 xmin=zmin, xmax=zmax, ymin=ymin, ymax=ymax,
199 199 xlabel='dB', ylabel='', title='',
200 200 ytick_visible=False,
201 201 grid='x')
202 202
203 203 noiseline = numpy.repeat(noisedB[index], len(y))
204 204 axes.addpline(noiseline, y, idline=1, color="black", linestyle="dashed", lw=2)
205 205
206 206 self.draw()
207 207
208 208 if figfile == None:
209 209 str_datetime = thisDatetime.strftime("%Y%m%d_%H%M%S")
210 210 name = str_datetime
211 211 if ((dataOut.azimuth!=None) and (dataOut.zenith!=None)):
212 212 name = name + '_az' + '_%2.2f'%(dataOut.azimuth) + '_zn' + '_%2.2f'%(dataOut.zenith)
213 213 figfile = self.getFilename(name)
214 214
215 215 self.save(figpath=figpath,
216 216 figfile=figfile,
217 217 save=save,
218 218 ftp=ftp,
219 219 wr_period=wr_period,
220 220 thisDatetime=thisDatetime)
221 221
222 222 class CrossSpectraPlot(Figure):
223 223
224 224 isConfig = None
225 225 __nsubplots = None
226 226
227 227 WIDTH = None
228 228 HEIGHT = None
229 229 WIDTHPROF = None
230 230 HEIGHTPROF = None
231 231 PREFIX = 'cspc'
232 232
233 233 def __init__(self, **kwargs):
234 234 Figure.__init__(self, **kwargs)
235 235 self.isConfig = False
236 236 self.__nsubplots = 4
237 237 self.counter_imagwr = 0
238 238 self.WIDTH = 250
239 239 self.HEIGHT = 250
240 240 self.WIDTHPROF = 0
241 241 self.HEIGHTPROF = 0
242 242
243 243 self.PLOT_CODE = CROSS_CODE
244 244 self.FTP_WEI = None
245 245 self.EXP_CODE = None
246 246 self.SUB_EXP_CODE = None
247 247 self.PLOT_POS = None
248 248
249 249 def getSubplots(self):
250 250
251 251 ncol = 4
252 252 nrow = self.nplots
253 253
254 254 return nrow, ncol
255 255
256 256 def setup(self, id, nplots, wintitle, showprofile=True, show=True):
257 257
258 258 self.__showprofile = showprofile
259 259 self.nplots = nplots
260 260
261 261 ncolspan = 1
262 262 colspan = 1
263 263
264 264 self.createFigure(id = id,
265 265 wintitle = wintitle,
266 266 widthplot = self.WIDTH + self.WIDTHPROF,
267 267 heightplot = self.HEIGHT + self.HEIGHTPROF,
268 268 show=True)
269 269
270 270 nrow, ncol = self.getSubplots()
271 271
272 272 counter = 0
273 273 for y in range(nrow):
274 274 for x in range(ncol):
275 275 self.addAxes(nrow, ncol*ncolspan, y, x*ncolspan, colspan, 1)
276 276
277 277 counter += 1
278 278
279 279 def run(self, dataOut, id, wintitle="", pairsList=None,
280 280 xmin=None, xmax=None, ymin=None, ymax=None, zmin=None, zmax=None,
281 281 coh_min=None, coh_max=None, phase_min=None, phase_max=None,
282 282 save=False, figpath='./', figfile=None, ftp=False, wr_period=1,
283 283 power_cmap='jet', coherence_cmap='jet', phase_cmap='RdBu_r', show=True,
284 284 server=None, folder=None, username=None, password=None,
285 285 ftp_wei=0, exp_code=0, sub_exp_code=0, plot_pos=0,
286 286 xaxis='frequency'):
287 287
288 288 """
289 289
290 290 Input:
291 291 dataOut :
292 292 id :
293 293 wintitle :
294 294 channelList :
295 295 showProfile :
296 296 xmin : None,
297 297 xmax : None,
298 298 ymin : None,
299 299 ymax : None,
300 300 zmin : None,
301 301 zmax : None
302 302 """
303 303
304 304 if pairsList == None:
305 305 pairsIndexList = dataOut.pairsIndexList
306 306 else:
307 307 pairsIndexList = []
308 308 for pair in pairsList:
309 309 if pair not in dataOut.pairsList:
310 310 raise ValueError, "Pair %s is not in dataOut.pairsList" %str(pair)
311 311 pairsIndexList.append(dataOut.pairsList.index(pair))
312 312
313 313 if not pairsIndexList:
314 314 return
315 315
316 316 if len(pairsIndexList) > 4:
317 317 pairsIndexList = pairsIndexList[0:4]
318 318
319 319 factor = dataOut.normFactor
320 320 x = dataOut.getVelRange(1)
321 321 y = dataOut.getHeiRange()
322 322 z = dataOut.data_spc[:,:,:]/factor
323 323 z = numpy.where(numpy.isfinite(z), z, numpy.NAN)
324 324
325 325 noise = dataOut.noise/factor
326 326
327 327 zdB = 10*numpy.log10(z)
328 328 noisedB = 10*numpy.log10(noise)
329 329
330 330 if coh_min == None:
331 331 coh_min = 0.0
332 332 if coh_max == None:
333 333 coh_max = 1.0
334 334
335 335 if phase_min == None:
336 336 phase_min = -180
337 337 if phase_max == None:
338 338 phase_max = 180
339 339
340 340 #thisDatetime = dataOut.datatime
341 341 thisDatetime = datetime.datetime.utcfromtimestamp(dataOut.getTimeRange()[0])
342 342 title = wintitle + " Cross-Spectra: %s" %(thisDatetime.strftime("%d-%b-%Y %H:%M:%S"))
343 343 # xlabel = "Velocity (m/s)"
344 344 ylabel = "Range (Km)"
345 345
346 346 if xaxis == "frequency":
347 347 x = dataOut.getFreqRange(1)/1000.
348 348 xlabel = "Frequency (kHz)"
349 349
350 350 elif xaxis == "time":
351 351 x = dataOut.getAcfRange(1)
352 352 xlabel = "Time (ms)"
353 353
354 354 else:
355 355 x = dataOut.getVelRange(1)
356 356 xlabel = "Velocity (m/s)"
357 357
358 358 if not self.isConfig:
359 359
360 360 nplots = len(pairsIndexList)
361 361
362 362 self.setup(id=id,
363 363 nplots=nplots,
364 364 wintitle=wintitle,
365 365 showprofile=False,
366 366 show=show)
367 367
368 368 avg = numpy.abs(numpy.average(z, axis=1))
369 369 avgdB = 10*numpy.log10(avg)
370 370
371 371 if xmin == None: xmin = numpy.nanmin(x)
372 372 if xmax == None: xmax = numpy.nanmax(x)
373 373 if ymin == None: ymin = numpy.nanmin(y)
374 374 if ymax == None: ymax = numpy.nanmax(y)
375 375 if zmin == None: zmin = numpy.floor(numpy.nanmin(noisedB)) - 3
376 376 if zmax == None: zmax = numpy.ceil(numpy.nanmax(avgdB)) + 3
377 377
378 378 self.FTP_WEI = ftp_wei
379 379 self.EXP_CODE = exp_code
380 380 self.SUB_EXP_CODE = sub_exp_code
381 381 self.PLOT_POS = plot_pos
382 382
383 383 self.isConfig = True
384 384
385 385 self.setWinTitle(title)
386 386
387 387 for i in range(self.nplots):
388 388 pair = dataOut.pairsList[pairsIndexList[i]]
389 389
390 390 chan_index0 = dataOut.channelList.index(pair[0])
391 391 chan_index1 = dataOut.channelList.index(pair[1])
392 392
393 393 str_datetime = '%s %s'%(thisDatetime.strftime("%Y/%m/%d"),thisDatetime.strftime("%H:%M:%S"))
394 394 title = "Ch%d: %4.2fdB: %s" %(pair[0], noisedB[chan_index0], str_datetime)
395 395 zdB = 10.*numpy.log10(dataOut.data_spc[chan_index0,:,:]/factor)
396 396 axes0 = self.axesList[i*self.__nsubplots]
397 397 axes0.pcolor(x, y, zdB,
398 398 xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax, zmin=zmin, zmax=zmax,
399 399 xlabel=xlabel, ylabel=ylabel, title=title,
400 400 ticksize=9, colormap=power_cmap, cblabel='')
401 401
402 402 title = "Ch%d: %4.2fdB: %s" %(pair[1], noisedB[chan_index1], str_datetime)
403 403 zdB = 10.*numpy.log10(dataOut.data_spc[chan_index1,:,:]/factor)
404 404 axes0 = self.axesList[i*self.__nsubplots+1]
405 405 axes0.pcolor(x, y, zdB,
406 406 xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax, zmin=zmin, zmax=zmax,
407 407 xlabel=xlabel, ylabel=ylabel, title=title,
408 408 ticksize=9, colormap=power_cmap, cblabel='')
409 409
410 410 coherenceComplex = dataOut.data_cspc[pairsIndexList[i],:,:]/numpy.sqrt(dataOut.data_spc[chan_index0,:,:]*dataOut.data_spc[chan_index1,:,:])
411 411 coherence = numpy.abs(coherenceComplex)
412 412 # phase = numpy.arctan(-1*coherenceComplex.imag/coherenceComplex.real)*180/numpy.pi
413 413 phase = numpy.arctan2(coherenceComplex.imag, coherenceComplex.real)*180/numpy.pi
414 414
415 415 title = "Coherence Ch%d * Ch%d" %(pair[0], pair[1])
416 416 axes0 = self.axesList[i*self.__nsubplots+2]
417 417 axes0.pcolor(x, y, coherence,
418 418 xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax, zmin=coh_min, zmax=coh_max,
419 419 xlabel=xlabel, ylabel=ylabel, title=title,
420 420 ticksize=9, colormap=coherence_cmap, cblabel='')
421 421
422 422 title = "Phase Ch%d * Ch%d" %(pair[0], pair[1])
423 423 axes0 = self.axesList[i*self.__nsubplots+3]
424 424 axes0.pcolor(x, y, phase,
425 425 xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax, zmin=phase_min, zmax=phase_max,
426 426 xlabel=xlabel, ylabel=ylabel, title=title,
427 427 ticksize=9, colormap=phase_cmap, cblabel='')
428 428
429 429
430 430
431 431 self.draw()
432 432
433 433 self.save(figpath=figpath,
434 434 figfile=figfile,
435 435 save=save,
436 436 ftp=ftp,
437 437 wr_period=wr_period,
438 438 thisDatetime=thisDatetime)
439 439
440 440
441 441 class RTIPlot(Figure):
442 442
443 443 __isConfig = None
444 444 __nsubplots = None
445 445
446 446 WIDTHPROF = None
447 447 HEIGHTPROF = None
448 448 PREFIX = 'rti'
449 449
450 450 def __init__(self, **kwargs):
451 451
452 452 Figure.__init__(self, **kwargs)
453 453 self.timerange = None
454 454 self.isConfig = False
455 455 self.__nsubplots = 1
456 456
457 457 self.WIDTH = 800
458 458 self.HEIGHT = 180
459 459 self.WIDTHPROF = 120
460 460 self.HEIGHTPROF = 0
461 461 self.counter_imagwr = 0
462 462
463 463 self.PLOT_CODE = RTI_CODE
464 464
465 465 self.FTP_WEI = None
466 466 self.EXP_CODE = None
467 467 self.SUB_EXP_CODE = None
468 468 self.PLOT_POS = None
469 469 self.tmin = None
470 470 self.tmax = None
471 471
472 472 self.xmin = None
473 473 self.xmax = None
474 474
475 475 self.figfile = None
476 476
477 477 def getSubplots(self):
478 478
479 479 ncol = 1
480 480 nrow = self.nplots
481 481
482 482 return nrow, ncol
483 483
484 484 def setup(self, id, nplots, wintitle, showprofile=True, show=True):
485 485
486 486 self.__showprofile = showprofile
487 487 self.nplots = nplots
488 488
489 489 ncolspan = 1
490 490 colspan = 1
491 491 if showprofile:
492 492 ncolspan = 7
493 493 colspan = 6
494 494 self.__nsubplots = 2
495 495
496 496 self.createFigure(id = id,
497 497 wintitle = wintitle,
498 498 widthplot = self.WIDTH + self.WIDTHPROF,
499 499 heightplot = self.HEIGHT + self.HEIGHTPROF,
500 500 show=show)
501 501
502 502 nrow, ncol = self.getSubplots()
503 503
504 504 counter = 0
505 505 for y in range(nrow):
506 506 for x in range(ncol):
507 507
508 508 if counter >= self.nplots:
509 509 break
510 510
511 511 self.addAxes(nrow, ncol*ncolspan, y, x*ncolspan, colspan, 1)
512 512
513 513 if showprofile:
514 514 self.addAxes(nrow, ncol*ncolspan, y, x*ncolspan+colspan, 1, 1)
515 515
516 516 counter += 1
517 517
518 518 def run(self, dataOut, id, wintitle="", channelList=None, showprofile='True',
519 519 xmin=None, xmax=None, ymin=None, ymax=None, zmin=None, zmax=None,
520 520 timerange=None,
521 521 save=False, figpath='./', lastone=0,figfile=None, ftp=False, wr_period=1, show=True,
522 522 server=None, folder=None, username=None, password=None,
523 523 ftp_wei=0, exp_code=0, sub_exp_code=0, plot_pos=0, **kwargs):
524 524
525 525 """
526 526
527 527 Input:
528 528 dataOut :
529 529 id :
530 530 wintitle :
531 531 channelList :
532 532 showProfile :
533 533 xmin : None,
534 534 xmax : None,
535 535 ymin : None,
536 536 ymax : None,
537 537 zmin : None,
538 538 zmax : None
539 539 """
540 540
541 541 colormap = kwargs.get('colormap', 'jet')
542 542 if not isTimeInHourRange(dataOut.datatime, xmin, xmax):
543 543 return
544 544
545 545 if channelList == None:
546 546 channelIndexList = dataOut.channelIndexList
547 547 else:
548 548 channelIndexList = []
549 549 for channel in channelList:
550 550 if channel not in dataOut.channelList:
551 551 raise ValueError, "Channel %d is not in dataOut.channelList"
552 552 channelIndexList.append(dataOut.channelList.index(channel))
553 553
554 554 if hasattr(dataOut, 'normFactor'):
555 555 factor = dataOut.normFactor
556 556 else:
557 557 factor = 1
558 558
559 559 # factor = dataOut.normFactor
560 560 x = dataOut.getTimeRange()
561 561 y = dataOut.getHeiRange()
562 562
563 563 # z = dataOut.data_spc/factor
564 564 # z = numpy.where(numpy.isfinite(z), z, numpy.NAN)
565 565 # avg = numpy.average(z, axis=1)
566 566 # avgdB = 10.*numpy.log10(avg)
567 567 avgdB = dataOut.getPower()
568 568
569 569 thisDatetime = dataOut.datatime
570 570 # thisDatetime = datetime.datetime.utcfromtimestamp(dataOut.getTimeRange()[0])
571 571 title = wintitle + " RTI" #: %s" %(thisDatetime.strftime("%d-%b-%Y"))
572 572 xlabel = ""
573 573 ylabel = "Range (Km)"
574 574
575 575 update_figfile = False
576 576
577 577 if dataOut.ltctime >= self.xmax:
578 578 self.counter_imagwr = wr_period
579 579 self.isConfig = False
580 580 update_figfile = True
581 581
582 582 if not self.isConfig:
583 583
584 584 nplots = len(channelIndexList)
585 585
586 586 self.setup(id=id,
587 587 nplots=nplots,
588 588 wintitle=wintitle,
589 589 showprofile=showprofile,
590 590 show=show)
591 591
592 592 if timerange != None:
593 593 self.timerange = timerange
594 594
595 595 self.xmin, self.xmax = self.getTimeLim(x, xmin, xmax, timerange)
596 596
597 597 noise = dataOut.noise/factor
598 598 noisedB = 10*numpy.log10(noise)
599 599
600 600 if ymin == None: ymin = numpy.nanmin(y)
601 601 if ymax == None: ymax = numpy.nanmax(y)
602 602 if zmin == None: zmin = numpy.floor(numpy.nanmin(noisedB)) - 3
603 603 if zmax == None: zmax = numpy.ceil(numpy.nanmax(avgdB)) + 3
604 604
605 605 self.FTP_WEI = ftp_wei
606 606 self.EXP_CODE = exp_code
607 607 self.SUB_EXP_CODE = sub_exp_code
608 608 self.PLOT_POS = plot_pos
609 609
610 610 self.name = thisDatetime.strftime("%Y%m%d_%H%M%S")
611 611 self.isConfig = True
612 612 self.figfile = figfile
613 613 update_figfile = True
614 614
615 615 self.setWinTitle(title)
616 616
617 617 for i in range(self.nplots):
618 618 index = channelIndexList[i]
619 619 title = "Channel %d: %s" %(dataOut.channelList[index], thisDatetime.strftime("%Y/%m/%d %H:%M:%S"))
620 620 if ((dataOut.azimuth!=None) and (dataOut.zenith!=None)):
621 621 title = title + '_' + 'azimuth,zenith=%2.2f,%2.2f'%(dataOut.azimuth, dataOut.zenith)
622 622 axes = self.axesList[i*self.__nsubplots]
623 623 zdB = avgdB[index].reshape((1,-1))
624 624 axes.pcolorbuffer(x, y, zdB,
625 625 xmin=self.xmin, xmax=self.xmax, ymin=ymin, ymax=ymax, zmin=zmin, zmax=zmax,
626 626 xlabel=xlabel, ylabel=ylabel, title=title, rti=True, XAxisAsTime=True,
627 627 ticksize=9, cblabel='', cbsize="1%", colormap=colormap)
628 628
629 629 if self.__showprofile:
630 630 axes = self.axesList[i*self.__nsubplots +1]
631 631 axes.pline(avgdB[index], y,
632 632 xmin=zmin, xmax=zmax, ymin=ymin, ymax=ymax,
633 633 xlabel='dB', ylabel='', title='',
634 634 ytick_visible=False,
635 635 grid='x')
636 636
637 637 self.draw()
638 638
639 639 self.save(figpath=figpath,
640 640 figfile=figfile,
641 641 save=save,
642 642 ftp=ftp,
643 643 wr_period=wr_period,
644 644 thisDatetime=thisDatetime,
645 645 update_figfile=update_figfile)
646 646
647 647 class CoherenceMap(Figure):
648 648 isConfig = None
649 649 __nsubplots = None
650 650
651 651 WIDTHPROF = None
652 652 HEIGHTPROF = None
653 653 PREFIX = 'cmap'
654 654
655 655 def __init__(self, **kwargs):
656 656 Figure.__init__(self, **kwargs)
657 657 self.timerange = 2*60*60
658 658 self.isConfig = False
659 659 self.__nsubplots = 1
660 660
661 661 self.WIDTH = 800
662 662 self.HEIGHT = 180
663 663 self.WIDTHPROF = 120
664 664 self.HEIGHTPROF = 0
665 665 self.counter_imagwr = 0
666 666
667 667 self.PLOT_CODE = COH_CODE
668 668
669 669 self.FTP_WEI = None
670 670 self.EXP_CODE = None
671 671 self.SUB_EXP_CODE = None
672 672 self.PLOT_POS = None
673 673 self.counter_imagwr = 0
674 674
675 675 self.xmin = None
676 676 self.xmax = None
677 677
678 678 def getSubplots(self):
679 679 ncol = 1
680 680 nrow = self.nplots*2
681 681
682 682 return nrow, ncol
683 683
684 684 def setup(self, id, nplots, wintitle, showprofile=True, show=True):
685 685 self.__showprofile = showprofile
686 686 self.nplots = nplots
687 687
688 688 ncolspan = 1
689 689 colspan = 1
690 690 if showprofile:
691 691 ncolspan = 7
692 692 colspan = 6
693 693 self.__nsubplots = 2
694 694
695 695 self.createFigure(id = id,
696 696 wintitle = wintitle,
697 697 widthplot = self.WIDTH + self.WIDTHPROF,
698 698 heightplot = self.HEIGHT + self.HEIGHTPROF,
699 699 show=True)
700 700
701 701 nrow, ncol = self.getSubplots()
702 702
703 703 for y in range(nrow):
704 704 for x in range(ncol):
705 705
706 706 self.addAxes(nrow, ncol*ncolspan, y, x*ncolspan, colspan, 1)
707 707
708 708 if showprofile:
709 709 self.addAxes(nrow, ncol*ncolspan, y, x*ncolspan+colspan, 1, 1)
710 710
711 711 def run(self, dataOut, id, wintitle="", pairsList=None, showprofile='True',
712 712 xmin=None, xmax=None, ymin=None, ymax=None, zmin=None, zmax=None,
713 713 timerange=None, phase_min=None, phase_max=None,
714 714 save=False, figpath='./', figfile=None, ftp=False, wr_period=1,
715 715 coherence_cmap='jet', phase_cmap='RdBu_r', show=True,
716 716 server=None, folder=None, username=None, password=None,
717 717 ftp_wei=0, exp_code=0, sub_exp_code=0, plot_pos=0):
718 718
719 719 if not isTimeInHourRange(dataOut.datatime, xmin, xmax):
720 720 return
721 721
722 722 if pairsList == None:
723 723 pairsIndexList = dataOut.pairsIndexList
724 724 else:
725 725 pairsIndexList = []
726 726 for pair in pairsList:
727 727 if pair not in dataOut.pairsList:
728 728 raise ValueError, "Pair %s is not in dataOut.pairsList" %(pair)
729 729 pairsIndexList.append(dataOut.pairsList.index(pair))
730 730
731 731 if pairsIndexList == []:
732 732 return
733 733
734 734 if len(pairsIndexList) > 4:
735 735 pairsIndexList = pairsIndexList[0:4]
736 736
737 737 if phase_min == None:
738 738 phase_min = -180
739 739 if phase_max == None:
740 740 phase_max = 180
741 741
742 742 x = dataOut.getTimeRange()
743 743 y = dataOut.getHeiRange()
744 744
745 745 thisDatetime = dataOut.datatime
746 746
747 747 title = wintitle + " CoherenceMap" #: %s" %(thisDatetime.strftime("%d-%b-%Y"))
748 748 xlabel = ""
749 749 ylabel = "Range (Km)"
750 750 update_figfile = False
751 751
752 752 if not self.isConfig:
753 753 nplots = len(pairsIndexList)
754 754 self.setup(id=id,
755 755 nplots=nplots,
756 756 wintitle=wintitle,
757 757 showprofile=showprofile,
758 758 show=show)
759 759
760 760 if timerange != None:
761 761 self.timerange = timerange
762 762
763 763 self.xmin, self.xmax = self.getTimeLim(x, xmin, xmax, timerange)
764 764
765 765 if ymin == None: ymin = numpy.nanmin(y)
766 766 if ymax == None: ymax = numpy.nanmax(y)
767 767 if zmin == None: zmin = 0.
768 768 if zmax == None: zmax = 1.
769 769
770 770 self.FTP_WEI = ftp_wei
771 771 self.EXP_CODE = exp_code
772 772 self.SUB_EXP_CODE = sub_exp_code
773 773 self.PLOT_POS = plot_pos
774 774
775 775 self.name = thisDatetime.strftime("%Y%m%d_%H%M%S")
776 776
777 777 self.isConfig = True
778 778 update_figfile = True
779 779
780 780 self.setWinTitle(title)
781 781
782 782 for i in range(self.nplots):
783 783
784 784 pair = dataOut.pairsList[pairsIndexList[i]]
785 785
786 786 ccf = numpy.average(dataOut.data_cspc[pairsIndexList[i],:,:],axis=0)
787 787 powa = numpy.average(dataOut.data_spc[pair[0],:,:],axis=0)
788 788 powb = numpy.average(dataOut.data_spc[pair[1],:,:],axis=0)
789 789
790 790
791 791 avgcoherenceComplex = ccf/numpy.sqrt(powa*powb)
792 792 coherence = numpy.abs(avgcoherenceComplex)
793 793
794 794 z = coherence.reshape((1,-1))
795 795
796 796 counter = 0
797 797
798 798 title = "Coherence Ch%d * Ch%d: %s" %(pair[0], pair[1], thisDatetime.strftime("%d-%b-%Y %H:%M:%S"))
799 799 axes = self.axesList[i*self.__nsubplots*2]
800 800 axes.pcolorbuffer(x, y, z,
801 801 xmin=self.xmin, xmax=self.xmax, ymin=ymin, ymax=ymax, zmin=zmin, zmax=zmax,
802 802 xlabel=xlabel, ylabel=ylabel, title=title, rti=True, XAxisAsTime=True,
803 803 ticksize=9, cblabel='', colormap=coherence_cmap, cbsize="1%")
804 804
805 805 if self.__showprofile:
806 806 counter += 1
807 807 axes = self.axesList[i*self.__nsubplots*2 + counter]
808 808 axes.pline(coherence, y,
809 809 xmin=zmin, xmax=zmax, ymin=ymin, ymax=ymax,
810 810 xlabel='', ylabel='', title='', ticksize=7,
811 811 ytick_visible=False, nxticks=5,
812 812 grid='x')
813 813
814 814 counter += 1
815 815
816 816 phase = numpy.arctan2(avgcoherenceComplex.imag, avgcoherenceComplex.real)*180/numpy.pi
817 817
818 818 z = phase.reshape((1,-1))
819 819
820 820 title = "Phase Ch%d * Ch%d: %s" %(pair[0], pair[1], thisDatetime.strftime("%d-%b-%Y %H:%M:%S"))
821 821 axes = self.axesList[i*self.__nsubplots*2 + counter]
822 822 axes.pcolorbuffer(x, y, z,
823 823 xmin=self.xmin, xmax=self.xmax, ymin=ymin, ymax=ymax, zmin=phase_min, zmax=phase_max,
824 824 xlabel=xlabel, ylabel=ylabel, title=title, rti=True, XAxisAsTime=True,
825 825 ticksize=9, cblabel='', colormap=phase_cmap, cbsize="1%")
826 826
827 827 if self.__showprofile:
828 828 counter += 1
829 829 axes = self.axesList[i*self.__nsubplots*2 + counter]
830 830 axes.pline(phase, y,
831 831 xmin=phase_min, xmax=phase_max, ymin=ymin, ymax=ymax,
832 832 xlabel='', ylabel='', title='', ticksize=7,
833 833 ytick_visible=False, nxticks=4,
834 834 grid='x')
835 835
836 836 self.draw()
837 837
838 838 if dataOut.ltctime >= self.xmax:
839 839 self.counter_imagwr = wr_period
840 840 self.isConfig = False
841 841 update_figfile = True
842 842
843 843 self.save(figpath=figpath,
844 844 figfile=figfile,
845 845 save=save,
846 846 ftp=ftp,
847 847 wr_period=wr_period,
848 848 thisDatetime=thisDatetime,
849 849 update_figfile=update_figfile)
850 850
851 851 class PowerProfilePlot(Figure):
852 852
853 853 isConfig = None
854 854 __nsubplots = None
855 855
856 856 WIDTHPROF = None
857 857 HEIGHTPROF = None
858 858 PREFIX = 'spcprofile'
859 859
860 860 def __init__(self, **kwargs):
861 861 Figure.__init__(self, **kwargs)
862 862 self.isConfig = False
863 863 self.__nsubplots = 1
864 864
865 865 self.PLOT_CODE = POWER_CODE
866 866
867 867 self.WIDTH = 300
868 868 self.HEIGHT = 500
869 869 self.counter_imagwr = 0
870 870
871 871 def getSubplots(self):
872 872 ncol = 1
873 873 nrow = 1
874 874
875 875 return nrow, ncol
876 876
877 877 def setup(self, id, nplots, wintitle, show):
878 878
879 879 self.nplots = nplots
880 880
881 881 ncolspan = 1
882 882 colspan = 1
883 883
884 884 self.createFigure(id = id,
885 885 wintitle = wintitle,
886 886 widthplot = self.WIDTH,
887 887 heightplot = self.HEIGHT,
888 888 show=show)
889 889
890 890 nrow, ncol = self.getSubplots()
891 891
892 892 counter = 0
893 893 for y in range(nrow):
894 894 for x in range(ncol):
895 895 self.addAxes(nrow, ncol*ncolspan, y, x*ncolspan, colspan, 1)
896 896
897 897 def run(self, dataOut, id, wintitle="", channelList=None,
898 898 xmin=None, xmax=None, ymin=None, ymax=None,
899 899 save=False, figpath='./', figfile=None, show=True,
900 900 ftp=False, wr_period=1, server=None,
901 901 folder=None, username=None, password=None):
902 902
903 903
904 904 if channelList == None:
905 905 channelIndexList = dataOut.channelIndexList
906 906 channelList = dataOut.channelList
907 907 else:
908 908 channelIndexList = []
909 909 for channel in channelList:
910 910 if channel not in dataOut.channelList:
911 911 raise ValueError, "Channel %d is not in dataOut.channelList"
912 912 channelIndexList.append(dataOut.channelList.index(channel))
913 913
914 914 factor = dataOut.normFactor
915 915
916 916 y = dataOut.getHeiRange()
917 917
918 918 #for voltage
919 919 if dataOut.type == 'Voltage':
920 920 x = dataOut.data[channelIndexList,:] * numpy.conjugate(dataOut.data[channelIndexList,:])
921 921 x = x.real
922 922 x = numpy.where(numpy.isfinite(x), x, numpy.NAN)
923 923
924 924 #for spectra
925 925 if dataOut.type == 'Spectra':
926 926 x = dataOut.data_spc[channelIndexList,:,:]/factor
927 927 x = numpy.where(numpy.isfinite(x), x, numpy.NAN)
928 928 x = numpy.average(x, axis=1)
929 929
930 930
931 931 xdB = 10*numpy.log10(x)
932 932
933 933 thisDatetime = datetime.datetime.utcfromtimestamp(dataOut.getTimeRange()[0])
934 934 title = wintitle + " Power Profile %s" %(thisDatetime.strftime("%d-%b-%Y"))
935 935 xlabel = "dB"
936 936 ylabel = "Range (Km)"
937 937
938 938 if not self.isConfig:
939 939
940 940 nplots = 1
941 941
942 942 self.setup(id=id,
943 943 nplots=nplots,
944 944 wintitle=wintitle,
945 945 show=show)
946 946
947 947 if ymin == None: ymin = numpy.nanmin(y)
948 948 if ymax == None: ymax = numpy.nanmax(y)
949 949 if xmin == None: xmin = numpy.nanmin(xdB)*0.9
950 950 if xmax == None: xmax = numpy.nanmax(xdB)*1.1
951 951
952 952 self.isConfig = True
953 953
954 954 self.setWinTitle(title)
955 955
956 956 title = "Power Profile: %s" %(thisDatetime.strftime("%d-%b-%Y %H:%M:%S"))
957 957 axes = self.axesList[0]
958 958
959 959 legendlabels = ["channel %d"%x for x in channelList]
960 960 axes.pmultiline(xdB, y,
961 961 xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax,
962 962 xlabel=xlabel, ylabel=ylabel, title=title, legendlabels=legendlabels,
963 963 ytick_visible=True, nxticks=5,
964 964 grid='x')
965 965
966 966 self.draw()
967 967
968 968 self.save(figpath=figpath,
969 969 figfile=figfile,
970 970 save=save,
971 971 ftp=ftp,
972 972 wr_period=wr_period,
973 973 thisDatetime=thisDatetime)
974 974
975 975 class SpectraCutPlot(Figure):
976 976
977 977 isConfig = None
978 978 __nsubplots = None
979 979
980 980 WIDTHPROF = None
981 981 HEIGHTPROF = None
982 982 PREFIX = 'spc_cut'
983 983
984 984 def __init__(self, **kwargs):
985 985 Figure.__init__(self, **kwargs)
986 986 self.isConfig = False
987 987 self.__nsubplots = 1
988 988
989 989 self.PLOT_CODE = POWER_CODE
990 990
991 991 self.WIDTH = 700
992 992 self.HEIGHT = 500
993 993 self.counter_imagwr = 0
994 994
995 995 def getSubplots(self):
996 996 ncol = 1
997 997 nrow = 1
998 998
999 999 return nrow, ncol
1000 1000
1001 1001 def setup(self, id, nplots, wintitle, show):
1002 1002
1003 1003 self.nplots = nplots
1004 1004
1005 1005 ncolspan = 1
1006 1006 colspan = 1
1007 1007
1008 1008 self.createFigure(id = id,
1009 1009 wintitle = wintitle,
1010 1010 widthplot = self.WIDTH,
1011 1011 heightplot = self.HEIGHT,
1012 1012 show=show)
1013 1013
1014 1014 nrow, ncol = self.getSubplots()
1015 1015
1016 1016 counter = 0
1017 1017 for y in range(nrow):
1018 1018 for x in range(ncol):
1019 1019 self.addAxes(nrow, ncol*ncolspan, y, x*ncolspan, colspan, 1)
1020 1020
1021 1021 def run(self, dataOut, id, wintitle="", channelList=None,
1022 1022 xmin=None, xmax=None, ymin=None, ymax=None,
1023 1023 save=False, figpath='./', figfile=None, show=True,
1024 1024 ftp=False, wr_period=1, server=None,
1025 1025 folder=None, username=None, password=None,
1026 1026 xaxis="frequency"):
1027 1027
1028 1028
1029 1029 if channelList == None:
1030 1030 channelIndexList = dataOut.channelIndexList
1031 1031 channelList = dataOut.channelList
1032 1032 else:
1033 1033 channelIndexList = []
1034 1034 for channel in channelList:
1035 1035 if channel not in dataOut.channelList:
1036 1036 raise ValueError, "Channel %d is not in dataOut.channelList"
1037 1037 channelIndexList.append(dataOut.channelList.index(channel))
1038 1038
1039 1039 factor = dataOut.normFactor
1040 1040
1041 1041 y = dataOut.getHeiRange()
1042 1042
1043 1043 z = dataOut.data_spc/factor
1044 1044 z = numpy.where(numpy.isfinite(z), z, numpy.NAN)
1045 1045
1046 1046 hei_index = numpy.arange(25)*3 + 20
1047 1047
1048 1048 if xaxis == "frequency":
1049 1049 x = dataOut.getFreqRange()/1000.
1050 1050 zdB = 10*numpy.log10(z[0,:,hei_index])
1051 1051 xlabel = "Frequency (kHz)"
1052 1052 ylabel = "Power (dB)"
1053 1053
1054 1054 elif xaxis == "time":
1055 1055 x = dataOut.getAcfRange()
1056 1056 zdB = z[0,:,hei_index]
1057 1057 xlabel = "Time (ms)"
1058 1058 ylabel = "ACF"
1059 1059
1060 1060 else:
1061 1061 x = dataOut.getVelRange()
1062 1062 zdB = 10*numpy.log10(z[0,:,hei_index])
1063 1063 xlabel = "Velocity (m/s)"
1064 1064 ylabel = "Power (dB)"
1065 1065
1066 1066 thisDatetime = datetime.datetime.utcfromtimestamp(dataOut.getTimeRange()[0])
1067 1067 title = wintitle + " Range Cuts %s" %(thisDatetime.strftime("%d-%b-%Y"))
1068 1068
1069 1069 if not self.isConfig:
1070 1070
1071 1071 nplots = 1
1072 1072
1073 1073 self.setup(id=id,
1074 1074 nplots=nplots,
1075 1075 wintitle=wintitle,
1076 1076 show=show)
1077 1077
1078 1078 if xmin == None: xmin = numpy.nanmin(x)*0.9
1079 1079 if xmax == None: xmax = numpy.nanmax(x)*1.1
1080 1080 if ymin == None: ymin = numpy.nanmin(zdB)
1081 1081 if ymax == None: ymax = numpy.nanmax(zdB)
1082 1082
1083 1083 self.isConfig = True
1084 1084
1085 1085 self.setWinTitle(title)
1086 1086
1087 1087 title = "Spectra Cuts: %s" %(thisDatetime.strftime("%d-%b-%Y %H:%M:%S"))
1088 1088 axes = self.axesList[0]
1089 1089
1090 1090 legendlabels = ["Range = %dKm" %y[i] for i in hei_index]
1091 1091
1092 1092 axes.pmultilineyaxis( x, zdB,
1093 1093 xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax,
1094 1094 xlabel=xlabel, ylabel=ylabel, title=title, legendlabels=legendlabels,
1095 1095 ytick_visible=True, nxticks=5,
1096 1096 grid='x')
1097 1097
1098 1098 self.draw()
1099 1099
1100 1100 self.save(figpath=figpath,
1101 1101 figfile=figfile,
1102 1102 save=save,
1103 1103 ftp=ftp,
1104 1104 wr_period=wr_period,
1105 1105 thisDatetime=thisDatetime)
1106 1106
1107 1107 class Noise(Figure):
1108 1108
1109 1109 isConfig = None
1110 1110 __nsubplots = None
1111 1111
1112 1112 PREFIX = 'noise'
1113 1113
1114 1114 def __init__(self, **kwargs):
1115 1115 Figure.__init__(self, **kwargs)
1116 1116 self.timerange = 24*60*60
1117 1117 self.isConfig = False
1118 1118 self.__nsubplots = 1
1119 1119 self.counter_imagwr = 0
1120 1120 self.WIDTH = 800
1121 1121 self.HEIGHT = 400
1122 1122 self.WIDTHPROF = 120
1123 1123 self.HEIGHTPROF = 0
1124 1124 self.xdata = None
1125 1125 self.ydata = None
1126 1126
1127 1127 self.PLOT_CODE = NOISE_CODE
1128 1128
1129 1129 self.FTP_WEI = None
1130 1130 self.EXP_CODE = None
1131 1131 self.SUB_EXP_CODE = None
1132 1132 self.PLOT_POS = None
1133 1133 self.figfile = None
1134 1134
1135 1135 self.xmin = None
1136 1136 self.xmax = None
1137 1137
1138 1138 def getSubplots(self):
1139 1139
1140 1140 ncol = 1
1141 1141 nrow = 1
1142 1142
1143 1143 return nrow, ncol
1144 1144
1145 1145 def openfile(self, filename):
1146 1146 dirname = os.path.dirname(filename)
1147 1147
1148 1148 if not os.path.exists(dirname):
1149 1149 os.mkdir(dirname)
1150 1150
1151 1151 f = open(filename,'w+')
1152 1152 f.write('\n\n')
1153 1153 f.write('JICAMARCA RADIO OBSERVATORY - Noise \n')
1154 1154 f.write('DD MM YYYY HH MM SS Channel0 Channel1 Channel2 Channel3\n\n' )
1155 1155 f.close()
1156 1156
1157 1157 def save_data(self, filename_phase, data, data_datetime):
1158 1158
1159 1159 f=open(filename_phase,'a')
1160 1160
1161 1161 timetuple_data = data_datetime.timetuple()
1162 1162 day = str(timetuple_data.tm_mday)
1163 1163 month = str(timetuple_data.tm_mon)
1164 1164 year = str(timetuple_data.tm_year)
1165 1165 hour = str(timetuple_data.tm_hour)
1166 1166 minute = str(timetuple_data.tm_min)
1167 1167 second = str(timetuple_data.tm_sec)
1168 1168
1169 1169 data_msg = ''
1170 1170 for i in range(len(data)):
1171 1171 data_msg += str(data[i]) + ' '
1172 1172
1173 1173 f.write(day+' '+month+' '+year+' '+hour+' '+minute+' '+second+' ' + data_msg + '\n')
1174 1174 f.close()
1175 1175
1176 1176
1177 1177 def setup(self, id, nplots, wintitle, showprofile=True, show=True):
1178 1178
1179 1179 self.__showprofile = showprofile
1180 1180 self.nplots = nplots
1181 1181
1182 1182 ncolspan = 7
1183 1183 colspan = 6
1184 1184 self.__nsubplots = 2
1185 1185
1186 1186 self.createFigure(id = id,
1187 1187 wintitle = wintitle,
1188 1188 widthplot = self.WIDTH+self.WIDTHPROF,
1189 1189 heightplot = self.HEIGHT+self.HEIGHTPROF,
1190 1190 show=show)
1191 1191
1192 1192 nrow, ncol = self.getSubplots()
1193 1193
1194 1194 self.addAxes(nrow, ncol*ncolspan, 0, 0, colspan, 1)
1195 1195
1196 1196
1197 1197 def run(self, dataOut, id, wintitle="", channelList=None, showprofile='True',
1198 1198 xmin=None, xmax=None, ymin=None, ymax=None,
1199 1199 timerange=None,
1200 1200 save=False, figpath='./', figfile=None, show=True, ftp=False, wr_period=1,
1201 1201 server=None, folder=None, username=None, password=None,
1202 1202 ftp_wei=0, exp_code=0, sub_exp_code=0, plot_pos=0):
1203 1203
1204 1204 if not isTimeInHourRange(dataOut.datatime, xmin, xmax):
1205 1205 return
1206 1206
1207 1207 if channelList == None:
1208 1208 channelIndexList = dataOut.channelIndexList
1209 1209 channelList = dataOut.channelList
1210 1210 else:
1211 1211 channelIndexList = []
1212 1212 for channel in channelList:
1213 1213 if channel not in dataOut.channelList:
1214 1214 raise ValueError, "Channel %d is not in dataOut.channelList"
1215 1215 channelIndexList.append(dataOut.channelList.index(channel))
1216 1216
1217 1217 x = dataOut.getTimeRange()
1218 1218 #y = dataOut.getHeiRange()
1219 1219 factor = dataOut.normFactor
1220 1220 noise = dataOut.noise[channelIndexList]/factor
1221 1221 noisedB = 10*numpy.log10(noise)
1222 1222
1223 1223 thisDatetime = dataOut.datatime
1224 1224
1225 1225 title = wintitle + " Noise" # : %s" %(thisDatetime.strftime("%d-%b-%Y"))
1226 1226 xlabel = ""
1227 1227 ylabel = "Intensity (dB)"
1228 1228 update_figfile = False
1229 1229
1230 1230 if not self.isConfig:
1231 1231
1232 1232 nplots = 1
1233 1233
1234 1234 self.setup(id=id,
1235 1235 nplots=nplots,
1236 1236 wintitle=wintitle,
1237 1237 showprofile=showprofile,
1238 1238 show=show)
1239 1239
1240 1240 if timerange != None:
1241 1241 self.timerange = timerange
1242 1242
1243 1243 self.xmin, self.xmax = self.getTimeLim(x, xmin, xmax, timerange)
1244 1244
1245 1245 if ymin == None: ymin = numpy.floor(numpy.nanmin(noisedB)) - 10.0
1246 1246 if ymax == None: ymax = numpy.nanmax(noisedB) + 10.0
1247 1247
1248 1248 self.FTP_WEI = ftp_wei
1249 1249 self.EXP_CODE = exp_code
1250 1250 self.SUB_EXP_CODE = sub_exp_code
1251 1251 self.PLOT_POS = plot_pos
1252 1252
1253 1253
1254 1254 self.name = thisDatetime.strftime("%Y%m%d_%H%M%S")
1255 1255 self.isConfig = True
1256 1256 self.figfile = figfile
1257 1257 self.xdata = numpy.array([])
1258 1258 self.ydata = numpy.array([])
1259 1259
1260 1260 update_figfile = True
1261 1261
1262 1262 #open file beacon phase
1263 1263 path = '%s%03d' %(self.PREFIX, self.id)
1264 1264 noise_file = os.path.join(path,'%s.txt'%self.name)
1265 1265 self.filename_noise = os.path.join(figpath,noise_file)
1266 1266
1267 1267 self.setWinTitle(title)
1268 1268
1269 1269 title = "Noise %s" %(thisDatetime.strftime("%Y/%m/%d %H:%M:%S"))
1270 1270
1271 1271 legendlabels = ["channel %d"%(idchannel) for idchannel in channelList]
1272 1272 axes = self.axesList[0]
1273 1273
1274 1274 self.xdata = numpy.hstack((self.xdata, x[0:1]))
1275 1275
1276 1276 if len(self.ydata)==0:
1277 1277 self.ydata = noisedB.reshape(-1,1)
1278 1278 else:
1279 1279 self.ydata = numpy.hstack((self.ydata, noisedB.reshape(-1,1)))
1280 1280
1281 1281
1282 1282 axes.pmultilineyaxis(x=self.xdata, y=self.ydata,
1283 1283 xmin=self.xmin, xmax=self.xmax, ymin=ymin, ymax=ymax,
1284 1284 xlabel=xlabel, ylabel=ylabel, title=title, legendlabels=legendlabels, marker='x', markersize=8, linestyle="solid",
1285 1285 XAxisAsTime=True, grid='both'
1286 1286 )
1287 1287
1288 1288 self.draw()
1289 1289
1290 1290 if dataOut.ltctime >= self.xmax:
1291 1291 self.counter_imagwr = wr_period
1292 1292 self.isConfig = False
1293 1293 update_figfile = True
1294 1294
1295 1295 self.save(figpath=figpath,
1296 1296 figfile=figfile,
1297 1297 save=save,
1298 1298 ftp=ftp,
1299 1299 wr_period=wr_period,
1300 1300 thisDatetime=thisDatetime,
1301 1301 update_figfile=update_figfile)
1302 1302
1303 1303 #store data beacon phase
1304 1304 if save:
1305 1305 self.save_data(self.filename_noise, noisedB, thisDatetime)
1306 1306
1307 1307 class BeaconPhase(Figure):
1308 1308
1309 1309 __isConfig = None
1310 1310 __nsubplots = None
1311 1311
1312 1312 PREFIX = 'beacon_phase'
1313 1313
1314 1314 def __init__(self, **kwargs):
1315 1315 Figure.__init__(self, **kwargs)
1316 1316 self.timerange = 24*60*60
1317 1317 self.isConfig = False
1318 1318 self.__nsubplots = 1
1319 1319 self.counter_imagwr = 0
1320 1320 self.WIDTH = 800
1321 1321 self.HEIGHT = 400
1322 1322 self.WIDTHPROF = 120
1323 1323 self.HEIGHTPROF = 0
1324 1324 self.xdata = None
1325 1325 self.ydata = None
1326 1326
1327 1327 self.PLOT_CODE = BEACON_CODE
1328 1328
1329 1329 self.FTP_WEI = None
1330 1330 self.EXP_CODE = None
1331 1331 self.SUB_EXP_CODE = None
1332 1332 self.PLOT_POS = None
1333 1333
1334 1334 self.filename_phase = None
1335 1335
1336 1336 self.figfile = None
1337 1337
1338 1338 self.xmin = None
1339 1339 self.xmax = None
1340 1340
1341 1341 def getSubplots(self):
1342 1342
1343 1343 ncol = 1
1344 1344 nrow = 1
1345 1345
1346 1346 return nrow, ncol
1347 1347
1348 1348 def setup(self, id, nplots, wintitle, showprofile=True, show=True):
1349 1349
1350 1350 self.__showprofile = showprofile
1351 1351 self.nplots = nplots
1352 1352
1353 1353 ncolspan = 7
1354 1354 colspan = 6
1355 1355 self.__nsubplots = 2
1356 1356
1357 1357 self.createFigure(id = id,
1358 1358 wintitle = wintitle,
1359 1359 widthplot = self.WIDTH+self.WIDTHPROF,
1360 1360 heightplot = self.HEIGHT+self.HEIGHTPROF,
1361 1361 show=show)
1362 1362
1363 1363 nrow, ncol = self.getSubplots()
1364 1364
1365 1365 self.addAxes(nrow, ncol*ncolspan, 0, 0, colspan, 1)
1366 1366
1367 1367 def save_phase(self, filename_phase):
1368 1368 f = open(filename_phase,'w+')
1369 1369 f.write('\n\n')
1370 1370 f.write('JICAMARCA RADIO OBSERVATORY - Beacon Phase \n')
1371 1371 f.write('DD MM YYYY HH MM SS pair(2,0) pair(2,1) pair(2,3) pair(2,4)\n\n' )
1372 1372 f.close()
1373 1373
1374 1374 def save_data(self, filename_phase, data, data_datetime):
1375 1375 f=open(filename_phase,'a')
1376 1376 timetuple_data = data_datetime.timetuple()
1377 1377 day = str(timetuple_data.tm_mday)
1378 1378 month = str(timetuple_data.tm_mon)
1379 1379 year = str(timetuple_data.tm_year)
1380 1380 hour = str(timetuple_data.tm_hour)
1381 1381 minute = str(timetuple_data.tm_min)
1382 1382 second = str(timetuple_data.tm_sec)
1383 1383 f.write(day+' '+month+' '+year+' '+hour+' '+minute+' '+second+' '+str(data[0])+' '+str(data[1])+' '+str(data[2])+' '+str(data[3])+'\n')
1384 1384 f.close()
1385 1385
1386 1386
1387 1387 def run(self, dataOut, id, wintitle="", pairsList=None, showprofile='True',
1388 1388 xmin=None, xmax=None, ymin=None, ymax=None, hmin=None, hmax=None,
1389 1389 timerange=None,
1390 1390 save=False, figpath='./', figfile=None, show=True, ftp=False, wr_period=1,
1391 1391 server=None, folder=None, username=None, password=None,
1392 1392 ftp_wei=0, exp_code=0, sub_exp_code=0, plot_pos=0):
1393 1393
1394 1394 if not isTimeInHourRange(dataOut.datatime, xmin, xmax):
1395 1395 return
1396 1396
1397 1397 if pairsList == None:
1398 1398 pairsIndexList = dataOut.pairsIndexList[:10]
1399 1399 else:
1400 1400 pairsIndexList = []
1401 1401 for pair in pairsList:
1402 1402 if pair not in dataOut.pairsList:
1403 1403 raise ValueError, "Pair %s is not in dataOut.pairsList" %(pair)
1404 1404 pairsIndexList.append(dataOut.pairsList.index(pair))
1405 1405
1406 1406 if pairsIndexList == []:
1407 1407 return
1408 1408
1409 1409 # if len(pairsIndexList) > 4:
1410 1410 # pairsIndexList = pairsIndexList[0:4]
1411 1411
1412 1412 hmin_index = None
1413 1413 hmax_index = None
1414 1414
1415 1415 if hmin != None and hmax != None:
1416 1416 indexes = numpy.arange(dataOut.nHeights)
1417 1417 hmin_list = indexes[dataOut.heightList >= hmin]
1418 1418 hmax_list = indexes[dataOut.heightList <= hmax]
1419 1419
1420 1420 if hmin_list.any():
1421 1421 hmin_index = hmin_list[0]
1422 1422
1423 1423 if hmax_list.any():
1424 1424 hmax_index = hmax_list[-1]+1
1425 1425
1426 1426 x = dataOut.getTimeRange()
1427 1427 #y = dataOut.getHeiRange()
1428 1428
1429 1429
1430 1430 thisDatetime = dataOut.datatime
1431 1431
1432 1432 title = wintitle + " Signal Phase" # : %s" %(thisDatetime.strftime("%d-%b-%Y"))
1433 1433 xlabel = "Local Time"
1434 1434 ylabel = "Phase (degrees)"
1435 1435
1436 1436 update_figfile = False
1437 1437
1438 1438 nplots = len(pairsIndexList)
1439 1439 #phase = numpy.zeros((len(pairsIndexList),len(dataOut.beacon_heiIndexList)))
1440 1440 phase_beacon = numpy.zeros(len(pairsIndexList))
1441 1441 for i in range(nplots):
1442 1442 pair = dataOut.pairsList[pairsIndexList[i]]
1443 1443 ccf = numpy.average(dataOut.data_cspc[pairsIndexList[i], :, hmin_index:hmax_index], axis=0)
1444 1444 powa = numpy.average(dataOut.data_spc[pair[0], :, hmin_index:hmax_index], axis=0)
1445 1445 powb = numpy.average(dataOut.data_spc[pair[1], :, hmin_index:hmax_index], axis=0)
1446 1446 avgcoherenceComplex = ccf/numpy.sqrt(powa*powb)
1447 1447 phase = numpy.arctan2(avgcoherenceComplex.imag, avgcoherenceComplex.real)*180/numpy.pi
1448 1448
1449 1449 #print "Phase %d%d" %(pair[0], pair[1])
1450 1450 #print phase[dataOut.beacon_heiIndexList]
1451 1451
1452 1452 if dataOut.beacon_heiIndexList:
1453 1453 phase_beacon[i] = numpy.average(phase[dataOut.beacon_heiIndexList])
1454 1454 else:
1455 1455 phase_beacon[i] = numpy.average(phase)
1456 1456
1457 1457 if not self.isConfig:
1458 1458
1459 1459 nplots = len(pairsIndexList)
1460 1460
1461 1461 self.setup(id=id,
1462 1462 nplots=nplots,
1463 1463 wintitle=wintitle,
1464 1464 showprofile=showprofile,
1465 1465 show=show)
1466 1466
1467 1467 if timerange != None:
1468 1468 self.timerange = timerange
1469 1469
1470 1470 self.xmin, self.xmax = self.getTimeLim(x, xmin, xmax, timerange)
1471 1471
1472 1472 if ymin == None: ymin = 0
1473 1473 if ymax == None: ymax = 360
1474 1474
1475 1475 self.FTP_WEI = ftp_wei
1476 1476 self.EXP_CODE = exp_code
1477 1477 self.SUB_EXP_CODE = sub_exp_code
1478 1478 self.PLOT_POS = plot_pos
1479 1479
1480 1480 self.name = thisDatetime.strftime("%Y%m%d_%H%M%S")
1481 1481 self.isConfig = True
1482 1482 self.figfile = figfile
1483 1483 self.xdata = numpy.array([])
1484 1484 self.ydata = numpy.array([])
1485 1485
1486 1486 update_figfile = True
1487 1487
1488 1488 #open file beacon phase
1489 1489 path = '%s%03d' %(self.PREFIX, self.id)
1490 1490 beacon_file = os.path.join(path,'%s.txt'%self.name)
1491 1491 self.filename_phase = os.path.join(figpath,beacon_file)
1492 1492 #self.save_phase(self.filename_phase)
1493 1493
1494 1494
1495 1495 #store data beacon phase
1496 1496 #self.save_data(self.filename_phase, phase_beacon, thisDatetime)
1497 1497
1498 1498 self.setWinTitle(title)
1499 1499
1500 1500
1501 1501 title = "Phase Plot %s" %(thisDatetime.strftime("%Y/%m/%d %H:%M:%S"))
1502 1502
1503 1503 legendlabels = ["Pair (%d,%d)"%(pair[0], pair[1]) for pair in dataOut.pairsList]
1504 1504
1505 1505 axes = self.axesList[0]
1506 1506
1507 1507 self.xdata = numpy.hstack((self.xdata, x[0:1]))
1508 1508
1509 1509 if len(self.ydata)==0:
1510 1510 self.ydata = phase_beacon.reshape(-1,1)
1511 1511 else:
1512 1512 self.ydata = numpy.hstack((self.ydata, phase_beacon.reshape(-1,1)))
1513 1513
1514 1514
1515 1515 axes.pmultilineyaxis(x=self.xdata, y=self.ydata,
1516 1516 xmin=self.xmin, xmax=self.xmax, ymin=ymin, ymax=ymax,
1517 1517 xlabel=xlabel, ylabel=ylabel, title=title, legendlabels=legendlabels, marker='x', markersize=8, linestyle="solid",
1518 1518 XAxisAsTime=True, grid='both'
1519 1519 )
1520 1520
1521 1521 self.draw()
1522 1522
1523 1523 if dataOut.ltctime >= self.xmax:
1524 1524 self.counter_imagwr = wr_period
1525 1525 self.isConfig = False
1526 1526 update_figfile = True
1527 1527
1528 1528 self.save(figpath=figpath,
1529 1529 figfile=figfile,
1530 1530 save=save,
1531 1531 ftp=ftp,
1532 1532 wr_period=wr_period,
1533 1533 thisDatetime=thisDatetime,
1534 1534 update_figfile=update_figfile)
Show file before | Show file after | Hide comments
@@ -1,240 +1,240
1 1 '''
2 2 Created on Jul 9, 2014
3 3
4 4 @author: roj-idl71
5 5 '''
6 6 import os, sys
7 7 import datetime
8 8 import numpy
9 9 import traceback
10 10
11 11 from time import sleep
12 12 from threading import Lock
13 13 # from threading import Thread
14 14
15 15 import schainpy
16 16 import schainpy.admin
17 17
18 18 from schainpy.model.proc.jroproc_base import Operation
19 19 from schainpy.model.serializer.data import obj2Dict, dict2Obj
20 20 from jroplot_correlation import *
21 21 from jroplot_heispectra import *
22 22 from jroplot_parameters import *
23 23 from jroplot_spectra import *
24 24 from jroplot_voltage import *
25 25
26 26
27 27 class Plotter(Operation):
28 28
29 29 isConfig = None
30 30 name = None
31 31 __queue = None
32 32
33 33 def __init__(self, plotter_name, plotter_queue=None, **kwargs):
34 34
35 35 Operation.__init__(self, **kwargs)
36 36
37 37 self.isConfig = False
38 38 self.name = plotter_name
39 39 self.__queue = plotter_queue
40 40
41 41 def getSubplots(self):
42 42
43 43 nrow = self.nplots
44 44 ncol = 1
45 45 return nrow, ncol
46 46
47 47 def setup(self, **kwargs):
48 48
49 49 print "Initializing ..."
50 50
51 51
52 52 def run(self, dataOut, id=None, **kwargs):
53 53
54 54 """
55 55
56 56 Input:
57 57 dataOut :
58 58 id :
59 59 """
60 60
61 61 packDict = {}
62 62
63 63 packDict['id'] = id
64 64 packDict['name'] = self.name
65 65 packDict['kwargs'] = kwargs
66 66
67 67 # packDict['data'] = obj2Dict(dataOut)
68 68 packDict['data'] = dataOut
69 69
70 70 self.__queue.put(packDict)
71 71
72 72 # class PlotManager(Thread):
73 73 class PlotManager():
74 74
75 75 __err = False
76 76 __stop = False
77 77 __realtime = False
78 78
79 79 controllerThreadObj = None
80 80
81 81 plotterList = ['Scope',
82 82 'SpectraPlot', 'RTIPlot',
83 83 'SpectraCutPlot',
84 84 'CrossSpectraPlot', 'CoherenceMap',
85 85 'PowerProfilePlot', 'Noise', 'BeaconPhase',
86 86 'CorrelationPlot',
87 87 'SpectraHeisScope', 'RTIfromSpectraHeis']
88 88
89 89 def __init__(self, plotter_queue):
90 90
91 91 # Thread.__init__(self)
92 92 # self.setDaemon(True)
93 93
94 94 self.__queue = plotter_queue
95 95 self.__lock = Lock()
96 96
97 97 self.plotInstanceDict = {}
98 98
99 99 self.__err = False
100 100 self.__stop = False
101 101 self.__realtime = False
102 102
103 103 def __handleError(self, name="", send_email=False):
104 104
105 105 err = traceback.format_exception(sys.exc_info()[0],
106 106 sys.exc_info()[1],
107 107 sys.exc_info()[2])
108 108
109 109 print "***** Error occurred in PlotManager *****"
110 110 print "***** [%s]: %s" %(name, err[-1])
111 111
112 112 message = "\nError ocurred in %s:\n" %name
113 113 message += "".join(err)
114 114
115 115 sys.stderr.write(message)
116 116
117 117 if not send_email:
118 118 return
119 119
120 120 import socket
121 121
122 122 subject = "SChain v%s: Error running %s\n" %(schainpy.__version__, name)
123 123
124 124 subtitle = "%s:\n" %(name)
125 125 subtitle += "Hostname: %s\n" %socket.gethostbyname(socket.gethostname())
126 126 subtitle += "Working directory: %s\n" %os.path.abspath("./")
127 # subtitle += "Configuration file: %s\n" %self.filename
127 # subtitle += "Configuration file: %s\n" %self.filename
128 128 subtitle += "Time: %s\n" %str(datetime.datetime.now())
129 129
130 130 adminObj = schainpy.admin.SchainNotify()
131 131 adminObj.sendAlert(message=message,
132 132 subject=subject,
133 133 subtitle=subtitle)
134 134
135 135 def run(self):
136 136
137 137 if self.__queue.empty():
138 138 return
139 139
140 140 if self.__err:
141 141 serial_data = self.__queue.get()
142 142 self.__queue.task_done()
143 143 return
144 144
145 145 self.__lock.acquire()
146 146
147 147 # if self.__queue.full():
148 148 # for i in range(int(self.__queue.qsize()/2)):
149 149 # serial_data = self.__queue.get()
150 150 # self.__queue.task_done()
151 151
152 152 n = int(self.__queue.qsize()/3 + 1)
153 153
154 154 for i in range(n):
155 155
156 156 if self.__queue.empty():
157 157 break
158 158
159 159 serial_data = self.__queue.get()
160 160 self.__queue.task_done()
161 161
162 162 plot_id = serial_data['id']
163 163 plot_name = serial_data['name']
164 164 kwargs = serial_data['kwargs']
165 165 # dataDict = serial_data['data']
166 166 #
167 167 # dataPlot = dict2Obj(dataDict)
168 168
169 169 dataPlot = serial_data['data']
170 170
171 171 if plot_id not in self.plotInstanceDict.keys():
172 172 className = eval(plot_name)
173 173 self.plotInstanceDict[plot_id] = className(**kwargs)
174 174
175 175 plotter = self.plotInstanceDict[plot_id]
176 176 try:
177 177 plotter.run(dataPlot, plot_id, **kwargs)
178 178 except:
179 179 self.__err = True
180 180 self.__handleError(plot_name, send_email=True)
181 181 break
182 182
183 183 self.__lock.release()
184 184
185 185 def isEmpty(self):
186 186
187 187 return self.__queue.empty()
188 188
189 189 def stop(self):
190 190
191 191 self.__lock.acquire()
192 192
193 193 self.__stop = True
194 194
195 195 self.__lock.release()
196 196
197 197 def close(self):
198 198
199 199 self.__lock.acquire()
200 200
201 201 for plot_id in self.plotInstanceDict.keys():
202 202 plotter = self.plotInstanceDict[plot_id]
203 203 plotter.close()
204 204
205 205 self.__lock.release()
206 206
207 207 def setController(self, controllerThreadObj):
208 208
209 209 self.controllerThreadObj = controllerThreadObj
210 210
211 211 def start(self):
212 212
213 213 if not self.controllerThreadObj.isRunning():
214 214 raise RuntimeError, "controllerThreadObj has not been initialized. Use controllerThreadObj.start() before call this method"
215 215
216 216 self.join()
217 217
218 218 def join(self):
219 219
220 220 #Execute plotter while controller is running
221 221 while self.controllerThreadObj.isRunning():
222 222 self.run()
223 223
224 224 self.controllerThreadObj.stop()
225 225
226 226 #Wait until plotter queue is empty
227 227 while not self.isEmpty():
228 228 self.run()
229 229
230 230 self.close()
231 231
232 232 def isErrorDetected(self):
233 233
234 234 self.__lock.acquire()
235 235
236 236 err = self.__err
237 237
238 238 self.__lock.release()
239 239
240 240 return err
Show file before | Show file after | Hide comments
@@ -1,1812 +1,1779
1 1 '''
2 2 Created on Jul 2, 2014
3 3
4 4 @author: roj-idl71
5 5 '''
6 6 import os
7 7 import sys
8 8 import glob
9 9 import time
10 10 import numpy
11 11 import fnmatch
12 12 import inspect
13 13 import time, datetime
14 14 import traceback
15 15 import zmq
16 16
17 17 try:
18 18 from gevent import sleep
19 19 except:
20 20 from time import sleep
21 21
22 22 from schainpy.model.data.jroheaderIO import PROCFLAG, BasicHeader, SystemHeader, RadarControllerHeader, ProcessingHeader
23 23 from schainpy.model.data.jroheaderIO import get_dtype_index, get_numpy_dtype, get_procflag_dtype, get_dtype_width
24 24
25 25 LOCALTIME = True
26 26
27 27 def isNumber(cad):
28 28 """
29 29 Chequea si el conjunto de caracteres que componen un string puede ser convertidos a un numero.
30 30
31 31 Excepciones:
32 32 Si un determinado string no puede ser convertido a numero
33 33 Input:
34 34 str, string al cual se le analiza para determinar si convertible a un numero o no
35 35
36 36 Return:
37 37 True : si el string es uno numerico
38 38 False : no es un string numerico
39 39 """
40 40 try:
41 41 float( cad )
42 42 return True
43 43 except:
44 44 return False
45 45
46 46 def isFileInEpoch(filename, startUTSeconds, endUTSeconds):
47 47 """
48 48 Esta funcion determina si un archivo de datos se encuentra o no dentro del rango de fecha especificado.
49 49
50 50 Inputs:
51 51 filename : nombre completo del archivo de datos en formato Jicamarca (.r)
52 52
53 53 startUTSeconds : fecha inicial del rango seleccionado. La fecha esta dada en
54 54 segundos contados desde 01/01/1970.
55 55 endUTSeconds : fecha final del rango seleccionado. La fecha esta dada en
56 56 segundos contados desde 01/01/1970.
57 57
58 58 Return:
59 59 Boolean : Retorna True si el archivo de datos contiene datos en el rango de
60 60 fecha especificado, de lo contrario retorna False.
61 61
62 62 Excepciones:
63 63 Si el archivo no existe o no puede ser abierto
64 64 Si la cabecera no puede ser leida.
65 65
66 66 """
67 67 basicHeaderObj = BasicHeader(LOCALTIME)
68 68
69 69 try:
70 70 fp = open(filename,'rb')
71 71 except IOError:
72 72 print "The file %s can't be opened" %(filename)
73 73 return 0
74 74
75 75 sts = basicHeaderObj.read(fp)
76 76 fp.close()
77 77
78 78 if not(sts):
79 79 print "Skipping the file %s because it has not a valid header" %(filename)
80 80 return 0
81 81
82 82 if not ((startUTSeconds <= basicHeaderObj.utc) and (endUTSeconds > basicHeaderObj.utc)):
83 83 return 0
84 84
85 85 return 1
86 86
87 87 def isTimeInRange(thisTime, startTime, endTime):
88 88
89 89 if endTime >= startTime:
90 90 if (thisTime < startTime) or (thisTime > endTime):
91 91 return 0
92 92
93 93 return 1
94 94 else:
95 95 if (thisTime < startTime) and (thisTime > endTime):
96 96 return 0
97 97
98 98 return 1
99 99
100 100 def isFileInTimeRange(filename, startDate, endDate, startTime, endTime):
101 101 """
102 102 Retorna 1 si el archivo de datos se encuentra dentro del rango de horas especificado.
103 103
104 104 Inputs:
105 105 filename : nombre completo del archivo de datos en formato Jicamarca (.r)
106 106
107 107 startDate : fecha inicial del rango seleccionado en formato datetime.date
108 108
109 109 endDate : fecha final del rango seleccionado en formato datetime.date
110 110
111 111 startTime : tiempo inicial del rango seleccionado en formato datetime.time
112 112
113 113 endTime : tiempo final del rango seleccionado en formato datetime.time
114 114
115 115 Return:
116 116 Boolean : Retorna True si el archivo de datos contiene datos en el rango de
117 117 fecha especificado, de lo contrario retorna False.
118 118
119 119 Excepciones:
120 120 Si el archivo no existe o no puede ser abierto
121 121 Si la cabecera no puede ser leida.
122 122
123 123 """
124 124
125 125
126 126 try:
127 127 fp = open(filename,'rb')
128 128 except IOError:
129 129 print "The file %s can't be opened" %(filename)
130 130 return None
131 131
132 132 firstBasicHeaderObj = BasicHeader(LOCALTIME)
133 133 systemHeaderObj = SystemHeader()
134 134 radarControllerHeaderObj = RadarControllerHeader()
135 135 processingHeaderObj = ProcessingHeader()
136 136
137 137 lastBasicHeaderObj = BasicHeader(LOCALTIME)
138 138
139 139 sts = firstBasicHeaderObj.read(fp)
140 140
141 141 if not(sts):
142 142 print "[Reading] Skipping the file %s because it has not a valid header" %(filename)
143 143 return None
144 144
145 145 if not systemHeaderObj.read(fp):
146 146 return None
147 147
148 148 if not radarControllerHeaderObj.read(fp):
149 149 return None
150 150
151 151 if not processingHeaderObj.read(fp):
152 152 return None
153 153
154 154 filesize = os.path.getsize(filename)
155 155
156 156 offset = processingHeaderObj.blockSize + 24 #header size
157 157
158 158 if filesize <= offset:
159 159 print "[Reading] %s: This file has not enough data" %filename
160 160 return None
161 161
162 162 fp.seek(-offset, 2)
163 163
164 164 sts = lastBasicHeaderObj.read(fp)
165 165
166 166 fp.close()
167 167
168 168 thisDatetime = lastBasicHeaderObj.datatime
169 169 thisTime_last_block = thisDatetime.time()
170 170
171 171 thisDatetime = firstBasicHeaderObj.datatime
172 172 thisDate = thisDatetime.date()
173 173 thisTime_first_block = thisDatetime.time()
174 174
175 175 #General case
176 176 # o>>>>>>>>>>>>>><<<<<<<<<<<<<<o
177 177 #-----------o----------------------------o-----------
178 178 # startTime endTime
179 179
180 180 if endTime >= startTime:
181 181 if (thisTime_last_block < startTime) or (thisTime_first_block > endTime):
182 182 return None
183 183
184 184 return thisDatetime
185 185
186 186 #If endTime < startTime then endTime belongs to the next day
187 187
188 188
189 189 #<<<<<<<<<<<o o>>>>>>>>>>>
190 190 #-----------o----------------------------o-----------
191 191 # endTime startTime
192 192
193 193 if (thisDate == startDate) and (thisTime_last_block < startTime):
194 194 return None
195 195
196 196 if (thisDate == endDate) and (thisTime_first_block > endTime):
197 197 return None
198 198
199 199 if (thisTime_last_block < startTime) and (thisTime_first_block > endTime):
200 200 return None
201 201
202 202 return thisDatetime
203 203
204 204 def isFolderInDateRange(folder, startDate=None, endDate=None):
205 205 """
206 206 Retorna 1 si el archivo de datos se encuentra dentro del rango de horas especificado.
207 207
208 208 Inputs:
209 209 folder : nombre completo del directorio.
210 210 Su formato deberia ser "/path_root/?YYYYDDD"
211 211
212 212 siendo:
213 213 YYYY : Anio (ejemplo 2015)
214 214 DDD : Dia del anio (ejemplo 305)
215 215
216 216 startDate : fecha inicial del rango seleccionado en formato datetime.date
217 217
218 218 endDate : fecha final del rango seleccionado en formato datetime.date
219 219
220 220 Return:
221 221 Boolean : Retorna True si el archivo de datos contiene datos en el rango de
222 222 fecha especificado, de lo contrario retorna False.
223 223 Excepciones:
224 224 Si el directorio no tiene el formato adecuado
225 225 """
226 226
227 227 basename = os.path.basename(folder)
228 228
229 229 if not isRadarFolder(basename):
230 230 print "The folder %s has not the rigth format" %folder
231 231 return 0
232 232
233 233 if startDate and endDate:
234 234 thisDate = getDateFromRadarFolder(basename)
235 235
236 236 if thisDate < startDate:
237 237 return 0
238 238
239 239 if thisDate > endDate:
240 240 return 0
241 241
242 242 return 1
243 243
244 244 def isFileInDateRange(filename, startDate=None, endDate=None):
245 245 """
246 246 Retorna 1 si el archivo de datos se encuentra dentro del rango de horas especificado.
247 247
248 248 Inputs:
249 249 filename : nombre completo del archivo de datos en formato Jicamarca (.r)
250 250
251 251 Su formato deberia ser "?YYYYDDDsss"
252 252
253 253 siendo:
254 254 YYYY : Anio (ejemplo 2015)
255 255 DDD : Dia del anio (ejemplo 305)
256 256 sss : set
257 257
258 258 startDate : fecha inicial del rango seleccionado en formato datetime.date
259 259
260 260 endDate : fecha final del rango seleccionado en formato datetime.date
261 261
262 262 Return:
263 263 Boolean : Retorna True si el archivo de datos contiene datos en el rango de
264 264 fecha especificado, de lo contrario retorna False.
265 265 Excepciones:
266 266 Si el archivo no tiene el formato adecuado
267 267 """
268 268
269 269 basename = os.path.basename(filename)
270 270
271 271 if not isRadarFile(basename):
272 272 print "The filename %s has not the rigth format" %filename
273 273 return 0
274 274
275 275 if startDate and endDate:
276 276 thisDate = getDateFromRadarFile(basename)
277 277
278 278 if thisDate < startDate:
279 279 return 0
280 280
281 281 if thisDate > endDate:
282 282 return 0
283 283
284 284 return 1
285 285
286 286 def getFileFromSet(path, ext, set):
287 287 validFilelist = []
288 288 fileList = os.listdir(path)
289 289
290 290 # 0 1234 567 89A BCDE
291 291 # H YYYY DDD SSS .ext
292 292
293 293 for thisFile in fileList:
294 294 try:
295 295 year = int(thisFile[1:5])
296 296 doy = int(thisFile[5:8])
297 297 except:
298 298 continue
299 299
300 300 if (os.path.splitext(thisFile)[-1].lower() != ext.lower()):
301 301 continue
302 302
303 303 validFilelist.append(thisFile)
304 304
305 305 myfile = fnmatch.filter(validFilelist,'*%4.4d%3.3d%3.3d*'%(year,doy,set))
306 306
307 307 if len(myfile)!= 0:
308 308 return myfile[0]
309 309 else:
310 310 filename = '*%4.4d%3.3d%3.3d%s'%(year,doy,set,ext.lower())
311 311 print 'the filename %s does not exist'%filename
312 312 print '...going to the last file: '
313 313
314 314 if validFilelist:
315 315 validFilelist = sorted( validFilelist, key=str.lower )
316 316 return validFilelist[-1]
317 317
318 318 return None
319 319
320 320 def getlastFileFromPath(path, ext):
321 321 """
322 322 Depura el fileList dejando solo los que cumplan el formato de "PYYYYDDDSSS.ext"
323 323 al final de la depuracion devuelve el ultimo file de la lista que quedo.
324 324
325 325 Input:
326 326 fileList : lista conteniendo todos los files (sin path) que componen una determinada carpeta
327 327 ext : extension de los files contenidos en una carpeta
328 328
329 329 Return:
330 330 El ultimo file de una determinada carpeta, no se considera el path.
331 331 """
332 332 validFilelist = []
333 333 fileList = os.listdir(path)
334 334
335 335 # 0 1234 567 89A BCDE
336 336 # H YYYY DDD SSS .ext
337 337
338 338 for thisFile in fileList:
339 339
340 340 year = thisFile[1:5]
341 341 if not isNumber(year):
342 342 continue
343 343
344 344 doy = thisFile[5:8]
345 345 if not isNumber(doy):
346 346 continue
347 347
348 348 year = int(year)
349 349 doy = int(doy)
350 350
351 351 if (os.path.splitext(thisFile)[-1].lower() != ext.lower()):
352 352 continue
353 353
354 354 validFilelist.append(thisFile)
355 355
356 356 if validFilelist:
357 357 validFilelist = sorted( validFilelist, key=str.lower )
358 358 return validFilelist[-1]
359 359
360 360 return None
361 361
362 362 def checkForRealPath(path, foldercounter, year, doy, set, ext):
363 363 """
364 364 Por ser Linux Case Sensitive entonces checkForRealPath encuentra el nombre correcto de un path,
365 365 Prueba por varias combinaciones de nombres entre mayusculas y minusculas para determinar
366 366 el path exacto de un determinado file.
367 367
368 368 Example :
369 369 nombre correcto del file es .../.../D2009307/P2009307367.ext
370 370
371 371 Entonces la funcion prueba con las siguientes combinaciones
372 372 .../.../y2009307367.ext
373 373 .../.../Y2009307367.ext
374 374 .../.../x2009307/y2009307367.ext
375 375 .../.../x2009307/Y2009307367.ext
376 376 .../.../X2009307/y2009307367.ext
377 377 .../.../X2009307/Y2009307367.ext
378 378 siendo para este caso, la ultima combinacion de letras, identica al file buscado
379 379
380 380 Return:
381 381 Si encuentra la cobinacion adecuada devuelve el path completo y el nombre del file
382 382 caso contrario devuelve None como path y el la ultima combinacion de nombre en mayusculas
383 383 para el filename
384 384 """
385 385 fullfilename = None
386 386 find_flag = False
387 387 filename = None
388 388
389 389 prefixDirList = [None,'d','D']
390 390 if ext.lower() == ".r": #voltage
391 391 prefixFileList = ['d','D']
392 392 elif ext.lower() == ".pdata": #spectra
393 393 prefixFileList = ['p','P']
394 394 else:
395 395 return None, filename
396 396
397 397 #barrido por las combinaciones posibles
398 398 for prefixDir in prefixDirList:
399 399 thispath = path
400 400 if prefixDir != None:
401 401 #formo el nombre del directorio xYYYYDDD (x=d o x=D)
402 402 if foldercounter == 0:
403 403 thispath = os.path.join(path, "%s%04d%03d" % ( prefixDir, year, doy ))
404 404 else:
405 405 thispath = os.path.join(path, "%s%04d%03d_%02d" % ( prefixDir, year, doy , foldercounter))
406 406 for prefixFile in prefixFileList: #barrido por las dos combinaciones posibles de "D"
407 407 filename = "%s%04d%03d%03d%s" % ( prefixFile, year, doy, set, ext ) #formo el nombre del file xYYYYDDDSSS.ext
408 408 fullfilename = os.path.join( thispath, filename ) #formo el path completo
409 409
410 410 if os.path.exists( fullfilename ): #verifico que exista
411 411 find_flag = True
412 412 break
413 413 if find_flag:
414 414 break
415 415
416 416 if not(find_flag):
417 417 return None, filename
418 418
419 419 return fullfilename, filename
420 420
421 421 def isRadarFolder(folder):
422 422 try:
423 423 year = int(folder[1:5])
424 424 doy = int(folder[5:8])
425 425 except:
426 426 return 0
427 427
428 428 return 1
429 429
430 430 def isRadarFile(file):
431 431 try:
432 432 year = int(file[1:5])
433 433 doy = int(file[5:8])
434 434 set = int(file[8:11])
435 435 except:
436 436 return 0
437 437
438 438 return 1
439 439
440 440 def getDateFromRadarFile(file):
441 441 try:
442 442 year = int(file[1:5])
443 443 doy = int(file[5:8])
444 444 set = int(file[8:11])
445 445 except:
446 446 return None
447 447
448 448 thisDate = datetime.date(year, 1, 1) + datetime.timedelta(doy-1)
449 449 return thisDate
450 450
451 451 def getDateFromRadarFolder(folder):
452 452 try:
453 453 year = int(folder[1:5])
454 454 doy = int(folder[5:8])
455 455 except:
456 456 return None
457 457
458 458 thisDate = datetime.date(year, 1, 1) + datetime.timedelta(doy-1)
459 459 return thisDate
460 460
461 461 class JRODataIO:
462 462
463 463 c = 3E8
464 464
465 465 isConfig = False
466 466
467 467 basicHeaderObj = None
468 468
469 469 systemHeaderObj = None
470 470
471 471 radarControllerHeaderObj = None
472 472
473 473 processingHeaderObj = None
474 474
475 475 dtype = None
476 476
477 477 pathList = []
478 478
479 479 filenameList = []
480 480
481 481 filename = None
482 482
483 483 ext = None
484 484
485 485 flagIsNewFile = 1
486 486
487 487 flagDiscontinuousBlock = 0
488 488
489 489 flagIsNewBlock = 0
490 490
491 491 fp = None
492 492
493 493 firstHeaderSize = 0
494 494
495 495 basicHeaderSize = 24
496 496
497 497 versionFile = 1103
498 498
499 499 fileSize = None
500 500
501 501 # ippSeconds = None
502 502
503 503 fileSizeByHeader = None
504 504
505 505 fileIndex = None
506 506
507 507 profileIndex = None
508 508
509 509 blockIndex = None
510 510
511 511 nTotalBlocks = None
512 512
513 513 maxTimeStep = 30
514 514
515 515 lastUTTime = None
516 516
517 517 datablock = None
518 518
519 519 dataOut = None
520 520
521 521 blocksize = None
522 522
523 523 getByBlock = False
524 524
525 525 def __init__(self):
526 526
527 527 raise NotImplementedError
528 528
529 529 def run(self):
530 530
531 531 raise NotImplementedError
532 532
533 533 def getDtypeWidth(self):
534 534
535 535 dtype_index = get_dtype_index(self.dtype)
536 536 dtype_width = get_dtype_width(dtype_index)
537 537
538 538 return dtype_width
539 539
540 540 def getAllowedArgs(self):
541 541 return inspect.getargspec(self.run).args
542 542
543 543 class JRODataReader(JRODataIO):
544 544
545 545
546 546 online = 0
547 547
548 548 realtime = 0
549 549
550 550 nReadBlocks = 0
551 551
552 552 delay = 10 #number of seconds waiting a new file
553 553
554 554 nTries = 3 #quantity tries
555 555
556 556 nFiles = 3 #number of files for searching
557 557
558 558 path = None
559 559
560 560 foldercounter = 0
561 561
562 562 flagNoMoreFiles = 0
563 563
564 564 datetimeList = []
565 565
566 566 __isFirstTimeOnline = 1
567 567
568 568 __printInfo = True
569 569
570 570 profileIndex = None
571 571
572 572 nTxs = 1
573 573
574 574 txIndex = None
575 575
576 576 #Added--------------------
577 577
578 578 selBlocksize = None
579 579
580 580 selBlocktime = None
581 581
582 582
583 583 def __init__(self):
584 584
585 585 """
586 586 This class is used to find data files
587 587
588 588 Example:
589 589 reader = JRODataReader()
590 590 fileList = reader.findDataFiles()
591 591
592 592 """
593 593 pass
594 594
595 595
596 596 def createObjByDefault(self):
597 597 """
598 598
599 599 """
600 600 raise NotImplementedError
601 601
602 602 def getBlockDimension(self):
603 603
604 604 raise NotImplementedError
605 605
606 606 def __searchFilesOffLine(self,
607 607 path,
608 608 startDate=None,
609 609 endDate=None,
610 610 startTime=datetime.time(0,0,0),
611 611 endTime=datetime.time(23,59,59),
612 612 set=None,
613 613 expLabel='',
614 614 ext='.r',
615 615 queue=None,
616 616 cursor=None,
617 617 skip=None,
618 618 walk=True):
619 619
620 620 self.filenameList = []
621 621 self.datetimeList = []
622 622
623 623 pathList = []
624 624
625 625 dateList, pathList = self.findDatafiles(path, startDate, endDate, expLabel, ext, walk, include_path=True)
626 626
627 627 if dateList == []:
628 # print "[Reading] Date range selected invalid [%s - %s]: No *%s files in %s)" %(startDate, endDate, ext, path)
628 # print "[Reading] Date range selected invalid [%s - %s]: No *%s files in %s)" %(startDate, endDate, ext, path)
629 629 return None, None
630 630
631 631 if len(dateList) > 1:
632 632 print "[Reading] Data found for date range [%s - %s]: total days = %d" %(startDate, endDate, len(dateList))
633 633 else:
634 634 print "[Reading] Data found for date range [%s - %s]: date = %s" %(startDate, endDate, dateList[0])
635 635
636 636 filenameList = []
637 637 datetimeList = []
638 638
639 639 for thisPath in pathList:
640 # thisPath = pathList[pathDict[file]]
641
640 # thisPath = pathList[pathDict[file]]
641
642 642 fileList = glob.glob1(thisPath, "*%s" %ext)
643 643 fileList.sort()
644 644
645 645 skippedFileList = []
646 646
647 647 if cursor is not None and skip is not None:
648 648 # if cursor*skip > len(fileList):
649 649 if skip == 0:
650 650 if queue is not None:
651 651 queue.put(len(fileList))
652 652 skippedFileList = []
653 653 else:
654 654 skippedFileList = fileList[cursor*skip: cursor*skip + skip]
655 655
656 656 else:
657 657 skippedFileList = fileList
658 658
659 659 for file in skippedFileList:
660 660
661 661 filename = os.path.join(thisPath,file)
662 662
663 663 if not isFileInDateRange(filename, startDate, endDate):
664 664 continue
665 665
666 666 thisDatetime = isFileInTimeRange(filename, startDate, endDate, startTime, endTime)
667 667
668 668 if not(thisDatetime):
669 669 continue
670 670
671 671 filenameList.append(filename)
672 672 datetimeList.append(thisDatetime)
673 673
674 674 if not(filenameList):
675 675 print "[Reading] Time range selected invalid [%s - %s]: No *%s files in %s)" %(startTime, endTime, ext, path)
676 676 return None, None
677 677
678 678 print "[Reading] %d file(s) was(were) found in time range: %s - %s" %(len(filenameList), startTime, endTime)
679 679 print
680 680
681 681 for i in range(len(filenameList)):
682 682 print "[Reading] %s -> [%s]" %(filenameList[i], datetimeList[i].ctime())
683 683
684 684 self.filenameList = filenameList
685 685 self.datetimeList = datetimeList
686 686
687 687 return pathList, filenameList
688 688
689 689 def __searchFilesOnLine(self, path, expLabel = "", ext = None, walk=True, set=None):
690 690
691 691 """
692 692 Busca el ultimo archivo de la ultima carpeta (determinada o no por startDateTime) y
693 693 devuelve el archivo encontrado ademas de otros datos.
694 694
695 695 Input:
696 696 path : carpeta donde estan contenidos los files que contiene data
697 697
698 698 expLabel : Nombre del subexperimento (subfolder)
699 699
700 700 ext : extension de los files
701 701
702 702 walk : Si es habilitado no realiza busquedas dentro de los ubdirectorios (doypath)
703 703
704 704 Return:
705 705 directory : eL directorio donde esta el file encontrado
706 706 filename : el ultimo file de una determinada carpeta
707 707 year : el anho
708 708 doy : el numero de dia del anho
709 709 set : el set del archivo
710 710
711 711
712 712 """
713 713 if not os.path.isdir(path):
714 714 return None, None, None, None, None, None
715 715
716 716 dirList = []
717 717
718 718 if not walk:
719 719 fullpath = path
720 720 foldercounter = 0
721 721 else:
722 722 #Filtra solo los directorios
723 723 for thisPath in os.listdir(path):
724 724 if not os.path.isdir(os.path.join(path,thisPath)):
725 725 continue
726 726 if not isRadarFolder(thisPath):
727 727 continue
728 728
729 729 dirList.append(thisPath)
730 730
731 731 if not(dirList):
732 732 return None, None, None, None, None, None
733 733
734 734 dirList = sorted( dirList, key=str.lower )
735 735
736 736 doypath = dirList[-1]
737 737 foldercounter = int(doypath.split('_')[1]) if len(doypath.split('_'))>1 else 0
738 738 fullpath = os.path.join(path, doypath, expLabel)
739 739
740 740
741 741 print "[Reading] %s folder was found: " %(fullpath )
742 742
743 743 if set == None:
744 744 filename = getlastFileFromPath(fullpath, ext)
745 745 else:
746 746 filename = getFileFromSet(fullpath, ext, set)
747 747
748 748 if not(filename):
749 749 return None, None, None, None, None, None
750 750
751 751 print "[Reading] %s file was found" %(filename)
752 752
753 753 if not(self.__verifyFile(os.path.join(fullpath, filename))):
754 754 return None, None, None, None, None, None
755 755
756 756 year = int( filename[1:5] )
757 757 doy = int( filename[5:8] )
758 758 set = int( filename[8:11] )
759 759
760 760 return fullpath, foldercounter, filename, year, doy, set
761 761
762 762 def __setNextFileOffline(self):
763 763
764 764 idFile = self.fileIndex
765 765
766 766 while (True):
767 767 idFile += 1
768 768 if not(idFile < len(self.filenameList)):
769 769 self.flagNoMoreFiles = 1
770 # print "[Reading] No more Files"
770 # print "[Reading] No more Files"
771 771 return 0
772 772
773 773 filename = self.filenameList[idFile]
774 774
775 775 if not(self.__verifyFile(filename)):
776 776 continue
777 777
778 778 fileSize = os.path.getsize(filename)
779 779 fp = open(filename,'rb')
780 780 break
781 781
782 782 self.flagIsNewFile = 1
783 783 self.fileIndex = idFile
784 784 self.filename = filename
785 785 self.fileSize = fileSize
786 786 self.fp = fp
787 787
788 # print "[Reading] Setting the file: %s"%self.filename
788 # print "[Reading] Setting the file: %s"%self.filename
789 789
790 790 return 1
791 791
792 792 def __setNextFileOnline(self):
793 793 """
794 794 Busca el siguiente file que tenga suficiente data para ser leida, dentro de un folder especifico, si
795 795 no encuentra un file valido espera un tiempo determinado y luego busca en los posibles n files
796 796 siguientes.
797 797
798 798 Affected:
799 799 self.flagIsNewFile
800 800 self.filename
801 801 self.fileSize
802 802 self.fp
803 803 self.set
804 804 self.flagNoMoreFiles
805 805
806 806 Return:
807 807 0 : si luego de una busqueda del siguiente file valido este no pudo ser encontrado
808 808 1 : si el file fue abierto con exito y esta listo a ser leido
809 809
810 810 Excepciones:
811 811 Si un determinado file no puede ser abierto
812 812 """
813 813 nFiles = 0
814 814 fileOk_flag = False
815 815 firstTime_flag = True
816 816
817 817 self.set += 1
818 818
819 819 if self.set > 999:
820 820 self.set = 0
821 821 self.foldercounter += 1
822 822
823 823 #busca el 1er file disponible
824 824 fullfilename, filename = checkForRealPath( self.path, self.foldercounter, self.year, self.doy, self.set, self.ext )
825 825 if fullfilename:
826 826 if self.__verifyFile(fullfilename, False):
827 827 fileOk_flag = True
828 828
829 829 #si no encuentra un file entonces espera y vuelve a buscar
830 830 if not(fileOk_flag):
831 831 for nFiles in range(self.nFiles+1): #busco en los siguientes self.nFiles+1 files posibles
832 832
833 833 if firstTime_flag: #si es la 1era vez entonces hace el for self.nTries veces
834 834 tries = self.nTries
835 835 else:
836 836 tries = 1 #si no es la 1era vez entonces solo lo hace una vez
837 837
838 838 for nTries in range( tries ):
839 839 if firstTime_flag:
840 840 print "\t[Reading] Waiting %0.2f sec for the next file: \"%s\" , try %03d ..." % ( self.delay, filename, nTries+1 )
841 841 sleep( self.delay )
842 842 else:
843 843 print "\t[Reading] Searching the next \"%s%04d%03d%03d%s\" file ..." % (self.optchar, self.year, self.doy, self.set, self.ext)
844 844
845 845 fullfilename, filename = checkForRealPath( self.path, self.foldercounter, self.year, self.doy, self.set, self.ext )
846 846 if fullfilename:
847 847 if self.__verifyFile(fullfilename):
848 848 fileOk_flag = True
849 849 break
850 850
851 851 if fileOk_flag:
852 852 break
853 853
854 854 firstTime_flag = False
855 855
856 856 print "\t[Reading] Skipping the file \"%s\" due to this file doesn't exist" % filename
857 857 self.set += 1
858 858
859 859 if nFiles == (self.nFiles-1): #si no encuentro el file buscado cambio de carpeta y busco en la siguiente carpeta
860 860 self.set = 0
861 861 self.doy += 1
862 862 self.foldercounter = 0
863 863
864 864 if fileOk_flag:
865 865 self.fileSize = os.path.getsize( fullfilename )
866 866 self.filename = fullfilename
867 867 self.flagIsNewFile = 1
868 868 if self.fp != None: self.fp.close()
869 869 self.fp = open(fullfilename, 'rb')
870 870 self.flagNoMoreFiles = 0
871 # print '[Reading] Setting the file: %s' % fullfilename
871 # print '[Reading] Setting the file: %s' % fullfilename
872 872 else:
873 873 self.fileSize = 0
874 874 self.filename = None
875 875 self.flagIsNewFile = 0
876 876 self.fp = None
877 877 self.flagNoMoreFiles = 1
878 # print '[Reading] No more files to read'
878 # print '[Reading] No more files to read'
879 879
880 880 return fileOk_flag
881 881
882 882 def setNextFile(self):
883 883 if self.fp != None:
884 884 self.fp.close()
885 885
886 886 if self.online:
887 887 newFile = self.__setNextFileOnline()
888 888 else:
889 889 newFile = self.__setNextFileOffline()
890 890
891 891 if not(newFile):
892 892 print '[Reading] No more files to read'
893 893 return 0
894 894
895 895 if self.verbose:
896 896 print '[Reading] Setting the file: %s' % self.filename
897 897
898 898 self.__readFirstHeader()
899 899 self.nReadBlocks = 0
900 900 return 1
901 901
902 902 def __waitNewBlock(self):
903 903 """
904 904 Return 1 si se encontro un nuevo bloque de datos, 0 de otra forma.
905 905
906 906 Si el modo de lectura es OffLine siempre retorn 0
907 907 """
908 908 if not self.online:
909 909 return 0
910 910
911 911 if (self.nReadBlocks >= self.processingHeaderObj.dataBlocksPerFile):
912 912 return 0
913 913
914 914 currentPointer = self.fp.tell()
915 915
916 916 neededSize = self.processingHeaderObj.blockSize + self.basicHeaderSize
917 917
918 918 for nTries in range( self.nTries ):
919 919
920 920 self.fp.close()
921 921 self.fp = open( self.filename, 'rb' )
922 922 self.fp.seek( currentPointer )
923 923
924 924 self.fileSize = os.path.getsize( self.filename )
925 925 currentSize = self.fileSize - currentPointer
926 926
927 927 if ( currentSize >= neededSize ):
928 928 self.basicHeaderObj.read(self.fp)
929 929 return 1
930 930
931 931 if self.fileSize == self.fileSizeByHeader:
932 # self.flagEoF = True
932 # self.flagEoF = True
933 933 return 0
934 934
935 935 print "[Reading] Waiting %0.2f seconds for the next block, try %03d ..." % (self.delay, nTries+1)
936 936 sleep( self.delay )
937 937
938 938
939 939 return 0
940 940
941 941 def waitDataBlock(self,pointer_location):
942 942
943 943 currentPointer = pointer_location
944 944
945 945 neededSize = self.processingHeaderObj.blockSize #+ self.basicHeaderSize
946 946
947 947 for nTries in range( self.nTries ):
948 948 self.fp.close()
949 949 self.fp = open( self.filename, 'rb' )
950 950 self.fp.seek( currentPointer )
951 951
952 952 self.fileSize = os.path.getsize( self.filename )
953 953 currentSize = self.fileSize - currentPointer
954 954
955 955 if ( currentSize >= neededSize ):
956 956 return 1
957 957
958 958 print "[Reading] Waiting %0.2f seconds for the next block, try %03d ..." % (self.delay, nTries+1)
959 959 sleep( self.delay )
960 960
961 961 return 0
962 962
963 963 def __jumpToLastBlock(self):
964 964
965 965 if not(self.__isFirstTimeOnline):
966 966 return
967 967
968 968 csize = self.fileSize - self.fp.tell()
969 969 blocksize = self.processingHeaderObj.blockSize
970 970
971 971 #salta el primer bloque de datos
972 972 if csize > self.processingHeaderObj.blockSize:
973 973 self.fp.seek(self.fp.tell() + blocksize)
974 974 else:
975 975 return
976 976
977 977 csize = self.fileSize - self.fp.tell()
978 978 neededsize = self.processingHeaderObj.blockSize + self.basicHeaderSize
979 979 while True:
980 980
981 981 if self.fp.tell()<self.fileSize:
982 982 self.fp.seek(self.fp.tell() + neededsize)
983 983 else:
984 984 self.fp.seek(self.fp.tell() - neededsize)
985 985 break
986
987 # csize = self.fileSize - self.fp.tell()
988 # neededsize = self.processingHeaderObj.blockSize + self.basicHeaderSize
989 # factor = int(csize/neededsize)
990 # if factor > 0:
991 # self.fp.seek(self.fp.tell() + factor*neededsize)
992
986
987 # csize = self.fileSize - self.fp.tell()
988 # neededsize = self.processingHeaderObj.blockSize + self.basicHeaderSize
989 # factor = int(csize/neededsize)
990 # if factor > 0:
991 # self.fp.seek(self.fp.tell() + factor*neededsize)
992
993 993 self.flagIsNewFile = 0
994 994 self.__isFirstTimeOnline = 0
995 995
996 996 def __setNewBlock(self):
997 997 #if self.server is None:
998 998 if self.fp == None:
999 999 return 0
1000
1001 # if self.online:
1002 # self.__jumpToLastBlock()
1003
1000
1001 # if self.online:
1002 # self.__jumpToLastBlock()
1003
1004 1004 if self.flagIsNewFile:
1005 1005 self.lastUTTime = self.basicHeaderObj.utc
1006 1006 return 1
1007 1007
1008 1008 if self.realtime:
1009 1009 self.flagDiscontinuousBlock = 1
1010 1010 if not(self.setNextFile()):
1011 1011 return 0
1012 1012 else:
1013 1013 return 1
1014 1014 #if self.server is None:
1015 1015 currentSize = self.fileSize - self.fp.tell()
1016 1016 neededSize = self.processingHeaderObj.blockSize + self.basicHeaderSize
1017 1017 if (currentSize >= neededSize):
1018 1018 self.basicHeaderObj.read(self.fp)
1019 1019 self.lastUTTime = self.basicHeaderObj.utc
1020 1020 return 1
1021 1021 # else:
1022 1022 # self.basicHeaderObj.read(self.zHeader)
1023 1023 # self.lastUTTime = self.basicHeaderObj.utc
1024 1024 # return 1
1025 1025 if self.__waitNewBlock():
1026 1026 self.lastUTTime = self.basicHeaderObj.utc
1027 1027 return 1
1028 1028 #if self.server is None:
1029 1029 if not(self.setNextFile()):
1030 1030 return 0
1031 1031
1032 1032 deltaTime = self.basicHeaderObj.utc - self.lastUTTime #
1033 1033 self.lastUTTime = self.basicHeaderObj.utc
1034 1034
1035 1035 self.flagDiscontinuousBlock = 0
1036 1036
1037 1037 if deltaTime > self.maxTimeStep:
1038 1038 self.flagDiscontinuousBlock = 1
1039 1039
1040 1040 return 1
1041 1041
1042 1042 def readNextBlock(self):
1043 1043
1044 1044 #Skip block out of startTime and endTime
1045 1045 while True:
1046 1046 if not(self.__setNewBlock()):
1047 1047 print 'returning'
1048 1048 return 0
1049 1049 if not(self.readBlock()):
1050 1050 return 0
1051 1051
1052 1052 self.getBasicHeader()
1053 1053
1054 1054 if not isTimeInRange(self.dataOut.datatime.time(), self.startTime, self.endTime):
1055 1055
1056 1056 print "[Reading] Block No. %d/%d -> %s [Skipping]" %(self.nReadBlocks,
1057 1057 self.processingHeaderObj.dataBlocksPerFile,
1058 1058 self.dataOut.datatime.ctime())
1059 1059 continue
1060 1060
1061 1061 break
1062 1062
1063 1063 if self.verbose:
1064 1064 print "[Reading] Block No. %d/%d -> %s" %(self.nReadBlocks,
1065 1065 self.processingHeaderObj.dataBlocksPerFile,
1066 1066 self.dataOut.datatime.ctime())
1067 1067 return 1
1068 1068
1069 1069 def __readFirstHeader(self):
1070 1070
1071 1071 self.basicHeaderObj.read(self.fp)
1072 1072 self.systemHeaderObj.read(self.fp)
1073 1073 self.radarControllerHeaderObj.read(self.fp)
1074 1074 self.processingHeaderObj.read(self.fp)
1075 1075
1076 1076 self.firstHeaderSize = self.basicHeaderObj.size
1077 1077
1078 1078 datatype = int(numpy.log2((self.processingHeaderObj.processFlags & PROCFLAG.DATATYPE_MASK))-numpy.log2(PROCFLAG.DATATYPE_CHAR))
1079 1079 if datatype == 0:
1080 1080 datatype_str = numpy.dtype([('real','<i1'),('imag','<i1')])
1081 1081 elif datatype == 1:
1082 1082 datatype_str = numpy.dtype([('real','<i2'),('imag','<i2')])
1083 1083 elif datatype == 2:
1084 1084 datatype_str = numpy.dtype([('real','<i4'),('imag','<i4')])
1085 1085 elif datatype == 3:
1086 1086 datatype_str = numpy.dtype([('real','<i8'),('imag','<i8')])
1087 1087 elif datatype == 4:
1088 1088 datatype_str = numpy.dtype([('real','<f4'),('imag','<f4')])
1089 1089 elif datatype == 5:
1090 1090 datatype_str = numpy.dtype([('real','<f8'),('imag','<f8')])
1091 1091 else:
1092 1092 raise ValueError, 'Data type was not defined'
1093 1093
1094 1094 self.dtype = datatype_str
1095 1095 #self.ippSeconds = 2 * 1000 * self.radarControllerHeaderObj.ipp / self.c
1096 1096 self.fileSizeByHeader = self.processingHeaderObj.dataBlocksPerFile * self.processingHeaderObj.blockSize + self.firstHeaderSize + self.basicHeaderSize*(self.processingHeaderObj.dataBlocksPerFile - 1)
1097 # self.dataOut.channelList = numpy.arange(self.systemHeaderObj.numChannels)
1098 # self.dataOut.channelIndexList = numpy.arange(self.systemHeaderObj.numChannels)
1097 # self.dataOut.channelList = numpy.arange(self.systemHeaderObj.numChannels)
1098 # self.dataOut.channelIndexList = numpy.arange(self.systemHeaderObj.numChannels)
1099 1099 self.getBlockDimension()
1100 1100
1101 1101 def __verifyFile(self, filename, msgFlag=True):
1102 1102
1103 1103 msg = None
1104 1104
1105 1105 try:
1106 1106 fp = open(filename, 'rb')
1107 1107 except IOError:
1108 1108
1109 1109 if msgFlag:
1110 1110 print "[Reading] File %s can't be opened" % (filename)
1111 1111
1112 1112 return False
1113 1113
1114 1114 currentPosition = fp.tell()
1115 1115 neededSize = self.processingHeaderObj.blockSize + self.firstHeaderSize
1116 1116
1117 1117 if neededSize == 0:
1118 1118 basicHeaderObj = BasicHeader(LOCALTIME)
1119 1119 systemHeaderObj = SystemHeader()
1120 1120 radarControllerHeaderObj = RadarControllerHeader()
1121 1121 processingHeaderObj = ProcessingHeader()
1122 1122
1123 1123 if not( basicHeaderObj.read(fp) ):
1124 1124 fp.close()
1125 1125 return False
1126 1126
1127 1127 if not( systemHeaderObj.read(fp) ):
1128 1128 fp.close()
1129 1129 return False
1130 1130
1131 1131 if not( radarControllerHeaderObj.read(fp) ):
1132 1132 fp.close()
1133 1133 return False
1134 1134
1135 1135 if not( processingHeaderObj.read(fp) ):
1136 1136 fp.close()
1137 1137 return False
1138 1138
1139 1139 neededSize = processingHeaderObj.blockSize + basicHeaderObj.size
1140 1140 else:
1141 1141 msg = "[Reading] Skipping the file %s due to it hasn't enough data" %filename
1142 1142
1143 1143 fp.close()
1144 1144
1145 1145 fileSize = os.path.getsize(filename)
1146 1146 currentSize = fileSize - currentPosition
1147 1147
1148 1148 if currentSize < neededSize:
1149 1149 if msgFlag and (msg != None):
1150 1150 print msg
1151 1151 return False
1152 1152
1153 1153 return True
1154 1154
1155 1155 def findDatafiles(self, path, startDate=None, endDate=None, expLabel='', ext='.r', walk=True, include_path=False):
1156 1156
1157 1157 path_empty = True
1158 1158
1159 1159 dateList = []
1160 1160 pathList = []
1161 1161
1162 1162 multi_path = path.split(',')
1163 1163
1164 1164 if not walk:
1165 1165
1166 1166 for single_path in multi_path:
1167 1167
1168 1168 if not os.path.isdir(single_path):
1169 1169 continue
1170 1170
1171 1171 fileList = glob.glob1(single_path, "*"+ext)
1172 1172
1173 1173 if not fileList:
1174 1174 continue
1175 1175
1176 1176 path_empty = False
1177 1177
1178 1178 fileList.sort()
1179 1179
1180 1180 for thisFile in fileList:
1181 1181
1182 1182 if not os.path.isfile(os.path.join(single_path, thisFile)):
1183 1183 continue
1184 1184
1185 1185 if not isRadarFile(thisFile):
1186 1186 continue
1187 1187
1188 1188 if not isFileInDateRange(thisFile, startDate, endDate):
1189 1189 continue
1190 1190
1191 1191 thisDate = getDateFromRadarFile(thisFile)
1192 1192
1193 1193 if thisDate in dateList:
1194 1194 continue
1195 1195
1196 1196 dateList.append(thisDate)
1197 1197 pathList.append(single_path)
1198 1198
1199 1199 else:
1200 1200 for single_path in multi_path:
1201 1201
1202 1202 if not os.path.isdir(single_path):
1203 1203 continue
1204 1204
1205 1205 dirList = []
1206 1206
1207 1207 for thisPath in os.listdir(single_path):
1208 1208
1209 1209 if not os.path.isdir(os.path.join(single_path,thisPath)):
1210 1210 continue
1211 1211
1212 1212 if not isRadarFolder(thisPath):
1213 1213 continue
1214 1214
1215 1215 if not isFolderInDateRange(thisPath, startDate, endDate):
1216 1216 continue
1217 1217
1218 1218 dirList.append(thisPath)
1219 1219
1220 1220 if not dirList:
1221 1221 continue
1222 1222
1223 1223 dirList.sort()
1224 1224
1225 1225 for thisDir in dirList:
1226 1226
1227 1227 datapath = os.path.join(single_path, thisDir, expLabel)
1228 1228 fileList = glob.glob1(datapath, "*"+ext)
1229 1229
1230 1230 if not fileList:
1231 1231 continue
1232 1232
1233 1233 path_empty = False
1234 1234
1235 1235 thisDate = getDateFromRadarFolder(thisDir)
1236 1236
1237 1237 pathList.append(datapath)
1238 1238 dateList.append(thisDate)
1239 1239
1240 1240 dateList.sort()
1241 1241
1242 1242 if walk:
1243 1243 pattern_path = os.path.join(multi_path[0], "[dYYYYDDD]", expLabel)
1244 1244 else:
1245 1245 pattern_path = multi_path[0]
1246 1246
1247 1247 if path_empty:
1248 1248 print "[Reading] No *%s files in %s for %s to %s" %(ext, pattern_path, startDate, endDate)
1249 1249 else:
1250 1250 if not dateList:
1251 1251 print "[Reading] Date range selected invalid [%s - %s]: No *%s files in %s)" %(startDate, endDate, ext, path)
1252 1252
1253 1253 if include_path:
1254 1254 return dateList, pathList
1255 1255
1256 1256 return dateList
1257 1257
1258 1258 def setup(self,
1259 1259 path=None,
1260 1260 startDate=None,
1261 1261 endDate=None,
1262 1262 startTime=datetime.time(0,0,0),
1263 1263 endTime=datetime.time(23,59,59),
1264 1264 set=None,
1265 1265 expLabel = "",
1266 1266 ext = None,
1267 1267 online = False,
1268 1268 delay = 60,
1269 1269 walk = True,
1270 1270 getblock = False,
1271 1271 nTxs = 1,
1272 1272 realtime=False,
1273 1273 blocksize=None,
1274 1274 blocktime=None,
1275 1275 queue=None,
1276 1276 skip=None,
1277 1277 cursor=None,
1278 1278 warnings=True,
1279 1279 verbose=True,
1280 1280 server=None):
1281 1281 if server is not None:
1282 1282 if 'tcp://' in server:
1283 1283 address = server
1284 1284 else:
1285 1285 address = 'ipc:///tmp/%s' % server
1286 1286 self.server = address
1287 1287 self.context = zmq.Context()
1288 1288 self.receiver = self.context.socket(zmq.PULL)
1289 1289 self.receiver.connect(self.server)
1290 1290 time.sleep(0.5)
1291 1291 print '[Starting] ReceiverData from {}'.format(self.server)
1292 1292 else:
1293 1293 self.server = None
1294 1294 if path == None:
1295 1295 raise ValueError, "[Reading] The path is not valid"
1296 1296
1297 1297 if ext == None:
1298 1298 ext = self.ext
1299 1299
1300 1300 if online:
1301 1301 print "[Reading] Searching files in online mode..."
1302 1302
1303 1303 for nTries in range( self.nTries ):
1304 1304 fullpath, foldercounter, file, year, doy, set = self.__searchFilesOnLine(path=path, expLabel=expLabel, ext=ext, walk=walk, set=set)
1305 1305
1306 1306 if fullpath:
1307 1307 break
1308 1308
1309 1309 print '[Reading] Waiting %0.2f sec for an valid file in %s: try %02d ...' % (self.delay, path, nTries+1)
1310 1310 sleep( self.delay )
1311 1311
1312 1312 if not(fullpath):
1313 1313 print "[Reading] There 'isn't any valid file in %s" % path
1314 1314 return
1315 1315
1316 1316 self.year = year
1317 1317 self.doy = doy
1318 1318 self.set = set - 1
1319 1319 self.path = path
1320 1320 self.foldercounter = foldercounter
1321 1321 last_set = None
1322 1322 else:
1323 1323 print "[Reading] Searching files in offline mode ..."
1324 1324 pathList, filenameList = self.__searchFilesOffLine(path, startDate=startDate, endDate=endDate,
1325 1325 startTime=startTime, endTime=endTime,
1326 1326 set=set, expLabel=expLabel, ext=ext,
1327 1327 walk=walk, cursor=cursor,
1328 1328 skip=skip, queue=queue)
1329 1329
1330 1330 if not(pathList):
1331 1331 # print "[Reading] No *%s files in %s (%s - %s)"%(ext, path,
1332 1332 # datetime.datetime.combine(startDate,startTime).ctime(),
1333 1333 # datetime.datetime.combine(endDate,endTime).ctime())
1334 1334
1335 1335 # sys.exit(-1)
1336 1336
1337 1337 self.fileIndex = -1
1338 1338 self.pathList = []
1339 1339 self.filenameList = []
1340 1340 return
1341 1341
1342 1342 self.fileIndex = -1
1343 self.pathList = pathList
1344 self.filenameList = filenameList
1345 file_name = os.path.basename(filenameList[-1])
1346 basename, ext = os.path.splitext(file_name)
1347 last_set = int(basename[-3:])
1348
1349 self.online = online
1350 self.realtime = realtime
1351 self.delay = delay
1352 ext = ext.lower()
1353 self.ext = ext
1354 self.getByBlock = getblock
1355 self.nTxs = nTxs
1356 self.startTime = startTime
1357 self.endTime = endTime
1358
1359 #Added-----------------
1360 self.selBlocksize = blocksize
1361 self.selBlocktime = blocktime
1362
1363 # Verbose-----------
1364 self.verbose = verbose
1365 self.warnings = warnings
1366
1367 if not(self.setNextFile()):
1368 if (startDate!=None) and (endDate!=None):
1369 print "[Reading] No files in range: %s - %s" %(datetime.datetime.combine(startDate,startTime).ctime(), datetime.datetime.combine(endDate,endTime).ctime())
1370 elif startDate != None:
1371 print "[Reading] No files in range: %s" %(datetime.datetime.combine(startDate,startTime).ctime())
1372 else:
1373 print "[Reading] No files"
1374
1375 self.fileIndex = -1
1376 self.pathList = []
1377 self.filenameList = []
1378 return
1379
1380 # self.getBasicHeader()
1381
1382 if last_set != None:
1383 self.dataOut.last_block = last_set * self.processingHeaderObj.dataBlocksPerFile + self.basicHeaderObj.dataBlock
1343 self.pathList = []
1344 self.filenameList = []
1345 return
1346
1347 # self.getBasicHeader()
1348
1349 if last_set != None:
1350 self.dataOut.last_block = last_set * self.processingHeaderObj.dataBlocksPerFile + self.basicHeaderObj.dataBlock
1384 1351 return
1385 1352
1386 1353 def getBasicHeader(self):
1387 1354
1388 1355 self.dataOut.utctime = self.basicHeaderObj.utc + self.basicHeaderObj.miliSecond/1000. + self.profileIndex * self.radarControllerHeaderObj.ippSeconds
1389 1356
1390 1357 self.dataOut.flagDiscontinuousBlock = self.flagDiscontinuousBlock
1391 1358
1392 1359 self.dataOut.timeZone = self.basicHeaderObj.timeZone
1393 1360
1394 1361 self.dataOut.dstFlag = self.basicHeaderObj.dstFlag
1395 1362
1396 1363 self.dataOut.errorCount = self.basicHeaderObj.errorCount
1397 1364
1398 1365 self.dataOut.useLocalTime = self.basicHeaderObj.useLocalTime
1399 1366
1400 1367 self.dataOut.ippSeconds = self.radarControllerHeaderObj.ippSeconds/self.nTxs
1401 1368
1402 1369 # self.dataOut.nProfiles = self.processingHeaderObj.profilesPerBlock*self.nTxs
1403 1370
1404 1371
1405 1372 def getFirstHeader(self):
1406 1373
1407 1374 raise NotImplementedError
1408 1375
1409 1376 def getData(self):
1410 1377
1411 1378 raise NotImplementedError
1412 1379
1413 1380 def hasNotDataInBuffer(self):
1414 1381
1415 1382 raise NotImplementedError
1416 1383
1417 1384 def readBlock(self):
1418 1385
1419 1386 raise NotImplementedError
1420 1387
1421 1388 def isEndProcess(self):
1422 1389
1423 1390 return self.flagNoMoreFiles
1424 1391
1425 1392 def printReadBlocks(self):
1426 1393
1427 1394 print "[Reading] Number of read blocks per file %04d" %self.nReadBlocks
1428 1395
1429 1396 def printTotalBlocks(self):
1430 1397
1431 1398 print "[Reading] Number of read blocks %04d" %self.nTotalBlocks
1432 1399
1433 1400 def printNumberOfBlock(self):
1434 1401
1435 1402 if self.flagIsNewBlock:
1436 1403 print "[Reading] Block No. %d/%d -> %s" %(self.nReadBlocks,
1437 1404 self.processingHeaderObj.dataBlocksPerFile,
1438 1405 self.dataOut.datatime.ctime())
1439 1406
1440 1407 def printInfo(self):
1441 1408
1442 1409 if self.__printInfo == False:
1443 1410 return
1444 1411
1445 1412 self.basicHeaderObj.printInfo()
1446 1413 self.systemHeaderObj.printInfo()
1447 1414 self.radarControllerHeaderObj.printInfo()
1448 1415 self.processingHeaderObj.printInfo()
1449 1416
1450 1417 self.__printInfo = False
1451 1418
1452 1419
1453 1420 def run(self,
1454 1421 path=None,
1455 1422 startDate=None,
1456 1423 endDate=None,
1457 1424 startTime=datetime.time(0,0,0),
1458 1425 endTime=datetime.time(23,59,59),
1459 1426 set=None,
1460 1427 expLabel = "",
1461 1428 ext = None,
1462 1429 online = False,
1463 1430 delay = 60,
1464 1431 walk = True,
1465 1432 getblock = False,
1466 1433 nTxs = 1,
1467 1434 realtime=False,
1468 1435 blocksize=None,
1469 1436 blocktime=None,
1470 1437 queue=None,
1471 1438 skip=None,
1472 1439 cursor=None,
1473 1440 warnings=True,
1474 1441 server=None,
1475 1442 verbose=True, **kwargs):
1476 1443
1477 1444 if not(self.isConfig):
1478 1445 # self.dataOut = dataOut
1479 1446 self.setup( path=path,
1480 1447 startDate=startDate,
1481 1448 endDate=endDate,
1482 1449 startTime=startTime,
1483 1450 endTime=endTime,
1484 1451 set=set,
1485 1452 expLabel=expLabel,
1486 1453 ext=ext,
1487 1454 online=online,
1488 1455 delay=delay,
1489 1456 walk=walk,
1490 1457 getblock=getblock,
1491 1458 nTxs=nTxs,
1492 1459 realtime=realtime,
1493 1460 blocksize=blocksize,
1494 1461 blocktime=blocktime,
1495 1462 queue=queue,
1496 1463 skip=skip,
1497 1464 cursor=cursor,
1498 1465 warnings=warnings,
1499 1466 server=server,
1500 1467 verbose=verbose)
1501 1468 self.isConfig = True
1502 1469 if server is None:
1503 1470 self.getData()
1504 1471 else:
1505 1472 self.getFromServer()
1506 1473
1507 1474 class JRODataWriter(JRODataIO):
1508 1475
1509 1476 """
1510 1477 Esta clase permite escribir datos a archivos procesados (.r o ,pdata). La escritura
1511 1478 de los datos siempre se realiza por bloques.
1512 1479 """
1513 1480
1514 1481 blockIndex = 0
1515 1482
1516 1483 path = None
1517 1484
1518 1485 setFile = None
1519 1486
1520 1487 profilesPerBlock = None
1521 1488
1522 1489 blocksPerFile = None
1523 1490
1524 1491 nWriteBlocks = 0
1525 1492
1526 1493 fileDate = None
1527 1494
1528 1495 def __init__(self, dataOut=None):
1529 1496 raise NotImplementedError
1530 1497
1531 1498
1532 1499 def hasAllDataInBuffer(self):
1533 1500 raise NotImplementedError
1534 1501
1535 1502
1536 1503 def setBlockDimension(self):
1537 1504 raise NotImplementedError
1538 1505
1539 1506
1540 1507 def writeBlock(self):
1541 1508 raise NotImplementedError
1542 1509
1543 1510
1544 1511 def putData(self):
1545 1512 raise NotImplementedError
1546 1513
1547 1514
1548 1515 def getProcessFlags(self):
1549 1516
1550 1517 processFlags = 0
1551 1518
1552 1519 dtype_index = get_dtype_index(self.dtype)
1553 1520 procflag_dtype = get_procflag_dtype(dtype_index)
1554 1521
1555 1522 processFlags += procflag_dtype
1556 1523
1557 1524 if self.dataOut.flagDecodeData:
1558 1525 processFlags += PROCFLAG.DECODE_DATA
1559 1526
1560 1527 if self.dataOut.flagDeflipData:
1561 1528 processFlags += PROCFLAG.DEFLIP_DATA
1562 1529
1563 1530 if self.dataOut.code is not None:
1564 1531 processFlags += PROCFLAG.DEFINE_PROCESS_CODE
1565 1532
1566 1533 if self.dataOut.nCohInt > 1:
1567 1534 processFlags += PROCFLAG.COHERENT_INTEGRATION
1568 1535
1569 1536 if self.dataOut.type == "Spectra":
1570 1537 if self.dataOut.nIncohInt > 1:
1571 1538 processFlags += PROCFLAG.INCOHERENT_INTEGRATION
1572 1539
1573 1540 if self.dataOut.data_dc is not None:
1574 1541 processFlags += PROCFLAG.SAVE_CHANNELS_DC
1575 1542
1576 1543 if self.dataOut.flagShiftFFT:
1577 1544 processFlags += PROCFLAG.SHIFT_FFT_DATA
1578 1545
1579 1546 return processFlags
1580 1547
1581 1548 def setBasicHeader(self):
1582 1549
1583 1550 self.basicHeaderObj.size = self.basicHeaderSize #bytes
1584 1551 self.basicHeaderObj.version = self.versionFile
1585 1552 self.basicHeaderObj.dataBlock = self.nTotalBlocks
1586 1553
1587 1554 utc = numpy.floor(self.dataOut.utctime)
1588 1555 milisecond = (self.dataOut.utctime - utc)* 1000.0
1589 1556
1590 1557 self.basicHeaderObj.utc = utc
1591 1558 self.basicHeaderObj.miliSecond = milisecond
1592 1559 self.basicHeaderObj.timeZone = self.dataOut.timeZone
1593 1560 self.basicHeaderObj.dstFlag = self.dataOut.dstFlag
1594 1561 self.basicHeaderObj.errorCount = self.dataOut.errorCount
1595 1562
1596 1563 def setFirstHeader(self):
1597 1564 """
1598 1565 Obtiene una copia del First Header
1599 1566
1600 1567 Affected:
1601 1568
1602 1569 self.basicHeaderObj
1603 1570 self.systemHeaderObj
1604 1571 self.radarControllerHeaderObj
1605 1572 self.processingHeaderObj self.
1606 1573
1607 1574 Return:
1608 1575 None
1609 1576 """
1610 1577
1611 1578 raise NotImplementedError
1612 1579
1613 1580 def __writeFirstHeader(self):
1614 1581 """
1615 1582 Escribe el primer header del file es decir el Basic header y el Long header (SystemHeader, RadarControllerHeader, ProcessingHeader)
1616 1583
1617 1584 Affected:
1618 1585 __dataType
1619 1586
1620 1587 Return:
1621 1588 None
1622 1589 """
1623 1590
1624 1591 # CALCULAR PARAMETROS
1625 1592
1626 1593 sizeLongHeader = self.systemHeaderObj.size + self.radarControllerHeaderObj.size + self.processingHeaderObj.size
1627 1594 self.basicHeaderObj.size = self.basicHeaderSize + sizeLongHeader
1628 1595
1629 1596 self.basicHeaderObj.write(self.fp)
1630 1597 self.systemHeaderObj.write(self.fp)
1631 1598 self.radarControllerHeaderObj.write(self.fp)
1632 1599 self.processingHeaderObj.write(self.fp)
1633 1600
1634 1601 def __setNewBlock(self):
1635 1602 """
1636 1603 Si es un nuevo file escribe el First Header caso contrario escribe solo el Basic Header
1637 1604
1638 1605 Return:
1639 1606 0 : si no pudo escribir nada
1640 1607 1 : Si escribio el Basic el First Header
1641 1608 """
1642 1609 if self.fp == None:
1643 1610 self.setNextFile()
1644 1611
1645 1612 if self.flagIsNewFile:
1646 1613 return 1
1647 1614
1648 1615 if self.blockIndex < self.processingHeaderObj.dataBlocksPerFile:
1649 1616 self.basicHeaderObj.write(self.fp)
1650 1617 return 1
1651 1618
1652 1619 if not( self.setNextFile() ):
1653 1620 return 0
1654 1621
1655 1622 return 1
1656 1623
1657 1624
1658 1625 def writeNextBlock(self):
1659 1626 """
1660 1627 Selecciona el bloque siguiente de datos y los escribe en un file
1661 1628
1662 1629 Return:
1663 1630 0 : Si no hizo pudo escribir el bloque de datos
1664 1631 1 : Si no pudo escribir el bloque de datos
1665 1632 """
1666 1633 if not( self.__setNewBlock() ):
1667 1634 return 0
1668 1635
1669 1636 self.writeBlock()
1670 1637
1671 1638 print "[Writing] Block No. %d/%d" %(self.blockIndex,
1672 1639 self.processingHeaderObj.dataBlocksPerFile)
1673 1640
1674 1641 return 1
1675 1642
1676 1643 def setNextFile(self):
1677 1644 """
1678 1645 Determina el siguiente file que sera escrito
1679 1646
1680 1647 Affected:
1681 1648 self.filename
1682 1649 self.subfolder
1683 1650 self.fp
1684 1651 self.setFile
1685 1652 self.flagIsNewFile
1686 1653
1687 1654 Return:
1688 1655 0 : Si el archivo no puede ser escrito
1689 1656 1 : Si el archivo esta listo para ser escrito
1690 1657 """
1691 1658 ext = self.ext
1692 1659 path = self.path
1693 1660
1694 1661 if self.fp != None:
1695 1662 self.fp.close()
1696 1663
1697 1664 timeTuple = time.localtime( self.dataOut.utctime)
1698 1665 subfolder = 'd%4.4d%3.3d' % (timeTuple.tm_year,timeTuple.tm_yday)
1699 1666
1700 1667 fullpath = os.path.join( path, subfolder )
1701 1668 setFile = self.setFile
1702 1669
1703 1670 if not( os.path.exists(fullpath) ):
1704 1671 os.mkdir(fullpath)
1705 1672 setFile = -1 #inicializo mi contador de seteo
1706 1673 else:
1707 1674 filesList = os.listdir( fullpath )
1708 1675 if len( filesList ) > 0:
1709 1676 filesList = sorted( filesList, key=str.lower )
1710 1677 filen = filesList[-1]
1711 1678 # el filename debera tener el siguiente formato
1712 1679 # 0 1234 567 89A BCDE (hex)
1713 1680 # x YYYY DDD SSS .ext
1714 1681 if isNumber( filen[8:11] ):
1715 1682 setFile = int( filen[8:11] ) #inicializo mi contador de seteo al seteo del ultimo file
1716 1683 else:
1717 1684 setFile = -1
1718 1685 else:
1719 1686 setFile = -1 #inicializo mi contador de seteo
1720 1687
1721 1688 setFile += 1
1722 1689
1723 1690 #If this is a new day it resets some values
1724 1691 if self.dataOut.datatime.date() > self.fileDate:
1725 1692 setFile = 0
1726 1693 self.nTotalBlocks = 0
1727 1694
1728 1695 filen = '%s%4.4d%3.3d%3.3d%s' % (self.optchar, timeTuple.tm_year, timeTuple.tm_yday, setFile, ext )
1729 1696
1730 1697 filename = os.path.join( path, subfolder, filen )
1731 1698
1732 1699 fp = open( filename,'wb' )
1733 1700
1734 1701 self.blockIndex = 0
1735 1702
1736 1703 #guardando atributos
1737 1704 self.filename = filename
1738 1705 self.subfolder = subfolder
1739 1706 self.fp = fp
1740 1707 self.setFile = setFile
1741 1708 self.flagIsNewFile = 1
1742 1709 self.fileDate = self.dataOut.datatime.date()
1743 1710
1744 1711 self.setFirstHeader()
1745 1712
1746 1713 print '[Writing] Opening file: %s'%self.filename
1747 1714
1748 1715 self.__writeFirstHeader()
1749 1716
1750 1717 return 1
1751 1718
1752 1719 def setup(self, dataOut, path, blocksPerFile, profilesPerBlock=64, set=None, ext=None, datatype=4):
1753 1720 """
1754 1721 Setea el tipo de formato en la cual sera guardada la data y escribe el First Header
1755 1722
1756 1723 Inputs:
1757 1724 path : directory where data will be saved
1758 1725 profilesPerBlock : number of profiles per block
1759 1726 set : initial file set
1760 1727 datatype : An integer number that defines data type:
1761 1728 0 : int8 (1 byte)
1762 1729 1 : int16 (2 bytes)
1763 1730 2 : int32 (4 bytes)
1764 1731 3 : int64 (8 bytes)
1765 1732 4 : float32 (4 bytes)
1766 1733 5 : double64 (8 bytes)
1767 1734
1768 1735 Return:
1769 1736 0 : Si no realizo un buen seteo
1770 1737 1 : Si realizo un buen seteo
1771 1738 """
1772 1739
1773 1740 if ext == None:
1774 1741 ext = self.ext
1775 1742
1776 1743 self.ext = ext.lower()
1777 1744
1778 1745 self.path = path
1779 1746
1780 1747 if set is None:
1781 1748 self.setFile = -1
1782 1749 else:
1783 1750 self.setFile = set - 1
1784 1751
1785 1752 self.blocksPerFile = blocksPerFile
1786 1753
1787 1754 self.profilesPerBlock = profilesPerBlock
1788 1755
1789 1756 self.dataOut = dataOut
1790 1757 self.fileDate = self.dataOut.datatime.date()
1791 1758 #By default
1792 1759 self.dtype = self.dataOut.dtype
1793 1760
1794 1761 if datatype is not None:
1795 1762 self.dtype = get_numpy_dtype(datatype)
1796 1763
1797 1764 if not(self.setNextFile()):
1798 1765 print "[Writing] There isn't a next file"
1799 1766 return 0
1800 1767
1801 1768 self.setBlockDimension()
1802 1769
1803 1770 return 1
1804 1771
1805 1772 def run(self, dataOut, path, blocksPerFile, profilesPerBlock=64, set=None, ext=None, datatype=4, **kwargs):
1806 1773
1807 1774 if not(self.isConfig):
1808 1775
1809 1776 self.setup(dataOut, path, blocksPerFile, profilesPerBlock=profilesPerBlock, set=set, ext=ext, datatype=datatype, **kwargs)
1810 1777 self.isConfig = True
1811 1778
1812 1779 self.putData()
Show file before | Show file after | Hide comments
@@ -1,729 +1,751
1 1
2 2 '''
3 3 Created on Jul 3, 2014
4 4
5 5 @author: roj-idl71
6 6 '''
7 7 # SUBCHANNELS EN VEZ DE CHANNELS
8 8 # BENCHMARKS -> PROBLEMAS CON ARCHIVOS GRANDES -> INCONSTANTE EN EL TIEMPO
9 9 # ACTUALIZACION DE VERSION
10 10 # HEADERS
11 11 # MODULO DE ESCRITURA
12 12 # METADATA
13 13
14 14 import os
15 15 import datetime
16 16 import numpy
17 17 import timeit
18 18 from profilehooks import coverage, profile
19 19 from fractions import Fraction
20 20
21 21 try:
22 22 from gevent import sleep
23 23 except:
24 24 from time import sleep
25 25
26 26 from schainpy.model.data.jroheaderIO import RadarControllerHeader, SystemHeader
27 27 from schainpy.model.data.jrodata import Voltage
28 28 from schainpy.model.proc.jroproc_base import ProcessingUnit, Operation
29 29 from time import time
30 30
31 31 import cPickle
32 32 try:
33 33 import digital_rf
34 34 except:
35 35 print 'You should install "digital_rf" module if you want to read Digital RF data'
36 36
37 37 class DigitalRFReader(ProcessingUnit):
38 38 '''
39 39 classdocs
40 40 '''
41 41
42 42 def __init__(self, **kwargs):
43 43 '''
44 44 Constructor
45 45 '''
46 46
47 47 ProcessingUnit.__init__(self, **kwargs)
48 48
49 49 self.dataOut = Voltage()
50 50 self.__printInfo = True
51 51 self.__flagDiscontinuousBlock = False
52 52 self.__bufferIndex = 9999999
53 53 self.__ippKm = None
54 54 self.__codeType = 0
55 55 self.__nCode = None
56 56 self.__nBaud = None
57 57 self.__code = None
58 self.dtype = None
58 59
59 60 def close(self):
60 61 print 'Average of writing to digital rf format is ', self.oldAverage * 1000
61 62 return
62 63
63 64 def __getCurrentSecond(self):
64 65
65 66 return self.__thisUnixSample/self.__sample_rate
66 67
67 68 thisSecond = property(__getCurrentSecond, "I'm the 'thisSecond' property.")
68 69
69 70 def __setFileHeader(self):
70 71 '''
71 72 In this method will be initialized every parameter of dataOut object (header, no data)
72 73 '''
73 74 ippSeconds = 1.0*self.__nSamples/self.__sample_rate
74 75
75 76 nProfiles = 1.0/ippSeconds # Number of profiles in one second
76
77 self.dataOut.radarControllerHeaderObj = RadarControllerHeader(self.__radarControllerHeader)
78
79 self.dataOut.systemHeaderObj = SystemHeader(self.__systemHeader)
80
77
78 try:
79 self.dataOut.radarControllerHeaderObj = RadarControllerHeader(self.__radarControllerHeader)
80 except:
81 self.dataOut.radarControllerHeaderObj = RadarControllerHeader(
82 txA=0,
83 txB=0,
84 nWindows=1,
85 nHeights=self.__nSamples,
86 firstHeight=self.__firstHeigth,
87 deltaHeight=self.__deltaHeigth,
88 codeType=self.__codeType,
89 nCode=self.__nCode, nBaud=self.__nBaud,
90 code = self.__code)
91
92 try:
93 self.dataOut.systemHeaderObj = SystemHeader(self.__systemHeader)
94 except:
95 self.dataOut.systemHeaderObj = SystemHeader(nSamples=self.__nSamples,
96 nProfiles=nProfiles,
97 nChannels=len(self.__channelList),
98 adcResolution=14)
81 99 self.dataOut.type = "Voltage"
82
100
83 101 self.dataOut.data = None
84 102
85 103 self.dataOut.dtype = self.dtype
86 104
87 105 # self.dataOut.nChannels = 0
88 106
89 107 # self.dataOut.nHeights = 0
90 108
91 109 self.dataOut.nProfiles = nProfiles
92 110
93 111 self.dataOut.heightList = self.__firstHeigth + numpy.arange(self.__nSamples, dtype = numpy.float)*self.__deltaHeigth
94 112
95 113 self.dataOut.channelList = range(self.__num_subchannels)
96 114
97 115 self.dataOut.blocksize = self.dataOut.getNChannels() * self.dataOut.getNHeights()
98 116
99 117 # self.dataOut.channelIndexList = None
100 118
101 119 self.dataOut.flagNoData = True
102 120
103 121 self.dataOut.flagDataAsBlock = False
104 122 # Set to TRUE if the data is discontinuous
105 123 self.dataOut.flagDiscontinuousBlock = False
106 124
107 125 self.dataOut.utctime = None
108 126
109 127 self.dataOut.timeZone = self.__timezone/60 # timezone like jroheader, difference in minutes between UTC and localtime
110 128
111 129 self.dataOut.dstFlag = 0
112 130
113 131 self.dataOut.errorCount = 0
114 132
115 self.dataOut.nCohInt = self.fixed_metadata_dict['nCohInt']
133 try:
134 self.dataOut.nCohInt = self.fixed_metadata_dict.get('nCohInt', 1)
116 135
117 self.dataOut.flagDecodeData = self.fixed_metadata_dict['flagDecodeData'] # asumo que la data esta decodificada
136 self.dataOut.flagDecodeData = self.fixed_metadata_dict['flagDecodeData'] # asumo que la data esta decodificada
118 137
119 self.dataOut.flagDeflipData = self.fixed_metadata_dict['flagDeflipData'] # asumo que la data esta sin flip
138 self.dataOut.flagDeflipData = self.fixed_metadata_dict['flagDeflipData'] # asumo que la data esta sin flip
120 139
121 self.dataOut.flagShiftFFT = self.fixed_metadata_dict['flagShiftFFT']
140 self.dataOut.flagShiftFFT = self.fixed_metadata_dict['flagShiftFFT']
122 141
123 self.dataOut.useLocalTime = self.fixed_metadata_dict['useLocalTime']
142 self.dataOut.useLocalTime = self.fixed_metadata_dict['useLocalTime']
143 except:
144 pass
145
124 146
125 147 self.dataOut.ippSeconds = ippSeconds
126 148
127 149 # Time interval between profiles
128 150 # self.dataOut.timeInterval = self.dataOut.ippSeconds * self.dataOut.nCohInt
129 151
130 152 self.dataOut.frequency = self.__frequency
131 153
132 154 self.dataOut.realtime = self.__online
133 155
134 156 def findDatafiles(self, path, startDate=None, endDate=None):
135 157
136 158 if not os.path.isdir(path):
137 159 return []
138 160
139 161 try:
140 162 digitalReadObj = digital_rf.DigitalRFReader(path, load_all_metadata=True)
141 163 except:
142 164 digitalReadObj = digital_rf.DigitalRFReader(path)
143 165
144 166 channelNameList = digitalReadObj.get_channels()
145 167
146 168 if not channelNameList:
147 169 return []
148 170
149 171 metadata_dict = digitalReadObj.get_rf_file_metadata(channelNameList[0])
150 172
151 173 sample_rate = metadata_dict['sample_rate'][0]
152 174
153 175 this_metadata_file = digitalReadObj.get_metadata(channelNameList[0])
154 176
155 177 try:
156 178 timezone = this_metadata_file['timezone'].value
157 179 except:
158 180 timezone = 0
159 181
160 182 startUTCSecond, endUTCSecond = digitalReadObj.get_bounds(channelNameList[0])/sample_rate - timezone
161 183
162 184 startDatetime = datetime.datetime.utcfromtimestamp(startUTCSecond)
163 185 endDatatime = datetime.datetime.utcfromtimestamp(endUTCSecond)
164 186
165 187 if not startDate:
166 188 startDate = startDatetime.date()
167 189
168 190 if not endDate:
169 191 endDate = endDatatime.date()
170 192
171 193 dateList = []
172 194
173 195 thisDatetime = startDatetime
174 196
175 197 while(thisDatetime<=endDatatime):
176 198
177 199 thisDate = thisDatetime.date()
178 200
179 201 if thisDate < startDate:
180 202 continue
181 203
182 204 if thisDate > endDate:
183 205 break
184 206
185 207 dateList.append(thisDate)
186 208 thisDatetime += datetime.timedelta(1)
187 209
188 210 return dateList
189 211
190 212 def setup(self, path = None,
191 213 startDate = None,
192 214 endDate = None,
193 215 startTime = datetime.time(0,0,0),
194 216 endTime = datetime.time(23,59,59),
195 217 channelList = None,
196 218 nSamples = None,
197 219 online = False,
198 220 delay = 60,
199 221 buffer_size = 1024,
200 222 ippKm=None,
201 223 **kwargs):
202 224 '''
203 225 In this method we should set all initial parameters.
204 226
205 227 Inputs:
206 228 path
207 229 startDate
208 230 endDate
209 231 startTime
210 232 endTime
211 233 set
212 234 expLabel
213 235 ext
214 236 online
215 237 delay
216 238 '''
217 239 self.i = 0
218 240 if not os.path.isdir(path):
219 241 raise ValueError, "[Reading] Directory %s does not exist" %path
220 242
221 243 try:
222 244 self.digitalReadObj = digital_rf.DigitalRFReader(path, load_all_metadata=True)
223 245 except:
224 246 self.digitalReadObj = digital_rf.DigitalRFReader(path)
225 247
226 248 channelNameList = self.digitalReadObj.get_channels()
227 249
228 250 if not channelNameList:
229 251 raise ValueError, "[Reading] Directory %s does not have any files" %path
230 252
231 253 if not channelList:
232 254 channelList = range(len(channelNameList))
233 255
234 256
235 257 ########## Reading metadata ######################
236 258
237 259 top_properties = self.digitalReadObj.get_properties(channelNameList[channelList[0]])
238 260
239 261
240
241
242
243
244 262 self.__num_subchannels = top_properties['num_subchannels']
245 263 self.__sample_rate = 1.0 * top_properties['sample_rate_numerator'] / top_properties['sample_rate_denominator']
246
247 264 # self.__samples_per_file = top_properties['samples_per_file'][0]
248 265 self.__deltaHeigth = 1e6*0.15/self.__sample_rate ## why 0.15?
249 266
250 267 this_metadata_file = self.digitalReadObj.get_digital_metadata(channelNameList[channelList[0]])
251 268 metadata_bounds = this_metadata_file.get_bounds()
252 269 self.fixed_metadata_dict = this_metadata_file.read(metadata_bounds[0])[metadata_bounds[0]] ## GET FIRST HEADER
253 self.__processingHeader = self.fixed_metadata_dict['processingHeader']
254 self.__radarControllerHeader = self.fixed_metadata_dict['radarControllerHeader']
255 self.__systemHeader = self.fixed_metadata_dict['systemHeader']
256 self.dtype = cPickle.loads(self.fixed_metadata_dict['dtype'])
270
271 try:
272 self.__processingHeader = self.fixed_metadata_dict['processingHeader']
273 self.__radarControllerHeader = self.fixed_metadata_dict['radarControllerHeader']
274 self.__systemHeader = self.fixed_metadata_dict['systemHeader']
275 self.dtype = cPickle.loads(self.fixed_metadata_dict['dtype'])
276 except:
277 pass
278
257 279
258 280 self.__frequency = None
259 281
260 try:
261 self.__frequency = self.fixed_metadata_dict['frequency']
262 except:
263 self.__frequency = None
282 self.__frequency = self.fixed_metadata_dict.get('frequency', 1)
264 283
265 try:
266 self.__timezone = self.fixed_metadata_dict['timezone'] * 60
267 except:
268 self.__timezone = 0
284 self.__timezone = self.fixed_metadata_dict.get('timezone', 300)
269 285
270 286
271 287 try:
272 288 nSamples = self.fixed_metadata_dict['nSamples']
273 289 except:
274 290 nSamples = None
275 291
276 292 self.__firstHeigth = 0
277 293
278 294 try:
279 295 codeType = self.__radarControllerHeader['codeType']
280 296 except:
281 297 codeType = 0
282 298
283 299 nCode = 1
284 300 nBaud = 1
285 301 code = numpy.ones((nCode, nBaud), dtype=numpy.int)
286 302
287 if codeType:
288 nCode = self.__radarControllerHeader['nCode']
289 nBaud = self.__radarControllerHeader['nBaud']
290 code = self.__radarControllerHeader['code']
291
303 try:
304 if codeType:
305 nCode = self.__radarControllerHeader['nCode']
306 nBaud = self.__radarControllerHeader['nBaud']
307 code = self.__radarControllerHeader['code']
308 except:
309 pass
310
311
292 312 if not ippKm:
293 313 try:
294 314 # seconds to km
295 315 ippKm = self.__radarControllerHeader['ipp']
296 316 except:
297 317 ippKm = None
298 318 ####################################################
319 self.__ippKm = ippKm
299 320 startUTCSecond = None
300 321 endUTCSecond = None
301 322
302 323 if startDate:
303 324 startDatetime = datetime.datetime.combine(startDate, startTime)
304 325 startUTCSecond = (startDatetime-datetime.datetime(1970,1,1)).total_seconds() + self.__timezone
305 326
306 327 if endDate:
307 328 endDatetime = datetime.datetime.combine(endDate, endTime)
308 329 endUTCSecond = (endDatetime-datetime.datetime(1970,1,1)).total_seconds() + self.__timezone
309 330
310 331 start_index, end_index = self.digitalReadObj.get_bounds(channelNameList[channelList[0]])
311 332
312 333 if not startUTCSecond:
313 334 startUTCSecond = start_index/self.__sample_rate
314 335
315 336 if start_index > startUTCSecond*self.__sample_rate:
316 337 startUTCSecond = start_index/self.__sample_rate
317 338
318 339 if not endUTCSecond:
319 340 endUTCSecond = end_index/self.__sample_rate
320 341
321 342 if end_index < endUTCSecond*self.__sample_rate:
322 343 endUTCSecond = end_index/self.__sample_rate
323 344 if not nSamples:
324 345 if not ippKm:
325 346 raise ValueError, "[Reading] nSamples or ippKm should be defined"
326 347 nSamples = int(ippKm / (1e6*0.15/self.__sample_rate))
327 348 channelBoundList = []
328 349 channelNameListFiltered = []
329 350
330 351 for thisIndexChannel in channelList:
331 352 thisChannelName = channelNameList[thisIndexChannel]
332 353 start_index, end_index = self.digitalReadObj.get_bounds(thisChannelName)
333 354 channelBoundList.append((start_index, end_index))
334 355 channelNameListFiltered.append(thisChannelName)
335 356
336 357 self.profileIndex = 0
337 358 self.i= 0
338 359 self.__delay = delay
339 self.__ippKm = ippKm
360
340 361 self.__codeType = codeType
341 362 self.__nCode = nCode
342 363 self.__nBaud = nBaud
343 364 self.__code = code
344 365
345 366 self.__datapath = path
346 367 self.__online = online
347 368 self.__channelList = channelList
348 369 self.__channelNameList = channelNameListFiltered
349 370 self.__channelBoundList = channelBoundList
350 371 self.__nSamples = nSamples
351 372 self.__samples_to_read = long(nSamples) # FIJO: AHORA 40
352 373 self.__nChannels = len(self.__channelList)
353 374
354 375 self.__startUTCSecond = startUTCSecond
355 376 self.__endUTCSecond = endUTCSecond
356 377
357 378 self.__timeInterval = 1.0 * self.__samples_to_read/self.__sample_rate # Time interval
358 379
359 380 if online:
360 381 # self.__thisUnixSample = int(endUTCSecond*self.__sample_rate - 4*self.__samples_to_read)
361 382 startUTCSecond = numpy.floor(endUTCSecond)
362 383
363 384 self.__thisUnixSample = long(startUTCSecond*self.__sample_rate) - self.__samples_to_read ## por que en el otro metodo lo primero q se hace es sumar samplestoread
364 385
365 386 self.__data_buffer = numpy.zeros((self.__num_subchannels, self.__samples_to_read), dtype = numpy.complex)
366 387
367 388 self.__setFileHeader()
368 389 self.isConfig = True
369 390
370 391 print "[Reading] Digital RF Data was found from %s to %s " %(
371 392 datetime.datetime.utcfromtimestamp(self.__startUTCSecond - self.__timezone),
372 393 datetime.datetime.utcfromtimestamp(self.__endUTCSecond - self.__timezone)
373 394 )
374 395
375 396 print "[Reading] Starting process from %s to %s" %(datetime.datetime.utcfromtimestamp(startUTCSecond - self.__timezone),
376 397 datetime.datetime.utcfromtimestamp(endUTCSecond - self.__timezone)
377 398 )
378 399 self.oldAverage = None
379 400 self.count = 0
380 401 self.executionTime = 0
381 402 def __reload(self):
382 403 # print
383 404 # print "%s not in range [%s, %s]" %(
384 405 # datetime.datetime.utcfromtimestamp(self.thisSecond - self.__timezone),
385 406 # datetime.datetime.utcfromtimestamp(self.__startUTCSecond - self.__timezone),
386 407 # datetime.datetime.utcfromtimestamp(self.__endUTCSecond - self.__timezone)
387 408 # )
388 409 print "[Reading] reloading metadata ..."
389 410
390 411 try:
391 412 self.digitalReadObj.reload(complete_update=True)
392 413 except:
393 414 self.digitalReadObj.reload()
394 415
395 416 start_index, end_index = self.digitalReadObj.get_bounds(self.__channelNameList[self.__channelList[0]])
396 417
397 418 if start_index > self.__startUTCSecond*self.__sample_rate:
398 419 self.__startUTCSecond = 1.0*start_index/self.__sample_rate
399 420
400 421 if end_index > self.__endUTCSecond*self.__sample_rate:
401 422 self.__endUTCSecond = 1.0*end_index/self.__sample_rate
402 423 print
403 424 print "[Reading] New timerange found [%s, %s] " %(
404 425 datetime.datetime.utcfromtimestamp(self.__startUTCSecond - self.__timezone),
405 426 datetime.datetime.utcfromtimestamp(self.__endUTCSecond - self.__timezone)
406 427 )
407 428
408 429 return True
409 430
410 431 return False
411 432
412 433 def timeit(self, toExecute):
413 434 t0 = time()
414 435 toExecute()
415 436 self.executionTime = time() - t0
416 437 if self.oldAverage is None: self.oldAverage = self.executionTime
417 438 self.oldAverage = (self.executionTime + self.count*self.oldAverage) / (self.count + 1.0)
418 439 self.count = self.count + 1.0
419 440 return
420 441
421 442 def __readNextBlock(self, seconds=30, volt_scale = 1):
422 443 '''
423 444 '''
424 445
425 446 # Set the next data
426 447 self.__flagDiscontinuousBlock = False
427 448 self.__thisUnixSample += self.__samples_to_read
428 449
429 450 if self.__thisUnixSample + 2*self.__samples_to_read > self.__endUTCSecond*self.__sample_rate:
430 451 print "[Reading] There are no more data into selected time-range"
431 452 if self.__online:
432 453 self.__reload()
433 454 else:
434 455 return False
435 456
436 457 if self.__thisUnixSample + 2*self.__samples_to_read > self.__endUTCSecond*self.__sample_rate:
437 458 return False
438 459 self.__thisUnixSample -= self.__samples_to_read
439 460
440 461 indexChannel = 0
441 462
442 463 dataOk = False
443 464 for thisChannelName in self.__channelNameList: ##TODO VARIOS CHANNELS?
444 465 for indexSubchannel in range(self.__num_subchannels):
445 466 try:
446 467 t0 = time()
447 468 result = self.digitalReadObj.read_vector_c81d(self.__thisUnixSample,
448 469 self.__samples_to_read,
449 470 thisChannelName, sub_channel=indexSubchannel)
450 471 self.executionTime = time() - t0
451 472 if self.oldAverage is None: self.oldAverage = self.executionTime
452 473 self.oldAverage = (self.executionTime + self.count*self.oldAverage) / (self.count + 1.0)
453 474 self.count = self.count + 1.0
454 475
455 476 except IOError, e:
456 477 #read next profile
457 478 self.__flagDiscontinuousBlock = True
458 479 print "[Reading] %s" %datetime.datetime.utcfromtimestamp(self.thisSecond - self.__timezone), e
459 480 break
460 481
461 482 if result.shape[0] != self.__samples_to_read:
462 483 self.__flagDiscontinuousBlock = True
463 484 print "[Reading] %s: Too few samples were found, just %d/%d samples" %(datetime.datetime.utcfromtimestamp(self.thisSecond - self.__timezone),
464 485 result.shape[0],
465 486 self.__samples_to_read)
466 487 break
467 488
468 489 self.__data_buffer[indexSubchannel,:] = result*volt_scale
469 490
470 491 indexChannel += 1
471 492
472 493 dataOk = True
473 494
474 495 self.__utctime = self.__thisUnixSample/self.__sample_rate
475 496
476 497 if not dataOk:
477 498 return False
478 499
479 500 print "[Reading] %s: %d samples <> %f sec" %(datetime.datetime.utcfromtimestamp(self.thisSecond - self.__timezone),
480 501 self.__samples_to_read,
481 502 self.__timeInterval)
482 503
483 504 self.__bufferIndex = 0
484 505
485 506 return True
486 507
487 508 def __isBufferEmpty(self):
488 509 return self.__bufferIndex > self.__samples_to_read - self.__nSamples #40960 - 40
489 510
490 511 def getData(self, seconds=30, nTries=5):
491 512
492 513 '''
493 514 This method gets the data from files and put the data into the dataOut object
494 515
495 516 In addition, increase el the buffer counter in one.
496 517
497 518 Return:
498 519 data : retorna un perfil de voltages (alturas * canales) copiados desde el
499 520 buffer. Si no hay mas archivos a leer retorna None.
500 521
501 522 Affected:
502 523 self.dataOut
503 524 self.profileIndex
504 525 self.flagDiscontinuousBlock
505 526 self.flagIsNewBlock
506 527 '''
507 528
508 529 err_counter = 0
509 530 self.dataOut.flagNoData = True
510 531
511 532 if self.__isBufferEmpty():
512 533 self.__flagDiscontinuousBlock = False
513 534
514 535 while True:
515 536 if self.__readNextBlock():
516 537 break
517 538 if self.__thisUnixSample > self.__endUTCSecond*self.__sample_rate:
518 539 return False
519 540
520 541 if self.__flagDiscontinuousBlock:
521 542 print '[Reading] discontinuous block found ... continue with the next block'
522 543 continue
523 544
524 545 if not self.__online:
525 546 return False
526 547
527 548 err_counter += 1
528 549 if err_counter > nTries:
529 550 return False
530 551
531 552 print '[Reading] waiting %d seconds to read a new block' %seconds
532 553 sleep(seconds)
533 554
534 555 self.dataOut.data = self.__data_buffer[:,self.__bufferIndex:self.__bufferIndex+self.__nSamples]
535 556 self.dataOut.utctime = (self.__thisUnixSample + self.__bufferIndex)/self.__sample_rate
536 557 self.dataOut.flagNoData = False
537 558 self.dataOut.flagDiscontinuousBlock = self.__flagDiscontinuousBlock
538 559 self.dataOut.profileIndex = self.profileIndex
539 560
540 561 self.__bufferIndex += self.__nSamples
541 562 self.profileIndex += 1
542 563
543 564 if self.profileIndex == self.dataOut.nProfiles:
544 565 self.profileIndex = 0
545 566
546 567 return True
547 568
548 569 def printInfo(self):
549 570 '''
550 571 '''
551 572 if self.__printInfo == False:
552 573 return
553 574
554 575 # self.systemHeaderObj.printInfo()
555 576 # self.radarControllerHeaderObj.printInfo()
556 577
557 578 self.__printInfo = False
558 579
559 580 def printNumberOfBlock(self):
560 581 '''
561 582 '''
562 583 return
563 584 # print self.profileIndex
564 585
565 586
566 587 def run(self, **kwargs):
567 588 '''
568 589 This method will be called many times so here you should put all your code
569 590 '''
570 591
571 592 if not self.isConfig:
572 593 self.setup(**kwargs)
573 594 #self.i = self.i+1
574 595 self.getData(seconds=self.__delay)
575 596
576 597 return
577 598
578 599 class DigitalRFWriter(Operation):
579 600 '''
580 601 classdocs
581 602 '''
582 603
583 604 def __init__(self, **kwargs):
584 605 '''
585 606 Constructor
586 607 '''
587 608 Operation.__init__(self, **kwargs)
588 609 self.metadata_dict = {}
589 610 self.dataOut = None
611 self.dtype = None
590 612
591 613 def setHeader(self):
592 614
593 615 self.metadata_dict['frequency'] = self.dataOut.frequency
594 616 self.metadata_dict['timezone'] = self.dataOut.timeZone
595 617 self.metadata_dict['dtype'] = cPickle.dumps(self.dataOut.dtype)
596 618 self.metadata_dict['nProfiles'] = self.dataOut.nProfiles
597 619 self.metadata_dict['heightList'] = self.dataOut.heightList
598 620 self.metadata_dict['channelList'] = self.dataOut.channelList
599 621 self.metadata_dict['flagDecodeData'] = self.dataOut.flagDecodeData
600 622 self.metadata_dict['flagDeflipData'] = self.dataOut.flagDeflipData
601 623 self.metadata_dict['flagShiftFFT'] = self.dataOut.flagShiftFFT
602 624 self.metadata_dict['flagDataAsBlock'] = self.dataOut.flagDataAsBlock
603 625 self.metadata_dict['useLocalTime'] = self.dataOut.useLocalTime
604 626 self.metadata_dict['nCohInt'] = self.dataOut.nCohInt
605 627
606 628 return
607 629
608 630 def setup(self, dataOut, path, frequency, fileCadence, dirCadence, metadataCadence, set=0, metadataFile='metadata', ext='.h5'):
609 631 '''
610 632 In this method we should set all initial parameters.
611 633 Input:
612 634 dataOut: Input data will also be outputa data
613 635 '''
614 636 self.setHeader()
615 637 self.__ippSeconds = dataOut.ippSeconds
616 638 self.__deltaH = dataOut.getDeltaH()
617 639 self.__sample_rate = 1e6*0.15/self.__deltaH
618 640 self.__dtype = dataOut.dtype
619 641 if len(dataOut.dtype) == 2:
620 642 self.__dtype = dataOut.dtype[0]
621 643 self.__nSamples = dataOut.systemHeaderObj.nSamples
622 644 self.__nProfiles = dataOut.nProfiles
623 645 self.__blocks_per_file = dataOut.processingHeaderObj.dataBlocksPerFile
624 646
625 647 self.arr_data = arr_data = numpy.ones((self.__nSamples, len(self.dataOut.channelList)), dtype=[('r', self.__dtype), ('i', self.__dtype)])
626 648
627 649 file_cadence_millisecs = long(1.0 * self.__blocks_per_file * self.__nProfiles * self.__nSamples / self.__sample_rate) * 1000
628 650 sub_cadence_secs = file_cadence_millisecs / 500
629 651
630 652 sample_rate_fraction = Fraction(self.__sample_rate).limit_denominator()
631 653 sample_rate_numerator = long(sample_rate_fraction.numerator)
632 654 sample_rate_denominator = long(sample_rate_fraction.denominator)
633 655 start_global_index = dataOut.utctime * self.__sample_rate
634 656
635 657 uuid = 'prueba'
636 658 compression_level = 1
637 659 checksum = False
638 660 is_complex = True
639 661 num_subchannels = len(dataOut.channelList)
640 662 is_continuous = True
641 663 marching_periods = False
642 664
643 665 self.digitalWriteObj = digital_rf.DigitalRFWriter(path, self.__dtype, dirCadence,
644 666 fileCadence, start_global_index,
645 667 sample_rate_numerator, sample_rate_denominator, uuid, compression_level, checksum,
646 668 is_complex, num_subchannels, is_continuous, marching_periods)
647 669
648 670 metadata_dir = os.path.join(path, 'metadata')
649 671 os.system('mkdir %s' % (metadata_dir))
650 672
651 673 self.digitalMetadataWriteObj = digital_rf.DigitalMetadataWriter(metadata_dir, dirCadence, 1, ##236, file_cadence_millisecs / 1000
652 674 sample_rate_numerator, sample_rate_denominator,
653 675 metadataFile)
654 676
655 677
656 678 self.isConfig = True
657 679 self.currentSample = 0
658 680 self.oldAverage = 0
659 681 self.count = 0
660 682 return
661 683
662 684 def writeMetadata(self):
663 685 print '[Writing] - Writing metadata'
664 686 start_idx = self.__sample_rate * self.dataOut.utctime
665 687
666 688 self.metadata_dict['processingHeader'] = self.dataOut.processingHeaderObj.getAsDict()
667 689 self.metadata_dict['radarControllerHeader'] = self.dataOut.radarControllerHeaderObj.getAsDict()
668 690 self.metadata_dict['systemHeader'] = self.dataOut.systemHeaderObj.getAsDict()
669 691 self.digitalMetadataWriteObj.write(start_idx, self.metadata_dict)
670 692 return
671 693
672 694
673 695 def timeit(self, toExecute):
674 696 t0 = time()
675 697 toExecute()
676 698 self.executionTime = time() - t0
677 699 if self.oldAverage is None: self.oldAverage = self.executionTime
678 700 self.oldAverage = (self.executionTime + self.count*self.oldAverage) / (self.count + 1.0)
679 701 self.count = self.count + 1.0
680 702 return
681 703
682 704
683 705 def writeData(self):
684 706 for i in range(self.dataOut.systemHeaderObj.nSamples):
685 707 for channel in self.dataOut.channelList:
686 708 self.arr_data[i][channel]['r'] = self.dataOut.data[channel][i].real
687 709 self.arr_data[i][channel]['i'] = self.dataOut.data[channel][i].imag
688 710
689 711 def f(): return self.digitalWriteObj.rf_write(self.arr_data)
690 712 self.timeit(f)
691 713
692 714 return
693 715
694 716 def run(self, dataOut, frequency=49.92e6, path=None, fileCadence=100, dirCadence=25, metadataCadence=1, **kwargs):
695 717 '''
696 718 This method will be called many times so here you should put all your code
697 719 Inputs:
698 720 dataOut: object with the data
699 721 '''
700 722 # print dataOut.__dict__
701 723 self.dataOut = dataOut
702 724 if not self.isConfig:
703 725 self.setup(dataOut, path, frequency, fileCadence, dirCadence, metadataCadence, **kwargs)
704 726 self.writeMetadata()
705 727
706 728 self.writeData()
707 729
708 730 ## self.currentSample += 1
709 731 ## if self.dataOut.flagDataAsBlock or self.currentSample == 1:
710 732 ## self.writeMetadata()
711 733 ## if self.currentSample == self.__nProfiles: self.currentSample = 0
712 734
713 735 def close(self):
714 736 print '[Writing] - Closing files '
715 737 print 'Average of writing to digital rf format is ', self.oldAverage * 1000
716 738 try:
717 739 self.digitalWriteObj.close()
718 740 except:
719 741 pass
720 742
721 743 # raise
722 744 if __name__ == '__main__':
723 745
724 746 readObj = DigitalRFReader()
725 747
726 748 while True:
727 749 readObj.run(path='/home/jchavez/jicamarca/mocked_data/')
728 750 # readObj.printInfo()
729 751 # readObj.printNumberOfBlock()
Show file before | Show file after | Hide comments
@@ -1,904 +1,903
1 1 import numpy
2 2
3 3 from jroproc_base import ProcessingUnit, Operation
4 4 from schainpy.model.data.jrodata import Spectra
5 5 from schainpy.model.data.jrodata import hildebrand_sekhon
6 6
7 7 class SpectraProc(ProcessingUnit):
8 8
9 9 def __init__(self, **kwargs):
10 10
11 11 ProcessingUnit.__init__(self, **kwargs)
12 12
13 13 self.buffer = None
14 14 self.firstdatatime = None
15 15 self.profIndex = 0
16 16 self.dataOut = Spectra()
17 17 self.id_min = None
18 18 self.id_max = None
19 19
20 20 def __updateSpecFromVoltage(self):
21 21
22 22 self.dataOut.timeZone = self.dataIn.timeZone
23 23 self.dataOut.dstFlag = self.dataIn.dstFlag
24 24 self.dataOut.errorCount = self.dataIn.errorCount
25 25 self.dataOut.useLocalTime = self.dataIn.useLocalTime
26 26
27 27 self.dataOut.radarControllerHeaderObj = self.dataIn.radarControllerHeaderObj.copy()
28 28 self.dataOut.systemHeaderObj = self.dataIn.systemHeaderObj.copy()
29 29 self.dataOut.channelList = self.dataIn.channelList
30 30 self.dataOut.heightList = self.dataIn.heightList
31 31 self.dataOut.dtype = numpy.dtype([('real','<f4'),('imag','<f4')])
32 32
33 33 self.dataOut.nBaud = self.dataIn.nBaud
34 34 self.dataOut.nCode = self.dataIn.nCode
35 35 self.dataOut.code = self.dataIn.code
36 36 self.dataOut.nProfiles = self.dataOut.nFFTPoints
37 37
38 38 self.dataOut.flagDiscontinuousBlock = self.dataIn.flagDiscontinuousBlock
39 39 self.dataOut.utctime = self.firstdatatime
40 40 self.dataOut.flagDecodeData = self.dataIn.flagDecodeData #asumo q la data esta decodificada
41 41 self.dataOut.flagDeflipData = self.dataIn.flagDeflipData #asumo q la data esta sin flip
42 42 self.dataOut.flagShiftFFT = False
43 43
44 44 self.dataOut.nCohInt = self.dataIn.nCohInt
45 45 self.dataOut.nIncohInt = 1
46 46
47 47 self.dataOut.windowOfFilter = self.dataIn.windowOfFilter
48 48
49 49 self.dataOut.frequency = self.dataIn.frequency
50 50 self.dataOut.realtime = self.dataIn.realtime
51 51
52 52 self.dataOut.azimuth = self.dataIn.azimuth
53 53 self.dataOut.zenith = self.dataIn.zenith
54 54
55 55 self.dataOut.beam.codeList = self.dataIn.beam.codeList
56 56 self.dataOut.beam.azimuthList = self.dataIn.beam.azimuthList
57 57 self.dataOut.beam.zenithList = self.dataIn.beam.zenithList
58 58
59 59 def __getFft(self):
60 60 """
61 61 Convierte valores de Voltaje a Spectra
62 62
63 63 Affected:
64 64 self.dataOut.data_spc
65 65 self.dataOut.data_cspc
66 66 self.dataOut.data_dc
67 67 self.dataOut.heightList
68 68 self.profIndex
69 69 self.buffer
70 70 self.dataOut.flagNoData
71 71 """
72 72 fft_volt = numpy.fft.fft(self.buffer,n=self.dataOut.nFFTPoints,axis=1)
73 73 fft_volt = fft_volt.astype(numpy.dtype('complex'))
74 74 dc = fft_volt[:,0,:]
75 75
76 76 #calculo de self-spectra
77 77 fft_volt = numpy.fft.fftshift(fft_volt,axes=(1,))
78 78 spc = fft_volt * numpy.conjugate(fft_volt)
79 79 spc = spc.real
80 80
81 81 blocksize = 0
82 82 blocksize += dc.size
83 83 blocksize += spc.size
84 84
85 85 cspc = None
86 86 pairIndex = 0
87 87 if self.dataOut.pairsList != None:
88 88 #calculo de cross-spectra
89 89 cspc = numpy.zeros((self.dataOut.nPairs, self.dataOut.nFFTPoints, self.dataOut.nHeights), dtype='complex')
90 90 for pair in self.dataOut.pairsList:
91 91 if pair[0] not in self.dataOut.channelList:
92 92 raise ValueError, "Error getting CrossSpectra: pair 0 of %s is not in channelList = %s" %(str(pair), str(self.dataOut.channelList))
93 93 if pair[1] not in self.dataOut.channelList:
94 94 raise ValueError, "Error getting CrossSpectra: pair 1 of %s is not in channelList = %s" %(str(pair), str(self.dataOut.channelList))
95 95
96 96 cspc[pairIndex,:,:] = fft_volt[pair[0],:,:] * numpy.conjugate(fft_volt[pair[1],:,:])
97 97 pairIndex += 1
98 98 blocksize += cspc.size
99 99
100 100 self.dataOut.data_spc = spc
101 101 self.dataOut.data_cspc = cspc
102 102 self.dataOut.data_dc = dc
103 103 self.dataOut.blockSize = blocksize
104 104 self.dataOut.flagShiftFFT = True
105 105
106 106 def run(self, nProfiles=None, nFFTPoints=None, pairsList=[], ippFactor=None):
107 107
108 108 self.dataOut.flagNoData = True
109 109
110 110 if self.dataIn.type == "Spectra":
111 111 self.dataOut.copy(self.dataIn)
112 # self.__selectPairs(pairsList)
112 # self.__selectPairs(pairsList)
113 113 return True
114 114
115 115 if self.dataIn.type == "Voltage":
116 116
117 117 if nFFTPoints == None:
118 118 raise ValueError, "This SpectraProc.run() need nFFTPoints input variable"
119 119
120 120 if nProfiles == None:
121 121 nProfiles = nFFTPoints
122 122
123 123 if ippFactor == None:
124 124 ippFactor = 1
125 125
126 126 self.dataOut.ippFactor = ippFactor
127 127
128 128 self.dataOut.nFFTPoints = nFFTPoints
129 129 self.dataOut.pairsList = pairsList
130 130
131 131 if self.buffer is None:
132 132 self.buffer = numpy.zeros( (self.dataIn.nChannels,
133 133 nProfiles,
134 134 self.dataIn.nHeights),
135 135 dtype='complex')
136 136
137 137 if self.dataIn.flagDataAsBlock:
138 138 #data dimension: [nChannels, nProfiles, nSamples]
139 139 nVoltProfiles = self.dataIn.data.shape[1]
140 140 # nVoltProfiles = self.dataIn.nProfiles
141 141
142 142 if nVoltProfiles == nProfiles:
143 143 self.buffer = self.dataIn.data.copy()
144 144 self.profIndex = nVoltProfiles
145 145
146 146 elif nVoltProfiles < nProfiles:
147 147
148 148 if self.profIndex == 0:
149 149 self.id_min = 0
150 150 self.id_max = nVoltProfiles
151 151
152 152 self.buffer[:,self.id_min:self.id_max,:] = self.dataIn.data
153 153 self.profIndex += nVoltProfiles
154 154 self.id_min += nVoltProfiles
155 155 self.id_max += nVoltProfiles
156 156 else:
157 157 raise ValueError, "The type object %s has %d profiles, it should just has %d profiles"%(self.dataIn.type,self.dataIn.data.shape[1],nProfiles)
158 158 self.dataOut.flagNoData = True
159 159 return 0
160 160 else:
161 161 self.buffer[:,self.profIndex,:] = self.dataIn.data.copy()
162 162 self.profIndex += 1
163 163
164 164 if self.firstdatatime == None:
165 165 self.firstdatatime = self.dataIn.utctime
166 166
167 167 if self.profIndex == nProfiles:
168 168 self.__updateSpecFromVoltage()
169 169 self.__getFft()
170 170
171 171 self.dataOut.flagNoData = False
172 172 self.firstdatatime = None
173 173 self.profIndex = 0
174 174
175 175 return True
176 176
177 177 raise ValueError, "The type of input object '%s' is not valid"%(self.dataIn.type)
178 178
179 179 def __selectPairs(self, pairsList):
180 180
181 181 if channelList == None:
182 182 return
183 183
184 184 pairsIndexListSelected = []
185 185
186 186 for thisPair in pairsList:
187 187
188 188 if thisPair not in self.dataOut.pairsList:
189 189 continue
190 190
191 191 pairIndex = self.dataOut.pairsList.index(thisPair)
192 192
193 193 pairsIndexListSelected.append(pairIndex)
194 194
195 195 if not pairsIndexListSelected:
196 196 self.dataOut.data_cspc = None
197 197 self.dataOut.pairsList = []
198 198 return
199 199
200 200 self.dataOut.data_cspc = self.dataOut.data_cspc[pairsIndexListSelected]
201 201 self.dataOut.pairsList = [self.dataOut.pairsList[i] for i in pairsIndexListSelected]
202 202
203 203 return
204 204
205 205 def __selectPairsByChannel(self, channelList=None):
206 206
207 207 if channelList == None:
208 208 return
209 209
210 210 pairsIndexListSelected = []
211 211 for pairIndex in self.dataOut.pairsIndexList:
212 212 #First pair
213 213 if self.dataOut.pairsList[pairIndex][0] not in channelList:
214 214 continue
215 215 #Second pair
216 216 if self.dataOut.pairsList[pairIndex][1] not in channelList:
217 217 continue
218 218
219 219 pairsIndexListSelected.append(pairIndex)
220 220
221 221 if not pairsIndexListSelected:
222 222 self.dataOut.data_cspc = None
223 223 self.dataOut.pairsList = []
224 224 return
225 225
226 226 self.dataOut.data_cspc = self.dataOut.data_cspc[pairsIndexListSelected]
227 227 self.dataOut.pairsList = [self.dataOut.pairsList[i] for i in pairsIndexListSelected]
228 228
229 229 return
230 230
231 231 def selectChannels(self, channelList):
232 232
233 233 channelIndexList = []
234 234
235 235 for channel in channelList:
236 236 if channel not in self.dataOut.channelList:
237 237 raise ValueError, "Error selecting channels, Channel %d is not valid.\nAvailable channels = %s" %(channel, str(self.dataOut.channelList))
238 238
239 239 index = self.dataOut.channelList.index(channel)
240 240 channelIndexList.append(index)
241 241
242 242 self.selectChannelsByIndex(channelIndexList)
243 243
244 244 def selectChannelsByIndex(self, channelIndexList):
245 245 """
246 246 Selecciona un bloque de datos en base a canales segun el channelIndexList
247 247
248 248 Input:
249 249 channelIndexList : lista sencilla de canales a seleccionar por ej. [2,3,7]
250 250
251 251 Affected:
252 252 self.dataOut.data_spc
253 253 self.dataOut.channelIndexList
254 254 self.dataOut.nChannels
255 255
256 256 Return:
257 257 None
258 258 """
259 259
260 260 for channelIndex in channelIndexList:
261 261 if channelIndex not in self.dataOut.channelIndexList:
262 262 raise ValueError, "Error selecting channels: The value %d in channelIndexList is not valid.\nAvailable channel indexes = " %(channelIndex, self.dataOut.channelIndexList)
263 263
264 264 # nChannels = len(channelIndexList)
265 265
266 266 data_spc = self.dataOut.data_spc[channelIndexList,:]
267 267 data_dc = self.dataOut.data_dc[channelIndexList,:]
268 268
269 269 self.dataOut.data_spc = data_spc
270 270 self.dataOut.data_dc = data_dc
271 271
272 272 self.dataOut.channelList = [self.dataOut.channelList[i] for i in channelIndexList]
273 273 # self.dataOut.nChannels = nChannels
274 274
275 275 self.__selectPairsByChannel(self.dataOut.channelList)
276 276
277 277 return 1
278 278
279 279 def selectHeights(self, minHei, maxHei):
280 280 """
281 281 Selecciona un bloque de datos en base a un grupo de valores de alturas segun el rango
282 282 minHei <= height <= maxHei
283 283
284 284 Input:
285 285 minHei : valor minimo de altura a considerar
286 286 maxHei : valor maximo de altura a considerar
287 287
288 288 Affected:
289 289 Indirectamente son cambiados varios valores a travez del metodo selectHeightsByIndex
290 290
291 291 Return:
292 292 1 si el metodo se ejecuto con exito caso contrario devuelve 0
293 293 """
294 294
295 295 if (minHei > maxHei):
296 296 raise ValueError, "Error selecting heights: Height range (%d,%d) is not valid" % (minHei, maxHei)
297 297
298 298 if (minHei < self.dataOut.heightList[0]):
299 299 minHei = self.dataOut.heightList[0]
300 300
301 301 if (maxHei > self.dataOut.heightList[-1]):
302 302 maxHei = self.dataOut.heightList[-1]
303 303
304 304 minIndex = 0
305 305 maxIndex = 0
306 306 heights = self.dataOut.heightList
307 307
308 308 inda = numpy.where(heights >= minHei)
309 309 indb = numpy.where(heights <= maxHei)
310 310
311 311 try:
312 312 minIndex = inda[0][0]
313 313 except:
314 314 minIndex = 0
315 315
316 316 try:
317 317 maxIndex = indb[0][-1]
318 318 except:
319 319 maxIndex = len(heights)
320 320
321 321 self.selectHeightsByIndex(minIndex, maxIndex)
322 322
323 323 return 1
324 324
325 325 def getBeaconSignal(self, tauindex = 0, channelindex = 0, hei_ref=None):
326 326 newheis = numpy.where(self.dataOut.heightList>self.dataOut.radarControllerHeaderObj.Taus[tauindex])
327 327
328 328 if hei_ref != None:
329 329 newheis = numpy.where(self.dataOut.heightList>hei_ref)
330 330
331 331 minIndex = min(newheis[0])
332 332 maxIndex = max(newheis[0])
333 333 data_spc = self.dataOut.data_spc[:,:,minIndex:maxIndex+1]
334 334 heightList = self.dataOut.heightList[minIndex:maxIndex+1]
335 335
336 336 # determina indices
337 337 nheis = int(self.dataOut.radarControllerHeaderObj.txB/(self.dataOut.heightList[1]-self.dataOut.heightList[0]))
338 338 avg_dB = 10*numpy.log10(numpy.sum(data_spc[channelindex,:,:],axis=0))
339 339 beacon_dB = numpy.sort(avg_dB)[-nheis:]
340 340 beacon_heiIndexList = []
341 341 for val in avg_dB.tolist():
342 342 if val >= beacon_dB[0]:
343 343 beacon_heiIndexList.append(avg_dB.tolist().index(val))
344 344
345 345 #data_spc = data_spc[:,:,beacon_heiIndexList]
346 346 data_cspc = None
347 347 if self.dataOut.data_cspc is not None:
348 348 data_cspc = self.dataOut.data_cspc[:,:,minIndex:maxIndex+1]
349 349 #data_cspc = data_cspc[:,:,beacon_heiIndexList]
350 350
351 351 data_dc = None
352 352 if self.dataOut.data_dc is not None:
353 353 data_dc = self.dataOut.data_dc[:,minIndex:maxIndex+1]
354 354 #data_dc = data_dc[:,beacon_heiIndexList]
355 355
356 356 self.dataOut.data_spc = data_spc
357 357 self.dataOut.data_cspc = data_cspc
358 358 self.dataOut.data_dc = data_dc
359 359 self.dataOut.heightList = heightList
360 360 self.dataOut.beacon_heiIndexList = beacon_heiIndexList
361 361
362 362 return 1
363 363
364 364
365 365 def selectHeightsByIndex(self, minIndex, maxIndex):
366 366 """
367 367 Selecciona un bloque de datos en base a un grupo indices de alturas segun el rango
368 368 minIndex <= index <= maxIndex
369 369
370 370 Input:
371 371 minIndex : valor de indice minimo de altura a considerar
372 372 maxIndex : valor de indice maximo de altura a considerar
373 373
374 374 Affected:
375 375 self.dataOut.data_spc
376 376 self.dataOut.data_cspc
377 377 self.dataOut.data_dc
378 378 self.dataOut.heightList
379 379
380 380 Return:
381 381 1 si el metodo se ejecuto con exito caso contrario devuelve 0
382 382 """
383 383
384 384 if (minIndex < 0) or (minIndex > maxIndex):
385 385 raise ValueError, "Error selecting heights: Index range (%d,%d) is not valid" % (minIndex, maxIndex)
386 386
387 387 if (maxIndex >= self.dataOut.nHeights):
388 388 maxIndex = self.dataOut.nHeights-1
389 389
390 390 #Spectra
391 391 data_spc = self.dataOut.data_spc[:,:,minIndex:maxIndex+1]
392 392
393 393 data_cspc = None
394 394 if self.dataOut.data_cspc is not None:
395 395 data_cspc = self.dataOut.data_cspc[:,:,minIndex:maxIndex+1]
396 396
397 397 data_dc = None
398 398 if self.dataOut.data_dc is not None:
399 399 data_dc = self.dataOut.data_dc[:,minIndex:maxIndex+1]
400 400
401 401 self.dataOut.data_spc = data_spc
402 402 self.dataOut.data_cspc = data_cspc
403 403 self.dataOut.data_dc = data_dc
404 404
405 405 self.dataOut.heightList = self.dataOut.heightList[minIndex:maxIndex+1]
406 406
407 407 return 1
408 408
409 409 def removeDC(self, mode = 2):
410 410 jspectra = self.dataOut.data_spc
411 411 jcspectra = self.dataOut.data_cspc
412 412
413 413
414 414 num_chan = jspectra.shape[0]
415 415 num_hei = jspectra.shape[2]
416 416
417 417 if jcspectra is not None:
418 418 jcspectraExist = True
419 419 num_pairs = jcspectra.shape[0]
420 420 else: jcspectraExist = False
421 421
422 422 freq_dc = jspectra.shape[1]/2
423 423 ind_vel = numpy.array([-2,-1,1,2]) + freq_dc
424 424
425 425 if ind_vel[0]<0:
426 426 ind_vel[range(0,1)] = ind_vel[range(0,1)] + self.num_prof
427 427
428 428 if mode == 1:
429 429 jspectra[:,freq_dc,:] = (jspectra[:,ind_vel[1],:] + jspectra[:,ind_vel[2],:])/2 #CORRECCION
430 430
431 431 if jcspectraExist:
432 432 jcspectra[:,freq_dc,:] = (jcspectra[:,ind_vel[1],:] + jcspectra[:,ind_vel[2],:])/2
433 433
434 434 if mode == 2:
435 435
436 436 vel = numpy.array([-2,-1,1,2])
437 437 xx = numpy.zeros([4,4])
438 438
439 439 for fil in range(4):
440 440 xx[fil,:] = vel[fil]**numpy.asarray(range(4))
441 441
442 442 xx_inv = numpy.linalg.inv(xx)
443 443 xx_aux = xx_inv[0,:]
444 444
445 445 for ich in range(num_chan):
446 446 yy = jspectra[ich,ind_vel,:]
447 447 jspectra[ich,freq_dc,:] = numpy.dot(xx_aux,yy)
448 448
449 449 junkid = jspectra[ich,freq_dc,:]<=0
450 450 cjunkid = sum(junkid)
451 451
452 452 if cjunkid.any():
453 453 jspectra[ich,freq_dc,junkid.nonzero()] = (jspectra[ich,ind_vel[1],junkid] + jspectra[ich,ind_vel[2],junkid])/2
454 454
455 455 if jcspectraExist:
456 456 for ip in range(num_pairs):
457 457 yy = jcspectra[ip,ind_vel,:]
458 458 jcspectra[ip,freq_dc,:] = numpy.dot(xx_aux,yy)
459 459
460 460
461 461 self.dataOut.data_spc = jspectra
462 462 self.dataOut.data_cspc = jcspectra
463 463
464 464 return 1
465 465
466 466 def removeInterference(self, interf = 2,hei_interf = None, nhei_interf = None, offhei_interf = None):
467 467
468 468 jspectra = self.dataOut.data_spc
469 469 jcspectra = self.dataOut.data_cspc
470 470 jnoise = self.dataOut.getNoise()
471 471 num_incoh = self.dataOut.nIncohInt
472 472
473 473 num_channel = jspectra.shape[0]
474 474 num_prof = jspectra.shape[1]
475 475 num_hei = jspectra.shape[2]
476 476
477 477 #hei_interf
478 478 if hei_interf is None:
479 479 count_hei = num_hei/2 #Como es entero no importa
480 480 hei_interf = numpy.asmatrix(range(count_hei)) + num_hei - count_hei
481 481 hei_interf = numpy.asarray(hei_interf)[0]
482 482 #nhei_interf
483 483 if (nhei_interf == None):
484 484 nhei_interf = 5
485 485 if (nhei_interf < 1):
486 486 nhei_interf = 1
487 487 if (nhei_interf > count_hei):
488 488 nhei_interf = count_hei
489 489 if (offhei_interf == None):
490 490 offhei_interf = 0
491 491
492 492 ind_hei = range(num_hei)
493 493 # mask_prof = numpy.asarray(range(num_prof - 2)) + 1
494 494 # mask_prof[range(num_prof/2 - 1,len(mask_prof))] += 1
495 495 mask_prof = numpy.asarray(range(num_prof))
496 496 num_mask_prof = mask_prof.size
497 497 comp_mask_prof = [0, num_prof/2]
498 498
499 499
500 500 #noise_exist: Determina si la variable jnoise ha sido definida y contiene la informacion del ruido de cada canal
501 501 if (jnoise.size < num_channel or numpy.isnan(jnoise).any()):
502 502 jnoise = numpy.nan
503 503 noise_exist = jnoise[0] < numpy.Inf
504 504
505 505 #Subrutina de Remocion de la Interferencia
506 506 for ich in range(num_channel):
507 507 #Se ordena los espectros segun su potencia (menor a mayor)
508 508 power = jspectra[ich,mask_prof,:]
509 509 power = power[:,hei_interf]
510 510 power = power.sum(axis = 0)
511 511 psort = power.ravel().argsort()
512 512
513 513 #Se estima la interferencia promedio en los Espectros de Potencia empleando
514 514 junkspc_interf = jspectra[ich,:,hei_interf[psort[range(offhei_interf, nhei_interf + offhei_interf)]]]
515 515
516 516 if noise_exist:
517 517 # tmp_noise = jnoise[ich] / num_prof
518 518 tmp_noise = jnoise[ich]
519 519 junkspc_interf = junkspc_interf - tmp_noise
520 520 #junkspc_interf[:,comp_mask_prof] = 0
521 521
522 522 jspc_interf = junkspc_interf.sum(axis = 0) / nhei_interf
523 523 jspc_interf = jspc_interf.transpose()
524 524 #Calculando el espectro de interferencia promedio
525 525 noiseid = numpy.where(jspc_interf <= tmp_noise/ numpy.sqrt(num_incoh))
526 526 noiseid = noiseid[0]
527 527 cnoiseid = noiseid.size
528 528 interfid = numpy.where(jspc_interf > tmp_noise/ numpy.sqrt(num_incoh))
529 529 interfid = interfid[0]
530 530 cinterfid = interfid.size
531 531
532 532 if (cnoiseid > 0): jspc_interf[noiseid] = 0
533 533
534 534 #Expandiendo los perfiles a limpiar
535 535 if (cinterfid > 0):
536 536 new_interfid = (numpy.r_[interfid - 1, interfid, interfid + 1] + num_prof)%num_prof
537 537 new_interfid = numpy.asarray(new_interfid)
538 538 new_interfid = {x for x in new_interfid}
539 539 new_interfid = numpy.array(list(new_interfid))
540 540 new_cinterfid = new_interfid.size
541 541 else: new_cinterfid = 0
542 542
543 543 for ip in range(new_cinterfid):
544 544 ind = junkspc_interf[:,new_interfid[ip]].ravel().argsort()
545 545 jspc_interf[new_interfid[ip]] = junkspc_interf[ind[nhei_interf/2],new_interfid[ip]]
546 546
547 547
548 548 jspectra[ich,:,ind_hei] = jspectra[ich,:,ind_hei] - jspc_interf #Corregir indices
549 549
550 550 #Removiendo la interferencia del punto de mayor interferencia
551 551 ListAux = jspc_interf[mask_prof].tolist()
552 552 maxid = ListAux.index(max(ListAux))
553 553
554 554
555 555 if cinterfid > 0:
556 556 for ip in range(cinterfid*(interf == 2) - 1):
557 557 ind = (jspectra[ich,interfid[ip],:] < tmp_noise*(1 + 1/numpy.sqrt(num_incoh))).nonzero()
558 558 cind = len(ind)
559 559
560 560 if (cind > 0):
561 561 jspectra[ich,interfid[ip],ind] = tmp_noise*(1 + (numpy.random.uniform(cind) - 0.5)/numpy.sqrt(num_incoh))
562 562
563 563 ind = numpy.array([-2,-1,1,2])
564 564 xx = numpy.zeros([4,4])
565 565
566 566 for id1 in range(4):
567 567 xx[:,id1] = ind[id1]**numpy.asarray(range(4))
568 568
569 569 xx_inv = numpy.linalg.inv(xx)
570 570 xx = xx_inv[:,0]
571 571 ind = (ind + maxid + num_mask_prof)%num_mask_prof
572 572 yy = jspectra[ich,mask_prof[ind],:]
573 573 jspectra[ich,mask_prof[maxid],:] = numpy.dot(yy.transpose(),xx)
574 574
575 575
576 576 indAux = (jspectra[ich,:,:] < tmp_noise*(1-1/numpy.sqrt(num_incoh))).nonzero()
577 577 jspectra[ich,indAux[0],indAux[1]] = tmp_noise * (1 - 1/numpy.sqrt(num_incoh))
578 578
579 579 #Remocion de Interferencia en el Cross Spectra
580 580 if jcspectra is None: return jspectra, jcspectra
581 581 num_pairs = jcspectra.size/(num_prof*num_hei)
582 582 jcspectra = jcspectra.reshape(num_pairs, num_prof, num_hei)
583 583
584 584 for ip in range(num_pairs):
585 585
586 586 #-------------------------------------------
587 587
588 588 cspower = numpy.abs(jcspectra[ip,mask_prof,:])
589 589 cspower = cspower[:,hei_interf]
590 590 cspower = cspower.sum(axis = 0)
591 591
592 592 cspsort = cspower.ravel().argsort()
593 593 junkcspc_interf = jcspectra[ip,:,hei_interf[cspsort[range(offhei_interf, nhei_interf + offhei_interf)]]]
594 594 junkcspc_interf = junkcspc_interf.transpose()
595 595 jcspc_interf = junkcspc_interf.sum(axis = 1)/nhei_interf
596 596
597 597 ind = numpy.abs(jcspc_interf[mask_prof]).ravel().argsort()
598 598
599 599 median_real = numpy.median(numpy.real(junkcspc_interf[mask_prof[ind[range(3*num_prof/4)]],:]))
600 600 median_imag = numpy.median(numpy.imag(junkcspc_interf[mask_prof[ind[range(3*num_prof/4)]],:]))
601 601 junkcspc_interf[comp_mask_prof,:] = numpy.complex(median_real, median_imag)
602 602
603 603 for iprof in range(num_prof):
604 604 ind = numpy.abs(junkcspc_interf[iprof,:]).ravel().argsort()
605 605 jcspc_interf[iprof] = junkcspc_interf[iprof, ind[nhei_interf/2]]
606 606
607 607 #Removiendo la Interferencia
608 608 jcspectra[ip,:,ind_hei] = jcspectra[ip,:,ind_hei] - jcspc_interf
609 609
610 610 ListAux = numpy.abs(jcspc_interf[mask_prof]).tolist()
611 611 maxid = ListAux.index(max(ListAux))
612 612
613 613 ind = numpy.array([-2,-1,1,2])
614 614 xx = numpy.zeros([4,4])
615 615
616 616 for id1 in range(4):
617 617 xx[:,id1] = ind[id1]**numpy.asarray(range(4))
618 618
619 619 xx_inv = numpy.linalg.inv(xx)
620 620 xx = xx_inv[:,0]
621 621
622 622 ind = (ind + maxid + num_mask_prof)%num_mask_prof
623 623 yy = jcspectra[ip,mask_prof[ind],:]
624 624 jcspectra[ip,mask_prof[maxid],:] = numpy.dot(yy.transpose(),xx)
625 625
626 626 #Guardar Resultados
627 627 self.dataOut.data_spc = jspectra
628 628 self.dataOut.data_cspc = jcspectra
629 629
630 630 return 1
631 631
632 632 def setRadarFrequency(self, frequency=None):
633 633
634 634 if frequency != None:
635 635 self.dataOut.frequency = frequency
636 636
637 637 return 1
638 638
639 639 def getNoise(self, minHei=None, maxHei=None, minVel=None, maxVel=None):
640 640 #validacion de rango
641 641 if minHei == None:
642 642 minHei = self.dataOut.heightList[0]
643 643
644 644 if maxHei == None:
645 645 maxHei = self.dataOut.heightList[-1]
646 646
647 647 if (minHei < self.dataOut.heightList[0]) or (minHei > maxHei):
648 648 print 'minHei: %.2f is out of the heights range'%(minHei)
649 649 print 'minHei is setting to %.2f'%(self.dataOut.heightList[0])
650 650 minHei = self.dataOut.heightList[0]
651 651
652 652 if (maxHei > self.dataOut.heightList[-1]) or (maxHei < minHei):
653 653 print 'maxHei: %.2f is out of the heights range'%(maxHei)
654 654 print 'maxHei is setting to %.2f'%(self.dataOut.heightList[-1])
655 655 maxHei = self.dataOut.heightList[-1]
656 656
657 657 # validacion de velocidades
658 658 velrange = self.dataOut.getVelRange(1)
659 659
660 660 if minVel == None:
661 661 minVel = velrange[0]
662 662
663 663 if maxVel == None:
664 664 maxVel = velrange[-1]
665 665
666 666 if (minVel < velrange[0]) or (minVel > maxVel):
667 667 print 'minVel: %.2f is out of the velocity range'%(minVel)
668 668 print 'minVel is setting to %.2f'%(velrange[0])
669 669 minVel = velrange[0]
670 670
671 671 if (maxVel > velrange[-1]) or (maxVel < minVel):
672 672 print 'maxVel: %.2f is out of the velocity range'%(maxVel)
673 673 print 'maxVel is setting to %.2f'%(velrange[-1])
674 674 maxVel = velrange[-1]
675 675
676 676 # seleccion de indices para rango
677 677 minIndex = 0
678 678 maxIndex = 0
679 679 heights = self.dataOut.heightList
680 680
681 681 inda = numpy.where(heights >= minHei)
682 682 indb = numpy.where(heights <= maxHei)
683 683
684 684 try:
685 685 minIndex = inda[0][0]
686 686 except:
687 687 minIndex = 0
688 688
689 689 try:
690 690 maxIndex = indb[0][-1]
691 691 except:
692 692 maxIndex = len(heights)
693 693
694 694 if (minIndex < 0) or (minIndex > maxIndex):
695 695 raise ValueError, "some value in (%d,%d) is not valid" % (minIndex, maxIndex)
696 696
697 697 if (maxIndex >= self.dataOut.nHeights):
698 698 maxIndex = self.dataOut.nHeights-1
699 699
700 700 # seleccion de indices para velocidades
701 701 indminvel = numpy.where(velrange >= minVel)
702 702 indmaxvel = numpy.where(velrange <= maxVel)
703 703 try:
704 704 minIndexVel = indminvel[0][0]
705 705 except:
706 706 minIndexVel = 0
707 707
708 708 try:
709 709 maxIndexVel = indmaxvel[0][-1]
710 710 except:
711 711 maxIndexVel = len(velrange)
712 712
713 713 #seleccion del espectro
714 714 data_spc = self.dataOut.data_spc[:,minIndexVel:maxIndexVel+1,minIndex:maxIndex+1]
715 715 #estimacion de ruido
716 716 noise = numpy.zeros(self.dataOut.nChannels)
717 717
718 718 for channel in range(self.dataOut.nChannels):
719 719 daux = data_spc[channel,:,:]
720 720 noise[channel] = hildebrand_sekhon(daux, self.dataOut.nIncohInt)
721 721
722 722 self.dataOut.noise_estimation = noise.copy()
723 723
724 724 return 1
725 725
726 726 class IncohInt(Operation):
727 727
728 728
729 729 __profIndex = 0
730 730 __withOverapping = False
731 731
732 732 __byTime = False
733 733 __initime = None
734 734 __lastdatatime = None
735 735 __integrationtime = None
736 736
737 737 __buffer_spc = None
738 738 __buffer_cspc = None
739 739 __buffer_dc = None
740 740
741 741 __dataReady = False
742 742
743 743 __timeInterval = None
744 744
745 745 n = None
746 746
747 747
748 748
749 749 def __init__(self, **kwargs):
750 750
751 751 Operation.__init__(self, **kwargs)
752 752 # self.isConfig = False
753 753
754 754 def setup(self, n=None, timeInterval=None, overlapping=False):
755 755 """
756 756 Set the parameters of the integration class.
757 757
758 758 Inputs:
759 759
760 760 n : Number of coherent integrations
761 761 timeInterval : Time of integration. If the parameter "n" is selected this one does not work
762 762 overlapping :
763 763
764 764 """
765 765
766 766 self.__initime = None
767 767 self.__lastdatatime = 0
768 768
769 769 self.__buffer_spc = 0
770 770 self.__buffer_cspc = 0
771 771 self.__buffer_dc = 0
772 772
773 773 self.__profIndex = 0
774 774 self.__dataReady = False
775 775 self.__byTime = False
776 776
777 777 if n is None and timeInterval is None:
778 778 raise ValueError, "n or timeInterval should be specified ..."
779 779
780 780 if n is not None:
781 781 self.n = int(n)
782 782 else:
783 783 self.__integrationtime = int(timeInterval) #if (type(timeInterval)!=integer) -> change this line
784 784 self.n = None
785 785 self.__byTime = True
786 786
787 787 def putData(self, data_spc, data_cspc, data_dc):
788 788
789 789 """
790 790 Add a profile to the __buffer_spc and increase in one the __profileIndex
791 791
792 792 """
793 793
794 794 self.__buffer_spc += data_spc
795 795
796 796 if data_cspc is None:
797 797 self.__buffer_cspc = None
798 798 else:
799 799 self.__buffer_cspc += data_cspc
800 800
801 801 if data_dc is None:
802 802 self.__buffer_dc = None
803 803 else:
804 804 self.__buffer_dc += data_dc
805 805
806 806 self.__profIndex += 1
807 807
808 808 return
809 809
810 810 def pushData(self):
811 811 """
812 812 Return the sum of the last profiles and the profiles used in the sum.
813 813
814 814 Affected:
815 815
816 816 self.__profileIndex
817 817
818 818 """
819 819
820 820 data_spc = self.__buffer_spc
821 821 data_cspc = self.__buffer_cspc
822 822 data_dc = self.__buffer_dc
823 823 n = self.__profIndex
824 824
825 825 self.__buffer_spc = 0
826 826 self.__buffer_cspc = 0
827 827 self.__buffer_dc = 0
828 828 self.__profIndex = 0
829 829
830 830 return data_spc, data_cspc, data_dc, n
831 831
832 832 def byProfiles(self, *args):
833 833
834 834 self.__dataReady = False
835 835 avgdata_spc = None
836 836 avgdata_cspc = None
837 837 avgdata_dc = None
838 838
839 839 self.putData(*args)
840 840
841 841 if self.__profIndex == self.n:
842 842
843 843 avgdata_spc, avgdata_cspc, avgdata_dc, n = self.pushData()
844 844 self.n = n
845 845 self.__dataReady = True
846 846
847 847 return avgdata_spc, avgdata_cspc, avgdata_dc
848 848
849 849 def byTime(self, datatime, *args):
850 850
851 851 self.__dataReady = False
852 852 avgdata_spc = None
853 853 avgdata_cspc = None
854 854 avgdata_dc = None
855 855
856 856 self.putData(*args)
857 857
858 858 if (datatime - self.__initime) >= self.__integrationtime:
859 859 avgdata_spc, avgdata_cspc, avgdata_dc, n = self.pushData()
860 860 self.n = n
861 861 self.__dataReady = True
862 862
863 863 return avgdata_spc, avgdata_cspc, avgdata_dc
864 864
865 865 def integrate(self, datatime, *args):
866 866
867 867 if self.__profIndex == 0:
868 868 self.__initime = datatime
869 869
870 870 if self.__byTime:
871 871 avgdata_spc, avgdata_cspc, avgdata_dc = self.byTime(datatime, *args)
872 872 else:
873 873 avgdata_spc, avgdata_cspc, avgdata_dc = self.byProfiles(*args)
874 874
875 875 if not self.__dataReady:
876 876 return None, None, None, None
877 877
878 878 return self.__initime, avgdata_spc, avgdata_cspc, avgdata_dc
879 879
880 880 def run(self, dataOut, n=None, timeInterval=None, overlapping=False):
881
882 881 if n==1:
883 882 return
884 883
885 884 dataOut.flagNoData = True
886 885
887 886 if not self.isConfig:
888 887 self.setup(n, timeInterval, overlapping)
889 888 self.isConfig = True
890 889
891 890 avgdatatime, avgdata_spc, avgdata_cspc, avgdata_dc = self.integrate(dataOut.utctime,
892 891 dataOut.data_spc,
893 892 dataOut.data_cspc,
894 893 dataOut.data_dc)
895 894
896 895 if self.__dataReady:
897 896
898 897 dataOut.data_spc = avgdata_spc
899 898 dataOut.data_cspc = avgdata_cspc
900 899 dataOut.data_dc = avgdata_dc
901 900
902 901 dataOut.nIncohInt *= self.n
903 902 dataOut.utctime = avgdatatime
904 903 dataOut.flagNoData = False
Show file before | Show file after | Hide comments
@@ -1,1007 +1,1007
1 1 '''
2 2 @author: Daniel Suarez
3 3 '''
4 4 import os
5 5 import glob
6 6 import ftplib
7 7
8 8 try:
9 9 import paramiko
10 10 import scp
11 11 except:
12 12 print "You should install paramiko and scp libraries \nif you want to use SSH protocol to upload files to the server"
13 13
14 14 import time
15 15
16 16 import threading
17 17 Thread = threading.Thread
18 18
19 19 # try:
20 20 # from gevent import sleep
21 21 # except:
22 22 from time import sleep
23 23
24 24 from schainpy.model.proc.jroproc_base import ProcessingUnit, Operation
25 25
26 26 class Remote(Thread):
27 27 """
28 28 Remote is a parent class used to define the behaviour of FTP and SSH class. These clases are
29 29 used to upload or download files remotely.
30 30
31 31 Non-standard Python modules used:
32 32 None
33 33
34 34 Written by:
35
36 "Miguel Urco":mailto:miguel.urco@jro.igp.gob.pe Jun. 03, 2015
37
35 "Miguel Urco":mailto:miguel.urco@jro.igp.gob.pe Jun. 03, 2015
38 36 """
39 37
40 38 server = None
41 39 username = None
42 40 password = None
43 41 remotefolder = None
44 42
45 43 period = 60
46 44 fileList = []
47 45 bussy = False
48 46
49 47 def __init__(self, server, username, password, remotefolder, period=60):
50 48
51 49 Thread.__init__(self)
52 50
53 51 self.setDaemon(True)
54 52
55 53 self.status = 0
56 54
57 55 self.__server = server
58 56 self.__username = username
59 57 self.__password = password
60 58 self.__remotefolder = remotefolder
61 59
62 60 self.period = period
63 61
64 62 self.fileList = []
65 63 self.bussy = False
66 64
67 65 self.stopFlag = False
68 66
69 67 print "[Remote Server] Opening server: %s" %self.__server
70 68 if self.open(self.__server, self.__username, self.__password, self.__remotefolder):
71 69 print "[Remote Server] %s server was opened successfully" %self.__server
72 70
73 71 self.close()
74 72
75 73 self.mutex = threading.Lock()
76 74
77 75 def stop(self):
78 76
79 77 self.stopFlag = True
80 78 self.join(10)
81 79
82 80 def open(self):
83 81 """
84 82 Connect to server and create a connection class (FTP or SSH) to remote server.
85 83 """
86 84 raise NotImplementedError, "Implement this method in child class"
87 85
88 86 def close(self):
89 87 """
90 88 Close connection to server
91 89 """
92 90 raise NotImplementedError, "Implement this method in child class"
93 91
94 92 def mkdir(self, remotefolder):
95 93 """
96 94 Create a folder remotely
97 95 """
98 96 raise NotImplementedError, "Implement this method in child class"
99 97
100 98 def cd(self, remotefolder):
101 99 """
102 100 Change working directory in remote server
103 101 """
104 102 raise NotImplementedError, "Implement this method in child class"
105 103
106 104 def download(self, filename, localfolder=None):
107 105 """
108 106 Download a file from server to local host
109 107 """
110 108 raise NotImplementedError, "Implement this method in child class"
111 109
112 110 def sendFile(self, fullfilename):
113 111 """
114 112 sendFile method is used to upload a local file to the current directory in remote server
115 113
116 114 Inputs:
117 115 fullfilename - full path name of local file to store in remote directory
118 116
119 117 Returns:
120 118 0 in error case else 1
121 119 """
122 120 raise NotImplementedError, "Implement this method in child class"
123 121
124 122 def upload(self, fullfilename, remotefolder=None):
125 123 """
126 124 upload method is used to upload a local file to remote directory. This method changes
127 125 working directory before sending a file.
128 126
129 127 Inputs:
130 128 fullfilename - full path name of local file to store in remote directory
131 129
132 130 remotefolder - remote directory
133 131
134 132 Returns:
135 133 0 in error case else 1
136 134 """
137 135 print "[Remote Server] Uploading %s to %s:%s" %(fullfilename, self.server, self.remotefolder)
138 136
139 137 if not self.status:
140 138 return 0
141 139
142 140 if remotefolder == None:
143 141 remotefolder = self.remotefolder
144 142
145 143 if not self.cd(remotefolder):
146 144 return 0
147 145
148 146 if not self.sendFile(fullfilename):
149 147 print "[Remote Server] Error uploading file %s" %fullfilename
150 148 return 0
151 149
152 150 print "[Remote Server] upload finished successfully"
153 151
154 152 return 1
155 153
156 154 def delete(self, filename):
157 155 """
158 156 Remove a file from remote server
159 157 """
160 158 pass
161 159
162 160 def updateFileList(self, fileList):
163 161 """
164 162 Remove a file from remote server
165 163 """
166 164
167 165 if fileList == self.fileList:
168 166 return 0
169 167
170 168 self.mutex.acquire()
171 # init = time.time()
172 #
173 # while(self.bussy):
174 # sleep(0.1)
175 # if time.time() - init > 2*self.period:
176 # return 0
177
169 # init = time.time()
170 #
171 # while(self.bussy):
172 # sleep(0.1)
173 # if time.time() - init > 2*self.period:
174 # return 0
175
178 176 self.fileList = fileList
179 177 self.mutex.release()
180 178 return 1
181 179
182 180 def run(self):
183 181
184 182 if not self.status:
185 183 print "Finishing FTP service"
186 184 return
187 185
188 186 if not self.cd(self.remotefolder):
189 187 raise ValueError, "Could not access to the new remote directory: %s" %self.remotefolder
190 188
191 189 while True:
192 190
193 191 for i in range(self.period):
194 192 if self.stopFlag:
195 193 break
196 194 sleep(1)
197 195
198 196 if self.stopFlag:
199 197 break
200
201 # self.bussy = True
198
199 # self.bussy = True
202 200 self.mutex.acquire()
203 201
204 202 print "[Remote Server] Opening %s" %self.__server
205 203 if not self.open(self.__server, self.__username, self.__password, self.__remotefolder):
206 204 self.mutex.release()
207 205 continue
208 206
209 207 for thisFile in self.fileList:
210 208 self.upload(thisFile, self.remotefolder)
211 209
212 210 print "[Remote Server] Closing %s" %self.__server
213 211 self.close()
214 212
215 213 self.mutex.release()
216 214 # self.bussy = False
217 215
218 216 print "[Remote Server] Thread stopped successfully"
219 217
220 218 class FTPClient(Remote):
221 219
222 220 __ftpClientObj = None
223 221
224 222 def __init__(self, server, username, password, remotefolder, period=60):
225 223 """
226 224 """
227 225 Remote.__init__(self, server, username, password, remotefolder, period)
228 226
229 227 def open(self, server, username, password, remotefolder):
230 228
231 229 """
232 230 This method is used to set FTP parameters and establish a connection to remote server
233 231
234 232 Inputs:
235 233 server - remote server IP Address
236 234
237 235 username - remote server Username
238 236
239 237 password - remote server password
240 238
241 239 remotefolder - remote server current working directory
242 240
243 241 Return:
244 242 Boolean - Returns 1 if a connection has been established, 0 otherwise
245 243
246 244 Affects:
247 245 self.status - in case of error or fail connection this parameter is set to 0 else 1
248 246
249 247 """
250 248
251 249 if server == None:
252 250 raise ValueError, "FTP server should be defined"
253 251
254 252 if username == None:
255 253 raise ValueError, "FTP username should be defined"
256 254
257 255 if password == None:
258 256 raise ValueError, "FTP password should be defined"
259 257
260 258 if remotefolder == None:
261 259 raise ValueError, "FTP remote folder should be defined"
262 260
263 261 try:
264 262 ftpClientObj = ftplib.FTP(server)
265 263 except ftplib.all_errors, e:
266 264 print "[FTP Server]: FTP server connection fail: %s" %server
267 265 print "[FTP Server]:", e
268 266 self.status = 0
269 267 return 0
270 268
271 269 try:
272 270 ftpClientObj.login(username, password)
273 271 except ftplib.all_errors:
274 272 print "[FTP Server]: FTP username or password are incorrect"
275 273 self.status = 0
276 274 return 0
277 275
278 276 if remotefolder == None:
279 277 remotefolder = ftpClientObj.pwd()
280 278 else:
281 279 try:
282 280 ftpClientObj.cwd(remotefolder)
283 281 except ftplib.all_errors:
284 282 print "[FTP Server]: FTP remote folder is invalid: %s" %remotefolder
285 283 remotefolder = ftpClientObj.pwd()
286 284
287 285 self.server = server
288 286 self.username = username
289 287 self.password = password
290 288 self.remotefolder = remotefolder
291 289 self.__ftpClientObj = ftpClientObj
292 290 self.status = 1
293 291
294 292 return 1
295 293
296 294 def close(self):
297 295 """
298 296 Close connection to remote server
299 297 """
300 298 if not self.status:
301 299 return 0
302 300
303 301 self.__ftpClientObj.close()
304 302
305 303 def mkdir(self, remotefolder):
306 304 """
307 305 mkdir is used to make a new directory in remote server
308 306
309 307 Input:
310 308 remotefolder - directory name
311 309
312 310 Return:
313 311 0 in error case else 1
314 312 """
315 313 if not self.status:
316 314 return 0
317 315
318 316 try:
319 317 self.__ftpClientObj.mkd(dirname)
320 318 except ftplib.all_errors:
321 319 print "[FTP Server]: Error creating remote folder: %s" %remotefolder
322 320 return 0
323 321
324 322 return 1
325 323
326 324 def cd(self, remotefolder):
327 325 """
328 326 cd is used to change remote working directory on server
329 327
330 328 Input:
331 329 remotefolder - current working directory
332 330
333 331 Affects:
334 332 self.remotefolder
335 333
336 334 Return:
337 335 0 in case of error else 1
338 336 """
339 337 if not self.status:
340 338 return 0
341 339
342 340 if remotefolder == self.remotefolder:
343 341 return 1
344 342
345 343 try:
346 344 self.__ftpClientObj.cwd(remotefolder)
347 345 except ftplib.all_errors:
348 346 print '[FTP Server]: Error changing to %s' %remotefolder
349 347 print '[FTP Server]: Trying to create remote folder'
350 348
351 349 if not self.mkdir(remotefolder):
352 350 print '[FTP Server]: Remote folder could not be created'
353 351 return 0
354 352
355 353 try:
356 354 self.__ftpClientObj.cwd(remotefolder)
357 355 except ftplib.all_errors:
358 356 return 0
359 357
360 358 self.remotefolder = remotefolder
361 359
362 360 return 1
363 361
364 362 def sendFile(self, fullfilename):
365 363
366 364 if not self.status:
367 365 return 0
368 366
369 367 fp = open(fullfilename, 'rb')
370 368
371 369 filename = os.path.basename(fullfilename)
372 370
373 371 command = "STOR %s" %filename
374 372
375 373 try:
376 374 self.__ftpClientObj.storbinary(command, fp)
377 375 except ftplib.all_errors, e:
378 376 print "[FTP Server]:", e
379 377 return 0
380 378
381 379 try:
382 380 self.__ftpClientObj.sendcmd('SITE CHMOD 755 ' + filename)
383 381 except ftplib.all_errors, e:
384 382 print "[FTP Server]:", e
385 383
386 384 fp.close()
387 385
388 386 return 1
389 387
390 388 class SSHClient(Remote):
391 389
392 390 __sshClientObj = None
393 391 __scpClientObj = None
394 392
395 393 def __init__(self, server, username, password, remotefolder, period=60):
396 394 """
397 395 """
398 396 Remote.__init__(self, server, username, password, remotefolder, period)
399 397
400 398 def open(self, server, username, password, remotefolder, port=22):
401 399
402 400 """
403 This method is used to set SSH parameters and establish a connection to a remote server
404
405 Inputs:
406 server - remote server IP Address
407
408 username - remote server Username
409
410 password - remote server password
411
412 remotefolder - remote server current working directory
413
414 Return: void
415
416 Affects:
417 self.status - in case of error or fail connection this parameter is set to 0 else 1
401 This method is used to set SSH parameters and establish a connection to a remote server
402
403 Inputs:
404 server - remote server IP Address
405
406 username - remote server Username
407
408 password - remote server password
409
410 remotefolder - remote server current working directory
411
412 Return: void
413
414 Affects:
415 self.status - in case of error or fail connection this parameter is set to 0 else 1
418 416
419 417 """
420 418 import socket
421 419
422 420 if server == None:
423 421 raise ValueError, "SSH server should be defined"
424 422
425 423 if username == None:
426 424 raise ValueError, "SSH username should be defined"
427 425
428 426 if password == None:
429 427 raise ValueError, "SSH password should be defined"
430 428
431 429 if remotefolder == None:
432 430 raise ValueError, "SSH remote folder should be defined"
433 431
434 432 sshClientObj = paramiko.SSHClient()
435 433
436 434 sshClientObj.load_system_host_keys()
437 435 sshClientObj.set_missing_host_key_policy(paramiko.WarningPolicy())
438 436
439 437 self.status = 0
440 438 try:
441 439 sshClientObj.connect(server, username=username, password=password, port=port)
442 440 except paramiko.AuthenticationException, e:
443 # print "SSH username or password are incorrect: %s"
441 # print "SSH username or password are incorrect: %s"
444 442 print "[SSH Server]:", e
445 443 return 0
446 444 except SSHException, e:
447 445 print "[SSH Server]:", e
448 446 return 0
449 447 except socket.error:
450 448 self.status = 0
451 449 print "[SSH Server]:", e
452 450 return 0
453 451
454 452 self.status = 1
455 453 scpClientObj = scp.SCPClient(sshClientObj.get_transport(), socket_timeout=30)
456 454
457 455 if remotefolder == None:
458 456 remotefolder = self.pwd()
459 457
460 458 self.server = server
461 459 self.username = username
462 460 self.password = password
463 461 self.__sshClientObj = sshClientObj
464 462 self.__scpClientObj = scpClientObj
465 463 self.status = 1
466 464
467 465 if not self.cd(remotefolder):
468 466 raise ValueError, "[SSH Server]: Could not access to remote folder: %s" %remotefolder
469 467 return 0
470 468
471 469 self.remotefolder = remotefolder
472 470
473 471 return 1
474 472
475 473 def close(self):
476 474 """
477 Close connection to remote server
475 Close connection to remote server
478 476 """
479 477 if not self.status:
480 478 return 0
481 479
482 480 self.__scpClientObj.close()
483 481 self.__sshClientObj.close()
484 482
485 483 def __execute(self, command):
486 484 """
487 __execute a command on remote server
488
489 Input:
490 command - Exmaple 'ls -l'
491
492 Return:
493 0 in error case else 1
485 __execute a command on remote server
486
487 Input:
488 command - Exmaple 'ls -l'
489
490 Return:
491 0 in error case else 1
494 492 """
495 493 if not self.status:
496 494 return 0
497 495
498 496 stdin, stdout, stderr = self.__sshClientObj.exec_command(command)
499 497
500 498 result = stderr.readlines()
501 499 if len(result) > 1:
502 500 return 0
503 501
504 502 result = stdout.readlines()
505 503 if len(result) > 1:
506 504 return result[0][:-1]
507 505
508 506 return 1
509 507
510 508 def mkdir(self, remotefolder):
511 509 """
512 mkdir is used to make a new directory in remote server
513
514 Input:
515 remotefolder - directory name
516
517 Return:
518 0 in error case else 1
510 mkdir is used to make a new directory in remote server
511
512 Input:
513 remotefolder - directory name
514
515 Return:
516 0 in error case else 1
519 517 """
520 518
521 519 command = 'mkdir %s' %remotefolder
522 520
523 521 return self.__execute(command)
524 522
525 523 def pwd(self):
526 524
527 525 command = 'pwd'
528 526
529 527 return self.__execute(command)
530 528
531 529 def cd(self, remotefolder):
532 530 """
533 cd is used to change remote working directory on server
534
535 Input:
536 remotefolder - current working directory
537
538 Affects:
539 self.remotefolder
540
541 Return:
542 0 in case of error else 1
531 cd is used to change remote working directory on server
532
533 Input:
534 remotefolder - current working directory
535
536 Affects:
537 self.remotefolder
538
539 Return:
540 0 in case of error else 1
543 541 """
544 542 if not self.status:
545 543 return 0
546 544
547 545 if remotefolder == self.remotefolder:
548 546 return 1
549 547
550 548 chk_command = "cd %s; pwd" %remotefolder
551 549 mkdir_command = "mkdir %s" %remotefolder
552 550
553 551 if not self.__execute(chk_command):
554 552 if not self.__execute(mkdir_command):
555 553 self.remotefolder = None
556 554 return 0
557 555
558 556 self.remotefolder = remotefolder
559 557
560 558 return 1
561 559
562 560 def sendFile(self, fullfilename):
563 561
564 562 if not self.status:
565 563 return 0
566 564
567 565 try:
568 566 self.__scpClientObj.put(fullfilename, remote_path=self.remotefolder)
569 567 except scp.ScpError, e:
570 568 print "[SSH Server]", str(e)
571 569 return 0
572 570
573 571 remotefile = os.path.join(self.remotefolder, os.path.split(fullfilename)[-1])
574 572 command = 'chmod 775 %s' %remotefile
575 573
576 574 return self.__execute(command)
577 575
578 576 class SendToServer(ProcessingUnit):
579 577
580 578 def __init__(self, **kwargs):
581 579
582 580 ProcessingUnit.__init__(self, **kwargs)
583 581
584 582 self.isConfig = False
585 self.clientObj = None
586
583 self.clientObj = None
584
587 585 def setup(self, server, username, password, remotefolder, localfolder, ext='.png', period=60, protocol='ftp', **kwargs):
588 586
589 587 self.clientObj = None
590 588 self.localfolder = localfolder
591 589 self.ext = ext
592 590 self.period = period
593 591
594 592 if str.lower(protocol) == 'ftp':
595 593 self.clientObj = FTPClient(server, username, password, remotefolder, period)
596 594
597 595 if str.lower(protocol) == 'ssh':
598 596 self.clientObj = SSHClient(server, username, password, remotefolder, period)
599 597
600 598 if not self.clientObj:
601 599 raise ValueError, "%s has been chosen as remote access protocol but it is not valid" %protocol
602 600
603 601 self.clientObj.start()
604 602
605 603 def findFiles(self):
606 604
607 605 if not type(self.localfolder) == list:
608 606 folderList = [self.localfolder]
609 607 else:
610 608 folderList = self.localfolder
611 609
612 610 #Remove duplicate items
613 611 folderList = list(set(folderList))
614 612
615 613 fullfilenameList = []
616 614
617 615 for thisFolder in folderList:
618 616
619 617 print "[Remote Server]: Searching files on %s" %thisFolder
620 618
621 619 filenameList = glob.glob1(thisFolder, '*%s' %self.ext)
622 620
623 621 if len(filenameList) < 1:
624 622 continue
625 623
626 624 for thisFile in filenameList:
627 625 fullfilename = os.path.join(thisFolder, thisFile)
628 626
629 627 if fullfilename in fullfilenameList:
630 628 continue
631 629
632 630 #Only files modified in the last 30 minutes are considered
633 631 if os.path.getmtime(fullfilename) < time.time() - 30*60:
634 632 continue
635 633
636 634 fullfilenameList.append(fullfilename)
637 635
638 636 return fullfilenameList
639 637
640 638 def run(self, **kwargs):
641
642 639 if not self.isConfig:
643 640 self.init = time.time()
644 641 self.setup(**kwargs)
645 642 self.isConfig = True
646
643
644 if not self.clientObj.is_alive():
645 print "[Remote Server]: Restarting connection "
646 self.setup(**kwargs)
647
647 648 if time.time() - self.init >= self.period:
648 649 fullfilenameList = self.findFiles()
649 650
650 651 if self.clientObj.updateFileList(fullfilenameList):
651 652 print "[Remote Server]: Sending the next files ", str(fullfilenameList)
652
653 653 self.init = time.time()
654 654
655 655 def close(self):
656 656 print "[Remote Server] Stopping thread"
657 657 self.clientObj.stop()
658 658
659 659
660 660 class FTP(object):
661 661 """
662 662 Ftp is a public class used to define custom File Transfer Protocol from "ftplib" python module
663 663
664 664 Non-standard Python modules used: None
665 665
666 666 Written by "Daniel Suarez":mailto:daniel.suarez@jro.igp.gob.pe Oct. 26, 2010
667 667 """
668 668
669 669 def __init__(self,server = None, username=None, password=None, remotefolder=None):
670 670 """
671 671 This method is used to setting parameters for FTP and establishing connection to remote server
672 672
673 673 Inputs:
674 674 server - remote server IP Address
675 675
676 676 username - remote server Username
677 677
678 678 password - remote server password
679 679
680 680 remotefolder - remote server current working directory
681 681
682 682 Return: void
683 683
684 684 Affects:
685 685 self.status - in Error Case or Connection Failed this parameter is set to 1 else 0
686 686
687 687 self.folderList - sub-folder list of remote folder
688 688
689 689 self.fileList - file list of remote folder
690 690
691 691
692 692 """
693 693
694 694 if ((server == None) and (username==None) and (password==None) and (remotefolder==None)):
695 695 server, username, password, remotefolder = self.parmsByDefault()
696 696
697 697 self.server = server
698 698 self.username = username
699 699 self.password = password
700 700 self.remotefolder = remotefolder
701 701 self.file = None
702 702 self.ftp = None
703 703 self.status = 0
704 704
705 705 try:
706 706 self.ftp = ftplib.FTP(self.server)
707 707 self.ftp.login(self.username,self.password)
708 self.ftp.cwd(self.remotefolder)
709 # print 'Connect to FTP Server: Successfully'
710
708 self.ftp.cwd(self.remotefolder)
709 # print 'Connect to FTP Server: Successfully'
710
711 711 except ftplib.all_errors:
712 712 print 'Error FTP Service'
713 713 self.status = 1
714 714 return
715 715
716 716
717 717
718 718 self.dirList = []
719 719
720 720 try:
721 721 self.dirList = self.ftp.nlst()
722 722
723 723 except ftplib.error_perm, resp:
724 724 if str(resp) == "550 No files found":
725 725 print "no files in this directory"
726 726 self.status = 1
727 727 return
728 728
729 729 except ftplib.all_errors:
730 730 print 'Error Displaying Dir-Files'
731 731 self.status = 1
732 732 return
733 733
734 734 self.fileList = []
735 735 self.folderList = []
736 736 #only for test
737 737 for f in self.dirList:
738 738 name, ext = os.path.splitext(f)
739 739 if ext != '':
740 740 self.fileList.append(f)
741 # print 'filename: %s - size: %d'%(f,self.ftp.size(f))
741 # print 'filename: %s - size: %d'%(f,self.ftp.size(f))
742 742
743 743 def parmsByDefault(self):
744 744 server = 'jro-app.igp.gob.pe'
745 745 username = 'wmaster'
746 746 password = 'mst2010vhf'
747 747 remotefolder = '/home/wmaster/graficos'
748 748
749 749 return server, username, password, remotefolder
750 750
751 751
752 752 def mkd(self,dirname):
753 753 """
754 754 mkd is used to make directory in remote server
755 755
756 756 Input:
757 757 dirname - directory name
758 758
759 759 Return:
760 760 1 in error case else 0
761 761 """
762 762 try:
763 763 self.ftp.mkd(dirname)
764 764 except:
765 765 print 'Error creating remote folder:%s'%dirname
766 766 return 1
767 767
768 768 return 0
769 769
770 770
771 771 def delete(self,filename):
772 772 """
773 773 delete is used to delete file in current working directory of remote server
774 774
775 775 Input:
776 776 filename - filename to delete in remote folder
777 777
778 778 Return:
779 779 1 in error case else 0
780 780 """
781 781
782 782 try:
783 783 self.ftp.delete(filename)
784 784 except:
785 785 print 'Error deleting remote file:%s'%filename
786 786 return 1
787 787
788 788 return 0
789 789
790 790 def download(self,filename,localfolder):
791 791 """
792 792 download is used to downloading file from remote folder into local folder
793 793
794 794 Inputs:
795 795 filename - filename to donwload
796 796
797 797 localfolder - directory local to store filename
798 798
799 799 Returns:
800 800 self.status - 1 in error case else 0
801 801 """
802 802
803 803 self.status = 0
804 804
805 805
806 806 if not(filename in self.fileList):
807 807 print 'filename:%s not exists'%filename
808 808 self.status = 1
809 809 return self.status
810 810
811 811 newfilename = os.path.join(localfolder,filename)
812 812
813 813 self.file = open(newfilename, 'wb')
814 814
815 815 try:
816 816 print 'Download: ' + filename
817 817 self.ftp.retrbinary('RETR ' + filename, self.__handleDownload)
818 818 print 'Download Complete'
819 819 except ftplib.all_errors:
820 820 print 'Error Downloading ' + filename
821 821 self.status = 1
822 822 return self.status
823 823
824 824 self.file.close()
825 825
826 826 return self.status
827 827
828 828
829 829 def __handleDownload(self,block):
830 830 """
831 831 __handleDownload is used to handle writing file
832 832 """
833 833 self.file.write(block)
834 834
835 835
836 836 def upload(self,filename,remotefolder=None):
837 837 """
838 838 upload is used to uploading local file to remote directory
839 839
840 840 Inputs:
841 841 filename - full path name of local file to store in remote directory
842 842
843 843 remotefolder - remote directory
844 844
845 845 Returns:
846 846 self.status - 1 in error case else 0
847 847 """
848 848
849 849 if remotefolder == None:
850 850 remotefolder = self.remotefolder
851 851
852 852 self.status = 0
853 853
854 854 try:
855 855 self.ftp.cwd(remotefolder)
856 856
857 857 self.file = open(filename, 'rb')
858 858
859 859 (head, tail) = os.path.split(filename)
860 860
861 861 command = "STOR " + tail
862 862
863 863 print 'Uploading: ' + tail
864 864 self.ftp.storbinary(command, self.file)
865 865 print 'Upload Completed'
866 866
867 867 except ftplib.all_errors:
868 868 print 'Error Uploading ' + tail
869 869 self.status = 1
870 870 return self.status
871 871
872 872 self.file.close()
873 873
874 874 #back to initial directory in __init__()
875 875 self.ftp.cwd(self.remotefolder)
876 876
877 877 return self.status
878 878
879 879
880 880 def dir(self,remotefolder):
881 881 """
882 882 dir is used to change working directory of remote server and get folder and file list
883 883
884 884 Input:
885 885 remotefolder - current working directory
886 886
887 887 Affects:
888 888 self.fileList - file list of working directory
889 889
890 890 Return:
891 891 infoList - list with filenames and size of file in bytes
892 892
893 893 self.folderList - folder list
894 894 """
895 895
896 896 self.remotefolder = remotefolder
897 897 print 'Change to ' + self.remotefolder
898 898 try:
899 899 self.ftp.cwd(remotefolder)
900 900 except ftplib.all_errors:
901 901 print 'Error Change to ' + self.remotefolder
902 902 infoList = None
903 903 self.folderList = None
904 904 return infoList,self.folderList
905 905
906 906 self.dirList = []
907 907
908 908 try:
909 909 self.dirList = self.ftp.nlst()
910 910
911 911 except ftplib.error_perm, resp:
912 912 if str(resp) == "550 No files found":
913 913 print "no files in this directory"
914 914 infoList = None
915 915 self.folderList = None
916 916 return infoList,self.folderList
917 917 except ftplib.all_errors:
918 918 print 'Error Displaying Dir-Files'
919 919 infoList = None
920 920 self.folderList = None
921 921 return infoList,self.folderList
922 922
923 923 infoList = []
924 924 self.fileList = []
925 925 self.folderList = []
926 926 for f in self.dirList:
927 927 name,ext = os.path.splitext(f)
928 928 if ext != '':
929 929 self.fileList.append(f)
930 930 value = (f,self.ftp.size(f))
931 931 infoList.append(value)
932 932
933 933 if ext == '':
934 934 self.folderList.append(f)
935 935
936 936 return infoList,self.folderList
937 937
938 938
939 939 def close(self):
940 940 """
941 941 close is used to close and end FTP connection
942 942
943 943 Inputs: None
944 944
945 945 Return: void
946 946
947 947 """
948 948 self.ftp.close()
949 949
950 950 class SendByFTP(Operation):
951 951
952 952 def __init__(self, **kwargs):
953 953 Operation.__init__(self, **kwargs)
954 954 self.status = 1
955 955 self.counter = 0
956 956
957 957 def error_print(self, ValueError):
958 958
959 959 print ValueError, 'Error FTP'
960 960 print "don't worry the program is running..."
961 961
962 962 def worker_ftp(self, server, username, password, remotefolder, filenameList):
963 963
964 964 self.ftpClientObj = FTP(server, username, password, remotefolder)
965 965 for filename in filenameList:
966 966 self.ftpClientObj.upload(filename)
967 967 self.ftpClientObj.close()
968 968
969 969 def ftp_thread(self, server, username, password, remotefolder):
970 970 if not(self.status):
971 971 return
972 972
973 973 import multiprocessing
974 974
975 975 p = multiprocessing.Process(target=self.worker_ftp, args=(server, username, password, remotefolder, self.filenameList,))
976 976 p.start()
977 977
978 978 p.join(3)
979 979
980 980 if p.is_alive():
981 981 p.terminate()
982 982 p.join()
983 983 print 'killing ftp process...'
984 984 self.status = 0
985 985 return
986 986
987 987 self.status = 1
988 988 return
989 989
990 990 def filterByExt(self, ext, localfolder):
991 991 fnameList = glob.glob1(localfolder,ext)
992 992 self.filenameList = [os.path.join(localfolder,x) for x in fnameList]
993 993
994 994 if len(self.filenameList) == 0:
995 995 self.status = 0
996 996
997 997 def run(self, dataOut, ext, localfolder, remotefolder, server, username, password, period=1):
998 998
999 999 self.counter += 1
1000 1000 if self.counter >= period:
1001 1001 self.filterByExt(ext, localfolder)
1002 1002
1003 1003 self.ftp_thread(server, username, password, remotefolder)
1004 1004
1005 1005 self.counter = 0
1006 1006
1007 1007 self.status = 1
Show file before | Show file after | Hide comments
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: modified file
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
Show file before | Show file after | Hide comments
1 NO CONTENT: file was removed
The requested commit or file is too big and content was truncated. Show full diff
General Comments 0
You need to be logged in to leave comments. Login now