##// END OF EJS Templates
schain
RSS
HDFWriter create metadata if not given, update setup files
jespinoza -
r1339:2c655c5faa65 v3.0.0b4
parent child
Show More
Side by Side Unified
Context file:
r1339:2c655c5faa65 Loading diff...:
Show file before | Show file after | Hide comments
@@ -1,143 +1,141
1 1 # CHANGELOG
2 2
3 3 ## 3.0.0
4 4
5 5 * Python 3.x & 2.X compatible
6 6 * New architecture with multiprocessing support
7 7 * Add @MPDecorator for multiprocessing Operations (Plots, Writers and Publishers)
8 8 * Added new type of operation `external` for non-locking operations
9 9 * New plotting architecture with buffering/throttle capabilities to speed up plots
10 10 * Clean controller to optimize scripts (format & optype are no longer required)
11 * Replace ParamReader and ParamWriter with new flexible HDFReader and HDFWriter
12 * New GUI with dinamic load of Units and operations (use Kivy framework)
11 * Replace `ParamReader` and `ParamWriter` with new flexible `HDFReader` and `HDFWriter`
12 * New GUI with dynamic load of Units and operations (use Kivy framework)
13 13 * Clean code
14 14
15 15 ## 2.3
16 16
17 17 * Added support for Madrigal formats (reading/writing).
18 18 * Added support for reading BLTR parameters (*.sswma).
19 19 * Added support for reading Julia format (*.dat).
20 20 * Added high order function `MPProject` for multiprocessing scripts.
21 21 * Added two new Processing Units `PublishData` and `ReceiverData` for receiving and sending dataOut through multiple ways (tcp, ipc, inproc).
22 22 * Added a new graphics Processing Unit `PlotterReceiver`. It is decoupled from normal processing sequence with support for data generated by multiprocessing scripts.
23 23 * Added support for sending realtime graphic to web server.
24 24 * GUI command `schain` is now `schainGUI`.
25 25 * Added a CLI tool named `schain`.
26 26 * Scripts templates can be now generated with `schain generate`.
27 27 * Now it is possible to search Processing Units and Operations with `schain search [module]` to get the right name and its allowed parameters.
28 28 * `schain xml` to run xml scripts.
29 29 * Added suggestions when parameters are poorly written.
30 30 * `Controller.start()` now runs in a different process than the process calling it.
31 31 * Added `schainpy.utils.log` for log standarization.
32 32 * Running script on online mode no longer ignores date and hour. Issue #1109.
33 33 * Added support for receving voltage data directly from JARS (tcp, ipc).
34 34 * Updated README for MAC OS GUI installation.
35 35 * Setup now installs numpy.
36 36
37 37 ## 2.2.6
38 38
39 39 * Graphics generated by the GUI are now the same as generated by scripts. Issue #1074.
40 40 * Added support for C extensions.
41 41 * Function `hildebrand_sehkon` optimized with a C wrapper.
42 42 * Numpy version updated.
43 43 * Migration to GIT.
44 44
45 ## 2.2.5:
45 ## 2.2.5
46 46
47 47 * splitProfiles and combineProfiles modules were added to VoltageProc and Signal Chain GUI.
48 48 * nProfiles of USRP data (hdf5) is the number of profiles thera are in one second.
49 49 * jroPlotter works directly with data objects instead of dictionaries
50 50 * script "schain" was added to Signal Chain installer
51 51
52 ## 2.2.4.1:
52 ## 2.2.4.1
53 53
54 * jroIO_usrp.py is update to read Sandra's data
54 * jroIO_usrp.py is update to read Sandra's data
55 55 * decimation in Spectra and RTI plots is always enabled.
56 56 * time* window option added to GUI
57 57
58 ## 2.2.4:
58 ## 2.2.4
59 59
60 60 * jroproc_spectra_lags.py added to schainpy
61 61 * Bug fixed in schainGUI: ProcUnit was created with the same id in some cases.
62 62 * Bug fixed in jroHeaderIO: Header size validation.
63 63
64 ## 2.2.3.1:
64 ## 2.2.3.1
65 65
66 66 * Filtering block by time has been added.
67 67 * Bug fixed plotting RTI, CoherenceMap and others using xmin and xmax parameters. The first day worked
68 68 properly but the next days did not.
69 69
70 ## 2.2.3:
70 ## 2.2.3
71 71
72 72 * Bug fixed in GUI: Error getting(reading) Code value
73 73 * Bug fixed in GUI: Flip option always needs channelList field
74 74 * Bug fixed in jrodata: when one branch modified a value in "dataOut" (example: dataOut.code) this value
75 75 was modified for every branch (because this was a reference). It was modified in data.copy()
76 76 * Bug fixed in jroproc_voltage.profileSelector(): rangeList replaces to profileRangeList.
77 77
78 ## 2.2.2:
78 ## 2.2.2
79 79
80 80 * VoltageProc: ProfileSelector, Reshape, Decoder with nTxs!=1 and getblock=True was tested
81 81 * Rawdata and testRawdata.py added to Signal Chain project
82 82
83 ## 2.2.1:
83 ## 2.2.1
84 84
85 85 * Bugs fixed in GUI
86 86 * Views were improved in GUI
87 87 * Support to MST* ISR experiments
88 88 * Bug fixed getting noise using hyldebrant. (minimum number of points > 20%)
89 89 * handleError added to jroplotter.py
90 90
91 ## 2.2.0:
91 ## 2.2.0
92 92
93 93 * GUI: use of external plotter
94 94 * Compatible with matplotlib 1.5.0
95 95
96 ## 2.1.5:
96 ## 2.1.5
97 97
98 98 * serializer module added to Signal Chain
99 99 * jroplotter.py added to Signal Chain
100 100
101 ## 2.1.4.2:
101 ## 2.1.4.2
102 102
103 103 * A new Plotter Class was added
104 104 * Project.start() does not accept filename as a parameter anymore
105 105
106 ## 2.1.4.1:
106 ## 2.1.4.1
107 107
108 108 * Send notifications when an error different to ValueError is detected
109 109
110 ## 2.1.4:
110 ## 2.1.4
111 111
112 112 * Sending error notifications to signal chain administrator
113 113 * Login to email server added
114 114
115 ## 2.1.3.3:
115 ## 2.1.3.3
116 116
117 117 * Colored Button Icons were added to GUI
118 118
119 ## 2.1.3.2:
119 ## 2.1.3.2
120 120
121 121 * GUI: user interaction enhanced
122 122 * controller_api.py: Safe access to ControllerThead
123 123
124 ## 2.1.3.1:
124 ## 2.1.3.1
125 125
126 126 * GUI: every icon were resized
127 127 * jroproc_voltage.py: Print a message when "Read from code" option is selected and the code is not defined inside data file
128 128
129 ## 2.1.3:
129 ## 2.1.3
130 130
131 131 * jroplot_heispectra.py: SpectraHeisScope was not showing the right channels
132 * jroproc_voltage.py: Bug fixed selecting profiles (self.nProfiles took a wrong value),
133 Bug fixed selecting heights by block (selecting profiles instead heights)
132 * jroproc_voltage.py: Bug fixed selecting profiles (self.nProfiles took a wrong value), Bug fixed selecting heights by block (selecting profiles instead heights)
134 133 * jroproc_voltage.py: New feature added: decoding data by block using FFT.
135 134 * jroIO_heispectra.py: Bug fixed in FitsReader. Using local Fits instance instead schainpy.mode.data.jrodata.Fits.
136 135 * jroIO_heispectra.py: Channel index list does not exist.
137 136
138 ## 2.1.2:
137 ## 2.1.2
139 138
140 * jroutils_ftp.py: Bug fixed, Any error sending file stopped the Server Thread
141 Server thread opens and closes remote server each time file list is sent
139 * jroutils_ftp.py: Bug fixed, Any error sending file stopped the Server Thread Server thread opens and closes remote server each time file list is sent
142 140 * jroplot_spectra.py: Noise path was not being created when noise data is saved.
143 * jroIO_base.py: startTime can be greater than endTime. Example: SpreadF [18:00 * 07:00] No newline at end of file
141 * jroIO_base.py: startTime can be greater than endTime. Example: SpreadF
Show file before | Show file after | Hide comments
@@ -1,102 +1,104
1 1 # Signal Chain
2 2
3 3 Signal Chain is a radar data processing library wich includes modules to read,
4 4 and write different files formats, besides modules to process and visualize the
5 5 data.
6 6
7 7 ## Dependencies
8 8
9 9 - GCC (gcc or gfortran)
10 10 - Python.h (python-dev or python-devel)
11 11 - Python-TK (python-tk)
12 12 - HDF5 libraries (libhdf5-dev)
13 13
14 14 ## Installation
15 15
16 To get started the easiest way to install it is through [PyPI](https://pypi.org/project/schainpy/) with pip:
16 To get started the easiest way to install it is through
17 [PyPI](https://pypi.org/project/schainpy/) with pip. We strongly recommend to
18 use an virtual environment like virtualenv or anaconda.
17 19
18 20 ```bash
19 21 pip install schainpy
20 22 ```
21 23
22 24 ### From source
23 25
24 26 First, ensure that you have the above-listed dependencies installed, then clone
25 27 the repository and install as normal python package:
26 28
27 29 ```bash
28 git clone https://github.com/JRO-Peru/schain.git
30 git clone https://github.com/JRO-Peru/schainpy.git
29 31 cd schain
30 32 git checkout `branch-name` (optional)
31 33 sudo pip install ./
32 34 ```
33 35
34 36 ### Using Docker
35 37
36 38 Download Dockerfile from the repository, and create a docker image:
37 39
38 40 ```bash
39 41 docker build -t schain .
40 42 ```
41 43
42 44 You can run a container using an xml file or a schain script also you need to
43 45 mount a volume for the data input and for the output files/plots:
44 46
45 47 ```bash
46 48 docker run -it --rm --volume /path/to/host/data:/data schain xml /data/test.xml
47 49 docker run -it --rm --volume /path/to/host/data:/data --entrypoint /urs/local/bin/python schain /data/test.py
48 50 ```
49 51
50 52 ## CLI (command line interface)
51 53
52 54 Signal Chain provides the following commands:
53 55
54 56 - schainGUI: Open the GUI
55 57 - schain: Signal chain command line
56 58
57 59 ## Example
58 60
59 61 Here you can find an script to read Spectra data (.pdata), remove dc and plot
60 62 self-spectra & RTI:
61 63
62 64 ```python
63 65 #!/usr/bin/python
64 66
65 67 from schainpy.controller import Project
66 68
67 69 prj = Project()
68 70
69 71 read_unit = prj.addReadUnit(
70 72 datatype='Spectra',
71 73 path='/path/to/pdata/',
72 74 startDate='2014/01/31',
73 75 endDate='2014/03/31',
74 76 startTime='00:00:00',
75 77 endTime='23:59:59',
76 78 online=0,
77 79 walk=0
78 80 )
79 81
80 82 proc_unit = prj.addProcUnit(
81 83 datatype='Spectra',
82 84 inputId=read_unit.getId()
83 85 )
84 86
85 87 op = proc_unit.addOperation(name='selectChannels')
86 88 op.addParameter(name='channelList', value='0,1')
87 89
88 90 op = proc_unit.addOperation(name='selectHeights')
89 91 op.addParameter(name='minHei', value='80')
90 92 op.addParameter(name='maxHei', value='200')
91 93
92 94 op = proc_unit.addOperation(name='removeDC')
93 95
94 96 op = proc_unit.addOperation(name='SpectraPlot')
95 97 op.addParameter(name='wintitle', value='Spectra', format='str')
96 98
97 99 op = proc_unit.addOperation(name='RTIPlot')
98 100 op.addParameter(name='wintitle', value='RTI', format='str')
99 101
100 102 prj.start()
101 103
102 104 ```
Show file before | Show file after | Hide comments
@@ -1,8 +1,8
1 1 """Signal chain python package"""
2 2
3 3 try:
4 4 from .controller import Project
5 5 except:
6 6 pass
7 7
8 __version__ = '3.0.0b3'
8 __version__ = '3.0.0b4'
Show file before | Show file after | Hide comments
@@ -1,620 +1,626
1 1 import os
2 2 import time
3 3 import datetime
4 4
5 5 import numpy
6 6 import h5py
7 7
8 8 import schainpy.admin
9 9 from schainpy.model.data.jrodata import *
10 10 from schainpy.model.proc.jroproc_base import ProcessingUnit, Operation, MPDecorator
11 11 from schainpy.model.io.jroIO_base import *
12 12 from schainpy.utils import log
13 13
14 14
15 15 class HDFReader(Reader, ProcessingUnit):
16 16 """Processing unit to read HDF5 format files
17 17
18 18 This unit reads HDF5 files created with `HDFWriter` operation contains
19 19 by default two groups Data and Metadata all variables would be saved as `dataOut`
20 20 attributes.
21 21 It is possible to read any HDF5 file by given the structure in the `description`
22 22 parameter, also you can add extra values to metadata with the parameter `extras`.
23 23
24 24 Parameters:
25 25 -----------
26 26 path : str
27 27 Path where files are located.
28 28 startDate : date
29 29 Start date of the files
30 30 endDate : list
31 31 End date of the files
32 32 startTime : time
33 33 Start time of the files
34 34 endTime : time
35 35 End time of the files
36 36 description : dict, optional
37 37 Dictionary with the description of the HDF5 file
38 38 extras : dict, optional
39 39 Dictionary with extra metadata to be be added to `dataOut`
40 40
41 41 Examples
42 42 --------
43 43
44 44 desc = {
45 45 'Data': {
46 46 'data_output': ['u', 'v', 'w'],
47 47 'utctime': 'timestamps',
48 48 } ,
49 49 'Metadata': {
50 50 'heightList': 'heights'
51 51 }
52 52 }
53 53
54 54 desc = {
55 55 'Data': {
56 56 'data_output': 'winds',
57 57 'utctime': 'timestamps'
58 58 },
59 59 'Metadata': {
60 60 'heightList': 'heights'
61 61 }
62 62 }
63 63
64 64 extras = {
65 65 'timeZone': 300
66 66 }
67 67
68 68 reader = project.addReadUnit(
69 69 name='HDFReader',
70 70 path='/path/to/files',
71 71 startDate='2019/01/01',
72 72 endDate='2019/01/31',
73 73 startTime='00:00:00',
74 74 endTime='23:59:59',
75 75 # description=json.dumps(desc),
76 76 # extras=json.dumps(extras),
77 77 )
78 78
79 79 """
80 80
81 81 __attrs__ = ['path', 'startDate', 'endDate', 'startTime', 'endTime', 'description', 'extras']
82 82
83 83 def __init__(self):
84 84 ProcessingUnit.__init__(self)
85 85 self.dataOut = Parameters()
86 86 self.ext = ".hdf5"
87 87 self.optchar = "D"
88 88 self.meta = {}
89 89 self.data = {}
90 90 self.open_file = h5py.File
91 91 self.open_mode = 'r'
92 92 self.description = {}
93 93 self.extras = {}
94 94 self.filefmt = "*%Y%j***"
95 95 self.folderfmt = "*%Y%j"
96 96
97 97 def setup(self, **kwargs):
98 98
99 99 self.set_kwargs(**kwargs)
100 100 if not self.ext.startswith('.'):
101 101 self.ext = '.{}'.format(self.ext)
102 102
103 103 if self.online:
104 104 log.log("Searching files in online mode...", self.name)
105 105
106 106 for nTries in range(self.nTries):
107 107 fullpath = self.searchFilesOnLine(self.path, self.startDate,
108 108 self.endDate, self.expLabel, self.ext, self.walk,
109 109 self.filefmt, self.folderfmt)
110 110 try:
111 111 fullpath = next(fullpath)
112 112 except:
113 113 fullpath = None
114 114
115 115 if fullpath:
116 116 break
117 117
118 118 log.warning(
119 119 'Waiting {} sec for a valid file in {}: try {} ...'.format(
120 120 self.delay, self.path, nTries + 1),
121 121 self.name)
122 122 time.sleep(self.delay)
123 123
124 124 if not(fullpath):
125 125 raise schainpy.admin.SchainError(
126 126 'There isn\'t any valid file in {}'.format(self.path))
127 127
128 128 pathname, filename = os.path.split(fullpath)
129 129 self.year = int(filename[1:5])
130 130 self.doy = int(filename[5:8])
131 131 self.set = int(filename[8:11]) - 1
132 132 else:
133 133 log.log("Searching files in {}".format(self.path), self.name)
134 134 self.filenameList = self.searchFilesOffLine(self.path, self.startDate,
135 135 self.endDate, self.expLabel, self.ext, self.walk, self.filefmt, self.folderfmt)
136 136
137 137 self.setNextFile()
138 138
139 139 return
140 140
141 141 def readFirstHeader(self):
142 142 '''Read metadata and data'''
143 143
144 144 self.__readMetadata()
145 145 self.__readData()
146 146 self.__setBlockList()
147 147
148 148 if 'type' in self.meta:
149 149 self.dataOut = eval(self.meta['type'])()
150 150
151 151 for attr in self.meta:
152 152 setattr(self.dataOut, attr, self.meta[attr])
153 153
154 154 self.blockIndex = 0
155 155
156 156 return
157 157
158 158 def __setBlockList(self):
159 159 '''
160 160 Selects the data within the times defined
161 161
162 162 self.fp
163 163 self.startTime
164 164 self.endTime
165 165 self.blockList
166 166 self.blocksPerFile
167 167
168 168 '''
169 169
170 170 startTime = self.startTime
171 171 endTime = self.endTime
172 172
173 173 thisUtcTime = self.data['utctime']
174 174 self.interval = numpy.min(thisUtcTime[1:] - thisUtcTime[:-1])
175 175
176 176 thisDatetime = datetime.datetime.utcfromtimestamp(thisUtcTime[0])
177 177
178 178 thisDate = thisDatetime.date()
179 179 thisTime = thisDatetime.time()
180 180
181 181 startUtcTime = (datetime.datetime.combine(thisDate, startTime) - datetime.datetime(1970, 1, 1)).total_seconds()
182 182 endUtcTime = (datetime.datetime.combine(thisDate, endTime) - datetime.datetime(1970, 1, 1)).total_seconds()
183 183
184 184 ind = numpy.where(numpy.logical_and(thisUtcTime >= startUtcTime, thisUtcTime < endUtcTime))[0]
185 185
186 186 self.blockList = ind
187 187 self.blocksPerFile = len(ind)
188 188 return
189 189
190 190 def __readMetadata(self):
191 191 '''
192 192 Reads Metadata
193 193 '''
194 194
195 195 meta = {}
196 196
197 197 if self.description:
198 198 for key, value in self.description['Metadata'].items():
199 199 meta[key] = self.fp[value].value
200 200 else:
201 201 grp = self.fp['Metadata']
202 202 for name in grp:
203 203 meta[name] = grp[name].value
204 204
205 205 if self.extras:
206 206 for key, value in self.extras.items():
207 207 meta[key] = value
208 208 self.meta = meta
209 209
210 210 return
211 211
212 212 def __readData(self):
213 213
214 214 data = {}
215 215
216 216 if self.description:
217 217 for key, value in self.description['Data'].items():
218 218 if isinstance(value, str):
219 219 if isinstance(self.fp[value], h5py.Dataset):
220 220 data[key] = self.fp[value].value
221 221 elif isinstance(self.fp[value], h5py.Group):
222 222 array = []
223 223 for ch in self.fp[value]:
224 224 array.append(self.fp[value][ch].value)
225 225 data[key] = numpy.array(array)
226 226 elif isinstance(value, list):
227 227 array = []
228 228 for ch in value:
229 229 array.append(self.fp[ch].value)
230 230 data[key] = numpy.array(array)
231 231 else:
232 232 grp = self.fp['Data']
233 233 for name in grp:
234 234 if isinstance(grp[name], h5py.Dataset):
235 235 array = grp[name].value
236 236 elif isinstance(grp[name], h5py.Group):
237 237 array = []
238 238 for ch in grp[name]:
239 239 array.append(grp[name][ch].value)
240 240 array = numpy.array(array)
241 241 else:
242 242 log.warning('Unknown type: {}'.format(name))
243 243
244 244 if name in self.description:
245 245 key = self.description[name]
246 246 else:
247 247 key = name
248 248 data[key] = array
249 249
250 250 self.data = data
251 251 return
252 252
253 253 def getData(self):
254 254
255 255 for attr in self.data:
256 256 if self.data[attr].ndim == 1:
257 257 setattr(self.dataOut, attr, self.data[attr][self.blockIndex])
258 258 else:
259 259 setattr(self.dataOut, attr, self.data[attr][:, self.blockIndex])
260 260
261 261 self.dataOut.flagNoData = False
262 262 self.blockIndex += 1
263 263
264 264 log.log("Block No. {}/{} -> {}".format(
265 265 self.blockIndex,
266 266 self.blocksPerFile,
267 267 self.dataOut.datatime.ctime()), self.name)
268 268
269 269 return
270 270
271 271 def run(self, **kwargs):
272 272
273 273 if not(self.isConfig):
274 274 self.setup(**kwargs)
275 275 self.isConfig = True
276 276
277 277 if self.blockIndex == self.blocksPerFile:
278 278 self.setNextFile()
279 279
280 280 self.getData()
281 281
282 282 return
283 283
284 284 @MPDecorator
285 285 class HDFWriter(Operation):
286 286 """Operation to write HDF5 files.
287 287
288 288 The HDF5 file contains by default two groups Data and Metadata where
289 289 you can save any `dataOut` attribute specified by `dataList` and `metadataList`
290 290 parameters, data attributes are normaly time dependent where the metadata
291 291 are not.
292 292 It is possible to customize the structure of the HDF5 file with the
293 293 optional description parameter see the examples.
294 294
295 295 Parameters:
296 296 -----------
297 297 path : str
298 298 Path where files will be saved.
299 299 blocksPerFile : int
300 300 Number of blocks per file
301 301 metadataList : list
302 302 List of the dataOut attributes that will be saved as metadata
303 303 dataList : int
304 304 List of the dataOut attributes that will be saved as data
305 305 setType : bool
306 306 If True the name of the files corresponds to the timestamp of the data
307 307 description : dict, optional
308 308 Dictionary with the desired description of the HDF5 file
309 309
310 310 Examples
311 311 --------
312 312
313 313 desc = {
314 314 'data_output': {'winds': ['z', 'w', 'v']},
315 315 'utctime': 'timestamps',
316 316 'heightList': 'heights'
317 317 }
318 318 desc = {
319 319 'data_output': ['z', 'w', 'v'],
320 320 'utctime': 'timestamps',
321 321 'heightList': 'heights'
322 322 }
323 323 desc = {
324 324 'Data': {
325 325 'data_output': 'winds',
326 326 'utctime': 'timestamps'
327 327 },
328 328 'Metadata': {
329 329 'heightList': 'heights'
330 330 }
331 331 }
332 332
333 333 writer = proc_unit.addOperation(name='HDFWriter')
334 334 writer.addParameter(name='path', value='/path/to/file')
335 335 writer.addParameter(name='blocksPerFile', value='32')
336 336 writer.addParameter(name='metadataList', value='heightList,timeZone')
337 337 writer.addParameter(name='dataList',value='data_output,utctime')
338 338 # writer.addParameter(name='description',value=json.dumps(desc))
339 339
340 340 """
341 341
342 342 ext = ".hdf5"
343 343 optchar = "D"
344 344 filename = None
345 345 path = None
346 346 setFile = None
347 347 fp = None
348 348 firsttime = True
349 349 #Configurations
350 350 blocksPerFile = None
351 351 blockIndex = None
352 352 dataOut = None
353 353 #Data Arrays
354 354 dataList = None
355 355 metadataList = None
356 356 currentDay = None
357 357 lastTime = None
358 358
359 359 def __init__(self):
360 360
361 361 Operation.__init__(self)
362 362 return
363 363
364 364 def setup(self, path=None, blocksPerFile=10, metadataList=None, dataList=None, setType=None, description=None):
365 365 self.path = path
366 366 self.blocksPerFile = blocksPerFile
367 367 self.metadataList = metadataList
368 self.dataList = dataList
368 self.dataList = [s.strip() for s in dataList]
369 369 self.setType = setType
370 370 self.description = description
371 371
372 for s in ['type', 'timeZone', 'useLocalTime']:
373 if s not in self.metadataList:
374 self.metadataList.append(s)
372 if self.metadataList is None:
373 self.metadataList = self.dataOut.metadata_list
375 374
376 375 tableList = []
377 376 dsList = []
378 377
379 378 for i in range(len(self.dataList)):
380 379 dsDict = {}
381 dataAux = getattr(self.dataOut, self.dataList[i])
382 dsDict['variable'] = self.dataList[i]
380 if hasattr(self.dataOut, self.dataList[i]):
381 dataAux = getattr(self.dataOut, self.dataList[i])
382 dsDict['variable'] = self.dataList[i]
383 else:
384 log.warning('Attribute {} not found in dataOut', self.name)
385 continue
383 386
384 387 if dataAux is None:
385 388 continue
386 389 elif isinstance(dataAux, (int, float, numpy.integer, numpy.float)):
387 390 dsDict['nDim'] = 0
388 391 else:
389 392 dsDict['nDim'] = len(dataAux.shape)
390 393 dsDict['shape'] = dataAux.shape
391 394 dsDict['dsNumber'] = dataAux.shape[0]
392 395 dsDict['dtype'] = dataAux.dtype
393 396
394 397 dsList.append(dsDict)
395 398
396 399 self.dsList = dsList
397 400 self.currentDay = self.dataOut.datatime.date()
398 401
399 402 def timeFlag(self):
400 403 currentTime = self.dataOut.utctime
401 404 timeTuple = time.localtime(currentTime)
402 405 dataDay = timeTuple.tm_yday
403 406
404 407 if self.lastTime is None:
405 408 self.lastTime = currentTime
406 409 self.currentDay = dataDay
407 410 return False
408 411
409 412 timeDiff = currentTime - self.lastTime
410 413
411 414 #Si el dia es diferente o si la diferencia entre un dato y otro supera la hora
412 415 if dataDay != self.currentDay:
413 416 self.currentDay = dataDay
414 417 return True
415 418 elif timeDiff > 3*60*60:
416 419 self.lastTime = currentTime
417 420 return True
418 421 else:
419 422 self.lastTime = currentTime
420 423 return False
421 424
422 def run(self, dataOut, path, blocksPerFile=10, metadataList=[],
425 def run(self, dataOut, path, blocksPerFile=10, metadataList=None,
423 426 dataList=[], setType=None, description={}):
424 427
425 428 self.dataOut = dataOut
426 429 if not(self.isConfig):
427 430 self.setup(path=path, blocksPerFile=blocksPerFile,
428 431 metadataList=metadataList, dataList=dataList,
429 432 setType=setType, description=description)
430 433
431 434 self.isConfig = True
432 435 self.setNextFile()
433 436
434 437 self.putData()
435 438 return
436 439
437 440 def setNextFile(self):
438 441
439 442 ext = self.ext
440 443 path = self.path
441 444 setFile = self.setFile
442 445
443 446 timeTuple = time.localtime(self.dataOut.utctime)
444 447 subfolder = 'd%4.4d%3.3d' % (timeTuple.tm_year,timeTuple.tm_yday)
445 448 fullpath = os.path.join(path, subfolder)
446 449
447 450 if os.path.exists(fullpath):
448 451 filesList = os.listdir(fullpath)
449 452 filesList = [k for k in filesList if k.startswith(self.optchar)]
450 453 if len( filesList ) > 0:
451 454 filesList = sorted(filesList, key=str.lower)
452 455 filen = filesList[-1]
453 456 # el filename debera tener el siguiente formato
454 457 # 0 1234 567 89A BCDE (hex)
455 458 # x YYYY DDD SSS .ext
456 459 if isNumber(filen[8:11]):
457 460 setFile = int(filen[8:11]) #inicializo mi contador de seteo al seteo del ultimo file
458 461 else:
459 462 setFile = -1
460 463 else:
461 464 setFile = -1 #inicializo mi contador de seteo
462 465 else:
463 466 os.makedirs(fullpath)
464 467 setFile = -1 #inicializo mi contador de seteo
465 468
466 469 if self.setType is None:
467 470 setFile += 1
468 471 file = '%s%4.4d%3.3d%03d%s' % (self.optchar,
469 472 timeTuple.tm_year,
470 473 timeTuple.tm_yday,
471 474 setFile,
472 475 ext )
473 476 else:
474 477 setFile = timeTuple.tm_hour*60+timeTuple.tm_min
475 478 file = '%s%4.4d%3.3d%04d%s' % (self.optchar,
476 479 timeTuple.tm_year,
477 480 timeTuple.tm_yday,
478 481 setFile,
479 482 ext )
480 483
481 484 self.filename = os.path.join( path, subfolder, file )
482 485
483 486 #Setting HDF5 File
484 487 self.fp = h5py.File(self.filename, 'w')
485 488 #write metadata
486 489 self.writeMetadata(self.fp)
487 490 #Write data
488 491 self.writeData(self.fp)
489 492
490 493 def getLabel(self, name, x=None):
491 494
492 495 if x is None:
493 496 if 'Data' in self.description:
494 497 data = self.description['Data']
495 498 if 'Metadata' in self.description:
496 499 data.update(self.description['Metadata'])
497 500 else:
498 501 data = self.description
499 502 if name in data:
500 503 if isinstance(data[name], str):
501 504 return data[name]
502 505 elif isinstance(data[name], list):
503 506 return None
504 507 elif isinstance(data[name], dict):
505 508 for key, value in data[name].items():
506 509 return key
507 510 return name
508 511 else:
509 512 if 'Metadata' in self.description:
510 513 meta = self.description['Metadata']
511 514 else:
512 515 meta = self.description
513 516 if name in meta:
514 517 if isinstance(meta[name], list):
515 518 return meta[name][x]
516 519 elif isinstance(meta[name], dict):
517 520 for key, value in meta[name].items():
518 521 return value[x]
519 return 'channel{:02d}'.format(x)
522 if 'cspc' in name:
523 return 'pair{:02d}'.format(x)
524 else:
525 return 'channel{:02d}'.format(x)
520 526
521 527 def writeMetadata(self, fp):
522 528
523 529 if self.description:
524 530 if 'Metadata' in self.description:
525 531 grp = fp.create_group('Metadata')
526 532 else:
527 533 grp = fp
528 534 else:
529 535 grp = fp.create_group('Metadata')
530 536
531 537 for i in range(len(self.metadataList)):
532 538 if not hasattr(self.dataOut, self.metadataList[i]):
533 539 log.warning('Metadata: `{}` not found'.format(self.metadataList[i]), self.name)
534 540 continue
535 541 value = getattr(self.dataOut, self.metadataList[i])
536 542 if isinstance(value, bool):
537 543 if value is True:
538 544 value = 1
539 545 else:
540 546 value = 0
541 547 grp.create_dataset(self.getLabel(self.metadataList[i]), data=value)
542 548 return
543 549
544 550 def writeData(self, fp):
545 551
546 552 if self.description:
547 553 if 'Data' in self.description:
548 554 grp = fp.create_group('Data')
549 555 else:
550 556 grp = fp
551 557 else:
552 558 grp = fp.create_group('Data')
553 559
554 560 dtsets = []
555 561 data = []
556 562
557 563 for dsInfo in self.dsList:
558 564 if dsInfo['nDim'] == 0:
559 565 ds = grp.create_dataset(
560 566 self.getLabel(dsInfo['variable']),
561 567 (self.blocksPerFile, ),
562 568 chunks=True,
563 569 dtype=numpy.float64)
564 570 dtsets.append(ds)
565 571 data.append((dsInfo['variable'], -1))
566 572 else:
567 573 label = self.getLabel(dsInfo['variable'])
568 574 if label is not None:
569 575 sgrp = grp.create_group(label)
570 576 else:
571 577 sgrp = grp
572 578 for i in range(dsInfo['dsNumber']):
573 579 ds = sgrp.create_dataset(
574 580 self.getLabel(dsInfo['variable'], i),
575 581 (self.blocksPerFile, ) + dsInfo['shape'][1:],
576 582 chunks=True,
577 583 dtype=dsInfo['dtype'])
578 584 dtsets.append(ds)
579 585 data.append((dsInfo['variable'], i))
580 586 fp.flush()
581 587
582 588 log.log('Creating file: {}'.format(fp.filename), self.name)
583 589
584 590 self.ds = dtsets
585 591 self.data = data
586 592 self.firsttime = True
587 593 self.blockIndex = 0
588 594 return
589 595
590 596 def putData(self):
591 597
592 598 if (self.blockIndex == self.blocksPerFile) or self.timeFlag():
593 599 self.closeFile()
594 600 self.setNextFile()
595 601
596 602 for i, ds in enumerate(self.ds):
597 603 attr, ch = self.data[i]
598 604 if ch == -1:
599 605 ds[self.blockIndex] = getattr(self.dataOut, attr)
600 606 else:
601 607 ds[self.blockIndex] = getattr(self.dataOut, attr)[ch]
602 608
603 609 self.fp.flush()
604 610 self.blockIndex += 1
605 611 log.log('Block No. {}/{}'.format(self.blockIndex, self.blocksPerFile), self.name)
606 612
607 613 return
608 614
609 615 def closeFile(self):
610 616
611 617 if self.blockIndex != self.blocksPerFile:
612 618 for ds in self.ds:
613 619 ds.resize(self.blockIndex, axis=0)
614 620
615 621 self.fp.flush()
616 622 self.fp.close()
617 623
618 624 def close(self):
619 625
620 626 self.closeFile()
Show file before | Show file after | Hide comments
@@ -1,86 +1,86
1 1 # Copyright (c) 2012-2020 Jicamarca Radio Observatory
2 2 # All rights reserved.
3 3 #
4 4 # Distributed under the terms of the BSD 3-clause license.
5 5 """schainpy is an open source library to read, write and process radar data
6 6
7 7 Signal Chain is a radar data processing library wich includes modules to read,
8 8 and write different files formats, besides modules to process and visualize the
9 9 data.
10 10 """
11 11
12 12 import os
13 13 from setuptools import setup, Extension
14 14 from setuptools.command.build_ext import build_ext as _build_ext
15 15 from schainpy import __version__
16 16
17 17 DOCLINES = __doc__.split("\n")
18 18
19 19 class build_ext(_build_ext):
20 20 def finalize_options(self):
21 21 _build_ext.finalize_options(self)
22 22 # Prevent numpy from thinking it is still in its setup process:
23 23 __builtins__.__NUMPY_SETUP__ = False
24 24 import numpy
25 25 self.include_dirs.append(numpy.get_include())
26 26
27 27 setup(
28 28 name = "schainpy",
29 29 version = __version__,
30 30 description = DOCLINES[0],
31 31 long_description = "\n".join(DOCLINES[2:]),
32 url = "https://github.com/JRO-Peru/schain",
32 url = "https://github.com/JRO-Peru/schainpy",
33 33 author = "Jicamarca Radio Observatory",
34 author_email = "jro-developers@igp.gob.pe",
34 author_email = "jro-developers@jro.igp.gob.pe",
35 35 license="BSD-3-Clause",
36 36 classifiers=[
37 37 "Development Status :: 4 - Beta",
38 38 "Environment :: Console",
39 39 "Intended Audience :: Science/Research",
40 40 "License :: OSI Approved :: BSD License",
41 41 "Operating System :: MacOS :: MacOS X",
42 42 "Operating System :: POSIX :: Linux",
43 43 "Programming Language :: Python :: 2",
44 44 "Programming Language :: Python :: 2.7",
45 45 "Programming Language :: Python :: 3",
46 46 "Programming Language :: Python :: 3.5",
47 47 "Programming Language :: Python :: 3.6",
48 48 "Programming Language :: Python :: 3.7",
49 49 "Topic :: Scientific/Engineering",
50 50 ],
51 51 packages = {
52 52 'schainpy',
53 53 'schainpy.model',
54 54 'schainpy.model.data',
55 55 'schainpy.model.graphics',
56 56 'schainpy.model.io',
57 57 'schainpy.model.proc',
58 58 'schainpy.model.utils',
59 59 'schainpy.utils',
60 60 'schainpy.gui',
61 61 'schainpy.cli',
62 62 },
63 63 package_data = {'': ['schain.conf.template'],
64 64 'schainpy.files': ['*.oga']
65 65 },
66 66 include_package_data = False,
67 67 scripts = ['schainpy/gui/schainGUI'],
68 68 entry_points = {
69 69 'console_scripts': [
70 70 'schain = schainpy.cli.cli:main',
71 71 ],
72 72 },
73 73 cmdclass = {'build_ext': build_ext},
74 74 ext_modules=[
75 75 Extension("schainpy.model.data._noise", ["schainc/_noise.c"]),
76 76 ],
77 77 setup_requires = ["numpy"],
78 78 install_requires = [
79 79 "scipy",
80 80 "h5py",
81 81 "matplotlib",
82 82 "pyzmq",
83 83 "fuzzywuzzy",
84 84 "click",
85 85 ],
86 86 )
General Comments 0
You need to be logged in to leave comments. Login now