##// END OF EJS Templates
schain
RSS
Add license, update setup and changelog
Juan C. Espinoza -
r1329:16056ee07511
parent child
Show More
Side by Side Unified
Context file:
r1329:16056ee07511 Loading diff...:
Show file before | Show file after | Hide comments
@@ -0,0 +1,28
1 This software, unless otherwise noted, is licensed under the BSD 3-clause.
2
3 Copyright (c) 2012-2020 Jicamarca Radio Observatory
4
5 Redistribution and use in source and binary forms, with or without
6 modification, are permitted provided that the following conditions are met:
7
8 * Redistributions of source code must retain the above copyright notice, this
9 list of conditions and the following disclaimer.
10
11 * Redistributions in binary form must reproduce the above copyright notice,
12 this list of conditions and the following disclaimer in the documentation
13 and/or other materials provided with the distribution.
14
15 * Neither the name of the copyright holder nor the names of its
16 contributors may be used to endorse or promote products derived from
17 this software without specific prior written permission.
18
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
20 AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
22 DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
23 FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
24 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
25 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
26 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
27 OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
28 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Show file before | Show file after | Hide comments
@@ -1,114 +1,102
1 1 # Byte-compiled / optimized / DLL files
2 2 __pycache__/
3 3 *.py[cod]
4 4 *$py.class
5 5
6 6 # C extensions
7 7 *.so
8 8
9 9 # Distribution / packaging
10 10 .Python
11 11 env/
12 12 build/
13 13 develop-eggs/
14 14 dist/
15 15 downloads/
16 16 eggs/
17 17 .eggs/
18 18 lib/
19 19 lib64/
20 20 parts/
21 21 sdist/
22 22 var/
23 23 wheels/
24 24 *.egg-info/
25 25 .installed.cfg
26 26 *.egg
27 27
28 28 # PyInstaller
29 29 # Usually these files are written by a python script from a template
30 30 # before PyInstaller builds the exe, so as to inject date/other infos into it.
31 31 *.manifest
32 32 *.spec
33 33
34 34 # Installer logs
35 35 pip-log.txt
36 36 pip-delete-this-directory.txt
37 37
38 38 # Unit test / coverage reports
39 39 htmlcov/
40 40 .tox/
41 41 .coverage
42 42 .coverage.*
43 43 .cache
44 44 nosetests.xml
45 45 coverage.xml
46 46 *,cover
47 47 .hypothesis/
48 48
49 49 # Translations
50 50 *.mo
51 51 *.pot
52 52
53 53 # Django stuff:
54 54 *.log
55 55 local_settings.py
56 56
57 57 # Flask stuff:
58 58 instance/
59 59 .webassets-cache
60 60
61 61 # Scrapy stuff:
62 62 .scrapy
63 63
64 64 # Sphinx documentation
65 65 docs/_build/
66 66
67 67 # PyBuilder
68 68 target/
69 69
70 70 # Jupyter Notebook
71 71 .ipynb_checkpoints
72 72
73 73 # pyenv
74 74 .python-version
75 75
76 76 # celery beat schedule file
77 77 celerybeat-schedule
78 78
79 79 # SageMath parsed files
80 80 *.sage.py
81 81
82 82 # dotenv
83 83 .env
84 84
85 85 # virtualenv
86 86 .venv
87 87 venv/
88 88 ENV/
89 89
90 90 # Spyder project settings
91 91 .spyderproject
92 92 .spyproject
93 93
94 94 # Rope project settings
95 95 .ropeproject
96 96
97 97 # mkdocs documentation
98 98 /site
99 99
100 100 # eclipse
101 101 .project
102 102 .pydevproject
103 # vscode
104
105 .vscode
106
107 schaingui/node_modules/
108 schainpy/scripts/
109 .svn/
110 *.png
111 *.pyc
112 .vscode
113 trash
114 *.log
Show file before | Show file after | Hide comments
@@ -1,121 +1,143
1 # CHANGELOG:
1 # CHANGELOG
2
3 ## 3.0.0
2 4
3 ## 3.0
4 5 * Python 3.x & 2.X compatible
5 6 * New architecture with multiprocessing support
6 7 * Add @MPDecorator for multiprocessing Operations (Plots, Writers and Publishers)
7 8 * Added new type of operation `external` for non-locking operations
8 9 * New plotting architecture with buffering/throttle capabilities to speed up plots
9 10 * Clean controller to optimize scripts (format & optype are no longer required)
11 * Replace ParamReader and ParamWriter with new flexible HDFReader and HDFWriter
10 12 * New GUI with dinamic load of Units and operations (use Kivy framework)
13 * Clean code
11 14
12 15 ## 2.3
16
13 17 * Added support for Madrigal formats (reading/writing).
14 18 * Added support for reading BLTR parameters (*.sswma).
15 19 * Added support for reading Julia format (*.dat).
16 20 * Added high order function `MPProject` for multiprocessing scripts.
17 21 * Added two new Processing Units `PublishData` and `ReceiverData` for receiving and sending dataOut through multiple ways (tcp, ipc, inproc).
18 22 * Added a new graphics Processing Unit `PlotterReceiver`. It is decoupled from normal processing sequence with support for data generated by multiprocessing scripts.
19 23 * Added support for sending realtime graphic to web server.
20 * GUI command `schain` is now `schainGUI`.
21 * Added a CLI tool named `schain`.
22 * Scripts templates can be now generated with `schain generate`.
23 * Now it is possible to search Processing Units and Operations with `schain search [module]` to get the right name and its allowed parameters.
24 * `schain xml` to run xml scripts.
24 * GUI command `schain` is now `schainGUI`.
25 * Added a CLI tool named `schain`.
26 * Scripts templates can be now generated with `schain generate`.
27 * Now it is possible to search Processing Units and Operations with `schain search [module]` to get the right name and its allowed parameters.
28 * `schain xml` to run xml scripts.
25 29 * Added suggestions when parameters are poorly written.
26 30 * `Controller.start()` now runs in a different process than the process calling it.
27 31 * Added `schainpy.utils.log` for log standarization.
28 32 * Running script on online mode no longer ignores date and hour. Issue #1109.
29 33 * Added support for receving voltage data directly from JARS (tcp, ipc).
30 34 * Updated README for MAC OS GUI installation.
31 35 * Setup now installs numpy.
32 36
33 37 ## 2.2.6
38
34 39 * Graphics generated by the GUI are now the same as generated by scripts. Issue #1074.
35 40 * Added support for C extensions.
36 41 * Function `hildebrand_sehkon` optimized with a C wrapper.
37 42 * Numpy version updated.
38 43 * Migration to GIT.
39 44
40 45 ## 2.2.5:
46
41 47 * splitProfiles and combineProfiles modules were added to VoltageProc and Signal Chain GUI.
42 48 * nProfiles of USRP data (hdf5) is the number of profiles thera are in one second.
43 49 * jroPlotter works directly with data objects instead of dictionaries
44 50 * script "schain" was added to Signal Chain installer
45 51
46 52 ## 2.2.4.1:
53
47 54 * jroIO_usrp.py is update to read Sandra's data
48 55 * decimation in Spectra and RTI plots is always enabled.
49 56 * time* window option added to GUI
50 57
51 58 ## 2.2.4:
59
52 60 * jroproc_spectra_lags.py added to schainpy
53 61 * Bug fixed in schainGUI: ProcUnit was created with the same id in some cases.
54 62 * Bug fixed in jroHeaderIO: Header size validation.
55 63
56 64 ## 2.2.3.1:
65
57 66 * Filtering block by time has been added.
58 67 * Bug fixed plotting RTI, CoherenceMap and others using xmin and xmax parameters. The first day worked
59 68 properly but the next days did not.
60 69
61 70 ## 2.2.3:
71
62 72 * Bug fixed in GUI: Error getting(reading) Code value
63 73 * Bug fixed in GUI: Flip option always needs channelList field
64 74 * Bug fixed in jrodata: when one branch modified a value in "dataOut" (example: dataOut.code) this value
65 75 was modified for every branch (because this was a reference). It was modified in data.copy()
66 76 * Bug fixed in jroproc_voltage.profileSelector(): rangeList replaces to profileRangeList.
67 77
68 78 ## 2.2.2:
79
69 80 * VoltageProc: ProfileSelector, Reshape, Decoder with nTxs!=1 and getblock=True was tested
70 81 * Rawdata and testRawdata.py added to Signal Chain project
71 82
72 83 ## 2.2.1:
84
73 85 * Bugs fixed in GUI
74 86 * Views were improved in GUI
75 87 * Support to MST* ISR experiments
76 88 * Bug fixed getting noise using hyldebrant. (minimum number of points > 20%)
77 89 * handleError added to jroplotter.py
78 90
79 91 ## 2.2.0:
92
80 93 * GUI: use of external plotter
81 94 * Compatible with matplotlib 1.5.0
82 95
83 96 ## 2.1.5:
97
84 98 * serializer module added to Signal Chain
85 99 * jroplotter.py added to Signal Chain
86 100
87 101 ## 2.1.4.2:
102
88 103 * A new Plotter Class was added
89 104 * Project.start() does not accept filename as a parameter anymore
90 105
91 106 ## 2.1.4.1:
107
92 108 * Send notifications when an error different to ValueError is detected
93 109
94 110 ## 2.1.4:
111
95 112 * Sending error notifications to signal chain administrator
96 113 * Login to email server added
97 114
98 115 ## 2.1.3.3:
116
99 117 * Colored Button Icons were added to GUI
100 118
101 119 ## 2.1.3.2:
120
102 121 * GUI: user interaction enhanced
103 122 * controller_api.py: Safe access to ControllerThead
104 123
105 124 ## 2.1.3.1:
125
106 126 * GUI: every icon were resized
107 127 * jroproc_voltage.py: Print a message when "Read from code" option is selected and the code is not defined inside data file
108 128
109 129 ## 2.1.3:
130
110 131 * jroplot_heispectra.py: SpectraHeisScope was not showing the right channels
111 132 * jroproc_voltage.py: Bug fixed selecting profiles (self.nProfiles took a wrong value),
112 133 Bug fixed selecting heights by block (selecting profiles instead heights)
113 134 * jroproc_voltage.py: New feature added: decoding data by block using FFT.
114 135 * jroIO_heispectra.py: Bug fixed in FitsReader. Using local Fits instance instead schainpy.mode.data.jrodata.Fits.
115 136 * jroIO_heispectra.py: Channel index list does not exist.
116 137
117 138 ## 2.1.2:
139
118 140 * jroutils_ftp.py: Bug fixed, Any error sending file stopped the Server Thread
119 141 Server thread opens and closes remote server each time file list is sent
120 142 * jroplot_spectra.py: Noise path was not being created when noise data is saved.
121 143 * jroIO_base.py: startTime can be greater than endTime. Example: SpreadF [18:00 * 07:00] No newline at end of file
Show file before | Show file after | Hide comments
@@ -1,158 +1,102
1 1 # Signal Chain
2 2
3 ## Introduction
3 Signal Chain is a radar data processing library wich includes modules to read,
4 and write different files formats, besides modules to process and visualize the
5 data.
4 6
5 Signal Chain (SCh) is a radar data processing library developed using [Python](www.python.org) at JRO. SCh provides modules to read, write, process and plot data.
7 ## Dependencies
6 8
7 ## Installation
8
9 Install system dependencies, clone the latest version from [here](http://jro-dev.igp.gob.pe/rhodecode/schain/) and install it as a normal python package, we strongly recommend to use Anaconda or a virtual environment for the installation.
10
11 ### Dependencies
12 9 - GCC (gcc or gfortran)
13 10 - Python.h (python-dev or python-devel)
14 11 - Python-TK (python-tk)
15 12 - HDF5 libraries (libhdf5-dev)
16 13
17 ### Linux based system (e.g. ubuntu)
18 ```
19 $ git clone http://jro-dev.igp.gob.pe/rhodecode/schain/
20 $ cd schain
21 $ git checkout `schain-branch` (optional)
22 $ sudo pip install ./
23 ```
14 ## Installation
24 15
25 ### MAC Os
26 ```
27 $ brew install python
28 $ git clone http://jro-dev.igp.gob.pe/rhodecode/schain/
29 $ cd schain
30 $ git checkout `schain-branch` (optional)
31 $ sudo pip install ./
16 To get started the easiest way to install it is through [PyPI](https://pypi.org/project/schainpy/) with pip:
17
18 ```bash
19 pip install schainpy
32 20 ```
33 21
34 ### Docker
22 ### From source
35 23
36 Download Dockerfile from the repository, and create a docker image
24 First, ensure that you have the above-listed dependencies installed, then clone
25 the repository and install as normal python package:
37 26
27 ```bash
28 git clone https://github.com/JRO-Peru/schain.git
29 cd schain
30 git checkout `branch-name` (optional)
31 sudo pip install ./
38 32 ```
39 $ docker build -t schain .
40 ```
41 33
42 You can run a container using an xml file or a schain script also you need to mount a volume for the data input and for the output files/plots
34 ### Using Docker
35
36 Download Dockerfile from the repository, and create a docker image:
37
38 ```bash
39 docker build -t schain .
43 40 ```
44 $ docker run -it --rm --volume /path/to/host/data:/data schain xml /data/test.xml
45 $ docker run -it --rm --volume /path/to/host/data:/data --entrypoint /urs/local/bin/python schain /data/test.py
41
42 You can run a container using an xml file or a schain script also you need to
43 mount a volume for the data input and for the output files/plots:
44
45 ```bash
46 docker run -it --rm --volume /path/to/host/data:/data schain xml /data/test.xml
47 docker run -it --rm --volume /path/to/host/data:/data --entrypoint /urs/local/bin/python schain /data/test.py
46 48 ```
47 49
48 50 ## CLI (command line interface)
49 51
50 52 Signal Chain provides the following commands:
51 53
52 54 - schainGUI: Open the GUI
53 55 - schain: Signal chain command line
54 56
57 ## Example
55 58
56 ## First Script
57
58 Here you can find an script to read Spectra data (.pdata), remove dc and plot spectra & RTI
59
60 First import SCh and creating a project
59 Here you can find an script to read Spectra data (.pdata), remove dc and plot
60 self-spectra & RTI:
61 61
62 62 ```python
63 63 #!/usr/bin/python
64 64
65 65 from schainpy.controller import Project
66 66
67 67 prj = Project()
68 prj.setup(
69 id = '100',
70 name='test',
71 description='Basic experiment'
72 )
73 ```
74 68
75 Add read unit and operations
76
77 ```python
78 69 read_unit = prj.addReadUnit(
79 70 datatype='Spectra',
80 71 path='/path/to/pdata/',
81 72 startDate='2014/01/31',
82 73 endDate='2014/03/31',
83 74 startTime='00:00:00',
84 75 endTime='23:59:59',
85 76 online=0,
86 77 walk=0
87 78 )
88 79
89 proc_unit = prj.addProcUnit(datatype='Spectra', inputId=read_unit.getId())
90
91 op = proc_unit.addOperation(name='selectChannels')
92 op.addParameter(name='channelList', value='0,1')
93
94 op = proc_unit.addOperation(name='selectHeights')
95 op.addParameter(name='minHei', value='80')
96 op.addParameter(name='maxHei', value='200')
97
98 op = proc_unit.addOperation(name='removeDC')
99
100 ```
101
102 Plot data & start project
103
104 ```python
105 op = proc_unit.addOperation(name='SpectraPlot')
106 op.addParameter(name='id', value='1')
107 op.addParameter(name='wintitle', value='Spectra')
108
109 op = procUnitConfObj1.addOperation(name='RTIPlot')
110 op.addParameter(name='id', value='2')
111 op.addParameter(name='wintitle', value='RTI')
112
113 prj.start()
114
115 ```
116
117 Full script
118
119
120 ```python
121 #!/usr/bin/python
122
123 from schainpy.prj import Project
124
125 prj = Project()
126 prj.setup(id = '100',
127 name='test',
128 description='Basic experiment')
129 read_unit = prj.addReadUnit(datatype='Spectra',
130 path='/path/to/pdata/',
131 startDate='2014/01/31',
132 endDate='2014/03/31',
133 startTime='00:00:00',
134 endTime='23:59:59',
135 online=0,
136 walk=0)
137
138 proc_unit = prj.addProcUnit(datatype='Spectra',
139 inputId=read_unit.getId())
80 proc_unit = prj.addProcUnit(
81 datatype='Spectra',
82 inputId=read_unit.getId()
83 )
140 84
141 85 op = proc_unit.addOperation(name='selectChannels')
142 86 op.addParameter(name='channelList', value='0,1')
143 87
144 88 op = proc_unit.addOperation(name='selectHeights')
145 89 op.addParameter(name='minHei', value='80')
146 90 op.addParameter(name='maxHei', value='200')
147 91
148 92 op = proc_unit.addOperation(name='removeDC')
149 93
150 94 op = proc_unit.addOperation(name='SpectraPlot')
151 95 op.addParameter(name='wintitle', value='Spectra', format='str')
152 96
153 97 op = procUnitConfObj1.addOperation(name='RTIPlot')
154 98 op.addParameter(name='wintitle', value='RTI', format='str')
155 99
156 100 prj.start()
157 101
158 ``` No newline at end of file
102 ```
Show file before | Show file after | Hide comments
@@ -1,7 +1,5
1 '''
2 Created on Jul 3, 2018
1 """Signal chain python package"""
3 2
4 @author $Author$
5 @version $Id$
6 '''
7 __version__ = '3.0'
3 from .controller import Project
4
5 __version__ = '3.0.0'
Show file before | Show file after | Hide comments
@@ -1,648 +1,656
1 '''
2 Main routines to create a Signal Chain project
3 '''
1 # Copyright (c) 2012-2020 Jicamarca Radio Observatory
2 # All rights reserved.
3 #
4 # Distributed under the terms of the BSD 3-clause license.
5 """API to create signal chain projects
6
7 The API is provide through class: Project
8 """
4 9
5 10 import re
6 11 import sys
7 12 import ast
8 13 import datetime
9 14 import traceback
10 15 import time
11 16 from multiprocessing import Process, Queue
12 17 from threading import Thread
13 18 from xml.etree.ElementTree import ElementTree, Element, SubElement
14 19
15 20 from schainpy.admin import Alarm, SchainWarning
16 21 from schainpy.model import *
17 22 from schainpy.utils import log
18 23
19 24
20 25 class ConfBase():
21 26
22 27 def __init__(self):
23 28
24 29 self.id = '0'
25 30 self.name = None
26 31 self.priority = None
27 32 self.parameters = {}
28 33 self.object = None
29 34 self.operations = []
30 35
31 36 def getId(self):
32 37
33 38 return self.id
34 39
35 40 def getNewId(self):
36 41
37 42 return int(self.id) * 10 + len(self.operations) + 1
38 43
39 44 def updateId(self, new_id):
40 45
41 46 self.id = str(new_id)
42 47
43 48 n = 1
44 49 for conf in self.operations:
45 50 conf_id = str(int(new_id) * 10 + n)
46 51 conf.updateId(conf_id)
47 52 n += 1
48 53
49 54 def getKwargs(self):
50 55
51 56 params = {}
52 57
53 58 for key, value in self.parameters.items():
54 59 if value not in (None, '', ' '):
55 60 params[key] = value
56 61
57 62 return params
58 63
59 64 def update(self, **kwargs):
60 65
61 66 for key, value in kwargs.items():
62 67 self.addParameter(name=key, value=value)
63 68
64 69 def addParameter(self, name, value, format=None):
65 70 '''
66 71 '''
67 72
68 73 if isinstance(value, str) and re.search(r'(\d+/\d+/\d+)', value):
69 74 self.parameters[name] = datetime.date(*[int(x) for x in value.split('/')])
70 75 elif isinstance(value, str) and re.search(r'(\d+:\d+:\d+)', value):
71 76 self.parameters[name] = datetime.time(*[int(x) for x in value.split(':')])
72 77 else:
73 78 try:
74 79 self.parameters[name] = ast.literal_eval(value)
75 80 except:
76 81 if isinstance(value, str) and ',' in value:
77 82 self.parameters[name] = value.split(',')
78 83 else:
79 84 self.parameters[name] = value
80 85
81 86 def getParameters(self):
82 87
83 88 params = {}
84 89 for key, value in self.parameters.items():
85 90 s = type(value).__name__
86 91 if s == 'date':
87 92 params[key] = value.strftime('%Y/%m/%d')
88 93 elif s == 'time':
89 94 params[key] = value.strftime('%H:%M:%S')
90 95 else:
91 96 params[key] = str(value)
92 97
93 98 return params
94 99
95 100 def makeXml(self, element):
96 101
97 102 xml = SubElement(element, self.ELEMENTNAME)
98 103 for label in self.xml_labels:
99 104 xml.set(label, str(getattr(self, label)))
100 105
101 106 for key, value in self.getParameters().items():
102 107 xml_param = SubElement(xml, 'Parameter')
103 108 xml_param.set('name', key)
104 109 xml_param.set('value', value)
105 110
106 111 for conf in self.operations:
107 112 conf.makeXml(xml)
108 113
109 114 def __str__(self):
110 115
111 116 if self.ELEMENTNAME == 'Operation':
112 117 s = ' {}[id={}]\n'.format(self.name, self.id)
113 118 else:
114 119 s = '{}[id={}, inputId={}]\n'.format(self.name, self.id, self.inputId)
115 120
116 121 for key, value in self.parameters.items():
117 122 if self.ELEMENTNAME == 'Operation':
118 123 s += ' {}: {}\n'.format(key, value)
119 124 else:
120 125 s += ' {}: {}\n'.format(key, value)
121 126
122 127 for conf in self.operations:
123 128 s += str(conf)
124 129
125 130 return s
126 131
127 132 class OperationConf(ConfBase):
128 133
129 134 ELEMENTNAME = 'Operation'
130 135 xml_labels = ['id', 'name']
131 136
132 137 def setup(self, id, name, priority, project_id, err_queue):
133 138
134 139 self.id = str(id)
135 140 self.project_id = project_id
136 141 self.name = name
137 142 self.type = 'other'
138 143 self.err_queue = err_queue
139 144
140 145 def readXml(self, element, project_id, err_queue):
141 146
142 147 self.id = element.get('id')
143 148 self.name = element.get('name')
144 149 self.type = 'other'
145 150 self.project_id = str(project_id)
146 151 self.err_queue = err_queue
147 152
148 153 for elm in element.iter('Parameter'):
149 154 self.addParameter(elm.get('name'), elm.get('value'))
150 155
151 156 def createObject(self):
152 157
153 158 className = eval(self.name)
154 159
155 160 if 'Plot' in self.name or 'Writer' in self.name or 'Send' in self.name or 'print' in self.name:
156 161 kwargs = self.getKwargs()
157 162 opObj = className(self.id, self.id, self.project_id, self.err_queue, **kwargs)
158 163 opObj.start()
159 164 self.type = 'external'
160 165 else:
161 166 opObj = className()
162 167
163 168 self.object = opObj
164 169 return opObj
165 170
166 171 class ProcUnitConf(ConfBase):
167 172
168 173 ELEMENTNAME = 'ProcUnit'
169 174 xml_labels = ['id', 'inputId', 'name']
170 175
171 176 def setup(self, project_id, id, name, datatype, inputId, err_queue):
172 177 '''
173 178 '''
174 179
175 180 if datatype == None and name == None:
176 181 raise ValueError('datatype or name should be defined')
177 182
178 183 if name == None:
179 184 if 'Proc' in datatype:
180 185 name = datatype
181 186 else:
182 187 name = '%sProc' % (datatype)
183 188
184 189 if datatype == None:
185 190 datatype = name.replace('Proc', '')
186 191
187 192 self.id = str(id)
188 193 self.project_id = project_id
189 194 self.name = name
190 195 self.datatype = datatype
191 196 self.inputId = inputId
192 197 self.err_queue = err_queue
193 198 self.operations = []
194 199 self.parameters = {}
195 200
196 201 def removeOperation(self, id):
197 202
198 203 i = [1 if x.id==id else 0 for x in self.operations]
199 204 self.operations.pop(i.index(1))
200 205
201 206 def getOperation(self, id):
202 207
203 208 for conf in self.operations:
204 209 if conf.id == id:
205 210 return conf
206 211
207 212 def addOperation(self, name, optype='self'):
208 213 '''
209 214 '''
210 215
211 216 id = self.getNewId()
212 217 conf = OperationConf()
213 218 conf.setup(id, name=name, priority='0', project_id=self.project_id, err_queue=self.err_queue)
214 219 self.operations.append(conf)
215 220
216 221 return conf
217 222
218 223 def readXml(self, element, project_id, err_queue):
219 224
220 225 self.id = element.get('id')
221 226 self.name = element.get('name')
222 227 self.inputId = None if element.get('inputId') == 'None' else element.get('inputId')
223 228 self.datatype = element.get('datatype', self.name.replace(self.ELEMENTNAME.replace('Unit', ''), ''))
224 229 self.project_id = str(project_id)
225 230 self.err_queue = err_queue
226 231 self.operations = []
227 232 self.parameters = {}
228 233
229 234 for elm in element:
230 235 if elm.tag == 'Parameter':
231 236 self.addParameter(elm.get('name'), elm.get('value'))
232 237 elif elm.tag == 'Operation':
233 238 conf = OperationConf()
234 239 conf.readXml(elm, project_id, err_queue)
235 240 self.operations.append(conf)
236 241
237 242 def createObjects(self):
238 243 '''
239 244 Instancia de unidades de procesamiento.
240 245 '''
241 246
242 247 className = eval(self.name)
243 248 kwargs = self.getKwargs()
244 249 procUnitObj = className()
245 250 procUnitObj.name = self.name
246 251 log.success('creating process...', self.name)
247 252
248 253 for conf in self.operations:
249 254
250 255 opObj = conf.createObject()
251 256
252 257 log.success('adding operation: {}, type:{}'.format(
253 258 conf.name,
254 259 conf.type), self.name)
255 260
256 261 procUnitObj.addOperation(conf, opObj)
257 262
258 263 self.object = procUnitObj
259 264
260 265 def run(self):
261 266 '''
262 267 '''
263 268
264 269 return self.object.call(**self.getKwargs())
265 270
266 271
267 272 class ReadUnitConf(ProcUnitConf):
268 273
269 274 ELEMENTNAME = 'ReadUnit'
270 275
271 276 def __init__(self):
272 277
273 278 self.id = None
274 279 self.datatype = None
275 280 self.name = None
276 281 self.inputId = None
277 282 self.operations = []
278 283 self.parameters = {}
279 284
280 285 def setup(self, project_id, id, name, datatype, err_queue, path='', startDate='', endDate='',
281 286 startTime='', endTime='', server=None, **kwargs):
282 287
283 288 if datatype == None and name == None:
284 289 raise ValueError('datatype or name should be defined')
285 290 if name == None:
286 291 if 'Reader' in datatype:
287 292 name = datatype
288 293 datatype = name.replace('Reader','')
289 294 else:
290 295 name = '{}Reader'.format(datatype)
291 296 if datatype == None:
292 297 if 'Reader' in name:
293 298 datatype = name.replace('Reader','')
294 299 else:
295 300 datatype = name
296 301 name = '{}Reader'.format(name)
297 302
298 303 self.id = id
299 304 self.project_id = project_id
300 305 self.name = name
301 306 self.datatype = datatype
302 307 self.err_queue = err_queue
303 308
304 309 self.addParameter(name='path', value=path)
305 310 self.addParameter(name='startDate', value=startDate)
306 311 self.addParameter(name='endDate', value=endDate)
307 312 self.addParameter(name='startTime', value=startTime)
308 313 self.addParameter(name='endTime', value=endTime)
309 314
310 315 for key, value in kwargs.items():
311 316 self.addParameter(name=key, value=value)
312 317
313 318
314 319 class Project(Process):
320 """API to create signal chain projects"""
315 321
316 322 ELEMENTNAME = 'Project'
317 323
318 324 def __init__(self):
319 325
320 Process.__init__(self)
321 self.id = None
326 Process.__init__(self, name='')
327 self.id = '1'
328 if name:
329 self.name = '{} ({})'.format(Process.__name__, name)
322 330 self.filename = None
323 331 self.description = None
324 332 self.email = None
325 333 self.alarm = []
326 334 self.configurations = {}
327 335 # self.err_queue = Queue()
328 336 self.err_queue = None
329 337 self.started = False
330 338
331 339 def getNewId(self):
332 340
333 341 idList = list(self.configurations.keys())
334 342 id = int(self.id) * 10
335 343
336 344 while True:
337 345 id += 1
338 346
339 347 if str(id) in idList:
340 348 continue
341 349
342 350 break
343 351
344 352 return str(id)
345 353
346 354 def updateId(self, new_id):
347 355
348 356 self.id = str(new_id)
349 357
350 358 keyList = list(self.configurations.keys())
351 359 keyList.sort()
352 360
353 361 n = 1
354 362 new_confs = {}
355 363
356 364 for procKey in keyList:
357 365
358 366 conf = self.configurations[procKey]
359 367 idProcUnit = str(int(self.id) * 10 + n)
360 368 conf.updateId(idProcUnit)
361 369 new_confs[idProcUnit] = conf
362 370 n += 1
363 371
364 372 self.configurations = new_confs
365 373
366 374 def setup(self, id=1, name='', description='', email=None, alarm=[]):
367 375
368 376 self.id = str(id)
369 377 self.description = description
370 378 self.email = email
371 379 self.alarm = alarm
372 380 if name:
373 381 self.name = '{} ({})'.format(Process.__name__, name)
374 382
375 383 def update(self, **kwargs):
376 384
377 385 for key, value in kwargs.items():
378 386 setattr(self, key, value)
379 387
380 388 def clone(self):
381 389
382 390 p = Project()
383 391 p.id = self.id
384 392 p.name = self.name
385 393 p.description = self.description
386 394 p.configurations = self.configurations.copy()
387 395
388 396 return p
389 397
390 398 def addReadUnit(self, id=None, datatype=None, name=None, **kwargs):
391 399
392 400 '''
393 401 '''
394 402
395 403 if id is None:
396 404 idReadUnit = self.getNewId()
397 405 else:
398 406 idReadUnit = str(id)
399 407
400 408 conf = ReadUnitConf()
401 409 conf.setup(self.id, idReadUnit, name, datatype, self.err_queue, **kwargs)
402 410 self.configurations[conf.id] = conf
403 411
404 412 return conf
405 413
406 414 def addProcUnit(self, id=None, inputId='0', datatype=None, name=None):
407 415
408 416 '''
409 417 '''
410 418
411 419 if id is None:
412 420 idProcUnit = self.getNewId()
413 421 else:
414 422 idProcUnit = id
415 423
416 424 conf = ProcUnitConf()
417 425 conf.setup(self.id, idProcUnit, name, datatype, inputId, self.err_queue)
418 426 self.configurations[conf.id] = conf
419 427
420 428 return conf
421 429
422 430 def removeProcUnit(self, id):
423 431
424 432 if id in self.configurations:
425 433 self.configurations.pop(id)
426 434
427 435 def getReadUnit(self):
428 436
429 437 for obj in list(self.configurations.values()):
430 438 if obj.ELEMENTNAME == 'ReadUnit':
431 439 return obj
432 440
433 441 return None
434 442
435 443 def getProcUnit(self, id):
436 444
437 445 return self.configurations[id]
438 446
439 447 def getUnits(self):
440 448
441 449 keys = list(self.configurations)
442 450 keys.sort()
443 451
444 452 for key in keys:
445 453 yield self.configurations[key]
446 454
447 455 def updateUnit(self, id, **kwargs):
448 456
449 457 conf = self.configurations[id].update(**kwargs)
450 458
451 459 def makeXml(self):
452 460
453 461 xml = Element('Project')
454 462 xml.set('id', str(self.id))
455 463 xml.set('name', self.name)
456 464 xml.set('description', self.description)
457 465
458 466 for conf in self.configurations.values():
459 467 conf.makeXml(xml)
460 468
461 469 self.xml = xml
462 470
463 471 def writeXml(self, filename=None):
464 472
465 473 if filename == None:
466 474 if self.filename:
467 475 filename = self.filename
468 476 else:
469 477 filename = 'schain.xml'
470 478
471 479 if not filename:
472 480 print('filename has not been defined. Use setFilename(filename) for do it.')
473 481 return 0
474 482
475 483 abs_file = os.path.abspath(filename)
476 484
477 485 if not os.access(os.path.dirname(abs_file), os.W_OK):
478 486 print('No write permission on %s' % os.path.dirname(abs_file))
479 487 return 0
480 488
481 489 if os.path.isfile(abs_file) and not(os.access(abs_file, os.W_OK)):
482 490 print('File %s already exists and it could not be overwriten' % abs_file)
483 491 return 0
484 492
485 493 self.makeXml()
486 494
487 495 ElementTree(self.xml).write(abs_file, method='xml')
488 496
489 497 self.filename = abs_file
490 498
491 499 return 1
492 500
493 501 def readXml(self, filename):
494 502
495 503 abs_file = os.path.abspath(filename)
496 504
497 505 self.configurations = {}
498 506
499 507 try:
500 508 self.xml = ElementTree().parse(abs_file)
501 509 except:
502 510 log.error('Error reading %s, verify file format' % filename)
503 511 return 0
504 512
505 513 self.id = self.xml.get('id')
506 514 self.name = self.xml.get('name')
507 515 self.description = self.xml.get('description')
508 516
509 517 for element in self.xml:
510 518 if element.tag == 'ReadUnit':
511 519 conf = ReadUnitConf()
512 520 conf.readXml(element, self.id, self.err_queue)
513 521 self.configurations[conf.id] = conf
514 522 elif element.tag == 'ProcUnit':
515 523 conf = ProcUnitConf()
516 524 input_proc = self.configurations[element.get('inputId')]
517 525 conf.readXml(element, self.id, self.err_queue)
518 526 self.configurations[conf.id] = conf
519 527
520 528 self.filename = abs_file
521 529
522 530 return 1
523 531
524 532 def __str__(self):
525 533
526 534 text = '\nProject[id=%s, name=%s, description=%s]\n\n' % (
527 535 self.id,
528 536 self.name,
529 537 self.description,
530 538 )
531 539
532 540 for conf in self.configurations.values():
533 541 text += '{}'.format(conf)
534 542
535 543 return text
536 544
537 545 def createObjects(self):
538 546
539 547 keys = list(self.configurations.keys())
540 548 keys.sort()
541 549 for key in keys:
542 550 conf = self.configurations[key]
543 551 conf.createObjects()
544 552 if conf.inputId is not None:
545 553 conf.object.setInput(self.configurations[conf.inputId].object)
546 554
547 555 def monitor(self):
548 556
549 557 t = Thread(target=self._monitor, args=(self.err_queue, self.ctx))
550 558 t.start()
551 559
552 560 def _monitor(self, queue, ctx):
553 561
554 562 import socket
555 563
556 564 procs = 0
557 565 err_msg = ''
558 566
559 567 while True:
560 568 msg = queue.get()
561 569 if '#_start_#' in msg:
562 570 procs += 1
563 571 elif '#_end_#' in msg:
564 572 procs -=1
565 573 else:
566 574 err_msg = msg
567 575
568 576 if procs == 0 or 'Traceback' in err_msg:
569 577 break
570 578 time.sleep(0.1)
571 579
572 580 if '|' in err_msg:
573 581 name, err = err_msg.split('|')
574 582 if 'SchainWarning' in err:
575 583 log.warning(err.split('SchainWarning:')[-1].split('\n')[0].strip(), name)
576 584 elif 'SchainError' in err:
577 585 log.error(err.split('SchainError:')[-1].split('\n')[0].strip(), name)
578 586 else:
579 587 log.error(err, name)
580 588 else:
581 589 name, err = self.name, err_msg
582 590
583 591 time.sleep(1)
584 592
585 593 ctx.term()
586 594
587 595 message = ''.join(err)
588 596
589 597 if err_msg:
590 598 subject = 'SChain v%s: Error running %s\n' % (
591 599 schainpy.__version__, self.name)
592 600
593 601 subtitle = 'Hostname: %s\n' % socket.gethostbyname(
594 602 socket.gethostname())
595 603 subtitle += 'Working directory: %s\n' % os.path.abspath('./')
596 604 subtitle += 'Configuration file: %s\n' % self.filename
597 605 subtitle += 'Time: %s\n' % str(datetime.datetime.now())
598 606
599 607 readUnitConfObj = self.getReadUnit()
600 608 if readUnitConfObj:
601 609 subtitle += '\nInput parameters:\n'
602 610 subtitle += '[Data path = %s]\n' % readUnitConfObj.parameters['path']
603 611 subtitle += '[Start date = %s]\n' % readUnitConfObj.parameters['startDate']
604 612 subtitle += '[End date = %s]\n' % readUnitConfObj.parameters['endDate']
605 613 subtitle += '[Start time = %s]\n' % readUnitConfObj.parameters['startTime']
606 614 subtitle += '[End time = %s]\n' % readUnitConfObj.parameters['endTime']
607 615
608 616 a = Alarm(
609 617 modes=self.alarm,
610 618 email=self.email,
611 619 message=message,
612 620 subject=subject,
613 621 subtitle=subtitle,
614 622 filename=self.filename
615 623 )
616 624
617 625 a.start()
618 626
619 627 def setFilename(self, filename):
620 628
621 629 self.filename = filename
622 630
623 631 def runProcs(self):
624 632
625 633 err = False
626 634 n = len(self.configurations)
627 635
628 636 while not err:
629 637 for conf in self.getUnits():
630 638 ok = conf.run()
631 639 if ok is 'Error':
632 640 n -= 1
633 641 continue
634 642 elif not ok:
635 643 break
636 644 if n == 0:
637 645 err = True
638 646
639 647 def run(self):
640 648
641 649 log.success('\nStarting Project {} [id={}]'.format(self.name, self.id), tag='')
642 650 self.started = True
643 651 self.start_time = time.time()
644 652 self.createObjects()
645 653 self.runProcs()
646 654 log.success('{} Done (Time: {:4.2f}s)'.format(
647 655 self.name,
648 656 time.time()-self.start_time), '')
Show file before | Show file after | Hide comments
@@ -1,63 +1,86
1 '''
2 Created on Jul 16, 2014
1 # Copyright (c) 2012-2020 Jicamarca Radio Observatory
2 # All rights reserved.
3 #
4 # Distributed under the terms of the BSD 3-clause license.
5 """schainpy is an open source library to read, write and process radar data
3 6
4 @author: Miguel Urco
5 @author: Juan C. Espinoza
6 '''
7 Signal Chain is a radar data processing library wich includes modules to read,
8 and write different files formats, besides modules to process and visualize the
9 data.
10 """
7 11
8 12 import os
9 13 from setuptools import setup, Extension
10 14 from setuptools.command.build_ext import build_ext as _build_ext
11 15 from schainpy import __version__
12 16
17 DOCLINES = __doc__.split("\n")
18
13 19 class build_ext(_build_ext):
14 20 def finalize_options(self):
15 21 _build_ext.finalize_options(self)
16 22 # Prevent numpy from thinking it is still in its setup process:
17 23 __builtins__.__NUMPY_SETUP__ = False
18 24 import numpy
19 25 self.include_dirs.append(numpy.get_include())
20 26
21 27 setup(
22 28 name = "schainpy",
23 29 version = __version__,
24 description = "Python tools to read, write and process Jicamarca data",
25 author = "Miguel Urco, Juan C. Espinoza",
26 author_email = "juan.espinoza@jro.igp.gob.pe",
27 url = "http://jro-dev.igp.gob.pe/rhodecode/schain",
30 description = DOCLINES[0],
31 long_description = "\n".join(DOCLINES[2:]),
32 url = "https://github.com/JRO-Peru/schain",
33 author = "Jicamarca Radio Observatory",
34 author_email = "jro-developers@igp.gob.pe",
35 license="BSD-3-Clause",
36 classifiers=[
37 "Development Status :: 4 - Beta",
38 "Environment :: Console",
39 "Intended Audience :: Science/Research",
40 "License :: OSI Approved :: BSD License",
41 "Operating System :: MacOS :: MacOS X",
42 "Operating System :: POSIX :: Linux",
43 "Programming Language :: Python :: 2",
44 "Programming Language :: Python :: 2.7",
45 "Programming Language :: Python :: 3",
46 "Programming Language :: Python :: 3.5",
47 "Programming Language :: Python :: 3.6",
48 "Programming Language :: Python :: 3.7",
49 "Topic :: Scientific/Engineering",
50 ],
28 51 packages = {
29 52 'schainpy',
30 53 'schainpy.model',
31 54 'schainpy.model.data',
32 55 'schainpy.model.graphics',
33 56 'schainpy.model.io',
34 57 'schainpy.model.proc',
35 58 'schainpy.model.utils',
36 59 'schainpy.utils',
37 60 'schainpy.gui',
38 61 'schainpy.cli',
39 62 },
40 63 package_data = {'': ['schain.conf.template'],
41 64 'schainpy.files': ['*.oga']
42 65 },
43 66 include_package_data = False,
44 67 scripts = ['schainpy/gui/schainGUI'],
45 68 entry_points = {
46 69 'console_scripts': [
47 70 'schain = schainpy.cli.cli:main',
48 71 ],
49 72 },
50 73 cmdclass = {'build_ext': build_ext},
51 74 ext_modules=[
52 75 Extension("schainpy.model.data._noise", ["schainc/_noise.c"]),
53 76 ],
54 77 setup_requires = ["numpy"],
55 78 install_requires = [
56 79 "scipy",
57 80 "h5py",
58 81 "matplotlib",
59 82 "pyzmq",
60 83 "fuzzywuzzy",
61 84 "click",
62 85 ],
63 86 )
General Comments 0
You need to be logged in to leave comments. Login now