Developer's manual¶
1. Introduction¶
Signal Chain (SCh) is a JRO ongoing project, which has the aim to develop open source libraries for the signal processing of the information acquired with scientific radars. The final purpose of these libraries is to share them with the scientific community that uses these devices, in order to foster collaboration between the different institutions engaged in this field of study.
2. Installation¶
- Install Python dependencies
- numpy
- matplotlib
- scipy
- h5py
- Download Eclipse: https://eclipse.org/downloads/. The recommended version is "Eclipse IDE for C/C++ Developers".
- Install Pydev
- Install Pydev plugin: http://www.pydev.org/manual_101_install.html
- Configure Pydev Interpreter: http://www.pydev.org/manual_101_interpreter.html
- Install Subclipse: http://dev.antoinesolutions.com/subclipse.
- Open the Subclipse repository perspective
- Select the Window > Open Perspective > Other… menu option
- Select the SVN Repository Exploring option
- Click OK
- Add Signal Chain Repository
- Right Click in the SVN Repositories panel
- Select the New > Repository Location… option
- Enter the repository URL: http://jro-dev.igp.gob.pe/svn/jro_soft/schain
- Click Finish
- Download Signal Chain Libraries
- Open Pydev perspective, analogously as Step 5
- Select the New > Project... option
- Select SVN > Checkout Projects from SVN
- Click Next
- Select "Use existing repository location"
- Select the Signal Chain repository
- Click Next
- Select http://jro-dev.igp.gob.pe/svn/jro_soft/schain > trunk > schainroot
- Click Next
- Select "Check out as a project in the workspace"
- Click Finish
3. Radar Basics¶
3.1 Radar principle¶
The basic principle of a radar can be observed in the animated gif below. The process starts with an electromagnetic pulse generated at the transmitter which is irradiated into the space through the antenna. The pulse travels across the space until it reaches the target. The pulse is intercepted by the target, which absorbs part of the energy and re-irradiates it in all directions. Finally, some of these energy is redirected back to the antenna, where the system receives it and the information is processed to estimate different parameters.
3.2 Transmission Pulse¶
The distance of the target can be determined with the time it takes the system to receive its echo. This time is how much it takes to the radar signal to travel from the radar to the target and back. Typically, in a radar experiment, multiple pulses are sent with an specific frequency which is called Pulse Repetition Frequency (PRF) and its inverse is called the Inter-Pulse Period (IPP). This period is selected according to the longest expected target's range. If the IPP is too short, a backscattered signal from a long-range target could arrive after the transmission of the next pulse.
3.3 Radar data¶
After each pulse, the echo is digitally sampled by the radar acquisition system. The samples at a fixed time after the pulse represent the response of the atmosphere at an specific height of the atmosphere . In this manner, this samples can be used to constuct a time series, as can be seen in the next figure. This series is then transformed into the frequency domain to visualize the doppler shifts.
4. Software Description¶
In Signal Chain, every signal processing algorithm is splitted into separated blocks. This approach can be explained in the image below, where the process of acquiring a digital image is illustrated as a pipeline. The process is splitted into five different blocks. Each one of them plays a different role. The first step is to transform the light into a voltage signal with a sensor array. Next, the analogue signal obtained is amplified with a low-noise amplifier block and, then, it is converted to digital through an ADC. The next block is a DSP, which involves many signal processing algorithms such as filtering or averaging. Finally, the last stage is to store the image obtained in a database.
Analogously, to develop SCh libraries, each algorithm is splitted into a sequence of independent operations such as data reading, interference filtering or results plotting. These agents are deliberately limited in scope and as generic as the algorithm allows. The purpose is to make the modules reusable in different processes and, in this manner, avoid repeating code. This trait is illustrated in the image below.
One of the most common experiments is the MST-ISR-EEJ. In this configuration, three different experiments (MST, ISR and EEJ) are executed simultaneously and their data are stored in the same raw files. MST stands for Mesosphere, Stratosphere, Troposphere and the experiment aims to study the low atmosphere echoes. ISR means Incoherent Scatter Radar and seeks to study the incoherent scatter echoes from the ionosphere. Finally, EEJ stands for Equatorial Electrojet, and the experiment studies the electrical properties of this phenomenum. To execute the experiment, the SCh script starts with the jointed data. The first step is to separate the profiles that belong to each one of the experiments. After this step, the image shows that each experiment runs independently. The MST experiment procceeds to decode the data and do a couple of incoherent integrations. Then, the data is converted to the frequency domain through the FFT. Finally, the data is integrated incoherently and it is plotted to show the results to the user. ISR does exactly the same operations; thus, they use the same operation libraries. The only difference is the parameters that each one of this operations have. Consequently, SCh methodology allows the user to process these two experiments with the same libraries, with only small variations in the input parameters.
4.1 Configuration layers¶
Every SCh experiment is configured in four basic steps or layers, which are described below.
User Interface.- The configuration can be done with the graphic interface (GUI) or a python script, which is the recommended option for a developer.
Project.- Next, a Project object should be created, which will have an ID, name and a brief description of the experiment being processed.
Units.- Inside the Project, the user will create Units, which will contain the different operations or tasks that SCh will performed to the data. There are three main types of Units:
- Reading Unit, this units read the data that is acquired by the radar system. This data can be voltages or spectral information.
- Writing Unit, once the data has been processed, the information obtained or the modified data can be stored in files that can be read later.
- Processing Unit, they contain the different mathematical operations or algorithms that will be performed to the data of an specific type. The type of operations that can be selected depend on the type of Processing Unit that it is being used. There are four main types of Processing Units: Voltage, Spectra, Correlations and Parameters.
Operations.- Finally, the user should add to the Units all the algorithms or mathematical operations. There are two type of operations:
- Processing Operation, algorithms or calculations to obtain specific information from the data, such as calculation of moments, winds estimation or decoding.
- Graphic Operation, plots to visualize the results of the signal processing algorithms
4.2 Basic structure¶
The basic pipeline of a SCh can be seen in the image below. The first module to be defined is the Reading module, which means that the first step is to define the data that the software will read and use as input. Next, a Processing Unit is defined to include all the operations that will be performed to the signal. Finally, a Writing Unit is added to output the results of the signal processing chain and to make further analysis.
4.3 Files distribution¶
Finally, to develop new libraries, it is necessary to be familiarized with the files distribution of Signal Chain. If the repository location was added correctly to the workspace, something as the next figure should appear. The files that will be used are all inside schainroot>source>schainpy. There are two main directores:
- model, contains the code of the signal processing libraries developed for the radar experiments and it is divided according to the type of data being analyzed.
- scripts, contains the python scripts that are executed to start the signal processing algorithms.
Likewise, the model directory is divided in different directories that organize the different libraries developed into four main categories:
- data, libraries that model the data passed through the different units and operations. There are four main types of data: voltage, spectra, correlation and parameters. Each one of them have different atrributes and methods according to their characteristics.
- graphics, graphic libraries to visualize the information obtain from the data in realtime. In the same way, this libraries are organized in different python scripts according to the type of data they plot.
- io, contains the reading and writing units code.
- proc, libraries that contain all the processing units and operations programming code.
5. Development examples¶
In this section, some examples to configure basic experiments will be addressed. The first one is a basic example to help understand the experiment configuration of Signal Chain. Then, the manual details how to create new operations and, finally, the user will understand how to develop his own reading and processing units.
5.1 Script Example¶
In this part, a simple example will be developed to understand the experiment configuration through a python script. In the figure below, the program structure can be seen. The script will start with a Reading Unit; in this case, Voltage data reader. Then, it will continue with a Voltage Processing Unit, which will contain two operations, a Decoder and Coherent Integrations. Next, a Spectra Processing Unit is added. Inside this Processing Unit, two more operations are added. One is Incoherent Integrations while the other one is a RTI Plot, to visualize the data being processed. Finally, a Writing Unit is created to store the processed data.
The first step is to set the search path for modules. The path for python libraries such as numpy, matplotlib is already set by default, so they only need to be imported. However, to use the Signal Chain modules as well, the workspace path needs to be added, which is done with the lines of code below.
import os, sys path = os.path.split(os.getcwd())[0] path = os.path.split(path)[0] sys.path.insert(0, path)
Next, some necessary variables are defined, such the XML file name, the path with the input files, and the path were the output files are going to be saves.
filename = "school_test.xml" path ="../../../data/rawdata/" pathfile = os.environ['HOME']
The Project object mentioned in 4.1 is created. The object's name is controllerObj and needs to have an id, name, and a brief description to identify it.
from schainpy.controller import Project controllerObj = Project() controllerObj.setup(id = '101', name='test01', description='Basic experiment')
Now, we proceed to configure the Reading Unit. In this case, the unit will read Voltage data so we need to specify the datatype as 'VoltageReader'. In this part, it is necessary to define the 'path' variable, which is the directory where the files processed are located.
readUnitConfObj = controllerObj.addReadUnit(datatype='VoltageReader',
path=path,
startDate='2014/01/31',
endDate='2014/03/31',
startTime='00:00:00',
endTime='23:59:59',
online=0,
delay=5,
walk=0)
The Voltage Processing Unit is created. As can be seen below, the input of this unit is the Reading Unit configured above. Next, the operations are added to this Processing Unit using the method 'addOperation'. The Decoder and CohInt operations are created. The CohInt operation integrates the data coherently, i.e. it averages the incoming data in order to increase the Signal to Noise Ratio (SNR). Consequently, the number of averages needs to be defined. This is done using the method 'addParameter'.
procUnitConfObj0 = controllerObj.addProcUnit(datatype='VoltageProc', inputId=readUnitConfObj.getId()) opObj01 = procUnitConfObj0.addOperation(name='Decoder', optype='other') opObj02 = procUnitConfObj0.addOperation(name='CohInt', optype='other') opObj02.addParameter(name='n', value='5', format='int')
The Spectra Processing Unit is created, and the input is the Voltage Processing Unit defined before. Signal Chain makes the conversion from time to frequency domain automatically. It just needs the user to specify the number of FFT points to be calculated for the spectrum, which is the same number of profiles to be stacked before converting to the frequency domain. Then, the Incoherent Integration operation is added and the number of integrations 'n' is specified as a parameter. To visualize the data, a RTI Plot is added.
procUnitConfObj1 = controllerObj.addProcUnit(datatype='SpectraProc', inputId=procUnitConfObj0.getId()) procUnitConfObj1.addParameter(name='nFFTPoints', value='16', format='int') opObj11 = procUnitConfObj1.addOperation(name='IncohInt', optype='other') opObj11.addParameter(name='n', value='2', format='int') opObj12 = procUnitConfObj1.addOperation(name='SpectraPlot', optype='other') opObj12.addParameter(name='id', value='1', format='int')
The spectra writer is introduced to store the processed the data. Two parameters are added to the operation. The first one is the path where the files will be stored, while the second one is the number of blocks or spectra that will be stored per file.
opObj13 = procUnitConfObj1.addOperation(name='SpectraWriter', optype='other') opObj13.addParameter(name='path', value=path) opObj13.addParameter(name='blocksPerFile', value='100', format='int')
After finishing the structure of the program, a XML file is created. This same XML file is produced when the GUIDE is being used.
print "Escribiendo el archivo XML" controllerObj.writeXml(filename) print "Leyendo el archivo XML" controllerObj.readXml(filename)
Finally, this is the code that Signal Chain actually executes. The Project, Processing Units and Operation objects are created, connected and the script is executed.
controllerObj.createObjects() controllerObj.connectObjects() controllerObj.run()
To run the program, write it in a .py file, and move the file into the folder: schainroot > source > schainpy > scripts. If everything works fine, a window showing the Spectral Plot should open.
5.2 Operation Example¶
In this section, an Operation Unit will be created, more specifically, a simpler version of the Coherent Integration operation that is already in the SCh libraries. This exercise will help to understand how to develop SCh libraries and how the software handles the data.
We will start with the script developed in section 5.1 and replace the Coherente Integration block. For this, write .py with the script
First, go to schainroot>source>schainpy>model>proc
class MyOperation(Operation):
nProfiles = 0
buffer = None
def run(self, dataOut, n):
#Doesnt fo any further
dataOut.flagNoData = True
#Copy the data
data = dataOut.data
#Set buffer
if self.buffer == None:
self.buffer = data/n
else:
self.buffer += data/n
self.nProfiles += 1
#Check if enough integrations
if self.nProfiles == n:
dataOut.data = self.buffer
dataOut.flagNoData = False
self.buffer = None
self.nProfiles = 0
5.3 Reading Unit Example¶
class MyReader(ProcessingUnit):
path = None
fileList = None
fileIndex = 0
nBlocks = 0
blockIndex = 0
#Metadata
timeZone = None
paramInterval = None
heightList = None
#Data
data = None
utctime = None
def __init__(self):
#Initialize dataOut as a type of data
self.dataOut = Parameters()
return
def run(self, **kwargs):
#Initial setup
if not(self.isConfig):
self.path = kwargs['path']
self.__setup()
#Have we read all blocks?
if self.blockIndex == self.nBlocks:
#Have we read all files?
if self.fileIndex == len(self.fileList):
#if yes, the program is terminated
self.dataOut.flagNoData = True
return 0
else:
#if not, we read next file
self.__readNextFile()
#Read next block
self.__readNextBlock()
return
Functions
def __setup(self):
#Search available files
fileList = glob.glob1(self.path, "*%s" %".hdf5")
fileList.sort()
#Save configuration
self.fileList = fileList
self.isConfig = True
def __readNextFile(self):
#Read file
filename = os.path.join(self.path,self.fileList[self.fileIndex])
fp = h5py.File(filename,'r')
#Read metadata files
grp1 = fp['Metadata']
self.timeZone = grp1['timeZone'].value
self.paramInterval = grp1['paramInterval'].value
self.heightList = grp1['heightList'].value
#Read data files
grp2 = fp['Data']
self.utctime = numpy.squeeze(grp2['utctime'].value)
grp22 = grp2['data_param']
ds0 = grp22['table0'].value
ds1 = grp22['table1'].value
ds2 = grp22['table2'].value
#Rearranging them for dataOut object
self.data = numpy.zeros((ds0.shape[0], 3, ds0.shape[1], ds0.shape[2]))
self.data[:,0,:,:] = ds0
self.data[:,1,:,:] = ds1
self.data[:,2,:,:] = ds2
#Setting variables
self.nBlocks = ds0.shape[2]
self.blockIndex = 0
self.fileIndex += 1
return
def __readNextBlock(self):
#Metadata
self.dataOut.timeZone = self.timeZone
self.dataOut.paramInterval = self.paramInterval
self.dataOut.heightList = self.heightList
self.dataOut.channelList = [0,1]
#Data
self.dataOut.utctimeInit = self.utctime[self.blockIndex]
self.dataOut.utctime = self.utctime[self.blockIndex]
self.dataOut.data_param = self.data[:,:,:,self.blockIndex]
self.dataOut.flagNoData = False
self.blockIndex += 1
return
Final script
import os, sys path = os.path.split(os.getcwd())[0] path = os.path.split(path)[0] sys.path.insert(0, path) filename = "school_test2.xml" figpath = os.path.join(os.environ['HOME'],'Pictures') path = os.path.join(os.environ['HOME'],'Pictures')
from schainpy.controller import Project controllerObj = Project() controllerObj.setup(id = '102', name='test02', description='Reader/Writer experiment')
readUnitConfObj = controllerObj.addReadUnit(datatype='MyReader',
path=path,
startTime = '00:00:00',
endTime = '23:59:59',
startDate = '2000/01/31',
endDate = '2012/01/31')
procUnitConfObj2 = controllerObj.addProcUnit(datatype='ParametersProc', inputId=readUnitConfObj.getId()) opObj21 = procUnitConfObj2.addOperation(name='ParametersPlot', optype='other') opObj21.addParameter(name='id', value='5', format='int') opObj21.addParameter(name='wintitle', value='Radial Velocity Plot', format='str') opObj21.addParameter(name='save', value='1', format='bool') opObj21.addParameter(name='figpath', value=figpath, format='str') opObj21.addParameter(name='xmin', value='0', format='float') opObj21.addParameter(name='xmax', value='0.5', format='float') opObj21.addParameter(name='zmin', value='-0.25', format='float') opObj21.addParameter(name='zmax', value='0.25', format='float') opObj21.addParameter(name='paramIndex', value='1', format='int') opObj21.addParameter(name='colormap', value='0', format='bool')
print "Escribiendo el archivo XML" controllerObj.writeXml(filename) print "Leyendo el archivo XML" controllerObj.readXml(filename) controllerObj.createObjects() controllerObj.connectObjects() controllerObj.run()
5.4 Plotting Example¶
class MyPlot(Figure):
def __init__(self):
return
def getSubplots(self):
ncol = 2
nrow = 1
return nrow, ncol
def run(self, dataOut, id, show = True, wintitle = None, figpath = None,
xmin = None, xmax = None, ymin = None, ymax = None, zmin = None, zmax = None,
xlabel = None, ylabel = None, title = None):
#Read data from dataOut object
x = dataOut.getFreqRange(1)
y = dataOut.heightList
zdB = numpy.log10(dataOut.data_spc)
thisDatetime = datetime.datetime.utcfromtimestamp(dataOut.getTimeRange()[0])
#Is it configured?
if not(self.isConfig):
#Create Figure
self.createFigure(id = id,
wintitle = "Plot Example",
widthplot = 250,
heightplot = 250,
show=show)
#Create Axes and adds them to axesList attribute
#nrow, ncol, row, col, colspan, rowspan
self.addAxes(1,2,0,0,1,1)
self.addAxes(1,2,0,1,1,1)
#Plot limits
xmin = x[0]
xmax = x[-1]
ymin = y[0]
ymax = y[-1]
zmin = numpy.nanmin(zdB)
zmax = numpy.nanmax(zdB)
#Labels
xlabel = "Frequency"
ylabel = "Heights"
title = "My Plot"
#Configure flag
self.iConfig = True
for i in range(2):
title = "Channel %d" %(i)
#Plot data
axes = self.axesList[i]
axes.pcolor(x, y, zdB[i,:,:],
xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax, zmin=zmin, zmax=zmax,
xlabel=xlabel, ylabel=ylabel, title=title,
ticksize=9, cblabel='')
#Show Figure
self.draw()
#Name of the File
name = thisDatetime.strftime("%Y%m%d_%H%M%S")
figfile = self.getFilename(name)
#Save Figure
self.save(figpath=figpath,
figfile=figfile,
save=True,
ftp="",
wr_period="",
thisDatetime=thisDatetime)
Script
import os, sys path = os.path.split(os.getcwd())[0] path = os.path.split(path)[0] sys.path.insert(0, path) path ="../../../data/pdata/" filename = "school_test3.xml" figpath = os.path.join(os.environ['HOME'],'Pictures/school/') #----------------------------------------------------------------------------------------------------------------- from schainpy.controller import Project controllerObj = Project() controllerObj.setup(id = '103', name='test03', description='Plot experiment')
readUnitConfObj = controllerObj.addReadUnit(datatype='Spectra',
path=path,
startDate='2010/12/18',
endDate='2015/12/22',
startTime='00:00:00',
endTime='23:59:59',
online=0,
walk=0,
expLabel='')
procUnitConfObj1 = controllerObj.addProcUnit(datatype='Spectra', inputId=readUnitConfObj.getId())
opObj11 = procUnitConfObj1.addOperation(name='MyPlot', optype='other') opObj11.addParameter(name='id', value='10', format='int') opObj11.addParameter(name='figpath', value=figpath)
print "Escribiendo el archivo XML" controllerObj.writeXml(filename) print "Leyendo el archivo XML" controllerObj.readXml(filename) controllerObj.createObjects() controllerObj.connectObjects() controllerObj.run()
6. Related Links¶
- Manual de Procesamiento de datos de JASMET: https://docs.google.com/document/d/1sxkEoHLzSm-nDbdg_FJmbip8PxFzz1TbNFh37VksRh4/edit?usp=sharing
