The requested changes are too big and content was truncated. Show full diff
| @@ -1,64 +1,64 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Is Madrigal appropriate</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_install.html" --><!-- InstanceParam name="href_back_top" type="text" value="admin.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_install.html" --><!-- InstanceParam name="href_prev_top" type="text" value="admin.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_install.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_install.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'ad_install.html' %}"><img src=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'ad_install.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Is Madrigal appropriate for my instrument(s)?<!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'ad_install.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Installing Madrigal <!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">Is Madrigal appropriate for my instrument(s)?</h1> |
|
|
40 | 40 | <p>The Madrigal database is designed to hold data about the upper atmosphere. One of the strengths of the Madrigal is that most of the measured parameters it contains are defined |
|
|
41 |
in a community standard, whose metadata can be found under the Access metadata tab. With the release of Madrigal 3.0, the file format is now Hdf2 with the restrictions defined in the <a href=" |
|
|
|
41 | in a community standard, whose metadata can be found under the Access metadata tab. With the release of Madrigal 3.0, the file format is now Hdf2 with the restrictions defined in the <a href="{% static 'CEDARMadrigalHdf5Format.pdf">CEDAR Madrigal Hdf5 file format</a>. This standard defines many frames of reference used commonly in atmospheric science, such as geodetic coordinates, geomagnetic coordinates, or radar (azimuth, elevation, and range) coordinates. Upper atmospheric data that can fit into one of these coordinate systems is a good candidate for Madrigal. Madrigal is not presently designed to handle spacecraft ephemeris in order to determine location, and to date Madrigal has only been used to hold spacecraft data that is independent of position (such as solar wind) or can be converted to one of the coordinate systems discussed above (such as total electron concentration in geodetic coordinates from GPS satellites).</p> | |
|
|
42 | 42 | <p>Generally, if the <a href="{% url 'parameter_metadata' %}">parameters found in the metadata</a> are appropriate for you, or only a few additions need to be made, then data from your instrument(s) is probably appropriate for Madrigal. Please feel free to contact the <a href="mailto:madrigal@haystack.mit.edu">OpenMadrigal administrator</a> if you have any questions about using Madrigal. </p> |
|
|
43 | 43 | <!-- InstanceEndEditable --> |
|
|
44 | 44 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
45 | 45 | <tr> |
|
|
46 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
47 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
48 |
<td width="5%"><a href="{% url 'docs' 'ad_install.html' %}"><img src=" |
|
|
|
46 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
47 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
48 | <td width="5%"><a href="{% url 'docs' 'ad_install.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
49 | 49 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Is Madrigal appropriate for my instrument(s)?<!-- InstanceEndEditable --></td> |
|
|
50 | 50 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
51 | 51 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
52 | 52 | </tr> |
|
|
53 | 53 | </table> |
|
|
54 | 54 | <div class='online-navigation'> |
|
|
55 | 55 | <b class="navlabel">Previous:</b> |
|
|
56 | 56 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
57 | 57 | <b class="navlabel"> Up:</b> |
|
|
58 | 58 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
59 | 59 | <b class="navlabel"> Next:</b> |
|
|
60 | 60 | <a class="sectref" href="{% url 'docs' 'ad_install.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Installing Madrigal <!-- InstanceEndEditable --></A></div> |
|
|
61 | 61 | <hr/> |
|
|
62 | 62 | <p> </p> |
|
|
63 | 63 | </body> |
|
|
64 | 64 | <!-- InstanceEnd --></html> |
| @@ -1,370 +1,370 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Creating and modifying Madrigal experiments</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_other.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_createFiles.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_createFiles.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_other.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_createFiles.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_other.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_createFiles.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_other.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'ad_createFiles.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'ad_other.html' %}"><img src=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'ad_createFiles.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'ad_other.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Creating and modifying Madrigal experiments <!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'ad_createFiles.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Creating Madrigal data files <!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'ad_other.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Other admin tasks <!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">Creating and modifying Madrigal experiments</h1> |
|
|
40 | 40 | <p>Adding data to Madrigal involves two steps. The first step is <a href="{% url 'docs' 'ad_createFiles.html' %}">creating Madrigal data files</a>, as discussed in the previous section. The next is installing them on the Madrigal database, with the appropriate metadata and any additional plots or other information you want to supply. </p> |
|
|
41 | 41 | <p>There are a number of scripts supplied with Madrigal to simplify the task of creating or modifying experiments on Madrigal. Use these scripts if you want to:</p> |
|
|
42 | 42 | <ul> |
|
|
43 | 43 | <li><a href="#createExpWithFile">Create an experiment using a pre-existing Madrigal file</a></li> |
|
|
44 | 44 | <li><a href="#createRTExp">Create an experiment when no Madrigal file yet exists</a> (such as to create a real-time file)</li> |
|
|
45 | 45 | <li><a href="#changeExpStatus">Modify the status of a given experiment</a></li> |
|
|
46 | 46 | <li><a href="#addFileToExp">Add a new file to an experiment</a></li> |
|
|
47 | 47 | <li><a href="#updateFileInExp">Modify an existing file in an experiment</a></li> |
|
|
48 | 48 | <li><a href="#changeFileStatus">Change the status of a file in an experiment</a></li> |
|
|
49 | 49 | <li><a href="#removeFileFromExp">Remove a file from an experiment</a></li> |
|
|
50 | 50 | <li><a href="#auxillary">Adding auxiliary plots and information to a Madrigal experiment</a></li> |
|
|
51 | 51 | </ul> |
|
|
52 | 52 | <p>After changing a Madrigal experiment using one or more of the scripts above, the script <em>madroot</em>/bin/updateMaster must be run for the changes to take effect. That's because these scripts change the metadata files in the local directory, and updateMaster combines all the local metadata files into the ones in <em>madroot</em>/metadata, which the used by the user interface to Madrigal. </p> |
|
|
53 | 53 | <p>This page also describes how you can add auxiliary information to a given Madrigal experiment. </p> |
|
|
54 | 54 | <h4><a name="createExpWithFile" id="createExpWithFile"></a>Create an experiment using a pre-existing Madrigal file</h4> |
|
|
55 | 55 | <p>If you have already created a Madrigal file for a given experiment, and want to add that new experiment to Madrigal, use the script createExpWithFile.py, located in <em>madroot</em>/bin. Usage:</p> |
|
|
56 | 56 | <pre> |
|
|
57 | 57 | createExpWithFile.py is a script used to create a new Madrigal experiment |
|
|
58 | 58 | based on an already existing file. Information such as the duration of the |
|
|
59 | 59 | experiment is obtained by analyzing the file. |
|
|
60 | 60 | |
|
|
61 | 61 | With Madrigal 3, accepts either old CEDAR database format files, or CEDAR Madrigal |
|
|
62 | 62 | Hdf5 files. |
|
|
63 | 63 | |
|
|
64 | 64 | Required arguments: |
|
|
65 | 65 | |
|
|
66 | 66 | --madFilename - full path to the complete Madrigal file. Basename will |
|
|
67 | 67 | be maintained. If old CEDAR databse format, .hdf5 will |
|
|
68 | 68 | be appended |
|
|
69 | 69 | |
|
|
70 | 70 | --expTitle - experiment title. Use quotes if title contains spaces. |
|
|
71 | 71 | |
|
|
72 | 72 | |
|
|
73 | 73 | --permission - 0 for public, 1 for private (restricted to certain IP range) |
|
|
74 | 74 | (both the experiment and the file will set) |
|
|
75 | 75 | |
|
|
76 | 76 | --fileDesc - file decription |
|
|
77 | 77 | |
|
|
78 | 78 | Optional arguments: |
|
|
79 | 79 | |
|
|
80 | 80 | --instCode - instrument code. If this argument missing, instrument code is |
|
|
81 | 81 | taken from file, but error is thrown if more than one kinst found. |
|
|
82 | 82 | |
|
|
83 | 83 | --category - 1=default, 2=variant, or 3=history If this argument is missing, |
|
|
84 | 84 | 1 (default) used. |
|
|
85 | 85 | |
|
|
86 | 86 | --dirName - directory name to use for experiment. If not given, the directory |
|
|
87 | 87 | name will be the default name DDmmmYY[optChar]. Cannot contain "/" |
|
|
88 | 88 | |
|
|
89 | 89 | --optChar - optional character to be added to experiment directory if no dirName |
|
|
90 | 90 | given. If dirName argument given, this argument ignored. optChar |
|
|
91 | 91 | is used if the default directory name DDmmmYY is used for |
|
|
92 | 92 | more than one experiment created for a given instrument on a given day. |
|
|
93 | 93 | For example, if --optChar=h for a MLH experiment on September 12, 2005, |
|
|
94 | 94 | then the experiment directory created would be experiments/2005/mlh/12sep05h. |
|
|
95 | 95 | |
|
|
96 | 96 | --kindat - Set file kindat independently from one (or more) in file |
|
|
97 | 97 | |
|
|
98 | 98 | --experimentsDirNum - the number to be appended to the experiments directory, if experiments |
|
|
99 | 99 | directory being used is of the form experiments[0-9]* instead of just |
|
|
100 | 100 | experiments. For example, if experimentsDirNum is 7, then the experiment |
|
|
101 | 101 | would be created in MADROOT/experiments7 instead of MADROOT/experiments. |
|
|
102 | 102 | Default is to create in experiments directory. |
|
|
103 | 103 | |
|
|
104 | 104 | --PI - set Principal Investigator for this experiment |
|
|
105 | 105 | |
|
|
106 | 106 | --PIEmail - set PI email for this experiment |
|
|
107 | 107 | |
|
|
108 | 108 | --fileAnalyst - set file analyst name for this file. This will default to blank. |
|
|
109 | 109 | |
|
|
110 | 110 | --fileAnalystEmail - set file analyst email for this file. This will default to blank. |
|
|
111 | 111 | |
|
|
112 | 112 | </pre> |
|
|
113 | 113 | <pre> </pre> |
|
|
114 | 114 | <p>Example: If you have already created the Madrigal file mlh060120g.001.hdf5 in the /tmp directory, |
|
|
115 | 115 | and the experiment name is Calibration, the file should be public, and the description is "Final", you would enter:</p> |
|
|
116 | 116 | <pre>/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.001.hdf5 \ |
|
|
117 | 117 | --expTitle="Calibration" --permission=0 --fileDesc="Final"</pre> |
|
|
118 | 118 | <p>If your input file is an old style CEDAR database file mlh060120g.001, the only difference is that there is not hdf5 extension:</p> |
|
|
119 | 119 | <pre>/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.001 \ |
|
|
120 | 120 | --expTitle="Calibration" --permission=0 --fileDesc="Final"</pre> |
|
|
121 | 121 | <h4><a name="createRTExp" id="createRTExp"></a>Create an experiment when no Madrigal file yet exists</h4> |
|
|
122 | 122 | <p>If you want to create an experiment for a Madrigal data yet to be created (such as when you want to create the Madrigal file in real-time), use the script createRTExp.py, located in <em>madroot</em>/bin. Usage: </p> |
|
|
123 | 123 | <pre>createRTExpWithFile.py is a script used to create a new Madrigal experiment |
|
|
124 | 124 | that will contain real-time files. These real-time files are assumed not to exist |
|
|
125 | 125 | yet.</pre> |
|
|
126 | 126 | <pre>Required arguments::</pre> |
|
|
127 | 127 | <pre> --startDate - experiment start date in form YYYY-MM-DD</pre> |
|
|
128 | 128 | <pre> --inst - instrument code or 3-letter Madrigal mnenonic</pre> |
|
|
129 | 129 | <pre> --expTitle - experiment title. Use quotes if title contains spaces.</pre> |
|
|
130 | 130 | <pre> --rtFiles - comma-separated list of realtime file basenames to be created</pre> |
|
|
131 | 131 | <pre> --kindats - comma-separated list of ints or single int of kindats for each realtime file. |
|
|
132 | 132 | The length and order must be the same as rtFiles. If only one |
|
|
133 | 133 | given, it is assumed that all rtFiles have the same kindat.</pre> |
|
|
134 | 134 | <pre> --fileDescs - comma-separated list of file descriptions. If the file description contains spaces, |
|
|
135 | 135 | quotes must be used.</pre> |
|
|
136 | 136 | <pre>Optional argument::</pre> |
|
|
137 | 137 | <pre> --numDays - number of days the experiment is estimated to run - if not given, and no start |
|
|
138 | 138 | and end times specified, defaults to one day. Error raised if endDate and endTime |
|
|
139 | 139 | also specified.</pre> |
|
|
140 | 140 | <pre> --startTime - start time in form HH:MM:DD. Defaults to 00:00:00</pre> |
|
|
141 | 141 | <pre> --endDate - end day in form YYYY-MM-DD. endTime must also be specified</pre> |
|
|
142 | 142 | <pre> --endTime - end time in form HH:MM:DD. endDate must also be specified</pre> |
|
|
143 | 143 | <pre> --permissions - comma-separated list of 0 for public, 1 for private (restricted to certain IP range). |
|
|
144 | 144 | If only one given, it is assumed it applied to all. If this argument is not |
|
|
145 | 145 | given, it defaults to 0 (public)</pre> |
|
|
146 | 146 | <pre> --dirName - directory name to use for experiment. If not given, the directory |
|
|
147 | 147 | name will be the default name DDmmmYY[optChar]. Cannot contain "/"</pre> |
|
|
148 | 148 | <pre> --optChar - optional character to be added to experiment directory if no dirName |
|
|
149 | 149 | given. If dirName argument given, this argument ignored. optChar |
|
|
150 | 150 | is used if the default directory name DDmmmYY is used for |
|
|
151 | 151 | more than one experiment created for a given instrument on a given day. |
|
|
152 | 152 | For example, if --optChar=h for a MLH experiment on September 12, 2005, |
|
|
153 | 153 | then the experiment directory created would be experiments/2005/mlh/12sep05h.</pre> |
|
|
154 | 154 | <pre> --security - overall experiment access. 0 for public, 1 for private, -1 for ignore. |
|
|
155 | 155 | Defaults to public (0)</pre> |
|
|
156 | 156 | <pre>--experimentsDirNum - the number to be appended to the experiments directory, if experiments |
|
|
157 | 157 | directory being used is of the form experiments[0-9]* instead of just |
|
|
158 | 158 | experiments. For example, if experimentsDirNum is 7, then the experiment |
|
|
159 | 159 | would be created in /Users/brideout/madroot31/experiments7 instead of MADROOT/experiments. |
|
|
160 | 160 | Default is to create in experiments directory.</pre> |
|
|
161 | 161 | |
|
|
162 | 162 | <pre> --PI - set Principal Investigator for this experiment</pre> |
|
|
163 | 163 | |
|
|
164 | 164 | <pre> --PIEmail - set PI email for this experiment</pre> |
|
|
165 | 165 | |
|
|
166 | 166 | <pre> --fileAnalyst - set file analyst name for these files (no commas allowed). |
|
|
167 | 167 | This will default to blank. If a different |
|
|
168 | 168 | fileAnalyst is desired for each file, the names must be comma separated.</pre> |
|
|
169 | 169 | |
|
|
170 | 170 | <pre> --fileAnalystEmail - set file analyst email for this file. This will default to blank. |
|
|
171 | 171 | If a different fileAnalyst email is desired for each file, the emails must be comma separated.</pre> |
|
|
172 | 172 | <p>Example: If you plan to create two real-time Madrigal files called mlh021001a.000.hdf5 and mlh021001b.000.hdf5 for the |
|
|
173 | 173 | Millstone Hill Radar for an experiment planned to run 2 days, with both files having the kindat 3410, you would enter:</p> |
|
|
174 | 174 | <pre>/opt/madrigal/bin/createRTExp.py --startDate=2002-10-01 --numDays=2 --inst=mlh --expTitle="test experiment" \ |
|
|
175 | 175 | --rtFiles=mlh021001a.000.hdf5,mlh021001b.000.hdf5 --kindats=3410 \ |
|
|
176 | 176 | --fileDescs="preliminary - single pulse,preliminary - alternating code" |
|
|
177 | 177 | </pre> |
|
|
178 | 178 | <h4><a name="changeExpStatus" id="changeExpStatus"></a>Modify the status of a given experiment</h4> |
|
|
179 | 179 | <p>All of the experiment attributes can be changed by the script changeExpStatus.py, located in <em>madroot</em>/bin. The most common reason you'd want to run this script is to change an experiment's security, with the options being 0 (public), 1 (<a href="ad_other.html#private">limited by IP address</a>), and -1 (ignored, or hidden from everyone). To completely remove an experiment, simply delete the directory. However, setting security to -1 allows you to bring the experiment back at some later time by running changeExpStatus.py again. </p> |
|
|
180 | 180 | <p>A number of other attributes can also be modified, but most are set automatically, and should not need modification.</p> |
|
|
181 | 181 | <p>Usage:</p> |
|
|
182 | 182 | <pre>changeExpStatus.py is a script used to change the status of an |
|
|
183 | 183 | existing Madrigal experiment. The following attributes can be changed:</pre> |
|
|
184 | 184 | <pre> expUrl |
|
|
185 | 185 | experiment name |
|
|
186 | 186 | siteID |
|
|
187 | 187 | start date |
|
|
188 | 188 | start time |
|
|
189 | 189 | end date |
|
|
190 | 190 | end time |
|
|
191 | 191 | instrument code |
|
|
192 | 192 | security (public, private. ignore) |
|
|
193 | 193 | PI (optional - overrides default for instrument) |
|
|
194 | 194 | PIEmail (optional - overrides default for instrument</pre> |
|
|
195 | 195 | <pre>Required argument: </pre> |
|
|
196 | 196 | <pre> --expDir - full path to experiment directory. Example: |
|
|
197 | 197 | "/opt/madrigal/experiments/1998/mlh/20jan98"</pre> |
|
|
198 | 198 | <pre>Optional arguments - set these to change an experiment attribute:</pre> |
|
|
199 | 199 | |
|
|
200 | 200 | <pre> --expUrl - must be in form <cgi base>/madtoc/YYYY/<3 letter lower case inst code>/<expDir> |
|
|
201 | 201 | example: http://www.haystack.mit.edu/cgi-bin/madtoc/1997/mlh/03dec97g</pre> |
|
|
202 | 202 | <pre> --expName - experiment name. Quotes required if contains spaces. Example: "World Day"</pre> |
|
|
203 | 203 | <pre> --siteID - Madrigal siteID of where data will be stored. Error raised if not the siteID |
|
|
204 | 204 | of the local Madrigal site. Example: 4</pre> |
|
|
205 | 205 | <pre> --startDate - new start date of experiment (UT). In form YYYY-MM-DD. Example: 1998-01-20</pre> |
|
|
206 | 206 | <pre> --startTime - new start time of experiment (UT). In form HH:MM:DD. Example: 12:30:00</pre> |
|
|
207 | 207 | <pre> --endDate - new end date of experiment (UT). In form YYYY-MM-DD. Example: 1998-01-21</pre> |
|
|
208 | 208 | <pre> --endTime - new end time of experiment (UT). In form HH:MM:DD. Example: 23:30:00</pre> |
|
|
209 | 209 | <pre> --inst - new instrument code. Example: 30</pre> |
|
|
210 | 210 | <pre> --security - new security code. Allowed values are 0 for public, 1 for private (limited IP range access) |
|
|
211 | 211 | -1 for ignore, 2 for archived experiment, 3 for private (limited IP range access) archived |
|
|
212 | 212 | experiment.</pre> |
|
|
213 | 213 | <pre> --PI - set Principal Investigator for this experiment</pre> |
|
|
214 | 214 | <pre> --PIEmail - set PI email for this experiment</pre> |
|
|
215 | 215 | <p>Example: to change to experiment /opt/madrigal/experiments/2006/mlh/20jan to be private, you would run: </p> |
|
|
216 | 216 | <pre>/opt/madrigal/bin/changeExpStatus.py --expDir=/opt/madrigal/experiments/2006/mlh/20jan06 --security=1 |
|
|
217 | 217 | |
|
|
218 | 218 | </pre> |
|
|
219 | 219 | <h4><a name="addFileToExp" id="addFileToExp"></a>Add a new file to an experiment</h4> |
|
|
220 | 220 | <p>If you want to add a new file to an existing Madrigal experiment, use addFileToExp.py, located in <em>madroot</em>/bin. You can use this script to add a completely new file, or to update an existing one. It you update an existing file and want to change the status of the older file to history or variant, use the script <a href="#changeFileStatus">changeFileStatus.py</a>. </p> |
|
|
221 | 221 | <p>Usage: </p> |
|
|
222 | 222 | <pre> |
|
|
223 | 223 | addFileToExp.py is a script used to add a new Madrigal file to an |
|
|
224 | 224 | existing experiment. Information such as the duration of the |
|
|
225 | 225 | experiment is updated by analyzing the file. |
|
|
226 | 226 | |
|
|
227 | 227 | With Madrigal 3, accepts either old CEDAR database format files, or CEDAR Madrigal |
|
|
228 | 228 | Hdf5 files. |
|
|
229 | 229 | |
|
|
230 | 230 | Required arguments: |
|
|
231 | 231 | |
|
|
232 | 232 | --madFilename - full path to the complete Madrigal file. Basename will |
|
|
233 | 233 | be maintained. If old CEDAR databse format, .hdf5 will |
|
|
234 | 234 | be appended |
|
|
235 | 235 | |
|
|
236 | 236 | --expDir - full path to experiment directory. Example: |
|
|
237 | 237 | "/opt/madrigal/experiments/1998/mlh/20jan98" |
|
|
238 | 238 | |
|
|
239 | 239 | --permission - 0 for public, 1 for private (restricted to certain IP range) |
|
|
240 | 240 | |
|
|
241 | 241 | --fileDesc - file decription |
|
|
242 | 242 | |
|
|
243 | 243 | Optional arguments: |
|
|
244 | 244 | |
|
|
245 | 245 | --category - 1=default, 2=variant, or 3=history If this argument is missing, |
|
|
246 | 246 | 1 (default) used. |
|
|
247 | 247 | |
|
|
248 | 248 | --kindat - Set file kindat independently from one (or more) in file |
|
|
249 | 249 | |
|
|
250 | 250 | --skipNotify - if this flag set (no arguments), registered users will NOT be notified |
|
|
251 | 251 | of this change. The default is to email all registered users. |
|
|
252 | 252 | |
|
|
253 | 253 | --fileAnalyst - set file analyst name for this file. This will default to blank. |
|
|
254 | 254 | |
|
|
255 | 255 | --fileAnalystEmail - set file analyst email for this file. This will default to blank. |
|
|
256 | 256 | </pre> |
|
|
257 | 257 | <p>Example: If you want to add the new file /tmp/mlh060120g.002.hdf to an existing experiment, you would enter:</p> |
|
|
258 | 258 | <pre>/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.002.hdf5 \ |
|
|
259 | 259 | --expDir=/opt/madrigal/experiments/2006/mlh/20jan06 --permission=0 \ |
|
|
260 | 260 | --fileDesc="alternative analysis" --category=1 |
|
|
261 | 261 | </pre> |
|
|
262 | 262 | <p>If you are adding an old style Cedar databse file /tmp/mlh060120g.002 to an existing experiment, you would enter the same command except without the .hdf5 extension:</p> |
|
|
263 | 263 | <pre>/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.002 \ |
|
|
264 | 264 | --expDir=/opt/madrigal/experiments/2006/mlh/20jan06 --permission=0 \ |
|
|
265 | 265 | --fileDesc="alternative analysis" --category=1 |
|
|
266 | 266 | </pre> |
|
|
267 | 267 | <h4><a name="updateFileInExp" id="updateFileInExp"></a>Modify an existing file in an experiment</h4> |
|
|
268 | 268 | <p>If you want to modify an existing file in a Madrigal experiment, use updateFileInExp.py, located in <em>madroot</em>/bin. Note that if the modification in the file is significant, it is preferable to make the old file a history file using <a href="#changeFileStatus">changeFileStatus.py</a>, and to add a new file with a different name using <a href="#addFileToExp">addFileToExp.py</a>.</p> |
|
|
269 | 269 | <p>Usage: </p> |
|
|
270 | 270 | <pre>updateFileInExp.py is a script used to update an existing Madrigal file in an |
|
|
271 | 271 | existing experiment. Information such as the duration of the |
|
|
272 | 272 | experiment is updated by analyzing the file. This script is use to replace |
|
|
273 | 273 | an existing Madrigal file. Use addFileToExp.py to add a new file, and |
|
|
274 | 274 | changeFileStatus.py to change any file attribute.</pre> |
|
|
275 | 275 | <pre>Required arguments:</pre> |
|
|
276 | 276 | <pre> --madFilename - full path to the new version of the Madrigal file. Basename will |
|
|
277 | 277 | be maintained.</pre> |
|
|
278 | 278 | <pre> --expDir - full path to experiment directory. Example: |
|
|
279 | 279 | "/opt/madrigal/experiments/1998/mlh/20jan98" |
|
|
280 | 280 | </pre> |
|
|
281 | 281 | <p>Example: To modify the existing file /opt/madrigal/experiments/2002/01oct02/mlh021001a.000.hdf5 with the file /tmp/mlh021001a.000.hdf5, you would enter:</p> |
|
|
282 | 282 | <pre>/opt/madrigal/bin/updateFileInExp.py --madFilename=/tmp/mlh021001a.000.hdf5 \ |
|
|
283 | 283 | --expDir=/opt/madrigal/experiments/2002/01oct02 </pre> |
|
|
284 | 284 | <h4><a name="changeFileStatus" id="changeFileStatus"></a>Change the status of a file in an experiment</h4> |
|
|
285 | 285 | <p>If you want to change the status of any existing file in an experiment, such as to make a default file into a history file, use changeFileStatus.py, located in <em>madroot</em>/bin. Usage: </p> |
|
|
286 | 286 | <pre> |
|
|
287 | 287 | changeFileStatus.py is a script used to change the status of an |
|
|
288 | 288 | existing Madrigal file. The file permission, the file description, |
|
|
289 | 289 | or the file category can be changed. |
|
|
290 | 290 | |
|
|
291 | 291 | Required arguments: |
|
|
292 | 292 | |
|
|
293 | 293 | --filename - basename of existing Madrigal file. |
|
|
294 | 294 | |
|
|
295 | 295 | --expDir - full path to experiment directory. Example: |
|
|
296 | 296 | "/opt/madrigal/experiments/1998/mlh/20jan98" |
|
|
297 | 297 | |
|
|
298 | 298 | Optional arguments - set these to change a file attribute: |
|
|
299 | 299 | |
|
|
300 | 300 | --permission - 0 for public, 1 for private (restricted to certain IP range) |
|
|
301 | 301 | |
|
|
302 | 302 | --fileDesc - file decription |
|
|
303 | 303 | |
|
|
304 | 304 | --category - 1=default, 2=variant, 3=history, or 4=realtime |
|
|
305 | 305 | |
|
|
306 | 306 | --fileAnalyst - set file analyst name for this file. |
|
|
307 | 307 | |
|
|
308 | 308 | --fileAnalystEmail - set file analyst email for this file. |
|
|
309 | 309 | </pre> |
|
|
310 | 310 | <p>Example: If you want to change the status of /opt/madrigal/experiments/2002/01oct02/mlh021001a.000.hdf5 to be a history file, you would enter:</p> |
|
|
311 | 311 | <pre> |
|
|
312 | 312 | /opt/madrigal/bin/changeFileStatus.py --filename=mlh021001a.000.hdf5 \ |
|
|
313 | 313 | --expDir=/opt/madrigal/experiments/2002/01oct02 \ |
|
|
314 | 314 | --category=3</pre> |
|
|
315 | 315 | <h4><a name="removeFileFromExp" id="removeFileFromExp"></a>Remove a file from an experiment</h4> |
|
|
316 | 316 | <p>To completely remove a file from an existing experiment, rather than simply make it a history file, use removeFileFromExp.py, located in <em>madroot</em>/bin. Usage: </p> |
|
|
317 | 317 | <pre> |
|
|
318 | 318 | removeFileFromExp.py is a script used to remove an existing Madrigal file from an |
|
|
319 | 319 | existing experiment. Information such as the duration of the |
|
|
320 | 320 | experiment is updated by analyzing the remaining files. </pre> |
|
|
321 | 321 | <pre>Required arguments:</pre> |
|
|
322 | 322 | <pre> --filename - basename of the Madrigal file to be removed.</pre> |
|
|
323 | 323 | <pre> --expDir - full path to experiment directory. Example: |
|
|
324 | 324 | "/opt/madrigal/experiments/1998/mlh/20jan98" |
|
|
325 | 325 | </pre> |
|
|
326 | 326 | <p>Example: To remove the file /opt/madrigal/experiments/2002/01oct02/mlh021001a.000.hdf5, you would enter:</p> |
|
|
327 | 327 | <pre>/opt/madrigal/bin/removeFileFromExp.py --filename=mlh021001a.000.hdf5 \ |
|
|
328 | 328 | --expDir=/opt/madrigal/experiments/2002/01oct02 |
|
|
329 | 329 | </pre> |
|
|
330 | 330 | <h4><a name="auxillary" id="auxillary"></a>Adding auxiliary plots and information to a Madrigal experiment</h4> |
|
|
331 | 331 | <p>Since Madrigal is a web-based application, you can also display auxiliary plots and information about your experiment as web pages. If you add additional web documents according to the rules below, these documents will also show up through the standard Madrigal interface. </p> |
|
|
332 | 332 | <ul> |
|
|
333 | 333 | <li>Subdirectories containing an html file named index.html . Links to these files will show up the experiment page. The |
|
|
334 | 334 | page title will be displayed as a link.</li> |
|
|
335 | 335 | <li>Html pages in the main directory, and again links to these files will show up the experiment page. The |
|
|
336 | 336 | page title will be displayed as a link.</li> |
|
|
337 | 337 | <li>Plots or other files related to individual records in the dataset. Links to these files appear next to the appropriate record in the summarizeCedarFile.cgi page. These files must be located in the subdirectory "plots/[file name]/records" under the experiment directory, where "file name" is the base name of the experiment record.. In order for the script to determine which file goes with which record, the files must include a five digit number somewhere in its name. For example, plot00027.gif would appear as a link next to record 27 in summarizeCedarFile.cgi. More than one file can be associated with a given record. |
|
|
338 | 338 | <ul> |
|
|
339 | 339 | <li>If you want to create individual record plots for an incoherent scatter radar with the standard parameters Te, Ti, Ne, and ion velocity, simply run the script createRecordPlots.py. For example, to create all the individual record plots for the Madrigal file mlh980120g.001.hdf5, you would run a command similar to the following:</li> |
|
|
340 | 340 | </ul> |
|
|
341 | 341 | <blockquote> |
|
|
342 | 342 | <pre>/opt/madrigal/bin/createRecordPlots.py /opt/madrigal/experiments/1998/mlh/20jan98/mil980120g.001.hdf5</pre> |
|
|
343 | 343 | </blockquote> |
|
|
344 | 344 | </li> |
|
|
345 | 345 | </ul> |
|
|
346 | <p>As an example of plots associated with individual records, lets say you have a Madrigal experiment in the directory /opt/madrigal/experiments/2002/01oct02, with a file named mlh021001a.001.hdf5. If this file had 10 records, you could then create 10 plot files called plot001.png through plot010.png. You would then create the subdirectories plots/mlh021001a.001.hdf5/ under the main directory /opt/madrigal/experiments/2002/01oct02/, and put those 10 plot files there. </p> | |
|
|
346 | <p>As an example of plots associated with individual records, lets say you have a Madrigal experiment in the directory /opt/madrigal/experiments/2002/01oct02, with a file named mlh021001a.001.hdf5. If this file had 10 records, you could then create 10 plot files called plot001.png' %} through plot010.png' %}. You would then create the subdirectories plots/mlh021001a.001.hdf5/ under the main directory /opt/madrigal/experiments/2002/01oct02/, and put those 10 plot files there. </p> | |
|
|
347 | 347 | <p><br> |
|
|
348 | 348 | </p> |
|
|
349 | 349 | <!-- InstanceEndEditable --> |
|
|
350 | 350 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
351 | 351 | <tr> |
|
|
352 |
<td width="5%"><a href="{% url 'docs' 'ad_createFiles.html' %}"><img src=" |
|
|
|
353 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
354 |
<td width="5%"><a href="{% url 'docs' 'ad_other.html' %}"><img src=" |
|
|
|
352 | <td width="5%"><a href="{% url 'docs' 'ad_createFiles.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
353 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
354 | <td width="5%"><a href="{% url 'docs' 'ad_other.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
355 | 355 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Creating and modifying Madrigal experiments<!-- InstanceEndEditable --></td> |
|
|
356 | 356 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
357 | 357 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
358 | 358 | </tr> |
|
|
359 | 359 | </table> |
|
|
360 | 360 | <div class='online-navigation'> |
|
|
361 | 361 | <b class="navlabel">Previous:</b> |
|
|
362 | 362 | <a class="sectref" href="{% url 'docs' 'ad_createFiles.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Creating Madrigal data files <!-- InstanceEndEditable --></A> |
|
|
363 | 363 | <b class="navlabel"> Up:</b> |
|
|
364 | 364 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
365 | 365 | <b class="navlabel"> Next:</b> |
|
|
366 | 366 | <a class="sectref" href="{% url 'docs' 'ad_other.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Other admin tasks <!-- InstanceEndEditable --></A></div> |
|
|
367 | 367 | <hr/> |
|
|
368 | 368 | <p> </p> |
|
|
369 | 369 | </body> |
|
|
370 | 370 | <!-- InstanceEnd --></html> |
| @@ -1,615 +1,615 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Creating and editing Madrigal data files </title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --> |
|
|
10 | 10 | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | td.linenos { background-color: #f0f0f0; padding-right: 10px; } |
|
|
13 | 13 | span.lineno { background-color: #f0f0f0; padding: 0 5px 0 5px; } |
|
|
14 | 14 | pre { line-height: 125%; } |
|
|
15 | 15 | body .hll { background-color: #ffffcc } |
|
|
16 | 16 | body { background: #f8f8f8; } |
|
|
17 | 17 | body .c { color: #408080; font-style: italic } /* Comment */ |
|
|
18 | 18 | body .err { border: 1px solid #FF0000 } /* Error */ |
|
|
19 | 19 | body .k { color: #008000; font-weight: bold } /* Keyword */ |
|
|
20 | 20 | body .o { color: #666666 } /* Operator */ |
|
|
21 | 21 | body .ch { color: #408080; font-style: italic } /* Comment.Hashbang */ |
|
|
22 | 22 | body .cm { color: #408080; font-style: italic } /* Comment.Multiline */ |
|
|
23 | 23 | body .cp { color: #BC7A00 } /* Comment.Preproc */ |
|
|
24 | 24 | body .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */ |
|
|
25 | 25 | body .c1 { color: #408080; font-style: italic } /* Comment.Single */ |
|
|
26 | 26 | body .cs { color: #408080; font-style: italic } /* Comment.Special */ |
|
|
27 | 27 | body .gd { color: #A00000 } /* Generic.Deleted */ |
|
|
28 | 28 | body .ge { font-style: italic } /* Generic.Emph */ |
|
|
29 | 29 | body .gr { color: #FF0000 } /* Generic.Error */ |
|
|
30 | 30 | body .gh { color: #000080; font-weight: bold } /* Generic.Heading */ |
|
|
31 | 31 | body .gi { color: #00A000 } /* Generic.Inserted */ |
|
|
32 | 32 | body .go { color: #888888 } /* Generic.Output */ |
|
|
33 | 33 | body .gp { color: #000080; font-weight: bold } /* Generic.Prompt */ |
|
|
34 | 34 | body .gs { font-weight: bold } /* Generic.Strong */ |
|
|
35 | 35 | body .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ |
|
|
36 | 36 | body .gt { color: #0044DD } /* Generic.Traceback */ |
|
|
37 | 37 | body .kc { color: #008000; font-weight: bold } /* Keyword.Constant */ |
|
|
38 | 38 | body .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */ |
|
|
39 | 39 | body .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */ |
|
|
40 | 40 | body .kp { color: #008000 } /* Keyword.Pseudo */ |
|
|
41 | 41 | body .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */ |
|
|
42 | 42 | body .kt { color: #B00040 } /* Keyword.Type */ |
|
|
43 | 43 | body .m { color: #666666 } /* Literal.Number */ |
|
|
44 | 44 | body .s { color: #BA2121 } /* Literal.String */ |
|
|
45 | 45 | body .na { color: #7D9029 } /* Name.Attribute */ |
|
|
46 | 46 | body .nb { color: #008000 } /* Name.Builtin */ |
|
|
47 | 47 | body .nc { color: #0000FF; font-weight: bold } /* Name.Class */ |
|
|
48 | 48 | body .no { color: #880000 } /* Name.Constant */ |
|
|
49 | 49 | body .nd { color: #AA22FF } /* Name.Decorator */ |
|
|
50 | 50 | body .ni { color: #999999; font-weight: bold } /* Name.Entity */ |
|
|
51 | 51 | body .ne { color: #D2413A; font-weight: bold } /* Name.Exception */ |
|
|
52 | 52 | body .nf { color: #0000FF } /* Name.Function */ |
|
|
53 | 53 | body .nl { color: #A0A000 } /* Name.Label */ |
|
|
54 | 54 | body .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */ |
|
|
55 | 55 | body .nt { color: #008000; font-weight: bold } /* Name.Tag */ |
|
|
56 | 56 | body .nv { color: #19177C } /* Name.Variable */ |
|
|
57 | 57 | body .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */ |
|
|
58 | 58 | body .w { color: #bbbbbb } /* Text.Whitespace */ |
|
|
59 | 59 | body .mb { color: #666666 } /* Literal.Number.Bin */ |
|
|
60 | 60 | body .mf { color: #666666 } /* Literal.Number.Float */ |
|
|
61 | 61 | body .mh { color: #666666 } /* Literal.Number.Hex */ |
|
|
62 | 62 | body .mi { color: #666666 } /* Literal.Number.Integer */ |
|
|
63 | 63 | body .mo { color: #666666 } /* Literal.Number.Oct */ |
|
|
64 | 64 | body .sb { color: #BA2121 } /* Literal.String.Backtick */ |
|
|
65 | 65 | body .sc { color: #BA2121 } /* Literal.String.Char */ |
|
|
66 | 66 | body .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */ |
|
|
67 | 67 | body .s2 { color: #BA2121 } /* Literal.String.Double */ |
|
|
68 | 68 | body .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */ |
|
|
69 | 69 | body .sh { color: #BA2121 } /* Literal.String.Heredoc */ |
|
|
70 | 70 | body .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */ |
|
|
71 | 71 | body .sx { color: #008000 } /* Literal.String.Other */ |
|
|
72 | 72 | body .sr { color: #BB6688 } /* Literal.String.Regex */ |
|
|
73 | 73 | body .s1 { color: #BA2121 } /* Literal.String.Single */ |
|
|
74 | 74 | body .ss { color: #19177C } /* Literal.String.Symbol */ |
|
|
75 | 75 | body .bp { color: #008000 } /* Name.Builtin.Pseudo */ |
|
|
76 | 76 | body .vc { color: #19177C } /* Name.Variable.Class */ |
|
|
77 | 77 | body .vg { color: #19177C } /* Name.Variable.Global */ |
|
|
78 | 78 | body .vi { color: #19177C } /* Name.Variable.Instance */ |
|
|
79 | 79 | body .il { color: #666666 } /* Literal.Number.Integer.Long */ |
|
|
80 | 80 | |
|
|
81 | 81 | </style> |
|
|
82 | 82 | |
|
|
83 | 83 | <!-- InstanceEndEditable --> |
|
|
84 |
<link href=" |
|
|
|
84 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
85 | 85 | <style type="text/css"> |
|
|
86 | 86 | html body { |
|
|
87 | 87 | background-color: {{bg_color}}; |
|
|
88 | 88 | } |
|
|
89 | 89 | </style> |
|
|
90 | 90 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_createExp.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_experiments.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_experiments.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_createExp.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_experiments.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_createExp.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_experiments.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_createExp.html" --> |
|
|
91 | 91 | </head> |
|
|
92 | 92 | |
|
|
93 | 93 | <body> |
|
|
94 | 94 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
95 | 95 | <tr> |
|
|
96 |
<td width="5%"><a href="{% url 'docs' 'ad_experiments.html' %}"><img src=" |
|
|
|
97 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
98 |
<td width="5%"><a href="{% url 'docs' 'ad_createExp.html' %}"><img src=" |
|
|
|
96 | <td width="5%"><a href="{% url 'docs' 'ad_experiments.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
97 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
98 | <td width="5%"><a href="{% url 'docs' 'ad_createExp.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
99 | 99 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Creating and editing Madrigal data files <!-- InstanceEndEditable --></td> |
|
|
100 | 100 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
101 | 101 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
102 | 102 | </tr> |
|
|
103 | 103 | </table> |
|
|
104 | 104 | <div class='online-navigation'> |
|
|
105 | 105 | <b class="navlabel">Previous:</b> |
|
|
106 | 106 | <a class="sectref" href="{% url 'docs' 'ad_experiments.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Madrigal data organization <!-- InstanceEndEditable --></A> |
|
|
107 | 107 | <b class="navlabel"> Up:</b> |
|
|
108 | 108 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
109 | 109 | <b class="navlabel"> Next:</b> |
|
|
110 | 110 | <a class="sectref" href="{% url 'docs' 'ad_createExp.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Creating Madrigal experiments <!-- InstanceEndEditable --></A></div> |
|
|
111 | 111 | <hr/> |
|
|
112 | 112 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
113 | 113 | <h1 align="center">Creating and editing Madrigal data files</h1> |
|
|
114 |
<p>A key element in administering the Madrigal database is the ability to create and edit Madrigal data files. An ambitious Madrigal administrator could read the <a href=" |
|
|
|
114 | <p>A key element in administering the Madrigal database is the ability to create and edit Madrigal data files. An ambitious Madrigal administrator could read the <a href="{% static 'CEDARMadrigalHdf5Format.pdf" target="_blank">CEDAR Madrigal Hdf5 format</a> description and write their own code from scratch. However, Madrigal provides API's and examples in two languages, Python and Matlab, to make this chore much, much easier. This section describes how to create Madrigal files using each of those two languages.</p> | |
|
|
115 | 115 | <ul> |
|
|
116 | 116 | <li><a href="#python">Python</a></li> |
|
|
117 | 117 | <li><a href="#matlab">Matlab</a></li> |
|
|
118 | 118 | </ul> |
|
|
119 | 119 | <h2><a name="python" id="python"></a>Python</h2> |
|
|
120 | 120 | <h3>How have things changed with Madrigal 3?</h3> |
|
|
121 | 121 | <p>For the most part, python scripts that created Madrigal 2 files will only need to be slightly modified to create Madrigal 3 files. With Madrigal 3, the independent parameters (excluding time) must be explicitly declared. This is because Madrigal 3 files contain both a table layout, and (if there are independent parameters) a grid layout, with the number of dimensions = 1 + (number of independent parameters). The first dimension is always time. The independent parameters are declared in the MadrigalDataRecord init method.</p> |
|
|
122 | 122 | <p>The second small change is the addition of an optional argument <em>arraySplitParms</em>. If one of more array splitting parameters are given, then multiple grids of data will be created, one grid for each unique combination of values of the array splitting parameters. The idea behind this argument is to make the gridded data less sparse and more useful to the end user. For example, a phased array incoherent scatter radar such as PFISR that simultaneously measures along different beams would split its arrays by the parameter beam_id. This would make the gridded data both dense and more user friendly - a user could simply open the data for the beam of interest.</p> |
|
|
123 | 123 | <h2>Instructions</h2> |
|
|
124 | 124 | <p>This section gives an introduction to using the madrigal python API to create new Cedar Madrigal Hdf5 files, and to edit existing Cedar Madrigal Hdf5 files. Examples are given of <a href="#create"> creating new normal-sized files, </a> <a href="#create_large">creating new large files</a>, and <a href="#edit">editing existing files.</a> Users creating Cedar Madrigal Hdf5 files with python can choose between two sightly different patterns, one which maximizes speed but could have memory issues with very large files, and another that limits memory use with very large files, but is somewhat slower. Complete <a href="madrigal/cedar.m.html">documentation</a> is available as part of the Madrigal Python API documentation. </p> |
|
|
125 | 125 | <p>The python cedar module was written to simplify the task of creating Cedar Madrigal Hdf5 files as much as possible. The user of this module only needs to know the following about the Cedar format: </p> |
|
|
126 | 126 | <ul> |
|
|
127 | 127 | <li>Cedar Madrigal Hdf5 files are made up of metadata and data records. Each data record consists of a measurement made with a single instrument over a single interval of time.</li> |
|
|
128 | 128 | <li>All parameters in Cedar Madrigal Hdf5 files are defined in the standard. Contact Bill Rideout if you can't find a parameter you need; more can be added.</li> |
|
|
129 | 129 | <li>Each data record in a Cedar Madrigal Hdf5 file has the same parameters. These parameters are either scalar (a single measurement per data record), or vectors (called 2D sometimes for historical reasons, but there is a limit of 5 for the number of independent spatial parameters).</li> |
|
|
130 | 130 | <li>If there are vector parameters, then the parameters which serve as independent parameters must be specified. Up to 5 may be given.</li> |
|
|
131 | 131 | <li>The Cedar Madrigal Hdf5 always contains a Table Layout, where the data is listed in a flat table, which means the scalar parameters are repeated for each row in a single data record.</li> |
|
|
132 | 132 | <li>If there are vector parameters, then there will also be one or more Array Layouts. The array layout is a multidimensional layout, with time in one dimension, and one dimension for each independent parameter. When you create this file, you can also choose to specify parameters used to split these arrays. For example, incoherent scatter radar taken with radars that measure different beams over the same time period may decide to have separate arrays for each beam code.</li> |
|
|
133 | 133 | <li>A list of Cedar parameters and their units can be found under<em> Access metadata</em> menu item from any Madrigal 3 site. A complete description of each parameter can also be seen by clicking on any parameter name. Parameters can be referred to in the python API either by case-insensitive mnemonics (e.g., "Gdalt") or by integer id. Parameters are set either to float, integer, or string values, according to their type, or to the special values 'missing', 'assumed', or 'knownbad' may be used for numeric values. The values 'assumed' and 'knownbad' can only be applied to error parameters.</li> |
|
|
134 | 134 | <li>In addition to data records, the Cedar format also allows two types of records that contain human-readable descriptions of the data: a catalog record and a header record. They are two only for historical reasons; each contains different metadata fields.</li> |
|
|
135 | 135 | </ul> |
|
|
136 | 136 | <blockquote> |
|
|
137 | 137 | <h3>MadrigalCedarFile</h3> |
|
|
138 | 138 | </blockquote> |
|
|
139 | 139 | <p>The high level object in the cedar module is <a href="madrigal/cedar.m.html"> MadrigalCedarFile</a>. This class emulates a python list, and so users may treat it just like a python list. The restriction enforced is that all items in the list must be either <a href="madrigal/cedar.m.html">MadrigalCatalogRecords</a>, <a href="madrigal/cedar.m.html">MadrigalHeaderRecords</a>, or <a href="madrigal/cedar.m.html">MadrigalDataRecords</a>. Each of these three classes supports the method getType(), which returns 'catalog', 'header', and 'data', respectively.</p> |
|
|
140 | 140 | <blockquote> |
|
|
141 | 141 | <a name="catHeadRec" id="catHeadRec"></a><h3>Using Python to add catalog and header records.</h3> |
|
|
142 | 142 | </blockquote> |
|
|
143 | 143 | <p>Cedar catalog and header records can be difficult to create. With the cedar module |
|
|
144 | 144 | <a href="madrigal/cedar.m.html"> CatalogHeaderCreator</a>, creating |
|
|
145 | 145 | these records should now be much easier. Catalog and header records contain a lot of information |
|
|
146 | 146 | that can be deduced from the data - this includes which parameters are present, minimum and maximums |
|
|
147 | 147 | of certain parameters, and start and stop times. The only parts of the catalog or header record that |
|
|
148 | 148 | can't be determined automatically are some optional descriptive text fields. With this new module, |
|
|
149 | 149 | you simply pass in strings of any length if you want to fill in one of those optional descriptive |
|
|
150 | 150 | text fields - the module will handle all formating for you.</p> |
|
|
151 | 151 | <p>Here's a list of the optional descriptive text fields for a <i>catalog</i> record:</p> |
|
|
152 | 152 | <ul> |
|
|
153 | 153 | <li><b>principleInvestigator: </b>names of responsible Principal Investigator(s) or others knowledgeable |
|
|
154 | 154 | about the experiment</li> |
|
|
155 | 155 | <li><b>expPurpose: </b>brief description of the experiment purpose</li> |
|
|
156 | 156 | <li><b>expMode: </b>further elaboration of meaning of experiment mode; e.g. antenna patterns and |
|
|
157 | 157 | pulse sequences</li> |
|
|
158 | 158 | <li><b>cycleTime: </b>minutes for one full measurement cycle (this needs to be a number)</li> |
|
|
159 | 159 | <li><b>correlativeExp: </b>any correlative experiments (experiments with related data)</li> |
|
|
160 | 160 | <li><b>sciRemarks: </b>scientific remarks</li> |
|
|
161 | 161 | <li><b>instRemarks: </b>instrument remarks</li> |
|
|
162 | 162 | </ul> |
|
|
163 | 163 | <p>Here's a list of the optional descriptive text fields for a <i>header</i> record:</p> |
|
|
164 | 164 | <ul> |
|
|
165 | 165 | <li><b>kindatDesc: </b>description of how this data was analyzed (the kind of data)</li> |
|
|
166 | 166 | <li><b>analyst: </b>name of person(s) who analyzed this data</li> |
|
|
167 | 167 | <li><b>comments: </b>additional comments about data (describe any instrument-specific parameters)</li> |
|
|
168 | 168 | <li><b>history: </b>a description of the history of the processing of this file</li> |
|
|
169 | 169 | </ul> |
|
|
170 | 170 | <p><a name="create" id="create"></a> </p> |
|
|
171 | 171 | <blockquote> |
|
|
172 | 172 | <h3>Creating a new normal-sized file example</h3> |
|
|
173 | 173 | </blockquote> |
|
|
174 | 174 | <div class="highlight"> |
|
|
175 | 175 | <pre><span></span><span class="sd">"""createSample.py shows an example of creating an entirely new Madrigal</span> |
|
|
176 | 176 | <span class="sd">file using the Python cedar module. In particular, it creates a file with</span> |
|
|
177 | 177 | <span class="sd">a catalog record, a header record, and two data records. The data records</span> |
|
|
178 | 178 | <span class="sd">contain two 1D parameters (System temperature - SYSTMP and Transmitter</span> |
|
|
179 | 179 | <span class="sd">Frequency TFREQ) and five 2D parameters (GDALT, GDLAT, GLON, and TR, and DTR)</span> |
|
|
180 | 180 | <span class="sd">The independent spatial parameter in this example is just GDALT.</span> |
|
|
181 | 181 | |
|
|
182 | 182 | <span class="sd">This example uses the normal-sized pattern for speed, where normal sized is considered |
|
|
183 | 183 | less than 3000 records</span><span class="sd">. In that pattern, all the records are appended to the MadrigalCedarFile </span> |
|
|
184 | 184 | <span class="sd">object, and then write is called to write everything to file at once, including both the Table and </span> |
|
|
185 | 185 | <span class="sd">Array Layouts.</span> |
|
|
186 | 186 | |
|
|
187 | 187 | <span class="sd">$Id$</span> |
|
|
188 | 188 | <span class="sd">"""</span> |
|
|
189 | 189 | |
|
|
190 | 190 | <span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">os.path</span> |
|
|
191 | 191 | <span class="kn">import</span> <span class="nn">types</span> |
|
|
192 | 192 | <span class="kn">import</span> <span class="nn">datetime</span> |
|
|
193 | 193 | |
|
|
194 | 194 | <span class="kn">import</span> <span class="nn">madrigal.metadata</span> |
|
|
195 | 195 | <span class="kn">import</span> <span class="nn">madrigal.cedar</span> |
|
|
196 | 196 | |
|
|
197 | 197 | <span class="c1">################# sample data #################</span> |
|
|
198 | 198 | |
|
|
199 | 199 | <span class="n">kinst</span> <span class="o">=</span> <span class="mi">30</span> <span class="c1"># instrument identifier of Millstone Hill ISR</span> |
|
|
200 | 200 | <span class="n">kindat</span> <span class="o">=</span> <span class="mi">3408</span> <span class="c1"># id of kind of data processing</span> |
|
|
201 | 201 | <span class="n">nrow</span> <span class="o">=</span> <span class="mi">5</span> <span class="c1"># all data records have 5 2D rows</span> |
|
|
202 | 202 | |
|
|
203 | 203 | <span class="n">SYSTMP</span> <span class="o">=</span> <span class="p">(</span><span class="mf">120.0</span><span class="p">,</span> <span class="mf">122.0</span><span class="p">)</span> |
|
|
204 | 204 | <span class="n">TFREQ</span> <span class="o">=</span> <span class="p">(</span><span class="mf">4.4E8</span><span class="p">,</span> <span class="mf">4.4E8</span><span class="p">)</span> |
|
|
205 | 205 | |
|
|
206 | 206 | <span class="n">GDALT</span> <span class="o">=</span> <span class="p">((</span><span class="mf">70.0</span><span class="p">,</span> <span class="mf">100.0</span><span class="p">,</span> <span class="mf">200.0</span><span class="p">,</span> <span class="mf">300.0</span><span class="p">,</span> <span class="mf">400.0</span><span class="p">),</span> |
|
|
207 | 207 | <span class="p">(</span><span class="mf">70.0</span><span class="p">,</span> <span class="mf">100.0</span><span class="p">,</span> <span class="mf">200.0</span><span class="p">,</span> <span class="mf">300.0</span><span class="p">,</span> <span class="mf">400.0</span><span class="p">))</span> |
|
|
208 | 208 | |
|
|
209 | 209 | <span class="n">GDLAT</span> <span class="o">=</span> <span class="p">((</span><span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">),</span> |
|
|
210 | 210 | <span class="p">(</span><span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">))</span> |
|
|
211 | 211 | |
|
|
212 | 212 | <span class="n">GLON</span> <span class="o">=</span> <span class="p">((</span><span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">),</span> |
|
|
213 | 213 | <span class="p">(</span><span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">))</span> |
|
|
214 | 214 | |
|
|
215 | 215 | <span class="n">TR</span> <span class="o">=</span> <span class="p">((</span><span class="s1">'missing'</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">2.3</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">),</span> |
|
|
216 | 216 | <span class="p">(</span><span class="s1">'missing'</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.7</span><span class="p">,</span> <span class="mf">2.4</span><span class="p">,</span> <span class="mf">3.1</span><span class="p">))</span> |
|
|
217 | 217 | |
|
|
218 | 218 | |
|
|
219 | 219 | <span class="n">DTR</span> <span class="o">=</span> <span class="p">((</span><span class="s1">'missing'</span><span class="p">,</span> <span class="s1">'assumed'</span><span class="p">,</span> <span class="s1">'assumed'</span><span class="p">,</span> <span class="mf">0.3</span><span class="p">,</span> <span class="mf">0.7</span><span class="p">),</span> |
|
|
220 | 220 | <span class="p">(</span><span class="s1">'missing'</span><span class="p">,</span> <span class="s1">'assumed'</span><span class="p">,</span> <span class="mf">0.7</span><span class="p">,</span> <span class="mf">0.4</span><span class="p">,</span> <span class="mf">0.5</span><span class="p">))</span> |
|
|
221 | 221 | |
|
|
222 | 222 | <span class="c1">################# end sample data #################</span> |
|
|
223 | 223 | |
|
|
224 | 224 | <span class="n">newFile</span> <span class="o">=</span> <span class="s1">'/tmp/testCedar.hdf5'</span> |
|
|
225 | 225 | |
|
|
226 | 226 | <span class="c1"># create a new Madrigal file </span> |
|
|
227 | 227 | <span class="n">cedarObj</span> <span class="o">=</span> <span class="n">madrigal</span><span class="o">.</span><span class="n">cedar</span><span class="o">.</span><span class="n">MadrigalCedarFile</span><span class="p">(</span><span class="n">newFile</span><span class="p">,</span> <span class="bp">True</span><span class="p">)</span> |
|
|
228 | 228 | |
|
|
229 | 229 | <span class="c1"># create all data records - each record lasts one minute</span> |
|
|
230 | 230 | <span class="n">startTime</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2005</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">19</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> |
|
|
231 | 231 | <span class="n">recTime</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">60</span><span class="p">)</span> |
|
|
232 | 232 | <span class="k">for</span> <span class="n">recno</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">):</span> |
|
|
233 | 233 | <span class="n">endTime</span> <span class="o">=</span> <span class="n">startTime</span> <span class="o">+</span> <span class="n">recTime</span> |
|
|
234 | 234 | <span class="n">dataRec</span> <span class="o">=</span> <span class="n">madrigal</span><span class="o">.</span><span class="n">cedar</span><span class="o">.</span><span class="n">MadrigalDataRecord</span><span class="p">(</span><span class="n">kinst</span><span class="p">,</span> |
|
|
235 | 235 | <span class="n">kindat</span><span class="p">,</span> |
|
|
236 | 236 | <span class="n">startTime</span><span class="o">.</span><span class="n">year</span><span class="p">,</span> |
|
|
237 | 237 | <span class="n">startTime</span><span class="o">.</span><span class="n">month</span><span class="p">,</span> |
|
|
238 | 238 | <span class="n">startTime</span><span class="o">.</span><span class="n">day</span><span class="p">,</span> |
|
|
239 | 239 | <span class="n">startTime</span><span class="o">.</span><span class="n">hour</span><span class="p">,</span> |
|
|
240 | 240 | <span class="n">startTime</span><span class="o">.</span><span class="n">minute</span><span class="p">,</span> |
|
|
241 | 241 | <span class="n">startTime</span><span class="o">.</span><span class="n">second</span><span class="p">,</span> |
|
|
242 | 242 | <span class="n">startTime</span><span class="o">.</span><span class="n">microsecond</span><span class="o">/</span><span class="mi">10000</span><span class="p">,</span> |
|
|
243 | 243 | <span class="n">endTime</span><span class="o">.</span><span class="n">year</span><span class="p">,</span> |
|
|
244 | 244 | <span class="n">endTime</span><span class="o">.</span><span class="n">month</span><span class="p">,</span> |
|
|
245 | 245 | <span class="n">endTime</span><span class="o">.</span><span class="n">day</span><span class="p">,</span> |
|
|
246 | 246 | <span class="n">endTime</span><span class="o">.</span><span class="n">hour</span><span class="p">,</span> |
|
|
247 | 247 | <span class="n">endTime</span><span class="o">.</span><span class="n">minute</span><span class="p">,</span> |
|
|
248 | 248 | <span class="n">endTime</span><span class="o">.</span><span class="n">second</span><span class="p">,</span> |
|
|
249 | 249 | <span class="n">endTime</span><span class="o">.</span><span class="n">microsecond</span><span class="o">/</span><span class="mi">10000</span><span class="p">,</span> |
|
|
250 | 250 | <span class="p">(</span><span class="s1">'systmp'</span><span class="p">,</span> <span class="s1">'tfreq'</span><span class="p">),</span> |
|
|
251 | 251 | <span class="p">(</span><span class="s1">'gdalt'</span><span class="p">,</span> <span class="s1">'gdlat'</span><span class="p">,</span> <span class="s1">'glon'</span><span class="p">,</span> <span class="s1">'tr'</span><span class="p">,</span> <span class="s1">'dtr'</span><span class="p">),</span> |
|
|
252 | 252 | <span class="n">nrow</span><span class="p">,</span> <span class="n">ind2DList</span><span class="o">=</span><span class="p">[</span><span class="s1">'gdalt'</span><span class="p">])</span> |
|
|
253 | 253 | |
|
|
254 | 254 | <span class="c1"># set 1d values</span> |
|
|
255 | 255 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set1D</span><span class="p">(</span><span class="s1">'systmp'</span><span class="p">,</span> <span class="n">SYSTMP</span><span class="p">[</span><span class="n">recno</span><span class="p">])</span> |
|
|
256 | 256 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set1D</span><span class="p">(</span><span class="s1">'tfreq'</span><span class="p">,</span> <span class="n">TFREQ</span><span class="p">[</span><span class="n">recno</span><span class="p">])</span> |
|
|
257 | 257 | |
|
|
258 | 258 | <span class="c1"># set 2d values</span> |
|
|
259 | 259 | <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nrow</span><span class="p">):</span> |
|
|
260 | 260 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'gdalt'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">GDALT</span><span class="p">[</span><span class="n">recno</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
261 | 261 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'gdlat'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">GDLAT</span><span class="p">[</span><span class="n">recno</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
262 | 262 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'glon'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">GLON</span><span class="p">[</span><span class="n">recno</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
263 | 263 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'tr'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">TR</span><span class="p">[</span><span class="n">recno</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
264 | 264 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'dtr'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">DTR</span><span class="p">[</span><span class="n">recno</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
265 | 265 | |
|
|
266 | 266 | <span class="c1"># append new data record</span> |
|
|
267 | 267 | <span class="n">cedarObj</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">dataRec</span><span class="p">)</span> |
|
|
268 | 268 | |
|
|
269 | 269 | <span class="n">startTime</span> <span class="o">+=</span> <span class="n">recTime</span> |
|
|
270 | 270 | |
|
|
271 | 271 | <span class="c1"># write new file</span> |
|
|
272 | 272 | <span class="n">cedarObj</span><span class="o">.</span><span class="n">write</span><span class="p">()</span> |
|
|
273 | 273 | |
|
|
274 | 274 | <span class="c1"># next, use the cedar.CatalogHeaderCreator class to add catalog and header </span> |
|
|
275 | 275 | <span class="n">catHeadObj</span> <span class="o">=</span> <span class="n">madrigal</span><span class="o">.</span><span class="n">cedar</span><span class="o">.</span><span class="n">CatalogHeaderCreator</span><span class="p">(</span><span class="n">newFile</span><span class="p">)</span> |
|
|
276 | 276 | <span class="n">catHeadObj</span><span class="o">.</span><span class="n">createCatalog</span><span class="p">(</span><span class="n">principleInvestigator</span><span class="o">=</span><span class="s2">"John Holt"</span><span class="p">,</span> <span class="n">sciRemarks</span><span class="o">=</span><span class="s2">"Test data only - do not use"</span><span class="p">)</span> |
|
|
277 | 277 | <span class="n">catHeadObj</span><span class="o">.</span><span class="n">createHeader</span><span class="p">(</span><span class="n">analyst</span><span class="o">=</span><span class="s2">"Bill Rideout"</span><span class="p">,</span> <span class="n">comments</span><span class="o">=</span><span class="s2">"Do not use this data"</span><span class="p">)</span> |
|
|
278 | 278 | <span class="n">catHeadObj</span><span class="o">.</span><span class="n">write</span><span class="p">()</span> |
|
|
279 | 279 | </pre></div> |
|
|
280 | 280 | |
|
|
281 | 281 | |
|
|
282 | 282 | <p><a name="create_large" id="create2"></a></p> |
|
|
283 | 283 | <h3>Creating a new large file example</h3> |
|
|
284 | 284 | <div class="highlight"><pre><span></span><span class="sd">"""createSampleLargeFile.py shows an example of creating an entirely new,</span> |
|
|
285 | 285 | <span class="sd">very large Madrigal file using the Python cedar module. </span> |
|
|
286 | 286 | |
|
|
287 | 287 | <span class="sd">In particular, it creates a file with</span> |
|
|
288 | 288 | <span class="sd">a catalog record, a header record, and one hundred thousand data records. Because this</span> |
|
|
289 | 289 | <span class="sd">is a large file, it calls dump for every five hundred records, which writes all the data</span> |
|
|
290 | 290 | <span class="sd">to file in the Table Layout. This limits the memory footprint to just over 1GB.</span> |
|
|
291 | 291 | <span class="sd">This example closes with a call to close(), which triggers the </span> |
|
|
292 | 292 | <span class="sd">creation of Array Layout, which again keeps the memory footprint down by only reading</span> |
|
|
293 | 293 | <span class="sd">in a set number of records at a time.</span> |
|
|
294 | 294 | |
|
|
295 | 295 | <span class="sd">The data records</span> |
|
|
296 | 296 | <span class="sd">contain two 1D parameters (System temperature - SYSTMP and Transmitter</span> |
|
|
297 | 297 | <span class="sd">Frequency TFREQ) and five 2D parameters (GDALT, GDLAT, GLON, and TR, and DTR)</span> |
|
|
298 | 298 | <span class="sd">The independent spatial parameter in this example is just GDALT.</span> |
|
|
299 | 299 | |
|
|
300 | 300 | <span class="sd">$Id$</span> |
|
|
301 | 301 | <span class="sd">"""</span> |
|
|
302 | 302 | |
|
|
303 | 303 | <span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">os.path</span> |
|
|
304 | 304 | <span class="kn">import</span> <span class="nn">types</span> |
|
|
305 | 305 | <span class="kn">import</span> <span class="nn">datetime</span> |
|
|
306 | 306 | |
|
|
307 | 307 | <span class="kn">import</span> <span class="nn">madrigal.metadata</span> |
|
|
308 | 308 | <span class="kn">import</span> <span class="nn">madrigal.cedar</span> |
|
|
309 | 309 | |
|
|
310 | 310 | <span class="c1">################# sample data #################</span> |
|
|
311 | 311 | |
|
|
312 | 312 | <span class="n">kinst</span> <span class="o">=</span> <span class="mi">30</span> <span class="c1"># instrument identifier of Millstone Hill ISR</span> |
|
|
313 | 313 | <span class="n">kindat</span> <span class="o">=</span> <span class="mi">3408</span> <span class="c1"># id of kind of data processing</span> |
|
|
314 | 314 | <span class="n">nrow</span> <span class="o">=</span> <span class="mi">5</span> <span class="c1"># all data records have 5 2D rows</span> |
|
|
315 | 315 | |
|
|
316 | 316 | <span class="n">SYSTMP</span> <span class="o">=</span> <span class="p">(</span><span class="mf">120.0</span><span class="p">,</span> <span class="mf">122.0</span><span class="p">)</span> |
|
|
317 | 317 | <span class="n">TFREQ</span> <span class="o">=</span> <span class="p">(</span><span class="mf">4.4E8</span><span class="p">,</span> <span class="mf">4.4E8</span><span class="p">)</span> |
|
|
318 | 318 | |
|
|
319 | 319 | <span class="n">GDALT</span> <span class="o">=</span> <span class="p">((</span><span class="mf">70.0</span><span class="p">,</span> <span class="mf">100.0</span><span class="p">,</span> <span class="mf">200.0</span><span class="p">,</span> <span class="mf">300.0</span><span class="p">,</span> <span class="mf">400.0</span><span class="p">),</span> |
|
|
320 | 320 | <span class="p">(</span><span class="mf">70.0</span><span class="p">,</span> <span class="mf">100.0</span><span class="p">,</span> <span class="mf">200.0</span><span class="p">,</span> <span class="mf">300.0</span><span class="p">,</span> <span class="mf">400.0</span><span class="p">))</span> |
|
|
321 | 321 | |
|
|
322 | 322 | <span class="n">GDLAT</span> <span class="o">=</span> <span class="p">((</span><span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">),</span> |
|
|
323 | 323 | <span class="p">(</span><span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">,</span> <span class="mf">42.0</span><span class="p">))</span> |
|
|
324 | 324 | |
|
|
325 | 325 | <span class="n">GLON</span> <span class="o">=</span> <span class="p">((</span><span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">),</span> |
|
|
326 | 326 | <span class="p">(</span><span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">,</span> <span class="mf">270.0</span><span class="p">))</span> |
|
|
327 | 327 | |
|
|
328 | 328 | <span class="n">TR</span> <span class="o">=</span> <span class="p">((</span><span class="s1">'missing'</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">2.3</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">),</span> |
|
|
329 | 329 | <span class="p">(</span><span class="s1">'missing'</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.7</span><span class="p">,</span> <span class="mf">2.4</span><span class="p">,</span> <span class="mf">3.1</span><span class="p">))</span> |
|
|
330 | 330 | |
|
|
331 | 331 | |
|
|
332 | 332 | <span class="n">DTR</span> <span class="o">=</span> <span class="p">((</span><span class="s1">'missing'</span><span class="p">,</span> <span class="s1">'assumed'</span><span class="p">,</span> <span class="s1">'assumed'</span><span class="p">,</span> <span class="mf">0.3</span><span class="p">,</span> <span class="mf">0.7</span><span class="p">),</span> |
|
|
333 | 333 | <span class="p">(</span><span class="s1">'missing'</span><span class="p">,</span> <span class="s1">'assumed'</span><span class="p">,</span> <span class="mf">0.7</span><span class="p">,</span> <span class="mf">0.4</span><span class="p">,</span> <span class="mf">0.5</span><span class="p">))</span> |
|
|
334 | 334 | |
|
|
335 | 335 | <span class="c1">################# end sample data #################</span> |
|
|
336 | 336 | |
|
|
337 | 337 | <span class="n">newFile</span> <span class="o">=</span> <span class="s1">'/tmp/testCedar.hdf5'</span> |
|
|
338 | 338 | <span class="k">try</span><span class="p">:</span> |
|
|
339 | 339 | <span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">newFile</span><span class="p">)</span> |
|
|
340 | 340 | <span class="k">except</span><span class="p">:</span> |
|
|
341 | 341 | <span class="k">pass</span> |
|
|
342 | 342 | |
|
|
343 | 343 | <span class="c1"># create a new Madrigal file </span> |
|
|
344 | 344 | <span class="n">cedarObj</span> <span class="o">=</span> <span class="n">madrigal</span><span class="o">.</span><span class="n">cedar</span><span class="o">.</span><span class="n">MadrigalCedarFile</span><span class="p">(</span><span class="n">newFile</span><span class="p">,</span> <span class="bp">True</span><span class="p">)</span> |
|
|
345 | 345 | |
|
|
346 | 346 | <span class="c1"># create all data records - each record lasts one minute</span> |
|
|
347 | 347 | <span class="n">startTime</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2005</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">19</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> |
|
|
348 | 348 | <span class="n">recTime</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">4</span><span class="p">)</span> |
|
|
349 | 349 | <span class="k">for</span> <span class="n">recno</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100000</span><span class="p">):</span> |
|
|
350 | 350 | <span class="n">endTime</span> <span class="o">=</span> <span class="n">startTime</span> <span class="o">+</span> <span class="n">recTime</span> |
|
|
351 | 351 | <span class="n">dataRec</span> <span class="o">=</span> <span class="n">madrigal</span><span class="o">.</span><span class="n">cedar</span><span class="o">.</span><span class="n">MadrigalDataRecord</span><span class="p">(</span><span class="n">kinst</span><span class="p">,</span> |
|
|
352 | 352 | <span class="n">kindat</span><span class="p">,</span> |
|
|
353 | 353 | <span class="n">startTime</span><span class="o">.</span><span class="n">year</span><span class="p">,</span> |
|
|
354 | 354 | <span class="n">startTime</span><span class="o">.</span><span class="n">month</span><span class="p">,</span> |
|
|
355 | 355 | <span class="n">startTime</span><span class="o">.</span><span class="n">day</span><span class="p">,</span> |
|
|
356 | 356 | <span class="n">startTime</span><span class="o">.</span><span class="n">hour</span><span class="p">,</span> |
|
|
357 | 357 | <span class="n">startTime</span><span class="o">.</span><span class="n">minute</span><span class="p">,</span> |
|
|
358 | 358 | <span class="n">startTime</span><span class="o">.</span><span class="n">second</span><span class="p">,</span> |
|
|
359 | 359 | <span class="n">startTime</span><span class="o">.</span><span class="n">microsecond</span><span class="o">/</span><span class="mi">10000</span><span class="p">,</span> |
|
|
360 | 360 | <span class="n">endTime</span><span class="o">.</span><span class="n">year</span><span class="p">,</span> |
|
|
361 | 361 | <span class="n">endTime</span><span class="o">.</span><span class="n">month</span><span class="p">,</span> |
|
|
362 | 362 | <span class="n">endTime</span><span class="o">.</span><span class="n">day</span><span class="p">,</span> |
|
|
363 | 363 | <span class="n">endTime</span><span class="o">.</span><span class="n">hour</span><span class="p">,</span> |
|
|
364 | 364 | <span class="n">endTime</span><span class="o">.</span><span class="n">minute</span><span class="p">,</span> |
|
|
365 | 365 | <span class="n">endTime</span><span class="o">.</span><span class="n">second</span><span class="p">,</span> |
|
|
366 | 366 | <span class="n">endTime</span><span class="o">.</span><span class="n">microsecond</span><span class="o">/</span><span class="mi">10000</span><span class="p">,</span> |
|
|
367 | 367 | <span class="p">(</span><span class="s1">'systmp'</span><span class="p">,</span> <span class="s1">'tfreq'</span><span class="p">),</span> |
|
|
368 | 368 | <span class="p">(</span><span class="s1">'gdalt'</span><span class="p">,</span> <span class="s1">'gdlat'</span><span class="p">,</span> <span class="s1">'glon'</span><span class="p">,</span> <span class="s1">'tr'</span><span class="p">,</span> <span class="s1">'dtr'</span><span class="p">),</span> |
|
|
369 | 369 | <span class="n">nrow</span><span class="p">,</span> <span class="n">ind2DList</span><span class="o">=</span><span class="p">[</span><span class="s1">'gdalt'</span><span class="p">])</span> |
|
|
370 | 370 | |
|
|
371 | 371 | <span class="c1"># set 1d values</span> |
|
|
372 | 372 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set1D</span><span class="p">(</span><span class="s1">'systmp'</span><span class="p">,</span> <span class="n">SYSTMP</span><span class="p">[</span><span class="n">recno</span> <span class="o">%</span> <span class="mi">2</span><span class="p">])</span> |
|
|
373 | 373 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set1D</span><span class="p">(</span><span class="s1">'tfreq'</span><span class="p">,</span> <span class="n">TFREQ</span><span class="p">[</span><span class="n">recno</span> <span class="o">%</span> <span class="mi">2</span><span class="p">])</span> |
|
|
374 | 374 | |
|
|
375 | 375 | <span class="c1"># set 2d values</span> |
|
|
376 | 376 | <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nrow</span><span class="p">):</span> |
|
|
377 | 377 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'gdalt'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">GDALT</span><span class="p">[</span><span class="n">recno</span> <span class="o">%</span> <span class="mi">2</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
378 | 378 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'gdlat'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">GDLAT</span><span class="p">[</span><span class="n">recno</span> <span class="o">%</span> <span class="mi">2</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
379 | 379 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'glon'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">GLON</span><span class="p">[</span><span class="n">recno</span> <span class="o">%</span> <span class="mi">2</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
380 | 380 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'tr'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">TR</span><span class="p">[</span><span class="n">recno</span> <span class="o">%</span> <span class="mi">2</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
381 | 381 | <span class="n">dataRec</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s1">'dtr'</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">DTR</span><span class="p">[</span><span class="n">recno</span> <span class="o">%</span> <span class="mi">2</span><span class="p">][</span><span class="n">n</span><span class="p">])</span> |
|
|
382 | 382 | |
|
|
383 | 383 | <span class="c1"># append new data record</span> |
|
|
384 | 384 | <span class="n">cedarObj</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">dataRec</span><span class="p">)</span> |
|
|
385 | 385 | |
|
|
386 | 386 | <span class="n">startTime</span> <span class="o">+=</span> <span class="n">recTime</span> |
|
|
387 | 387 | |
|
|
388 | 388 | <span class="k">if</span> <span class="n">recno</span> <span class="o">%</span> <span class="mi">500</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">recno</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span> |
|
|
389 | 389 | <span class="n">cedarObj</span><span class="o">.</span><span class="n">dump</span><span class="p">()</span> <span class="c1"># this puts everything on disk and removes it from RAM</span> |
|
|
390 | 390 | <span class="k">if</span> <span class="n">recno</span> <span class="o">%</span> <span class="mi">10000</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> |
|
|
391 | 391 | <span class="c1"># give some feedback</span> |
|
|
392 | 392 | <span class="k">print</span><span class="p">(</span><span class="s1">'At </span><span class="si">%i</span><span class="s1"> records'</span> <span class="o">%</span> <span class="p">(</span><span class="n">recno</span><span class="p">))</span> |
|
|
393 | 393 | |
|
|
394 | 394 | <span class="c1"># finished adding new records</span> |
|
|
395 | 395 | <span class="n">cedarObj</span><span class="o">.</span><span class="n">dump</span><span class="p">()</span> <span class="c1"># write whatever records are still in RAM</span> |
|
|
396 | 396 | <span class="k">print</span><span class="p">(</span><span class="s1">'about to call close, which will also create the array layout'</span><span class="p">)</span> |
|
|
397 | 397 | <span class="n">cedarObj</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="c1"># this triggers creation of the Array Layout, then closes file</span> |
|
|
398 | 398 | |
|
|
399 | 399 | <span class="k">print</span><span class="p">(</span><span class="s1">'Finally, using CatalogHeaderCreator to add some descriptive text.'</span><span class="p">)</span> |
|
|
400 | 400 | <span class="c1"># next, use the cedar.CatalogHeaderCreator class to add catalog and header </span> |
|
|
401 | 401 | <span class="n">catHeadObj</span> <span class="o">=</span> <span class="n">madrigal</span><span class="o">.</span><span class="n">cedar</span><span class="o">.</span><span class="n">CatalogHeaderCreator</span><span class="p">(</span><span class="n">newFile</span><span class="p">)</span> |
|
|
402 | 402 | <span class="n">catHeadObj</span><span class="o">.</span><span class="n">createCatalog</span><span class="p">(</span><span class="n">principleInvestigator</span><span class="o">=</span><span class="s2">"John Holt"</span><span class="p">,</span> <span class="n">sciRemarks</span><span class="o">=</span><span class="s2">"Test data only - do not use"</span><span class="p">)</span> |
|
|
403 | 403 | <span class="n">catHeadObj</span><span class="o">.</span><span class="n">createHeader</span><span class="p">(</span><span class="n">analyst</span><span class="o">=</span><span class="s2">"Bill Rideout"</span><span class="p">,</span> <span class="n">comments</span><span class="o">=</span><span class="s2">"Do not use this data"</span><span class="p">)</span> |
|
|
404 | 404 | <span class="n">catHeadObj</span><span class="o">.</span><span class="n">write</span><span class="p">()</span> |
|
|
405 | 405 | </pre></div> |
|
|
406 | 406 | <p><a name="edit" id="edit"></a> |
|
|
407 | 407 | </p> |
|
|
408 | 408 | <h2>Editing an existing file example</h2> |
|
|
409 | 409 | <div class="highlight"> |
|
|
410 | 410 | <pre><span class="sd">"""editSample.py shows an example of editing existing data in a Madrigal</span> |
|
|
411 | 411 | <span class="sd">file using the Python cedar module. In particular, it edits the sample file</span> |
|
|
412 | 412 | <span class="sd"><span class="s">mlh980120g.002.hdf5</span> to increase all Ti values by a factor of 1.2</span> |
|
|
413 | 413 | <span class="sd">"""</span> |
|
|
414 | 414 | |
|
|
415 | 415 | <span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">os.path</span> |
|
|
416 | 416 | <span class="kn">import</span> <span class="nn">types</span> |
|
|
417 | 417 | |
|
|
418 | 418 | <span class="kn">import</span> <span class="nn">madrigal.metadata</span> |
|
|
419 | 419 | <span class="kn">import</span> <span class="nn">madrigal.cedar</span> |
|
|
420 | 420 | |
|
|
421 | 421 | <span class="n">metaObj</span> <span class="o">=</span> <span class="n">madrigal</span><span class="o">.</span><span class="n">metadata</span><span class="o">.</span><span class="n">MadrigalDB</span><span class="p">()</span> |
|
|
422 | 422 | |
|
|
423 | 423 | <span class="n">orgFile</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">metaObj</span><span class="o">.</span><span class="n">getMadroot</span><span class="p">(),</span> <span class="s">'experiments/1998/mlh/20jan98/mlh980120g.002.hdf5'</span><span class="p">)</span> |
|
|
424 | 424 | <span class="n">newFile</span> <span class="o">=</span> <span class="s">'/tmp/mlh980120g.002.hdf5'</span> |
|
|
425 | 425 | |
|
|
426 | 426 | <span class="c"># read the Madrigal file into memory</span> |
|
|
427 | 427 | <span class="n">cedarObj</span> <span class="o">=</span> <span class="n">madrigal</span><span class="o">.</span><span class="n">cedar</span><span class="o">.</span><span class="n">MadrigalCedarFile</span><span class="p">(</span><span class="n">orgFile</span><span class="p">)</span> |
|
|
428 | 428 | |
|
|
429 | 429 | <span class="c"># loop through each record, increasing all Ti values by a factor of 1.2</span> |
|
|
430 | 430 | <span class="k">for</span> <span class="n">record</span> <span class="ow">in</span> <span class="n">cedarObj</span><span class="p">:</span> |
|
|
431 | 431 | <span class="c"># skip header and catalog records</span> |
|
|
432 | 432 | <span class="k">if</span> <span class="n">record</span><span class="o">.</span><span class="n">getType</span><span class="p">()</span> <span class="o">==</span> <span class="s">'data'</span><span class="p">:</span> |
|
|
433 | 433 | <span class="c"># loop through each 2D roow</span> |
|
|
434 | 434 | <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">record</span><span class="o">.</span><span class="n">getNrow</span><span class="p">()):</span> |
|
|
435 | 435 | <span class="n">presentTi</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="n">get2D</span><span class="p">(</span><span class="s">'Ti'</span><span class="p">,</span> <span class="n">row</span><span class="p">)</span> |
|
|
436 | 436 | <span class="c"># make sure its not a special string value, eg 'missing'</span> |
|
|
437 | 437 | <span class="k">if</span> <span class="nb">type</span><span class="p">(</span><span class="n">presentTi</span><span class="p">)</span> <span class="o">!=</span> <span class="n">types</span><span class="o">.</span><span class="n">StringType</span><span class="p">:</span> |
|
|
438 | 438 | <span class="n">record</span><span class="o">.</span><span class="n">set2D</span><span class="p">(</span><span class="s">'Ti'</span><span class="p">,</span> <span class="n">row</span><span class="p">,</span> <span class="n">presentTi</span><span class="o">*</span><span class="mf">1.2</span><span class="p">)</span> |
|
|
439 | 439 | |
|
|
440 | 440 | <span class="c"># write edited file</span> |
|
|
441 | 441 | <span class="n">cedarObj</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">newFilename</span><span class="o">=</span><span class="n">newFile</span><span class="p">)</span></pre></div> |
|
|
442 | 442 | |
|
|
443 | 443 | <p> </p> |
|
|
444 | 444 | <h2><a name="matlab" id="matlab"></a>Matlab</h2> |
|
|
445 | 445 | <p>To create CEDAR Madrigal Hdf5 files with Matlab, you will need to put the file MadrigalHdf5File.m on your Matlab path. You also need to run the Matlab script on a server with Madrigal installed, since ultimately Matlab will simply write your information to a *.mat file, and then call the python script createMadrigalHdf5FromMatlab.py to convert that *.mat file to the output Hdf5 file. The script createMadrigalHdf5FromMatlab.py is installed in MADROOT/bin as part of the Madrigal installation.</p> |
|
|
446 | 446 | <p>You may however, run MadrigalHdf5File.m on a non-Madrigal server, and save the output file as a *.mat file, rather than an Hdf5 file. You could then convert that *.mat file into a Madrigal file on a server with Madrigal installed, using the convertToMadrigal function also defined in MadrigalHdf5File.m.</p> |
|
|
447 | 447 | <p>There are only seven methods in the main class of this API, along with one helper method needed only if not running on a Madrigal server:</p> |
|
|
448 | 448 | <ol> |
|
|
449 | 449 | <li>MadrigalHdf5File (the init method)</li> |
|
|
450 | 450 | <li>appendRecord (called once for each Madrigal record added)</li> |
|
|
451 | 451 | <li>set1DParm (called for each one D parameter in each Madrigal record)</li> |
|
|
452 | 452 | <li>set2DParm (called for each two D parameter in each Madrigal record, passes in a vector of length equal to number of rows in record)</li> |
|
|
453 | 453 | <li>setCatalog - called once per file; set some metadata beyond the default automatically added</li> |
|
|
454 | 454 | <li>setHeader - called once per file; set the rest of the metadata</li> |
|
|
455 | 455 | <li>write - called when file complete</li> |
|
|
456 | 456 | </ol> |
|
|
457 | 457 | <p>These seven methods are documented below, along with the helper method convertToMadrigal. An <a href="#matlab_example">example Matlab script</a> that creates a CEDAR Madrigal Hdf5 file is also given.</p> |
|
|
458 | 458 | |
|
|
459 | 459 | <div class="highlight"> |
|
|
460 | 460 | <pre><span class="nn">classdef</span> MadrigalHdf5File<br /> <span class="nb">% class MadrigalHdf5File allows the creation of Madrigal Hdf5 files via<br /> % Matlab. The general idea of this class is to simply write out all<br /> % data for the file in a Matlab struct array. Then the python script<br /> % createMadrigalHdf5FromMatlab is called from this scriptto create all <br /> % metadata and alternate array layouts. This keeps the amount of Matlab <br /> % code here at a minimum. See file testMadrigalHdf5File.m for example<br /> % usage.</span> |
|
|
461 | 461 | |
|
|
462 | 462 | <span class="nn">methods<br /> function</span> madFile = MadrigalHdf5File(filename, oneDParms, ...<br /> independent2DParms, twoDParms, arraySplittingParms)<br /> <span class="nb">% Object constructor for MadrigalHdf5File<br /> % Inputs:<br /> % filename - the filename to write to. Must end *.hdf5,<br/> % .hdf, or .mat. If .mat, writes a Matlab file that must<br/> % later be converted to Madrigal using convertToMadrigal<br /> % oneDParms - a cell array of strings representing 1D parms. May be<br /> % empty. Example:<br /> % cellstr(char('azm', 'elm', 'sn', 'beamid'))<br /> % independent2DParms - a cell array of strings representing independent <br /> % 2D parms. May be empty (ie, {}). Examples:<br /> % cellstr(char('range'))<br /> % cellstr(char('gdlat', 'glon'))<br /> % cellstr(char())<br /> % twoDParms - a cell array of strings representing dependent <br /> % 2D parms. May be empty (ie, {}). Examples:<br /> % cellstr(char('ti', 'dti', 'ne', 'dne'))<br /> % cellstr(char())<br /> % arraySplittingParms - a cell array of strings representing <br /> % parameters whose values are used to split arrays. May <br /> % be empty, in which case set to {}. Example:<br /> % cellstr(char('beamid'))<br /> % skipArray - optional argument. If set to true, no array<br /> % layout created. If false or not passed in, array layout<br /> % created if any 2D variables.</span> |
|
|
463 | 463 | |
|
|
464 | 464 | <span class="nn">function</span> madFile = appendRecord(madFile, ut1_unix, ut2_unix, kindat, ...<br /> kinst, numRows)<br /> <span class="nb">% appendRecord adds a new record to MadrigalHdf5File. It<br /> % returns the record number of the present row (first will be<br /> % 0)<br /> % Inputs:<br /> % madFile - the created MadrigalHdf5File object<br /> % ut1_unix, ut2_unix - unix start and end time of record in<br /> % float seconds since 1970-01-01<br /> % kindat - integer kind of data code. See metadata.<br /> % kinst - integer instrument code. See metadata.<br /> % numRows - number of rows of 2D data. If all 1D data, set<br /> % to 1<br /> % Returns:<br /> % the record number of the present row (first will be 0)<br /> % Affects:<br /> % Updates madFile.recordCount, appends to madFile.data the<br /> % number of rows numRows with all data except stdParms set<br /> % to NaN. Use set1D and set2D to populate that record using<br /> % recNum as index.</span> |
|
|
465 | 465 | |
|
|
466 | 466 | <span class="nn">function</span> madFile = set1DParm(madFile, parm, value, lastRec)<br /> <span class="nb">% set1DParm sets the values of 1D parm parm to value value for<br /> % record with lastRecord value lastRec</span> |
|
|
467 | 467 | |
|
|
468 | 468 | <span class="nn">function</span> madFile = set2DParm(madFile, parm, values, lastRec)<br /> <span class="nb">% set2DParm sets the values of 2D parm parm to value values for<br /> % record with lastRecord value lastRec</span> |
|
|
469 | 469 | |
|
|
470 | 470 | <span class="nn">function</span> madFile = setCatalog(madFile, principleInvestigator, expPurpose, expMode, ...<br /> cycleTime, correlativeExp, sciRemarks, instRemarks)<br /> <span class="nb">% setCatalog allows setting extra information in the catalog<br /> % record. This method is optional. Even if this method is not<br /> % called, the catalog record will contain a description of the<br /> % instrument (kinst code and name) and kind of data brief<br /> % description, along with a list of description of the<br /> % parameters in the file, and the first and last times of the<br /> % measurements.<br /> %<br /> % Inputs:<br /> %<br /> % principleInvestigator - Names of responsible Principal Investigator(s) or <br /> % others knowledgeable about the experiment.<br /> % expPurpose - Brief description of the experiment purpose<br /> % expMode - Further elaboration of meaning of MODEXP; e.g. antenna patterns <br /> % and pulse sequences.<br /> % cycleTime - Minutes for one full measurement cycle - must<br /> % be numeric<br /> % correlativeExp - Correlative experiments (experiments with related data)<br /> % sciRemarks - scientific remarks<br /> % instRemarks - instrument remarks<br /> %</span> |
|
|
471 | 471 | |
|
|
472 | 472 | <span class="nn">function</span> madFile = setHeader(madFile, kindatDesc, analyst, comments, history)<br /> <span class="nb">% setHeader allows setting extra information in the header<br /> % record. This method is optional.<br /> %<br /> % Inputs:<br /> %<br /> % kindatDesc - description of how this data was analyzed (the kind of data)<br /> % analyst - name of person who analyzed this data<br /> % comments - additional comments about data (describe any instrument-specific parameters)<br /> % history - a description of the history of the processing of this file<br /> %</span> |
|
|
473 | 473 | |
|
|
474 | 474 | <span class="nn">function</span> write(madFile)<br /> <span class="nb">% write writes out the complete Hdf5 file to madFile.filename</span> |
|
|
475 | 475 | |
|
|
476 | 476 | <span class="nn">function</span> convertToMadrigal(matFile, madrigalFile)<br /> <span class="nb">% convertToMadrigal converts a matlab mat file to Madrigal Hdf5 file<br/> % Inputs:<br/> % matFile - existing Matlab .mat file created earlier<br/> % madrigalFile - madrigal file to create. Must end *.hdf5, .h5,<br/> % .hdf, or .mat.<br/></span> |
|
|
477 | 477 | </pre> |
|
|
478 | 478 | </div> |
|
|
479 | 479 | |
|
|
480 | 480 | |
|
|
481 | 481 | <a name="matlab_example"></a>Example usage of MadrigalHdf5File |
|
|
482 | 482 | <div class="highlight"> |
|
|
483 | 483 | <pre> |
|
|
484 | 484 | <span class="nb">% test/example script to exercise MadrigalHdf5File |
|
|
485 | 485 | % |
|
|
486 | 486 | % $Id: testMadrigalHdf5File.m 4644 2015-01-13 19:30:37Z brideout $</span> |
|
|
487 | 487 | filename = <span class="s">'/Users/brideout/Documents/workspace/mad3_0/madroot/source/madmatlab/example.h5'</span>; |
|
|
488 | 488 | oneDParms = cellstr(char(<span class="s">'azm'</span>, <span class="s">'elm'</span>, <span class="s">'sn'</span>, <span class="s">'beamid'</span>)); |
|
|
489 | 489 | independent2DParms = cellstr(char(<span class="s">'range'</span>)); |
|
|
490 | 490 | twoDParms = cellstr(char(<span class="s">'ti'</span>, <span class="s">'dti'</span>)); |
|
|
491 | 491 | <span class="nb">% Use {} for independent2DParms and twoDParms if all scalar parameters</span> |
|
|
492 | 492 | arraySplittingParms = cellstr(char(<span class="s">'beamid'</span>)); |
|
|
493 | 493 | <span class="nb">% Use arraySplittingParms = {}; for no splitting</span> |
|
|
494 | 494 | |
|
|
495 | 495 | <span class="nb">% some hard-coded fake data</span> |
|
|
496 | 496 | ut1_unix = 1.0E9; |
|
|
497 | 497 | ut2_unix = 1.0E9 + 100; |
|
|
498 | 498 | kindat = 3410; |
|
|
499 | 499 | kinst= 30; |
|
|
500 | 500 | numRows = 5; |
|
|
501 | 501 | azm = 100.0; |
|
|
502 | 502 | elm = 45.0; |
|
|
503 | 503 | sn = 0.5; |
|
|
504 | 504 | range = [100.0, 150.0, 200.0, 250.0, 300.0]; |
|
|
505 | 505 | ti = [1000.0, 1100.0, 1200.0, 1300.0, 1400.0]; |
|
|
506 | 506 | dti = [100.0, 150.0, 200.0, 250.0, 300.0]; |
|
|
507 | 507 | beamids = [1,2,1,2,1,2,1,2,1,2]; |
|
|
508 | 508 | |
|
|
509 | 509 | |
|
|
510 | 510 | madFile = MadrigalHdf5File(filename, oneDParms, ... |
|
|
511 | 511 | independent2DParms, twoDParms, arraySplittingParms); |
|
|
512 | 512 | |
|
|
513 | 513 | <span class="nn">for</span> rec = 1:10 |
|
|
514 | 514 | madFile = madFile.appendRecord(ut1_unix, ut2_unix, kindat, ... |
|
|
515 | 515 | kinst, numRows); |
|
|
516 | 516 | |
|
|
517 | 517 | <span class="nb">% set 1D values</span> |
|
|
518 | 518 | madFile = madFile.set1DParm(<span class="s">'azm'</span>, azm, madFile.lastRecord); |
|
|
519 | 519 | madFile = madFile.set1DParm(<span class="s">'elm'</span>, elm, madFile.lastRecord); |
|
|
520 | 520 | madFile = madFile.set1DParm(<span class="s">'sn'</span>, sn, madFile.lastRecord); |
|
|
521 | 521 | madFile = madFile.set1DParm(<span class="s">'beamid'</span>, beamids(rec), madFile.lastRecord); |
|
|
522 | 522 | |
|
|
523 | 523 | <span class="nb">% set 2D and independent variables</span> |
|
|
524 | 524 | madFile = madFile.set2DParm(<span class="s">'range'</span>, range, madFile.lastRecord); |
|
|
525 | 525 | madFile = madFile.set2DParm(<span class="s">'ti'</span>, ti, madFile.lastRecord); |
|
|
526 | 526 | madFile = madFile.set2DParm(<span class="s">'dti'</span>, dti, madFile.lastRecord); |
|
|
527 | 527 | |
|
|
528 | 528 | <span class="nn">end</span> |
|
|
529 | 529 | |
|
|
530 | 530 | <span class="nb">% add catalog and header info</span> |
|
|
531 | 531 | principleInvestigator = <span class="s">'Bill Rideout'</span>; |
|
|
532 | 532 | expPurpose = <span class="s">'Measure the ionosphere'</span>; |
|
|
533 | 533 | expMode = <span class="s">'Vector measurements'</span>; |
|
|
534 | 534 | cycleTime = 20.0; |
|
|
535 | 535 | correlativeExp = ''; |
|
|
536 | 536 | sciRemarks = <span class="s">'Big solar storm'</span>; |
|
|
537 | 537 | instRemarks = <span class="s">'Working well!!!'</span>; |
|
|
538 | 538 | madFile = madFile.setCatalog(principleInvestigator, expPurpose, expMode, ... |
|
|
539 | 539 | cycleTime, correlativeExp, sciRemarks, instRemarks); |
|
|
540 | 540 | |
|
|
541 | 541 | kindatDesc = <span class="s">'Regular processing'</span>; |
|
|
542 | 542 | analyst = <span class="s">'Phil Erickson'</span>; |
|
|
543 | 543 | comments = <span class="s">'Include unusual parameter description here'</span>; |
|
|
544 | 544 | history = <span class="s">'Reprocessed four times'</span>; |
|
|
545 | 545 | madFile = madFile.setHeader(kindatDesc, analyst, comments, history); |
|
|
546 | 546 | |
|
|
547 | 547 | write(madFile); |
|
|
548 | 548 | </pre> |
|
|
549 | 549 | |
|
|
550 | 550 | <h3>Converting an old Madrigal 2 file to Madrigal 3</h3> |
|
|
551 | 551 | <p>If you have old Madrigal 2 files you want to convert to Madrigal 3, you can use $MADROOT/bin/exportToHdf.py</p> |
|
|
552 | 552 | <pre> |
|
|
553 | 553 | $MADROOT/bin/exportToHdf.py -h |
|
|
554 | 554 | |
|
|
555 | 555 | exportToHdf.py is a script used to convert a Cedar file to hdf5 format. |
|
|
556 | 556 | |
|
|
557 | 557 | Required arguments: |
|
|
558 | 558 | |
|
|
559 | 559 | --cedarFilename - full path of existing Madrigal file. |
|
|
560 | 560 | |
|
|
561 | 561 | --hdf5Filename - full path of hdf5 file to write. |
|
|
562 | 562 | |
|
|
563 | 563 | Optional arguments - set these to add layouts, parameters or filters. Default is to use cachedFiles.ini: |
|
|
564 | 564 | |
|
|
565 | 565 | --independentSpatialParms - a comma separated list of parameters as mnemonics |
|
|
566 | 566 | that represent independent spatial variables. Causes array layout to be added to |
|
|
567 | 567 | output Hdf5 file. If not given, uses $/home/midasop/mad3/cachedFiles.ini |
|
|
568 | 568 | |
|
|
569 | 569 | --arraySplittingParms - a comma separated list of parameters as mnemonics used to split |
|
|
570 | 570 | arrays into subarrays. For example, beamcode would split data with separate beamcodes |
|
|
571 | 571 | into separate arrays. The number of separate arrays will be up to the product of the number of |
|
|
572 | 572 | unique values found for each parameter, with the restriction that combinations with no records will |
|
|
573 | 573 | not create a separate array. . If not given, uses $/home/midasop/mad3/cachedFiles.ini |
|
|
574 | 574 | |
|
|
575 | 575 | --extraParameters - These parameters will be added to the output file if |
|
|
576 | 576 | they are not already in the input file. Comma-delimited. Default is no |
|
|
577 | 577 | extra parameter |
|
|
578 | 578 | |
|
|
579 | 579 | --filter - Filter argument as in isprint command as string (eg, 'ti.500,2000') Only one allowed. |
|
|
580 | 580 | Default is no filtering |
|
|
581 | 581 | |
|
|
582 | 582 | --status - use to set status of file. Default is 1 (default). Use -1 to get from fileTab (error raised |
|
|
583 | 583 | if not available) |
|
|
584 | 584 | |
|
|
585 | 585 | Example: |
|
|
586 | 586 | exportToHdf --cedarFilename=/opt/madrigal/experiments/1998/mlh/20jan98/mil20100112.001 |
|
|
587 | 587 | --hdf5Filename=/home/user/data/mil20100112.hdf5 |
|
|
588 | 588 | --independentSpatialParms=range |
|
|
589 | 589 | --arraySplittingParms=kinst,pl,mdtyp |
|
|
590 | 590 | --extraParameters=ti,te |
|
|
591 | 591 | --filter=ti,500,1000 |
|
|
592 | 592 | </pre> |
|
|
593 | 593 | </div> |
|
|
594 | 594 | <!-- InstanceEndEditable --> |
|
|
595 | 595 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
596 | 596 | <tr> |
|
|
597 |
<td width="5%"><a href="{% url 'docs' 'ad_experiments.html' %}"><img src=" |
|
|
|
598 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
599 |
<td width="5%"><a href="{% url 'docs' 'ad_createExp.html' %}"><img src=" |
|
|
|
597 | <td width="5%"><a href="{% url 'docs' 'ad_experiments.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
598 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
599 | <td width="5%"><a href="{% url 'docs' 'ad_createExp.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
600 | 600 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Creating and editing Madrigal data files <!-- InstanceEndEditable --></td> |
|
|
601 | 601 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
602 | 602 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
603 | 603 | </tr> |
|
|
604 | 604 | </table> |
|
|
605 | 605 | <div class='online-navigation'> |
|
|
606 | 606 | <b class="navlabel">Previous:</b> |
|
|
607 | 607 | <a class="sectref" href="{% url 'docs' 'ad_experiments.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Madrigal data organization <!-- InstanceEndEditable --></A> |
|
|
608 | 608 | <b class="navlabel"> Up:</b> |
|
|
609 | 609 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
610 | 610 | <b class="navlabel"> Next:</b> |
|
|
611 | 611 | <a class="sectref" href="{% url 'docs' 'ad_createExp.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Creating Madrigal experiments <!-- InstanceEndEditable --></A></div> |
|
|
612 | 612 | <hr/> |
|
|
613 | 613 | <p> </p> |
|
|
614 | 614 | </body> |
|
|
615 | 615 | <!-- InstanceEnd --></html> |
| @@ -1,87 +1,87 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>How Madrigal data is organized</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_createFiles.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_metadata.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_metadata.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_createFiles.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_metadata.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_createFiles.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_metadata.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_createFiles.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'ad_metadata.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'ad_createFiles.html' %}"><img src=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'ad_metadata.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'ad_createFiles.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->How Madrigal data is organized <!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'ad_metadata.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Madrigal data model <!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'ad_createFiles.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Creating Madrigal data files <!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">How Madrigal data is organized </h1> |
|
|
40 | 40 | <p>In the Madrigal database, the data are organized by experiment. An experiment consists of data from a single instrument, and, as a rule, is meant to address a particular scientific goal. Most often an experiment will correspond to a particular set of operating modes run for a contiguous interval of time. For example, the data collected by an incoherent scatter radar during a world day would constitute an experiment. More complicated situations exist, and judgment may have to be exercised in determining what constitutes an experiment. For example, an experiment might be interrupted temporarily in order to use a different set of operating modes in support of a satellite overpass. In this case there could be two experiments which overlap in time.</p> |
|
|
41 | 41 | <p>To each experiment there corresponds a directory. These directories are of the form <em>madroot</em>/experiments[0-9]*/<year>/<instrument>/<<em>directory</em>>, where year is the four-digit year. The next level is the experiments[0-9]*, where the directory must begin "experiments" and then may be followed by any number of digits. This is to allow additional hard drives to be mounted to expand the directory space. The next level, instrument, is the 3-letter mnemonic for the instrument (set in the instTab.txt file), and <<em>directory</em>> is an arbitrary directory name. An example might be /opt/madrigal/experiments/1997/son/06jan1997_001, which would contain a 1997 Sondrestrom (son) experiment. The use of the starting date in the experiment directory name is not required. </p> |
|
|
42 | 42 | <p>Prior to Madrigal 2.5, the final directory name was required to be in the form DDmmmYY*, where the date was the start date of the experiment and * is an optional character to distinguish different experiments with the same start date. For example, /opt/madrigal/experiments/1997/son/06jan97 contained Sondrestrom data for an experiment beginning on 6 January, 1997, where year =1997, instrument =son, start_date =06jan97, and there was no optional character because there was only one Sondrestrom experiment starting on that date. This convention has been dropped as a requirement, although it is still commonly used as a convention.</p> |
|
|
43 | 43 | <p>Previously, experiments that did not follow the above naming convention were effectively "hidden". With Madrigal 2.5, the security field in the metadata file expTab.txt can be used to hide (or restrict access to) an experiment. (See the <a href="ad_createExp.html#changeExpStatus">change experiment status</a> tool). </p> |
|
|
44 | 44 | <p>Each experiment directory must contain: </p> |
|
|
45 | 45 | <ul> |
|
|
46 | 46 | <li>The Experiment Table entry (expTab.txt) for this experiment. This file is created automatically if you use the tools described in the <a href="{% url 'docs' 'ad_createExp.html' %}">creating experiments</a> section. </li> |
|
|
47 | 47 | </ul> |
|
|
48 | 48 | Each experiment directory may contain: |
|
|
49 | 49 | <ul> |
|
|
50 | 50 | <li>The metadata file fileTab.txt for this experiment. This table must have one entry for each Madrigal format file in the experiment. This file is created automatically if you use the tools described in the <a href="ad_createExp.html">creating experiments</a> section. </li> |
|
|
51 |
<li>One or more datasets in <a href=" |
|
|
|
51 | <li>One or more datasets in <a href="{% static 'CEDARMadrigalHdf5Format.pdf' %}" target="_blank">CEDAR Madrigal Hdf5 format</a>. </li> | |
|
|
52 | 52 | <li>Subdirectories containing an html file named index.html . Links to these files will show up the experiment page. </li> |
|
|
53 | 53 | <li>Html pages in the main directory, and again links to these files will show up the experiment page. </li> |
|
|
54 | 54 | <li>Plots or other files related to individual records in the dataset. Links to these files appear next to the appropriate record in the summarizeCedarFile.cgi page. These files must be located in the subdirectory "plots/[file name]/records" under the experiment directory. In order for the script to determine which file goes with which record, the files must include a five digit number somewhere in its name. For example, plot00027.gif would appear as a link next to record 27 in summarizeCedarFile.cgi. More than one file can be associated with a given record. </li> |
|
|
55 | 55 | </ul> |
|
|
56 | 56 | <p>An essential role for a Madrigal administrator is to run the script <em>madroot</em>/bin/updateMaster on a regular basis. This script performs a number of functions:</p> |
|
|
57 | 57 | <ul> |
|
|
58 | 58 | <li>It gathers all the separate metadata files in the individual experiment directories into the central directory <em>madroot</em>/metadata. </li> |
|
|
59 | 59 | <li>It gathers metadata files from other Madrigal sites into the central directory <em>madroot</em>/metadata. </li> |
|
|
60 | 60 | <li>It updates geophysical files from the OpenMadrigal site.</li> |
|
|
61 | 61 | </ul> |
|
|
62 | 62 | <p>It is recommended that this script be set as a cron job to run once a day. </p> |
|
|
63 | 63 | <p>For Madrigal sites with limited bandwidth, updateMaster can occasionally be run with the -s flag. This will skip any possible download of the latest geophysical files, and so maybe run faster. However, it is still important to run updateMaster without the -s flag on a regular basis.</p> |
|
|
64 | 64 | <p>A complete description of the various ways to add and modify experiments in a Madrigal database is given <a href="{% url 'docs' 'ad_createExp.html' %}">in the next section.</a> The updateMaster script must be run after each change described in that section for the changes to take effect. </p> |
|
|
65 | 65 | <p>Note that an experiment need not contain any Madrigal files. </p> |
|
|
66 | 66 | <!-- InstanceEndEditable --> |
|
|
67 | 67 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
68 | 68 | <tr> |
|
|
69 |
<td width="5%"><a href="{% url 'docs' 'ad_metadata.html' %}"><img src=" |
|
|
|
70 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
71 |
<td width="5%"><a href="{% url 'docs' 'ad_createFiles.html' %}"><img src=" |
|
|
|
69 | <td width="5%"><a href="{% url 'docs' 'ad_metadata.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
70 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
71 | <td width="5%"><a href="{% url 'docs' 'ad_createFiles.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
72 | 72 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->How Madrigal data is organized<!-- InstanceEndEditable --></td> |
|
|
73 | 73 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
74 | 74 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
75 | 75 | </tr> |
|
|
76 | 76 | </table> |
|
|
77 | 77 | <div class='online-navigation'> |
|
|
78 | 78 | <b class="navlabel">Previous:</b> |
|
|
79 | 79 | <a class="sectref" href="{% url 'docs' 'ad_metadata.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Madrigal data model <!-- InstanceEndEditable --></A> |
|
|
80 | 80 | <b class="navlabel"> Up:</b> |
|
|
81 | 81 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
82 | 82 | <b class="navlabel"> Next:</b> |
|
|
83 | 83 | <a class="sectref" href="{% url 'docs' 'ad_createFiles.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Creating Madrigal data files <!-- InstanceEndEditable --></A></div> |
|
|
84 | 84 | <hr/> |
|
|
85 | 85 | <p> </p> |
|
|
86 | 86 | </body> |
|
|
87 | 87 | <!-- InstanceEnd --></html> |
| @@ -1,181 +1,181 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Installing Madrigal for the first time</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_upgrade.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_appropriate.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_appropriate.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_upgrade.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_appropriate.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_upgrade.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_appropriate.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_upgrade.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'ad_appropriate.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'ad_upgrade.html' %}"><img src=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'ad_appropriate.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'ad_upgrade.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Installing Madrigal for the first time <!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'ad_appropriate.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Is Madrigal appropriate? <!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'ad_upgrade.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Upgrading Madrigal <!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">Installing Madrigal for the first time </h1> |
|
|
40 | 40 | <h4 align="left">Prerequisites</h4> |
|
|
41 | 41 | <p>Madrigal is meant to be installed on any Unix-like OS, including Mac and Linux. Most of the prerequisites for Madrigal are pre-installed in any modern unix distribution. They are: </p> |
|
|
42 | 42 | <ol> |
|
|
43 | 43 | <li>A C and a FORTRAN compiler. The free GNU compilers may be downloaded from the <a href="http://www.gnu.org/software/gcc/">GNU Website </a>.</li> |
|
|
44 | 44 | <li>Autotools (automake, autoconf, aclocal, libtoolize, autoheader)</li> |
|
|
45 | 45 | <li>A web server. See instructions below to run a Django application (which Madrigal is) on apache.</li> |
|
|
46 | 46 | <li>The environmental variable MADROOT should be set to the path to the Madrigal installation directory. Its recommended this be done in your login script (eg, .bash_profile) because you will need it to run administrative scripts later.</li> |
|
|
47 | 47 | <li>With Madrigal 3, the installation uses a version of python 3 you install yourself. The easiest way I have found to install python with scientific tools is to use either apt-get (for ubuntu), or the Anaconda python distribution, or pip. The following python modules need to be installed: |
|
|
48 | 48 | <ol> |
|
|
49 | 49 | <li>django (version 1.11 or greater)</li> |
|
|
50 | 50 | <li>pip and setuptools</li> |
|
|
51 | 51 | <li>numpy</li> |
|
|
52 | 52 | <li>scipy</li> |
|
|
53 | 53 | <li>matplotlib</li> |
|
|
54 | 54 | <li>netCDF4</li> |
|
|
55 | 55 | <li>h5py</li> |
|
|
56 | 56 | <li>sqlparse</li> |
|
|
57 | 57 | <li>filelock</li> |
|
|
58 | 58 | <li>bootstrap3 (can be installed via <em>pip install django-bootstrap3</em>)</li> |
|
|
59 | 59 | <li>aacgmv2 (can be installed via <em>pip install aacgmv2</em>)<br /> |
|
|
60 | 60 | </li> |
|
|
61 | 61 | </ol> |
|
|
62 | 62 | </li> |
|
|
63 | 63 | </ol> |
|
|
64 | 64 | <p>In addition, you need to have Autotools (automake, autoconf, aclocal, libtoolize, autoheader) installed, along with h5repack (may be part of h5utils package).</p> |
|
|
65 | 65 | <pre></pre> |
|
|
66 | 66 | <h4 align="left">Installation instructions </h4> |
|
|
67 | 67 | <p align="left">Madrigal can be installed on any unix server with a web server configured for Django. If you want to link your data in with data on other Madrigal servers, please notify the <a href="mailto:madrigal@haystack.mit.edu">OpenMadrigal administrator</a>. In general you do not need root permission to install madrigal once the prerequisites listed above are installed. </p> |
|
|
68 | 68 | <ol> |
|
|
69 | 69 | <li>Create a directory to be used as your Madrigal root directory on your unix server. This directory will be referred to as MADROOT. </li> |
|
|
70 | 70 | <li>Create the environment variable MADROOT with that directory path. </li> |
|
|
71 | 71 | <li>Download the latest Madrigal distribution file, madrigal*.tar.gz from the <a href="http://cedar.openmadrigal.org/madrigalDownload/">OpenMadrigal distribution page</a> to MADROOT. </li> |
|
|
72 | 72 | <li>gunzip madrigal*.tar.gz </li> |
|
|
73 | 73 | <li>tar -xf madrigal*.tar </li> |
|
|
74 | 74 | <li>Repeat the 3 steps above to get and untar the Sample Experiments file <em>experiments.tar.gz</em>. </li> |
|
|
75 | 75 | <li>Make sure $MADROOT/bin/python links to the version of python with the required modules listed above. For example, if you are using anaconda, you might run <<em>ln -s //anaconda/bin/python $MADROOT/bin/python</em>>.</li> |
|
|
76 | 76 | <li>If desired, copy any data from your old $MADROOT/experiments directory to this new $MADROOT/experiments directory.</li> |
|
|
77 | 77 | <li>In madroot, there will now be a file called madrigal.cfg.template. Copy it to madrigal.cfg, and edit all the configuration parameters, as described in the <a href="#edit">Editing madrigal.cfg</a> section below. </li> |
|
|
78 | 78 | <li>Configure your web server to run the Madrigal Django application. Instruction for apache can be found in the <a href="#django">Configuring Apache for a Madrigal Django Application</a> section below. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver. </li> |
|
|
79 | 79 | <li>Edit the file MADROOT/metadata/siteTab.txt with a unique id to include your site. The format of siteTab.txt is described <a href="ad_metadata.html#siteTab">here</a>. Note that if your site will be running https, the third field will begin https://. If you want to be an official Madrigal site that other Madrigal sites share data with, request the <a href="mailto:madrigal@haystack.mit.edu">OpenMadrigal administrator</a> to assign you this id. If this is just a test site, use any id not already in the siteTab.txt file. Finally, make sure the last column of the line for your site ends <em>,3.0</em> . This tells all sites that you site is now a Madrigal 3 site. For example: <pre>999,mad31,localhost,madrigal,cgi-bin/madrigal,madrigal/servlets,Bill Rideout,MIT Haystack Observatory,Route 40,,Westford,MA,01886,USA,1-781-981-5624,brideout@haystack.mit.edu,3.0</pre></li> |
|
|
80 | 80 | <li>Be sure to cd to MADROOT before running the following step. Then you should be able to complete the installation simply by typing <br /> |
|
|
81 | 81 | <pre><sudo> bash installMadrigal &> install.log &</pre> |
|
|
82 | 82 | |
|
|
83 | 83 | The <em>sudo</em> is required only if installing modules on your default version of python requires sudo. There may be a long pause when running updateMaster near the end of the installation since the instParmTab.txt metadata file is being built for the first time by examining every data file, but future calls to updateMaster will be much faster since only new experiments are examined. Help with any installation errors is available from the <a href="mailto:madrigal@haystack.mit.edu">OpenMadrigal administrator</a>. </li> |
|
|
84 | 84 | <li>If there were no errors, your madrigal installation should be running at the url given by MADSERVERROOT. </li> |
|
|
85 | 85 | <li>If you want to receive notices about updates to Madrigal, sign up for the openmadrigal-admin mailing list under http://cedar.openmadrigal.org/openmadrigal. </li> |
|
|
86 | 86 | <li>Set the script <em>madroot</em>/bin/updateMaster up as a cron job to run once a day. </li> |
|
|
87 | 87 | <li>If you want to add any documentation pages specific to your site to the Madrigal documentation pages, see the <a href="ad_other.html#siteSpecific">other admin tasks</a> section of this manual.</li> |
|
|
88 | 88 | <li>If you want to add your own rules of the road to the Madrigal experiment page, see the <a href="ad_other.html#rules">other admin tasks</a> section of this manual.</li> |
|
|
89 | 89 | </ol> |
|
|
90 | 90 | <p> </p> |
|
|
91 | 91 | <h4><a name="edit" id="edit"></a>Editing the madrigal.cfg file </h4> |
|
|
92 | 92 | The madrigal.cfg file contains all the configuration information specific to your installation. This section discusses how to edit that file for each parameter. The madrigal.cfg.template file contains examples of each parameter. |
|
|
93 | 93 | <ul> |
|
|
94 | 94 | <li>MADROOT - Madrigal root directory (absolute). This must be set as an environmental variable. </li> |
|
|
95 | 95 | <li>MAD<b></b>SERVER - Web server for accessing Madrigal. For https, add https:// to the beginning. If not given, uses http </li> |
|
|
96 | 96 | <li>MAD<b></b>SERVERROOT - document virtual directory relative to MAD<b></b>SERVER. |
|
|
97 | 97 | If the full Madrigal Url is the same as the MAD<b></b>SERVER field |
|
|
98 | 98 | (for example, if the url is <i>http://madrigal.hao.ucar.edu/</i>), then set this field to a period. (Example: |
|
|
99 | 99 | <i>MAD<b></b>SERVERROOT = .</i> ) </li> |
|
|
100 | 100 | <li>SITEID - Site ID - Must be unique and same as in siteTab.txt </li> |
|
|
101 | 101 | <li>HTMLSTYLE - Body style of html pages </li> |
|
|
102 | 102 | <li>INDEXHEAD - Heading in the top-level Madrigal page </li> |
|
|
103 | 103 | <li>CONTACT - Mailto link of contact person(s) for this madrigal installation. Multiple email addresses may be included if separated by commas. </li> |
|
|
104 | 104 | <li>MAILSERVER - Name of mailserver (possibly localhost if running sendmail) </li> |
|
|
105 | 105 | <li>PLOTBUTTONLABEL - By default the button to show additional documentation is labelled Plots/Docs. Change this line for a different label. <br> |
|
|
106 | 106 | </li> |
|
|
107 | 107 | </ul> |
|
|
108 | 108 | |
|
|
109 | 109 | <h4><a name="django" id="django"></a>Configuring Apache for a Madrigal Django Application</h4> |
|
|
110 | 110 | <p>Some configuration of apache is required to run a Django application like Madrigal. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver. We will configure apache to run Madrigal in mod_wsgi daemon mode for improved performance.</p> |
|
|
111 | 111 | <p>The first step is to make sure mod_wsgi is installed as part of your apache webserver. If not, see mod_wsgi installation directions. Once installed, add the line</p> |
|
|
112 | 112 | <pre>LoadModule wsgi_module modules/mod_wsgi.so</pre> |
|
|
113 | 113 | <p>near the top of the httpd.conf file. Next add the follow section, with <em>MADROOT</em>, <em>MADSERVERROOT</em> and <em>YOUR_SERVER</em> replaced with values for your server from the madrigal.cfg file described below. Note that if <em>MADSERVERROOT</em> is zero length (that is, if the full Madrigal Url is the same as the MAD<b></b>SERVER field |
|
|
114 | 114 | - for example, if the url is <i>http://madrigal.hao.ucar.edu/</i>) then /<em>MADSERVERROOT</em> just becomes / unlike in the madrigal.cfg file when it becomes a period. Set <em>NUM_CPU</em> to about one half available on your server.</p> |
|
|
115 | 115 | |
|
|
116 | 116 | <pre>SetEnv PYTHONPATH <em>MADROOT</em>/source/madpy/djangoMad/ |
|
|
117 | 117 | WSGIDaemonProcess <em>YOUR_SERVER</em> python-path=<em>MADROOT</em>/source/madpy/djangoMad processes=<em>NUM_CPU</em> display-name=%{GROUP} lang='en_US.UTF-8' locale='en_US.UTF-8' |
|
|
118 | 118 | WSGIProcessGroup <em>YOUR_SERVER</em> |
|
|
119 | 119 | WSGIScriptAlias /<em>MADSERVERROOT</em> <em>MADROOT</em>/source/madpy/djangoMad/djangoMad/wsgi.py process-group=<em>YOUR_SERVER</em> |
|
|
120 | 120 | Alias /static/ <em>MADROOT</em>/source/madpy/djangoMad/madweb/static/ |
|
|
121 | 121 | WSGIApplicationGroup %{GLOBAL}</pre> |
|
|
122 | 122 | |
|
|
123 | 123 | <pre> <Directory <em>MADROOT</em>/source/madpy/djangoMad/madweb/static></pre> |
|
|
124 | 124 | <blockquote> |
|
|
125 | 125 | <pre>Require all granted</pre> |
|
|
126 | 126 | </blockquote> |
|
|
127 | 127 | <pre> |
|
|
128 | 128 | </Directory> |
|
|
129 | 129 | <Directory <em>MADROOT</em>/source/madpy/djangoMad/djangoMad></pre> |
|
|
130 | 130 | <blockquote> |
|
|
131 | 131 | <pre><Files wsgi.py> </pre> |
|
|
132 | 132 | <blockquote> |
|
|
133 | 133 | <pre>Require all granted</pre> |
|
|
134 | 134 | </blockquote> |
|
|
135 | 135 | <pre></Files></pre> |
|
|
136 | 136 | </blockquote> |
|
|
137 | 137 | <pre></Directory> |
|
|
138 | 138 | </pre> |
|
|
139 | 139 | |
|
|
140 | 140 | <p>You may also need to add to your apache configuration file <em>SetEnv LD_LIBRARY_PATH $MADROOT/lib</em>, possibly along with other library paths.</p> |
|
|
141 | 141 | |
|
|
142 | 142 | <p>Finally, you need to edit the file <em>MADROOT</em>/source/madpy/djangoMad/djangoMad/settings_production_template.py, and save it as <em>MADROOT</em>/source/madpy/djangoMad/djangoMad/settings_production.py. The lines to be edited are:</p> |
|
|
143 | 143 | <pre>SECRET_KEY = '*****' |
|
|
144 | 144 | ALLOWED_HOSTS = ['madrigal3.haystack.mit.edu'] |
|
|
145 | 145 | ADMINS = (('Bill Rideout', 'brideout@haystack.mit.edu'),) |
|
|
146 | 146 | EMAIL_HOST = 'hyperion.haystack.mit.edu' |
|
|
147 | 147 | MANAGERS = (('Bill Rideout', 'brideout@haystack.mit.edu'),) |
|
|
148 | 148 | STATIC_URL = '/static/'</pre> |
|
|
149 | 149 | |
|
|
150 | 150 | <p>The STATIC_URL line only needs to be edited if MAD<b></b>SERVERROOT is not empty; that is, if your madrigal is installed in a subdirectory. |
|
|
151 | 151 | For example, if your url is http://madrigal.sri.com/madrigal, you would change that line to be STATIC_URL = '/madrigal/static/'. But if your |
|
|
152 | 152 | url is http://madrigal.sri.com, you would leave that STATIC_URL line as is.</p> |
|
|
153 | 153 | |
|
|
154 | 154 | <p>You can generate a new secret key for your site at http://www.miniwebtool.com/django-secret-key-generator/, which is recommended for security reasons. The other lines are self-explanatory. </p> |
|
|
155 | 155 | <p>When you are done, be sure to restart apache for the changes to take effect. Note that because this Madrigal Django application runs in daemon mode, it will be necessary to restart the web server if you ever update the python code.</p> |
|
|
156 | 156 | |
|
|
157 | 157 | <h3>Throttling Madrigal admin emails</h3> |
|
|
158 | 158 | <p>Sometimes Madrigal users write scripts that cause numerous admin emails, and in addition malicious users can also generate numerous admin emails. |
|
|
159 | 159 | If you would like to throttle the number of Madrigal admin emails you receive, see <a href="{% url 'docs' 'ad_other.html' %}#throttle">Throttling admin emails</a></p> |
|
|
160 | 160 | <!-- InstanceEndEditable --> |
|
|
161 | 161 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
162 | 162 | <tr> |
|
|
163 |
<td width="5%"><a href="{% url 'docs' 'ad_appropriate.html' %}"><img src=" |
|
|
|
164 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
165 |
<td width="5%"><a href="{% url 'docs' 'ad_upgrade.html' %}"><img src=" |
|
|
|
163 | <td width="5%"><a href="{% url 'docs' 'ad_appropriate.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
164 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
165 | <td width="5%"><a href="{% url 'docs' 'ad_upgrade.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
166 | 166 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Installing Madrigal for the first time <!-- InstanceEndEditable --></td> |
|
|
167 | 167 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
168 | 168 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
169 | 169 | </tr> |
|
|
170 | 170 | </table> |
|
|
171 | 171 | <div class='online-navigation'> |
|
|
172 | 172 | <b class="navlabel">Previous:</b> |
|
|
173 | 173 | <a class="sectref" href="{% url 'docs' 'ad_appropriate.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Is Madrigal appropriate? <!-- InstanceEndEditable --></A> |
|
|
174 | 174 | <b class="navlabel"> Up:</b> |
|
|
175 | 175 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
176 | 176 | <b class="navlabel"> Next:</b> |
|
|
177 | 177 | <a class="sectref" href="{% url 'docs' 'ad_upgrade.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Upgrading Madrigal <!-- InstanceEndEditable --></A></div> |
|
|
178 | 178 | <hr/> |
|
|
179 | 179 | <p> </p> |
|
|
180 | 180 | </body> |
|
|
181 | 181 | <!-- InstanceEnd --></html> |
| @@ -1,154 +1,154 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Using isprint</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="dev_contents.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_logging.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_logging.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="dev_contents.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_logging.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="dev_contents.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_logging.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="dev_contents.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'ad_links.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'dev_contents.html' %}"><img src=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'ad_links.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'dev_contents.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Using isprint <!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'ad_links.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Creating direct links<!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'dev_contents.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Developer's toc<!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">Admin command line tool: isprint</h1> |
|
|
40 | 40 | <p>Madrigal administrators can get a quick look print out of a CEDAR Madrigal using the command line tool isprint. This may be more convenient than using the web interface or using an Hdf5 viewer such as Hdfview.</p> |
|
|
41 | 41 | <p> The application isprint is used to display both measured and derived data from one particular Cedar/Madrigal file. The input file can be any valid Cedar format. It will generate a table of user selected parameters subject to user specified filters. The name isprint original referred to "<strong><u>I</u></strong>ncoherent <strong><u>s</u></strong>catter <strong><u>print</u></strong>" but the application now generically prints any data file in the Cedar format. The engine underlying isprint is the main method for outputting Madrigal data. This application is located in <em>madroot</em>/bin.</p> |
|
|
42 | 42 | <p>The isprint application: </p> |
|
|
43 | 43 | <ul> |
|
|
44 | 44 | <li>Treats measured and derived parameters in the same way. </li> |
|
|
45 | 45 | <li>Accepts any parameter (or two parameters added, subtracted, multiplied, or divided) as a filter, with any number of allowed ranges. </li> |
|
|
46 | 46 | <li>Keeps track of 1D versus 2D data, and only prints one line of data if only 1D parameters are requested. </li> |
|
|
47 | 47 | </ul> |
|
|
48 | 48 | <p>To use isprint, the user needs to specify:</p> |
|
|
49 | 49 | <ol> |
|
|
50 | 50 | <li>The full filename </li> |
|
|
51 | 51 | <li>Output file (if not given, prints ascii to stdout)</li> |
|
|
52 | 52 | <li>The parameters desired to be displayed (if any) </li> |
|
|
53 | 53 | <li>Any filters to limit the amount of data shown (if any) </li> |
|
|
54 | 54 | <li>Any non-default formatting options </li> |
|
|
55 | 55 | </ol> |
|
|
56 | 56 | <blockquote> |
|
|
57 | 57 | <h3>The full filename </h3> |
|
|
58 | 58 | </blockquote> |
|
|
59 | 59 | <p>file= path to file (this argument is required) <br /> |
|
|
60 | 60 | <em>Example: file=/opt/madrigal/experiments/1998/mlh/20jan98/mlh980120g.001.hdf5 </em> </p> |
|
|
61 | 61 | <blockquote> |
|
|
62 | 62 | <h3>The output file</h3> |
|
|
63 | 63 | </blockquote> |
|
|
64 | 64 | <p>output=<output file>. If not given, default is text file to stdout. If given and extension is one of<br /> |
|
|
65 | 65 | <.hdf5, .h5, .hdf>, then will save as Hdf5 file. If extension is .nc, will be saved as netCDF4. In<br /> |
|
|
66 | 66 | all other cases, will be saved as ascii text. If hdf5 or netCDF4, arguments (header, summary, badval,<br /> |
|
|
67 | 67 | assumed, and knownbad) ignored if given.</p> |
|
|
68 | 68 | <blockquote> |
|
|
69 | 69 | <h3>The parameters desired to be displayed </h3> |
|
|
70 | 70 | </blockquote>Simply enter the desired parameter mnemonic (case-insensitive). They will be displayed in the order entered. If none given, only the header records will be displayed. <br /> |
|
|
71 | 71 | <em>Example: azm gdalt Range ti Dti </em> |
|
|
72 | 72 | <blockquote> |
|
|
73 | 73 | <h3>Any filters to limit the amount of data shown </h3> |
|
|
74 | 74 | <h5>Time range </h5> |
|
|
75 | 75 | </blockquote> |
|
|
76 | 76 | <p>date1=mm/dd/yyyy - starting date to be examined. If time1 not given, defaults to 0 UT. <br /> |
|
|
77 | 77 | <em>Example: date1=01/20/1998 </em> </p> |
|
|
78 | 78 | <p>time1=hh:mm:ss - starting UT time to be examined. If date1 given, is applied to date1. If not, applies on the first day of the experiment. <br /> |
|
|
79 | 79 | <em>Example: time1=13:30:00 </em> </p> |
|
|
80 | 80 | <p>date2=mm/dd/yyyy - ending date to be examined. If time2 not given, defaults to 0 UT. <br /> |
|
|
81 | 81 | <em>Example: date2=01/21/1998 </em> </p> |
|
|
82 | 82 | <p>time2=hh:mm:ss - ending UT time to be examined - If date2 not given, refers to date1. If date1 and date2 not given, refers to 1st day. <br /> |
|
|
83 | 83 | <em>Example: time2=15:45:00 </em> </p> |
|
|
84 | 84 | <p><em><strong>In the follow arguments ranges are used. If any range value is not given, it may be used to indicate no lower or upper limit (but the comma is always required). Ranges are inclusive of the end points. </strong></em> </p> |
|
|
85 | 85 | <blockquote> |
|
|
86 | 86 | <h5>Geodetic altitude </h5> |
|
|
87 | 87 | </blockquote> |
|
|
88 | 88 | <p>z=lower alt limit1, upper alt limit1 [or lower alt limit2 , upper alt limit2 ...] (km) <br /> |
|
|
89 | 89 | <em>Example 1: z=100,500 </em> This would limit the geodetic altitude to 100 to 500 km. <br /> |
|
|
90 | 90 | <em>Example 2: z=100,200or300,400 </em> This would limit the geodetic altitude to 100 to 200 km or 300 to 400 km. <br /> |
|
|
91 | 91 | <em>Example 3: z=,200or300,400 </em> Since the lower limit of the first range is missing, this would limit the geodetic altitude to anything below 200 km or from 300 to 400 km. </p> |
|
|
92 | 92 | <blockquote> |
|
|
93 | 93 | <h5>Azimuth (from -180 to 180) </h5> |
|
|
94 | 94 | </blockquote> |
|
|
95 | 95 | <p>az=lower az limit1, upper az limit1 [or lower az limit2 , upper az limit2 ...] (from -180 to 180 degrees) <br /> |
|
|
96 | 96 | <em>Example 1: az=100,120 </em> This would limit the azimuth to 100 to 120 degrees. <br /> |
|
|
97 | 97 | <em>Example 2: az=-180,-90or90,180 </em> This would limit the azimuth to between -180 and -90 degrees or to between 90 and 180 degrees. Note this allows a filter to go through 180 degrees. </p> |
|
|
98 | 98 | <blockquote> |
|
|
99 | 99 | <h5>Elevation (from 0 to 90) </h5> |
|
|
100 | 100 | </blockquote> |
|
|
101 | 101 | <p>el=lower el limit1, upper el limit1 [or lower el limit2 , upper el limit2 ...] (from 0 to 90) <br /> |
|
|
102 | 102 | <em>Example 1: el=0,45 </em> This would limit the elevation from 0 to 45 degrees. </p> |
|
|
103 | 103 | <blockquote> |
|
|
104 | 104 | <h5>Pulse length (in seconds) </h5> |
|
|
105 | 105 | </blockquote> |
|
|
106 | 106 | <p>plen=lower pl limit1, upper pl limit1 [or lower pl limit2 , upper pl limit2 ...] (pulse len in sec) <br /> |
|
|
107 | 107 | <em>Example 1: plen=,5e-4 </em> This would limit the pulse length to 5e-4 seconds or less. </p> |
|
|
108 | 108 | <blockquote> |
|
|
109 | 109 | <h5>Free form filters </h5> |
|
|
110 | 110 | </blockquote> |
|
|
111 | 111 | <p>filter=[mnemonic] or [mnemonic1,[+-*/]mnemonic2] , lower limit1 , upper limit1 [or lower limit2 , upper limit2 ...] (any number of filters may be added) <br /> |
|
|
112 | 112 | <em>Example: filter=ti,500,1000or2000,3000 </em> Limits the data to points where Ti is between 500 and 1000 degrees or between 2000 and 3000 degrees. Note that the units are always those of the Cedar standard. <br /> |
|
|
113 | 113 | <em>Example: filter=gdalt,-,sdwht,0, </em> This filter implies "gdalt - sdwht" must be greater than 0.0. Since sdwht is shadow height (the distance above any point on the earth where the sun is first visible), this filter implies that only data in direct sunlight will be displayed. <br /> |
|
|
114 | 114 | <em>Example: filter=ti,/,Dti,100, </em> Limits the data to points where the ratio Ti/dTi is more than 100. </p> |
|
|
115 | 115 | <blockquote> |
|
|
116 | 116 | <h3>Format options </h3> |
|
|
117 | 117 | </blockquote> |
|
|
118 | 118 | <p>header=t or f (defaults to header=t, show headers at the beginning of each record) <br /> |
|
|
119 | 119 | <em>Example: header=f </em> </p> |
|
|
120 | 120 | <p>badval=bad value string (defaults to "missing") <br /> |
|
|
121 | 121 | <em>Example: badval=n/a </em> </p> |
|
|
122 | 122 | <p>assumed=assumed value string (defaults to "assumed") <br /> |
|
|
123 | 123 | <em>Example: assumed=-32766 </em> </p> |
|
|
124 | 124 | <p>knownbad=known bad value string (defaults to "knownbad") <br /> |
|
|
125 | 125 | <em>Example: knownbad=WARNING-BADVALUE </em> </p> |
|
|
126 | 126 | <p>mxchar=maximum characters per line (defaults to no maximum) <br /> |
|
|
127 | 127 | <em>Example: mxchar=80 </em></p> |
|
|
128 | 128 | <p>indSpatialParms=<comma separated list of parms>. Used only if saving to an hdf5 or netCDF4 file. If not given,<br /> |
|
|
129 | 129 | the default is to use the independent parms in the original file. Ignored if only 1D parms requested.</p> |
|
|
130 | 130 | <p>Example: <code>isprint file=/opt/madrigal/experiments/1998/mlh/20jan98/mil980120g.003.hdf5 date1=01/20/1998 time1=15:00:00 date2=01/20/1998 time2=16:00:00 z=200,300or500,600 badval=noData filter=gdalt,-,sdwht,0, filter=ti,500,1000 uth gdalt gdlat glon ti te </code></p> |
|
|
131 | 131 | <p>This example would show data from mil980120g.003.hdf5 between 15 and 16 UT on 01/20/1998 where altitude is either between 200 and 300 km or between 500 and 600 km, and where gdalt-sdwht is greater than 0 (point is in sunlight), and where ti is between 500 and 1000. "noData" would be printed if data was not available. </p> |
|
|
132 | 132 | <p> </p> |
|
|
133 | 133 | <!-- InstanceEndEditable --> |
|
|
134 | 134 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
135 | 135 | <tr> |
|
|
136 |
<td width="5%"><a href="{% url 'docs' 'ad_links.html' %}"><img src=" |
|
|
|
137 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
138 |
<td width="5%"><a href="{% url 'docs' 'dev_contents.html' %}"><img src=" |
|
|
|
136 | <td width="5%"><a href="{% url 'docs' 'ad_links.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
137 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
138 | <td width="5%"><a href="{% url 'docs' 'dev_contents.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
139 | 139 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Command line interface to Madrigal<!-- InstanceEndEditable --></td> |
|
|
140 | 140 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
141 | 141 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
142 | 142 | </tr> |
|
|
143 | 143 | </table> |
|
|
144 | 144 | <div class='online-navigation'> |
|
|
145 | 145 | <b class="navlabel">Previous:</b> |
|
|
146 | 146 | <a class="sectref" href="{% url 'docs' 'ad_links.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Creating direct links<!-- InstanceEndEditable --></A> |
|
|
147 | 147 | <b class="navlabel"> Up:</b> |
|
|
148 | 148 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
149 | 149 | <b class="navlabel"> Next:</b> |
|
|
150 | 150 | <a class="sectref" href="{% url 'docs' 'dev_contents.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Developer's toc <!-- InstanceEndEditable --></A></div> |
|
|
151 | 151 | <hr/> |
|
|
152 | 152 | <p> </p> |
|
|
153 | 153 | </body> |
|
|
154 | 154 | <!-- InstanceEnd --></html> |
| @@ -1,85 +1,85 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>User access logging</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_other.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_other.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_other.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_other.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_isprint.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'ad_logging.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'ad_isprint.html' %}"><img src=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'ad_logging.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'ad_isprint.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Creating direct links to experiments/files <!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'ad_logging.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->User access logging<!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'ad_isprint.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Using isprint<!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | |
|
|
40 | 40 | |
|
|
41 | 41 | <h1 align="center">Creating direct links to experiments/files</h1> |
|
|
42 | 42 | |
|
|
43 | 43 | |
|
|
44 | 44 | <p>There are times you as administrator might want to create urls that directly take a user to a particular experiment, or to a particular |
|
|
45 | 45 | file within an experiment. Below HOME refers to the url to your site's home page. In all cases, if a user has not yet registered, |
|
|
46 | 46 | they will first go to the register page, and then directly to the requested url.</p> |
|
|
47 | 47 | |
|
|
48 | 48 | <h3>To go directly to a particular Madrigal experiment, follow the example below:</h3> |
|
|
49 | 49 | |
|
|
50 | 50 | <pre>HOME/showExperiment?experiment_list=experiments/2017/arg/27jul17</pre> |
|
|
51 | 51 | |
|
|
52 | 52 | <h3>To go directly to a particular Madrigal experiment and activate the show plots/docs button, follow the example below:</h3> |
|
|
53 | 53 | |
|
|
54 | 54 | <pre>HOME/showExperiment?experiment_list=experiments/2017/arg/27jul17&show_plots</pre> |
|
|
55 | 55 | |
|
|
56 | 56 | <h3>To go directly to a particular Madrigal file, follow the example below:</h3> |
|
|
57 | 57 | |
|
|
58 | 58 | <pre>HOME/showExperiment?experiment_list=experiments/2017/arg/27jul17&file_list=one_ion_aro20170727topg.000.hdf5</pre> |
|
|
59 | 59 | |
|
|
60 | 60 | <h3>To go directly to a particular Madrigal file and activate the show plots/docs button, follow the example below:</h3> |
|
|
61 | 61 | |
|
|
62 | 62 | <pre>HOME/showExperiment?experiment_list=experiments/2017/arg/27jul17&file_list=one_ion_aro20170727topg.000.hdf5&show_plots</pre> |
|
|
63 | 63 | |
|
|
64 | 64 | <!-- InstanceEndEditable --> |
|
|
65 | 65 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
66 | 66 | <tr> |
|
|
67 |
<td width="5%"><a href="{% url 'docs' 'ad_logging.html' %}"><img src=" |
|
|
|
68 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
69 |
<td width="5%"><a href="{% url 'docs' 'ad_isprint.html' %}"><img src=" |
|
|
|
67 | <td width="5%"><a href="{% url 'docs' 'ad_logging.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
68 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
69 | <td width="5%"><a href="{% url 'docs' 'ad_isprint.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
70 | 70 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Creating direct links to experiments/files<!-- InstanceEndEditable --></td> |
|
|
71 | 71 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
72 | 72 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
73 | 73 | </tr> |
|
|
74 | 74 | </table> |
|
|
75 | 75 | <div class='online-navigation'> |
|
|
76 | 76 | <b class="navlabel">Previous:</b> |
|
|
77 | 77 | <a class="sectref" href="{% url 'docs' 'ad_logging.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->User access logging<!-- InstanceEndEditable --></A> |
|
|
78 | 78 | <b class="navlabel"> Up:</b> |
|
|
79 | 79 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
80 | 80 | <b class="navlabel"> Next:</b> |
|
|
81 | 81 | <a class="sectref" href="{% url 'docs' 'ad_isprint.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Using isprint <!-- InstanceEndEditable --></A></div> |
|
|
82 | 82 | <hr/> |
|
|
83 | 83 | <p> </p> |
|
|
84 | 84 | </body> |
|
|
85 | 85 | <!-- InstanceEnd --></html> |
| @@ -1,78 +1,78 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>User access logging</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_other.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_other.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_other.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_other.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_isprint.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'ad_other.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'ad_links.html' %}"><img src=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'ad_other.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'ad_links.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->User access logging <!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'ad_other.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Other admin tasks <!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'ad_links.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Creating direct links<!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">User access logging</h1> |
|
|
40 | 40 | <p>Madrigal now logs each time a user accesses data from a Madrigal data file. This log file is stored in <em>madroot</em>/metadata/userdata/access_<YYYY>.log. Each line in the log file has five comma-separated fields:</p> |
|
|
41 | 41 | <ol> |
|
|
42 | 42 | <li>User name</li> |
|
|
43 | 43 | <li>User email</li> |
|
|
44 | 44 | <li>User affiliation</li> |
|
|
45 | 45 | <li>Time stamp <YYYY-MM-DD HH-MM-SS> </li> |
|
|
46 | 46 | <li>Full path the Madrigal file accessed. </li> |
|
|
47 | 47 | </ol> |
|
|
48 | 48 | <p>Sample:</p> |
|
|
49 | 49 | <pre>Bill Rideout,brideout@haystack.mit.edu,MIT Haystack,2006-01-03 10-10-09,/opt/madrigal/experiments/2005/mlh/04sep05/mlh050904a.000.hdf5 |
|
|
50 | 50 | Bill Rideout,brideout@haystack.mit.edu,MIT Haystack,2006-01-03 10-10-10,/opt/madrigal/experiments/2005/mlh/04sep05/mlh050904b.000.hdf5 |
|
|
51 | 51 | Bill Rideout,brideout@haystack.mit.edu,MIT Haystack,2006-01-03 10-10-17,/opt/madrigal/experiments/2005/mlh/04sep05/mlh050904c.000.hdf5 |
|
|
52 | 52 | |
|
|
53 | 53 | </pre> |
|
|
54 | 54 | <p>When a user is accessing Madrigal through the web, they will be asked for this information the first time they use any Data Access <a href="wt_accessData.html"></a> page. No attempt is made to verify the data they enter. This data is then stored as a cookie, and information from that cookie is used for this logging.</p> |
|
|
55 | 55 | <p>All the latest releases of the <a href="rt_introduction.html">remote API</a> also require the caller to supply these additional fields. </p> |
|
|
56 | 56 | <p>The log file will automatically roll over once a year. </p> |
|
|
57 | 57 | <!-- InstanceEndEditable --> |
|
|
58 | 58 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
59 | 59 | <tr> |
|
|
60 |
<td width="5%"><a href="{% url 'docs' 'ad_other.html' %}"><img src=" |
|
|
|
61 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
62 |
<td width="5%"><a href="{% url 'docs' 'ad_links.html' %}"><img src=" |
|
|
|
60 | <td width="5%"><a href="{% url 'docs' 'ad_other.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
61 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
62 | <td width="5%"><a href="{% url 'docs' 'ad_links.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
63 | 63 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->User access logging <!-- InstanceEndEditable --></td> |
|
|
64 | 64 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
65 | 65 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
66 | 66 | </tr> |
|
|
67 | 67 | </table> |
|
|
68 | 68 | <div class='online-navigation'> |
|
|
69 | 69 | <b class="navlabel">Previous:</b> |
|
|
70 | 70 | <a class="sectref" href="{% url 'docs' 'ad_other.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Other admin tasks <!-- InstanceEndEditable --></A> |
|
|
71 | 71 | <b class="navlabel"> Up:</b> |
|
|
72 | 72 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
73 | 73 | <b class="navlabel"> Next:</b> |
|
|
74 | 74 | <a class="sectref" href="{% url 'docs' 'ad_links.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Creating direct links<!-- InstanceEndEditable --></A></div> |
|
|
75 | 75 | <hr/> |
|
|
76 | 76 | <p> </p> |
|
|
77 | 77 | </body> |
|
|
78 | 78 | <!-- InstanceEnd --></html> |
| @@ -1,255 +1,255 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Madrigal data model</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_experiments.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_upgrade.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_upgrade.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_experiments.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_upgrade.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_experiments.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_upgrade.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_experiments.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'ad_upgrade.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'ad_experiments.html' %}"><img src=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'ad_upgrade.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'ad_experiments.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->The Madrigal data model and metadata files <!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'ad_upgrade.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Upgrading Madrigal <!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'ad_experiments.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Madrigal data organization <!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">The Madrigal data model and metadata files </h1> |
|
|
40 | 40 | <p align="left">Understanding the Madrigal data model is an important step in understanding how Madrigal works. There is a correspondence between each level of the data model and the metadata files that are found in the MADROOT/metadata directory. In this section we describe each level of the Madrigal data model and the corresponding metadata file.</p> |
|
|
41 | 41 | <ul> |
|
|
42 | 42 | <li><a href="#siteTab">Madrigal site</a></li> |
|
|
43 | 43 | <li><a href="#instTab">Instrument</a></li> |
|
|
44 | 44 | <li><a href="#instType">Instrument type</a></li> |
|
|
45 | 45 | <li><a href="#expTab">Experiment</a></li> |
|
|
46 | 46 | <li><a href="#fileTab">Experiment files</a></li> |
|
|
47 | 47 | <li><a href="#parcods">Data parameters</a> |
|
|
48 | 48 | <ul> |
|
|
49 | 49 | <li><a href="#explanation">Parameter explanations</a></li> |
|
|
50 | 50 | </ul> |
|
|
51 | 51 | </li> |
|
|
52 | 52 | <li><a href="#madCatTab">Parameter categories</a></li> |
|
|
53 | 53 | <li><a href="#typeTab">Data types (kindat)</a></li> |
|
|
54 | 54 | <li><a href="#instParmTab">Instrument parameters</a></li> |
|
|
55 | 55 | <li><a href="#instKindatTab">Instrument kindats</a></li> |
|
|
56 | 56 | <li><a href="#instDataTab">Instrument data</a></li> |
|
|
57 | 57 | <li><a href="#data">Madrigal files</a></li> |
|
|
58 | 58 | </ul> |
|
|
59 | 59 | <h4 align="left"><a name="siteTab" id="siteTab"></a>Madrigal site - siteTab.txt </h4> |
|
|
60 | 60 | <p align="left">The highest level of Madrigal is a Madrigal site. A Madrigal site is one particular web site controlled by |
|
|
61 | 61 | one particular group, that holds all their own data. While each Madrigal site stores their own data locally, they also share metadata with all the other sites. This makes it possible for users to search for data at all the Madrigal sites at once no matter which site they visit, and simply follow links to the Madrigal site that has the data they are interested in.</p> |
|
|
62 | 62 | <p align="left">Metadata about all sites is stored in MADROOT/metadata/siteTab.txt. When new Madrigal sites are added, this table is updated and all Madrigal sites are notified so they can update this file. If a site is running Madrigal 2.5 or higher, this file will be automatically updated unless the file has been manually modified by the administrator in a way not reported to <a href="mailto:madrigal@haystack.mit.edu">OpenMadrigal administrator</a>. This file contains the following comma-separated fields: </p> |
|
|
63 | 63 | <ul> |
|
|
64 | 64 | <li>Site ID (e.g., 1) </li> |
|
|
65 | 65 | <li>Site Name (e.g., Millstone Hill Observatory)</li> |
|
|
66 | 66 | <li>Madrigal server (e.g., www.haystack.mit.edu). If https, then you need to append https:// (e.g., https://www.haystack.mit.edu)</li> |
|
|
67 | 67 | <li>Madrigal document root relative to server (e.g., madrigal) Leave empty if not needed.</li> |
|
|
68 | 68 | <li>Madrigal CGI directory relative to server (e.g., cgi-bin/madrigal) - This field is no longer used.</li> |
|
|
69 | 69 | <li>Madrigal servlet directory relative to server (e.g., madrigal/servlets) - This field is no longer used. </li> |
|
|
70 | 70 | <li>Contact name (e.g., John M. Holt)</li> |
|
|
71 | 71 | <li>Contact Address 1 (e.g., MIT Haystack Observatory)</li> |
|
|
72 | 72 | <li>Contact Address 2 (e.g., Route 40)</li> |
|
|
73 | 73 | <li>Contact Address 3 (e.g., "" )</li> |
|
|
74 | 74 | <li>Contact City (e.g., Westford)</li> |
|
|
75 | 75 | <li>Contact State/Province (e.g., MA)</li> |
|
|
76 | 76 | <li>Contact Postal Code (e.g., 01886)</li> |
|
|
77 | 77 | <li>Contact Country (e.g., USA)</li> |
|
|
78 | 78 | <li>Contact Telephone (e.g., 1-617-981-5624)</li> |
|
|
79 | 79 | <li>Contact email (e.g., mailto:jmh@haystack.mit.edu) Multiple addresses may be listed if separated by semicolons.</li> |
|
|
80 | 80 | <li>Site version (e.g. 3.0) Version of Madrigal installed as period seperated integers. If not given, default is 2.6.</li> |
|
|
81 | 81 | </ul> |
|
|
82 | 82 | <h4><a name="instTab" id="instTab"></a>Instrument - instTab.txt </h4> |
|
|
83 | 83 | <p>The next layer of the Madrigal data model is the instrument. All data in Madrigal is associated with one and only one instrument. Any given Madrigal site will hold data from one or more instruments. Since Madrigal focuses on ground-based instruments, most instruments have a particular location associated with them. However, some Madrigal data is based on measurements from multiple instruments, and so have no particular location. Some examples are "EISCAT Scientific Association IS Radars" which combine data from the multiple EISCAT radars, and "World-wide GPS Receiver Network", which consists of over a thousand individual GPS receivers distributed around the globe.</p> |
|
|
84 | 84 | <p>Metadata about all instruments is stored in MADROOT/metadata/instTab.txt. When new Madrigal instruments are added, this table is updated and all Madrigal sites are notified so they can update this file. If a site is running Madrigal 2.5 or higher, this file will be automatically updated unless the file has been manually modified by the administrator in a way not reported to <a href="mailto:madrigal@haystack.mit.edu">OpenMadrigal administrator</a>. This instrument code list is usually consistent with the Cedar instrument list. This file contains the following comma-separated fields: </p> |
|
|
85 | 85 | <ul> |
|
|
86 | 86 | <li>Instrument Code (e.g., 30) </li> |
|
|
87 | 87 | <li>Instrument 3-letter Mnemonic - either of second two characters may also be numbers (e.g., mlh or p4p)</li> |
|
|
88 | 88 | <li>Instrument Name (e.g., Millstone Hill Incoherent Scatter Radar)</li> |
|
|
89 | 89 | <li>Latitude (e.g., 42.5)</li> |
|
|
90 | 90 | <li>Longitude (e.g., -71.9)</li> |
|
|
91 | 91 | <li>Altitude in km above sea level (e.g., 0.146)</li> |
|
|
92 | 92 | <li>Contact name (e.g., John M. Holt)</li> |
|
|
93 | 93 | <li>Contact Address 1 (e.g., MIT Haystack Observatory)</li> |
|
|
94 | 94 | <li>Contact Address 2 (e.g., Route 40)</li> |
|
|
95 | 95 | <li>Contact Address 3 (e.g., "" )</li> |
|
|
96 | 96 | <li>Contact City (e.g., Westford)</li> |
|
|
97 | 97 | <li>Contact State/Province (e.g., MA)</li> |
|
|
98 | 98 | <li>Contact Postal Code (e.g., 01886)</li> |
|
|
99 | 99 | <li>Contact Country (e.g., USA)</li> |
|
|
100 | 100 | <li>Contact Telephone (e.g., 1-617-981-5625)</li> |
|
|
101 | 101 | <li>Contact email (e.g., mailto:jmh@haystack.mit.edu)</li> |
|
|
102 | 102 | <li>Instrument category id (e.g., 6)</li> |
|
|
103 | 103 | </ul> |
|
|
104 | 104 | <p> </p> |
|
|
105 | 105 | <h4><a name="instType" id="instType"></a>Instrument type - instType.txt </h4> |
|
|
106 | 106 | <p>The instrument type table lists categories of instruments, to allow the user to search instruments more easily. If a site is running Madrigal 2.5 or higher, this file will be automatically updated unless the file has been manually modified by the administrator in a way not reported to <a href="mailto:madrigal@haystack.mit.edu">OpenMadrigal administrator</a>. This file contains the following comma-separated fields: </p> |
|
|
107 | 107 | <ul> |
|
|
108 | 108 | <li>Instrument category id (e.g., 6) </li> |
|
|
109 | 109 | <li>Instrument category description (e.g., Incoherent Scatter Radar) </li> |
|
|
110 | 110 | </ul> |
|
|
111 | 111 | <p> </p> |
|
|
112 | 112 | <h4><a name="expTab" id="expTab"></a>Experiment - expTab.txt </h4> |
|
|
113 | 113 | <p>All the data from a given instrument is organized into experiments. An experiment consists of data from a single instrument covering a limited period of time, and, as a rule, is meant to address a particular scientific goal. Madrigal makes the assumption that instruments may be run in different modes, and so the data generated may vary from one experiment to another. By organizing one instrument's data into experiments, the purpose and limitations of each experiment can be made clearer. As a Madrigal administrator, you can also provide users with supplemental plots and documentation about that experiment, in addition to the standard Madrigal data files. See the section on <a href="ad_createExp.html">creating and updating Madrigal experiments</a> for more information.</p> |
|
|
114 | 114 | <p>Madrigal has a number of security codes for different types of experiments. Most experiments are public, which allows them to be accessed by everyone. An experiment can also be made private, which means that only users with set ranges of ip addresses can access them. (See<a href="ad_other.html#private"> Set private versus public access</a>.) Private experiments are never shared with other Madrigal sites. There is also a hidden experiment state to completely remove an experiment from access by Madrigal (if, for example, the data is discovered to be corrupt, but might be fixed in the future). </p> |
|
|
115 | 115 | <p>There are also two experiment states for archiving experiments. These are <em>public archive</em> and <em>private archive</em>. These states are meant to support the archiving of Madrigal data at a central Madrigal site, such as the one at NCAR. These archived experiments are duplicates of experiments found at other Madrigal sites. In general these archived experiments are ignored by any part of the user interface that searches all sites, because the user will only want to find the main data source. However, when the user interface is only accessing local data, these archived experiments will appear. A private archived experiment is subject to the same restriction as a regular private experiment. </p> |
|
|
116 | 116 | <p>Metadata about all experiments is stored in MADROOT/metadata/expTab.txt. This file is automatically generated from individual expTab.txt files located in each experiment directory, as will be described in the <a href="ad_experiments.html">next section on experiment organization</a>. This file contains the following comma-separated fields: </p> |
|
|
117 | 117 | <ul> |
|
|
118 | 118 | <li>Experiment ID (auto-generated) </li> |
|
|
119 | 119 | <li>Experiment URL (e.g., http://www.haystack.mit.edu/cgi-bin/madtoc/1997/mlh/03dec97g). Note this url is historical, and no longer works. </li> |
|
|
120 | 120 | <li>Experiment Name (e.g., Wide Latitude Substorm Study)</li> |
|
|
121 | 121 | <li>Site ID (e.g., 1)</li> |
|
|
122 | 122 | <li>Start Date (YYYYMMDD) (e.g., 19971203) </li> |
|
|
123 | 123 | <li>Start Time (HHMMSS)(e.g., 011356)</li> |
|
|
124 | 124 | <li>End Date (e.g., 19971205) (YYYYMMDD)</li> |
|
|
125 | 125 | <li>End Time (e.g., 123525) (HHMMSS)</li> |
|
|
126 | 126 | <li>Instrument Code (e.g., 30)</li> |
|
|
127 | 127 | <li>Security Code (e.g., 0) - 0 for public, 1 for private, -1 for completely ignored, 2 for public archive, 3 for private archive. </li> |
|
|
128 | 128 | <li>Principle Investigator (e.g., Phil Erickson) optional field - used to override the instrument PI </li> |
|
|
129 | 129 | <li>Principle Investigator email (e.g., perickson@haystack.mit.edu) optional - PI must also be given </li> |
|
|
130 | 130 | </ul> |
|
|
131 | 131 | <p>There is also a file called MADROOT/metadata/expTabAll.txt which is also automatically generated. If differs from expTab.txt in that it contains experiment metadata from all Madrigal sites, not just the local one. Any remote experiment with a non-zero security code will be excluded. </p> |
|
|
132 | 132 | <h4><a name="fileTab" id="fileTab"></a>Experiment Files - fileTab.txt </h4> |
|
|
133 | 133 | <p>The data from a given experiment is stored in one or more experiment files. There are a number of reasons |
|
|
134 | 134 | there may be more than one file for a given experiment. The first is that different kinds of data |
|
|
135 | 135 | may be stored in different files. Also, the experimental data may be analyzed in more than one way, leading |
|
|
136 | 136 | to files with different sets of measured parameters. For these two cases, each file should have its own |
|
|
137 | 137 | kindat code (see below). Another reason for multiple files is that older, historical files can be |
|
|
138 | 138 | kept on-line for reference purposes. </p> |
|
|
139 | 139 | |
|
|
140 |
<p>With Madrigal 3, the format of these files is Hdf5 as defined by the <a href=" |
|
|
|
140 | <p>With Madrigal 3, the format of these files is Hdf5 as defined by the <a href="{% static 'CEDARMadrigalHdf5Format.pdf' %}" target="_blank">CEDAR Madrigal Hdf5 file format</a>. Each file may contain only one kindat. The category field is used to distinguish | |
|
|
141 | 141 | files which are of historical interest only, e.g. a file which have been superseded by a file with an |
|
|
142 | 142 | improved electron density calibration. In some cases there may be more than one up-to-date variant of a |
|
|
143 | 143 | file, e.g. when different analysis options have been chosen. In this case one of these files is designated |
|
|
144 | 144 | the default, and the others are designated as variants.</p> |
|
|
145 | 145 | <p>Metadata about all experiment files is stored in MADROOT/metadata/fileTab.txt. This file is automatically generated from individual fileTab.txt files located in each experiment directory, as will be described in the <a href="ad_experiments.html">next section on experiment organization</a>. This file contains the following comma-separated fields: </p> |
|
|
146 | 146 | <ul> |
|
|
147 | 147 | <li>File Name (e.g., mil971203g.002) </li> |
|
|
148 | 148 | <li>Experiment ID (e.g., 10000125) </li> |
|
|
149 | 149 | <li>Data Type (e.g., 3001) </li> |
|
|
150 | 150 | <li>Category (1=default, 2=variant, 3=history, 4=real-time)</li> |
|
|
151 | 151 | <li>Size of File (e.g., 241920) </li> |
|
|
152 | 152 | <li>File contains at least one Catalog Record File (0 for no, 1 for yes)</li> |
|
|
153 | 153 | <li>File contains at least one Header Record File (0 for no, 1 for yes)</li> |
|
|
154 | 154 | <li>Analysis/modify Date (YYYYMMDD) (e.g., 19980101)</li> |
|
|
155 | 155 | <li>Analysis/modify Time (HHMMSS) (e.g., 115131) </li> |
|
|
156 | 156 | <li>File processing status description (preliminary, final, or any other description)</li> |
|
|
157 | 157 | <li>Permission flag: 0 for public, 1 for private</li> |
|
|
158 | 158 | <li>File analyst - who did the analysis of the data and created this file (e.g. John Holt) optional</li> |
|
|
159 | 159 | <li>File analyst email (e.g., jholt@haystack.mit.edu) optional - File analyst must also be given </li> |
|
|
160 | 160 | </ul> |
|
|
161 | 161 | <p>There is also a file called MADROOT/metadata/fileTabAll.txt which is also automatically generated. If differs from fileTab.txt in that it contains experiment file metadata from all Madrigal sites, not just the local one. </p> |
|
|
162 | 162 | <p> </p> |
|
|
163 | 163 | <h4><a name="parcods" id="parcods"></a>Data parameters - parmCodes.txt </h4> |
|
|
164 | 164 | <p>Any given file is made up a series of records holding measured parameters. Note that based on which parameters are in the file, Madrigal will automatically derive a large number of other parameters such as Kp and Magnetic field strength that aren't in the file itself. In the web browser, measured parameters are shown in bold, derived parameters in normal font. </p> |
|
|
165 | 165 | <p> The metadata file parmCodes.txt contains information about what Madrigal or Cedar parameters are supported. It replaces the old format file parmCodes.txt used in Madrigal 2 because that old file limited the size of id codes with its column delimited format. . If a parameter has a parameter code of 0, it cannot be stored in a Cedar file and is meant the be a derived value only. All Madrigal mnemonics must be unique. All non-zero parameter codes must be unique. If a new parameter is desired, it should be done in coordination with Bill Rideout at brideout@haystack.mit.edu.</p> |
|
|
166 | 166 | <p>The file parmCodes.txt is comma delimited. </p> |
|
|
167 | 167 | <ol> |
|
|
168 | 168 | <li>Parameter Code (may be zero) </li> |
|
|
169 | 169 | <li>Description (cannot include a comma)</li> |
|
|
170 | 170 | <li>Units</li> |
|
|
171 | 171 | <li>Mnemonic</li> |
|
|
172 | 172 | <li>Format (Now use C-style formatting)</li> |
|
|
173 | 173 | <li>Width</li> |
|
|
174 | 174 | <li>Category Id (see madCatTab.txt) </li> |
|
|
175 | 175 | <li>Mnemonic has Html description (1 or 0) - used to <a href="#explanation">display extra information</a> about the parameter </li> |
|
|
176 | 176 | <li>Err mnemonic has Html description (1 or 0) - used to <a href="#explanation">display extra information</a> about the error parameter</li> |
|
|
177 | 177 | <li>Is duplicate of code (optional field - used because old cedar format format had duplicate parameters to represent different dynamic range).</li> |
|
|
178 | 178 | </ol> |
|
|
179 | 179 | <blockquote> |
|
|
180 | 180 | <h5><a name="explanation" id="explanation"></a>Parameter explanations</h5> |
|
|
181 | 181 | <p>For parameters that cannot be fully described in the simple string in the parmCodes.txt file, additional explanation about the parameter or its corresponding error parameter can be added to the file madroot/doc/parmDesc.html. Simply create a new named anchor in that file, where the anchor name is the parameter mnemonic in all capitals. Following that, a description of arbitrary length can be given using html. Change one of the last two columns from 0 to 1 for that parameter in parmCodes.txt to let Madrigal know that this explanation exists. In general, the parameter order in parmDesc.html matches that of parmCodes.txt, but that is not a functional requirement. </p> |
|
|
182 | 182 | </blockquote> |
|
|
183 | 183 | <h4><a name="madCatTab" id="madCatTab"></a>Parameter categories - madCatTab.txt</h4> |
|
|
184 | 184 | <p>The Madrigal category metadata file(madCatTab.txt) contains information about what categories Madrigal parameters belong in. The categories are similar to the Cedar categories, but do not follow them exactly. This file does not change. This file contains the following comma-separated fields: </p> |
|
|
185 | 185 | <ul> |
|
|
186 | 186 | <li>Category id (integers, starting at 0) </li> |
|
|
187 | 187 | <li>Category name </li> |
|
|
188 | 188 | <li>Minimum parameter code (integer). Used only for codes not listed in parmCodes.txt. If category doesn't have a range of codes, use -1. </li> |
|
|
189 | 189 | <li>Maximum parameter code (integer). Used only for codes not listed in parmCodes.txt. If category doesn't have a range of codes, use -1.</li> |
|
|
190 | 190 | </ul> |
|
|
191 | 191 | <h4><a name="typeTab" id="typeTab"></a>Data type (kindat) table - typeTab.txt </h4> |
|
|
192 | 192 | <p>The Madrigal data type (also called kind of data or kindat) metadata file(typeTab.txt) contains a list of all data |
|
|
193 | 193 | types in the database. The purpose of the kindat is to uniquely identify the data processing algorithm used to compute |
|
|
194 | 194 | the parameters in the associated Madrigal file. Each individual Madrigal site is free to update this table, as long as |
|
|
195 | 195 | there are no duplicate kindats in your own file. It is best to simply keep the kindat codes in ascending order to aviod |
|
|
196 | 196 | any chance of a duplicate. The kindat description must not contain a comma.</p> |
|
|
197 | 197 | <ul> |
|
|
198 | 198 | <li>Data Type Code (e.g., 3001) </li> |
|
|
199 | 199 | <li>Data Type Brief Description (e.g., Basic Derived Parameters ) </li> |
|
|
200 | 200 | </ul> |
|
|
201 | 201 | <h4><a name="instParmTab" id="instParmTab"></a>Instrument parameter table - instParmTab.txt </h4> |
|
|
202 | 202 | <p>The instrument parameter metadata file (instParmTab.txt) contains information about what measured parameters are found in the data for any given instrument. This data is used to support the global database query web page, and is rebuild by updateMaster. |
|
|
203 | 203 | This file contains the following comma-separated fields:</p> |
|
|
204 | 204 | <ul> |
|
|
205 | 205 | <li>Instrument Code (e.g., 30) </li> |
|
|
206 | 206 | <li>Parameter mnemonic list (e.g., range rangei az1 az2 el1 el2 pl snp3 chisq mhdqc1 systmp systmi power tfreq popl dpopl ti dti tr dtr vo dvo ph+ dph+ pm dpm fa dfa pnrmd pnrmdi vdopp dvdopp) </li> |
|
|
207 | 207 | </ul> |
|
|
208 | 208 | <h4><a name="instKindatTab" id="instKindatTab"></a>Instrument kindat table - instKindatTab.txt </h4> |
|
|
209 | 209 | <p> The instrument kindat metadata file (instKindatTab.txt) contains information about what kindat codes are used with any given instrument. This data is used to support the global database query web page, and is rebuild by updateMaster. This file contains the following comma-separated fields:</p> |
|
|
210 | 210 | <ul> |
|
|
211 | 211 | <li>Instrument Code (e.g., 30) </li> |
|
|
212 | 212 | <li>Kindat code list (e.g., 3408 13204 13210 3001) </li> |
|
|
213 | 213 | </ul> |
|
|
214 | 214 | <h4><a name="instDataTab" id="instKindatTab2"></a>Instrument data table - instData*.txt </h4> |
|
|
215 | 215 | <p> The instrument data metadata files (instData.txt and instDataPriv.txt) contain information about for what years and instruments each Madrigal site has data. Effectively these file are summaries of data in expTabAll.txt, and only exist to speed UI performance. Note that there will be only one line per kinst value. If muliple sites have data for the same instrument, the rules as to which siteID is used are:</p> |
|
|
216 | 216 | <ol> |
|
|
217 | 217 | <li>Test experiments are ignored</li> |
|
|
218 | 218 | <li>If the local site has that instrument, the local site id is always used (thus archive sites will show data as local).</li> |
|
|
219 | 219 | <li>Non-local archived experiments are ignored in favor of the main site.</li> |
|
|
220 | 220 | <li>If the instrument is at multiple non-archived, non-local sites, the site with the largest number of experiments is used.</li> |
|
|
221 | 221 | </ol> |
|
|
222 | 222 | <p>The two files instData.txt and instDataPriv.txt diff only in that private data is included in instDataPriv.txt, whereas public data isd included in both.</p> |
|
|
223 | 223 | <p>This data is used to enhace performance of various web pages, and is rebuild by updateMaster. </p> |
|
|
224 | 224 | <p>This file contains the following comma-separated fields:</p> |
|
|
225 | 225 | <ul> |
|
|
226 | 226 | <li>Site ID (eg, 10)</li> |
|
|
227 | 227 | <li>Instrument Code (e.g., 30) </li> |
|
|
228 | 228 | <li>Year for whihc there is data list (e.g., 1998 1999 2002) </li> |
|
|
229 | 229 | </ul> |
|
|
230 | 230 | <p> </p> |
|
|
231 | 231 | <h4><a name="data" id="data"></a>File data </h4> |
|
|
232 |
<p>The bottom level of the Madrigal data model is of course the data itself. With Madrigal 3.0 and beyond, the actual file format is Hdf5 that follows the rules defined in the <a href=" |
|
|
|
232 | <p>The bottom level of the Madrigal data model is of course the data itself. With Madrigal 3.0 and beyond, the actual file format is Hdf5 that follows the rules defined in the <a href="{% static 'CEDARMadrigalHdf5Format.pdf' %}" target="_blank">CEDAR Madrigal Hdf5 format</a> document. A Madrigal file is made up of a series of records, each with a start and stop time, representing the integration period of measurement (Madrigal tries to enforce the idea that all measurements take a finite time, but sometimes the start time = the stop time).</p> | |
|
|
233 | 233 | <p>Each Cedar parameter can also have an associated error value. This error value can have the special values "missing", "assumed", or "known bad". If an error parameter is "assumed", the implication is that the measured value itself is assumed, and does not represent a measured value. If the error value is "known bad", the measured data is known to have a problem.</p> |
|
|
234 | 234 | <!-- InstanceEndEditable --> |
|
|
235 | 235 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
236 | 236 | <tr> |
|
|
237 |
<td width="5%"><a href="{% url 'docs' 'ad_upgrade.html' %}"><img src=" |
|
|
|
238 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
239 |
<td width="5%"><a href="{% url 'docs' 'ad_experiments.html' %}"><img src=" |
|
|
|
237 | <td width="5%"><a href="{% url 'docs' 'ad_upgrade.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
238 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
239 | <td width="5%"><a href="{% url 'docs' 'ad_experiments.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
240 | 240 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->The Madrigal data model and metadata files <!-- InstanceEndEditable --></td> |
|
|
241 | 241 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
242 | 242 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
243 | 243 | </tr> |
|
|
244 | 244 | </table> |
|
|
245 | 245 | <div class='online-navigation'> |
|
|
246 | 246 | <b class="navlabel">Previous:</b> |
|
|
247 | 247 | <a class="sectref" href="{% url 'docs' 'ad_upgrade.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Upgrading Madrigal <!-- InstanceEndEditable --></A> |
|
|
248 | 248 | <b class="navlabel"> Up:</b> |
|
|
249 | 249 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
250 | 250 | <b class="navlabel"> Next:</b> |
|
|
251 | 251 | <a class="sectref" href="{% url 'docs' 'ad_experiments.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Madrigal data organization <!-- InstanceEndEditable --></A></div> |
|
|
252 | 252 | <hr/> |
|
|
253 | 253 | <p> </p> |
|
|
254 | 254 | </body> |
|
|
255 | 255 | <!-- InstanceEnd --></html> |
| @@ -1,232 +1,232 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Other administrative tasks</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --> |
|
|
10 | 10 | |
|
|
11 | 11 | |
|
|
12 | 12 | <style type="text/css"> |
|
|
13 | 13 | <!-- |
|
|
14 | 14 | .style1 {font-family: "Courier New", Courier, monospace} |
|
|
15 | 15 | --> |
|
|
16 | 16 | </style> |
|
|
17 | 17 | <!-- InstanceEndEditable --> |
|
|
18 |
<link href=" |
|
|
|
18 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
19 | 19 | <style type="text/css"> |
|
|
20 | 20 | html body { |
|
|
21 | 21 | background-color: {{bg_color}}; |
|
|
22 | 22 | } |
|
|
23 | 23 | </style> |
|
|
24 | 24 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_logging.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_createExp.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_createExp.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_logging.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_createExp.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_logging.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_createExp.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_logging.html" --> |
|
|
25 | 25 | </head> |
|
|
26 | 26 | |
|
|
27 | 27 | <body> |
|
|
28 | 28 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
29 | 29 | <tr> |
|
|
30 |
<td width="5%"><a href="{% url 'docs' 'ad_createExp.html' %}"><img src=" |
|
|
|
31 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
32 |
<td width="5%"><a href="{% url 'docs' 'ad_logging.html' %}"><img src=" |
|
|
|
30 | <td width="5%"><a href="{% url 'docs' 'ad_createExp.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
31 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
32 | <td width="5%"><a href="{% url 'docs' 'ad_logging.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
33 | 33 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Other administrative tasks <!-- InstanceEndEditable --></td> |
|
|
34 | 34 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
35 | 35 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
36 | 36 | </tr> |
|
|
37 | 37 | </table> |
|
|
38 | 38 | <div class='online-navigation'> |
|
|
39 | 39 | <b class="navlabel">Previous:</b> |
|
|
40 | 40 | <a class="sectref" href="{% url 'docs' 'ad_createExp.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Creating Madrigal experiments <!-- InstanceEndEditable --></A> |
|
|
41 | 41 | <b class="navlabel"> Up:</b> |
|
|
42 | 42 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
43 | 43 | <b class="navlabel"> Next:</b> |
|
|
44 | 44 | <a class="sectref" href="{% url 'docs' 'ad_logging.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->User access logging <!-- InstanceEndEditable --></A></div> |
|
|
45 | 45 | <hr/> |
|
|
46 | 46 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
47 | 47 | <h1 align="center">Other administrative tasks</h1> |
|
|
48 | 48 | <p>Some other tasks a Madrigal administrator might need to do:</p> |
|
|
49 | 49 | <ul> |
|
|
50 | 50 | <li><a href="#cached_files">Create cached text or netCDF4 versions of CEDAR Madrigal Hdf5 files</a></li> |
|
|
51 | 51 | <li><a href="#private">Set private versus public access for Madrigal data files</a></li> |
|
|
52 | 52 | <li><a href="#links">Modifying the Madrigal homepage</a></li> |
|
|
53 | 53 | <li><a href="#madrigal_cfg">Modify settings in the madrigal.cfg file</a></li> |
|
|
54 | 54 | <li><a href="#siteSpecific">Add documentation specific to your site to Madrigal documentation page</a></li> |
|
|
55 | 55 | <li><a href="#rules">Add site-specific rules-of the-road to the Madrigal experiment page</a></li> |
|
|
56 | 56 | <li><a href="#static">Add static pages to your Madrigal site</a></li> |
|
|
57 | 57 | <li><a href="#copy_exp">Copy entire experiment directories from one Madrigal site to another</a></li> |
|
|
58 | 58 | <li><a href="#updateExpTimes">Reset experiment start and end times according to data in the experiment files</a></li> |
|
|
59 | 59 | <li><a href="#register">Registering interest in an experiment or an intrument</a></li> |
|
|
60 | 60 | <li><a href="#rebuilding">Rebuilding the madrigal python library</a></li> |
|
|
61 | 61 | <li><a href="#experiments">Mounting additional hard drives to expand the experiments disc capacity</a></li> |
|
|
62 | 62 | <li><a href="#throttle">Throttling admin emails if too many are sent</a></li> |
|
|
63 | 63 | <li><a href="#convert">Converting from http to https</a></li> |
|
|
64 | 64 | </ul> |
|
|
65 | 65 | <p>Contact the <a href="mailto:madrigal@haystack.mit.edu">OpenMadrigal administrator</a> <a href="mailto:brideout@haystack.mit.edu"></a> if some other administrative issue arises. </p> |
|
|
66 | 66 | <h4><a name="cached_files" id="cached_files"></a>Create cached text or netCDF4 versions of CEDAR Madrigal Hdf5 files</h4> |
|
|
67 | 67 | <p>When users download ascii or netCDF4 versions of CEDAR Madrigal Hdf5 files as is (that is, just the parameters in the file without filtering), these alternative file formats by default are dynamically created. If this takes too long, the Madrigal admistrator can speed that up by creating cached files in the <<em>experiment directory</em>>/overview directory. The cached text file has <em>'.txt.gz</em>' appended to the basename of the CEDAR Madrigal Hdf5 file, and the netCDF4 file has '<em>.nc</em>' appended to the basename of the CEDAR Madrigal Hdf5 file. These cached files are created with the $MADROOT/bin/createCachedFiles.py script. Here's the usage:</p> |
|
|
68 | 68 | <pre> |
|
|
69 | 69 | createCachedFiles.py [--excludeText --excludeNetCDF4 --inst=<instList> --path=<expPath> |
|
|
70 | 70 | --includeNonDefault --overwrite --includeGeo --numCPU=<numCPU> -h --help] |
|
|
71 | 71 | |
|
|
72 | 72 | By default, both text and netCDF4 files created. Use --excludeText or --excludeNetCDF4 to only create one type. |
|
|
73 | 73 | By default all instruments will be included. Use --inst=<comma delimited kinst list> to only include some instruments. |
|
|
74 | 74 | By default, all experiment directories will be included. Use --path to limit to a particular directory and all subdirectories. |
|
|
75 | 75 | By default only default files will be cached. Use --includeNonDefault to include all files. |
|
|
76 | 76 | Use --overwrite to overwrite all Hdf5 cached files. Default is to skip existing cached files. |
|
|
77 | 77 | Set --includeGeo to also convert geophysical files. Default is to skip them. |
|
|
78 | 78 | numCPU by default is the maximum of (1, numCPUs available - 2). Use --numCPU to override, but still will not be highter than default. |
|
|
79 | 79 | -h or --help - print usage and exit |
|
|
80 | 80 | </pre> |
|
|
81 | 81 | <h4><a name="private" id="private2"></a>Set private versus public access for Madrigal data files</h4> |
|
|
82 | 82 | <p>This new feature of Madrigal allows data files to be made either publicly available, or restricted to a limited set of IP address listed in a file. </p> |
|
|
83 | 83 | <p>To disable this feature and make all your data public, run the following from the $MADROOT directory: </p> |
|
|
84 | 84 | <pre>$MADROOT/bin/setAccess $MADROOT/experiments public </pre> |
|
|
85 | 85 | <p>To enable this feature, you must first create a file called "trustedIPs.txt" under MADROOT. This file should contain the IP addresses of hosts considered to be part of the private group, or partial IPs if whole sub-nets should be included. This file is not created during installation. Anyone whose browser's IP matches this list will be able to view files marked as private. For example, if trustedIPs.txt contains the line 132.197.*, any IP address that begins 132.197 will have private access. </p> |
|
|
86 | 86 | <p>Also, if the fileTab.txt file for that experiment can be overwritten by the web server, anyone whose browser's IP matches this list will be able to to change the access of any file between public and private from the madExperiments listing page. </p> |
|
|
87 | 87 | <p>To change a large number of experiments to be either public or private, run the following script from the $MADROOT directory: </p> |
|
|
88 | 88 | <pre>$MADROOT/bin/setAccess dirPath [public || private] </pre> |
|
|
89 | 89 | <p>where dirPath is the full path name of any directory in $MADROOT/experiments, and the second argument is either public or private. This script will set all experiments in dirPath and below to be public or private. </p> |
|
|
90 | 90 | <p>The access permission for any given file is set in the fileTab.txt file. See <a href="ad_metadata.html">metadata documentation </a> for details. </p> |
|
|
91 | 91 | |
|
|
92 | 92 | <h4><a name="links" id="links"></a>Modifying the Madrigal homepage</h4> |
|
|
93 | 93 | <p>You can easily modify the Madrigal homepage of you site by modifying the template file $MADROOT/source/madpy/djangoMad/madweb/templates/madweb/index.html.</p> |
|
|
94 | 94 | |
|
|
95 | 95 | <h4><a name="madrigal_cfg" id="madrigal_cfg"></a>Modify settings in the madrigal.cfg file</h4> |
|
|
96 | 96 | <p>The following settings in the <em>madroot</em>/madrigal.cfg file can be modified at any time and the changes will take effect immediately:</p> |
|
|
97 | 97 | <ul> |
|
|
98 | 98 | <li>CONTACT - Mailto link of contact person(s) for this madrigal installation. Multiple email addresses may be included if separated by commas. </li> |
|
|
99 | 99 | <li>MAILSERVER - Name of mailserver (possibly localhost if running sendmail) </li> |
|
|
100 | 100 | <li>NOTESMANAGER - If you want someone to be notified whenever a user adds a note to an experiment, uncomment this optional line and add the email address. See below for a discussion of the optional notes feature and how to configure it after installation.</li> |
|
|
101 | 101 | <li>HTMLSTYLE - Body style of html pages </li> |
|
|
102 | 102 | <li>INDEXHEAD - Heading in the top-level Madrigal page </li> |
|
|
103 | 103 | </ul> |
|
|
104 | 104 | |
|
|
105 | 105 | <h4><a name="siteSpecific" id="siteSpecific"></a>Add documentation specific to your site to Madrigal documentation page</h4> |
|
|
106 | 106 | <p>If you want to add any documentation pages specific to your site to the Madrigal documentation pages, simply put them at a url where they can be indexed. Then edit the file MADROOT/source/madpy/djangoMad/madweb/templates/madweb/madContents.html to modify the line</p> |
|
|
107 | 107 | <pre><a href="#"></pre> |
|
|
108 | 108 | <p>to replace # with the url where your site specific documentation is.</p> |
|
|
109 | 109 | |
|
|
110 | 110 | <h4><a name="rules" id="rules"></a>Add site-specific rules-of the-road to the Madrigal experiment page</h4> |
|
|
111 | 111 | <p>By default, users see only the Cedar rules-of-the-road on the data access page when they first enter the Madrigal site. If you want your users to see rules-of the-road specific to your Madrigal site when the reach the Madrigal experiment page, simply create a text file with your rules called <strong>local_rules_of_the_road.txt</strong> in the <em>madroot</em> directory. </p> |
|
|
112 | 112 | |
|
|
113 | 113 | <h4><a name="static" id="static"></a>Add static pages to your Madrigal site</h4> |
|
|
114 | 114 | <p>Soemtimes its helpful to add static pages to your Madrigal site, such as a page describing your instrument, that you can then simply reference from each individual experiment page. To do this, simply copy the static file to |
|
|
115 | 115 | MADROOT/source/madpy/djangoMad/madweb/static/siteSpecific, and use the url "/static/siteSpecific/[filename]" to refer to it.</p> |
|
|
116 | 116 | |
|
|
117 | 117 | |
|
|
118 | 118 | <h4><a name="copy_exp" id="copy_exp"></a>Copy entire experiment directories from one Madrigal site to another</h4> |
|
|
119 | 119 | <p> If you want to copy data from one Madrigal site into yours (such as is done during the standard installation with the test data from Haystack) copy all the desired data from another Madrigal site into the appropriate directory in $MADROOT/experiments. Then follow these steps: </p> |
|
|
120 | 120 | <ol> |
|
|
121 | 121 | <li>cd $MADROOT </li> |
|
|
122 | 122 | <li>./configureExperiments </li> |
|
|
123 | 123 | <li>$MADROOT/bin/updateMaster </li> |
|
|
124 | 124 | </ol> |
|
|
125 | 125 | <p>The script configureExperiments automatically edits the metadata files, changing the site id to your site id. This same procedure also works for <a href="#experiments">moving experiments between experiments[0-9]* directories</a>.</p> |
|
|
126 | 126 | <h4><a name="updateExpTimes" id="updateExpTimes"></a>Reset experiment start and end times according to data in the experiment files</h4> |
|
|
127 | 127 | <p>Sometimes the experiment start and end times in the metadata (the expTab.txt file) are manually edited, and do not match the start and end times of the data found in the files. The script updateExpTimes.py will examine the start and end times of all the files in an experiment, and set the experiment metadata start and end times to be the earliest and latest times found. Usage:</p> |
|
|
128 | 128 | <pre>$MADROOT/bin/updateExpTimes.py [experiment_directory]</pre> |
|
|
129 | 129 | <p>If optional argument <em>experiment_directory</em> given, then it will only update all experiments found in that directory. Default is to update the entire database. Only default or real time files are used to determine the times. Example:</p> |
|
|
130 | 130 | <pre>$MADROOT/bin/updateExpTimes.py $MADROOT/experiments/1998</pre> |
|
|
131 | 131 | <p>will update all experiment times for the year 1998. </p> |
|
|
132 | 132 | <p> </p> |
|
|
133 | 133 | <h4><a name="register" id="register"></a>Registering interest in an experiment or an instrument</h4> |
|
|
134 | 134 | <p>With Madrigal 2.6, users can register interest in an existing experiment or in an instrument. When a data file is updated |
|
|
135 | 135 | in that experiment using the standard admin scripts, by default all users registered for that experiment or instrument |
|
|
136 | 136 | are sent an email. To ensure that is possible, you simply need to make sure the files |
|
|
137 | 137 | <em>$MADROOT/metadata/userdata/regExp.txt</em> and <em>$MADROOT/metadata/userdata/regExp.txt</em> are writable by the web server. If you get a request from a user to unregister them from an experiment or an instrument, simply delete that line with their email from that file. With Madrigal 3.0, you can also use the utility<em> $MADROOT/bin/registrationAdmin.py</em>. Usage is:</p> |
|
|
138 | 138 | <pre>registrationAdmin.py [--unregister | --register] |
|
|
139 | 139 | [--inst=<kinst> | --exp=<expDir>] userEmail</pre> |
|
|
140 | 140 | <pre>Either unregister or register must be set. |
|
|
141 | 141 | Either --inst=<instrument code> or --exp=<expDir> must be set. |
|
|
142 | 142 | expDir is in form <experiments/2010/mlh/18jan10> |
|
|
143 | 143 | </pre> |
|
|
144 | 144 | <h4><a name="rebuilding" id="rebuilding"></a>Rebuilding the madrigal python library</h4> |
|
|
145 | 145 | If for any reason you rebuild the madrigal python library by running <em>MADROOT/bin/python setup.py install</em> in the MADROOT/sopurce/madpy directory, you will need to restart the web server after you are done. |
|
|
146 | 146 | <h4><a name="experiments" id="experiments"></a>Mounting additional hard drives to expand the experiments disc capacity</h4> |
|
|
147 | 147 | <p>With Madrigal 2.6, administrators can expand the disc space available for experiments by mounting a directory named <em>experiments[0-9]*</em> in the Madroot directory. For example, you can add a directory called <em>experiments2</em> in the Madroot directory and use it for a mount point for additional space for Madrigal experiments. After you create this additional mount, you will need to create a soft link in your web server document root with a soft link to that new directory. For example, say you do the following:</p> |
|
|
148 | 148 | <ul> |
|
|
149 | 149 | <li>You create a new mount called <em>experiments2</em> in your Madroot directory<em>.</em></li> |
|
|
150 | 150 | <li>Be sure to set the permission of this new directory so that the Madrigal administrator and the web server can read and write in it.</li> |
|
|
151 | 151 | <li>You need to create a soft link from <madroot>/source/madpy/djangoMad/madweb/static to <madroot>/experiments2. For example, <em>cd /opt/madrigal3/source/madpy/djangoMad/madweb/static</em> and then <em>ln -s /opt/madrigal3/experiments2 experiments2</em>.</li> |
|
|
152 | 152 | </ul> |
|
|
153 | 153 | <p>You can now make use of this new directory in two ways:</p> |
|
|
154 | 154 | <ol> |
|
|
155 | 155 | <li>You can move experiments or whole directory trees of experiments between different <em>experiments[0-9]*</em> directrories as described in <a href="#copy_exp">Copy entire experiement directories</a> section (remember to run configureExperiments and updateMaster afterwards). Note, however, the experiments with geophysical data should not be moved (years 1950, 1957, and 1963).</li> |
|
|
156 | 156 | <li>When running the scripts createExpWithFile.py or createRTExp.py, use the <em>--experimentsDirNum</em> argument to choose the new directory to use. By default the base <em>experiments</em> directory will be used. In the example above, the option <em>--experimentsDirNum</em>=2 would cause <em>experiments2</em> to be used.</li> |
|
|
157 | 157 | </ol> |
|
|
158 | 158 | <p> </p> |
|
|
159 | 159 | <h4><a name="throttle" id="throttle"></a>Throttling admin emails if too many sent</h4> |
|
|
160 | 160 | <p>If you receive too many emails, you can use a third party tool to throttle the amount. The tool I have used successfully is django-email-throttler, |
|
|
161 | 161 | which is described at https://github.com/robert-kisteleki/django-email-throttler. To install, use pip and then edit your settings_production.py file:</p> |
|
|
162 | 162 | |
|
|
163 | 163 | <pre>pip install django-email-throttler</pre> |
|
|
164 | 164 | <p>Then add the following lines to the end of your file MADROOT/source/madpy/djangoMad/djangoMad/settings_production.py. |
|
|
165 | 165 | The top lines can be adjusted to set exactly how many emails you get, but these numbers should work fine. When done, you will |
|
|
166 | 166 | need to restart apache for the changes to take affect.</p> |
|
|
167 | 167 | |
|
|
168 | 168 | <pre> |
|
|
169 | 169 | # stuff for email throttling and their defaults |
|
|
170 | 170 | EMAILTHROTTLER_TMPDIR = '/tmp' |
|
|
171 | 171 | EMAILTHROTTLER_PREFIX = 'emailthrottler-' |
|
|
172 | 172 | EMAILTHROTTLER_INTERVAL = 600 |
|
|
173 | 173 | EMAILTHROTTLER_TRUNCATE_SUBJECT = None # 128 is also ok |
|
|
174 | 174 | EMAILTHROTTLER_SUBJECT_THRESHOLD = 3 # 0 for no-check |
|
|
175 | 175 | EMAILTHROTTLER_OVERALL_THRESHOLD = 6 # 0 for no-check |
|
|
176 | 176 | |
|
|
177 | 177 | LOGGING = { |
|
|
178 | 178 | 'version': 1, |
|
|
179 | 179 | 'disable_existing_loggers': True, |
|
|
180 | 180 | 'handlers': { |
|
|
181 | 181 | 'mail_admins': { |
|
|
182 | 182 | 'level': 'ERROR', |
|
|
183 | 183 | 'class': 'django.utils.log.AdminEmailHandler', |
|
|
184 | 184 | 'email_backend': 'django_email_throttler.throttler.ThrottledEmailBackend', |
|
|
185 | 185 | 'include_html': False, |
|
|
186 | 186 | } |
|
|
187 | 187 | }, |
|
|
188 | 188 | 'loggers': { |
|
|
189 | 189 | 'django': { |
|
|
190 | 190 | 'handlers': ['mail_admins'], |
|
|
191 | 191 | 'level': 'ERROR', |
|
|
192 | 192 | 'propagate': True, |
|
|
193 | 193 | }, |
|
|
194 | 194 | } |
|
|
195 | 195 | } |
|
|
196 | 196 | </pre> |
|
|
197 | 197 | |
|
|
198 | 198 | <h4><a name="convert" id="convert"></a>Converting from http to https</h4> |
|
|
199 | 199 | <p>The following are the steps needed to change an existing Madrigal site from http to https</p> |
|
|
200 | 200 | <ol> |
|
|
201 | 201 | <li>In $MADROOT/metadata/siteTab.txt, include https:// in the third field of your site. For example, change millstone.haystack.mit.edu to https://millstone.haystack.mit.edu</li> |
|
|
202 | 202 | <li>Change the MAD<b></b>SERVER line in $MADROOT/madrigal.cfg to include https:// For example, change millstone.haystack.mit.edu to https://millstone.haystack.mit.edu</li> |
|
|
203 | 203 | <li>cd $MADROOT/source/madpy</li> |
|
|
204 | 204 | <li>$MADROOT/bin/python setup.py install</li> |
|
|
205 | 205 | <li>$MADROOT/configureExperiments</li> |
|
|
206 | 206 | <li>$MADROOT/bin/updateMaster</li> |
|
|
207 | 207 | <li>Let Bill Rideout (brideout@mit.edu) know about the change.</li> |
|
|
208 | 208 | </ol> |
|
|
209 | 209 | |
|
|
210 | 210 | |
|
|
211 | 211 | <!-- InstanceEndEditable --> |
|
|
212 | 212 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
213 | 213 | <tr> |
|
|
214 |
<td width="5%"><a href="{% url 'docs' 'ad_createExp.html' %}"><img src=" |
|
|
|
215 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
216 |
<td width="5%"><a href="{% url 'docs' 'ad_logging.html' %}"><img src=" |
|
|
|
214 | <td width="5%"><a href="{% url 'docs' 'ad_createExp.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
215 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
216 | <td width="5%"><a href="{% url 'docs' 'ad_logging.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
217 | 217 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Other administrative tasks <!-- InstanceEndEditable --></td> |
|
|
218 | 218 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
219 | 219 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
220 | 220 | </tr> |
|
|
221 | 221 | </table> |
|
|
222 | 222 | <div class='online-navigation'> |
|
|
223 | 223 | <b class="navlabel">Previous:</b> |
|
|
224 | 224 | <a class="sectref" href="{% url 'docs' 'ad_createExp.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Creating Madrigal experiments <!-- InstanceEndEditable --></A> |
|
|
225 | 225 | <b class="navlabel"> Up:</b> |
|
|
226 | 226 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
227 | 227 | <b class="navlabel"> Next:</b> |
|
|
228 | 228 | <a class="sectref" href="{% url 'docs' 'ad_logging.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->User access logging <!-- InstanceEndEditable --></A></div> |
|
|
229 | 229 | <hr/> |
|
|
230 | 230 | <p> </p> |
|
|
231 | 231 | </body> |
|
|
232 | 232 | <!-- InstanceEnd --></html> |
| @@ -1,165 +1,165 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Upgrading Madrigal</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_metadata.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_install.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_install.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_metadata.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_install.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_metadata.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_install.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_metadata.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'ad_install.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'ad_metadata.html' %}"><img src=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'ad_install.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'ad_metadata.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Upgrading to Madrigal 3<!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'ad_install.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Installing Madrigal <!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'ad_metadata.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Madrigal data model <!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">Upgrading Madrigal to version 3</h1> |
|
|
40 | 40 | <p align="left">The following directions assume you have a version of Madrigal 2 already running, and you want to upgrade it to Madrigal 3. It is recommended you back up your existing Madrigal installation before proceeding. You will also need to upgrade slightly any loading programs you are running to create the new Madrigal 3 Hdf4 CEDAR format. Contact Bill Rideout or go to <a href="{% url 'docs' 'ad_createFiles.html' %}">Create Madrigal files</a> to learn more.</p> |
|
|
41 | 41 | <h4 align="left">Prerequisites</h4> |
|
|
42 | 42 | <p>With Madrigal 3, the installation uses a version of python 3 you install yourself. The easiest way I have found to install python with scientific tools is to use either apt-get (for ubuntu), or the Anaconda python distribution, or pip. The following python modules need to be installed:</p> |
|
|
43 | 43 | <ol> |
|
|
44 | 44 | <li>django (version 1.11 or greater)</li> |
|
|
45 | 45 | <li>pip and setuptools</li> |
|
|
46 | 46 | <li>numpy</li> |
|
|
47 | 47 | <li>scipy</li> |
|
|
48 | 48 | <li>matplotlib</li> |
|
|
49 | 49 | <li>netCDF4</li> |
|
|
50 | 50 | <li>h5py</li> |
|
|
51 | 51 | <li>sqlparse</li> |
|
|
52 | 52 | <li>filelock</li> |
|
|
53 | 53 | <li>bootstrap3 (can be installed via <em>pip install django-bootstrap3</em>)</li> |
|
|
54 | 54 | <li>aacgmv2 (can be installed via <em>pip install aacgmv2</em>)<br /> |
|
|
55 | 55 | </li> |
|
|
56 | 56 | </ol> |
|
|
57 | 57 | <p>In addition, you need to have Autotools (automake, autoconf, aclocal, libtoolize, autoheader) installed, along with h5repack (may be part of h5utils package).</p> |
|
|
58 | 58 | |
|
|
59 | 59 | <pre></pre> |
|
|
60 | 60 | <h4 align="left">Installation instructions</h4> |
|
|
61 | 61 | <ol> |
|
|
62 | 62 | <li>Be sure the environmental variable MADROOT is set to your main Madrigal directory (where madrigal.cfg is locatied). </li> |
|
|
63 | 63 | <li>Make sure $MADROOT/bin/python links to the version of python with the required modules listed above. For example, if you are using anaconda, you might run <<em>ln -s //anaconda/bin/python $MADROOT/bin/python</em>>.</li> |
|
|
64 | 64 | <li>Download the latest Madrigal 3 distribution file, madrigal*.tar.gz from the <a href="http://cedar.openmadrigal.org/madrigalDownload/">OpenMadrigal distribution page</a> to MADROOT. </li> |
|
|
65 | 65 | <li>tar -xzf madrigal*.tar.gz </li> |
|
|
66 | 66 | <li>Configure your web server to run the Madrigal Django application. Instruction for apache can be found in the <a href="#django">Configuring Apache for a Madrigal Django Application</a> section below. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver. </li> |
|
|
67 | 67 | <li>Edit your file $MADROOT/metadata/siteTab.txt. The format of siteTab.txt is described <a href="ad_metadata.html#siteTab">here</a>. Note that if your site will be running https, the third field will begin https://. Make sure the last column of that line for your site ends <em>,3.0</em> . This tells all sites that you site is now a Madrigal 3 site. For example: <pre>999,mad31,localhost,madrigal,cgi-bin/madrigal,madrigal/servlets,Bill Rideout,MIT Haystack Observatory,Route 40,,Westford,MA,01886,USA,1-781-981-5624,brideout@haystack.mit.edu,3.0</pre></li> |
|
|
68 | 68 | <li>Be sure to cd to MADROOT before running the following step. Then you should be able to complete the installation simply by typing <br /> |
|
|
69 | 69 | <pre>bash installMadrigal &> install.log &</pre> |
|
|
70 | 70 | Help with any installation errors is available from the <a href="mailto:madrigal@haystack.mit.edu">OpenMadrigal administrator</a>. </li> |
|
|
71 | 71 | <li>If there were no errors, your madrigal installation should be running at the url given by MADSERVERROOT. </li> |
|
|
72 | 72 | <li>Set the script <em>madroot</em>/bin/updateMaster up as a cron job to run once a day. </li> |
|
|
73 | 73 | <li>If you want to add any documentation pages specific to your site to the Madrigal documentation pages, see the <a href="ad_other.html#siteSpecific">other admin tasks</a> section of this manual.</li> |
|
|
74 | 74 | <li>If you want to add your own rules of the road to the Madrigal experiment page, see the <a href="ad_other.html#rules">other admin tasks</a> section of this manual.</li> |
|
|
75 | 75 | </ol> |
|
|
76 | 76 | <p> </p> |
|
|
77 | 77 | <h4><a name="edit" id="edit"></a>Editing the madrigal.cfg file </h4> |
|
|
78 | 78 | The madrigal.cfg file contains all the configuration information specific to your installation. This section discusses how to edit that file for each parameter. The madrigal.cfg.template file contains examples of each parameter. For an upgrade, this file does not generally need to be edited. |
|
|
79 | 79 | <ul> |
|
|
80 | 80 | <li>MADROOT - Madrigal root directory (absolute). This must be set as an environmental variable. </li> |
|
|
81 | 81 | <li>MAD<b></b>SERVER - Web server for accessing Madrigal. For https, add https:// to the beginning. If not given, uses http </li> |
|
|
82 | 82 | <li>MAD<b></b>SERVERROOT - document virtual directory relative to MAD<b></b>SERVER. |
|
|
83 | 83 | If the full Madrigal Url is the same as the MAD<b></b>SERVER field |
|
|
84 | 84 | (for example, if the url is <i>http://madrigal.hao.ucar.edu/</i>), then set this field to a period. (Example: <i>MAD<b></b>SERVERROOT = .</i> ) </li> |
|
|
85 | 85 | <li>SITEID - Site ID - Must be unique and same as in siteTab.txt </li> |
|
|
86 | 86 | <li>HTMLSTYLE - Body style of html pages </li> |
|
|
87 | 87 | <li>INDEXHEAD - Heading in the top-level Madrigal page </li> |
|
|
88 | 88 | <li>CONTACT - Mailto link of contact person(s) for this madrigal installation. Multiple email addresses may be included if separated by commas. </li> |
|
|
89 | 89 | <li>MAILSERVER - Name of mailserver (possibly localhost if running sendmail) </li> |
|
|
90 | 90 | <li>PLOTBUTTONLABEL - By default the button to show additional documentation is labelled Plots/Docs. Change this line for a different label. <br> |
|
|
91 | 91 | </li> |
|
|
92 | 92 | </ul> |
|
|
93 | 93 | <h4><a name="django" id="django"></a>Configuring Apache for a Madrigal Django Application</h4> |
|
|
94 | 94 | <p>Some configuration of apache is required to run a Django application like Madrigal. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver. We will configure apache to run Madrigal in mod_wsgi daemon mode for improved performance.</p> |
|
|
95 | 95 | <p>The first step is to make sure mod_wsgi is installed as part of your apache webserver. If not, see mod_wsgi installation directions. Once installed, add the line</p> |
|
|
96 | 96 | <pre>LoadModule wsgi_module modules/mod_wsgi.so</pre> |
|
|
97 | 97 | <p>near the top of the httpd.conf file. Next add the follow section, with <em>MADROOT</em>, <em>MADSERVERROOT</em> and <em>YOUR_SERVER</em> replaced with values for your server from the madrigal.cfg file described below. Note that if <em>MADSERVERROOT</em> is zero length (that is, if the full Madrigal Url is the same as the MAD<b></b>SERVER field |
|
|
98 | 98 | - for example, if the url is <i>http://madrigal.hao.ucar.edu/</i>) then /<em>MADSERVERROOT</em> just becomes / unlike in the madrigal.cfg file when it becomes a period. Set <em>NUM_CPU</em> to about one half available on your server.</p> |
|
|
99 | 99 | |
|
|
100 | 100 | <pre>SetEnv PYTHONPATH <em>MADROOT</em>/source/madpy/djangoMad/ |
|
|
101 | 101 | WSGIDaemonProcess <em>YOUR_SERVER</em> python-path=<em>MADROOT</em>/source/madpy/djangoMad processes=<em>NUM_CPU</em> display-name=%{GROUP} lang='en_US.UTF-8' locale='en_US.UTF-8' |
|
|
102 | 102 | WSGIProcessGroup <em>YOUR_SERVER</em> |
|
|
103 | 103 | WSGIScriptAlias /<em>MADSERVERROOT</em> <em>MADROOT</em>/source/madpy/djangoMad/djangoMad/wsgi.py process-group=<em>YOUR_SERVER</em> |
|
|
104 | 104 | Alias /static/ <em>MADROOT</em>/source/madpy/djangoMad/madweb/static/ |
|
|
105 | 105 | WSGIApplicationGroup %{GLOBAL}</pre> |
|
|
106 | 106 | |
|
|
107 | 107 | <pre> <Directory <em>MADROOT</em>/source/madpy/djangoMad/madweb/static></pre> |
|
|
108 | 108 | <blockquote> |
|
|
109 | 109 | <pre>Require all granted</pre> |
|
|
110 | 110 | </blockquote> |
|
|
111 | 111 | <pre> |
|
|
112 | 112 | </Directory> |
|
|
113 | 113 | <Directory <em>MADROOT</em>/source/madpy/djangoMad/djangoMad></pre> |
|
|
114 | 114 | <blockquote> |
|
|
115 | 115 | <pre><Files wsgi.py> </pre> |
|
|
116 | 116 | <blockquote> |
|
|
117 | 117 | <pre>Require all granted</pre> |
|
|
118 | 118 | </blockquote> |
|
|
119 | 119 | <pre></Files></pre> |
|
|
120 | 120 | </blockquote> |
|
|
121 | 121 | <pre></Directory></pre> |
|
|
122 | 122 | |
|
|
123 | 123 | <p>You may also need to add to your apache configuration file <em>SetEnv LD_LIBRARY_PATH $MADROOT/lib</em>, possibly along with other library paths.</p> |
|
|
124 | 124 | |
|
|
125 | 125 | <p>Finally, you need to edit the file <em>MADROOT</em>/source/madpy/djangoMad/djangoMad/settings_production_template.py, and save it as <em>MADROOT</em>/source/madpy/djangoMad/djangoMad/settings_production.py. The lines to be edited are:</p> |
|
|
126 | 126 | <pre>SECRET_KEY = '*****' |
|
|
127 | 127 | ALLOWED_HOSTS = ['madrigal3.haystack.mit.edu'] |
|
|
128 | 128 | ADMINS = (('Bill Rideout', 'brideout@haystack.mit.edu'),) |
|
|
129 | 129 | EMAIL_HOST = 'hyperion.haystack.mit.edu' |
|
|
130 | 130 | MANAGERS = (('Bill Rideout', 'brideout@haystack.mit.edu'),) |
|
|
131 | 131 | STATIC_URL = '/static/'</pre> |
|
|
132 | 132 | |
|
|
133 | 133 | <p>The STATIC_URL line only needs to be edited if MAD<b></b>SERVERROOT is not empty; that is, if your madrigal is installed in a subdirectory. |
|
|
134 | 134 | For example, if your url is http://madrigal.sri.com/madrigal, you would change that line to be STATIC_URL = '/madrigal/static/'. But if your |
|
|
135 | 135 | url is http://madrigal.sri.com, you would leave that STATIC_URL line as is.</p> |
|
|
136 | 136 | |
|
|
137 | 137 | <p>You can generate a new secret key for your site at http://www.miniwebtool.com/django-secret-key-generator/, which is recommended for security reasons. The other lines are self-explanatory. </p> |
|
|
138 | 138 | <p>When you are done, be sure to restart apache for the changes to take effect. Note that because this Madrigal Django application runs in daemon mode, it will be necessary to restart the web server if you ever update the python code.</p> |
|
|
139 | 139 | |
|
|
140 | 140 | <h3>Throttling Madrigal admin emails</h3> |
|
|
141 | 141 | <p>Sometimes Madrigal users write scripts that cause numerous admin emails, and in addition malicious users can also generate numerous admin emails. |
|
|
142 | 142 | If you would like to throttle the number of Madrigal admin emails you receive, see <a href="{% url 'docs' 'ad_other.html' %}#throttle">Throttling admin emails</a></p> |
|
|
143 | 143 | <h4> </h4> |
|
|
144 | 144 | <!-- InstanceEndEditable --> |
|
|
145 | 145 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
146 | 146 | <tr> |
|
|
147 |
<td width="5%"><a href="{% url 'docs' 'ad_install.html' %}"><img src=" |
|
|
|
148 |
<td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src=" |
|
|
|
149 |
<td width="5%"><a href="{% url 'docs' 'ad_metadata.html' %}"><img src=" |
|
|
|
147 | <td width="5%"><a href="{% url 'docs' 'ad_install.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
148 | <td width="5%"><a href="{% url 'docs' 'admin.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
149 | <td width="5%"><a href="{% url 'docs' 'ad_metadata.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
150 | 150 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Upgrading or moving Madrigal <!-- InstanceEndEditable --></td> |
|
|
151 | 151 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
152 | 152 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
153 | 153 | </tr> |
|
|
154 | 154 | </table> |
|
|
155 | 155 | <div class='online-navigation'> |
|
|
156 | 156 | <b class="navlabel">Previous:</b> |
|
|
157 | 157 | <a class="sectref" href="{% url 'docs' 'ad_install.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Installing Madrigal <!-- InstanceEndEditable --></A> |
|
|
158 | 158 | <b class="navlabel"> Up:</b> |
|
|
159 | 159 | <a class="sectref" href="{% url 'docs' 'admin.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Madrigal admin guide <!-- InstanceEndEditable --></A> |
|
|
160 | 160 | <b class="navlabel"> Next:</b> |
|
|
161 | 161 | <a class="sectref" href="{% url 'docs' 'ad_metadata.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Madrigal data model <!-- InstanceEndEditable --></A></div> |
|
|
162 | 162 | <hr/> |
|
|
163 | 163 | <p> </p> |
|
|
164 | 164 | </body> |
|
|
165 | 165 | <!-- InstanceEnd --></html> |
| @@ -1,75 +1,75 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Madrigal administrator's guide</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href="/madrigal |
|
|
|
10 | <link href="/madrigal{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="madContents.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_appropriate.html" --><!-- InstanceParam name="href_back_top" type="text" value="rr_fortran95.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="rr_fortran95.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="madContents.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_appropriate.html" --><!-- InstanceParam name="href_prev_top" type="text" value="rr_fortran95.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="madContents.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_appropriate.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="rr_fortran95.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="madContents.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_appropriate.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 | 22 | <td width="5%"><a href="{% url 'docs' 'rr_fortran95.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> |
|
|
23 | 23 | <td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> |
|
|
24 | 24 | <td width="5%"><a href="{% url 'docs' 'ad_appropriate.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Madrigal administrator's guide<!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'rr_fortran95.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Fortran95 remote API reference <!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'madContents.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Doc home <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'ad_appropriate.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Is Madrigal appropriate?<!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">Madrigal administrator's guide</h1> |
|
|
40 | 40 | <p>This section of the Madrigal documentation is meant for people considering installing Madrigal to hold data from their instruments, or those who have already installed Madrigal and are responsible for administering or updating it. This guide describes how to determine whether Madrigal is right for your data, and how to install it if it is. It also discusses how to create Madrigal data files, and how to add them to Madrigal. </p> |
|
|
41 | 41 | <ul> |
|
|
42 | 42 | <li><a href="{% url 'docs' 'ad_appropriate.html' %}">Is Madrigal appropriate for my instrument(s)?</a></li> |
|
|
43 | 43 | <li><a href="{% url 'docs' 'ad_install.html' %}">Installing Madrigal for the first time </a></li> |
|
|
44 | 44 | <li><a href="{% url 'docs' 'ad_upgrade.html' %}">Upgrading Madrigal to the latest release</a></li> |
|
|
45 | 45 | <li><a href="{% url 'docs' 'ad_metadata.html' %}">The Madrigal data model and metadata files</a></li> |
|
|
46 | 46 | <li><a href="{% url 'docs' 'ad_experiments.html' %}">How Madrigal data is organized</a></li> |
|
|
47 | 47 | <li><a href="{% url 'docs' 'ad_createFiles.html' %}">Creating Madrigal data files</a></li> |
|
|
48 | 48 | <li><a href="{% url 'docs' 'ad_createExp.html' %}">Creating and updating Madrigal experiments</a></li> |
|
|
49 | 49 | <li><a href="{% url 'docs' 'ad_other.html' %}">Other administrative tasks</a></li> |
|
|
50 | 50 | <li><a href="{% url 'docs' 'ad_logging.html' %}">User access logging</a></li> |
|
|
51 | 51 | <li><a href="{% url 'docs' 'ad_links.html' %}">Creating direct links to experiments/files</a></li> |
|
|
52 | 52 | <li><a href="{% url 'docs' 'ad_isprint.html' %}">Using isprint for file quick looks</a></li> |
|
|
53 | 53 | </ul> |
|
|
54 | 54 | <!-- InstanceEndEditable --> |
|
|
55 | 55 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
56 | 56 | <tr> |
|
|
57 | 57 | <td width="5%"><a href="{% url 'docs' 'rr_fortran95.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> |
|
|
58 | 58 | <td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> |
|
|
59 | 59 | <td width="5%"><a href="{% url 'docs' 'ad_appropriate.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> |
|
|
60 | 60 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Madrigal administrator's guide<!-- InstanceEndEditable --></td> |
|
|
61 | 61 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
62 | 62 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
63 | 63 | </tr> |
|
|
64 | 64 | </table> |
|
|
65 | 65 | <div class='online-navigation'> |
|
|
66 | 66 | <b class="navlabel">Previous:</b> |
|
|
67 | 67 | <a class="sectref" href="{% url 'docs' 'rr_fortran95.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Fortran95 remote API reference <!-- InstanceEndEditable --></A> |
|
|
68 | 68 | <b class="navlabel"> Up:</b> |
|
|
69 | 69 | <a class="sectref" href="{% url 'docs' 'madContents.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Doc home <!-- InstanceEndEditable --></A> |
|
|
70 | 70 | <b class="navlabel"> Next:</b> |
|
|
71 | 71 | <a class="sectref" href="{% url 'docs' 'ad_appropriate.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Is Madrigal appropriate? <!-- InstanceEndEditable --></A></div> |
|
|
72 | 72 | <hr/> |
|
|
73 | 73 | <p> </p> |
|
|
74 | 74 | </body> |
|
|
75 | 75 | <!-- InstanceEnd --></html> |
| @@ -1,87 +1,87 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Madrigal developer's guide</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 | 10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 | 16 | <!-- InstanceParam name="href_up_top" type="text" value="madContents.html" --><!-- InstanceParam name="href_next_top" type="text" value="madrigal/index.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="madContents.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="madrigal/index.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="madContents.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="madrigal/index.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="madContents.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="madrigal/index.html" --> |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 | 22 | <td width="5%"><a href="{% url 'docs' 'ad_isprint.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> |
|
|
23 | 23 | <td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> |
|
|
24 | 24 | <td width="5%"><a href="{% url 'docs' 'madrigal/index.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Madrigal developer's guide <!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'ad_isprint.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Using isprint<!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'madContents.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Doc home <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 | 36 | <a class="sectref" href="{% url 'docs' 'madrigal/index.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->Internal python API<!-- InstanceEndEditable --></A></div> |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">Madrigal developer's guide</h1> |
|
|
40 | 40 | <p>This section of the Madrigal documentation is meant for developers working to extend or modify Madrigal. It is not meant for users of Madrigal or Madrigal administrators. </p> |
|
|
41 | 41 | <p>With the release of Madrigal 3.0, Madrigal has only a single python API. The C and Fortran code installed in Madrigal is purely scientific code used in the derivation engine, and is exposed only as a python extension, madrigal._derive.</p> |
|
|
42 | 42 | <p>The web interface is now a Django web application, and no direct cgi scripts remain. The django application is contained in MADROOT/source/madpy/djangoMad.</p> |
|
|
43 | 43 | <ul> |
|
|
44 | 44 | <li><a href="madrigal/index.html">Internal Madrigal Python API</a> |
|
|
45 | 45 | <ul> |
|
|
46 | 46 | <li><a href="{% url 'docs' 'madrigal/admin.m.html' %}">madrigal.admin</a></li> |
|
|
47 | 47 | <li><a href="{% url 'docs' 'madrigal/cedar.m.html' %}">madrigal.cedar</a></li> |
|
|
48 | 48 | <li><a href="{% url 'docs' 'madrigal/data.m.html' %}">madrigal.data</a></li> |
|
|
49 | 49 | <li><a href="{% url 'docs' 'madrigal/derivation.m.html' %}">madrigal.derivation</a></li> |
|
|
50 | 50 | <li><a href="{% url 'docs' 'madrigal/isprint.m.html' %}">madrigal.isprint</a></li> |
|
|
51 | 51 | <li><a href="{% url 'docs' 'madrigal/metadata.m.html' %}">madrigal.metadata</a></li> |
|
|
52 | 52 | <li><a href="{% url 'docs' 'madrigal/openmadrigal.m.html' %}">madrigal.openmadrigal</a></li> |
|
|
53 | 53 | <li><a href="{% url 'docs' 'madrigal/ui/index.html' %}">madrigal.ui</a> |
|
|
54 | 54 | <ul> |
|
|
55 | 55 | <li><a href="{% url 'docs' 'madrigal/ui/madrigalPlot.m.html' %}">madrigal.ui.madrigalPlot</a></li> |
|
|
56 | 56 | <li><a href="{% url 'docs' 'madrigal/ui/userData.m.html' %}">madrigal.ui.userData</a></li> |
|
|
57 | 57 | <li><a href="{% url 'docs' 'madrigal/ui/web.m.html' %}">madrigal.ui.web</a></li> |
|
|
58 | 58 | </ul> |
|
|
59 | 59 | </li> |
|
|
60 | 60 | </ul> |
|
|
61 | 61 | </li> |
|
|
62 | 62 | <li><a href="{% url 'docs' 'dev_derivation.html' %}">Madrigal derivation engine</a></li> |
|
|
63 |
<li><a href=" |
|
|
|
64 |
<li><a href=" |
|
|
|
63 | <li><a href="{% static 'CEDARMadrigalHdf5Format.pdf' %}">CEDAR Madrigal Hdf5 file format</a></li> | |
|
|
64 | <li><a href="{% static 'cedarFormat.pdf' %}">Deprecated Cedar file format</a></li> | |
|
|
65 | 65 | </ul> |
|
|
66 | 66 | <!-- InstanceEndEditable --> |
|
|
67 | 67 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
68 | 68 | <tr> |
|
|
69 | 69 | <td width="5%"><a href="{% url 'docs' 'ad_isprint.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> |
|
|
70 | 70 | <td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> |
|
|
71 | 71 | <td width="5%"><a href="{% url 'docs' 'madrigal/index.html' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> |
|
|
72 | 72 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Madrigal developer's guide<!-- InstanceEndEditable --></td> |
|
|
73 | 73 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
74 | 74 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
75 | 75 | </tr> |
|
|
76 | 76 | </table> |
|
|
77 | 77 | <div class='online-navigation'> |
|
|
78 | 78 | <b class="navlabel">Previous:</b> |
|
|
79 | 79 | <a class="sectref" href="{% url 'docs' 'ad_isprint.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Using isprint <!-- InstanceEndEditable --></A> |
|
|
80 | 80 | <b class="navlabel"> Up:</b> |
|
|
81 | 81 | <a class="sectref" href="{% url 'docs' 'madContents.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Doc home <!-- InstanceEndEditable --></A> |
|
|
82 | 82 | <b class="navlabel"> Next:</b> |
|
|
83 | 83 | <a class="sectref" href="{% url 'docs' 'madrigal/index.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->Internal python API <!-- InstanceEndEditable --></A></div> |
|
|
84 | 84 | <hr/> |
|
|
85 | 85 | <p> </p> |
|
|
86 | 86 | </body> |
|
|
87 | 87 | <!-- InstanceEnd --></html> |
| @@ -1,95 +1,95 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Madrigal Derivation Engine</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --><!-- InstanceEndEditable --> |
|
|
10 |
<link href=" |
|
|
|
10 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
11 | 11 | <style type="text/css"> |
|
|
12 | 12 | html body { |
|
|
13 | 13 | background-color: {{bg_color}}; |
|
|
14 | 14 | } |
|
|
15 | 15 | </style> |
|
|
16 |
<!-- InstanceParam name="href_up_top" type="text" value="dev_contents.html" --><!-- InstanceParam name="href_next_top" type="text" value=" |
|
|
|
16 | <!-- InstanceParam name="href_up_top" type="text" value="dev_contents.html" --><!-- InstanceParam name="href_next_top" type="text" value="{% static 'CEDARMadrigalHdf5Format.pdf' %}" --><!-- InstanceParam name="href_back_top" type="text" value="madrigal/ui/web.m.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="madrigal/ui/web.m.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="dev_contents.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="{% static 'CEDARMadrigalHdf5Format.pdf' %}" --><!-- InstanceParam name="href_prev_top" type="text" value="madrigal/ui/web.m.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="dev_contents.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="{% static 'CEDARMadrigalHdf5Format.pdf' %}" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="madrigal/ui/web.m.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="dev_contents.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="{% static 'CEDARMadrigalHdf5Format.pdf' %}" --> | |
|
|
17 | 17 | </head> |
|
|
18 | 18 | |
|
|
19 | 19 | <body> |
|
|
20 | 20 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
21 | 21 | <tr> |
|
|
22 |
<td width="5%"><a href="{% url 'docs' 'madrigal/ui/web.m.html' %}"><img src=" |
|
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'dev_contents.html' %}"><img src=" |
|
|
|
24 |
<td width="5%"><a href=" |
|
|
|
22 | <td width="5%"><a href="{% url 'docs' 'madrigal/ui/web.m.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
23 | <td width="5%"><a href="{% url 'docs' 'dev_contents.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% static 'CEDARMadrigalHdf5Format.pdf' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
25 | 25 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Madrigal Derivation Engine<!-- InstanceEndEditable --></td> |
|
|
26 | 26 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
27 | 27 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
28 | 28 | </tr> |
|
|
29 | 29 | </table> |
|
|
30 | 30 | <div class='online-navigation'> |
|
|
31 | 31 | <b class="navlabel">Previous:</b> |
|
|
32 | 32 | <a class="sectref" href="{% url 'docs' 'madrigal/ui/web.m.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->madrigal.ui.web <!-- InstanceEndEditable --></A> |
|
|
33 | 33 | <b class="navlabel"> Up:</b> |
|
|
34 | 34 | <a class="sectref" href="{% url 'docs' 'dev_contents.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Developer's toc <!-- InstanceEndEditable --></A> |
|
|
35 | 35 | <b class="navlabel"> Next:</b> |
|
|
36 |
<a class="sectref" href=" |
|
|
|
36 | <a class="sectref" href="{% static 'CEDARMadrigalHdf5Format.pdf' %}"><!-- InstanceBeginEditable name="NextTitle" -->CEDAR Madrigal Hdf5 File Format<!-- InstanceEndEditable --></A></div> | |
|
|
37 | 37 | <hr/> |
|
|
38 | 38 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
39 | 39 | <h1 align="center">Madrigal Derivation Engine</h1> |
|
|
40 | 40 | <p>The Madrigal derivation is defined in the python module madrigal.derivation. In particular, that module begins with a collections.OrderedDict dictionary called MadrigalDerivedMethods. To add a new derivation method, simply define the new method in MadrigalDerivedMethods, and then implement it in C or python. If you add a new C method, you will also need to update the file source/madpy/madrigal/_derive.c, which is done simply by running <em>$MADROOT/bin/python $MADROOT/source/madpy/scripts/bin/updateMadDerivations.py</em>. This page goes into the derivation engine in detail.</p> |
|
|
41 | 41 | <h4>MadrigalDerivedMethods</h4> |
|
|
42 | 42 | <p>MadrigalDerivedMethods is a collections.OrderDict in madrigal.derivation. The name of each derivation method is the key, and so must be unique. The value is a list with between two and four items.</p> |
|
|
43 | 43 | <ol> |
|
|
44 | 44 | <li>The first item is the input mnemonic(s).</li> |
|
|
45 | 45 | <li>The second item is the output mnemonic(s).</li> |
|
|
46 | 46 | <li>The optional third parameter is either 'python' or 'C'. If only two parameters, 'C' is assumed. This determines whether the method is implementated on C (or Fortran wrapped in C), or python.</li> |
|
|
47 | 47 | <li>The optional fourth parameter is a list of kinst values for which the derivation is valid (used in statistical models, etc). This feature requires 'python' to be the implementation method.</li> |
|
|
48 | 48 | </ol> |
|
|
49 | 49 | <p>The order of these derivation methods matter - they will be called in a single pass from first to last. The MadrigalDerivationPlan |
|
|
50 | 50 | determines which need to be called, and what parameters are derivable given the initial measured parameters. |
|
|
51 | 51 | Any parameter will be derived by the first available method that has that parameter as an output and |
|
|
52 | 52 | for which all input parameters are measured or themselves derivable.</p> |
|
|
53 | 53 | |
|
|
54 | 54 | <p>For the C derivation methods, the methods are implemented in source/madc/madrec/madDeriveMethods.c. All methods must have a signature of the form:</p> |
|
|
55 | 55 | <pre>int methodName(int inCount, |
|
|
56 | 56 | double * inputArr, |
|
|
57 | 57 | int outCount, |
|
|
58 | 58 | double * outputArr, |
|
|
59 | 59 | FILE * errFile)</pre> |
|
|
60 | 60 | <p>where inCount is the number of input arguments, and inputArr is a double array of length inCount, with |
|
|
61 | 61 | values for each input variable. Likewise, outCount is the number of output arguments, and outputArr |
|
|
62 | 62 | is already allocated double array of length outCount, where the C method sets the values for each |
|
|
63 | 63 | output variable when it returns. If values cannot be calulated, that output value is set to nan.</p> |
|
|
64 | 64 | <p>When a new C derivation method is added, this method needs to be exposed to python. This is done through the file madpy/madrigal/_derive.c. However, you do not need to manually edit this file. Instead, cd $MADROOT/source/madpy/scripts/bin and then run:</p> |
|
|
65 | 65 | <pre>$MADROOT/bin/python $MADROOT/source/madpy/scripts/bin/updateMadDerivations.py</pre> |
|
|
66 | 66 | <p>This will create the file madpy/madrigal/_derive_new.c. Compare this new file with the existing file madpy/madrigal/_derive.c, and if the change looks reasonable, add it to source control.</p> |
|
|
67 | 67 | <p>For the python derivation methods, all methods are defined in madrigal.derivation.MadrigalDerivationMethods |
|
|
68 | 68 | in this module. Each method has two arguments - a double array of input values of length the number of input |
|
|
69 | 69 | variables, and the output array the length of the number of output variable. As in the C, the output |
|
|
70 | 70 | array is passed in to the derivation method preallocated.</p> |
|
|
71 | 71 | |
|
|
72 | 72 | |
|
|
73 | 73 | <P> |
|
|
74 | 74 | <!-- InstanceEndEditable --> |
|
|
75 | 75 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
76 | 76 | <tr> |
|
|
77 |
<td width="5%"><a href="{% url 'docs' 'madrigal/ui/web.m.html' %}"><img src=" |
|
|
|
78 |
<td width="5%"><a href="{% url 'docs' 'dev_contents.html' %}"><img src=" |
|
|
|
79 |
<td width="5%"><a href=" |
|
|
|
77 | <td width="5%"><a href="{% url 'docs' 'madrigal/ui/web.m.html' %}"><img src="{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
78 | <td width="5%"><a href="{% url 'docs' 'dev_contents.html' %}"><img src="{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
79 | <td width="5%"><a href="{% static 'CEDARMadrigalHdf5Format.pdf' %}"><img src="{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
80 | 80 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Madrigal C API<!-- InstanceEndEditable --></td> |
|
|
81 | 81 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
82 | 82 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
83 | 83 | </tr> |
|
|
84 | 84 | </table> |
|
|
85 | 85 | <div class='online-navigation'> |
|
|
86 | 86 | <b class="navlabel">Previous:</b> |
|
|
87 | 87 | <a class="sectref" href="{% url 'docs' 'madrigal/ui/web.m.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->madrigal.ui.web<!-- InstanceEndEditable --></A> |
|
|
88 | 88 | <b class="navlabel"> Up:</b> |
|
|
89 | 89 | <a class="sectref" href="{% url 'docs' 'dev_contents.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Developer's toc<!-- InstanceEndEditable --></A> |
|
|
90 | 90 | <b class="navlabel"> Next:</b> |
|
|
91 |
<a class="sectref" href=" |
|
|
|
91 | <a class="sectref" href="{% static 'CEDARMadrigalHdf5Format.pdf' %}"><!-- InstanceBeginEditable name="NextTitle2" -->CEDAR Madrigal Hdf5 File Format <!-- InstanceEndEditable --></A></div> | |
|
|
92 | 92 | <hr/> |
|
|
93 | 93 | <p> </p> |
|
|
94 | 94 | </body> |
|
|
95 | 95 | <!-- InstanceEnd --></html> |
| @@ -1,87 +1,87 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <title>Isprint filters description</title> |
|
|
8 |
<link href=" |
|
|
|
8 | <link href="{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
9 | 9 | <style type="text/css"> |
|
|
10 | 10 | html body { |
|
|
11 | 11 | background-color: {{bg_color}}; |
|
|
12 | 12 | } |
|
|
13 | 13 | </style> |
|
|
14 | 14 | <!-- InstanceParam name="href_up_top" type="text" value="admin.html" --><!-- InstanceParam name="href_next_top" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_back_top" type="text" value="ad_other.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="ad_other.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_prev_top" type="text" value="ad_other.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="ad_isprint.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="ad_other.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="admin.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="ad_isprint.html" --> |
|
|
15 | 15 | </head> |
|
|
16 | 16 | |
|
|
17 | 17 | <body> |
|
|
18 | 18 | |
|
|
19 | 19 | <h2>Isprint filter description</h2> |
|
|
20 | 20 | |
|
|
21 | 21 | <center><h4>Creating filter strings for isprint</h4></center> |
|
|
22 | 22 | |
|
|
23 | 23 | <p>Combine filters below into one string with spaces. Example:</p> |
|
|
24 | 24 | |
|
|
25 | 25 | |
|
|
26 | 26 | <pre> |
|
|
27 | 27 | "date1=12/01/2005 time1=12:00:00 date1=12/02/2005 time1=08:00:00 el=0,45 filter=ti,500,1000" |
|
|
28 | 28 | </pre> |
|
|
29 | 29 | |
|
|
30 | 30 | <h4>Time range</h4> |
|
|
31 | 31 | |
|
|
32 | 32 | <pre> |
|
|
33 | 33 | date1=mm/dd/yyyy - starting date to be examined. If time1 not given, defaults to 0 UT. |
|
|
34 | 34 | Example: date1=01/20/1998 |
|
|
35 | 35 | |
|
|
36 | 36 | time1=hh:mm:ss - starting UT time to be examined. If date1 given, is applied to date1. If not, applies on the first day of the experiment. |
|
|
37 | 37 | Example: time1=13:30:00 |
|
|
38 | 38 | |
|
|
39 | 39 | date2=mm/dd/yyyy - ending date to be examined. If time2 not given, defaults to 0 UT. |
|
|
40 | 40 | Example: date2=01/21/1998 |
|
|
41 | 41 | |
|
|
42 | 42 | time2=hh:mm:ss - ending UT time to be examined - If date2 not given, refers to date1. If date1 and date2 not given, refers to 1st day. |
|
|
43 | 43 | Example: time2=15:45:00 |
|
|
44 | 44 | </pre> |
|
|
45 | 45 | |
|
|
46 | 46 | <p>In the follow arguments ranges are used. If any range value is not given, it may be used to indicate no lower or upper limit (but the comma is always required). Ranges are inclusive of the end points.</p> |
|
|
47 | 47 | |
|
|
48 | 48 | <h4>Geodetic altitude</h4> |
|
|
49 | 49 | |
|
|
50 | 50 | <pre> |
|
|
51 | 51 | z=lower alt limit1, upper alt limit1 [or lower alt limit2 , upper alt limit2 ...] (km) |
|
|
52 | 52 | Example 1: z=100,500 This would limit the geodetic altitude to 100 to 500 km. |
|
|
53 | 53 | Example 2: z=100,200or300,400 This would limit the geodetic altitude to 100 to 200 km or 300 to 400 km. |
|
|
54 | 54 | Example 3: z=,200or300,400 Since the lower limit of the first range is missing, this would limit the geodetic altitude to anything below 200 km or from 300 to 400 km. |
|
|
55 | 55 | </pre> |
|
|
56 | 56 | |
|
|
57 | 57 | <h4>Azimuth (from -180 to 180)</h4> |
|
|
58 | 58 | |
|
|
59 | 59 | <pre> |
|
|
60 | 60 | az=lower az limit1, upper az limit1 [or lower az limit2 , upper az limit2 ...] (from -180 to 180 degrees) |
|
|
61 | 61 | Example 1: az=100,120 This would limit the azimuth to 100 to 120 degrees. |
|
|
62 | 62 | Example 2: az=-180,-90or90,180 This would limit the azimuth to between -180 and -90 degrees or to between 90 and 180 degrees. Note this allows a filter to go through 180 degrees. |
|
|
63 | 63 | </pre> |
|
|
64 | 64 | |
|
|
65 | 65 | <h4>Elevation (from 0 to 90)</h4> |
|
|
66 | 66 | |
|
|
67 | 67 | <pre> |
|
|
68 | 68 | el=lower el limit1, upper el limit1 [or lower el limit2 , upper el limit2 ...] (from 0 to 90) |
|
|
69 | 69 | Example 1: el=0,45 This would limit the elevation from 0 to 45 degrees. |
|
|
70 | 70 | </pre> |
|
|
71 | 71 | |
|
|
72 | 72 | <h4>Pulse length (in seconds)</h4> |
|
|
73 | 73 | |
|
|
74 | 74 | <pre> |
|
|
75 | 75 | plen=lower pl limit1, upper pl limit1 [or lower pl limit2 , upper pl limit2 ...] (pulse len in sec) |
|
|
76 | 76 | Example 1: plen=,5e-4 This would limit the pulse length to 5e-4 seconds or less. |
|
|
77 | 77 | </pre> |
|
|
78 | 78 | |
|
|
79 | 79 | <h4>Free form filters</h4> |
|
|
80 | 80 | |
|
|
81 | 81 | <pre> |
|
|
82 | 82 | filter=[mnemonic] or [mnemonic1,[+-*/]mnemonic2] , lower limit1 , upper limit1 [or lower limit2 , upper limit2 ...] (any number of filters may be added) |
|
|
83 | 83 | Example: filter=ti,500,1000or2000,3000 Limits the data to points where Ti is between 500 and 1000 degrees or between 2000 and 3000 degrees. Note that the units are always those of the Cedar standard. |
|
|
84 | 84 | Example: filter=gdalt,-,sdwht,0, This filter implies "gdalt - sdwht" must be greater than 0.0. Since sdwht is shadow height (the distance above any point on the earth where the sun is first visible), this filter implies that only data in direct sunlight will be displayed. |
|
|
85 | 85 | Example: filter=ti,/,Dti,100, Limits the data to points where the ratio Ti/dTi is more than 100. |
|
|
86 | 86 | </pre> |
|
|
87 | 87 | |
| @@ -1,50 +1,50 | |||
|
|
1 | 1 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
2 | 2 | <html xmlns="http://www.w3.org/1999/xhtml"> |
|
|
3 | 3 | <head> |
|
|
4 | 4 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
5 | 5 | <title>Madrigal documentation - v3.0</title> |
|
|
6 |
<link href="/madrigal |
|
|
|
6 | <link href="/madrigal{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
7 | 7 | <style type="text/css"> |
|
|
8 | 8 | html body { |
|
|
9 | 9 | background-color: {{bg_color}}; |
|
|
10 | 10 | } |
|
|
11 | 11 | </style> |
|
|
12 | 12 | </head> |
|
|
13 | 13 | |
|
|
14 | 14 | <body> |
|
|
15 | 15 | <center> |
|
|
16 | 16 | <h1>Madrigal Database v3.2.1 Documentation - Contents</h1> |
|
|
17 | 17 | <table width="100%" border="1"> |
|
|
18 | 18 | <tr> |
|
|
19 | 19 | <td class="navigation"><a href="/">Home</a></td> |
|
|
20 | 20 | </tr> |
|
|
21 | 21 | </table> |
|
|
22 | 22 | </center> |
|
|
23 | 23 | <hr size="4" /> |
|
|
24 | 24 | <ul> |
|
|
25 | 25 | <li><a href="{% url 'docs' 'madIntroduction.html' %}">1. Brief history of Madrigal</a></li> |
|
|
26 | 26 | </ul> |
|
|
27 | 27 | <ul> |
|
|
28 | 28 | <li><a href="{% url 'docs' 'whatsNew.html' %}">2. What's new in Madrigal 3.2.1</a></li> |
|
|
29 | 29 | </ul> |
|
|
30 | 30 | <ul> |
|
|
31 | 31 | <li><a href="{% url 'docs' 'wt_usersGuide.html' %}">3. Madrigal user's guide (How do I access Madrigal data?) </a> |
|
|
32 | 32 | <ul> |
|
|
33 | 33 | <li><a href="{% url 'docs' 'wt_contents.html' %}">2.1 Web interface tutorial </a></li> |
|
|
34 | 34 | <li><a href="{% url 'docs' 'rt_contents.html' %}">2.2 Remote data access programming tutorial</a></li> |
|
|
35 | 35 | <li><a href="{% url 'docs' 'rr_contents.html' %}">2.3 Remote data access programming reference guide</a></li> |
|
|
36 | 36 | </ul> |
|
|
37 | 37 | </li> |
|
|
38 | 38 | </ul> |
|
|
39 | 39 | <ul> |
|
|
40 | 40 | <li><a href="{% url 'docs' 'admin.html' %}">4. Madrigal Administrator's Guide</a></li> |
|
|
41 | 41 | </ul> |
|
|
42 | 42 | <ul> |
|
|
43 | 43 | <li><a href="{% url 'docs' 'dev_contents.html' %}">5. Madrigal Developer's Guide</a></li> |
|
|
44 | 44 | </ul> |
|
|
45 | 45 | <ul> |
|
|
46 | 46 | <li><a href="{{ siteSpecific }}">6. Site specific documentation</a></li> |
|
|
47 | 47 | </ul> |
|
|
48 | 48 | <p> </p> |
|
|
49 | 49 | </body> |
|
|
50 | 50 | </html> |
| @@ -1,71 +1,71 | |||
|
|
1 | 1 | {% load static %} |
|
|
2 | 2 | |
|
|
3 | 3 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|
|
4 | 4 | <html xmlns="http://www.w3.org/1999/xhtml"><!-- InstanceBegin template="/Templates/doc_template.dwt" codeOutsideHTMLIsLocked="false" --> |
|
|
5 | 5 | <head> |
|
|
6 | 6 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> |
|
|
7 | 7 | <!-- InstanceBeginEditable name="doctitle" --> |
|
|
8 | 8 | <title>Madrigal history</title> |
|
|
9 | 9 | <!-- InstanceEndEditable --><!-- InstanceBeginEditable name="head" --> |
|
|
10 | 10 | <!-- InstanceEndEditable --> |
|
|
11 |
<link href="/madrigal |
|
|
|
11 | <link href="/madrigal{% static 'madrigal.css' %}" rel="stylesheet" type="text/css" /> | |
|
|
12 | 12 | <style type="text/css"> |
|
|
13 | 13 | html body { |
|
|
14 | 14 | background-color: {{bg_color}}; |
|
|
15 | 15 | } |
|
|
16 | 16 | </style> |
|
|
17 | 17 | <!-- InstanceParam name="href_up_top" type="text" value="madContents.html" --><!-- InstanceParam name="href_next_top" type="text" value="whatsNew.html" --><!-- InstanceParam name="href_back_top" type="text" value="madContents.html" --><!-- InstanceParam name="href_back_bottom" type="text" value="madContents.html" --><!-- InstanceParam name="href_up_bottom" type="text" value="madContents.html" --><!-- InstanceParam name="href_next_bottom" type="text" value="whatsNew.html" --><!-- InstanceParam name="href_prev_top" type="text" value="madContents.html" --><!-- InstanceParam name="href_uptitle_top" type="text" value="madContents.html" --><!-- InstanceParam name="href_nexttitle_top" type="text" value="whatsNew.html" --><!-- InstanceParam name="href_prevtitle_bottom" type="text" value="madContents.html" --><!-- InstanceParam name="href_uptitle_bottom" type="text" value="madContents.html" --><!-- InstanceParam name="href_nexttitle_bottom" type="text" value="whatsNew.html" --> |
|
|
18 | 18 | </head> |
|
|
19 | 19 | |
|
|
20 | 20 | <body> |
|
|
21 | 21 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
22 | 22 | <tr> |
|
|
23 |
<td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="/madrigal |
|
|
|
24 |
<td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="/madrigal |
|
|
|
25 |
<td width="5%"><a href="{% url 'docs' 'whatsNew.html' %}"><img src="/madrigal |
|
|
|
23 | <td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="/madrigal{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
24 | <td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="/madrigal{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
25 | <td width="5%"><a href="{% url 'docs' 'whatsNew.html' %}"><img src="/madrigal{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
26 | 26 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleTop" -->Brief history of Madrigal <!-- InstanceEndEditable --></td> |
|
|
27 | 27 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
28 | 28 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
29 | 29 | </tr> |
|
|
30 | 30 | </table> |
|
|
31 | 31 | <div class='online-navigation'> |
|
|
32 | 32 | <b class="navlabel">Previous:</b> |
|
|
33 | 33 | <a class="sectref" href="{% url 'docs' 'madContents.html' %}"><!-- InstanceBeginEditable name="PreviousTitle" -->Doc home <!-- InstanceEndEditable --></A> |
|
|
34 | 34 | <b class="navlabel"> Up:</b> |
|
|
35 | 35 | <a class="sectref" href="{% url 'docs' 'madContents.html' %}"><!-- InstanceBeginEditable name="UpTitle" -->Doc home <!-- InstanceEndEditable --></A> |
|
|
36 | 36 | <b class="navlabel"> Next:</b> |
|
|
37 | 37 | <a class="sectref" href="{% url 'docs' 'whatsNew.html' %}"><!-- InstanceBeginEditable name="NextTitle" -->What's new in Madrigal 3.0? <!-- InstanceEndEditable --></A></div> |
|
|
38 | 38 | <hr/> |
|
|
39 | 39 | <!-- InstanceBeginEditable name="EditDoc" --> |
|
|
40 | 40 | <center> |
|
|
41 | 41 | <h1>Brief history of Madrigal </h1> |
|
|
42 | 42 | </center> |
|
|
43 | 43 | <p>Madrigal is a database of ground-based measurements and models of the Earth's upper atmosphere and ionosphere. It is the community database of the Coupling, Energetics and Dynamics of Atmospheric Regions (<a href="http://cedarweb.vsp.ucar.edu/wiki/index.php/Main_Page">CEDAR</a>) program, which is devoted to the characterization and understanding of the atmosphere above about 60 km, with emphasis on the various processes that determine the basic structure and composition of the atmosphere, and on the mechanisms that couple different atmospheric regions. Instruments developed or upgraded under CEDAR include interferometers, spectrometers, imagers, lidars and medium, high-frequency and incoherent scatter radars. The success of CEDAR has been due, in large measure, to its ability to encourage collaborative efforts coalescing observations, theory and modeling. The CEDAR community includes about 800 scientists and students from around the world.</p> |
|
|
44 | 44 | <p>From the inception of the CEDAR program in 1988, there has been a great concern among the members of the CEDAR community to make the data collected by the CEDAR instruments easily accessible for joint studies. Consequently, a high priority was placed on establishing a repository for CEDAR data and model results. An incoherent scatter radar database had been established at the National Center for Atmospheric Research (NCAR) in 1985, and this evolved into the CEDAR Database in 1989. By the end of 1997, it had grown to include data from 44 instruments and 16 models. Over 200 users have requested information from the Database. In 2012, the Madrigal database and the CEDAR database were combined into a single database system, the Madrigal CEDAR database, based at MIT Haystack Observatory.</p> |
|
|
45 |
<p>A central element of the CEDAR Database is a standard data format. With Madrigal 3.0 the file format is now Hdf5, with a well-defined layout and parameters. This new CEDAR Hdf5 format is described <a href="/madrigal |
|
|
|
45 | <p>A central element of the CEDAR Database is a standard data format. With Madrigal 3.0 the file format is now Hdf5, with a well-defined layout and parameters. This new CEDAR Hdf5 format is described <a href="/madrigal{% static 'CEDARMadrigalHdf5Format.pdf' %}" target="_blank">here</a>. This format replaced the old <a href="/madrigal{% static 'cedarFormat.pdf' %}">16 bit integer based format</a> that evolved from the format used by the earlier incoherent scatter database, which in turn evolved from an earlier version of Madrigal developed at the MIT Haystack Observatory in 1980. </p> | |
|
|
46 | 46 | <p>Haystack maintains and develops the Madrigal CEDAR database as an open-source project with community contributions. With the Madrigal CEDAR database, the site owner stores only their own data, which they can add to or update at any time. However, because the Madrigal database shares its metadata with all other Madrigal sites, users browsing any Madrigal site can search for data at any other Madrigal site. In addition, a central archival Madrigal site (<a href="http://cedar.openmadrigal.org" target="_self">cedar.openmadrigal.org</a>) archives all Madrigal sites and stores data not stored at other Madrigal sites.</p> |
|
|
47 | 47 | <p>Madrigal data are arranged into "experiments", which may contain data files, images, documentation, links, etc. A key feature of Madrigal is its seamless integration of archival and real-time data. A realtime file on Madrigal is accessed in exactly the same way as any archival file.</p> |
|
|
48 | 48 | <p>Madrigal has been installed at numerous locations in addition to Millstone Hill, including EISCAT, SRI International, and Jicamarca. The inventories of experiments available at each installation are available to the other installations through shared metadata. New Madrigal sites can be automatically added at any time.</p> |
|
|
49 | 49 | <p>Madrigal is an open source project with a central <a href="http://cedar.openmadrigal.org/openmadrigal">developer's web site</a> and a <a href="http://atlas.haystack.mit.edu/cgi-bin/millstone_viewvc.cgi/openmadrigal/" target="_self">central repository</a>. A complete Subversion archive of all Madrigal software, including software which is not included with the standard distribution, is available at the developer's web site. There is also an Open Madrigal mailing list and developer's forum. Any group wishing to install Madrigal to distribute their instrument's data is welcome to - see the http://cedar.openmadrigal.org/openmadrigal web site for details. The <a href="http://atlas.haystack.mit.edu/cgi-bin/millstone_viewvc.cgi/openmadrigal/" target="_self">central repository</a> link is a place where all data from all Madrigal sites is archived. It can also be used to access data, although accessing the local sites is preferred.</p> |
|
|
50 | 50 | <!-- InstanceEndEditable --> |
|
|
51 | 51 | <table width="100%" border="1" cellpadding="0" cellspacing="2" class="navigation"> |
|
|
52 | 52 | <tr> |
|
|
53 |
<td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="/madrigal |
|
|
|
54 |
<td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="/madrigal |
|
|
|
55 |
<td width="5%"><a href="{% url 'docs' 'whatsNew.html' %}"><img src="/madrigal |
|
|
|
53 | <td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="/madrigal{% static 'previous.png' %}" alt="previous" width="32" height="32" /></a></td> | |
|
|
54 | <td width="5%"><a href="{% url 'docs' 'madContents.html' %}"><img src="/madrigal{% static 'up.png' %}" alt="up" width="32" height="32" /></a></td> | |
|
|
55 | <td width="5%"><a href="{% url 'docs' 'whatsNew.html' %}"><img src="/madrigal{% static 'next.png' %}" alt="next" width="32" height="32" /></a></td> | |
|
|
56 | 56 | <td width="54%"><!-- InstanceBeginEditable name="EditTitleBottom" -->Brief overview of Madrigal <!-- InstanceEndEditable --></td> |
|
|
57 | 57 | <td width="13%"><a href="{% url 'docs' 'madContents.html' %}">Doc home </a></td> |
|
|
58 | 58 | <td width="18%"><a href="/">Madrigal home</a></td> |
|
|
59 | 59 | </tr> |
|
|
60 | 60 | </table> |
|
|
61 | 61 | <div class='online-navigation'> |
|
|
62 | 62 | <b class="navlabel">Previous:</b> |
|
|
63 | 63 | <a class="sectref" href="{% url 'docs' 'madContents.html' %}"><!-- InstanceBeginEditable name="PreviousTitle2" -->Doc Home <!-- InstanceEndEditable --></A> |
|
|
64 | 64 | <b class="navlabel"> Up:</b> |
|
|
65 | 65 | <a class="sectref" href="{% url 'docs' 'madContents.html' %}"><!-- InstanceBeginEditable name="UpTitle2" -->Doc home <!-- InstanceEndEditable --></A> |
|
|
66 | 66 | <b class="navlabel"> Next:</b> |
|
|
67 | 67 | <a class="sectref" href="{% url 'docs' 'whatsNew.html' %}"><!-- InstanceBeginEditable name="NextTitle2" -->What's new in Madrigal 3.0? <!-- InstanceEndEditable --></A></div> |
|
|
68 | 68 | <hr/> |
|
|
69 | 69 | <p> </p> |
|
|
70 | 70 | </body> |
|
|
71 | 71 | <!-- InstanceEnd --></html> |
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
|
|
1 | NO CONTENT: modified file | |
| The requested commit or file is too big and content was truncated. Show full diff | |||
General Comments 0
You need to be logged in to leave comments.
Login now