Automating electronic data sheets CCSDS Spring 2013 Sam Price.
-
Upload
spencer-stokes -
Category
Documents
-
view
219 -
download
0
Transcript of Automating electronic data sheets CCSDS Spring 2013 Sam Price.
Automating electronic data sheetsCCSDS Spring 2013
Sam Price
2 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
Navigator Background• Missions
– GPM– MMS – High Altitude 12Re– Hubble servicing mission
• Reflected GPS signals
• Embedded orbit propagator– Geons– 3rd Party software– 3rd party commands/telemetry
• 45 commands• 43 telemetry messages
GPS orbits, 3Re
Main lobe
First side lobe
42.6 degrees (L1)
GEO, 6Re
LEO (below 3000 km) MMS, 12Re
3 / 23NASA GSFC – CCSDS Spring 2013
Document Count
Document Message Size
Descriptions Member Layout
Meta dataunits,polys,limits
TLM RatePer Mission
Responsible Person5+ people
1,2MMS,GPM
ICDMSWord
(1) Device
3,4GPM,MMS
Telemetry mockup (Ada)
Maybe??? (2) Mockup
5,6MMS,GPM
Ground system (asist)
(3) Ground System owner
7 C++ Headers YesDoxygen
YesDoxygen
(1) Device
8 3rd Party C++Orbit Prop
Yes3rd party
Yes3rd party
(4) 3rd Party C/C++
9 3rd party ICD Orbit Prop
Yes3rd party
Yes3rd party
Yes3rd party
Yes3rd party
(4) 3rd Party ICD
10,11MMS,GPM
Bandwidth Rates (excel)
(5) Bandwidth person
12 C++ Command Verification
(1) Device - Algorithms
13 CSV Logging (1) Device - Logging
14 Analysis (Matlab)
(1) Device - Analysis
15 Endianess converters for CSV Logging
(1) Device - Logging
Navigator Electronic Documents for Command / Telemetry
4 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
Multiple documents are expensive $$$$
• 15 different electronic versions, – manually created
• Same data across all of the documents• 5+ different people maintaining separate
documents. • Changes propagate slowly throughout documents
($$$)• Errors occur from manual changes.
($$$$)• Naming conventions are not the same
5 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
• C/C++– Endianess conversion routines– Command validation– Header files
• Human readable Interface Control Document– Latex– HTML– Maybe MS Word …
• Ground system definitions• Bandwidth spreadsheet• Ada mockup of telemetry for testing.• Logging tools
– CSV Logging– Matlab import– Database logging
C++ headersTelemetry/Commands
3rd party C Headers
Some tools
XML Database
Why C/C++ as starting point– Widespread existence
• All of my Telemetry is in C• More people know C
– Machine readable– Exactly what is implemented– 3rd party libraries are in C– Many more reasons
A streamlined approach
6 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
Convert C/C++ to XML
Workflow
C++ headersTelemetry/Commands
3rd party C Headers
XML
XML Database
xml_to_asist.py
xml_to_ctypes.py
Asist
Asist RDLS
Python
Python CTypes
xml_to_latex.py
Latex ICD
7 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
What does it take to mimic C/C++ in XML• Basic Types
– bools– integers -- Use stdint.h
• int8_t,int16_t,int32_t, int64_t
– unsigned Integers• uint8_t,uint16_t,uint32_t, uint64_t
– floating point memory• float, double, long double
– strings• char arrays
– Arrays• Multidimensional
– void pointer, pointers (16,32,64) sizes
– Structures/classes– unions– enums
• No child / parents / inheritance
• No templates• C/C++ can have
alignment/padding problems• Metadata stored in doxygen
style comments– Units, Polynomials, warning
ranges, etc• Bit fields defined in
comments• No C bitfield yet…
8 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
Use a compiler for parsing• Clang/llvm
– C/C++ Compiler similar to GCC– Cross platform compatible(Windows,Linux,Mac)
• LibClang– Pass arguments to change C/C++ structure padding.
• -m32 , -m64 – Cross platform compatible (Linux, Mac, Windows)– Apple uses it in their developer tools for code completion,
syntax highlighting, etc…– Use it to find comments, structures, types, padding in
structure, etc.• http://clang.llvm.org/docs/Tooling.html• http://clang.llvm.org/get_started.html• https://github.com/trolldbois/clang <- padding/size info
9 / 23NASA GSFC – CCSDS Spring 2013
C Example CCSDS Header/*! * @brief CCSDS Telemetry Header * @TelemetryPacketHeader * */typedef struct CCSDS_Telemetry_Header{ /*! * @brief Destination address */ char p_sp_dest; /*! * @brief Protocol ID * @default 0xf2 */ char p_sp_prot; /*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @enum ENUM_PACKET_TYPE * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @enum CCSDS_Secondary_Header * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135.0-b-1 application ID's 2040- 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */ uint16_t p_ident;/*! . . . . */ uint16_t p_seq_cont;/*! . . . . */ uint16_t p_data_len;/*! . . . . */ uint64_t s_time;} CCSDS_Telemetry_Header;
Doxygen comments start with @marker_name marker value
Indented comments belong to parent comment
Could have used C bitfields(See backup slides)
10 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
C/C++ -> clang_to_json.py -> json_to_xml.py -> XML
• Current status– 500 lines of code to go from C Header to XML– Need to add enums, meta data, bit fields.– Implemented structures, unions, bit structures, arrays,
padding bytes, integers, unsigned integers, floating point numbers, pointers, comment parsing.
• https://github.com/nasa/SOIS-CCSDS-XML– NonNasa Contact for access, currently private project
until XML format is well defined.– NASA: fill out a NAMS/IDMAX request
11 / 23NASA GSFC – CCSDS Spring 2013
C Example p_ident
/*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135.0-b-1 application ID's 2040- 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */ uint16_t p_ident;
12 / 23NASA GSFC – CCSDS Spring 2013
Clang to JSON Output for p_ident
…..{'array': [], 'basetype': 'uint16_t', 'basetype_name': None, 'bitarray': True,'data': <clang.cindex.Cursor object at 0x10a7969e0>, 'is_pointer': False, 'kind': CursorKind.FIELD_DECL, 'name': 'p_ident', 'offset': 16, 'size': 2L,
'comments': [{'bitarray': 'msbf', 'children': [{'bits': '3', 'children': ['name': 'packet_version_number'}, {'brief': 'Pacet Version Number’}, {'default': '0'} ]
}, ], …. More Member Comments …..]
'members': [{'array': [], 'basetype_name': None, 'bitarray': False, 'bits': 3, 'comments': [{'name': 'packet_version_number'}, {'brief': 'Pacet Version Number’}, {'default': '0'}], 'data': None, 'is_pointer': False, 'kind': None, 'members': [], 'name': 'packet_version_number', 'size': -1},… More Members..],},
> python clang_to_json.py ../examples/src_cpp/ccsds.h -m32 –I../examples/src_cpp
• Offset, Sizes include padding information.
• Clang object used for further processing if needed.
• Bit arrays become children for bitstructs.
13 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
XML Layout
• Type database– Basic types
• Integers, floats
– Structures– Unions– Bit structures
• Telemetry database– Similar to structures
• Command database– Similar to structures
<TypeDatabase>
<typeInt/>
<typeFloat/>
<typeBitStruct>
<Members>
<variable/>
</Members>
<typeBitStruct\>
<typeStruct>
<Members>
<variable/>
</Members>
<typeStruct/>
</TypeDatabase>
<Telemetry>
</Telemetry>
<Commands>
</Commands>
14 / 23NASA GSFC – CCSDS Spring 2013
C Example/*! * @brief CCSDS Telemetry Header * @TelemetryPacketHeader * */
typedef struct CCSDS_Telemetry_Header{ /*! * @brief Destination address */ char p_sp_dest; /*! * @brief Protocol ID * @default 0xf2 */ char p_sp_prot; /*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @enum ENUM_PACKET_TYPE * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @enum CCSDS_Secondary_Header * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135.0-b-1 application ID's 2040- 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */
uint16_t p_ident;/*! . . . . */ uint16_t p_seq_cont;/*! . . . . */ uint16_t p_data_len;/*! . . . . */ uint64_t s_time;
} CCSDS_Telemetry_Header;
15 / 23NASA GSFC – CCSDS Spring 2013
Generated XML<TypeDatabase>
<typeInt bits="3" endianess="bigEndian" name="int3_be" signed="true"/>
…. More small types for int1, int11, int8, int16 .. etc ….
<typeBitStruct name="CCSDS_Telemetry_Header_p_ident" type="uint16_be">
<Members>
<variable name="packet_version_number" type="int3_be"/>
<variable name="packet_type" type="int1_be"/>
<variable name="secondary_header_present" type="int1_be"/>
<variable name="apid" type="int11_be"/>
</Members>
</typeBitStruct>
…. More typeBitStruct for p_seq_cont, s_time ….
<typeStruct name="CCSDS_Telemetry_Header">
<Members>
<variable bits="8" name="p_sp_dest" offset="0" type="int8_be"/>
<variable bits="8" name="p_sp_prot" offset="8" type="int8_be"/>
<variable bits="16" name="p_ident" offset="16" type="CCSDS_Telemetry_Header_p_ident"/>
<variable bits="16" name="p_seq_cont" offset="32" type="CCSDS_Telemetry_Header_p_seq_cont"/>
<variable bits="16" name="p_data_len" offset="48" type="uint16_be"/>
<variable bits="64" name="s_time" offset="64" type="CCSDS_Telemetry_Header_s_time"/>
</Members>
</typeStruct>
</TypeDatabase>
> python json_to_xml.py ../examples/src_cpp/messages.h -m32 –I../examples/src_cpp
16 / 23NASA GSFC – CCSDS Spring 2013
Writing XML with pythonfrom xml.etree.ElementTree import Element from xml.etree.ElementTree import ElementTree
parent_node = Element('EDS')
Messages = Element('Telemetry')parent_node.append(Messages)
#All attributes/text must textMessages.attrib['Count'] = str(5)Messages.text = “Hello”
document = ElementTree(parent_node)
#See json_to_xml for prettify functionprint prettify(document.getroot())
17 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
XML Output
<?xml version="1.0" ?><EDS> <Telemetry Count="5"> Hello </Telemetry></EDS>Or Root = document.getroot()print xml.etree.ElementTree.tostring(Root,'utf-8')'<EDS><Telemetry Count="5">Hello</Telemetry></EDS>'
18 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
json_to_xml.py
• XML to Python CTypes• Decode telemetry/send commands
C++ headersTelemetry/Commands
3rd party C Headers
XML
xml_to_ctypes.py
Python
Python CTypes
clang_to_json.py
19 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
Reading XML with pythonimport xml.etree.ElementTree as ETdoc = ET.parse(filename)root = doc.getroot()for node in root.findall('TypeDatabase/typeStruct'): print node.attrib['name’] print node.text
See xml_to_ctypes.py for complete listing200 lines of code to convert from XML to python ctypesOrhttp://docs.python.org/2/library/xml.etree.elementtree.html
20 / 23NASA GSFC – CCSDS Spring 2013
Python ctypes example
import ctypes
class CCSDS_Telemetry_Header_p_ident ( ctypes.LittleEndianStructure ): """""" #_packed_ _fields_ = [ ("apid" , ctypes.c_uint16 , 11 ), ("secondary_header_present” , ctypes.c_uint16 , 1 ), ("packet_type" , ctypes.c_uint16 , 1 ), ("packet_version_number" , ctypes.c_uint16 , 3)]
class CCSDS_Telemetry_Header ( ctypes.LittleEndianStructure ): ""”This is a comment about the CCSDS Packet Header""" _pack_ = 1 _fields_ = [ ("p_sp_dest" , ctypes.c_int8 ), ("p_sp_prot" , ctypes.c_int8 ), ("p_ident" , CCSDS_Telemetry_Header_p_ident ), ("p_seq_cont" , CCSDS_Telemetry_Header_p_seq_cont ), ("p_data_len” , ctypes.c_int16 ), ("s_time" , CCSDS_Telemetry_Header_s_time )]
python xml_to_ctypes.py test.xml > test_ctypes.py 205 lines of code
21 / 23NASA GSFC – CCSDS Spring 2013
Using python ctypes to decode / encode dataimport test_ctypes as tcfid = open('log.raw’)
#Buffer big enough to hold any command / telemetry data
big_buffer = bytearray(1024*10) write_buffer = memoryview(big_buffer)
#header data directly into the buffer. 8bytes.
fid.readinto(write_buffer[0:8])#big_buffer[0:8]#bytearray(b'\x00\xf2E\n\x00\xc0\x17\x00\')
header = tc.CCSDS_Telemetry_Header.from_buffer(big_buffer)#Read in the rest of the CCSDS packet.
fid.readinto(write_buffer[8:(header.p_data_len+1)])print "Apid is %d decode it." % header.p_ident.apid
#changing ctypes changes the binary data. Good for commanding
header.p_sp_dest = 5#big_buffer[0:8]#bytearray(b'\x05\xf2E\n\x00\xc0\x17\x00')
fid.close()
22 / 23NASA GSFC – CCSDS Spring 2013
ctypes cont…class CCSDS_Telemetry_Header ( ctypes.LittleEndianStructure ): "””Documentation about CCSDS Telemetry Header""" _pack_ = 1 _fields_ = [ ("p_sp_dest" , ctypes.c_int8 ), ("p_sp_prot" , ctypes.c_int8 ), ("p_ident" , CCSDS_Telemetry_Header_p_ident ), ("p_seq_cont" , CCSDS_Telemetry_Header_p_seq_cont ), ("_p_data_len" , ctypes.c_int16 ), ("s_time" , CCSDS_Telemetry_Header_s_time )] @property def p_data_len(self): “””More Documentation””” return self._p_data_len + 1 @p_data_len.setter def p_data_len(self, new_len): self._p_data_len = new_len – 1
h = CCSDS_Telemetry_Headerh.p_data_len = 8print h._p_data_len7print h.p_data_len8
CCSDS data length is 1 minus actual length of the message.
Provide documentation via docstring on telemetry value.
Telemetry: Polynomials, conversions, etc
Commanding:limit values, throw errors if not within ranges.
23 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary
Lessoned learned• Provide all of the standard C/C++ types
– As listed on “mimic C/C++ in XML” slide• Standardized naming of Integers, floats, bools, pointers.
(int32,int16,int8, …) – Allows for creating dictionary mappings between XML and
desired language• Mimic the structure of existing languages.
– Makes mapping from XML to existing language more strait forward
• Keep it simple • Develop tools with XML standard
– Kept having to change scheme while making tools to make processing easier
24 / 23NASA GSFC – CCSDS Spring 2013
Backup slides
25 / 23NASA GSFC – CCSDS Spring 2013
C bit field example(Not supporting, yet …)
struct CCSDS_Telemetry_header_p_ident {
uint16_t apid:11;uint16_t secondary_header_present:1;uint16_t packet_type:1;uint16_t packet_version_number:1;
};
http://msdn.microsoft.com/en-us/library/ewwyfdbe(v=vs.71).aspx
Least significant field first
26 / 23NASA GSFC – CCSDS Spring 2013
C Example cont/*! * @ingroup MESSAGES * @brief An example message for Power point. * @apid 135 */typedef struct Basic_M{
CCSDS_Telemetry_Header packet_header; //CCSDS Packet Header/*! * @brief Temp of the baseplate the cpu is attached to. * @polynomial [1566.7676,-18.85822,-.17372, 0.00229, 0.00001] * @unit fahrenheit
* @mode nominal * @red_limit [-Inf,-20), (120,Inf] * @yellow_limit [-20,-15), (115,120] */int32_t cpu_temp;
/*! * @brief Receiver Position * @coordinate_frame ECEF * @unit meter */ double position[3];};
Doxygen comments start with @marker_name marker value
Indented comments belong to parent comment
27 / 23NASA GSFC – CCSDS Spring 2013
Example basic example
<Telemetry>
<Message name="Basic_M">
<Members>
<variable bits="128" name="packet_header" offset="0" type="CCSDS_Telemetry_Header"/>
<variable bits="32" name="cpu_temp" offset="128" type="int32_be"/>
<variable bits="192" name="position" offset="192" type="float64_be">
<array>
<size>
3
</size>
</array>
</variable>
</Members>
</Message>
</Telemetry>