NT Data Logging: Difference between revisions

From STorM32-BGC Wiki
Jump to navigation Jump to search
(39 intermediate revisions by the same user not shown)
Line 1: Line 1:
A data logger, the NT logger, can be connected to the NT bus, which allows us to record on a micro SD card a substantial amount of data. The data are recorded for each cycle of the controller loop, i.e., with the maximal possible time resolution. The NT logger is also quite small and can be easily mounted on-board. This way one can get highly accurate and informative data taken during a real flight condition. The same data can also be recorded on the ground, without the need of a NT logger, using the live-recording feature.
''The information on this page refers to firmware v2.56e and higher.''


The data logging option hopefully will allow us to better diagnose any in-flight troubles, such as jello or micro vibrations in the video, and find solutions to resolve them. It also suggests many other applications. For instance, one may use it to dynamically balance the motors and propellers.  
A data logger, the NT Logger, can be connected to the NT bus, which allows us to record on a micro SD card a substantial amount of data. The data are recorded for each cycle of the controller loop, i.e., with the maximal possible time resolution. The NT Logger is quite small and can be easily mounted on any vehicle. This way one can get highly accurate and informative data taken during a real flight condition. The same data can also be recorded on the ground, without the need of a NT Logger, using the live-recording feature.


{{COMMENT|The data logger must be a NT logger, any other logger, such as the widely available OpenLog and its derivatives, cannot be used.}}
The data logging option allows us to diagnose any in-flight troubles, such as jello or micro vibrations in the video, and find solutions to resolve them. It also suggests many other applications. For instance, one may use it to dynamically balance the motors and propellers.  


== NT Logger Module ==
{{COMMENT|The data logger must be a NT Logger; any other logger, such as the widely available OpenLog and its derivatives, cannot be used.}}
 
<div class="toclimit-2">__TOC__</div>
 
== NT Logger Modules ==
 
=== Micro NT Logger Module v1.32 ===
 
The Micro NT Logger module has the RTC integrated and uses a replaceable battery cell.
 
:[[File:Storm32-micro-logger.jpg|680px]]
 
=== NT Logger Module v1.3 ===


[[File:P1100441rcg.JPG|342px]] [[File:P1100682-kl.JPG|240px]] [[File:P1100685-kl.JPG|240px]]
[[File:P1100441rcg.JPG|342px]] [[File:P1100682-kl.JPG|240px]] [[File:P1100685-kl.JPG|240px]]


The NT logger module is connected to the NT bus, like any other NT module.  
This NT Logger module can be extended by connecting a RTC (real time clock) module to its I2C port. Any RTC module which is based on the [https://datasheets.maximintegrated.com/en/ds/DS3231.pdf DS3231] chip can be used. However, the "Raspberry Pi Mini RTC module" is most recommended, because of its low cost and tiny size. In fact, the NT Logger v1.3 PCB is designed specifically for that module.
 
:[[File:P1120095kl.JPG|280px]]
 
{{COMMENT|The battery on these RTC modules lasts for about 1.5 years, but is not chargeable. That means that after that time the RTC will either not work anymore or needs to be replaced, which is quite a pain to do.}}
 
== SD Card ==


The micro SD card must be formatted with FAT32. The firmware author uses the Windows tool. Using the [https://www.sdcard.org/downloads/formatter_4/ SD Association formatting tool] didn't yielded any noticeable advantage.  
The micro SD card must be formatted with FAT32. The firmware author uses the Windows tool. Using the [https://www.sdcard.org/downloads/formatter_4/ SD Association formatting tool] didn't yielded any noticeable advantage.  


The micro SD card should be of very high speed. Unfortunately, not any card works equally well. Little experience is available so far:
The micro SD card should be of very high speed. Unfortunately, not any card works equally well. This experience is available:


{| class="wikitable"
{| class="wikitable"
Line 20: Line 38:
!reported by
!reported by
|-
|-
| SanDisk Extreme SDHC 16GB UHS-I Class 10 U3 || very good, missing frames of ca. 0.1% || OlliW, GekoCH
| SanDisk Extreme Micro SDHC 32GB Class 10 UHS-I UHS-Class 3 V30 A1-L (SDSQXAF-032G-GN6MA) || excellent, nearly no missing frames || OlliW
|-
| SanDisk Extreme Micro SDHC 16GB UHS-I Class 10 U3 || very good, missing frames of ca. 0.1% || OlliW, GekoCH
|-
|-
| SanDisk Ultra SDHC 16GB UHS-I Class 10 HC1 || very good, missing frames below 0.1% || OlliW
| SanDisk Ultra Micro SDHC 16GB UHS-I Class 10 HC1 || very good, missing frames below 0.1% || OlliW
|-
|-
| Kingston 8GB HC class 4 || bad, missing frames of ca. 20% || OlliW
| Kingston 8GB HC class 4 || bad, missing frames of ca. 20% || OlliW
Line 28: Line 48:
| Kingston 32GB HC class 4 || bad, missing frames of ca. 20% || OlliW
| Kingston 32GB HC class 4 || bad, missing frames of ca. 20% || OlliW
|}
|}
== RTC ==
The NT Logger module can be equipped with a RTC (real time clock) module to its I2C port, and the Micro NT Logger has it already on board. This makes it possible to append the data file names with a precise date&time stamp, which makes identifying the data files on the SD card much simpler. This little feature is in fact so useful, that one wouldn't want to miss it once one has used it once :).
The RTC module must be configured before it can be properly used, i.e., the current date and time must be written to it at least once. This can be conveniently done via the GUI: Go to the {{GUI|Tools}} menu and call the {{GUI|NTLogger RTC Tool}}.


== NTLoggerTool ==
== NTLoggerTool ==


For analyzing the recorded data, the NTLoggerTool is available, providing tools such as FFT, and others. It also allows us to convert the data to other formats, most notably to .cfl files. The .cfl files are suitable for the program BlackboxExplorer for STorM32, which is a decendant of [https://github.com/cleanflight/blackbox-log-viewer CleanFlight's BlackboxExplorer]. The NTLoggerTool also allows us to record directly the data on the NT bus using a USB-TTL adapter; a feature called live-recording. It enables many work-bench tests and procedures.
For analyzing the recorded data, the NTLoggerTool is available, providing tools such as FFT, and others. The NTLoggerTool also allows us to record directly the data on the NT bus using a USB-TTL adapter; a feature called live-recording. It enables many work-bench tests and procedures.
   
   
The NTLoggerTool is steadily advancing with fast pace; any manual would be quickly outdated. For a discussion it is best to join the STorM32 NT thread at rcgroups.
The NTLoggerTool is steadily advancing and any manual would be quickly outdated. For a discussion it is best to join the STorM32 NT thread at rcgroups.


The NTLoggerTool is open source (GPL3). Anyone is highly welcome to contribute to it.
The NTLoggerTool is open source (GPL3). Anyone is highly welcome to contribute to it.
Line 39: Line 65:
== Logged Data ==
== Logged Data ==


The data logger "spies" on the STorM32's NT Tx line, i.e., is capable of recording all data transmitted from the STorM32 to the NT modules. The points in the controller loop at which data are "sampled" are indicated in this sketch:
The NT Logger "spies" the STorM32's NT Tx line and records all data transmitted from the STorM32 to the NT modules.  
 
=== Controller ===
 
The points in the controller loop at which data are "sampled" are indicated in this sketch:


[[File:Storm32-nt-logger-data-sources-02.jpg|880px|center]]
[[File:Storm32-nt-logger-data-sources-02.jpg|880px|center]]
This is however not all data recorded by the NT Logger; it records many more.
=== Autopilot System Time ===
When BetaCopter/BetaPlane is used and a MAVLink connection is established, then the autopilot's system time is recorded in the NT logs (if the autopilot can provide this info, e.g., if a GPS is available and has a fix). This is highly useful for matching NT logs and autopilot logs.


== GUI Settings ==
== GUI Settings ==


However, not all data samples may be send out. Which data are emitted by the STorM32 can be controlled by the parameter {{PARAMNAME|NT Logging}}, found in the {{GUI|Setup}} window.  
The actual data which are emitted by the STorM32 controller can be adjusted with the parameter {{PARAMNAME|NT Logging}}, found in the {{GUI|Setup}} window. It is a bit mask, and allows combining these options:
 
:[[File:Storm32-nt-logger-gui.jpg|480px]]
 
The individual options enable these data packets:


{| class="wikitable" style="text-align: center;"
{| class="wikitable" style="text-align: center;"
!{{PARAMNAME|NT Logging}}
!{{PARAMNAME|NT Logging}}
!basic
!data
!pid
!accgyro
!accgyro_raw
|-
|
|SetLogger<br>CmdAhrs<br>SetMotorAll
|CmdPid
|CmdAccGyro
|CmdAccGyroRaw<br>Imu1 & Imu2 & Imu3
|-
| style="text-align:left;" | {{PARAMVALUE|off}} || - || - || - || -
|-
| style="text-align:left;" | {{PARAMVALUE|basic}} || style="background-color: lightgreen;"| x || - || - || -
|-
|-
| style="text-align:left;" | {{PARAMVALUE|basic + pid}} || style="background-color: lightgreen;"| x || style="background-color: lightgreen;"| x || - || -
|style="text-align:left;" | {{PARAMVALUE|basic}} || SetLogger, CmdAhrs, SetMotorAll, CmdEncoderData, CmdStorm32LinkData
|-
|-
| style="text-align:left;" | {{PARAMVALUE|basic + accgyro}} || style="background-color: lightgreen;"| x || - || style="background-color: lightgreen;"| x || -
|style="text-align:left;" | {{PARAMVALUE|acc + gyro raw}} || CmdAccGyroRaw for Imu1 & Imu2
|-
|-
| style="text-align:left;" | {{PARAMVALUE|basic + accgyro_raw}} || style="background-color: lightgreen;"| x || - || - || style="background-color: lightgreen;"| x
|style="text-align:left;" | {{PARAMVALUE|pid in}} || CmdPidIn
|-
|-
| style="text-align:left;" | {{PARAMVALUE|basic + pid + accgyro}} || style="background-color: lightgreen;"| x || style="background-color: lightgreen;"| x || style="background-color: lightgreen;"| x || -
|style="text-align:left;" | {{PARAMVALUE|pid}} || CmdPid
|-
|-
| style="text-align:left;" | {{PARAMVALUE|basic + pid + ag_raw}} || style="background-color: lightgreen;"| x || style="background-color: lightgreen;"| x || - || style="background-color: lightgreen;"| x
|style="text-align:left;" | {{PARAMVALUE|acc + gyro}} || CmdAccGyro for Imu1 & Imu2
|-
|-
| style="text-align:left;" | {{PARAMVALUE|full}} || style="background-color: lightgreen;"| x || style="background-color: lightgreen;"| x || style="background-color: lightgreen;"| x || style="background-color: lightgreen;"| x
|style="text-align:left;" | {{PARAMVALUE|inputs}} || CmdFunctionInputValues
|}
|}


It can happen that not all data can be transmitted in one cycle, if the computational load and data rate are too high. This mostly occurs with both 2nd IMU and full data logging enabled. In that case the accgyro data are distributed over several cycles.  
It can happen that not all data can be transmitted in one cycle, if the computational load in that cycle is too high to support the full data rate. This typically occurs with both 2nd IMU and full data logging enabled. In that case some data might be simply missing in the log file.
 
Also the data rate of the micro SD card represents an obvious limitation. If the SD card cannot keep up, frames (the data of one controller cycle) will be lost. Lowering the logging can alleviate that a lot. With modern state-of-the-art high-speed SD cards this is not much of an issue however.


Also the data rate of the micro SD card represents an obvious limitation. If the SD card can't keep up, frames (the data of one controller cycle) will be lost. Lowering the logging can alleviate that a lot.
For vibration analysis, the experience so far shows that the {{PARAMVALUE|acc + gyro raw}} and {{PARAMVALUE|pid in}} data are most useful. It is thus recommended to have these enabled. The {{PARAMVALUE|pid}} and {{PARAMVALUE|acc + gyro}} data are comparatively less useful for that purpose.


For vibration analysis, the experience so far shows that the accgyro_raw data are most useful. It is thus recommended to have these enabled. The pid and accgyro data are comparatively less useful for that purpose.
For analysing the stabilization performance of the gimbal, the {{PARAMVALUE|pid in}} data are by far most useful, as it gives access to the PID error signals. It is thus recommended to have these always enabled.


== Live Recording ==
== Live Recording ==
Line 103: Line 132:


{{DATACLASS|Imu1 Pitch,Roll,Yaw}}: <br>
{{DATACLASS|Imu1 Pitch,Roll,Yaw}}: <br>
Angles of IMU1, or the camera, respectively, as also visible in the GUI. These are the angles as they are coming out of the IMU1 AHRS and enter the PID.
Angles of IMU1, or the camera, respectively, as also visible in the GUI. These are the angles as they are coming out of the IMU1 AHRS and enter the PID and axis coupling calculations.


{{DATACLASS|Imu2 Pitch,Roll,Yaw}}: <br>
{{DATACLASS|Imu2 Pitch,Roll,Yaw}}: <br>
Angles of the 2nd IMU (IMU2), as also visible in the GUI. These are the angles as they are coming out of the IMU2 AHRS and enter the PID and 2nd IMU calculations.
Angles of the 2nd IMU (IMU2), as also visible in the GUI. These are the angles as they are coming out of the IMU2 AHRS and enter the PID and axis coupling calculations.
 
{{DATACLASS|Encoder Pitch,Roll,Yaw}}: <br>
Encoder angles, as also visible in the GUI.
 
{{DATACLASS|PID In Pitch,Roll,Yaw}}: <br>
PID controller error and setpoint angles.


{{DATACLASS|PID Pitch,Roll,Yaw}}: <br>
{{DATACLASS|PID Pitch,Roll,Yaw}}: <br>
PID controller output, as also visible in the GUI. These are the angles as they are coming out after the PID controller and enter the 2nd IMU calculations.
PID controller output.


{{DATACLASS|Ahrs1}}: <br>
{{DATACLASS|Ahrs1}}: <br>
Line 135: Line 170:
These are the raw accelerometer and gyroscope data, as they are coming out of IMU2, before any other processing. Note that therefore the x,y,z axes refer to the orientation of the IMU2 chip, and not the STorM32's x,y,z axes from which the angles are derived, as the orientation correction has not been passed by these data.
These are the raw accelerometer and gyroscope data, as they are coming out of IMU2, before any other processing. Note that therefore the x,y,z axes refer to the orientation of the IMU2 chip, and not the STorM32's x,y,z axes from which the angles are derived, as the orientation correction has not been passed by these data.


{{DATACLASS|Acc3 raw}}, {{DATACLASS|Gyro3 raw}}: <br>
{{DATACLASS|Temp 1+2}}: <br>
These are the raw accelerometer and gyroscope data, as they are coming out of IMU3, before any other processing. Note that therefore the x,y,z axes refer to the orientation of the IMU3 chip, and not the STorM32's x,y,z axes from which the angles are derived, as these data have not passed any orientation correction.
Temperature of IMU1, IMU2
 
{{DATACLASS|Temp 1+2+3}}: <br>
Temperature of IMU1, IMU2, IMU3
 
{{DATACLASS|PID Mot Pitch,Roll,Yaw}}: <br>
Motor angles as they are coming out after the 2nd IMU calculations, and enter the motor drivers.


{{DATACLASS|PID Mot Pitch,Roll,Yaw}}: <br>
{{DATACLASS|Mot Pitch,Roll,Yaw}}: <br>
Motor angles as they are send to the motor drivers.
Motor angles as they are send to the motor drivers.


{{DATACLASS|Vmax Pitch,Roll,Yaw}}: <br>
{{DATACLASS|Vmax Pitch,Roll,Yaw}}: <br>
Vmax for each motor axis.
Vmax for each motor axis.

Revision as of 08:57, 18 August 2020

The information on this page refers to firmware v2.56e and higher.

A data logger, the NT Logger, can be connected to the NT bus, which allows us to record on a micro SD card a substantial amount of data. The data are recorded for each cycle of the controller loop, i.e., with the maximal possible time resolution. The NT Logger is quite small and can be easily mounted on any vehicle. This way one can get highly accurate and informative data taken during a real flight condition. The same data can also be recorded on the ground, without the need of a NT Logger, using the live-recording feature.

The data logging option allows us to diagnose any in-flight troubles, such as jello or micro vibrations in the video, and find solutions to resolve them. It also suggests many other applications. For instance, one may use it to dynamically balance the motors and propellers.

Comment: The data logger must be a NT Logger; any other logger, such as the widely available OpenLog and its derivatives, cannot be used.

NT Logger Modules

Micro NT Logger Module v1.32

The Micro NT Logger module has the RTC integrated and uses a replaceable battery cell.

Storm32-micro-logger.jpg

NT Logger Module v1.3

P1100441rcg.JPG P1100682-kl.JPG P1100685-kl.JPG

This NT Logger module can be extended by connecting a RTC (real time clock) module to its I2C port. Any RTC module which is based on the DS3231 chip can be used. However, the "Raspberry Pi Mini RTC module" is most recommended, because of its low cost and tiny size. In fact, the NT Logger v1.3 PCB is designed specifically for that module.

P1120095kl.JPG

Comment: The battery on these RTC modules lasts for about 1.5 years, but is not chargeable. That means that after that time the RTC will either not work anymore or needs to be replaced, which is quite a pain to do.

SD Card

The micro SD card must be formatted with FAT32. The firmware author uses the Windows tool. Using the SD Association formatting tool didn't yielded any noticeable advantage.

The micro SD card should be of very high speed. Unfortunately, not any card works equally well. This experience is available:

micro SD card experience reported by
SanDisk Extreme Micro SDHC 32GB Class 10 UHS-I UHS-Class 3 V30 A1-L (SDSQXAF-032G-GN6MA) excellent, nearly no missing frames OlliW
SanDisk Extreme Micro SDHC 16GB UHS-I Class 10 U3 very good, missing frames of ca. 0.1% OlliW, GekoCH
SanDisk Ultra Micro SDHC 16GB UHS-I Class 10 HC1 very good, missing frames below 0.1% OlliW
Kingston 8GB HC class 4 bad, missing frames of ca. 20% OlliW
Kingston 32GB HC class 4 bad, missing frames of ca. 20% OlliW

RTC

The NT Logger module can be equipped with a RTC (real time clock) module to its I2C port, and the Micro NT Logger has it already on board. This makes it possible to append the data file names with a precise date&time stamp, which makes identifying the data files on the SD card much simpler. This little feature is in fact so useful, that one wouldn't want to miss it once one has used it once :).

The RTC module must be configured before it can be properly used, i.e., the current date and time must be written to it at least once. This can be conveniently done via the GUI: Go to the [GUI:Tools] menu and call the [GUI:NTLogger RTC Tool].

NTLoggerTool

For analyzing the recorded data, the NTLoggerTool is available, providing tools such as FFT, and others. The NTLoggerTool also allows us to record directly the data on the NT bus using a USB-TTL adapter; a feature called live-recording. It enables many work-bench tests and procedures.

The NTLoggerTool is steadily advancing and any manual would be quickly outdated. For a discussion it is best to join the STorM32 NT thread at rcgroups.

The NTLoggerTool is open source (GPL3). Anyone is highly welcome to contribute to it.

Logged Data

The NT Logger "spies" the STorM32's NT Tx line and records all data transmitted from the STorM32 to the NT modules.

Controller

The points in the controller loop at which data are "sampled" are indicated in this sketch:

Storm32-nt-logger-data-sources-02.jpg

This is however not all data recorded by the NT Logger; it records many more.

Autopilot System Time

When BetaCopter/BetaPlane is used and a MAVLink connection is established, then the autopilot's system time is recorded in the NT logs (if the autopilot can provide this info, e.g., if a GPS is available and has a fix). This is highly useful for matching NT logs and autopilot logs.

GUI Settings

The actual data which are emitted by the STorM32 controller can be adjusted with the parameter NT Logging, found in the [GUI:Setup] window. It is a bit mask, and allows combining these options:

Storm32-nt-logger-gui.jpg

The individual options enable these data packets:

NT Logging data
“basic” SetLogger, CmdAhrs, SetMotorAll, CmdEncoderData, CmdStorm32LinkData
“acc + gyro raw” CmdAccGyroRaw for Imu1 & Imu2
“pid in” CmdPidIn
“pid” CmdPid
“acc + gyro” CmdAccGyro for Imu1 & Imu2
“inputs” CmdFunctionInputValues

It can happen that not all data can be transmitted in one cycle, if the computational load in that cycle is too high to support the full data rate. This typically occurs with both 2nd IMU and full data logging enabled. In that case some data might be simply missing in the log file.

Also the data rate of the micro SD card represents an obvious limitation. If the SD card cannot keep up, frames (the data of one controller cycle) will be lost. Lowering the logging can alleviate that a lot. With modern state-of-the-art high-speed SD cards this is not much of an issue however.

For vibration analysis, the experience so far shows that the “acc + gyro raw” and “pid in” data are most useful. It is thus recommended to have these enabled. The “pid” and “acc + gyro” data are comparatively less useful for that purpose.

For analysing the stabilization performance of the gimbal, the “pid in” data are by far most useful, as it gives access to the PID error signals. It is thus recommended to have these always enabled.

Live Recording

It is also possible to "spy" on the STorM32's NT Tx line using a USB-TTL adapter, and record the data using the NTLoggerTool. The feature is somewhat analogous to the DataDisplay in the STorM32 GUI, except that it records much more data and moreover at full speed, with maximal time resolution.

The USB-TTL adapter must be capable of handling 2.000.000 bauds. Adapters with FT232RL (FTDI) work out of the box. Adapters with CP2102 may be configured to do so, see How to configure CP2102 USB adapters for high baud rates. The firmware author has much better results with a CP2102-based adapter. His FT232RL adapters appear to have problems with keeping up with the high data rate (it's about 1Mb/s), yielding many data gaps, while with a CP2102 adapter no data are lost.

The USB-TTL adapter is connected to the NT bus as follows:

USB-TTL adapter GND -> STorM32 NT GND
USB-TTL adapter Rx  -> STorM32 NT Tx
USB-TTL adapter Tx  -> do not connect

The USB-TTL adapter's Tx line must not be connected to the STorM32's NT Rx line, i.e., should be left floating.

Data Fields

Performance:
Various metrics measuring the execution time of some parts of the code.

Imu1 Pitch,Roll,Yaw:
Angles of IMU1, or the camera, respectively, as also visible in the GUI. These are the angles as they are coming out of the IMU1 AHRS and enter the PID and axis coupling calculations.

Imu2 Pitch,Roll,Yaw:
Angles of the 2nd IMU (IMU2), as also visible in the GUI. These are the angles as they are coming out of the IMU2 AHRS and enter the PID and axis coupling calculations.

Encoder Pitch,Roll,Yaw:
Encoder angles, as also visible in the GUI.

PID In Pitch,Roll,Yaw:
PID controller error and setpoint angles.

PID Pitch,Roll,Yaw:
PID controller output.

Ahrs1:
Various metrics related to the IMU1 AHRS.

State:
State of the controller, as also visible in the GUI. 0 = StrtMotors, 1 = Settle, 2 = Calibrate, 3 = Level, 4 = MotorDir, 5 = Relevel, 6 = Normal, 7 = FastLevel

Error:
Error counts on the NT bus, as also visible in the GUI.

Voltage:
Battery voltage, as also visible in the GUI.

Acc1, Gyro1:
Accelerometer and gyroscope measurements of IMU1. These are the data as they are coming out after the calibration, bias, filter, and orientation corrections, and enter the IMU1 AHRS.

Acc2, Gyro2:
Accelerometer and gyroscope measurements of IMU2. These are the data as they are coming out after the calibration, bias, and orientation corrections, and enter the IMU2 AHRS.

Acc1 raw, Gyro1 raw:
These are the raw accelerometer and gyroscope data, as they are coming out of IMU1, before any other processing. Note that therefore the x,y,z axes refer to the orientation of the IMU1 chip, and not the STorM32's x,y,z axes from which the angles are derived, as the orientation correction has not been passed by these data.

Acc2 raw, Gyro2 raw:
These are the raw accelerometer and gyroscope data, as they are coming out of IMU2, before any other processing. Note that therefore the x,y,z axes refer to the orientation of the IMU2 chip, and not the STorM32's x,y,z axes from which the angles are derived, as the orientation correction has not been passed by these data.

Temp 1+2:
Temperature of IMU1, IMU2

Mot Pitch,Roll,Yaw:
Motor angles as they are send to the motor drivers.

Vmax Pitch,Roll,Yaw:
Vmax for each motor axis.