Telemetry and units for altitude and speed

A pilot contacted me the other day to ask this:

Using the exact same GPX file generated from Bad Elf. If I create a flight from it on the iPhone app, by sharing it from the Bad Elf app, or as a file from Files app, the Altitude of meters and Speed of m/s both get converted properly to feet and knots when viewed on the web site. Using the exact same file on the web page will not cause the conversion to feet and knots to take place, and we get altitudes that are 1/3 the correct, and speeds about 1/2 correct. What is going on?

For the most part, Telemetry on MyFlightbook is unitless.  This helps in some areas but can be a bit  confusing in others.

Here is where it is important to distinguish a format from a schema.  A format is simply the way that data is represented in a file.  CSV and XML are both formats (and both text-file based).  CSV is a simple table with each line of text represents a row in a table, and columns within a row are separated by a comma (hence "Comma Separated Values", or CSV).  XML uses start and end tags (e.g., "<altitude>48.3</altitude>") to delimit data.  But note that while a format tells you how to read the data, it doesn't say anything about how the data should be interpreted.  It could be literally anything, as long as it's correctly formatted.

A schema is a convention for how data is interpreted.  E.g., declaring that an altitude is labeled "altitude" and is represented in meters.  GPX and KML are two common ways to represent telemetry, and these are both XML-based schemas.  That is, their format is XML, but the data is arranged in a specified manner that allows you to interpret it correctly.

GPX and KML use meters for altitude and meters/second for speed, and MyFlightbook (both web and iOS/Android) knows this when importing; it can thus infer reasonable speeds to use for takeoff or for landing.

CSV is a significantly more compact and extensible format than XML, but there is no "CSV Schema"; the data can be whatever you want in whatever units you want.  Because it is compact and can include MyFlightbook specific data (such as noting when a takeoff or landing occurs), the MyFlightbook mobile apps generate CSV.  And, since feet and knots are generally what aviators around the world are familiar with, those are the units that MyFlightbook uses in its CSV file. 

But since you can upload CSV from any number of sources (engine monitors, GPS logs, etc.), MyFlightbook can’t make any assumption about the units used; I just have a dictionary of column headings that have some semantic meaning to me (“Heading”, “Altitude”, “Speed”, “Oil Temp”), where I can at least determine the type of the data (number, date, time, latitude/longitude, etc.), but I can’t assume units. (OK, I suppose “Heading” is only in degrees, but my other examples here are inherently ambiguous).  Any data with a column heading that is not in the dictionary is simply ignored.

You may notice that when viewing telemetry on MyFlightbook, units are not displayed, and this is why.  It is also why MyFlightbook's default units in the CSV it generates are feet and kts; that way the numbers “look right” (even in the metric world, where people can work with altitudes in meters just fine, most people don’t think in terms of meters/second).

Both the iOS and Android apps can import GPX/KML (or even NMEA, on iOS, which is yet another schema) and create a flight from it.  So why do the units convert to ft/kts when doing this?

The iOS/Android apps import by treating the data that as an alternative GPS source.  So they're not directly attaching the data to the flight. Instead, the architecture is more like this:

I.e., the system for detecting takeoffs/landings receives data in real-time from the physical GPS (typical) OR from a fake GPS data stream that derives from a file; it doesn’t (in fact, can’t) distinguish the source.  Since the input file has a schema, the system knows which units to use and can convert as appropriate to determine takeoffs/landings/etc. 

As it is fed the data and detecting the flight, it is also optionally recording the flight, which as I described above, it does using CSV and using feet and knots; this is what is submitted up to the server when the flight is saved.  

So the original GPX/KML/NMEA used for import is lost; it’s a data source, but think of it not as a data source for your flight, but rather as a data source for the GPS emulator. 

On the web, the process is similar, but there are three key differences: 

• There isn't an actual GPS (it’s always from a file)
• You’re always attaching the file and optionally detecting, which is the reverse of the mobile apps (which always detect and only optionally recording the result)
• You can attach/autofill from CSV

This has a couple of implications.  One is that the web does not generate a telemetry file and instead attaches what you submit directly and with no translation.  

And while the input units for GPX/KML are specified, that's not true for CSV.  MyFlightbook's graphing system (which is web only) needs to work with all of these formats, so it has a layer of abstraction built in which converts whatever underlying the data format is into a simple table (rows/columns) of data, where each column has a label (e.g., “Speed”, “Oil Temperature”) and a data type (“number”, “timestamp”, etc.).  The graphing system at that point has no idea about semantics or units; it just graphs the raw data.  So a GPX file will graph in meters and meters/second, while a CSV file would (probably) graph in feet and knots, but the only way you'd be able to distinguish is to look at the values: if you see "35,000" in the altitude, it's probably feet.

(I have had a request to allow you to apply a conversion factor at display time, so that if you know the data is in metric you can convert to ft/kts at display time, but I haven’t gotten around to that.)

This works just fine until you go to download the data from the website.  Downloading in the original format is easy: no translation is required; if it was GPX, it is still GPX.  

But what if you want a KML version of your data (to display in Google Earth, for example) and the original data was CSV?  The problem is that KML has to be in meters and meters/second, but the CSV may have been feet and knots.  For this reason, when you download your data in a format that has a schema, you are asked to declare the units for speed/altitude in the original data. It will then load the data into a table (as described above), apply the conversions, and then render into the correct output. 

So, for example, if you initialized a flight on your iPhone from Bad Elf data and saved it, that should have saved in CSV format with ft/kts (which you can verify this by downloading in original format).  If you want a GPX version of that flight, you would download GPX and declare that the original units were knots and feet; that tells the system which conversion factor to apply to the CSV data to produce valid GPX.