Friday, July 19, 2019

Rounding and totals

Minutes and decimals don't always play nicely together, since minutes are a denominator of 60 when converted to decimal.

As a result, someone will occasionally catch what appears to be a math error.

Consider these totals, and the three flights that make it up that a pilot sent me this past week:

At first glance, this looks incorrect, since 1.37+1.70+1.32=4.39, not 4.38.  But it's actually the best math that I can do, given that I want to keep per-minute accuracy.
In this case, both 4.38 hours (=262.8 minutes) and 4.39 hours (=263.4 minutes) both round to 263 minutes, which then converts to 4.383333333 hours, which displays as 4.38 since I round to two decimal places.

So the good news is that the rounding is only ever off by a hundredth of an hour here or there, and it cancels out statistically.  

Actually, I can make a stronger statement: I deliberately maintain to-the-minute accuracy at all stages - including in totals - but I don't promise to maintain sub-minute accuracy.

Here's where it's worth getting into some of the esoterica, but the important thing is that I'm actually optimizing for the math to always work in the hours:minutes space over the decimal space.

In order to do this, I convert the decimal to hours/minutes before adding. So, for example, 1.32 hours is 79.2 minutes.  I round that to 79 (nearest minute) and then divide back by 60 to get the hours input to the totals, which is 1.31666667 hours.

So in the example above,

Flight, As Logged    Flight, rounded to nearest minute
1.37 1.366666667
1.7 1.7
1.32 1.316666667
Total, Logged:   Total, rounded:
4.39 4.383333333

The left column above is raw addition of the values as entered.  The right column has each of those numbers multiplied by 60 (to get minutes), rounded, and then converted back to hours.  That sums up to 4.38333, or 4.38 hours.  Because each of the numbers in the right column represents a precise minute count, the sum represents a precise minute count, so no minutes are lost or added; this is why I can make the claim that all values are accurate to-the-minute.

Doing it this way keeps the math working regardless of whether you use hh:mm (hours:minutes) or decimal hours. 

It's worth noting that if you're using decimal and only use one decimal place (which is 6 minutes), then there's no rounding issue whatsoever, since 6 minutes precisely equals 0.1 hour.  In hh:mm format, though, you need at least two decimal places to capture all 60 possible minutes in an hour.  But conversely, 2 digits actually gives you 100, not just 60, possible slices of an hour, so there's inevitably some rounding that happens somewhere.

Computers and databases only store so many digits of precision - I chose 2 digits because it's enough to accurately handle minutes in hh:mm.   The conversion I do above (multiplying by 60 and then rounding to get whole minutes) restores as many digits of precision as the computer can handle, which will always be enough for any actual logbook to not have any rounding errors.

Some logbooks store all times under-the-covers as minutes and divide by 60 in the display; that works fine (it's exactly the same as doing the right-hand column above without the division by 60), but it too would have rounding issues.  After all, imagine if you entered 1.31 hours (=78.6 minutes which becomes 79 if you're storing minutes in the database).  When reading from the database and converting for decimal display, that would display as 79 / 60 = 1.3166667 = 1.32 hours.

Yet another alternative is to store lots of digits of precision in the database, but that has its own problems: the math works fine, but putting long decimals into fields you can edit can be cumbersome for you, the user, and leads to other problems as well.  E.g., that 1.316666667 number above is too large to fit in many fields on a screen so it would need to get truncated to, say,
1.3167 - and if you then read that field back (for example, when updating a flight) then presto, you've lost the very precision you were trying to preserve.

So it's an interesting anomaly of decimal display that sometimes the hundredth's-place digit in totals appears to be off.  But yes: since I can promise that all numbers - both at the individual flight level and in totals - are accurate to the minute, then I can also promise that any accumulated rounding error - using to-the-minute flights - is also less than one-sixtieth of an hour.  And since .02 is more than one-sixtieth, and I only have two decimal places, any such rounding discrepancy (I shouldn't even call it "error") should thus actually be limited to 0.01.  And in hh:mm display, there will be no such rounding discrepancy.

Tuesday, June 18, 2019

Flight Templates

MyFlightbook has over 640 different "flight properties" (with new ones added all the time!), which are attributes that you can add to a flight, describing everything from Aircraft Carrier landings to Zero-visibility Takeoffs.  The idea is that any given pilot, on any given flight, uses only a tiny subset of these properties.  This, of course, presents a clutter problem: how do you manage such a large set of potential options?  Historically, I've done this automatically, showing you those properties that you've used before, with an option to explicitly exclude ones you no longer need.

Today, I'm adding new "template" functionality.  The idea is that you can define templates for various sorts of flights, and choose the template that is appropriate for a given entry.  E.g., if you fly for the airlines during the week, but fly gliders on the weekend, you can load up one template for each kind of flight, optimized with the fields you need most of the time.

A template is really nothing more than a collection of flight properties.  You can define your own templates, or choose from a library of templates authored by other pilots.  There are also a few "automatic" templates that the system makes available and applies automatically (discussed below):

Creating Templates

You can define templates by going to "Preferences" under the Profile tab on the website and expanding the "Flight Properties" section.  You'll see a list of your current templates:

Possibly the easiest way to get going with templates is to choose one from a library of templates shared by other pilots.  If you click "Brows available templates", you can browse this library and copy any that you like into your account.

If nothing in the library suits your needs, you can create a new template:

You can give your template a name, a description, and a category.  Then you'll see two lists.  On the left are the set of available properties; on the right are the properties in your template.  Drag-and-drop from left to right to select the properties you want (or from right to left to remove a property you no longer want).  You can also type in the search box to quickly filter properties by name.

The "Category" of a template is just a grouping mechanism.  At the moment, there are 5 categories, though I may add more over time if it makes sense.  Current categories are:
  • Automatic - These are implemented by the system; you can't create these.
  • Training - Flights where you are honing specific skills, such as performing aerobatic maneuvers or instrument approaches, go here.
  • Lesson Plans - Use for flights that are following a specific lesson of a syllabus.
  • Checkrides and Reviews - checkrides and other periodic flights to ensure proficiency like a regulatory/company/club-mandated periodic review of flight skills
  • Missions - Specific purposes for a flight might go here.  E.g., fire fighting, search-and-rescue, charity flights, and so forth.
  • Roles - These are for templates that vary based on your role in a given flight - e.g., are you PIC or SIC? Instructor or student?

Once you save the template, it is available for use.
The two checkboxes in the image above warrant additional explanation:
  • Checking the "Shared" checkbox makes the template available for other pilots; unchecking it removes it from the library of templates.  If you have made a template that works well for your scenario and might work for others, then I strongly encourage you to please check this box!
  • "Use by default" tells the system that you want that template's properties to appear by default for new flights.

Using Templates

Great, you've defined some templates.  So how do you use them?

On the website, you'll see a downward-facing arrow to the right of where you can select properties today:

When you click that arrow, you'll see the set of templates from which you can choose.  As you turn templates on or off, you'll see the set of properties displayed for editing change accordingly.  (Note: if you have any property that has a non-empty value, it will always be displayed, regardless of template selection.

You can choose from the set of templates that you've defined, plus the automatic templates I mentioned earlier.  At the moment, there are three of these:
  • Previously Used - This matches the pre-template functionality.  It contains all of the properties that you've used on prior flights, minus the ones you've explicitly excluded (in Preferences).  This is what is used if you don't check the "Use by default" box for any other templates.
  • Simulator / Training Device (Basic) - this is automatically selected whenever you choose a simulator as the aircraft for a flight.  It (currently) adds two properties ("Ground Instruction Received" and "Simulator/Training Device Identifier") to whatever other properties you are using.
  • Anonymous Aircraft (Basic) - this is automatically selected whenever you choose an anonymous aircraft, and adds a property ("Aircraft Registration") where you can record the specific aircraft used for a flight.
The iOS and Android apps are now updated as well to include template functionality.  On the iOS app, tap the information icon on the right side of the Properties header to view available templates (you may need to refresh your property list for this).  On Android, there is a "View Flight Templates" menu item.  The apps respect default templates, but to create/edit templates you do need to use the website.

"Power template tips"

  • Templates are not mutually exclusive - you can use more than one at a time, or specify that more than one should be used by default.  E.g., if you're a CFII, you might create an "Instrument Flight" template and an "Instructor" template and use them both at the same time.
  • You can attach a template to an aircraft.  Tap on the downward-facing arrow to the right of the aircraft (on the website) and you'll see any templates you've defined.  Whenever you select that aircraft for a flight, the specified template(s) will be used.  For example, if you fly aerobatics in one plane, but not in others, you might define an aerobatic template and use it for flights in that aircraft, but not clutter up flights with aerobatic maneuvers you won't perform in other aircraft:

  • The mobile apps refresh templates from the website when they download properties, which happens periodically.  You can force a refresh, though, by doing pull-to-refresh (try to scroll past the top of the screen) while viewing templates or properties.
Please send me any feedback about templates, and please share any useful ones that you create!

Tuesday, May 28, 2019

What constitutes a "valid" flight entry?

With any data system, one must be prepared to do validation on the data that goes in.  A logbook is no exception to that rule.

It turns out, though, that there are surprisingly few validation checks that can be universally applied to a flight - there are a lot of things that look like an error in some scenarios, but which are perfectly valid in others.

An example that I am asked about frequently is that the system does not flag flights where Dual plus PIC plus SIC does not equal Total time.  For much of flying in much of the world, that equation is satisfied, but (in particular) it is not here in the US, where you can simultaneously log Dual and PIC if you're appropriately rated for the aircraft yet receiving instruction.

Probably the most common scenario where a flight frustrates data validation is "catch-up" flights, where you make one or more entries in your logbook to represent totals from prior (typically paper) logbooks.  Such flights typically include large numbers (larger than any single flight could actually have), and a mix of things that might be difficult or impossible to do in a typical actual wheels-up-to-wheels-down scenario.

Over the years, it seems that whenever I put in a validation check, I soon find counter examples of perfectly acceptable flight entries that violate my rule.

So in an evolutionary process of "survival of the fittest", here are the checks that have withstood the test of time:

  • Valid aircraft - every flight must be associated with an aircraft (even if no actual flying was done; you can generally use any aircraft you like for ground sessions - since the times are zero, any aircraft will work).  This is because the aircraft's model determines all sorts of attributes about the flight, such as the category/class, whether it was complex or high performance or turbine, etc.
  • No negative numbers.  I've yet to find a scenario where negative numbers are allowed.
  • Hobbs start after hobbs end.  (I don't currently validate tach time because I don't currently perform any computations on tach time).
  • Flight start after flight end or engine start after engine end (I don’t currently validate block time) 
  • Full-stop night landings + full stop day landings greater than the total landings (unless total landings is 0, in which case I auto-fill it)
  • Full-stop Night landings indicated without any night flight.
  • Night takeoffs at towered airports + night takeoffs at non-towered airport (which, kinda by definition, is all of the possible night takeoffs) is greater than total night takeoffs
  • More described approaches (e.g., a property indicating two ILS approaches) than logged approaches
  • Comments or Route field too long (13K and 128 characters, respectively)
  • Date before 1900 or more than 2 days from "right now" in Pacific time

Wednesday, May 15, 2019

International functionality Part 3: Functionality

In my previous two posts, I discussed how internationalization affects languages and regional conventions; in this third and final post, I will discuss actual functional changes that vary based on different jurisdictions.

While most software functionality does not vary worldwide (think Microsoft Office or the Chrome browser, which do the exact same things wherever they are), other software must be responsive to local markets and the rules, regulations or conventions therein, which means actual changes in functionality depending on where you are.  While aviation is largely harmonized across the world, it is nevertheless governed by each country and that results in a wide variety of rules.

At one level, simple logging of data is not really dependent on local rules.  After all, what you did on a flight is what you did regardless of where you are. The complexities tend to arise when performing things computations or reports on that data.

A challenge here compared to the language or regional conventions discussed in the prior two posts is that those can typically be determined automatically.  But functional rules generally need to be declared explicitly, because it is not uncommon for someone in one jurisdiction to follow the rules from another jurisdiction.  As a result, all of the functionality I'll describe here are things that are configured explicitly (generally in Profile->Preferences on the website)

Probably the most important (or visible) functionality that can vary is currency rules. By default, MyFlightbook implements US (FAA) rules.  MyFlightbook does also support Canada currency and EASA LAPL (European Light Plane) currency rules.

The FAA "groups" currency by category/class/type, so if you do your takeoffs and landings in a Piper Archer, you're current in a Cessna 172 as well, and vice versa.  But some countries require currency to be specific in a model: you need to have performed your takeoffs and landings in that model in order to be current in that model.  MyFlightbook supports either model.  You can also choose to view your totals grouped by category/class/total, or by model.

Different countries have different certification rules for pilots.  In addition to FAA rules, MyFlightbook supports some EASA, Canadian, and Australian ratings.  Adding new ratings is generally pretty straightforward, so if you can point me to a reference document, send it my way and I'll see what I can do.

Another visible place where different regional conventions are on display is in printing formats.  Some regions, like the US, specify what you should record, but make no particular recommendation on the form or layout in which it is displayed.  Others (Europe) can be very prescriptive (see prior posts about printing here and here).  MyFlightbook currently supports layouts that approximate the EASA standard, as well as layouts that conform to typical layouts in South Africa, Canada, and New Zealand.  If you have an additional layout, send it to me; it's generally pretty easy to add.

There are a few other bits of functionality that, while not strictly international or regulatory, do tend to correlate with region:
  • Input or display of times in decimal vs. hour:minute (HH:MM) format.  You can turn this setting off/on in Profile->Preferences on the website, or in settings on the mobile apps.
  • How night flight and night landings are computed.  You can set this on the web site next to the "autofill" button, or in settings on the mobile apps.  Night can be computed based on sunrise/sunset, or an offset from the start/end of civil twilight.  Night landings can either be anything at night, or 1 hour offset from sunrise/sunset.
  • On the mobile apps, you can also adjust how total flight time is computed from flight/block/engine times.

I'm a US-based pilot and fly on FAA conventions, so that's what I know best, but my goal is for MyFlightbook to be useful to pilots worldwide.  I hope this series of posts help explain MyFlightbook's international support, and I hope you'll contact me if there's any functionality you'd like to see (or bugs I need to fix!)

International functionality Part 2: Local conventions

In my last post I discussed how MyFlightbook handles different languages.  In this one, I will discuss how MyFlightbook handles regional differences.

Language and Region are independent variables.  E.g., English is spoken in both the US and the UK, but we express our dates differently.  I am writing this on May 15, 2019.  In the US, the short form of that date would be written 5/15/2019 (i.e., month/day/year), but in the UK, it would be 15/5/2019 (day/month/year).

Besides dates, there are several other conventions that vary based on locale:
  • Times.  E.g., by default is 4:15pm expressed  as "4:15 PM" or as "16:15"?
  • Decimal points and thousands separators.  E.g., one-thousand two-hundred fifty-two and a half would be "1,252.5" in the US or "1 252,5" in France.
  • List separators: typically a comma or a semicolon.  This is particularly important for CSV
  • Currency symbols.  Obviously, the US uses $, but Europe uses €
All modern operating systems provide services that software like MyFlightbook can use that makes all of this automatic and transparent.  MyFlightbook uses those services, so all conventions should be followed appropriately.

The trick is to ensure that MyFlightbook knows which region should be used.

For the apps on iOS or Android, you can set your device's region by going into the device's settings app.  There's a language and region area where you can set both the preferred language (as discussed in the last post, the iOS and Android apps are translated into French as well as English) and your preferred region.  Most people do not need to do anything here because their device comes pre-configured for their language and region.

The website gets slightly more complicated because it is one instance of the software that is running on a machine in the US (in the cloud somewhere; I actually have no idea where), so it is using US conventions, but it needs to service people from all over the world.  How to solve this?

Whenever your browser requests a page, it sends a list of language-region codes along with the request.  A language region code looks like "en-us": the "en" part means it is requesting English language, and the "us" part means it wants to use US conventions.

The list of these codes is in priority order.  So a request might include the following list: "en-us", "en-gb", "fr-ca", "fr-fr".  This tells the server that English is preferred, then French.  And that the preferred regions are US if possible, then UK ("gb" = "Great Britain"), followed by Canada, then France.

Note that this is sent by the browser, so if you are using a Windows or Mac OS with, say, French Canadian conventions, but your browser is set up for English/US, then you will get English.  The MyFlightbook server can't see your OS's settings, it can only see what the browser sends.

This can be set in various ways, depending on the browser.  In Chrome, you'd go to settings (using the menu or the URL chrome://settings/), then go to advanced, and you can specify language/region there:

In Firefox, you'd go to Options (or about:preferences) and find Languages from there:

In my next post, I'll talk about regional functionality.

International functionality Part 1: Languages

In the software world from which I come (I'm an alum of Microsoft and Expedia), we would think of internationalization along three dimensions: translation into a local language, localization to use local conventions, and functionality that varies by locale.  I figured it might be worth a discussion of how MyFlightbook breaks down along these dimensions.

I'll talk about each of these in successive blog posts, starting here with the first: translation.

Translation is pretty straightforward, albeit very lightly implemented.  While I know varying degrees of several languages, I'm essentially monolingual.  The fact that English is the world-wide language of aviation certainly makes it easier for MyFlightbook to get away with being all English than for other software programs, but nevertheless I am rigorous about ensuring that all of the code - whether on the website or in the mobile apps - can be localized and is not tied to English, even if much of the time English happens to be the only available language. 

In fact, thanks to a French-speaking user who did the initial translation (and Google Translate, which helped with subsequent translations, which might be humorous, but I wouldn't know...), the iOS and Android apps are entirely bi-lingual.  If you set your iOS or Android device to use French, the MyFlightbook UI will appear in French.

It's not seamless, however: much of the text used in display does come from the server, and the server is only lightly translated.  So if you are using French settings, you may still see English inserted in numerous places because that text derives from the database or from the web site (which only has a few pages translated.)

The key thing, though, is that the code is structured to be easily translated.  Specifically, this means that all text is kept outside of the code and loaded (based on language) as it is needed.  Keeping the text separate from the code makes translation easier, since a linguist can take a file with text in it and just go through and translate each item, without having to have any coding capabilities.  And it means that new languages can be added simply by adding appropriate translated text files.

If anybody is interested in helping to translate the website, all of the text is separate from the code, so it's pretty straightforward to do.  (It's just a lot of text...)

Supporting different languages actually goes beyond just translation, though, particularly since MyFlightbook *pilots* speak and use a huge variety of languages.  Most of this is pretty transparent - the system uses Unicode so it can handle data in Kanji, Hindi, Hebrew, Arabic, etc. without any issues.  The main thing that the code needs to handle here is right-to-left languages like Hebrew or Arabic and ensuring that the appropriate alignment occurs based on language (i.e., not hardcoding left or right alignment, but instead keying off of the language).

In my next post, I'll talk about localization.

Wednesday, May 1, 2019

Drones and MyFlightbook

Probably the fastest growing area in aviation these days is in the unmanned aerial system ("UAS", aka "unmanned aerial vehicle" or "UAV", or more colloquially, "drone") space.  And many UAS pilots are using MyFlightbook to record their flights.

MyFlightbook supports UAS as a psuedo-category/class.  E.g., as a peer to AMEL or Helicopter.  You can create UAS aircraft using one of many UAS models on the system, and include drone flights next to manned flights.  Since they are separate category/class, flights in one should not pollute things like currency or totals in the other.

Of course, since these are flights in your logbook, any times you record will accrue in your totals.  But alas, it's also easy to back these out, if you like, by using the search functionality.

There are a variety of flight properties that are in the system today that are focused on UAS scenarios.  As of this writing (more are added all the time!), the system has the following:

Property NameDescription
UAS - AutonomousIndicates that the flight was an autonomous UAS flight
UAS - Hand-held TransmitterIndicates that the UAS flight was controlled by a hand-held transmitter
UAS - Lost Link ReturnIndicates that the UAS executed return procedures after a loss of communication
UAS LaunchesNumber of times that a UAS (drone) was launched
UAS RecoveriesNumber of times that a UAS (drone) was recovered
UAS: Air Vehicle Operator TimeTime spent operating a drone
UAS: Mission Payload Operator TimeTime spent managing the paylod in a drone
UAS: Ground Control Station TimeTime spent in ground control for a drone
UAS: Maritime Flight HoursTime spent flying a drone (UAS) over water
UAS: Electro-Optical Sensor TimeTime spent using an electro-optical sensor (drone/UAS)
UAS: Infrared Sensor TimeTime spent using an infrared sensor (drone/UAS)
UAS: Short Wave Infrared Sensor TimeTime spent using a short-wave infrared sensor (drone/UAS)
UAS: Medium Wave Infrared Sensor TimeTime spent using a medium-wave infrared sensor (drone/UAS)
UAS: Synthetic Aperture Radar Sensor TimeTime spent using a synthetic aperture operations with a drone (UAS)
UAS: Hyperspectral Imaging Sensor TimeTime spent doing imaging (hyperspectral) with a drone (UAS)
UAS: Multispectral Imaging Sensor TimeTime spent doing imaging (multispectral) with a drone (UAS)
UAS: Signals Intelligence TimeTime spent doing signals intelligence with a drone (UAS)
UAS - 107.73 - Aeronautical Knowledge TestIndicates that the pilot took and passed a knowledge test (initial or recurrent) covering the areas specified in 107.73
UAS - 107.74 - Training CourseIndicates that the pilot successfully passed a training course (initial or recurrent) covering the areas specified in 107.74

As part of ensuring that unmanned aircraft are utilized safely - and don't conflict with manned aircraft! - the FAA introduced part 14CFR107, governing unmanned systems. 107.65, in particular, defines recent experience required to operate a UAS under many circumstances.  It works like currency, except that it focuses on recent knowledge training rather than flight experience.

To be current per 107.65, you must have done one of the following three things in the preceding 24-calendar months:
  • Pass an initial knowledge test, 
  • Pass a recurrent knowledge test, or 
  • Have a valid flight review (of the manned flight variety, per 61.56) and take a knowledge course.

To indicate that you have done either of the first items above, add an entry into your logbook with the "UAS - 107.73 - Aeronautical Knowledge Test" property; if you take the appropriate course, add an entry with the "UAS - 107.74 Training Course" property.  And, of course, if you are a licensed pilot of manned aircraft, you'll want to show that you have a valid flight review anyhow; you can do this by adding a property to the appropriate flight: "Flight Review," "BFR", or one or the various checkrides that qualify.

MyFlightbook will show (and update) 107 currency once you have met any of the requirements above.

Tuesday, April 23, 2019

Adding and editing aircraft - Update

A few months ago, I discussed how models get assigned to aircraft when you add a new aircraft, or when you edit an existing one.  This bit of code has always been surprisingly complex and convoluted, but over this past weekend I went in to add some new intelligence to it and was able to clean it up in the process.

The whole complexity here arises from the fact that MyFlightbook tries hard to share aircraft among pilots (for reasons discussed here).  That is, if two or more pilots fly N12345, I want there to be only one instance of N12345 in the system, used by each of those pilots.

This leads to some interesting scenarios, so I thought I'd geek out a bit and describe them here, including the enhancements I just made.

The key to the new enhancement is the notion of finding a "close" or a "perfect" match for an aircraft within a set of aircraft that share a tailnumber: 
  • A "perfect" match means that the model of one aircraft is exactly the same as the model for the other.  E.g., a Cessna C-172N is a perfect match for a C-172N, but not a perfect match for a C-172S.  
  • A "close" match is one where the category/class of two aircraft is the same AND they have the same non-empty ICAO identifier.  E.g., a C-172N and a C-172S both have the ICAO identifier "C172", so they are a close match.
  • The "best" match out of a set of aircraft is a perfect match, if one exists, otherwise a close match, if one exists.  There is not always a "best" match.
OK, so what happens when you *add* an aircraft to your account?  There are now four possible outcomes:
  • If there is no aircraft in the system with that tail number, it is simply added and the model you specified is used.  Piece of cake, very simple, exactly as you'd expect.
  • If there are already one or more aircraft with the same tail number in the system, then I try to find a perfect or close match.  
    • If a perfect match is found, then that aircraft is added to your account.  Just like above, and as you'd expect, but you might acquire some images, notes, and maintenance in the process.
    • Otherwise, if a close match is found, then that aircraft is added to your account, but I send you an email telling you of the substitution (e.g., you added a PA28-161 but it's actually a PA28-181.)
    • Otherwise, I create a new version of the aircraft using the model you specified.  E.g., if you specified N12345 is a Boeing 737 but N12345 is already in the system as a C-172, then the C-172 is left alone and a new Boeing with tail N12345 is created and added to your account.  This is the new enhancement; previously, you'd have picked up the C-172 and received an email; you'd then have to edit N12345 to be the Boeing 737.
Now you have an aircraft in your account - what happens if you try to edit it?

Simple edits like adding notes, pictures, or maintenance events are straightforward - the underlying aircraft is edited.

But things get complicated in two additional scenarios: modifying the model of the aircraft, and/or modifying the tail number of the aircraft.

I discussed editing the model of the aircraft in my earlier blog post; this is essentially unchanged (and indeed used the notion of perfect/close match described above).  So if you edit a Super Cub to be on floats, or edit a Cirrus SR22 to be an Airbus A-320, the system will automatically clone the aircraft for you, with two minor qualifiers/exceptions:
  • If the alternative version already exists, then instead of cloning the aircraft, it will just re-use the existing clone.
  • If you're the only pilot using the aircraft, then no clone is created; I just edit the underlying aircraft, unless there is already an alternative that meets the "best match" criteria above, in which case that match is used.
There had been a corner case here that my enhancement has mitigated: suppose you owned an aircraft with vanity registration NABC, but then bought a new aircraft that isn't the same model, and carried NABC over to the new aircraft.  If you were the only pilot using NABC, there had been no way to get both versions of NABC into the system side by side without contacting me (I have an admin tool to do this).  But with the enhancement above, you can now just add the new version of NABC to your account, and the cloning should occur automatically (assuming there's no "best" match).

A similar issue arises when you edit the tail number of an aircraft in your account.  Usually, there is no reason to edit an aircraft's tail.  (After all, if an airframe gets a new registration, it generally makes the most sense to treat it in MyFlightbook as a new aircraft.)  So the primary reason for doing this is to handle a typo.

There are a few scenarios here, so let's assume you're editing N12345 to be N54321.
  • If you're the only one flying N12345:
    • If N54321 is not in the system, the existing aircraft simply has its tail changed.  Because the existing aircraft is changed, all of your flights in N12345 magically become flights in N54321, not because the flights themselves are changed (they aren't), but because the underlying aircraft they use has changed.
    • If one or more versions of N54321 are in the system, then the best match (as described above) is simply added to your account, or else a clone is created (if there is no best match).  No flights are modified, so in this scenario your flights do NOT reflect a change.
  • If other pilots are flying N12345, then I explicitly do NOT want to modify N12345 in the database (thereby changing their logbooks).  Therefore, I treat this as the creation of a new aircraft and it follows all of the rules for adding a new aircraft above.  Assuming that this is most likely due to a typo, though, the system detects this scenario and modifies your flights to be in the newly added/created aircraft.
Phew.  Who would have thought that adding/editing of aircraft would be so complicated?

Wednesday, March 20, 2019

Telemetry and units for altitude and speed

A pilot contacted me the other day to ask this:

Using the exact same GPX file generated from Bad Elf. If I create a flight from it on the iPhone app, by sharing it from the Bad Elf app, or as a file from Files app, the Altitude of meters and Speed of m/s both get converted properly to feet and knots when viewed on the web site. Using the exact same file on the web page will not cause the conversion to feet and knots to take place, and we get altitudes that are 1/3 the correct, and speeds about 1/2 correct. What is going on?

For the most part, Telemetry on MyFlightbook is unitless.  This helps in some areas but can be a bit  confusing in others.

Here is where it is important to distinguish a format from a schema.  A format is simply the way that data is represented in a file.  CSV and XML are both formats (and both text-file based).  CSV is a simple table with each line of text represents a row in a table, and columns within a row are separated by a comma (hence "Comma Separated Values", or CSV).  XML uses start and end tags (e.g., "<altitude>48.3</altitude>") to delimit data.  But note that while a format tells you how to read the data, it doesn't say anything about how the data should be interpreted.  It could be literally anything, as long as it's correctly formatted.

A schema is a convention for how data is interpreted.  E.g., declaring that an altitude is labeled "altitude" and is represented in meters.  GPX and KML are two common ways to represent telemetry, and these are both XML-based schemas.  That is, their format is XML, but the data is arranged in a specified manner that allows you to interpret it correctly.

GPX and KML use meters for altitude and meters/second for speed, and MyFlightbook (both web and iOS/Android) knows this when importing; it can thus infer reasonable speeds to use for takeoff or for landing.

CSV is a significantly more compact and extensible format than XML, but there is no "CSV Schema"; the data can be whatever you want in whatever units you want.  Because it is compact and can include MyFlightbook specific data (such as noting when a takeoff or landing occurs), the MyFlightbook mobile apps generate CSV.  And, since feet and knots are generally what aviators around the world are familiar with, those are the units that MyFlightbook uses in its CSV file. 

But since you can upload CSV from any number of sources (engine monitors, GPS logs, etc.), MyFlightbook can’t make any assumption about the units used; I just have a dictionary of column headings that have some semantic meaning to me (“Heading”, “Altitude”, “Speed”, “Oil Temp”), where I can at least determine the type of the data (number, date, time, latitude/longitude, etc.), but I can’t assume units. (OK, I suppose “Heading” is only in degrees, but my other examples here are inherently ambiguous).  Any data with a column heading that is not in the dictionary is simply ignored.

You may notice that when viewing telemetry on MyFlightbook, units are not displayed, and this is why.  It is also why MyFlightbook's default units in the CSV it generates are feet and kts; that way the numbers “look right” (even in the metric world, where people can work with altitudes in meters just fine, most people don’t think in terms of meters/second).

Both the iOS and Android apps can import GPX/KML (or even NMEA, on iOS, which is yet another schema) and create a flight from it.  So why do the units convert to ft/kts when doing this?

The iOS/Android apps import by treating the data that as an alternative GPS source.  So they're not directly attaching the data to the flight. Instead, the architecture is more like this:
I.e., the system for detecting takeoffs/landings receives data in real-time from the physical GPS (typical) OR from a fake GPS data stream that derives from a file; it doesn’t (in fact, can’t) distinguish the source.  Since the input file has a schema, the system knows which units to use and can convert as appropriate to determine takeoffs/landings/etc. 

As it is fed the data and detecting the flight, it is also optionally recording the flight, which as I described above, it does using CSV and using feet and knots; this is what is submitted up to the server when the flight is saved.  

So the original GPX/KML/NMEA used for import is lost; it’s a data source, but think of it not as a data source for your flight, but rather as a data source for the GPS emulator. 

On the web, the process is similar, but there are three key differences: 
  • There isn't an actual GPS (it’s always from a file)
  • You’re always attaching the file and optionally detecting, which is the reverse of the mobile apps (which always detect and only optionally recording the result)
  • You can attach/autofill from CSV
This has a couple of implications.  One is that the web does not generate a telemetry file and instead attaches what you submit directly and with no translation.  

And while the input units for GPX/KML are specified, that's not true for CSV.  MyFlightbook's graphing system (which is web only) needs to work with all of these formats, so it has a layer of abstraction built in which converts whatever underlying the data format is into a simple table (rows/columns) of data, where each column has a label (e.g., “Speed”, “Oil Temperature”) and a data type (“number”, “timestamp”, etc.).  The graphing system at that point has no idea about semantics or units; it just graphs the raw data.  So a GPX file will graph in meters and meters/second, while a CSV file would (probably) graph in feet and knots, but the only way you'd be able to distinguish is to look at the values: if you see "35,000" in the altitude, it's probably feet.

(I have had a request to allow you to apply a conversion factor at display time, so that if you know the data is in metric you can convert to ft/kts at display time, but I haven’t gotten around to that.)

This works just fine until you go to download the data from the website.  Downloading in the original format is easy: no translation is required; if it was GPX, it is still GPX.  

But what if you want a KML version of your data (to display in Google Earth, for example) and the original data was CSV?  The problem is that KML has to be in meters and meters/second, but the CSV may have been feet and knots.  For this reason, when you download your data in a format that has a schema, you are asked to declare the units for speed/altitude in the original data. It will then load the data into a table (as described above), apply the conversions, and then render into the correct output. 

So, for example, if you initialized a flight on your iPhone from Bad Elf data and saved it, that should have saved in CSV format with ft/kts (which you can verify this by downloading in original format).  If you want a GPX version of that flight, you would download GPX and declare that the original units were knots and feet; that tells the system which conversion factor to apply to the CSV data to produce valid GPX.