Testing procedure for keyboard layouts

Previous tests used a collection of input files, together with a suite of programs to do word-based analysis.

There are several problems with this approach.

• I used multiple input texts because none of them had the correct English character letter frequency. The hope was that they would balance each other out, with a net result having the correct character frequency. However, it did not work out like that. So the tests were biased to a degree.
• That testing procedure was tedious and time-consuming, there were 300-400 layouts and dozens of input texts. I used iMacros on Firefox to semi-automate it, but it still took a lot of time. Then Firefox changed what extensions are allowed to do, and this approach became non-viable.
• For the word tests. it became apparent that there is conflict between Layout Theory and the English language. For example, one test involved counting words typeable on the home keys. However, some good layouts do not put the letter O on the home row, and then get a poor score in this test. Or, a layout may put T and H on the same hand to get nice rolls, but will then do poorly in a test that looks at hand alternation.

So I went back to the drawing board. Earlier this year I developed the concept of chained bigrams, which allows us to create input texts with the correct input character and bigram frequency. For the purposes of this exercise, I created three input texts:

• 1 MB of Chained English Bigrams
• 1 MB of Chained Code Bigrams
• 1 MB of Chained Proglish Bigrams, where proglish is an 80:20 mix of English and code.

Each layout was evaluated on KLAnext, my fork of Patrick Gillespie's Keyboard Layout Analyzer. My version fixes various bugs and has a modified scoring system.

I then calculate each layout's score relative to the best score on each test, as a percentage.
Then the weighted average for "Fingers" is ((4 × English) + Code + Proglish) / 6.

For the Words tests, I previously took an average of these tests:

• Home Key Words
• Home Block Words
• Angle Mod Home Block Words
• Easy Block Words
• Left hand words
• Right hand words
• Home Key Bigrams
• Home Key Trigrams
• Home Key Quadgrams
• Home Key Common words
• Same finger bigrams
• Fraser Street

For this cycle, I initially took out some of those and replaced them with

• Kleinblock words, using the key blocks proposed by Arno Klein (see Step 1)
• Hand alternation
• A different Same finger bigrams test, based on the Linux word list.

Further testing and thought eventually led to two changes:

1. Dump the Linux word list, and use the 30k most common words from my English corpus.
2. Only have one test:
   1. Same-finger bigrams, using the 30k words.

The Linux word list contains a great many words, many of which you are unlikely to type on a regular basis (e.g. extraterritorialities, acoustoelectrically, acetylsalicylates, hippopotomonstrosesquipedalian, pneumonoultramicroscopicsilicovolcanoconiosis, Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch). So using common words is a more realistic approach.

For the word tests, I count unique letters in each word rather than just words directly. Counting words treats long words the same as short words, and counting letters allows words like "kindnesses" to score higher than "kindness", even though it's the same letters.

I removed the Home Key Words test (popularised by Maltron) because layouts with home keys on the thumb have an unfair advantage. The test also penalises those layouts that opt to keep O and/or U and/or H off the home keys, because other factors are more important.

Each layout's score is then converted to a percentage of the highest score, as its "Words" score.

Finally, the overall average is (Fingers * 2 + Words) / 3.