This is the first article in a short series about the different components of AXElements. AXElements has been rewritten to be much more modular and now various components may be used individually. The keyboard event generator was the first component to be polished and released as its own gem.

Event generation is exposed via the Accessibility::String mix-in. Using the mix-in, a simple string of human readable text is taken and turned it into a sequence of events which may be fed to one of the keyboard event posting APIs provided by OS X.

To get started you will need to have a working Ruby 1.9 implementation—I’ve tested this with MRI 1.9.3 and MacRuby (for MacRuby you will need a nightly build). Then you can install the AXTyper gem:

1

gem install AXTyper

UPDATE: The gem has been reported to not compile if your Ruby was compiled with GCC. The preferred solution would be for you to recompile Ruby with Clang or use MacRuby. 😝 I’m not a pro with C extensions, but I’ll look into whether or not I can force the extension to compile with Clang for you people who do not want to, or cannot, recompile with Clang.

UPDATE 2: AXTyper-0.7.4 now checks if Ruby’s CC is Clang. If it is not Clang then it tries to find Clang in your $PATH. A proper error message is given if Clang cannot be found. Clang is binary compatible with GCC, so this should be safe to do.

The quickest demonstration of event generation is through irb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'accessibility/string'
=> true
irb(main):003:0> include Accessibility::String
=> NSObject
irb(main):004:0> events = keyboard_events_for "Hello, #{ENV['USER']}."
=> [[56, true], [38, true], [38, false], [56, false], [2, true],
 [2, false], [35, true], [35, false], [35, true], [35, false],
 [1, true], [1, false], [13, true], [13, false], [49, true],
 [49, false], [46, true], [46, false], [31, true], [31, false],
 [0, true], [0, false], [4, true], [4, false], [0, true], [0, false],
 [14, true], [14, false]]
irb(main):005:0>

And now you have some events! Events are number/boolean tuples; the number is a key code and the boolean is the key state; there are more details regarding the key code, but I’ll get to that later.

Now that we have some events, we should figure out what to do with them since they are not of much use on their own. The most useful, and easiest, thing to do is post them to the system. This of course comes with a caveat: you will actually be causing the keys to be typed out to the system and the front most application will receive the events.

Be careful of the string you give to the generator!

The interface for posting events is different depending on if you are using MRI or MacRuby. With MacRuby, events can be posted natively in Ruby code, but for this demonstration the C extension provided by AXTyper includes a singleton method, KeyCoder.post_event, which takes a single event and posts it to the system. Using this method we can post our events:

1
2
3
4
5
6
7
8

irb(main):005:0> events.each do |event| KeyCoder.post_event event end
Hello, mrada.=> [[56, true], [38, true], [38, false], [56, false],
 [2, true], [2, false], [35, true], [35, false], [35, true],
 [35, false], [1, true], [1, false], [13, true], [13, false],
 [49, true], [49, false], [46, true], [46, false], [31, true],
 [31, false], [0, true], [0, false], [4, true], [4, false], [0, true],
 [0, false], [14, true], [14, false]]
irb(main):006:0> Hello, mrada.

You’ll notice that typing begins immediately when the post_event method is called. Typing is also really fast; in fact, it has to be slowed down for practical use.

Let’s add a wrapper to do generation and posting together, we’ll alsoadd a small time buffer so that you get a chance to release the return key after entering a command in irb:

1
2
3
4
5
6

def type string
  sleep 0.1
  keyboard_events_for(string).each do |event|
    KeyCoder.post_event event
  end
end

Now let’s try something a bit more complicated:

1

type "\\CONTROL+l `say 'ZOMG'`\n"

A Little More Depth

So what’s going on here? On the AXElements side of things there is just a basic lexer and generator. The lexer tokenizes the string input and the generator takes the tokens and generates one or more pairs of event tuples. The interesting detail is that a set of escape sequences for control keys has been added.

Lexing

Strings fed to the lexer support an extra set of escape sequences for the control keys like Control, Option, Command, etc.. The last example above used the "\\CONTROL" escape sequence to represent the left control key. Other than those special sequences, string formatting is straight forward and you can use all the letters, numbers, and symbols that you would in any other string. Uppercasing letters is automatically handled for you and so are all of the symbols that you would have to hold down the shift or option key in order to type. White space and line breaks in strings will get turned into tabs, spaces, and return/enter, and delete key presses appropriately, but you can also use "\t", "\s", "\r", "\n", and "\b" as you would in any other string.

Once the lexer is done lexing, its output is fed into the generator.

Event Generation

The event generator inspects each token and figures out what events need to be generated in order type the token.

As mentioned above, events are number/boolean tuples. They are fed to the CGEventCreateKeyboardEvent() function that is used in the above examples. The function takes a key code and a key state as parameters and so each event pair is simply what is required to be passed to that function.

Key codes are a mapping of numbers to keys on the keyboard; some mappings are static, such as the control keys; and some mappings are dynamic based on the keyboard layout. Since a code refers to a physical key and not the particular symbol they represent, upper case letters require the generation of events for pressing either the shift key before hitting the lower case letter.

The key state is simply a boolean value with true meaning that the key is in the keydown state and false meaning that the key is in the keyup state. This makes it possible to hold down the shift key and then press the "a" key to generate an "A"; it also makes it possible to simulate hotkeys and symbols.

Symbols

Symbols such as "!", "∑", and "]" are all supported by the event generator. Any symbol that you can type directly using one or more keys should Just Work™. Some contrived examples might look like this snippet:

1
2
3

type "@hash = { a: '∑', b: '™' }\n"
type "@hash.merge! { c: '£', d: '¢' }\n"
type "@hash.inspect\n"

A caveat to this is that some symbols can be on the keyboard twice, the obvious cases are the mathematical operations and numbers. The event generator does not use the keypad for plain symbols in strings. To use the keypad keys specifically you will need to use custom escape sequences.

Escape Characters

As mentioned above, standard escape codes (e.g. "\n", "\t") all still work, even using "\b" as the delete key. On top of the built in escape sequences, the lexer and generator have added an additional set of escape sequences for control keys and other static keys. The full list is located in the documentation here, but the naming convention should be obvious enough after a small demonstration.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

type "\\CONTROL"
# or
type "\\CTRL"

# using built in escape codes
type "The cake is a lie\b\b\b\b\b\b delicious"

# Use the CMD+a hotkey and then start typing
type "\\COMMAND+a I just deleted everything, didn't I?"

# type one key and then the other
type "\\1 \\+ \\2 \n"

# type all the keys as a hot key combination
type "\\CMD+\\SHIFT+s"
# or
type "\\CMD+S"

# type out a backslash
type "\\"

# type out a fake command key
type "\\CAKE"

There are two rules to using the custom escape sequences. First, a custom escape sequence must be terminated by an empty space or the end of the string.

Hot Keys

The second rule is an exception to the first rule; a custom escape sequence can end with a "+" if it is being used in a hot key combination and will be immediately followed by another key in the combination. In previous examples we combined two and three keys in order to make a combination, but an upper limit on the number of keys is not defined by the event generator.

Sending Events To A Specific Application

Though the easiest way to post events to the system was to use CGEventPost(), there are also APIs for posting events to a specific application regardless of which application is focused.

The API for doing this is located in the OS X Accessibility headers, but requires you to provide a reference to the application where you would like to post events. Using the accessibility APIs is a little tricky at first, especially in Ruby.

I’ll cover Accessibility API basics in the next part of this series: "Accessibility: From References To Objects".