Event generation is exposed via the Accessibility::String mix-in. Using the mix-in, a simple string of human readable text is taken and turned it into a sequence of events which may be fed to one of the keyboard event posting APIs provided by OS X.

To get started you will need to have a working Ruby 1.9 implementation—I’ve tested this with MRI 1.9.3 and MacRuby (for MacRuby you will need a nightly build). Then you can install the AXTyper gem:

1

gem install AXTyper

UPDATE: The gem has been reported to not compile if your Ruby was compiled with GCC. The preferred solution would be for you to recompile Ruby with Clang or use MacRuby. 😝 I’m not a pro with C extensions, but I’ll look into whether or not I can force the extension to compile with Clang for you people who do not want to, or cannot, recompile with Clang.

UPDATE 2: AXTyper-0.7.4 now checks if Ruby’s CC is Clang. If it is not Clang then it tries to find Clang in your $PATH. A proper error message is given if Clang cannot be found. Clang is binary compatible with GCC, so this should be safe to do.

The quickest demonstration of event generation is through irb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'accessibility/string'
=> true
irb(main):003:0> include Accessibility::String
=> NSObject
irb(main):004:0> events = keyboard_events_for "Hello, #{ENV['USER']}."
=> [[56, true], [38, true], [38, false], [56, false], [2, true],
 [2, false], [35, true], [35, false], [35, true], [35, false],
 [1, true], [1, false], [13, true], [13, false], [49, true],
 [49, false], [46, true], [46, false], [31, true], [31, false],
 [0, true], [0, false], [4, true], [4, false], [0, true], [0, false],
 [14, true], [14, false]]
irb(main):005:0>

And now you have some events! Events are number/boolean tuples; the number is a key code and the boolean is the key state; there are more details regarding the key code, but I’ll get to that later.

Now that we have some events, we should figure out what to do with them since they are not of much use on their own. The most useful, and easiest, thing to do is post them to the system. This of course comes with a caveat: you will actually be causing the keys to be typed out to the system and the front most application will receive the events.

Be careful of the string you give to the generator!

The interface for posting events is different depending on if you are using MRI or MacRuby. With MacRuby, events can be posted natively in Ruby code, but for this demonstration the C extension provided by AXTyper includes a singleton method, KeyCoder.post_event, which takes a single event and posts it to the system. Using this method we can post our events:

1
2
3
4
5
6
7
8

irb(main):005:0> events.each do |event| KeyCoder.post_event event end
Hello, mrada.=> [[56, true], [38, true], [38, false], [56, false],
 [2, true], [2, false], [35, true], [35, false], [35, true],
 [35, false], [1, true], [1, false], [13, true], [13, false],
 [49, true], [49, false], [46, true], [46, false], [31, true],
 [31, false], [0, true], [0, false], [4, true], [4, false], [0, true],
 [0, false], [14, true], [14, false]]
irb(main):006:0> Hello, mrada.

You’ll notice that typing begins immediately when the post_event method is called. Typing is also really fast; in fact, it has to be slowed down for practical use.

Let’s add a wrapper to do generation and posting together, we’ll alsoadd a small time buffer so that you get a chance to release the return key after entering a command in irb:

1
2
3
4
5
6

def type string
  sleep 0.1
  keyboard_events_for(string).each do |event|
    KeyCoder.post_event event
  end
end

Now let’s try something a bit more complicated:

1

type "\\CONTROL+l `say 'ZOMG'`\n"

A Little More Depth

So what’s going on here? On the AXElements side of things there is just a basic lexer and generator. The lexer tokenizes the string input and the generator takes the tokens and generates one or more pairs of event tuples. The interesting detail is that a set of escape sequences for control keys has been added.

Lexing

Strings fed to the lexer support an extra set of escape sequences for the control keys like Control, Option, Command, etc.. The last example above used the "\\CONTROL" escape sequence to represent the left control key. Other than those special sequences, string formatting is straight forward and you can use all the letters, numbers, and symbols that you would in any other string. Uppercasing letters is automatically handled for you and so are all of the symbols that you would have to hold down the shift or option key in order to type. White space and line breaks in strings will get turned into tabs, spaces, and return/enter, and delete key presses appropriately, but you can also use "\t", "\s", "\r", "\n", and "\b" as you would in any other string.

Once the lexer is done lexing, its output is fed into the generator.

Event Generation

The event generator inspects each token and figures out what events need to be generated in order type the token.

As mentioned above, events are number/boolean tuples. They are fed to the CGEventCreateKeyboardEvent() function that is used in the above examples. The function takes a key code and a key state as parameters and so each event pair is simply what is required to be passed to that function.

Key codes are a mapping of numbers to keys on the keyboard; some mappings are static, such as the control keys; and some mappings are dynamic based on the keyboard layout. Since a code refers to a physical key and not the particular symbol they represent, upper case letters require the generation of events for pressing either the shift key before hitting the lower case letter.

The key state is simply a boolean value with true meaning that the key is in the keydown state and false meaning that the key is in the keyup state. This makes it possible to hold down the shift key and then press the "a" key to generate an "A"; it also makes it possible to simulate hotkeys and symbols.

Symbols

Symbols such as "!", "∑", and "]" are all supported by the event generator. Any symbol that you can type directly using one or more keys should Just Work™. Some contrived examples might look like this snippet:

1
2
3

type "@hash = { a: '∑', b: '™' }\n"
type "@hash.merge! { c: '£', d: '¢' }\n"
type "@hash.inspect\n"

A caveat to this is that some symbols can be on the keyboard twice, the obvious cases are the mathematical operations and numbers. The event generator does not use the keypad for plain symbols in strings. To use the keypad keys specifically you will need to use custom escape sequences.

Escape Characters

As mentioned above, standard escape codes (e.g. "\n", "\t") all still work, even using "\b" as the delete key. On top of the built in escape sequences, the lexer and generator have added an additional set of escape sequences for control keys and other static keys. The full list is located in the documentation here, but the naming convention should be obvious enough after a small demonstration.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

type "\\CONTROL"
# or
type "\\CTRL"

# using built in escape codes
type "The cake is a lie\b\b\b\b\b\b delicious"

# Use the CMD+a hotkey and then start typing
type "\\COMMAND+a I just deleted everything, didn't I?"

# type one key and then the other
type "\\1 \\+ \\2 \n"

# type all the keys as a hot key combination
type "\\CMD+\\SHIFT+s"
# or
type "\\CMD+S"

# type out a backslash
type "\\"

# type out a fake command key
type "\\CAKE"

There are two rules to using the custom escape sequences. First, a custom escape sequence must be terminated by an empty space or the end of the string.

Hot Keys

The second rule is an exception to the first rule; a custom escape sequence can end with a "+" if it is being used in a hot key combination and will be immediately followed by another key in the combination. In previous examples we combined two and three keys in order to make a combination, but an upper limit on the number of keys is not defined by the event generator.

Sending Events To A Specific Application

Though the easiest way to post events to the system was to use CGEventPost(), there are also APIs for posting events to a specific application regardless of which application is focused.

The API for doing this is located in the OS X Accessibility headers, but requires you to provide a reference to the application where you would like to post events. Using the accessibility APIs is a little tricky at first, especially in Ruby.

I’ll cover Accessibility API basics in the next part of this series: "Accessibility: From References To Objects".

A Little History

jiraSOAP is my first real ruby project and it was how I learned ruby basics. My earlier experiences with ruby were small scripts to access the bug tracker at work and I was using a gem called jira4r to communicate with the bug tracker. This gem was limited in a few ways that became very irritating. It had no APIs from JIRA 4.x; the API it did expose was ugly and inflexible; and the gem does not run on Ruby 1.9 and thus could not run on MacRuby, which is a technology that my boss suggested looking into.

At the time, JIRA was beginning to develop a REST API, but it was pre-alpha quality. The available resources were limited and read-only. On the other hand the still supported, but not actively developed, SOAP interface had APIs for almost everything.

Enter handsoap, a gem that promised an easy way to make a custom wrapper around a SOAP API that would be very fast. As it turns out, the speed part was overly optimistic, but I’ll get to that later. The important feature was that I could customize the API that I provided instead of simply generating something from the WSDL.

After a couple of weeks of playing with the library (it was far from my primary concern at work), I had a working predecessor to jiraSOAP. At this point I asked my boss if we could open source my work; that way I would work on it more after hours and other people could use and hopefully contribute to the project as well.

jiraSOAP’s Early OSS Days

The early days were hectic, I was pretty much rewriting jiraSOAP for each release as I learned new things. I even had a silly contribution policy warning people about contributions being unlikely to apply cleanly.

Part of the problem was that the recommended practices for writing the SOAP client were about as vague as an East Texas software patent, or at least the step about refactoring was. I can’t blame handsoap for that problem, it was my first real ruby project and I was still figuring a lot things out.

The logic for building messages was pretty clear and simple, but the the parsing logic required a relatively large amount of code from me. It was the parsing logic that was changing often due to refactoring with the goal of being cleaner and faster.

How Slow Was It?

handsoap claims to be faster because it can use C extensions for HTTP communication and XML parsing, they even supply some benchmarks in the README, though they qualify the benchmarks as being a little unfair.

I guess the unfair part is that those numbers were not jiving with my personal experience; everything felt much slower than advertised. When I looked into it, I found that their benchmark code didn’t actually do any parsing!

Inspired by "ZOMG WHY IS THIS CODE SO SLOW?", I began my own investigation into why my gem was slow. Though, in my case I was working with a smaller and more sane code base.

Step 1 is figuring out what to measure and then step 2 is making the baseline measurements. Since I supplied most of the parsing logic, I figured that I should measure parsing time. I chose to measure parse time for Issue objects since it is the most common type of object to work with; but it is also the largest type, weighing in at 21 attributes, including collections of other data structures that need to be parsed.

Before I get to how the measurements were made, I should mention that when I originally performed benchmarks that I was not very rigorous about collecting data. For the sake of this blog post I went back in time and benchmarked things properly. I say properly, but I really mean "good enough" for my purposes.

Since I wanted to measure just parsing time, I needed to change the jiraSOAP code base a little. The gist shows how to modify the jiraSOAP 0.9 code, but the changes needed for an older version are almost the same:

As a baseline I measured jira4r-jh, a recent fork that includes an API that I wanted to use, but the changes needed for measurement happened in soap4r, a dependency:

Instead of using benchmark or minitest, I opted to just write a small script:

The oddity there is that I average a larger sample set for a smaller number of issues because I am trying to avoid outliers from the GC kicking in. I didn’t want to stop the GC as that seemed too unrealistic, but I did give a 1 second sleep period between benchmarking runs. Since I am trying to show the difference in performance between versions, I don’t think it will matter as long as I am consistent and the numbers don’t end up being very close together.

Due to differences in how I patched the projects, the benchmark script for jira4r is a little different, I won’t embed it here, but it’s in the gist with everything else.

If you are familiar with the JIRA version of SQL, you can see that I am always using the same set of issues for each test run; and the 1000 issues that were used take up 4.4 MB on disk when dumped to a file. Benchmarks on my 2.4 GHz C2D MacBook Pro running OS X Lion. jiraSOAP benchmarks were run using Ruby 1.9.3preview1 and MacRuby 0.10; jira4r benchmarks were run using Ruby 1.8.7-p249.

The Numbers

WHAT. THE. FRAK?! jiraSOAP was more than 2x slower than jira4r with CRuby; with MacRuby the the performance really bad—a whopping 7x slower than jira4r and more than 3x slower than the equivalent code running on CRuby.

Experienced MacRuby users will probably suggest that such low performance is a result of lots and lots of memory allocations—programs that allocate a lot of memory will usually be slower on MacRuby. Using that intuition I began tracing through the parsing logic and found that the problem was how each Issue was being initialized.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

  def initialize_with_xml_fragment(frag)
    super frag
    @key, @summary, @description, @type_id, @status_id,
    @assignee_name, @reporter_name, @priority_id, @project_name,
    @resolution_id, @environment, @votes, @last_updated, @create_date,
    @due_date, @affects_versions, @fix_versions, @components,
    @custom_field_values, @attachment_names =
      frag.nodes( ['key',                 :to_s],
                  ['summary',             :to_s],
                  ['description',         :to_s],
                  ['type',                :to_s],
                  ['status',              :to_s],
                  ['assignee',            :to_s],
                  ['reporter',            :to_s],
                  ['priority',            :to_s],
                  ['project',             :to_s],
                  ['resolution',          :to_s],
                  ['environment',         :to_s],
                  ['votes',               :to_i],
                  ['updated',             :to_date],
                  ['created',             :to_date],
                  ['duedate',             :to_date],
                  ['affectsVersions/*',   :to_objects, JIRA::Version],
                  ['fixVersions/*',       :to_objects, JIRA::Version],
                  ['components/*',        :to_objects, JIRA::Component],
                  ['customFieldValues/*', :to_objects, JIRA::CustomFieldValue],
                  ['attachmentNames/*',   :to_ss] )
  end

A quick glance at the code will tell you that it has to dup each array and each element of the array for each call. But wait, it gets worse! Those strings are used in an the XPath query which is run on the same XML fragment for each attribute. The symbol elements are then used as the name of the transformer to be used on the XML node’s inner text; the data transformers could be simple and just take the string from the inner text, or end up triggering the initialization of more objects that needed to be parsed (#to_objects), or it could initialize an array of Nazi soldiers (#to_ss).

So for each issue, the virtual machine was constantly reallocating a lot of constant data, and re-walking the XML fragment for each XPath query. That makes two separate problems that should be fixable and which will undoubtedly improve performance and get rid of some repetitive code.

Plan of Attack

To fix iterating through the XML fragment for each attribute, I had to stop using XPath. While XPath is easier to use it was a bad fit in the first place since data from every node of the fragment was being used. I’d say that putting XPath in the recommended steps without a caveat was a bit misleading.

Some quick research revealed that the best choice for this problem was a pull parser since it only iterates over the fragment once. The important thing to note here is that this change will cause the logic to switch around—instead of getting each attribute to ask for some XML it will now get each XML node to ask for an attribute. What makes this faster is that asking for a node required the use of XPath whereas asking for an attribute could be as simple as a hash look up.

Given that the XML should be highly structured, I could have iterated through both the XML nodes and an array with the attribute names to just match the nodes and attributes up; however I was not to sure on the ordering of nodes with respect to sub-classing, perhaps it was in alphabetical order, but I would have to go through the WSDL for each version of JIRA that I wanted to support. Since I didn’t want to do that took a potential performance hit and used a hash where the node name corresponded to a key.

Using this new strategy, the parsing logic was totally generic and I refactored initialization of all XML entities to a common method:

1
2
3
4
5
6
7

  def initialize_with_xml frag
    attributes = self.class.parse
    frag.children.each { |node|
      action = attributes[node.name]
      self.send action[0], (node.send *action[1..-1]) if action
    }
  end

Each class had its own parse map, and you can see from the code snippet. By declaring attributes using a data-mapper style of class method, the maps were built up for each class. The interesting part was getting inheritance working for parse maps; at first I had the obvious solution which looked a bit hacky but then I learned about some rarely used methods in ruby.

Ruby’s Less Frequently Used Metaprogramming Callbacks

Class#inherited, Module#included, and Module#extended are callbacks which are executed each time a Class/Module is subclassed, included, or extended. You can override the default implementation, which is an empty method, to provide custom inheritance/mixin rules.

This turns logic that looks like this:

1
2
3
4
5

  def property name, attr, transformer
    # ...
    @parse ||= superclass.parse.dup
    # ...
  end

into a one line method like this:

1
2
3

  def inherited subclass
    subclass.parse = @parse.dup
  end

Which I think is much nicer since it separates the logic of inheriting a parse map from adding a new property. It is also faster since it does not need to check if the value is already initialized which is a savings of a couple hundred checks at boot time.

I found out about this callback when I was going over the HotCocoa::Behaviors module, but I have since also found out that it is the technique used by minitest in order to keep track of which classes have tests.

Jumping Back To The Numbers

With the new parsing logic, how do the two gems stack up?

About a 2x improvement! CRuby jiraSOAP is just a bit slower than jira4r, and MacRuby jiraSOAP is about as fast as CRuby jiraSOAP was before the change.

Given the performance gap, I still suspected the issue is that there are too many allocations somewhere, but probably not in my code anymore. That is when I started looking into handsoap’s source code to see what goes on when I parse XML.

It turns out that their parsing abstraction was the gold mine I should have been looking for earlier. Specifically, their method for getting the inner text of an XML node, #to_s.

That method does a bit of work to handle cases that are not relevant to jiraSOAP, but that detail is very minor when compared to calling Node#serialize in nokogiri which ends up unpacking two options hashes before it gets to the point of extracting the inner text, which then returns to handsoap so that it can run 5 chained #gsub calls.

The real kicker is that all the data transformers call#to_s first to get the nodes inner text. For an issue with 21 attributes, including nested structures, that is at least 21 calls to #to_s which allocates, among other things, the following:

• #to_s
  • a Hash for options with a Symbol to String key-value pair (3)
  • 5 gsub calls with literal String arguments that need to be duped (15)
• #serialize
  • a String to setup the encoding information (1)
  • a StringIO to buffer the inner text into the string that was setup before (1)
• #write_to
  • a String will have to be duped for the indent_text option in this case (1)
  • a SaveOptions object (1)
  • a String that will be created by indent_text * indent_times (1)

That is at least 23 unneeded allocations per call of #to_s, which is 483 allocations per issue object and 483,000 allocations for the 1000 issue sample.

Keep in mind that that number is for the minimum number of attributes; an issue has two arrays of Version objects, one for affected versions and one for fix versions, and each version has 6 attributes; issues also have an array of CustomFieldValue objects with 3 attributes each; and issues also have an array of attachment name strings. Realistically, most issues will have an affects version and probably a fix version; many databases also use custom field. 483,000 allocations is actually a conservative estimate and the real number of wasteful allocations is is well over 500,000.

Fixing The Problem

Fixing things in handsoap was a possibility, but I had already opened a pull request in which the handsoap devs were responding too infrequently. I decided that handsoap was unfairly allocated some resources that I wanted and I took upon myself to fix the problem by liberating the oppressed methods—I went freedom patching.

As mentioned earlier, I didn’t need code to handle any of the special cases. I also think that calling gsub to swap out escaped characters is a waste since the data will end up in an HTML document more often than not.

After checking, I found that I could just call Node#content to get the inner text of an XML node. That method doesn’t do any of unnecessary work that I outlined earlier, it doesn’t even enforce an encoding.

Newest Numbers

How do the performance numbers look after bypassing the extraneous code paths of handsoap?

Really good! jiraSOAP on CRuby now runs so much faster that the scaling for the graphs is all messed up. Even jiraSOAP on MacRuby is now faster than jira4r.

I also haven’t had any bugs logged related to encoding problems or escaped characters, so I think this is a big win. I even went and benchmarked past 1000 issues.

How Does It Scale?

jiraSOAP now parses ~2250 issues/s, which is ~10MB/s, which I think is pretty good. It also looks like the new lazy-sweep GC changes in ruby 1.9.3 kick in somewhere between 3000-4000 issues which is very cool.

The parsing can still get faster without resorting to C (nokogiri excepted); there are other solutions I have not tried yet because I haven’t had the time lately.

One such solution would be to cache the XML fragment and only parse it on demand. I would have to go back to XPath in this case, but since parsing is not done until needed it will get the standard laziness trade off of maybe never having to parse the XML and saving potentially many allocations.

Benchmarking such a change would be more challenging since it would need to take into account a much larger sample of the different ways in which people are using jiraSOAP; how man attributes are used normally and how often?

Then there is the memory-time trade off to consider, the XML document is, at some level, a very large string, which is not nearly as memory efficient as the structures the data would be parsed into and would be expensive to keep around for a long period of time.

The lack of understanding on how other people use the gem is the reason why it would be difficult to tell whether or not it would be faster in the general case or at least worth the memory trade off. I know at least one person was using the gem as part of a rails app, but I don’t know if it was a main component or simply a way to log bugs when errors occurred.

I could at least go as far as to find the threshold at which it becomes worthwhile by actually implementing the change, but I don’t have the time right now and at this point the performance bottleneck is probably not parsing anymore.

Next Steps

The real performance bottleneck now is the use of net/http for the networking layer. With CRuby you can easily tell handsoap to use a different back end for HTTP communication. curb is handsoap’s recommended back end and is much faster than net/http, but it is not compatible with MacRuby which is why I changed the default to net/http.

If you are using CRuby then you are better off switching to curb or httpclient after loading jiraSOAP:

1
2
3
4

  require 'jiraSOAP'
  Handsoap.http_driver = :http_client
  # or
  Handsoap.http_driver = :curb

You can also use eventmachine with handsoap, but it will not work with jiraSOAP out of the box, you would need to reimplement the 4 private methods in lib/jiraSOAP/api.rb.

In the future I think I will be able to release the gem once for each RUBY_ENGINE so that each ruby can have proper defaults. It would also be nice to have a MacRuby specific back end that uses Cocoa for the networking layer.

Though, by this time next year I expect that the JIRA REST API will have caught up to the SOAP API and jiraSOAP will become obsolete. So I don’t think I will be spending too much more time on this project. Besides, jiraSOAP now implements 99% of the API that I care about and the implementation is simple enough and stable enough to have other APIs easily added by users who want them.

I’ve been thinking about starting a blog for a few months now, but I could never find a topic that I wanted to publicly write about.

Possibilities:

• announcements of things that I am working on
• interesting tech that I am playing with
• topics related to school work

I will probably make a blog post about my jiraSOAP gem in the next few days since I am releasing version 0.9 tonight.

Hopefully I don’t pollute the internet too much.