So, I spent a bit of time on the weekend playing a little more in-depth with the new UI Testing APIs in the Xcode 7 beta 2. Please note that this is beta software, so many of my observations may not be true for the final release or future betas.

The Good

UI testing feels like an effortless and natural part of the development process.

First off, let's start with the positive impressions. I think the greatest thing about the new UI Testing features is how integrated they are into the development process. Specifically, recording tests just by tapping, swiping and typing in the simulator, the ability to click on an element in the test editor and choose alternate queries that can describe the same element a different way, and the step-by-step logging of failing tests plus screenshots right inside Xcode are all really slick, and make UI testing feel like an effortless and natural part of the development process.

Also positive is Apple's approach to defining elements as queries that are evaluated when needed, rather than pointers to actual objects. Besides playing very nicely with Swift (XCUIElements are never nil, they can only fail to resolve to a currently existing UI object, which means that XCUIElements don't have to be optionals and unwrapped for each use), this pattern also enables a very clean approach for determining when a UI object starts or ceases to be present, using the XCUIElement exists property. This makes it really easy to add convenience methods for waiting until an element exists, etc. as I played around with here:

https://github.com/daniel-hall/DHTestingAdditions

Lastly, the ease of triggering events from XCUIElements is wonderful. You can take essentially any object in the UIView hierarchy of your app and in a single message simulate a tap, long press, two finger tap, or even holding and then dragging to another element.

Overall, the initial concepts and implementations feel very smart and like a great foundation for UI Testing. Unfortunately, that foundation is missing some key parts of the structure above it.

The Bad

I'd have to say without a doubt that the worst deficiency in Xcode UI testing as it exists right now is the inability to get information about the UIView object that an XCUIElement references. For example, if I use an XCUIElement to refer to a UILabel on the screen, I can find out what its frame is, and what the text in the label is. But a short list of just some the things I can't find out about that UILabel are: if it's hidden or not, the font family or size, the number of lines it has, its alpha value, the text color or background color, etc.

Making a broad estimate, I would say that the majority of UI Tests that I would want to write cannot be expressed using Xcode UI Testing because of this one deficiency. I'm not sure why Apple only allows a fixed set of attributes (frame, title, isEnabled, label, identifier, accessibility value and elementType) to be queried, and why those attributes are missing such important properties like color, isHidden and control state, but I can only hope that this is a temporary situation.

It's particularly interesting to note that the advanced UI Testing example that was covered in the "UI Testing in Xcode"session at WWDC was supposed to eventually verify that the color of various elements changed to match the color of a button that was pressed. In fact, there seems to be no way to accomplish this using Xcode UI Testing at this time.

UI Testing targets do not provide any access to the classes and objects in the application at runtime.

Perhaps more concerning than the examples of missing properties is that fact that the architecture and implementation of Xcode UI Testing seem to indicate that there is no intention of letting developers message or query the objects in their view hierarchies directly. So, assuming I created a custom control that had a "percentProgress" property, it seems very unlikely that I would ever be able to verify that property having a certain value following some set user interactions. Why? Well, not only is there no API in XCUIElement that allows you to get a reference to the actual object in the view hierarchy and message it, but UI Testing targets do not provide access to the classes and objects in the application at runtime.

The Unexpected

Yep, that was the sort unexpected, M. Night Shyamalan style twist for me. And it took a lot of head scratching before I realized what was happening. You see, normally, in a test target for unit tests, you can instantiate or get references to objects in the application you are testing, and message them, read property values, etc. in order to set up and verify the desired states and results. This is because the application itself hosts the tests target bundle, and the tests are executed from within the context of the running application.

However, try to instantiate a class from your running application in a UI Test and you'll get a build error because classes from your target application are not accessible in your test target. You could explicitly include your .m or .swift files in the test target to make that error go away, but when you instantiate those classes in the test, they are not being instantiated by or in the context of your actual application, but only the test process itself (not useful).

Going a step further, trying to access the view hierarchy of the running application is also futile, because [UIApplication sharedApplication].windows is empty (because you are getting the shared application for the test process itself, not the running app). Ditto for trying to reach the app delegate.

It's also worth noting that this approach means that break points in your application will not trigger during UI Tests, and you will not see any console logs from your application while UI Tests are running. (Update: As of Xcode 7 beta 4, breakpoints in your running application will now trigger during UI testing! Thanks to Joe Masilotti for pointing that out in the comments.)

This makes a sort of sense, since the UI Test target is running in a separate process and interacts with the main application from the outside, just like a real user poking it with a finger. However, some aspect of this limitation is not technical but clearly by design, since the UI Test target itself is able to walk the view hierarchy, execute queries, and message into the target application to synthesize events. So there would certainly be a way to expose similar access to developers, but Apple has for now decided not to. It's also worth noting that this approach means that break points in your application will not trigger during UI Tests, and you will not see any console logs from your application while UI Tests are running. (Update: As of Xcode 7 beta 4, breakpoints in your running application will now trigger during UI testing! Thanks to Joe Masilotti for pointing that out in the comments.) The benefit I assume is coming in trade for that blindfold is that your UI Tests can run against release / non-debug builds of your application, which is the absolute best approximation of the real user experience.

But all of this means simply that there is no API provided to interact with or get information from the application under test except through the XCUIApplication class (which only allows you to launch or terminate the application) and through the XCUIElement class (which only gives access to elementType, frame, label, title, accessibility value and isEnabled).

This would be less concerning if I could see some hints as to how Apple might allow access to read arbitrary properties from an XCUIElement, but the whole approach of having a protocol (XCUIElementAttributes) with a fixed list of pre-specifed properties that can be accessed (those named above) is pretty rigid and doesn't suggest an likely path for a developer who might want to access a unique property on a custom UIView subclass.

It also means that there is a hard line between functional / unit tests and UI tests, and there can be no using tools from Xcode UI Testing (like relaunching an app) with the sorts of API access you would want for integration and functional testing.

The Future

Despite the current shortcomings of UI Testing in Xcode (which again, makes it only able to handle a minority of the UI Tests I've previously written using KIF or Calabash), I'm quite optimistic about its future. There has clearly been a lot of good thought put into how UI Tests should work, and given the addition of some better methods to verify element state, I can even see how the total black box approach to the target application is a great practice for writing tests that truly simulate the user experience and nothing else.

The query engine for now seems pretty robust and intelligent, and it seems like getting more access to the represented UI Views isn't technically infeasible, but simply a matter of Apple's engineers working out an approach and API they like.

In its current state, I don't feel Xcode UI Testing is flexible or capable enough to replace any of the other solutions that currently exist. But with a relative small number of changes / fixes it could easily catch up with and surpass other UI automation and testing tools. If you are not using any UI Testing at all currently, then it's easy enough to crank out a few tests in Xcode 7 and cover some useful cases. But it won't be long before you run headfirst into the frustration of being unable to verify very much of what matters in your app.

So for now, I'm not planning to use the new UI Testing features of Xcode in my own projects, since their limitations and deficiencies seem to outweigh their otherwise exciting benefits. But, I'll be keeping track of the updates to Xcode UI Testing with great enthusiasm and posting about further developments as they arise.