Running Kotlin/Native unit tests on iOS Simulator
When developing Kotlin/Native solution one may need to run unit tests against a selected target platform to verify that the solution works correctly on all supported runtimes. Kotlin/Native already provides nice support for running tests on JVM, Android, Linux, Windows and macOS but it does not support iOS out of the box.
This story provides a quick-start guide on running Kotlin/Native unit tests on iOS simulator. We will first add some tests to a basic project and run them manually. Then we will see how to implement a simple Gradle task allowing running that process as a part of the build.
Initial project setup
You may find it easier to checkout the project of sample Github repository and follow the story.
Let's start simple. Our multiplatform solution is built as a simple Gradle-based project. Our starting directory structure looks like that:
.
├── KotlinNativeFramework
│ ├── build.gradle.kts
│ └── src
│ └── commonMain
│ └── kotlin
│ └── KotlinNativeFramework.kt
├── build.gradle.kts
├── gradle
│ └── wrapper
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew
├── local.properties
└── settings.gradle.kts
The major configuration is done inside KotlinNativeFramework/build.gradle.kts. Whole project is configured in a quite minimalistic variant allowing building for iOS.
While using Kotlin/Native typically we define targets separately for each supported architecture: arm64, armX64 respectively for iOS, watchOS and tvOS. With version 1.3.60, the kotlin-multiplatform plugin provides shortcuts that automate such a configuration: they let users create a group of targets along with a common source set for them with a single method. Therefore we use ios shortcut which defines targets for both iOS device and simulator architectures.
plugins {
kotlin("multiplatform") version "1.3.61"
}
repositories {
mavenCentral()
}
kotlin {
ios {
binaries {
framework {
baseName = "KotlinNativeFramework"
}
}
}
}
tasks.withType<Wrapper> {
gradleVersion = "5.3.1"
distributionType = Wrapper.DistributionType.ALL
}
Source code for Kotlin/Native solution contains one class KotlinNativeFramework exposing a method helloFromKotlin(), returning some nice greeting.
class KotlinNativeFramework {
fun helloFromKotlin(name: String) = "Hello from Kotlin, $name!"
}
In Kotlin/Native projects, running tests in a Gradle build is currently supported by default for JVM, Android, Linux, Windows and macOS. Other Kotlin/Native targets like JS or iOS need to be manually configured to run the tests with an appropriate environment, an emulator or a test framework. Apart from that limitation, the default build configuration already does a lot for the developer wanting to test his Kotlin/Native code.
As it could be observed in ./gradlew tasks output - our configuration, apart from already mentioned targets - generates some other useful tasks.
Build tasks
-----------
assemble - Assembles the outputs of this project.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
clean - Deletes the build directory.
...
linkDebugFrameworkIosArm64 - Links a framework 'debugFramework' for a target 'iosArm64'.
linkDebugFrameworkIosX64 - Links a framework 'debugFramework' for a target 'iosX64'.
linkDebugTestIosArm64 - Links a test executable 'debugTest' for a target 'iosArm64'.
linkDebugTestIosX64 - Links a test executable 'debugTest' for a target 'iosX64'.
linkReleaseFrameworkIosArm64 - Links a framework 'releaseFramework' for a target 'iosArm64'.
linkReleaseFrameworkIosX64 - Links a framework 'releaseFramework' for a target 'iosX64'.
metadataJar - Assembles a jar archive containing the main classes.
metadataMainClasses - Assembles outputs for compilation 'main' of target 'metadata'
We will focus on linkDebugTestIosX64 task, which generates a binary that targets iOS simululator. That binary is all we need as it is already ready to be deployed on iOS Simulator. It even contains debug symbols, so using the approach I have described in my previous story Xcode debugger could be attached to it.
Let's add some unit tests
Kotlin/Native project comes with some useful mechanics for source code organization called sourceSets. A Kotlin source set is a collection of Kotlin sources, along with their resources, dependencies, and language settings, which may take part in Kotlin compilations of one or more targets. Some source sets like commonMain, commonTest are created, configured and available by default.
Source set configuration can specify its dependencies by defining them in dependencies{...} closure. To start autoring our tests we will need to depend on test-common and test-annotations-common which provide convenient test annotations and assertions API.
Our dependency definition will look like that:
sourceSets {
commonTest {
dependencies {
implementation(kotlin("test-common"))
implementation(kotlin("test-annotations-common"))
}
}
}
After applying that changes, kotlin{...} block in our build.gradle.kts should look like the one below.
kotlin {
sourceSets {
commonTest {
dependencies {
implementation(kotlin("test-common"))
implementation(kotlin("test-annotations-common"))
}
}
}
ios {
binaries {
framework {
baseName = "KotlinNativeFramework"
}
}
}
}
Then, let's write some test code. First, we will create SampleTest.kt at KotlinNativeFramework/src/commonTest/src/main/kotlin. Then, we are going to write the actual test code there. It is going to be a simple test case with an assertion on returned value.
import kotlin.test.Test
import kotlin.test.assertEquals
class SampleTest {
@Test
fun testReturnValue() {
assertEquals("Hello from Kotlin, Joe",
KotlinNativeFramework().helloFromKotlin("John"))
}
}
Let's run it!
As we already found out - binary, that is ready for testing could be easily generated ....
Let's then use simctl to check if it works. First, we would need to find out, what simulators we already have available in our system. We can use list switch.
xcrun simctl list
It should print the list of entries similiar to the ones below.
== Devices ==
-- iOS 12.2 --
iPhone 8 Plus (40FE8851-A2CB-45E4-9821-C46A7D9F9357) (Shutdown)
iPhone X (5DE07911-BE32-43AB-834D-607423A1E076) (Shutdown)
iPhone Xs (E6A910F0-F569-47D4-9F49-165676A35E0A) (Shutdown)
-- iOS 13.4 --
iPhone 8 (9EBC054E-8555-4B4B-9F8C-9872C6D1063C) (Shutdown)
iPhone 8 Plus (FB908D91-D738-4199-BF1C-B2DD5D13BD9E) (Shutdown)
Let's then pick one (let it be iPhone Xs with iOS 12.2) from the list and boot it using simctl command-line tool. In our case, the invocation will look like that
xcrun simctl boot E6A910F0-F569-47D4-9F49-165676A35E0A
Once the device is booted, the final step is launching the binary Gradle just generated.
xcrun simctl spawn E6A910F0-F569-47D4-9F49-165676A35E0A KotlinNativeFramework/build/bin/iosX64/debugTest/test.kexe
As the run is finished - we can see a nice table with tests summary and as we could expect - our test failed.
[==========] Running 1 tests from 1 test cases.
[----------] Global test environment set-up.
[----------] 1 tests from SampleTest
[ RUN ] SampleTest.testReturnValue
[ FAILED ] SampleTest.testReturnValue (0 ms)
[----------] 1 tests from SampleTest (6 ms total)
[----------] Global test environment tear-down
[==========] 1 tests from 1 test cases ran. (6 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 tests, listed below:
[ FAILED ] SampleTest.testReturnValue
It was expected, but actually the fix for that is pretty simple, so let's apply it.
import kotlin.test.Test
import kotlin.test.assertEquals
class SampleTest {
@Test
fun testReturnValue() {
assertEquals("Hello from Kotlin, John!",
KotlinNativeFramework().helloFromKotlin("John"))
}
}
Making things reusable
Now, we would need to re-run linkDebugTestIosX64 and launch the binary on the simulator using the commands above. Let's instead define a simple Gradle task wrapping these operations. Doing that will allow us to reuse the logic and make things simpler.
That is a nice use case for buildSrc, which is a Gradle location indented to help developers in extracting imperative build logic and encapsulating it into separate tasks. It is treated as an included build, which means that code in buildSrc is compiled before the rest of build scripts and is put in the classpath, so other tasks can later reuse entities defined in buildSrc. To get it working, let's create that simple file structure at the root of our project.
.
└── buildSrc
└── src
└── main
└── kotlin
└── SimulatorTestsTask.kt
The build.gradle.kts content we need could be as simple as the one in below fragment.
plugins {
`kotlin-dsl`
}
repositories {
jcenter()
}
Configured source tree at this point looks like that:
.
├── KotlinNativeFramework
│ ├── build.gradle.kts
│ └── src
│ ├── commonMain
│ │ └── kotlin
│ │ └── KotlinNativeFramework.kt
│ └── commonTest
│ └── kotlin
│ └── SampleTest.kt
├── build.gradle.kts
├── buildSrc
│ ├── build.gradle.kts
│ └── src
│ └── main
│ └── kotlin
│ └── SimulatorTestsTask.kt
├── gradle
│ └── wrapper
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew
├── kotlin-native-ios-tests.iml
├── local.properties
└── settings.gradle.kts
Now we can start authoring our task. It is going to take test binary to run and target simulator identifier, boot the simulator and spawn the binary process, so we may start defining it as follows
import org.gradle.api.DefaultTask
import org.gradle.api.tasks.Input
import org.gradle.api.tasks.InputFile
import org.gradle.api.tasks.TaskAction
open class SimulatorTestsTask: DefaultTask() {
@InputFile
val testExecutable = project.objects.fileProperty()
@Input
val simulatorId = project.objects.property(String::class.java)
@TaskAction
fun runTests() {
...
}
}
Then, we can focus on the task action itself. Gradle comes with useful Exec API (see docs) that allows executing command line processes, which we will utilize here. What is important in our solution, apart from mapping the commands we have previously executed to project.exec() invocations, we should also pay attention to shutting the booted simulator down when we finish using it.
@TaskAction
fun runTests() {
val device = simulatorId.get()
val bootResult = project.exec { commandLine("xcrun", "simctl", "boot", device) }
try {
println(device)
print(testExecutable.get())
val spawnResult = project.exec { commandLine("xcrun", "simctl", "spawn", device, testExecutable.get()) }
spawnResult.assertNormalExitValue()
} finally {
if (bootResult.exitValue == 0) {
project.exec { commandLine("xcrun", "simctl", "shutdown", device) }
}
}
}
The next thing to do is actually applying the SimulatorTestsTask in our build. We can do it in KotlinNativeFramework's build.gradle.kts. First, we need to access test binary. We can do it via accessing KotlinNativeTarget from kotlin.targets. Then, from its binaries we can get the reference for a debuggable test binary. Returned object is an instance of TestExecutable and exposes outputFile and linkTask properties. We are going to use them to define dependency on linkTask, so we will be sure that task that builds binary available as outputFile is actually invoked before the test-running task we have just defined. Additionally, we are making check task, which is a built-in verification task, dependent on runIosTests so our tests will be invoked with any other tests.
kotlin {
...
val testBinary = kotlin.targets.getByName<KotlinNativeTarget>("iosX64").binaries.getTest("DEBUG")
val runIosTests by project.tasks.creating(SimulatorTestsTask::class) {
dependsOn(testBinary.linkTask)
testExecutable.set(testBinary.outputFile)
simulatorId.set("E6A910F0-F569-47D4-9F49-165676A35E0A")
}
project.tasks["check"].dependsOn(runIosTests)
}
When we run the task we have just defined using
./gradlew check
we can see that this time everything passed now 🎉
[==========] Running 1 tests from 1 test cases.
[----------] Global test environment set-up.
[----------] 1 tests from SampleTest
[ RUN ] SampleTest.testReturnValue
[ OK ] SampleTest.testReturnValue (0 ms)
[----------] 1 tests from SampleTest (0 ms total)
[----------] Global test environment tear-down
[==========] 1 tests from 1 test cases ran. (0 ms total)
[ PASSED ] 1 tests.
Where to go now?
That story presented the basic integration, which makes use of some assumptions related to simulator presence and state. However, simctl is a powerful tool allowing simulator management that could be explored further, especially if you plan to include your Kotlin/Native unit tests running on iOS simulator into your CI pipeline. It may be worth considering creating a fresh simulator instance to reduce the risk of tests flakiness caused by environmental issues. Also, you may want to check my previous story, where I touched various aspects of integrating Kotlin/Native framework with the Xcode project and debugging it on iOS devices.
Wrap up
To sum up, today we learned how to run Kotlin/Native unit tests on iOS Simulator. We have also touched the topic of integrating the run process with Gradle-based build. I hope that was a helpful, valuable and interesting read. Thanks for reading!