Goals, Projects and Tests

Goals

This initial post is a scattering of content to help set the stage for future writing. It talks about some things I want to achieve, my current project spread, why I'm doing certain things, and touches on the early stages of design for my test framework. It's not incredibly well thought out as its primary purpose is to force me to write something and in turn commit me to publishing this content online.

Goals

It's been some time since I've had goals for personal projects or writing, largely due to time and availability. As my life stabilizes, I want to establish manageable goals towards which I can make slow progress and eventually achieve. Some of these will be concrete artifacts (e.g. a library) and others will be behaviors.

• Write a single blog post per month. These do not need to be technical in nature, but likely will be. I want to establish a routine where I write regularly.
• Create foundational Scala libraries for application development. I have several ideas that I want to help mature into useable libraries. I also want to use these libraries as the backbone of any applications that I create. Examples include a test framework, a configuration library, and a logging API.
• Create small applications based on my libraries. This will help me adopt a feedback cycle based on usage, and will also give me minor tools to use. An example is pasta, which I envision as a trivial persistent copy/paste store.
• Develop note taking software suitable for long-term use. Taking notes is a valuable tool in my life, and in the past couple of years I've also started using it for longer term knowledge management. I want performant tools that don't disrupt my writing and help facilitate knowledge retrieval.

Why?

Some of my goals - notably the libraries and note taking software - deserve some explanation. There are plenty of libraries that do what I'm building. There are endless note taking tools. Why build more? I think it's important to acknowledge that a sizeable chunk of the reason is: "Because I can, and I find it gratifying." Aside from this, I want to test design concepts rather than wait for someone else to employ them. I believe that I might be able to produce something useful that isn't identical to existing tools.

Projects

The projects I care about right now are foundational. I want to operate as much within a stack that I have designed as possible. I am explicitly more focused on my personal experience than I am on general (or any) outside adoption. These are some of the key components I am currently building:

Projects - Test Framework

Existing test frameworks overcomplicate tests in certain ways or don't support concepts that I care about (e.g. effect types). I've also found the reporting capabilities to be lackluster. I envision a test framework that grants power in its simplicity and can be easily used outside of a build tool.

Projects - Logging

In JVM land, why do loggers so often assume static instantiation? Why are globals a requirement? Why is security of logged information not a concern? Why is a data-oriented approach not supported? Why is context restricted to thread- oriented tools like the MDC? Why are files the defacto configuration mechanism? I have ideas for logging that stem from real-life production environments in the health care domain, and I believe that Scala is an excellent language in which to trial them.

Projects - Configuration

Configuration libraries are numerous. What can I offer? One major concept is that of a configuration digest that tracks what is loaded (or was not!) and how it is used. I also think that avoiding the notion of "a file" as the base construct can be useful. In general I find that in practice, application startup is often ignored or hand-waved away. Improving the configuration experience and reporting on it can help ease the debugging experience and ease the ability of engineers to maintain an application over time. I believe that I can improve the experience and increase the value of well-organized configuration.

Tests

I have found a number of things lacking in test frameworks that I have used. Recently, especially, I have found that detailed reporting is either not available or difficult to achieve. I want to build my own test framework to test my ideas and also to learn about the challenges that come with building one.

It's worth keeping in mind that I'm coming to the table with my particular experience and biases: I've been working primarily on the JVM and primarily with Scala. I've had experiences with too many low-value tests, excessive mocking, poor result management, and more that influence my views.

Complexity

Many test frameworks introduce unnecessary (in my eyes!) complexity. This is a very subjective topic, and I suspect that many will find my views perhaps too reductionist. Some things that I have found that feel excessive:

• Too many ways to do one thing. ScalaTest (which I have used extensively, and successfully) is notorious for this.
• Many options for assertions. Reducing to a basic assert with perhaps a single alternative for exceptions is desirable.
• Nested or otherwise complex groupings of tests. Each class can represent a single, flat set of tests.

Test Identification

What is a test? Usually I've found a test to be a name, perhaps combined with a class or source file. Test organization tools at times rely on users manually naming and/or categorizing tests, which feels completely wrong to me. Any sort of test management outside of where the tests are defined is doomed to become out of date or mismatched. Tracking tests over time in general can be painful, especially if something like a class or a name changes.

How can I improve on this? The key is a notion of a test-defined Permanent Identifier. This by definition cannot change over time and is used for external tracking. It also serves as a central point of reference for reporting.

import MyTests.ID._

class MyTests extends Tests.Pure("Foo"):
  test(FOO_0001, "should bar some baz") { implicit ctx =>
    assert(1 == 1)
  }

object MyTests:

  object ID:
    val FOO_0001: PermanentId = "FOO_0001".p
    end ID

end MyTests

Reporting

Test reports can be more powerful than they are in many cases. Why not offer ways to trace and capture details about things like HTTP requests? Why not make that power extensible, so another user can add Kafka tracing? This information can be presented in a normalized form within the test report. Logging within tests should be trivial, and should retain order. Users of a test framework should be able to choose how results are reported, and should (in my opinion) be able to add their own report formats if needed.

Another related thought is with regards to macro-level tracing. How do we identify a particular execution of a test? What about a particular execution of a suite of tests? This metadata can be extremely useful and can empower tooling that processes test results.

Effect Type Support

This point is rather specific to Scala (or languages with similar paradigms). I find that test frameworks often assume synchronous test code or provide questionable async support. There are some test frameworks that explicitly support effect types or streaming as well. When building my own, I want to support effect types (notably Cats Effect) and synchronous code.

Discourage Mocking

While libraries such as Mockito and ScalaMock might be quality pieces of software and useful to many people, I generally reject this type of mocking in my tests. I strongly prefer to write stub implementations. My test framework will make no effort whatsoever to support mocks or any mocking library.

Flexibility in Usage

I see test frameworks advertised as "unit testing" frameworks at times. Why not use them for more? Why does execution need to be bound to some build tool? Why can't I easily run tests as part of some generic application in the main function? I want my test framework to support different use cases (unit, functional tests) and different methods of invocation (build tool, main).

Note that this also includes things like how tests are selected and instantiated. Many (most? all?) test frameworks I've used seriously expect static instantiation of tests classes and often rely on things like assumptions about no-arg constructors. I'm interested in challenging some of these assumptions.

Prior Art

weaver-test seems to do a number of things right, though I lack extensive experience using it. uTest is another cool project that pushes towards simplicity, but also does things I don't necessarily care for such as arbitrary test nesting (which I can just not use). I've had a generally good experience with it. ScalaTest is quite a battle-tested option that I have used extensively, and it's just too big -- I use a very restricted set of features and am somewhat happy with it.

Where do I go from here?

I'm going to push to write one or two more posts in June. On top of that, I'm intending them to be more focused than this post. I'd like to get into some specifics with code in something near-term. This entry took me a couple of hours to draft, so I'm expecting a greater investment for something with more purpose and polish.