Tag Archives: kotlin

iggy: ignore file processor for jvm

As a core team member for Swagger Codegen, one of the major features I’ve added in the last year was support for .swagger-codegen-ignore. This ignore file allows for defining more complex rules to prevent Swagger Codegen from overwriting existing files.

Previously, we only supported conditionally generating apis, models, their respective docs, and supporting files. You could list zero files (i.e. skip everything in that category) or list one or more filename in csv format. There were a few problems with this previous approach:

  • Users didn’t know how to define the excludes.
    • Should they be absolute or relative paths?
    • Should they include file extensions?
    • Just the Swagger Definition’s schema name?
    • Should it be lowercase, title cased, or lower camel cased.
  • It didn’t allow for excluding other files or folders unknown to Swagger Codegen at the time of the definition. This means that a generator in Swagger Codegen could create a new filename in a subsequent release, which may or may not be overwritten.
  • There was no support for pattern-based exclusions

To account for these issues, I followed the manpage for .gitignore to design a simple package for excluding files. In less than 1000 LOC (including headers and comments), I created a solution that I was pretty happy with.

Since then, I’ve extracted the module to a project called iggy and published to Maven Central so other projects can benefit from the code. Check out the README, tests, and examples for full features.

iggy allows you to define the filename used for ignore. For example, the code included in Swagger Codegen supports only .swagger-codegen-ignore in the directory root of the generated code, but we had an issue filed about how to allow for complex exclusions on initial generation from the maven plugin. I submitted a fix that allows users to define an external ignore file to be consumed by the maven plugin, and this file is used as an override to the generated root’s ignore file if users generate over existing code.

It’s pretty nifty.

Usage

Given a directory structure:

.
├── level1
│   ├── a.txt
│   ├── level1file
│   └── level2
│       ├── b.txt
│       ├── level2file
│       └── level3
│           ├── c.txt
│           └── level3file
└── rootedfile

And the following ignore file:

# This is an example ignore file

# This matches only a.txt under level1 directory (relative to .ignore)
level1/a.txt

# This recursively matches: c.text
**/c.txt

# This recursively matches files ending in: file
**/*file

# This matches only against the directory in which .ignore exists, negating from the recursive *file pattern above
!/rootedfile

# This allows any file recursively named b with a three-character extension
!**/b.???

We can easily evaluate whether or not files in the hierarchy are marked for inclusion or exclusion.

This example is included in the repo under examples/example and is a kotlin script:

#!/bin/sh
exec kotlins -cp `mvncp us.jimschubert:iggy:1.0.0` "$0" "[email protected]"
!#
// To run, see https://github.com/andrewoma/kotlin-script
// NOTE: This file will have apparent syntax issues in your IDE.
import us.jimschubert.iggy.IgnoreProcessor
import java.io.File

fun main(args: Array<String>) {
    val processor = IgnoreProcessor(".")
    val currentDirectory = File(".")

    fun walk(current: File) {
        current.listFiles().forEach { file ->
            if(file.isDirectory()) walk(file)
            else System.out.println("$file is allowed: ${processor.allowsFile(file)}")
        }
    }

    walk(currentDirectory)
}

Running this example gives the following output:

$ ./example
/Users/jim/bin/mvncp:267: warning: shadowing outer local variable - opts
./.ignore is allowed: true
./example is allowed: true
./level1/a.txt is allowed: false
./level1/level1file is allowed: false
./level1/level2/b.txt is allowed: true
./level1/level2/level2file is allowed: false
./level1/level2/level3/c.txt is allowed: false
./level1/level2/level3/level3file is allowed: false
./rootedfile is allowed: true

The API for IgnoreProcessor is pretty simple. First, you construct the class with the directory location of the ignore file (and optionally an ignore filename override). Then, you provide a file to IgnoreProcessor#allowsFile and it returns true or false based on your ignore file’s definitions.

You can also construct IgnoreProcessor with a File.

Notes

Unlike gitignore, iggy doesn’t traverse all directories to the defined hierarchy root. There is a single source of ignore definitions.

iggy uses java.nio.file.PathMatcher with conditionally defined glob patterns based on lines defined in your ignore file. There are some OS-specific things (with Windows, really) where case sensitive evaluation may be unintuitive.

Code

The official repository is on Gitlab at https://gitlab.com/jimschubert/iggy. There is a “mirror” on GitHub.

Prototyping a Flags/Bitmasks implementation in Kotlin 1.1.1

One of my more favorite features of C# is the use of FlagsAttribute. If you’re not familiar with the concept, [Flags] allows you to treat an integral enum type as a bitmask. In other words, you can store 64 different on/off settings in a 64-bit integer.

The example from MSDN shows the storage of Tuesday + Thursday in a variable called meetingDays:

    [Flags]
    enum Days2
    {
        None = 0x0,
        Sunday = 0x1,
        Monday = 0x2,
        Tuesday = 0x4,
        Wednesday = 0x8,
        Thursday = 0x10,
        Friday = 0x20,
        Saturday = 0x40
    }
    class MyClass
    {
        Days2 meetingDays = Days2.Tuesday | Days2.Thursday;
    }

In C#, the operators are built into the compiler, so meetingDays becomes an integer with the 3rd and 5th bits set. With too many flags options, it can become difficult to keep track of the values, so I prefer using a left shift operator:

    [Flags]
    enum Days2
    {
        None = 0,
        Sunday = 1 << 0,
        Monday = 1 << 1,
        Tuesday = 1 << 2,
        Wednesday = 1 << 3,
        Thursday = 1 << 4,
        Friday = 1 << 5,
        Saturday = 1 << 6
    }
    
    class MyClass
    {
        public Days2 meetingDays = Days2.Tuesday | Days2.Thursday;
    }
    

Because Kotlin doesn’t support flags directly, to add programmatic support for this, we’ll need to implement some operators or infix operators. Unfortunately, as of right now Kotlin doesn’t support overloading the bitwise OR (|) operator. A workaround is to use a specialized operator that means “set”, or to explicitly use the named “or” operator.

The below example doesn’t support all bitwise operators, but it should be enough to get a project moving forward with a generalized Flags implementation.

fun main(args : Array<String>){
    val mask: BitMask = ParameterFeature.Path + 
        ParameterFeature.Query + 
        ParameterFeature.Header;
    
    val enabled = enabledValues<ParameterFeature>(mask)
    
    println("flags enabled: $enabled")
    println("flags disabled: ${enumValues<ParameterFeature>().filterNot { enabled.contains(it) } }")
    println("mask hasFlag ParameterFeature.Query: ${mask hasFlag ParameterFeature.Query}")
    println("mask hasFlag ParameterFeature.Body: ${mask hasFlag ParameterFeature.Body}")
}

An interesting point to notice here is that setting multiple flags results in a BitMask instance rather than a ParameterFeature instance as you’d have in C#. Another point is that, like C#, there’s no compiler control that requires each enum entry to be a single bit or even to be contiguous bits. I actually prefer the Kotlin syntax here for shifting a bit to the left: 1 shl 1.

The implementation of a flag would look like this:

enum class ParameterFeature(override val bit: Long) : Flags {
    Undefined(0),
    Path(1 shl 0),
    Query(1 shl 1),
    Header(1 shl 2),
    Body(1 shl 3),
    FormUnencoded(1 shl 4),
    FormMultipart(1 shl 5);
}

Notice how this requires extending from the Flags interface and overriding the bit constructor parameter. Unlike C#, which allows you to define flags using any integral type, my prototype is limited to Long.

Everything else is handled by jugging between the two types of Flags and BitMask using a combination of operators and infix functions, for example:

infix fun Flags.and(other: Long): BitMask = BitMask(bit and other)
infix fun <T: Flags> Flags.or(other: T): BitMask = BitMask(bit or other.bit)

Here’s the complete example as a gist: