iggy: ignore file processor for jvm

As a core team member for Swagger Codegen, one of the major features I’ve added in the last year was support for .swagger-codegen-ignore. This ignore file allows for defining more complex rules to prevent Swagger Codegen from overwriting existing files.

Previously, we only supported conditionally generating apis, models, their respective docs, and supporting files. You could list zero files (i.e. skip everything in that category) or list one or more filename in csv format. There were a few problems with this previous approach:

  • Users didn’t know how to define the excludes.
    • Should they be absolute or relative paths?
    • Should they include file extensions?
    • Just the Swagger Definition’s schema name?
    • Should it be lowercase, title cased, or lower camel cased.
  • It didn’t allow for excluding other files or folders unknown to Swagger Codegen at the time of the definition. This means that a generator in Swagger Codegen could create a new filename in a subsequent release, which may or may not be overwritten.
  • There was no support for pattern-based exclusions

To account for these issues, I followed the manpage for .gitignore to design a simple package for excluding files. In less than 1000 LOC (including headers and comments), I created a solution that I was pretty happy with.

Since then, I’ve extracted the module to a project called iggy and published to Maven Central so other projects can benefit from the code. Check out the README, tests, and examples for full features.

iggy allows you to define the filename used for ignore. For example, the code included in Swagger Codegen supports only .swagger-codegen-ignore in the directory root of the generated code, but we had an issue filed about how to allow for complex exclusions on initial generation from the maven plugin. I submitted a fix that allows users to define an external ignore file to be consumed by the maven plugin, and this file is used as an override to the generated root’s ignore file if users generate over existing code.

It’s pretty nifty.

Usage

Given a directory structure:

.
├── level1
│   ├── a.txt
│   ├── level1file
│   └── level2
│       ├── b.txt
│       ├── level2file
│       └── level3
│           ├── c.txt
│           └── level3file
└── rootedfile

And the following ignore file:

# This is an example ignore file

# This matches only a.txt under level1 directory (relative to .ignore)
level1/a.txt

# This recursively matches: c.text
**/c.txt

# This recursively matches files ending in: file
**/*file

# This matches only against the directory in which .ignore exists, negating from the recursive *file pattern above
!/rootedfile

# This allows any file recursively named b with a three-character extension
!**/b.???

We can easily evaluate whether or not files in the hierarchy are marked for inclusion or exclusion.

This example is included in the repo under examples/example and is a kotlin script:

#!/bin/sh
exec kotlins -cp `mvncp us.jimschubert:iggy:1.0.0` "$0" "$@"
!#
// To run, see https://github.com/andrewoma/kotlin-script
// NOTE: This file will have apparent syntax issues in your IDE.
import us.jimschubert.iggy.IgnoreProcessor
import java.io.File

fun main(args: Array<String>) {
    val processor = IgnoreProcessor(".")
    val currentDirectory = File(".")

    fun walk(current: File) {
        current.listFiles().forEach { file ->
            if(file.isDirectory()) walk(file)
            else System.out.println("$file is allowed: ${processor.allowsFile(file)}")
        }
    }

    walk(currentDirectory)
}

Running this example gives the following output:

$ ./example
/Users/jim/bin/mvncp:267: warning: shadowing outer local variable - opts
./.ignore is allowed: true
./example is allowed: true
./level1/a.txt is allowed: false
./level1/level1file is allowed: false
./level1/level2/b.txt is allowed: true
./level1/level2/level2file is allowed: false
./level1/level2/level3/c.txt is allowed: false
./level1/level2/level3/level3file is allowed: false
./rootedfile is allowed: true

The API for IgnoreProcessor is pretty simple. First, you construct the class with the directory location of the ignore file (and optionally an ignore filename override). Then, you provide a file to IgnoreProcessor#allowsFile and it returns true or false based on your ignore file’s definitions.

You can also construct IgnoreProcessor with a File.

Notes

Unlike gitignore, iggy doesn’t traverse all directories to the defined hierarchy root. There is a single source of ignore definitions.

iggy uses java.nio.file.PathMatcher with conditionally defined glob patterns based on lines defined in your ignore file. There are some OS-specific things (with Windows, really) where case sensitive evaluation may be unintuitive.

Code

The official repository is on Gitlab at https://gitlab.com/jimschubert/iggy. There is a “mirror” on GitHub.

Prototyping a Flags/Bitmasks implementation in Kotlin 1.1.1

One of my more favorite features of C# is the use of FlagsAttribute. If you’re not familiar with the concept, [Flags] allows you to treat an integral enum type as a bitmask. In other words, you can store 64 different on/off settings in a 64-bit integer.

The example from MSDN shows the storage of Tuesday + Thursday in a variable called meetingDays:

    [Flags]
    enum Days2
    {
        None = 0x0,
        Sunday = 0x1,
        Monday = 0x2,
        Tuesday = 0x4,
        Wednesday = 0x8,
        Thursday = 0x10,
        Friday = 0x20,
        Saturday = 0x40
    }
    class MyClass
    {
        Days2 meetingDays = Days2.Tuesday | Days2.Thursday;
    }

In C#, the operators are built into the compiler, so meetingDays becomes an integer with the 3rd and 5th bits set. With too many flags options, it can become difficult to keep track of the values, so I prefer using a left shift operator:

    [Flags]
    enum Days2
    {
        None = 0,
        Sunday = 1 << 0,
        Monday = 1 << 1,
        Tuesday = 1 << 2,
        Wednesday = 1 << 3,
        Thursday = 1 << 4,
        Friday = 1 << 5,
        Saturday = 1 << 6
    }
    
    class MyClass
    {
        public Days2 meetingDays = Days2.Tuesday | Days2.Thursday;
    }
    

Because Kotlin doesn’t support flags directly, to add programmatic support for this, we’ll need to implement some operators or infix operators. Unfortunately, as of right now Kotlin doesn’t support overloading the bitwise OR (|) operator. A workaround is to use a specialized operator that means “set”, or to explicitly use the named “or” operator.

The below example doesn’t support all bitwise operators, but it should be enough to get a project moving forward with a generalized Flags implementation.

fun main(args : Array<String>){
    val mask: BitMask = ParameterFeature.Path + 
        ParameterFeature.Query + 
        ParameterFeature.Header;
    
    val enabled = enabledValues<ParameterFeature>(mask)
    
    println("flags enabled: $enabled")
    println("flags disabled: ${enumValues<ParameterFeature>().filterNot { enabled.contains(it) } }")
    println("mask hasFlag ParameterFeature.Query: ${mask hasFlag ParameterFeature.Query}")
    println("mask hasFlag ParameterFeature.Body: ${mask hasFlag ParameterFeature.Body}")
}

An interesting point to notice here is that setting multiple flags results in a BitMask instance rather than a ParameterFeature instance as you’d have in C#. Another point is that, like C#, there’s no compiler control that requires each enum entry to be a single bit or even to be contiguous bits. I actually prefer the Kotlin syntax here for shifting a bit to the left: 1 shl 1.

The implementation of a flag would look like this:

enum class ParameterFeature(override val bit: Long) : Flags {
    Undefined(0),
    Path(1 shl 0),
    Query(1 shl 1),
    Header(1 shl 2),
    Body(1 shl 3),
    FormUnencoded(1 shl 4),
    FormMultipart(1 shl 5);
}

Notice how this requires extending from the Flags interface and overriding the bit constructor parameter. Unlike C#, which allows you to define flags using any integral type, my prototype is limited to Long.

Everything else is handled by jugging between the two types of Flags and BitMask using a combination of operators and infix functions, for example:

infix fun Flags.and(other: Long): BitMask = BitMask(bit and other)
infix fun <T: Flags> Flags.or(other: T): BitMask = BitMask(bit or other.bit)

Here’s the complete example as a gist:

Coding against a composed docker environment

Microservices are pretty big these days. Unfortunately, they add complexity to development, deployment, and maintenance.

For instance, how do you code against multiple dependencies locally? What if your architecture gets larger and includes things like service discovery via consul, a database, long-lived TCP connections with a backend service? It would be a pain to spin all of those dependencies up locally, and it would be costly to run them all in the cloud for development. It is absolutely possible for your architecture to be too large to follow this pattern– fair warning.

It’s easy to put together a virtual network using Docker Compose which collects microservices into a single environment. Docker also makes it easy to do containerized development. I feel like this is a huge trend, and we’ll probably see IDEs or other tooling spring up to support this.

This isn’t a tutorial about Docker. If you’re not familiar with Docker, walk through some tutorials first. This is a quick example of connecting a development container to an existing docker network.

The example is taken from an example in jimschubert/sbt-scala, the repo for my SBT/Scala docker image.

QOTD Service/Client Example

This example will create a composed environment consisting of two Quote of the Day (QOTD) services. These services will exist on the same network, exposing port 17 only within the network. The client is the hypothetical piece under development which queries the QOTD from one (or both) services using only the internal host and port.

The QOTD service code is very simple, requiring only nmap-ncat and fortune (in alpine linux):

#!/bin/sh
exec ncat -l 17 --keep-open --send-only --exec "/foo.sh"

The script that generates the QOTD is also simple:

#!/bin/sh
echo $HOSTNAME: $(/usr/bin/fortune)

Check out the full Dockerfile. Remember, this post is more about the service/client communication within a docker composed network.

The key here is in how docker-compose.yml is defined:

version: '2'
services:
  fortune1:
    image: fortune
    hostname: fortune1
    mem_limit: 64MB
    expose:
     - "17"
    networks:
      - fortune
  fortune2:
    image: fortune
    hostname: fortune2
    mem_limit: 64MB
    expose:
     - "17"
    networks:
      - fortune
networks:
  fortune:
    driver: bridge

Each service is assigned to an internal network called fortune. This is how the development container will associate with these services. To run the services, you’d just build the Dockerfile which runs each service and execute

docker-compose up

Do that in one terminal, then open another terminal for the next part.

Client

The client is written in Scala. For simplicity, it is a command line tool which queries one or more hosts (on port 17) provided as an argument to the script. For kicks, consider how the following code (that is, querying against fortune1:17 and fortune2:17 from a single client) could be done without docker on a development machine.

Save this somewhere as src/Example.scala:

import java.net._
import java.io._
import scala.io._

object Example { 
    def qotd(server: String): Unit = {
        val s = new Socket(InetAddress.getByName(server), 17)
        val in = new BufferedSource(s.getInputStream()).getLines()
        println(in.next())
        s.close()
    }
    
    def main(args: Array[String]): Unit = args.foreach(qotd) 
}

If there’s a suitable public image available, you don’t need a Dockerfile for this code. Here’s how you can do local development against the above client:

docker run -it --net=fortune_fortune -v $(pwd)/src:/src --workdir=/src jimschubert/sbt-scala:latest

The docker run -it … jimschubert/sbt-scala:latest parts above just execute an interactive shell in an alpine linux container that I maintain for SBT/Scala development.

–net=fortune_fortune joins this container to the network created by the docker-compose.yml. The network is created automatically when you bring up the environment with docker-compose up. It’s in the format project_network where project is pulled from the directory name, COMPOSE_PROJECT_NAME environment variable or the optional docker-compose

-p, --project-name NAME

switch. If you’re not running against the code from github, you can run

docker network list

to get the generated network name.

-v $(pwd)/src:/src mounts the src directory relative to where you’ve run the command to the /src directory within the container.

–workdir=/src is here for convenience, so you’re starting in the right folder in the container.

Once run the full command and you’re in the container, you can execute the client directly:

scala Example.scala fortune1 fortune2 fortune1

Go ahead and add a println(server) to the top of the qotd method in Example.scala and run the command again. No need for restarting the container!

Conclusion

This post demonstrates how to do containerized development of a simple client. There are multiple instances of the QOTD service (port 17) consumed by the client running on a single development machine.

The ports of these services can’t be exposed to port 17 on the host machine because this would result in a port conflict. It doesn’t make a lot of sense to remap port 17 to some other port on the host because port 17 is an explicitly defined port for QOTD. There may be issues if you do attempt to remap the port, like what if you choose a port that is randomly acquired by some other application?

Running the client in a container allows us to access the services as they would exist in the wild. Mapping the src directory from the host to the container allows us to quickly iterate on code without restarting the container. This is important, especially if you’re using a much larger base image than alpine.

If you’re doing Scala development, a recommended next step would be to use the spray/revolver plugin. Once setup, you would just start an sbt REPL and run ~re-start. Any local code changes are automatically detected and the plugin restarts the JVM process running your application.

Finding Wakelocks in Android 7 using Google’s Battery Historian

One of my favorite apps prior to my upgrade to Android 7 (Nougat) was GSam Battery Monitor (the paid version is fully worth it). Unfortunately, Android 7 no longer allows apps like GSam Battery Monitor to track individual application battery usage, CPU usage, or wakelocks. That sucks*.

In Android 7 with Doze, you should see periods of marginal battery usage. The more apps you have, the more background services/sync operations/location broadcasts are running and you’ll see plateaus in battery usage be more slanted. Up until a few weeks ago, battery usage would basically flatline between screen time. I set out to figure out what was causing this. I found a thread on Reddit discussing Battery Historian, so I decided to check it out. This post is somewhat of a review of this tool.

Battery Historian analyzes bug report files and presents the information in an interactive graph with categorized breakdowns of battery usage. Battery Historian is a web application written in Go, and doesn’t appear to have an official release (binary or hosted). This means it’s basically limited to power users to evaluate. Since it’s licensed under Apache 2.0, I compiled and hosted this for my own use at https://historian.jimschubert.us/. Before using my hosted instance of Battery Historian, please heed Google’s warning about bug report files:

Bug reports contain data from the system’s various log files, including personal and private information. Only share bug reports with apps and people you trust.

Running Battery Historian locally isn’t too difficult. I definitely recommend going this route if you’re more of a power user; you don’t need programming experience, only comfort on the command line.

Once you’ve taken a bug report on your device and uploaded it to Battery Historian, you’re presented with a graph of all activity found in the bug report.

battery_historian

The declining black line is the same battery usage visualization you’d see in your Battery Usage application or widget. Other groups of battery draining events are presented as horizontal time-oriented lines; some are either on or off (screen), some show gradients for battery draining in the category (temperature), while others display different colors for different states of the event (Mobile signal strength).

Scroll below the graph and you’ll find a breakdown of system statistics. After a little sorting and investigation, I found that JuiceSSH had locked WiFi for 1.5 hours when I hadn’t used the application for over a week:

battery_historian_-_juicessh_wifi_lock

After selecting the JuiceSSH app in the left dropdown, this takes you to the App Stats tab which presents some interesting battery usage statistics:

Wakelocks:
Wakelock Name      | Full Time | Full Count | Partial Time | Partial Count | Window Time | Window Count
JuiceSSH CloudSync |           | 0          | 8m 0s 50ms   | 1             |             | 0           
*alarm*            |           | 0          | 684ms        | 1             |             | 0           
Services:
Service Name                            | Time spent started | # starts | # launches
com.sonelli.juicessh.services.CloudSync | 2h 28m 15s 789ms   | 1        | 1         
Process info:
Time spent running actively in the foreground       | 0ms             
Time spent with a service running in the foreground | 0ms             
Time spent on top                                   | 0ms             
Time spent on top while the device was sleeping     | 0ms             
Time spent running actively in the background       | 1h 33m 10s 343ms
Time spent cached                                   | 38ms            

Looks like, in my case, JuiceSSH Cloud Sync had 8 minutes worth of wakelocks, 2.5 hours of a service running in which 1.5 hours of that was actively running in the background. JuiceSSH unfortunately doesn’t have configuration options for Cloud Sync, so I’ve disabled it. I’ve noticed an improvement since disabling JuiceSSH Cloud Sync.

Although I was a little miffed that the Android 7 update borked GSam’s ability to report on per-application usage, I found Battery Historian to be a really insightful tool. From and Android Development perspective it would be a nice utility to evaluate your application’s effect on the system through normal use. It may be a little overkill from a user or power user’s perspective, though. My Nexus 5X was getting about 18 hours of pretty heavy usage (including my son watching YouTube videos). After disabling JuiceSSH Cloud Sync, I’m seeing about 20 hours of battery out of the same usage. Does that really matter if you’re like me and you charge your phone every night? 👻

* I’ve read that GSam Battery Monitor with root still works as expected on Nougat.

Ignoring files with Swagger Codegen

I’m proud to announce that I’m now a member of the Swagger Codegen Core team.

I’ve previously refactored the C# client, the maven plugin, added an ASP.NET 5 Web API server generator, and fixed some bugs.

I recently added support for .swagger-codegen-ignore. A few weeks ago, I updated the C# client project structure to be a more standard structure:

    .
    ├── IO.Swagger.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

With this breaking change and the .swagger-codegen-ignore, users can now generate into existing repositories. Suppose your project is setup this way:

    .
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        └── MyProject.Test
            └── packages.config

You may want your generated client code to be output under src/MyProject.Client, but you don’t want build.bat, build.sh, or docs to get blown away. Last week, I partially resolved this by adding options to disable api and model doc generation for all languages… but there was no clean way to ignore other files like build.bat or build.sh.

Now, you can add the following .swagger-codegen-ignore to your existing directory structure:

# Swagger Codegen Ignore
/README.md
/build.*
docs/
src/
!src/MyProject.Client/
!src/MyProject.Client.Test/

Just to be safe (this feature is brand new), commit and push your changes before running swagger-codegen using your repository as the output directory. You should end up with the following structure:

    .
    ├── .swagger-codegen-ignore
    ├── IO.Swagger.sln
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        ├── MyProject.Test
        │   └── packages.config
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

As expected, the only generated content should be the IO.Swagger.sln, src/IO.Swagger and src/IO.Swagger.Test.

All of the functionality in this post should land in version 2.2.0. Until then, you can build directly against the master branch for these features.

Bash function: recent

#!/bin/env bash

# Display recently changed files recursively below the current directory
# recent [minutes]
#    minutes: The number of minutes to display. Default 5.
# 
# Credit to find usage goes to
# F. Hauri (http://superuser.com/users/178656/f-hauri)
# see: http://superuser.com/a/621727/60571
function recent {
   local mins="-$((${1:-5}+0))"
    
   find . -type f -mmin $mins -print0 | xargs -0 /bin/ls -ltr
}

Example:

jim at macbert in ~/projects/swagger-codegen on cs/req_properties_2584 705ed78
$ recent
-rw-r--r--@ 1 jim  staff  6148 May  8 11:05 ./samples/client/petstore/csharp/SwaggerClient/.DS_Store
jim at macbert in ~/projects/swagger-codegen on cs/req_properties_2584 705ed78
$ recent 120
-rw-r--r--  1 jim  staff     149 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/obj/Debug/.NETFramework,Version=v4.5.AssemblyAttribute.cs
-rwxr-xr-x  1 jim  staff  131072 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/bin/Debug/Swagger Library.dll
-rwxr-xr-x  1 jim  staff  131072 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/obj/Debug/Swagger Library.dll
-rwxr-xr-x  1 jim  staff  131072 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/bin/Debug/Swagger Library.dll
-rw-r--r--  1 jim  staff   44083 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/bin/Debug/Swagger Library.dll.mdb
-rw-r--r--  1 jim  staff   44083 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/obj/Debug/Swagger Library.dll.mdb
-rw-r--r--  1 jim  staff     940 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/obj/Debug/IO.Swagger.csproj.FilesWrittenAbsolute.txt
-rw-r--r--  1 jim  staff   44083 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/bin/Debug/Swagger Library.dll.mdb
-rw-r--r--  1 jim  staff    4365 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/obj/Debug/SwaggerClientTest.dll.mdb
-rwxr-xr-x  1 jim  staff   28160 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/obj/Debug/SwaggerClientTest.dll
-rw-r--r--  1 jim  staff    2875 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/obj/Debug/SwaggerClientTest.csproj.FilesWrittenAbsolute.txt
-rw-r--r--  1 jim  staff    4365 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/bin/Debug/SwaggerClientTest.dll.mdb
-rwxr-xr-x  1 jim  staff   28160 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/bin/Debug/SwaggerClientTest.dll
-rw-r--r--  1 jim  staff    8467 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/TestResult.xml
-rw-r--r--  1 jim  staff  431527 May  8 09:11 ./.git/index
-rw-r--r--@ 1 jim  staff    6148 May  8 11:05 ./samples/client/petstore/csharp/SwaggerClient/.DS_Store

The gist is available here.

Developer James Schubert shares his code and his thoughts.