Tag Archives: github

iggy: ignore file processor for jvm

As a core team member for Swagger Codegen, one of the major features I’ve added in the last year was support for .swagger-codegen-ignore. This ignore file allows for defining more complex rules to prevent Swagger Codegen from overwriting existing files.

Previously, we only supported conditionally generating apis, models, their respective docs, and supporting files. You could list zero files (i.e. skip everything in that category) or list one or more filename in csv format. There were a few problems with this previous approach:

  • Users didn’t know how to define the excludes.
    • Should they be absolute or relative paths?
    • Should they include file extensions?
    • Just the Swagger Definition’s schema name?
    • Should it be lowercase, title cased, or lower camel cased.
  • It didn’t allow for excluding other files or folders unknown to Swagger Codegen at the time of the definition. This means that a generator in Swagger Codegen could create a new filename in a subsequent release, which may or may not be overwritten.
  • There was no support for pattern-based exclusions

To account for these issues, I followed the manpage for .gitignore to design a simple package for excluding files. In less than 1000 LOC (including headers and comments), I created a solution that I was pretty happy with.

Since then, I’ve extracted the module to a project called iggy and published to Maven Central so other projects can benefit from the code. Check out the README, tests, and examples for full features.

iggy allows you to define the filename used for ignore. For example, the code included in Swagger Codegen supports only .swagger-codegen-ignore in the directory root of the generated code, but we had an issue filed about how to allow for complex exclusions on initial generation from the maven plugin. I submitted a fix that allows users to define an external ignore file to be consumed by the maven plugin, and this file is used as an override to the generated root’s ignore file if users generate over existing code.

It’s pretty nifty.

Usage

Given a directory structure:

.
├── level1
│   ├── a.txt
│   ├── level1file
│   └── level2
│       ├── b.txt
│       ├── level2file
│       └── level3
│           ├── c.txt
│           └── level3file
└── rootedfile

And the following ignore file:

# This is an example ignore file

# This matches only a.txt under level1 directory (relative to .ignore)
level1/a.txt

# This recursively matches: c.text
**/c.txt

# This recursively matches files ending in: file
**/*file

# This matches only against the directory in which .ignore exists, negating from the recursive *file pattern above
!/rootedfile

# This allows any file recursively named b with a three-character extension
!**/b.???

We can easily evaluate whether or not files in the hierarchy are marked for inclusion or exclusion.

This example is included in the repo under examples/example and is a kotlin script:

#!/bin/sh
exec kotlins -cp `mvncp us.jimschubert:iggy:1.0.0` "$0" "[email protected]"
!#
// To run, see https://github.com/andrewoma/kotlin-script
// NOTE: This file will have apparent syntax issues in your IDE.
import us.jimschubert.iggy.IgnoreProcessor
import java.io.File

fun main(args: Array<String>) {
    val processor = IgnoreProcessor(".")
    val currentDirectory = File(".")

    fun walk(current: File) {
        current.listFiles().forEach { file ->
            if(file.isDirectory()) walk(file)
            else System.out.println("$file is allowed: ${processor.allowsFile(file)}")
        }
    }

    walk(currentDirectory)
}

Running this example gives the following output:

$ ./example
/Users/jim/bin/mvncp:267: warning: shadowing outer local variable - opts
./.ignore is allowed: true
./example is allowed: true
./level1/a.txt is allowed: false
./level1/level1file is allowed: false
./level1/level2/b.txt is allowed: true
./level1/level2/level2file is allowed: false
./level1/level2/level3/c.txt is allowed: false
./level1/level2/level3/level3file is allowed: false
./rootedfile is allowed: true

The API for IgnoreProcessor is pretty simple. First, you construct the class with the directory location of the ignore file (and optionally an ignore filename override). Then, you provide a file to IgnoreProcessor#allowsFile and it returns true or false based on your ignore file’s definitions.

You can also construct IgnoreProcessor with a File.

Notes

Unlike gitignore, iggy doesn’t traverse all directories to the defined hierarchy root. There is a single source of ignore definitions.

iggy uses java.nio.file.PathMatcher with conditionally defined glob patterns based on lines defined in your ignore file. There are some OS-specific things (with Windows, really) where case sensitive evaluation may be unintuitive.

Code

The official repository is on Gitlab at https://gitlab.com/jimschubert/iggy. There is a “mirror” on GitHub.

Prototyping a Flags/Bitmasks implementation in Kotlin 1.1.1

One of my more favorite features of C# is the use of FlagsAttribute. If you’re not familiar with the concept, [Flags] allows you to treat an integral enum type as a bitmask. In other words, you can store 64 different on/off settings in a 64-bit integer.

The example from MSDN shows the storage of Tuesday + Thursday in a variable called meetingDays:

    [Flags]
    enum Days2
    {
        None = 0x0,
        Sunday = 0x1,
        Monday = 0x2,
        Tuesday = 0x4,
        Wednesday = 0x8,
        Thursday = 0x10,
        Friday = 0x20,
        Saturday = 0x40
    }
    class MyClass
    {
        Days2 meetingDays = Days2.Tuesday | Days2.Thursday;
    }

In C#, the operators are built into the compiler, so meetingDays becomes an integer with the 3rd and 5th bits set. With too many flags options, it can become difficult to keep track of the values, so I prefer using a left shift operator:

    [Flags]
    enum Days2
    {
        None = 0,
        Sunday = 1 << 0,
        Monday = 1 << 1,
        Tuesday = 1 << 2,
        Wednesday = 1 << 3,
        Thursday = 1 << 4,
        Friday = 1 << 5,
        Saturday = 1 << 6
    }
    
    class MyClass
    {
        public Days2 meetingDays = Days2.Tuesday | Days2.Thursday;
    }
    

Because Kotlin doesn’t support flags directly, to add programmatic support for this, we’ll need to implement some operators or infix operators. Unfortunately, as of right now Kotlin doesn’t support overloading the bitwise OR (|) operator. A workaround is to use a specialized operator that means “set”, or to explicitly use the named “or” operator.

The below example doesn’t support all bitwise operators, but it should be enough to get a project moving forward with a generalized Flags implementation.

fun main(args : Array<String>){
    val mask: BitMask = ParameterFeature.Path + 
        ParameterFeature.Query + 
        ParameterFeature.Header;
    
    val enabled = enabledValues<ParameterFeature>(mask)
    
    println("flags enabled: $enabled")
    println("flags disabled: ${enumValues<ParameterFeature>().filterNot { enabled.contains(it) } }")
    println("mask hasFlag ParameterFeature.Query: ${mask hasFlag ParameterFeature.Query}")
    println("mask hasFlag ParameterFeature.Body: ${mask hasFlag ParameterFeature.Body}")
}

An interesting point to notice here is that setting multiple flags results in a BitMask instance rather than a ParameterFeature instance as you’d have in C#. Another point is that, like C#, there’s no compiler control that requires each enum entry to be a single bit or even to be contiguous bits. I actually prefer the Kotlin syntax here for shifting a bit to the left: 1 shl 1.

The implementation of a flag would look like this:

enum class ParameterFeature(override val bit: Long) : Flags {
    Undefined(0),
    Path(1 shl 0),
    Query(1 shl 1),
    Header(1 shl 2),
    Body(1 shl 3),
    FormUnencoded(1 shl 4),
    FormMultipart(1 shl 5);
}

Notice how this requires extending from the Flags interface and overriding the bit constructor parameter. Unlike C#, which allows you to define flags using any integral type, my prototype is limited to Long.

Everything else is handled by jugging between the two types of Flags and BitMask using a combination of operators and infix functions, for example:

infix fun Flags.and(other: Long): BitMask = BitMask(bit and other)
infix fun <T: Flags> Flags.or(other: T): BitMask = BitMask(bit or other.bit)

Here’s the complete example as a gist:

Coding against a composed docker environment

Microservices are pretty big these days. Unfortunately, they add complexity to development, deployment, and maintenance.

For instance, how do you code against multiple dependencies locally? What if your architecture gets larger and includes things like service discovery via consul, a database, long-lived TCP connections with a backend service? It would be a pain to spin all of those dependencies up locally, and it would be costly to run them all in the cloud for development. It is absolutely possible for your architecture to be too large to follow this pattern– fair warning.

It’s easy to put together a virtual network using Docker Compose which collects microservices into a single environment. Docker also makes it easy to do containerized development. I feel like this is a huge trend, and we’ll probably see IDEs or other tooling spring up to support this.

This isn’t a tutorial about Docker. If you’re not familiar with Docker, walk through some tutorials first. This is a quick example of connecting a development container to an existing docker network.

The example is taken from an example in jimschubert/sbt-scala, the repo for my SBT/Scala docker image.

QOTD Service/Client Example

This example will create a composed environment consisting of two Quote of the Day (QOTD) services. These services will exist on the same network, exposing port 17 only within the network. The client is the hypothetical piece under development which queries the QOTD from one (or both) services using only the internal host and port.

The QOTD service code is very simple, requiring only nmap-ncat and fortune (in alpine linux):

#!/bin/sh
exec ncat -l 17 --keep-open --send-only --exec "/foo.sh"

The script that generates the QOTD is also simple:

#!/bin/sh
echo $HOSTNAME: $(/usr/bin/fortune)

Check out the full Dockerfile. Remember, this post is more about the service/client communication within a docker composed network.

The key here is in how docker-compose.yml is defined:

version: '2'
services:
  fortune1:
    image: fortune
    hostname: fortune1
    mem_limit: 64MB
    expose:
     - "17"
    networks:
      - fortune
  fortune2:
    image: fortune
    hostname: fortune2
    mem_limit: 64MB
    expose:
     - "17"
    networks:
      - fortune
networks:
  fortune:
    driver: bridge

Each service is assigned to an internal network called fortune. This is how the development container will associate with these services. To run the services, you’d just build the Dockerfile which runs each service and execute

docker-compose up

Do that in one terminal, then open another terminal for the next part.

Client

The client is written in Scala. For simplicity, it is a command line tool which queries one or more hosts (on port 17) provided as an argument to the script. For kicks, consider how the following code (that is, querying against fortune1:17 and fortune2:17 from a single client) could be done without docker on a development machine.

Save this somewhere as src/Example.scala:

import java.net._
import java.io._
import scala.io._

object Example { 
    def qotd(server: String): Unit = {
        val s = new Socket(InetAddress.getByName(server), 17)
        val in = new BufferedSource(s.getInputStream()).getLines()
        println(in.next())
        s.close()
    }
    
    def main(args: Array[String]): Unit = args.foreach(qotd) 
}

If there’s a suitable public image available, you don’t need a Dockerfile for this code. Here’s how you can do local development against the above client:

docker run -it --net=fortune_fortune -v $(pwd)/src:/src --workdir=/src jimschubert/sbt-scala:latest

The docker run -it … jimschubert/sbt-scala:latest parts above just execute an interactive shell in an alpine linux container that I maintain for SBT/Scala development.

–net=fortune_fortune joins this container to the network created by the docker-compose.yml. The network is created automatically when you bring up the environment with docker-compose up. It’s in the format project_network where project is pulled from the directory name, COMPOSE_PROJECT_NAME environment variable or the optional docker-compose

-p, --project-name NAME

switch. If you’re not running against the code from github, you can run

docker network list

to get the generated network name.

-v $(pwd)/src:/src mounts the src directory relative to where you’ve run the command to the /src directory within the container.

–workdir=/src is here for convenience, so you’re starting in the right folder in the container.

Once run the full command and you’re in the container, you can execute the client directly:

scala Example.scala fortune1 fortune2 fortune1

Go ahead and add a println(server) to the top of the qotd method in Example.scala and run the command again. No need for restarting the container!

Conclusion

This post demonstrates how to do containerized development of a simple client. There are multiple instances of the QOTD service (port 17) consumed by the client running on a single development machine.

The ports of these services can’t be exposed to port 17 on the host machine because this would result in a port conflict. It doesn’t make a lot of sense to remap port 17 to some other port on the host because port 17 is an explicitly defined port for QOTD. There may be issues if you do attempt to remap the port, like what if you choose a port that is randomly acquired by some other application?

Running the client in a container allows us to access the services as they would exist in the wild. Mapping the src directory from the host to the container allows us to quickly iterate on code without restarting the container. This is important, especially if you’re using a much larger base image than alpine.

If you’re doing Scala development, a recommended next step would be to use the spray/revolver plugin. Once setup, you would just start an sbt REPL and run ~re-start. Any local code changes are automatically detected and the plugin restarts the JVM process running your application.

Ignoring files with Swagger Codegen

I’m proud to announce that I’m now a member of the Swagger Codegen Core team.

I’ve previously refactored the C# client, the maven plugin, added an ASP.NET 5 Web API server generator, and fixed some bugs.

I recently added support for .swagger-codegen-ignore. A few weeks ago, I updated the C# client project structure to be a more standard structure:

    .
    ├── IO.Swagger.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

With this breaking change and the .swagger-codegen-ignore, users can now generate into existing repositories. Suppose your project is setup this way:

    .
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        └── MyProject.Test
            └── packages.config

You may want your generated client code to be output under src/MyProject.Client, but you don’t want build.bat, build.sh, or docs to get blown away. Last week, I partially resolved this by adding options to disable api and model doc generation for all languages… but there was no clean way to ignore other files like build.bat or build.sh.

Now, you can add the following .swagger-codegen-ignore to your existing directory structure:

# Swagger Codegen Ignore
/README.md
/build.*
docs/
src/
!src/MyProject.Client/
!src/MyProject.Client.Test/

Just to be safe (this feature is brand new), commit and push your changes before running swagger-codegen using your repository as the output directory. You should end up with the following structure:

    .
    ├── .swagger-codegen-ignore
    ├── IO.Swagger.sln
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        ├── MyProject.Test
        │   └── packages.config
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

As expected, the only generated content should be the IO.Swagger.sln, src/IO.Swagger and src/IO.Swagger.Test.

All of the functionality in this post should land in version 2.2.0. Until then, you can build directly against the master branch for these features.

Create configurable Angular services using providers

With Angular 2 on its way, it seems like a lot of developers are now afraid of Angular 1.x. I have nothing bad to say about Angular. Sure, some things could be made easier but that’s why Angular 2 is on its way. I’ve played with Angular 2 a bit, and it only works in Chrome. I’m fairly confident Angular 1.x will be around for quite a while.

In that light, let’s talk about creating configurable services. One reason why AngularJS is so popular is because you can easily create modular, reusable code. This code can encapsulate logic, directives, extensible services and factories (via decorators), as well as configurability. I’ll talk about creating directives and decorators in future articles.

This example will be a simple URL constructor for placekitten.com. Any AngularJS service can be written to be configurable (if there’s reason to configure the service, of course).

First of all, let’s see how this service will be used.

.controller('View1Ctrl', ['$scope', 'kittyService', function ($scope, kittyService) {
    $scope.kittycat = kittyService.getUrl();
}])
  <p>How about a kitty cat?</p>
  <p><img ng-src="{{kittycat}}"/></p>

Pretty simple. We’ll call kittyService.getUrl() and use the string returned from the service as the src for an image.

To create a configurable service, we’ll use an angular provider instead of a service or factory. The structure I use for a provider is this:

(function () {
    'use strict';
    var module = angular.module('myApp.services', []);

    module.provider('serviceName', function ServiceName() {
        // use `this` for configurable properties or functions

        this.$get = [function () {
            var service = {};
            // This can be any valid return that you'd normally use
            // with module.service or module.factory
            return service;
        }];
    });
})();

This structure is only a little different from a service or factory. Rather than return an object or constructor function, you use this.$get to create your reusable code. What you assign to this property is the array injector for your service definition. You’ll feel comfortable with this syntax if you define your services using the array-based dependency injection:

angular.module('myApp')
.service('serviceName', [function (){
            var service = {};
            // This can be any valid return that you'd normally use
            // with module.service or module.factory
            return service;
}]);

You can use instance properties and methods on the provider definition to configure your service, then use the private context of your provider to share the configuration with the service. Here’s the full code for the kittyService:

(function () {
    'use strict';
    var module = angular.module('myApp.services', []);

    module.provider('kittyService', function KittyServiceProvider() {
        var height = 100,
            width = 100;

        this.setHeight = function (h) {
            height = h;
        };
        this.setWidth = function (w) {
            width = w;
        };

        this.$get = [function () {
            var service = {};

            service.getUrl = function () {
                return 'http://placekitten.com/g/' + width + '/' + height;
            };

            // This can be any valid return that you'd normally use
            // with module.service or module.factory
            return service;
        }];
    });
})();

The provider has two private variables: height and width. The two methods on the provider object allow you to update these values, but only during the config phase. After your application’s config function has completed, your provider’s functions are no longer accessible.

One problem with this service is that placekitten.com shows the same image for a given combination of height and width. In the config phase of our application, the service’s height and width can be configured randomly:

.config(['kittyServiceProvider', function (kittyServiceProvider) {
    var widths = [200,300,400,500,600];
    var heights = [300,400,500,600,700];
    kittyServiceProvider.setHeight(heights[Math.floor(Math.random()*heights.length)]);
    kittyServiceProvider.setWidth(widths[Math.floor(Math.random()*widths.length)]);
}]);

The name injected into the config phase is kittyServiceProvider and not kittyService. This is important. AngularJS knows that when you request kittyServiceProvider in the config phase, you want the object containing your service definition (the KittyServiceProvider function in the example above). Once the config phase completes, the $injector service will prepare your service from the this.$get definition of your service.

If you load the example code, refresh the page a few times and you’ll receive a new kitty image.

NOTE: The heights and widths may not always combine in a way that retrieves an image. Just refresh for another example.

Why not do this in a service?

True, you could do this directly in the service if this was just your code. The benefit here is when you want to share a service with other developers, perhaps on a different project within your company or publicly on GitHub. What if your team wants a 200×300 kitty and HR wants a 500×500 kitty? In these situations you can’t just modify the code with your hard-coded values. True, in this example you could use constants or values instead but then your service would have a dependency on the consumer (this is bad).

A service that dumps a string for displaying a cat image is probably not that useful in the real world. Consider other possibilities. I’ve used providers for creating a reusable notification display which allows you to define whether a notification will automatically close and how long notifications will be displayed if they auto-close. Suppose your application often receives 40x errors when users leave their browsers unattended long enough for an authenticated session to time out; you could create a provider which caches failed requests and retries up to X (configurable) times before giving up. Or, you could create a configurable service which caches those failed requests, displays a login dialog, then automatically retries the failed requests X (configurable) milliseconds after logging in successfully. Those examples would require the use of an http interceptor, and would have been quite a bit more complex than a kitty image linker.

Code

Sample code for this blog post is available on github at jimschubert/blogs.

Further Reading

If you really want to dig into the guts of Providers (versus services, factories, constants, or values), check out Adrian Castillo’s blog post, AngularJS Providers under the hood.

For reference to each recipe, check out AngularJS Provider Recipes on the official docs.

Decorating directives with isolate scope in Angular 1.3

A question from reddit excited my interest. Angular 1.3 apparently has broken the ability to decorate isolate scopes.

I tried version of Angular between the 1.2.26 known working version to the 1.3.0 broken version and found the issue was introduced in 1.3.0-rc.2.

The way to get around this is hacky:

app.config(function($provide) {
    $provide.decorator('mycustomDirective',['$delegate','$controller', function($delegate, $controller) {
        var directive = $delegate[0];
        directive.$$isolateBindings['othervar'] = {
          attrName: 'othervar',
          mode: '=',
          optional: true
        };
        
        return $delegate;  
    }]);
});   

Here’s a working example: http://plnkr.co/edit/CRxhX6?p=preview

Your First App: Node.js is complete!

I’m stoked to announce that I’ve finished writing my first self-published book, Your First App: node.js.

The book is a full-stack application development tutorial using what is commonly known as the ‘MEAN’ stack, but with a heavier focus on the backend technologies: node.js, express.js, and MongoDB. The book ends with a starter AngularJS project of an HTML5 instant messaging chat application.

While following along in the book, it may be useful to run test queries in a REST client. Postman is available for Google Chrome as a packaged application. Import `yfa-nodejs.postman_dump.json` (located in the root of the supplemental code’s repository) into Postman to evaluate the application’s API.

Check out the code on GitHub