Tag Archives: Examples

iggy: ignore file processor for jvm

As a core team member for Swagger Codegen, one of the major features I’ve added in the last year was support for .swagger-codegen-ignore. This ignore file allows for defining more complex rules to prevent Swagger Codegen from overwriting existing files.

Previously, we only supported conditionally generating apis, models, their respective docs, and supporting files. You could list zero files (i.e. skip everything in that category) or list one or more filename in csv format. There were a few problems with this previous approach:

  • Users didn’t know how to define the excludes.
    • Should they be absolute or relative paths?
    • Should they include file extensions?
    • Just the Swagger Definition’s schema name?
    • Should it be lowercase, title cased, or lower camel cased.
  • It didn’t allow for excluding other files or folders unknown to Swagger Codegen at the time of the definition. This means that a generator in Swagger Codegen could create a new filename in a subsequent release, which may or may not be overwritten.
  • There was no support for pattern-based exclusions

To account for these issues, I followed the manpage for .gitignore to design a simple package for excluding files. In less than 1000 LOC (including headers and comments), I created a solution that I was pretty happy with.

Since then, I’ve extracted the module to a project called iggy and published to Maven Central so other projects can benefit from the code. Check out the README, tests, and examples for full features.

iggy allows you to define the filename used for ignore. For example, the code included in Swagger Codegen supports only .swagger-codegen-ignore in the directory root of the generated code, but we had an issue filed about how to allow for complex exclusions on initial generation from the maven plugin. I submitted a fix that allows users to define an external ignore file to be consumed by the maven plugin, and this file is used as an override to the generated root’s ignore file if users generate over existing code.

It’s pretty nifty.

Usage

Given a directory structure:

.
├── level1
│   ├── a.txt
│   ├── level1file
│   └── level2
│       ├── b.txt
│       ├── level2file
│       └── level3
│           ├── c.txt
│           └── level3file
└── rootedfile

And the following ignore file:

# This is an example ignore file

# This matches only a.txt under level1 directory (relative to .ignore)
level1/a.txt

# This recursively matches: c.text
**/c.txt

# This recursively matches files ending in: file
**/*file

# This matches only against the directory in which .ignore exists, negating from the recursive *file pattern above
!/rootedfile

# This allows any file recursively named b with a three-character extension
!**/b.???

We can easily evaluate whether or not files in the hierarchy are marked for inclusion or exclusion.

This example is included in the repo under examples/example and is a kotlin script:

#!/bin/sh
exec kotlins -cp `mvncp us.jimschubert:iggy:1.0.0` "$0" "[email protected]"
!#
// To run, see https://github.com/andrewoma/kotlin-script
// NOTE: This file will have apparent syntax issues in your IDE.
import us.jimschubert.iggy.IgnoreProcessor
import java.io.File

fun main(args: Array<String>) {
    val processor = IgnoreProcessor(".")
    val currentDirectory = File(".")

    fun walk(current: File) {
        current.listFiles().forEach { file ->
            if(file.isDirectory()) walk(file)
            else System.out.println("$file is allowed: ${processor.allowsFile(file)}")
        }
    }

    walk(currentDirectory)
}

Running this example gives the following output:

$ ./example
/Users/jim/bin/mvncp:267: warning: shadowing outer local variable - opts
./.ignore is allowed: true
./example is allowed: true
./level1/a.txt is allowed: false
./level1/level1file is allowed: false
./level1/level2/b.txt is allowed: true
./level1/level2/level2file is allowed: false
./level1/level2/level3/c.txt is allowed: false
./level1/level2/level3/level3file is allowed: false
./rootedfile is allowed: true

The API for IgnoreProcessor is pretty simple. First, you construct the class with the directory location of the ignore file (and optionally an ignore filename override). Then, you provide a file to IgnoreProcessor#allowsFile and it returns true or false based on your ignore file’s definitions.

You can also construct IgnoreProcessor with a File.

Notes

Unlike gitignore, iggy doesn’t traverse all directories to the defined hierarchy root. There is a single source of ignore definitions.

iggy uses java.nio.file.PathMatcher with conditionally defined glob patterns based on lines defined in your ignore file. There are some OS-specific things (with Windows, really) where case sensitive evaluation may be unintuitive.

Code

The official repository is on Gitlab at https://gitlab.com/jimschubert/iggy. There is a “mirror” on GitHub.

Coding against a composed docker environment

Microservices are pretty big these days. Unfortunately, they add complexity to development, deployment, and maintenance.

For instance, how do you code against multiple dependencies locally? What if your architecture gets larger and includes things like service discovery via consul, a database, long-lived TCP connections with a backend service? It would be a pain to spin all of those dependencies up locally, and it would be costly to run them all in the cloud for development. It is absolutely possible for your architecture to be too large to follow this pattern– fair warning.

It’s easy to put together a virtual network using Docker Compose which collects microservices into a single environment. Docker also makes it easy to do containerized development. I feel like this is a huge trend, and we’ll probably see IDEs or other tooling spring up to support this.

This isn’t a tutorial about Docker. If you’re not familiar with Docker, walk through some tutorials first. This is a quick example of connecting a development container to an existing docker network.

The example is taken from an example in jimschubert/sbt-scala, the repo for my SBT/Scala docker image.

QOTD Service/Client Example

This example will create a composed environment consisting of two Quote of the Day (QOTD) services. These services will exist on the same network, exposing port 17 only within the network. The client is the hypothetical piece under development which queries the QOTD from one (or both) services using only the internal host and port.

The QOTD service code is very simple, requiring only nmap-ncat and fortune (in alpine linux):

#!/bin/sh
exec ncat -l 17 --keep-open --send-only --exec "/foo.sh"

The script that generates the QOTD is also simple:

#!/bin/sh
echo $HOSTNAME: $(/usr/bin/fortune)

Check out the full Dockerfile. Remember, this post is more about the service/client communication within a docker composed network.

The key here is in how docker-compose.yml is defined:

version: '2'
services:
  fortune1:
    image: fortune
    hostname: fortune1
    mem_limit: 64MB
    expose:
     - "17"
    networks:
      - fortune
  fortune2:
    image: fortune
    hostname: fortune2
    mem_limit: 64MB
    expose:
     - "17"
    networks:
      - fortune
networks:
  fortune:
    driver: bridge

Each service is assigned to an internal network called fortune. This is how the development container will associate with these services. To run the services, you’d just build the Dockerfile which runs each service and execute

docker-compose up

Do that in one terminal, then open another terminal for the next part.

Client

The client is written in Scala. For simplicity, it is a command line tool which queries one or more hosts (on port 17) provided as an argument to the script. For kicks, consider how the following code (that is, querying against fortune1:17 and fortune2:17 from a single client) could be done without docker on a development machine.

Save this somewhere as src/Example.scala:

import java.net._
import java.io._
import scala.io._

object Example { 
    def qotd(server: String): Unit = {
        val s = new Socket(InetAddress.getByName(server), 17)
        val in = new BufferedSource(s.getInputStream()).getLines()
        println(in.next())
        s.close()
    }
    
    def main(args: Array[String]): Unit = args.foreach(qotd) 
}

If there’s a suitable public image available, you don’t need a Dockerfile for this code. Here’s how you can do local development against the above client:

docker run -it --net=fortune_fortune -v $(pwd)/src:/src --workdir=/src jimschubert/sbt-scala:latest

The docker run -it … jimschubert/sbt-scala:latest parts above just execute an interactive shell in an alpine linux container that I maintain for SBT/Scala development.

–net=fortune_fortune joins this container to the network created by the docker-compose.yml. The network is created automatically when you bring up the environment with docker-compose up. It’s in the format project_network where project is pulled from the directory name, COMPOSE_PROJECT_NAME environment variable or the optional docker-compose

-p, --project-name NAME

switch. If you’re not running against the code from github, you can run

docker network list

to get the generated network name.

-v $(pwd)/src:/src mounts the src directory relative to where you’ve run the command to the /src directory within the container.

–workdir=/src is here for convenience, so you’re starting in the right folder in the container.

Once run the full command and you’re in the container, you can execute the client directly:

scala Example.scala fortune1 fortune2 fortune1

Go ahead and add a println(server) to the top of the qotd method in Example.scala and run the command again. No need for restarting the container!

Conclusion

This post demonstrates how to do containerized development of a simple client. There are multiple instances of the QOTD service (port 17) consumed by the client running on a single development machine.

The ports of these services can’t be exposed to port 17 on the host machine because this would result in a port conflict. It doesn’t make a lot of sense to remap port 17 to some other port on the host because port 17 is an explicitly defined port for QOTD. There may be issues if you do attempt to remap the port, like what if you choose a port that is randomly acquired by some other application?

Running the client in a container allows us to access the services as they would exist in the wild. Mapping the src directory from the host to the container allows us to quickly iterate on code without restarting the container. This is important, especially if you’re using a much larger base image than alpine.

If you’re doing Scala development, a recommended next step would be to use the spray/revolver plugin. Once setup, you would just start an sbt REPL and run ~re-start. Any local code changes are automatically detected and the plugin restarts the JVM process running your application.

Ignoring files with Swagger Codegen

I’m proud to announce that I’m now a member of the Swagger Codegen Core team.

I’ve previously refactored the C# client, the maven plugin, added an ASP.NET 5 Web API server generator, and fixed some bugs.

I recently added support for .swagger-codegen-ignore. A few weeks ago, I updated the C# client project structure to be a more standard structure:

    .
    ├── IO.Swagger.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

With this breaking change and the .swagger-codegen-ignore, users can now generate into existing repositories. Suppose your project is setup this way:

    .
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        └── MyProject.Test
            └── packages.config

You may want your generated client code to be output under src/MyProject.Client, but you don’t want build.bat, build.sh, or docs to get blown away. Last week, I partially resolved this by adding options to disable api and model doc generation for all languages… but there was no clean way to ignore other files like build.bat or build.sh.

Now, you can add the following .swagger-codegen-ignore to your existing directory structure:

# Swagger Codegen Ignore
/README.md
/build.*
docs/
src/
!src/MyProject.Client/
!src/MyProject.Client.Test/

Just to be safe (this feature is brand new), commit and push your changes before running swagger-codegen using your repository as the output directory. You should end up with the following structure:

    .
    ├── .swagger-codegen-ignore
    ├── IO.Swagger.sln
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        ├── MyProject.Test
        │   └── packages.config
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

As expected, the only generated content should be the IO.Swagger.sln, src/IO.Swagger and src/IO.Swagger.Test.

All of the functionality in this post should land in version 2.2.0. Until then, you can build directly against the master branch for these features.

Create configurable Angular services using providers

With Angular 2 on its way, it seems like a lot of developers are now afraid of Angular 1.x. I have nothing bad to say about Angular. Sure, some things could be made easier but that’s why Angular 2 is on its way. I’ve played with Angular 2 a bit, and it only works in Chrome. I’m fairly confident Angular 1.x will be around for quite a while.

In that light, let’s talk about creating configurable services. One reason why AngularJS is so popular is because you can easily create modular, reusable code. This code can encapsulate logic, directives, extensible services and factories (via decorators), as well as configurability. I’ll talk about creating directives and decorators in future articles.

This example will be a simple URL constructor for placekitten.com. Any AngularJS service can be written to be configurable (if there’s reason to configure the service, of course).

First of all, let’s see how this service will be used.

.controller('View1Ctrl', ['$scope', 'kittyService', function ($scope, kittyService) {
    $scope.kittycat = kittyService.getUrl();
}])
  <p>How about a kitty cat?</p>
  <p><img ng-src="{{kittycat}}"/></p>

Pretty simple. We’ll call kittyService.getUrl() and use the string returned from the service as the src for an image.

To create a configurable service, we’ll use an angular provider instead of a service or factory. The structure I use for a provider is this:

(function () {
    'use strict';
    var module = angular.module('myApp.services', []);

    module.provider('serviceName', function ServiceName() {
        // use `this` for configurable properties or functions

        this.$get = [function () {
            var service = {};
            // This can be any valid return that you'd normally use
            // with module.service or module.factory
            return service;
        }];
    });
})();

This structure is only a little different from a service or factory. Rather than return an object or constructor function, you use this.$get to create your reusable code. What you assign to this property is the array injector for your service definition. You’ll feel comfortable with this syntax if you define your services using the array-based dependency injection:

angular.module('myApp')
.service('serviceName', [function (){
            var service = {};
            // This can be any valid return that you'd normally use
            // with module.service or module.factory
            return service;
}]);

You can use instance properties and methods on the provider definition to configure your service, then use the private context of your provider to share the configuration with the service. Here’s the full code for the kittyService:

(function () {
    'use strict';
    var module = angular.module('myApp.services', []);

    module.provider('kittyService', function KittyServiceProvider() {
        var height = 100,
            width = 100;

        this.setHeight = function (h) {
            height = h;
        };
        this.setWidth = function (w) {
            width = w;
        };

        this.$get = [function () {
            var service = {};

            service.getUrl = function () {
                return 'http://placekitten.com/g/' + width + '/' + height;
            };

            // This can be any valid return that you'd normally use
            // with module.service or module.factory
            return service;
        }];
    });
})();

The provider has two private variables: height and width. The two methods on the provider object allow you to update these values, but only during the config phase. After your application’s config function has completed, your provider’s functions are no longer accessible.

One problem with this service is that placekitten.com shows the same image for a given combination of height and width. In the config phase of our application, the service’s height and width can be configured randomly:

.config(['kittyServiceProvider', function (kittyServiceProvider) {
    var widths = [200,300,400,500,600];
    var heights = [300,400,500,600,700];
    kittyServiceProvider.setHeight(heights[Math.floor(Math.random()*heights.length)]);
    kittyServiceProvider.setWidth(widths[Math.floor(Math.random()*widths.length)]);
}]);

The name injected into the config phase is kittyServiceProvider and not kittyService. This is important. AngularJS knows that when you request kittyServiceProvider in the config phase, you want the object containing your service definition (the KittyServiceProvider function in the example above). Once the config phase completes, the $injector service will prepare your service from the this.$get definition of your service.

If you load the example code, refresh the page a few times and you’ll receive a new kitty image.

NOTE: The heights and widths may not always combine in a way that retrieves an image. Just refresh for another example.

Why not do this in a service?

True, you could do this directly in the service if this was just your code. The benefit here is when you want to share a service with other developers, perhaps on a different project within your company or publicly on GitHub. What if your team wants a 200×300 kitty and HR wants a 500×500 kitty? In these situations you can’t just modify the code with your hard-coded values. True, in this example you could use constants or values instead but then your service would have a dependency on the consumer (this is bad).

A service that dumps a string for displaying a cat image is probably not that useful in the real world. Consider other possibilities. I’ve used providers for creating a reusable notification display which allows you to define whether a notification will automatically close and how long notifications will be displayed if they auto-close. Suppose your application often receives 40x errors when users leave their browsers unattended long enough for an authenticated session to time out; you could create a provider which caches failed requests and retries up to X (configurable) times before giving up. Or, you could create a configurable service which caches those failed requests, displays a login dialog, then automatically retries the failed requests X (configurable) milliseconds after logging in successfully. Those examples would require the use of an http interceptor, and would have been quite a bit more complex than a kitty image linker.

Code

Sample code for this blog post is available on github at jimschubert/blogs.

Further Reading

If you really want to dig into the guts of Providers (versus services, factories, constants, or values), check out Adrian Castillo’s blog post, AngularJS Providers under the hood.

For reference to each recipe, check out AngularJS Provider Recipes on the official docs.

Your IDE’s font matters: Fantasque Sans Mono

A good development environment is crucial to your productivity, there’s no question about it. The phrase ‘good development environment’ is subjective. For example, I’m pretty efficient using WebStorm and VisualStudio. I use vim pretty often but I’m nowhere near ‘efficient’. Matias Niemelä, on the other hand can easily manipulate text in vim.

A year or two ago, I read a blog post somewhere that suggested a font is just as important as your tool of choice. I thought, “yeah, right!”, but downloaded the font anyway. After a day of using the font, I realized how much more quickly I could read through code and how much easier it was to visually identify code blocks. I know that sentence sounds trivial, but after two years of using what I consider a better font I definitely notice a difference when I’m not using the font.

My font of choice these days is called Fantasque Sans Mono.

Here’s a screenshot comparison of Arial and Fantasque Sans Mono, both at 14pt.

Atom: arial 14pt
Atom: arial 14pt
Atom: fantasque sans mono 14pt
Atom: fantasque sans mono 14pt

The first thing to notice is how uncluttered letters are in Fantasque. On line 17, the word ‘function’ feels like it runs together in Arial but the letters clearly stand out in Fantasque. Also notice how the letter ‘g’ is unmistakable on line 18. On line 21, the kerning between parentheses, braces, brackets, and vertical pipes all make the line of code slightly easier to read. On line 23, the zero in Arial and many other fonts often resembles the capital letter ‘O’. For literals, this is obviously a number but it can be an easy typo to overlook within a string. In Fantasque, a zero has a line through it to clearly differentiate the number from the letter. Line 25 is easily consumed in the Arial screenshot mostly because the Atom editor clearly distinguishes between patterns and operations in the regex literal. Take a look at the difference without JavaScript syntax highlighting:

Atom: arial 14pt plain text
Atom: arial 14pt plain text
Atom: fantasque sans mono 14pt
Atom: fantasque sans mono 14pt

The actual code tends to be wider than other fonts. In the Arial screenshot, the image stops at the 80 character vertical indicator, which can easily be seen in the Fantasque screenshot. I actually like to use a 12pt or 13pt font size with Fantasque, which gives me a little more code on screen.

Although these points may seem trivial, I challenge you to at least give a few different fonts a try. You’ll be as surprised as I was to find that subtle differences in your font can improve productivity.

Download Fantasque Sans Mono from OpenFontLibrary.

Manually debugging Angular scope

While a utility like Angular Batarang is invaluable, you’ll sometimes want to quickly and effectively inspect what exists on a given scope. If you’ve used Angular Batarang, you know that can be a pain in the ass. Here’s a quick alternative that works in Google Chrome, Firefox, and Safari.

Right-click on some element you want to inspect in your AngularJS-backed page. Select Inspect Element. This will open an ‘Elements’ inspector tab. Leave the element selected in that tab and go to ‘Console’. Type out the following:

angular.element($0).scope();

This will dump the scope to the console. You can assign this to a variable or navigate the parent/child scopes from here. This works because most browsers’ developer consoles support some extras in their Command Line API. The $0 refers to the most recently selected DOM element (try $0 through $4!).

You can test this out on the AngularJS Phone Catalog Tutorial. Here’s what you should see:

Android element debugging.

Data at the root level is invalid. Line 1, position 1.

Recently, I encountered a really weird problem with an XML document. I was trying to load a document from a string:

var doc = XDocument.parse(someString);

I received this unhelpful exception message:

Data at the root level is invalid. Line 1, position 1.

I verified the XML document and retried two or three times with and without the XML declaration (both of which should work with XDocument). Nothing helped, so I googled for an answer. I found the following answer on StackOverflow by James Brankin:

I eventually figured out there was a byte mark exception and removed it using this code:

string _byteOrderMarkUtf8 = Encoding.UTF8.GetString(Encoding.UTF8.GetPreamble());
if (xml.StartsWith(_byteOrderMarkUtf8))
{
    xml = xml.Remove(0, _byteOrderMarkUtf8.Length);
}

This solution worked. I was happy. I discussed it with a coworker and he had never heard of a BOM character before, so I thought “I should blog about this”.

Byte-Order Mark

The BOM is the character returned by Encoding.UTF8.GetPreamble(). Microsoft’s documentation explains:

The Unicode byte order mark (BOM) is serialized as follows (in hexadecimal):

  • UTF-8: EF BB BF
  • UTF-16 big endian byte order: FE FF
  • UTF-16 little endian byte order: FF FE
  • UTF-32 big endian byte order: 00 00 FE FF
  • UTF-32 little endian byte order: FF FE 00 00

Converting these bytes to a string (Encoding.UTF8.GetString) allows us to check if the xml string starts with the BOM or not. The code then removes that BOM from the xml string.

A BOM is a bunch of characters, so what? What does it do?

From Wikipedia:

The byte order mark (BOM) is a Unicode character used to signal the endianness (byte order) of a text file or stream. It is encoded at U+FEFF byte order mark (BOM). BOM use is optional, and, if used, should appear at the start of the text stream. Beyond its specific use as a byte-order indicator, the BOM character may also indicate which of the several Unicode representations the text is encoded in.

This explanation is better than the explanation from Microsoft. The BOM is (1) an indicator that a stream of bytes is Unicode and (2) a reference to the endianess of the encoding. UTF8 is agnostic of endianness (reference), so the fact that the BOM is there and causing problems in C# code is annoying. I didn’t research why the UTF8 BOM wasn’t stripped from the string (XML is coming directly from SQL Server).

What is ‘endianness’?

Text is a string of bytes, where one or more bytes represents a single character. When text is transferred from one medium to another (from a flash drive to a hard drive, across the internet, between web services, etc.), it is transferred as stream of bytes. Not all machines understand bytes in the same way, though. Some machines are ‘little-endian’ and some are ‘big-endian’.

Wikipedia explains the etymology of ‘endianness’:

In 1726, Jonathan Swift described in his satirical novel Gulliver’s Travels tensions in Lilliput and Blefuscu: whereas royal edict in Lilliput requires cracking open one’s soft-boiled egg at the small end, inhabitants of the rival kingdom of Blefuscu crack theirs at the big end (giving them the moniker Big-endians).

For text encoding, ‘endianness’ simply means ‘which end goes first into memory’. Think of this as a direction for a set of bytes. The word ‘Example’ can be represented by the following bytes (example taken from StackOverflow):

45 78 61 6d 70 6c 65

‘Big Endian’ means the first bytes go first into memory:

45 78 61 6d 70 6c 65
<-------------------

‘Little Endian’ means the text goes into memory with the small-end first:

45 78 61 6d 70 6c 65
------------------->

So, when ‘Example’ is transferred as ‘Big-Endian’, it looks exactly as the bytes in the above examples:

45 78 61 6d 70 6c 65

But, when it’s transferred in ‘Little Endian’, it looks like this:

65 6c 70 6d 61 78 45

Users of digital technologies don’t need to care about this, as long as they see ‘Example’ where they should see ‘Example’. Many engineers don’t need to worry about endianness because it is abstracted away by many frameworks to the point of only needing to know which type of encoding (UTF8 vs UTF16, for example). If you’re into network communications or dabbling in device programming, you’ll almost definitely need to be aware of endianness.

In fact, the endianness of text isn’t constrained by the system interacting with the text. You can work on a Big Endian operating system and install VoIP software that transmits Little Endian data. Understanding endianness also makes you cool.

Summary

I don’t have any code to accompany this post, but I hope the discussion of BOM and endianness made for a great read!