Coding against a composed docker environment

Microservices are pretty big these days. Unfortunately, they add complexity to development, deployment, and maintenance.

For instance, how do you code against multiple dependencies locally? What if your architecture gets larger and includes things like service discovery via consul, a database, long-lived TCP connections with a backend service? It would be a pain to spin all of those dependencies up locally, and it would be costly to run them all in the cloud for development. It is absolutely possible for your architecture to be too large to follow this pattern– fair warning.

It’s easy to put together a virtual network using Docker Compose which collects microservices into a single environment. Docker also makes it easy to do containerized development. I feel like this is a huge trend, and we’ll probably see IDEs or other tooling spring up to support this.

This isn’t a tutorial about Docker. If you’re not familiar with Docker, walk through some tutorials first. This is a quick example of connecting a development container to an existing docker network.

The example is taken from an example in jimschubert/sbt-scala, the repo for my SBT/Scala docker image.

QOTD Service/Client Example

This example will create a composed environment consisting of two Quote of the Day (QOTD) services. These services will exist on the same network, exposing port 17 only within the network. The client is the hypothetical piece under development which queries the QOTD from one (or both) services using only the internal host and port.

The QOTD service code is very simple, requiring only nmap-ncat and fortune (in alpine linux):

#!/bin/sh
exec ncat -l 17 --keep-open --send-only --exec "/foo.sh"

The script that generates the QOTD is also simple:

#!/bin/sh
echo $HOSTNAME: $(/usr/bin/fortune)

Check out the full Dockerfile. Remember, this post is more about the service/client communication within a docker composed network.

The key here is in how docker-compose.yml is defined:

version: '2'
services:
  fortune1:
    image: fortune
    hostname: fortune1
    mem_limit: 64MB
    expose:
     - "17"
    networks:
      - fortune
  fortune2:
    image: fortune
    hostname: fortune2
    mem_limit: 64MB
    expose:
     - "17"
    networks:
      - fortune
networks:
  fortune:
    driver: bridge

Each service is assigned to an internal network called fortune. This is how the development container will associate with these services. To run the services, you’d just build the Dockerfile which runs each service and execute

docker-compose up

Do that in one terminal, then open another terminal for the next part.

Client

The client is written in Scala. For simplicity, it is a command line tool which queries one or more hosts (on port 17) provided as an argument to the script. For kicks, consider how the following code (that is, querying against fortune1:17 and fortune2:17 from a single client) could be done without docker on a development machine.

Save this somewhere as src/Example.scala:

import java.net._
import java.io._
import scala.io._

object Example { 
    def qotd(server: String): Unit = {
        val s = new Socket(InetAddress.getByName(server), 17)
        val in = new BufferedSource(s.getInputStream()).getLines()
        println(in.next())
        s.close()
    }
    
    def main(args: Array[String]): Unit = args.foreach(qotd) 
}

If there’s a suitable public image available, you don’t need a Dockerfile for this code. Here’s how you can do local development against the above client:

docker run -it --net=fortune_fortune -v $(pwd)/src:/src --workdir=/src jimschubert/sbt-scala:latest

The docker run -it … jimschubert/sbt-scala:latest parts above just execute an interactive shell in an alpine linux container that I maintain for SBT/Scala development.

–net=fortune_fortune joins this container to the network created by the docker-compose.yml. The network is created automatically when you bring up the environment with docker-compose up. It’s in the format project_network where project is pulled from the directory name, COMPOSE_PROJECT_NAME environment variable or the optional docker-compose

-p, --project-name NAME

switch. If you’re not running against the code from github, you can run

docker network list

to get the generated network name.

-v $(pwd)/src:/src mounts the src directory relative to where you’ve run the command to the /src directory within the container.

–workdir=/src is here for convenience, so you’re starting in the right folder in the container.

Once run the full command and you’re in the container, you can execute the client directly:

scala Example.scala fortune1 fortune2 fortune1

Go ahead and add a println(server) to the top of the qotd method in Example.scala and run the command again. No need for restarting the container!

Conclusion

This post demonstrates how to do containerized development of a simple client. There are multiple instances of the QOTD service (port 17) consumed by the client running on a single development machine.

The ports of these services can’t be exposed to port 17 on the host machine because this would result in a port conflict. It doesn’t make a lot of sense to remap port 17 to some other port on the host because port 17 is an explicitly defined port for QOTD. There may be issues if you do attempt to remap the port, like what if you choose a port that is randomly acquired by some other application?

Running the client in a container allows us to access the services as they would exist in the wild. Mapping the src directory from the host to the container allows us to quickly iterate on code without restarting the container. This is important, especially if you’re using a much larger base image than alpine.

If you’re doing Scala development, a recommended next step would be to use the spray/revolver plugin. Once setup, you would just start an sbt REPL and run ~re-start. Any local code changes are automatically detected and the plugin restarts the JVM process running your application.

Flattr this!

Finding Wakelocks in Android 7 using Google’s Battery Historian

One of my favorite apps prior to my upgrade to Android 7 (Nougat) was GSam Battery Monitor (the paid version is fully worth it). Unfortunately, Android 7 no longer allows apps like GSam Battery Monitor to track individual application battery usage, CPU usage, or wakelocks. That sucks*.

In Android 7 with Doze, you should see periods of marginal battery usage. The more apps you have, the more background services/sync operations/location broadcasts are running and you’ll see plateaus in battery usage be more slanted. Up until a few weeks ago, battery usage would basically flatline between screen time. I set out to figure out what was causing this. I found a thread on Reddit discussing Battery Historian, so I decided to check it out. This post is somewhat of a review of this tool.

Battery Historian analyzes bug report files and presents the information in an interactive graph with categorized breakdowns of battery usage. Battery Historian is a web application written in Go, and doesn’t appear to have an official release (binary or hosted). This means it’s basically limited to power users to evaluate. Since it’s licensed under Apache 2.0, I compiled and hosted this for my own use at https://historian.jimschubert.us/. Before using my hosted instance of Battery Historian, please heed Google’s warning about bug report files:

Bug reports contain data from the system’s various log files, including personal and private information. Only share bug reports with apps and people you trust.

Running Battery Historian locally isn’t too difficult. I definitely recommend going this route if you’re more of a power user; you don’t need programming experience, only comfort on the command line.

Once you’ve taken a bug report on your device and uploaded it to Battery Historian, you’re presented with a graph of all activity found in the bug report.

battery_historian

The declining black line is the same battery usage visualization you’d see in your Battery Usage application or widget. Other groups of battery draining events are presented as horizontal time-oriented lines; some are either on or off (screen), some show gradients for battery draining in the category (temperature), while others display different colors for different states of the event (Mobile signal strength).

Scroll below the graph and you’ll find a breakdown of system statistics. After a little sorting and investigation, I found that JuiceSSH had locked WiFi for 1.5 hours when I hadn’t used the application for over a week:

battery_historian_-_juicessh_wifi_lock

After selecting the JuiceSSH app in the left dropdown, this takes you to the App Stats tab which presents some interesting battery usage statistics:

Wakelocks:
Wakelock Name      | Full Time | Full Count | Partial Time | Partial Count | Window Time | Window Count
JuiceSSH CloudSync |           | 0          | 8m 0s 50ms   | 1             |             | 0           
*alarm*            |           | 0          | 684ms        | 1             |             | 0           
Services:
Service Name                            | Time spent started | # starts | # launches
com.sonelli.juicessh.services.CloudSync | 2h 28m 15s 789ms   | 1        | 1         
Process info:
Time spent running actively in the foreground       | 0ms             
Time spent with a service running in the foreground | 0ms             
Time spent on top                                   | 0ms             
Time spent on top while the device was sleeping     | 0ms             
Time spent running actively in the background       | 1h 33m 10s 343ms
Time spent cached                                   | 38ms            

Looks like, in my case, JuiceSSH Cloud Sync had 8 minutes worth of wakelocks, 2.5 hours of a service running in which 1.5 hours of that was actively running in the background. JuiceSSH unfortunately doesn’t have configuration options for Cloud Sync, so I’ve disabled it. I’ve noticed an improvement since disabling JuiceSSH Cloud Sync.

Although I was a little miffed that the Android 7 update borked GSam’s ability to report on per-application usage, I found Battery Historian to be a really insightful tool. From and Android Development perspective it would be a nice utility to evaluate your application’s effect on the system through normal use. It may be a little overkill from a user or power user’s perspective, though. My Nexus 5X was getting about 18 hours of pretty heavy usage (including my son watching YouTube videos). After disabling JuiceSSH Cloud Sync, I’m seeing about 20 hours of battery out of the same usage. Does that really matter if you’re like me and you charge your phone every night? 👻

* I’ve read that GSam Battery Monitor with root still works as expected on Nougat.

Flattr this!

Ignoring files with Swagger Codegen

I’m proud to announce that I’m now a member of the Swagger Codegen Core team.

I’ve previously refactored the C# client, the maven plugin, added an ASP.NET 5 Web API server generator, and fixed some bugs.

I recently added support for .swagger-codegen-ignore. A few weeks ago, I updated the C# client project structure to be a more standard structure:

    .
    ├── IO.Swagger.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

With this breaking change and the .swagger-codegen-ignore, users can now generate into existing repositories. Suppose your project is setup this way:

    .
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        └── MyProject.Test
            └── packages.config

You may want your generated client code to be output under src/MyProject.Client, but you don’t want build.bat, build.sh, or docs to get blown away. Last week, I partially resolved this by adding options to disable api and model doc generation for all languages… but there was no clean way to ignore other files like build.bat or build.sh.

Now, you can add the following .swagger-codegen-ignore to your existing directory structure:

# Swagger Codegen Ignore
/README.md
/build.*
docs/
src/
!src/MyProject.Client/
!src/MyProject.Client.Test/

Just to be safe (this feature is brand new), commit and push your changes before running swagger-codegen using your repository as the output directory. You should end up with the following structure:

    .
    ├── .swagger-codegen-ignore
    ├── IO.Swagger.sln
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        ├── MyProject.Test
        │   └── packages.config
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

As expected, the only generated content should be the IO.Swagger.sln, src/IO.Swagger and src/IO.Swagger.Test.

All of the functionality in this post should land in version 2.2.0. Until then, you can build directly against the master branch for these features.

Flattr this!

Bash function: recent

#!/bin/env bash

# Display recently changed files recursively below the current directory
# recent [minutes]
#    minutes: The number of minutes to display. Default 5.
# 
# Credit to find usage goes to
# F. Hauri (http://superuser.com/users/178656/f-hauri)
# see: http://superuser.com/a/621727/60571
function recent {
   local mins="-$((${1:-5}+0))"
    
   find . -type f -mmin $mins -print0 | xargs -0 /bin/ls -ltr
}

Example:

jim at macbert in ~/projects/swagger-codegen on cs/req_properties_2584 705ed78
$ recent
-rw-r--r--@ 1 jim  staff  6148 May  8 11:05 ./samples/client/petstore/csharp/SwaggerClient/.DS_Store
jim at macbert in ~/projects/swagger-codegen on cs/req_properties_2584 705ed78
$ recent 120
-rw-r--r--  1 jim  staff     149 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/obj/Debug/.NETFramework,Version=v4.5.AssemblyAttribute.cs
-rwxr-xr-x  1 jim  staff  131072 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/bin/Debug/Swagger Library.dll
-rwxr-xr-x  1 jim  staff  131072 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/obj/Debug/Swagger Library.dll
-rwxr-xr-x  1 jim  staff  131072 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/bin/Debug/Swagger Library.dll
-rw-r--r--  1 jim  staff   44083 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/bin/Debug/Swagger Library.dll.mdb
-rw-r--r--  1 jim  staff   44083 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/obj/Debug/Swagger Library.dll.mdb
-rw-r--r--  1 jim  staff     940 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/obj/Debug/IO.Swagger.csproj.FilesWrittenAbsolute.txt
-rw-r--r--  1 jim  staff   44083 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClient/src/IO.Swagger/bin/Debug/Swagger Library.dll.mdb
-rw-r--r--  1 jim  staff    4365 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/obj/Debug/SwaggerClientTest.dll.mdb
-rwxr-xr-x  1 jim  staff   28160 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/obj/Debug/SwaggerClientTest.dll
-rw-r--r--  1 jim  staff    2875 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/obj/Debug/SwaggerClientTest.csproj.FilesWrittenAbsolute.txt
-rw-r--r--  1 jim  staff    4365 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/bin/Debug/SwaggerClientTest.dll.mdb
-rwxr-xr-x  1 jim  staff   28160 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/bin/Debug/SwaggerClientTest.dll
-rw-r--r--  1 jim  staff    8467 May  8 09:11 ./samples/client/petstore/csharp/SwaggerClientTest/TestResult.xml
-rw-r--r--  1 jim  staff  431527 May  8 09:11 ./.git/index
-rw-r--r--@ 1 jim  staff    6148 May  8 11:05 ./samples/client/petstore/csharp/SwaggerClient/.DS_Store

The gist is available here.

Flattr this!

Docker Kitematic (Beta) and iTerm2 3.0 Beta

I guess using everything in beta can cause issues, but whatever.

Kitematic has an ‘Exec’ option for containers which opens a terminal connected to that container. If you have iTerm2 v3.0 beta installed, you’ve probably noticed this opens in Terminal.app instead of iTerm2.

The button in Kitematic executes a bash script at

/Applications/Docker/Kitematic (Beta).app/Contents/Resources/resources/terminal

. The bash script executes AppleScript, and the AppleScript syntax for iTerm2 in 3.0 has changed.

Here’s a replacement for that terminal bash script, with the new iTerm2 syntax (minus extra error handling).

#!/bin/bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
CMD="clear && $*"

ITERM_EXISTS=`osascript <<EOF
set doesExist to false
try
    do shell script "osascript -e 'exists application \"iTerm2\"'"
    set doesExist to true
end try
return doesExist
EOF`

function open_iterm () {
  osascript > /dev/null <<EOF
  tell application "iTerm2"
    activate
    set newWindow to (create window with default profile)
    select first window

    tell current window
      tell current session
        write text "bash -c \"$CMD\""
      end tell
    end tell
  end tell
EOF
}

function open_terminal () {
  osascript > /dev/null <<EOF
  tell application "Terminal" to activate
  delay 0.4
  tell application "System Events" to keystroke "t" using command down
  tell application "Terminal"
    do script "bash -c \"$CMD\"" in window 1
  end tell
EOF
}

if [ "$ITERM_EXISTS" == "true" ]; then
  open_iterm "$@" || open_terminal "$@"
else
  open_terminal "$@"
fi

This may not be 100% correct, but it works.

Flattr this!

ES6 Template Strings without Indent via tags

ES6 is pretty cool. One new feature is template strings which allow for multiline interpolated strings.

For example, from MDN:

console.log(`string text line 1
string text line 2`);
// "string text line 1
// string text line 2"

They also allow for evaluated variables embedded within the string (again from MDN):

var a = 5;
var b = 10;
console.log(`Fifteen is ${a + b} and\nnot ${2 * a + b}.`);
// "Fifteen is 15 and
// not 20."

The problem is that you may be using these strings within a class that already has indentation.

export class App {
    text =`
       This is an example of what could be considered a 
       "problem" with ES6 multiline strings.
       
       Each of these lines will have 7 characters of whitespace
       at the beginning of the line.`;
    }
}

This is probably not what you want in every situation.

Luckily, ES6 supports something called tagged template strings. Using a special function syntax, you can hook into the template functionality and either change how the template is evaluate or modify the output of the template. You can use these tagged template strings to achieve something like Scala’s stripMargin function. If you’re not familiar with stripMargin, you basically put a pipe (the | character) at the start of the line and indentation before the pipe is ignored. You can achieve this with tag template that performs a regex replace:

Here’s how this baby works:

See the Pen xZZmVV by Jim Schubert (@jimschubert) on CodePen.

Flattr this!

Google Now, the new NSA

I’m going to make a bold claim here, but I don’t care. Google Now is essentially a mechanism to spy on people and I think everyone should disable the feature immediately.

I’ve had a lot of respect for Google over the years since I started using Google Search back in 1998. I really appreciated the “do no evil” mantra, as well as the focus on open source technology and giving back to the community.

Then something happened and a lot of free services started getting cancelled. Next there seemed to be a huge push toward paid goods and services (Play Store, Play Music, Fiber, Fi, and now YouTub Red). Meanwhile many of the open source technologies seem to be self-serving– sure I like AngularJS and am intrigued by Go, but I have no reason to believe Google won’t drop these like hot potatoes.

Despite the progression of the company, back in 2012 I decided to try my hand at interviewing with Google. It literally started as the rudest interview I’ve had to date. At the time, I had only had .NET programming experience in my professional career. The interviewer obviously hadn’t researched me as a candidate to understand that my passion lies in Unix-based operating systems and open source technologies*. The interviewer’s introduction went like this:

So… you’re a .NET Developer? *3s long sigh* Uh.. yeah.. I guess let’s get started.

* After an 18 month hiatus, I returned to a .NET position solely because Microsoft announced plans to open source .NET. That’s how serious I am about it.

I’m sure there’s a bad egg or two at Google, but my experiences haven’t left a positive impression on me. Then, this happened…

I’m a software architect, working on a new web module for club management software. On October 6, I entered some tasks into our project management system by listing out a rough model of some things to code. A few of these tasks included variations on the word “slots”, since this is a module for scheduling players to tees at a golf course.

On October 22, I got around to the task related to implementing slots. I looked over my initial model and felt like there was something missing, so I discussed it verbally with a veteran programmer at the company. This was a 15 minute, in person conversation in which the word “slot” probably came up 50 or 60 times. Later that night, while browsing Google’s Play Store on my phone:

Google is always listening.
Google is always listening.

What. The. Hell.

I immediately texted that screenshot to the guy I talked with that day. We both thought it was weird and a little creepy.

I had previously noticed ads targeted based on things I knew I had conversations about, but thought little of it. “I’m sure I googled it if I was talking about it,” I’d tell myself. This time, I’m 100% positive I hadn’t googled anything about a ‘slot’. The concept of a ‘slot’ in the system I’m working with is an abstract way to relate a time (07:15am) to a tee (Tee 1) on an certain day (10/22/2015). I would have had about as much reason to search for ‘slot’ as I would have to search for ‘turkey sandwich’. Just to be sure, I reviewed my search history at https://www.google.com/history as well as locally in my browser. As I thought, there was no recent activity for the word ‘slot’ aside from 10-15 entries on 10/16 about an Amazon deal of the day.

Google-History-Search-Slot

Being a technologist has its perks. I had recently installed CyanogenMod on my phone and it exposes functionality to individually manage permissions for an app. I went to Settings -> Apps -> Google App and clicked the ‘Modify’ button. I then disabled permissions for the microphone (which at the time had been allowed thousands of times). I started watching TV to let my phone sit with the screen off. After 19 minutes, I checked back and the Google App microphone permission had been denied 43 times. In 19 minutes.

I left the ‘OK Google’ detection enabled on my phone for the next day to see how much Google was actually listening to me. This turned out to be 1185 times.

2015-10-24 15.20.01

But, this whole “Google is a bad guy” thing is just conspiracy theory, right?

Consider the fact that within the past year, Google introduced “Google Now” into their Chrome browser. Earlier this year, users became outraged to find out a secret component in the browser allowed Google to listen to everything going on in your home or office. Google’s response was that they don’t care about your conversations and that the recording is only available on google.com or on the new tab page of the browser. Basically, “Don’t worry about it.”

But then, why do they hold a patent for delivering ads based on environmental conditions and background noise? The first image of the patent clearly displays a “client terminal” as a browser or mobile device.

There are a few issues with the Google Now service and the patent that Google holds for targeting ads based on my ‘environmental conditions’. First of all, I opted into letting Google listen to my microphone at regular intervals for “OK Google”. I don’t deny that. I did not opt into Google using my conversations while my phone is off to deliver personalized ads. Maybe I’m wrong and these targeted ads weren’t based on audio recorded while my phone was laying, turned off, on my desk at work while I was having a conversation? If Google Chrome is the culprit, the same issue exists. I’ve opted into Google targeting ads based on my search history. I did not opt into Google intercepting every keystroke in my browser (especially not for private intranet sites) to target me with ads.

Google’s service is apparently still a part of the Chrome browser (navigate to chrome://voicesearch and see for yourself). However, I don’t see a way to disable this. From what I can tell in searching it seems like Google has at least temporarily removed Google Now functionality from Chrome. I can’t tell whether or not the browser has access to randomly record audio while the browser is open.

I wouldn’t be so bothered by this, except that Google hasn’t been completely transparent about what’s being recorded from my phone or browser and transferred to their servers. What if I worked in the US Government? How could Google not be held accountable for their actions when the NSA’s actions have been deemed unconstitutional?

You can disable lots of Google’s search and ad targeting services in your Account Settings. I link to it directly because, as you probably guess, it’s not exactly easy to find.

Flattr this!

Packtpub.com Free Learning

Over at Packtpub.com, they’re offering a free book every day as part of a free learning initiative.

They’ve had some good books in the past for free (R Starter, Node Web Development, Mastering NInject for Dependency Management). You’ve missed those (Sorry!), but there are plenty more coming out each day. They’re only available for 24 hours each.

Click the link below.

FreeLearning

Flattr this!

Developer James Schubert shares his code and his thoughts.