Tag Archives: Advanced

Prototyping a Flags/Bitmasks implementation in Kotlin 1.1.1

One of my more favorite features of C# is the use of FlagsAttribute. If you’re not familiar with the concept, [Flags] allows you to treat an integral enum type as a bitmask. In other words, you can store 64 different on/off settings in a 64-bit integer.

The example from MSDN shows the storage of Tuesday + Thursday in a variable called meetingDays:

    [Flags]
    enum Days2
    {
        None = 0x0,
        Sunday = 0x1,
        Monday = 0x2,
        Tuesday = 0x4,
        Wednesday = 0x8,
        Thursday = 0x10,
        Friday = 0x20,
        Saturday = 0x40
    }
    class MyClass
    {
        Days2 meetingDays = Days2.Tuesday | Days2.Thursday;
    }

In C#, the operators are built into the compiler, so meetingDays becomes an integer with the 3rd and 5th bits set. With too many flags options, it can become difficult to keep track of the values, so I prefer using a left shift operator:

    [Flags]
    enum Days2
    {
        None = 0,
        Sunday = 1 << 0,
        Monday = 1 << 1,
        Tuesday = 1 << 2,
        Wednesday = 1 << 3,
        Thursday = 1 << 4,
        Friday = 1 << 5,
        Saturday = 1 << 6
    }
    
    class MyClass
    {
        public Days2 meetingDays = Days2.Tuesday | Days2.Thursday;
    }
    

Because Kotlin doesn’t support flags directly, to add programmatic support for this, we’ll need to implement some operators or infix operators. Unfortunately, as of right now Kotlin doesn’t support overloading the bitwise OR (|) operator. A workaround is to use a specialized operator that means “set”, or to explicitly use the named “or” operator.

The below example doesn’t support all bitwise operators, but it should be enough to get a project moving forward with a generalized Flags implementation.

fun main(args : Array<String>){
    val mask: BitMask = ParameterFeature.Path + 
        ParameterFeature.Query + 
        ParameterFeature.Header;
    
    val enabled = enabledValues<ParameterFeature>(mask)
    
    println("flags enabled: $enabled")
    println("flags disabled: ${enumValues<ParameterFeature>().filterNot { enabled.contains(it) } }")
    println("mask hasFlag ParameterFeature.Query: ${mask hasFlag ParameterFeature.Query}")
    println("mask hasFlag ParameterFeature.Body: ${mask hasFlag ParameterFeature.Body}")
}

An interesting point to notice here is that setting multiple flags results in a BitMask instance rather than a ParameterFeature instance as you’d have in C#. Another point is that, like C#, there’s no compiler control that requires each enum entry to be a single bit or even to be contiguous bits. I actually prefer the Kotlin syntax here for shifting a bit to the left: 1 shl 1.

The implementation of a flag would look like this:

enum class ParameterFeature(override val bit: Long) : Flags {
    Undefined(0),
    Path(1 shl 0),
    Query(1 shl 1),
    Header(1 shl 2),
    Body(1 shl 3),
    FormUnencoded(1 shl 4),
    FormMultipart(1 shl 5);
}

Notice how this requires extending from the Flags interface and overriding the bit constructor parameter. Unlike C#, which allows you to define flags using any integral type, my prototype is limited to Long.

Everything else is handled by jugging between the two types of Flags and BitMask using a combination of operators and infix functions, for example:

infix fun Flags.and(other: Long): BitMask = BitMask(bit and other)
infix fun <T: Flags> Flags.or(other: T): BitMask = BitMask(bit or other.bit)

Here’s the complete example as a gist:

Finding Wakelocks in Android 7 using Google’s Battery Historian

One of my favorite apps prior to my upgrade to Android 7 (Nougat) was GSam Battery Monitor (the paid version is fully worth it). Unfortunately, Android 7 no longer allows apps like GSam Battery Monitor to track individual application battery usage, CPU usage, or wakelocks. That sucks*.

In Android 7 with Doze, you should see periods of marginal battery usage. The more apps you have, the more background services/sync operations/location broadcasts are running and you’ll see plateaus in battery usage be more slanted. Up until a few weeks ago, battery usage would basically flatline between screen time. I set out to figure out what was causing this. I found a thread on Reddit discussing Battery Historian, so I decided to check it out. This post is somewhat of a review of this tool.

Battery Historian analyzes bug report files and presents the information in an interactive graph with categorized breakdowns of battery usage. Battery Historian is a web application written in Go, and doesn’t appear to have an official release (binary or hosted). This means it’s basically limited to power users to evaluate. Since it’s licensed under Apache 2.0, I compiled and hosted this for my own use at https://historian.jimschubert.us/. Before using my hosted instance of Battery Historian, please heed Google’s warning about bug report files:

Bug reports contain data from the system’s various log files, including personal and private information. Only share bug reports with apps and people you trust.

Running Battery Historian locally isn’t too difficult. I definitely recommend going this route if you’re more of a power user; you don’t need programming experience, only comfort on the command line.

Once you’ve taken a bug report on your device and uploaded it to Battery Historian, you’re presented with a graph of all activity found in the bug report.

battery_historian

The declining black line is the same battery usage visualization you’d see in your Battery Usage application or widget. Other groups of battery draining events are presented as horizontal time-oriented lines; some are either on or off (screen), some show gradients for battery draining in the category (temperature), while others display different colors for different states of the event (Mobile signal strength).

Scroll below the graph and you’ll find a breakdown of system statistics. After a little sorting and investigation, I found that JuiceSSH had locked WiFi for 1.5 hours when I hadn’t used the application for over a week:

battery_historian_-_juicessh_wifi_lock

After selecting the JuiceSSH app in the left dropdown, this takes you to the App Stats tab which presents some interesting battery usage statistics:

Wakelocks:
Wakelock Name      | Full Time | Full Count | Partial Time | Partial Count | Window Time | Window Count
JuiceSSH CloudSync |           | 0          | 8m 0s 50ms   | 1             |             | 0           
*alarm*            |           | 0          | 684ms        | 1             |             | 0           
Services:
Service Name                            | Time spent started | # starts | # launches
com.sonelli.juicessh.services.CloudSync | 2h 28m 15s 789ms   | 1        | 1         
Process info:
Time spent running actively in the foreground       | 0ms             
Time spent with a service running in the foreground | 0ms             
Time spent on top                                   | 0ms             
Time spent on top while the device was sleeping     | 0ms             
Time spent running actively in the background       | 1h 33m 10s 343ms
Time spent cached                                   | 38ms            

Looks like, in my case, JuiceSSH Cloud Sync had 8 minutes worth of wakelocks, 2.5 hours of a service running in which 1.5 hours of that was actively running in the background. JuiceSSH unfortunately doesn’t have configuration options for Cloud Sync, so I’ve disabled it. I’ve noticed an improvement since disabling JuiceSSH Cloud Sync.

Although I was a little miffed that the Android 7 update borked GSam’s ability to report on per-application usage, I found Battery Historian to be a really insightful tool. From and Android Development perspective it would be a nice utility to evaluate your application’s effect on the system through normal use. It may be a little overkill from a user or power user’s perspective, though. My Nexus 5X was getting about 18 hours of pretty heavy usage (including my son watching YouTube videos). After disabling JuiceSSH Cloud Sync, I’m seeing about 20 hours of battery out of the same usage. Does that really matter if you’re like me and you charge your phone every night? 👻

* I’ve read that GSam Battery Monitor with root still works as expected on Nougat.

Ignoring files with Swagger Codegen

I’m proud to announce that I’m now a member of the Swagger Codegen Core team.

I’ve previously refactored the C# client, the maven plugin, added an ASP.NET 5 Web API server generator, and fixed some bugs.

I recently added support for .swagger-codegen-ignore. A few weeks ago, I updated the C# client project structure to be a more standard structure:

    .
    ├── IO.Swagger.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

With this breaking change and the .swagger-codegen-ignore, users can now generate into existing repositories. Suppose your project is setup this way:

    .
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        └── MyProject.Test
            └── packages.config

You may want your generated client code to be output under src/MyProject.Client, but you don’t want build.bat, build.sh, or docs to get blown away. Last week, I partially resolved this by adding options to disable api and model doc generation for all languages… but there was no clean way to ignore other files like build.bat or build.sh.

Now, you can add the following .swagger-codegen-ignore to your existing directory structure:

# Swagger Codegen Ignore
/README.md
/build.*
docs/
src/
!src/MyProject.Client/
!src/MyProject.Client.Test/

Just to be safe (this feature is brand new), commit and push your changes before running swagger-codegen using your repository as the output directory. You should end up with the following structure:

    .
    ├── .swagger-codegen-ignore
    ├── IO.Swagger.sln
    ├── MyProject.sln
    ├── README.md
    ├── bin
    ├── build.bat
    ├── build.sh
    ├── docs
    ├── packages
    └── src
        ├── MyProject
        │   └── packages.config
        ├── MyProject.Test
        │   └── packages.config
        ├── IO.Swagger
        │   └── packages.config
        └── IO.Swagger.Test
            └── packages.config

As expected, the only generated content should be the IO.Swagger.sln, src/IO.Swagger and src/IO.Swagger.Test.

All of the functionality in this post should land in version 2.2.0. Until then, you can build directly against the master branch for these features.

[javascript] Be sure to read documentation carefully…

I have a love-hate relationship with JavaScript. I love the language, but I hate the idiosyncrasies. And like many others, I use MDN pretty regularly to verify function signatures, browser support, or to find example code or shims.

The recently posted article on MDN, “A re-introduction to JavaScript”, demonstrates something I wish JavaScript developers would stop demonstrating: cleverness. The article links to one of Crockford’s posts about JavaScript being the most misunderstood language. Sure, there are quirks, but they’re defined quirks in most cases.

I showed a coworker an example of an oddity in JavaScript the other day.

var x = "016";
var y = 0|x;
console.log(y);
// 16

var x = "0o16";
var y = 0|x;
console.log(y);
// 14

console.log(016);
// 14

console.log(0o16);
// 14

This example works differently in ECMAScript 3 and ECMAScript 5, as beautifully explained by CMS on StackOverflow.

Those are ‘gotchas’ of the language that you just have to know. Yes, it sucks. The big problem is when people share code that attempts to be ‘faster’ or ‘more efficient’. Not only do JavaScript developers have to weed out unexpected behavior from the language, but they also have to be diligent to not assume every other JavaScript developer is sharing bug-free code.

This is what you’d see if you were skimming through the recent re-introduction article:

A_re-introduction_to_JavaScript__JS_tutorial__-_JavaScript___MDN

If you’re in a hurry, you’ll probably not see the important bit of text immediately following the examples:

Here we are setting up two variables. The assignment in the middle part of the for loop is also tested for truthfulness — if it succeeds, the loop continues. Since i is incremented each time, items from the array will be assigned to item in sequential order. The loop stops when a “falsy” item is found (such as undefined).

Note that this trick should only be used for arrays which you know do not contain “falsy” values (arrays of objects or DOM nodes for example). If you are iterating over numeric data that might include a 0 or string data that might include the empty string you should use the i, len idiom instead.

If you’re like me and skimming the article for anything interesting, you’ll have probably also missed the text but immediately caught the possible bugginess of the “nicer idiom”. A less experienced skimmer will see the “nicer idiom”, think it’s clever, and use it somewhere it shouldn’t be used. That’s just how the JavaScript community works, even when code on a trusted website like MDN doesn’t work in a given situation.

For a clear example of the problem, consider this code that uses the “nicer idiom” in the wrong way:

// Correction on Mozilla's 're-introduction'
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/A_re-introduction_to_JavaScript
var arr = [],
    terminate = 20,
    i = 0;

while(i <= terminate){
    arr[i] = i++;
}

delete arr[5];

// NOTE: arr[0] is 0, which is falsey. This won't print!!!
// change idx=1 and this terminates at index 5.
for (var idx = 0, item; item = arr[idx++];) {
    console.log(item);
}

/*
// proper iteration
for (var i = 0; i < arr.length; i++) {
    var item = arr[i];   
    console.log(item);
}
*/

Play with this in jsfiddle

The issue here is that the array must not contain falsy values. This includes undefined values in sparse arrays, null, zero, "0", "0.0", "", NaN, document.all, or any other values that coerce to false. The proper way to iterate over an array is to explicitly visit every index unless you have a clearly defined and documented reason not to continue through the array.

The lesson to learn here is twofold: actually read documentation and articles, and don’t just assume JavaScript code is usable code.

Review: Microsoft .NET – Architecting Applications for the Enterprise, 2nd Edition

I’ve recently finished reading Microsoft .NET – Architecting Applications for the Enterprise, 2nd Edition by Dino Esposito and Andrea Saltarello. This book caught my attention for two reasons. First, I really enjoy software architecture. Second, I’ve read lots of work by Dino and always found them enjoyable.

The book gives examples of a few architectural patterns like Domain Driven Design (DDD), Command Query Responsibility Segregation (CQRS), and Event Sourcing. It starts with a pretty heavy focus on DDD and seems to assume that you’ve read *The* Domain Driven Design book by Eric Evans. I’ve only skimmed through that book, but it was enough to get many of the references. This book does cover the high level definitions of DDD in a way that the reader can jump right in. The accompanying source code even provides the same example in DDD and CQRS (the two main architectures being discussed).

There were two major things I liked about this book. I really enjoyed how the book isn’t presented as ‘this is the best software architecture, so you should use it.’ Things are presented in a way that clearly states the authors’ position on each architecture. For example, DDD is discussed as a the typical go-to architecture for enterprise systems. It’s a proven pattern that ‘just works’ in many cases. The discussion of CQRS clearly identifies some weaknesses in DDD and supplies an alternative with the explicit caveat that a CQRS-based architecture will change how you think about data.

The other main point I liked about the book is how real-world the discussions are. I usually get that feeling from Microsoft Press books. Early on the authors say, “To design a system that solves a problem, you must fully understand the problem and its domain.” In one of my previous positions, one of the first things I asked when joining the team was, “How have you documented what we’re trying to build?” I received a weird non-answer from the team’s senior developer. I went to the team’s manager and asked the same thing so I could understand the requirements gathering and technical design analysis that went into the product. He said, “We don’t have any of that — we’re Agile.” I thought he was joking, but he wasn’t. Obviously it would strike a chord with me when the authors said, “Agile architecture is sometimes presented as an oxymoron, like saying that if you’re agile you don’t do any architecture analysis, you just start coding…” Elsewhere they described the businessman-developer disconnect that occurs when requirements are incomplete, which is exactly what happened on the previous team I mentioned. It’s like I always say, “If you don’t know where you’re going, how will you get there?”

Another aspect of the book that I think can get buried in reading is the point about developing for a task-based user experience. Too often do developers start coding at the database and work their ways up to the user interface. Without a clean architecture, this can lead to situations where a column in your relational database has to change all the way to the UI. That’s silly. With a good architecture that clearly identifies boundaries in code, you inherently create more reusable and more maintainable code. I think this should be everyone’s main goal while writing code.

I also thought it was cool how the book and accompanying example code demonstrate using a NoSQL database (RavenDB) for an event sourcing application. While I don’t know where I’d ever use Event Sourcing over a more common architecture (mainly because I have to account for the expertise of other developers), I really like the Event Sourcing pattern. Now that Akka.NET is gaining traction, I wonder if Event Sourcing will become the way of the future.

One minor issue I had with the companion content was that it didn’t run as-is. There was a problem with the NuGet packages and MVC5 for which I found a solution at https://naa4e.codeplex.com/workitem/1.

I would definitely recommend this book for senior developers and engineers. It provides concrete examples and guidance from experienced architects toward more solid enterprise application design.

You can download the companion code for the book at https://naa4e.codeplex.com/

Tips for debugging a WiX MSI

WiX is an excellent technology that simplifies the creation of MSI files using an XML abstraction on top of the Windows Installer APIs. WiX doesn’t change the underlying MSI architecture and it can be a huge pain in the ass sometimes. I thought I would share some tips for debugging what’s going on with an MSI. These tips aren’t specific to WiX, that’s just the technology I’m familiar with.

Logging

The first thing you should try is to add the log switch when you run your installer. To do this, open a command prompt in the directory where your MSI file is located and run it with the following parameters:

msiexec yourinstaller.msi /L*v install.log

The /L*v says to log verbose messages to a file.

For a full list of msiexec switches, check out MSDN’s documentation for msiexec.

Re-cache MSI

If you think your package was somehow corrupted or you’ve made a simple such as changing a file’s guid or a feature condition, you can re-cache the package with the following command:

msiexec /fv yourinstaller.msi

This is generally used when you install a product and then thing “oops, I forgot something…”. You don’t want to create a whole new release, so you can update the installed/cached MSI.

Be careful to read the documentation. With the /f parameter, you can’t pass properties on the command line.

Increased Debug Messages

You can actually increase the amount of messages exposed to MSI logs by setting a machine policy for Windows Installer.

Open regedit and go to HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows\Installer (you may need to create this key). Add a DWORD key called Debug with a value of 7.

The value of 7 will write all common debugging messages to the log as well as command line information. If you’re passing sensitive data to the installer on the command line, be warned that this information gets dumped via the OutputDebugString function. This means that anyone with physical access to the machine or access to the machine via TCP/IP may be able to read these messages. The documentation for the policy settings warns that this setting is for debugging purposes only and may not exist in future versions of Windows Installer.

For a full list of available settings, check out Microsoft’s Windows Installer Machine Policies.

System Debugging

Passing the logging switch and log file name to an MSI every time you run an MSI can be annoying. You can set the Logging machine policy to have Windows Installer write a log to a temp file, but then you have to track down the temp file.

Instead, you can run Sysinternals’ DebugView. This utility allows you to catch debugging information exposed by the Win32 OutputDebugString function on local and remote machines. This doesn’t just gather information from MSIs. For example, you can open a TFS query editor in Visual Studio 2013 and inspect the queries made via TFS.

Orca

Lastly, the Windows SDK contains a file called Orca.msi. Orca is a tool to inspect the contents of an MSI. An MSI is essentially just a database of tables and fields with values. You can open an existing MSI, edit a condition, and save it. This can make investigation a little easier. For instance, if you’re trying to figure out why a condition on a custom action doesn’t seem to be passing, you can edit the MSI and modify the condition instead of rebuilding the MSI. It’s also helpful if you’re trying to figure out how another MSI implements some logic.

I’m sure there are plenty of other techniques for debugging MSI files.