Tag Archives: linux

Data at the root level is invalid. Line 1, position 1.

Recently, I encountered a really weird problem with an XML document. I was trying to load a document from a string:

var doc = XDocument.parse(someString);

I received this unhelpful exception message:

Data at the root level is invalid. Line 1, position 1.

I verified the XML document and retried two or three times with and without the XML declaration (both of which should work with XDocument). Nothing helped, so I googled for an answer. I found the following answer on StackOverflow by James Brankin:

I eventually figured out there was a byte mark exception and removed it using this code:

string _byteOrderMarkUtf8 = Encoding.UTF8.GetString(Encoding.UTF8.GetPreamble());
if (xml.StartsWith(_byteOrderMarkUtf8))
{
    xml = xml.Remove(0, _byteOrderMarkUtf8.Length);
}

This solution worked. I was happy. I discussed it with a coworker and he had never heard of a BOM character before, so I thought “I should blog about this”.

Byte-Order Mark

The BOM is the character returned by Encoding.UTF8.GetPreamble(). Microsoft’s documentation explains:

The Unicode byte order mark (BOM) is serialized as follows (in hexadecimal):

  • UTF-8: EF BB BF
  • UTF-16 big endian byte order: FE FF
  • UTF-16 little endian byte order: FF FE
  • UTF-32 big endian byte order: 00 00 FE FF
  • UTF-32 little endian byte order: FF FE 00 00

Converting these bytes to a string (Encoding.UTF8.GetString) allows us to check if the xml string starts with the BOM or not. The code then removes that BOM from the xml string.

A BOM is a bunch of characters, so what? What does it do?

From Wikipedia:

The byte order mark (BOM) is a Unicode character used to signal the endianness (byte order) of a text file or stream. It is encoded at U+FEFF byte order mark (BOM). BOM use is optional, and, if used, should appear at the start of the text stream. Beyond its specific use as a byte-order indicator, the BOM character may also indicate which of the several Unicode representations the text is encoded in.

This explanation is better than the explanation from Microsoft. The BOM is (1) an indicator that a stream of bytes is Unicode and (2) a reference to the endianess of the encoding. UTF8 is agnostic of endianness (reference), so the fact that the BOM is there and causing problems in C# code is annoying. I didn’t research why the UTF8 BOM wasn’t stripped from the string (XML is coming directly from SQL Server).

What is ‘endianness’?

Text is a string of bytes, where one or more bytes represents a single character. When text is transferred from one medium to another (from a flash drive to a hard drive, across the internet, between web services, etc.), it is transferred as stream of bytes. Not all machines understand bytes in the same way, though. Some machines are ‘little-endian’ and some are ‘big-endian’.

Wikipedia explains the etymology of ‘endianness’:

In 1726, Jonathan Swift described in his satirical novel Gulliver’s Travels tensions in Lilliput and Blefuscu: whereas royal edict in Lilliput requires cracking open one’s soft-boiled egg at the small end, inhabitants of the rival kingdom of Blefuscu crack theirs at the big end (giving them the moniker Big-endians).

For text encoding, ‘endianness’ simply means ‘which end goes first into memory’. Think of this as a direction for a set of bytes. The word ‘Example’ can be represented by the following bytes (example taken from StackOverflow):

45 78 61 6d 70 6c 65

‘Big Endian’ means the first bytes go first into memory:

45 78 61 6d 70 6c 65
<-------------------

‘Little Endian’ means the text goes into memory with the small-end first:

45 78 61 6d 70 6c 65
------------------->

So, when ‘Example’ is transferred as ‘Big-Endian’, it looks exactly as the bytes in the above examples:

45 78 61 6d 70 6c 65

But, when it’s transferred in ‘Little Endian’, it looks like this:

65 6c 70 6d 61 78 45

Users of digital technologies don’t need to care about this, as long as they see ‘Example’ where they should see ‘Example’. Many engineers don’t need to worry about endianness because it is abstracted away by many frameworks to the point of only needing to know which type of encoding (UTF8 vs UTF16, for example). If you’re into network communications or dabbling in device programming, you’ll almost definitely need to be aware of endianness.

In fact, the endianness of text isn’t constrained by the system interacting with the text. You can work on a Big Endian operating system and install VoIP software that transmits Little Endian data. Understanding endianness also makes you cool.

Summary

I don’t have any code to accompany this post, but I hope the discussion of BOM and endianness made for a great read!

Flattr this!

Hard link in OS X

Under linux, I used mount --bind olddir newdir to link a source code directory from a git repository in my Dropbox folder to another public git repository. I did this to avoid using git submodules and, for whatever reason, git had trouble following hard links under Ubuntu.

I recently purchased a Macbook Pro, and I was trying to use the same bound mount point. The mount provided with OS X doesn’t offer --bind. A main suggestion I found online is to use bindfs. bindfs didn’t work for my needs, and I didn’t want to manually compile a link/unlink application as others suggested.

I found that the homebrew package coreutils provides the gnu version of ln. To install:

brew install coreutils

This package doesn’t overwrite OS X default packages unless you run the above command with the --default-names switch, otherwise all utilities are prefixed with a g. Using the gnu ln program, I was able to hard-link successfully:

gln -d src target

I don’t know what the difference is between Ubuntu and OS X (both running git 1.8.4), but I can now work properly under git with hard links as expected.

Installing more gnu utilities

You can go one step further and install more gnu utilities (found on StackExchange):

brew install coreutils findutils gnu-tar gnu-sed gawk gnutls gnu-indent gnu-getopt

Then, add the following to your .bashrc:

PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
MANPATH="/usr/local/opt/coreutils/libexec/gnuman:$MANPATH"

Thanks to the Apple StackExchange user lri for the excellent post.

Flattr this!

System76 notebooks… are they worth the money?

On June 19 2012, I purchased a 15″ Pangolin Performance with the following configuration:

  • Ubuntu 12.04 LTS 64 bit
  • 15.6″ 720p High Definition LED Backlit Display ( 1366 x 768 )
  • Intel HD Graphics 4000
  • 3rd Generation Intel Core i7-3610QM Processor ( 2.30GHz 6MB L3 Cache – 4 Cores plus Hyperthreading )
  • 16 GB Dual Channel DDR3 SDRAM at 1600MHz – 2 X 8 GB
  • 750 GB 7200 RPM SATA II
  • 8X DVD±R/RW/4X +DL Super-Multi Drive
  • Intel Centrino 1030 – 802.11 b/g/n Wireless LAN + Bluetooth Combo Module

Total price: $1,038.00

The price was higher than I wanted to spend, but I considered this purchase an investment. After all, I had previously owned a Sony Vaio for almost 5 years, and which I had purchased for $800. The reviews for System76 seemed a bit mixed. I’ve been using Linux or BSD as my main desktop environment since 1998, so I took a chance on the mixed reviews and made the investment.

Issue #1

After two months, I had to open a ticket because the Ubuntu version preinstalled on the machine displayed the graphics card as ‘Unknown’ and the screen itself exhibited a sort of wave of energy which was giving me headaches. I was told by the support engineer that the fix would be to install the mesa-utils package. Why is mesa-utils not installed by default if it’s a necessary package? I have no idea, but here’s the response I received from my support ticket:

It’s actually a known issue without having the mesa-utils installed. Apparently, the info is actually pulled from glxinfo.

Mesa-Utils isn’t part of the default install, and admittedly, we do not add it either. I know there’s talk of trying to add it as part of the install.

Sure.

The graphics issues were actually resolved by an early upgrade to Ubuntu 13.04.

Issue #2

In October 2012, I decide to dual boot a copy of Windows 8 from MSDN. There were numerous issues with drivers. After opening a support ticket, I was told that System76 doesn’t have access to drivers for their hardware from their vendors (bison or chicony hardware). That seems really shady, considering a machine for such a high price should have quality hardware. I ended up removing Windows 8 and installing Windows 7 within a VirtualBox VM. The hardware is beefy enough to host multiple concurrent VMs, so this is actually my preferred method for cross-platform development. However, a machine that can’t dual boot Windows basically renders the machine useless for developers who need to code at a low level like packet processing or driver development.

Issue #3

On June 15 2014, just shy of two years after purchasing this machine, I decided to open a ticket regarding battery issues I started having after installing Ubuntu 14.04. My battery began to hold no more than 75% or so charge. I inquired how to troubleshoot whether or not there were some other factors causing battery issues. For reference, here are screenshots of the battery statistics:

I was told by the support team that my battery was causing problems because:

The statistics indicated this is within your battery. The energy designed (48.8 W/hr) and full (40.0 W/hr) is where your discrepancy lies.

So, I opened a ticket saying that my machine had a max of 76% charge after hours of charging (both booted and off) and support replied saying my problem was that my battery could only hold a max charge of 83.3%. The resolution offered was to purchase a new battery at $105.

That’s not right. Every two years of ownership, I will need to purchase the same low quality battery at $105 out of pocket? In a sense, I have purchased a machine that also requires a $50/year battery fee. This seems really silly.

I admit that I had no battery issues for about two years. In fact, I was surprised by the battery life (2.5 to 3 hours) early on. Machines with a mechanical hard drive and 16GB of RAM are generally considered battery hogs. I don’t know of anyone who actually works on such a machine on battery.

Conclusion

Although my machine has worked well for two years, I don’t like the quality of the hardware or the poor response from the company about the issues I have had. I probably wouldn’t mind it if the quality of support offset the quality of the hardware. For example, if System76 actually stood behind their product and replied “Wow, our batteries should last more than 1.95 years. We’ll send you a replacement immediately,” I would recommend purchasing from the company. You do get a pretty beefy machine for relatively cheap. As a software engineer with multiple side projects and a 9 month old son, I don’t have time to continuously try to troubleshoot issues that I really shouldn’t be having with a quality piece of hardware.

My own resolution was to purchase a 13″ Macbook Pro. This machine has 8GB RAM and a 256GB SSD Hard Drive. I have been getting 8 or 9 hours of battery life, and that’s with the power-sucking Google Chrome open at all times. I became accustomed to the OS X environment while working at Expedia. Mac OS X is a UNIX certified OS, so I feel at home in the environment. The only issue I had with OS X when I first started working in the environment was the difference between COMMAND, OPTION, and CONTROL. This took a week or two to become habit. My wife also left her Sony VAIO for a Macbook Air last year and she loves it. You’d be hard-pressed to find a Linux distribution which allows for such a smooth computing experience for non-geeks.

I will be selling my System76 machine after backing up all important information. If you’re interested, let me know. You’d be getting a beefy machine and not taking along any of the annoyance of paying over $1000 for subpar customer support.

To answer the question “Are they worth it?”, I would have to say “It depends.” If you want a machine that would cost $2500-3000 elsewhere for around $1000, then yes it is definitely worth it. If you don’t mind replacing a battery after two years, then yes it’s worth it. If you’ve used Linux for a long time like I have and don’t mind spending an hour or two every time a ‘surprise’ surfaces, then yes it’s definitely worth it. If you’re like me and you have little free time, then you’ll want to open your machine and expect everything to work as expected with little or no interaction with customer support. In my case, it’s just not worth it to own a machine that requires so much maintenance. If this system was a car, I would sell it as a car that runs well and needs little or no work. That is, if you don’t work off battery you’d be all set.

Flattr this!

Install Balsamiq, Ubuntu 14.04

Balsamiq for Desktop requires Adobe AIR, which is no longer supported.

Here are the steps I took to get AIR and Balsamiq to install.

gnome-keyring is required by AIR, gdebi is required to install the balsamiq package which will fail as “bad quality” in the Ubuntu Software Center. You can skip the gdebi install if you’re using X File Package (or probably anything other than Ubuntu Software Center) to install .deb packages.

sudo apt-get install gnome-keyring gdebi gdebi-core

Lots of sites say you need to install ia32-libs, which don’t exist in the Ubuntu repositories. Here is what’s required (taken from this post on askubuntu.com and this other post on askubuntu.com )

sudo apt-get install gtk2-engines:i386 libart-2.0-2:i386 libcairo2:i386 libcanberra-gtk0:i386 libdatrie1:i386 libgail-common:i386 libgconf2-4:i386 libgtk2.0-0:i386 liblua5.1-0:i386 libpango1.0-0:i386 libpixman-1-0:i386 libqt4-network:i386 libqt4-test:i386 libqtcore4:i386 libthai0:i386 libbonobo2-0:i386 libglade2-0:i386 libgnomecanvas2-0:i386 libidl0:i386 liborbit2:i386 libwmf0.2-7:i386 gtk2-engines-murrine:i386 libxml2:i386 libxslt1.1:i386 libxt6:i386 lib32nss-mdns libnspr4-0d:i386 libnss3-1d:i386 

Now, download AdobeAIRInstaller.bin version 2.6 and MockupsForDesktop64bit.deb. (If MockupsForDesktop64bit.deb link is dead, please download from here.)

cd ~/Downloads
chmod +x AdobeAIRInstaller.bin # make Adobe AIR executable
LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu ./AdobeAIRInstaller.bin # install Adobe AIR

Once AIR is installed, open your file manager, right-click on the balsamiq package and choose open with -> gdebi. The install should now complete.

Flattr this!

Disable scrollbar overlays in Ubuntu 13.10

Working with Tizen IDE, I noticed the overlay doesn’t actually display for the scrollbar in Ubuntu 13.10. This isn’t so bad for vertical scrolling because I use the touchpad to scroll. For horizontal scrolling, it is a huge pain. I’ve never liked the overlays, and they don’t really fit well with Cinnamon.

Here’s how to set the scrollbars back to normal:

$ gsettings set com.canonical.desktop.interface scrollbar-mode normal

If you discover the normal mode causes problems for you, you can reset it to the overlay:

$ gsettings set com.canonical.desktop.interface scrollbar-mode overlay-auto

Or, if you prefer a GUI, open dconf-editor and expand the node on the left to com→canonical→desktop→interface, then select the scrollbar style you like best.

Flattr this!

Tizen SDK 2.2.1 on Ubuntu 13.10

I’ve registered for the Tizen Seattle Devlab & Hack which takes place on December 6-7 2013.  To prepare, I downloaded and installed the Tizen SDK. I’m really enjoying it so far, but I had a couple of issues with installation and startup on Ubuntu 13.10.

Remove Google Talk

First of all, I had issues starting Tizen IDE. It turned out that I need to uninstall google-talk

$ sudo apt-get remove google-talkplugin

Install Oracle JDK

Then, I had to install Oracle’s JDK as mentioned on Tizen’s prerequisites page.

$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java6-installer

The last thing to do is to make sure your system is configured to use Oracle’s JDK instead of OpenJDK.

$ sudo update-alternatives --config java

Just select the number of the java6 JDK. For example, in the following display you would choose 2 to switch from Oracle JDK7 to JDK6:

There are 4 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-6-oracle/jre/bin/java          1076      auto mode
  1            /usr/bin/gij-4.8                                 1048      manual mode
  2            /usr/lib/jvm/java-6-oracle/jre/bin/java          1076      manual mode
  3            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1071      manual mode
* 4            /usr/lib/jvm/java-7-oracle/jre/bin/java          1075      manual mode

Press enter to keep the current choice[*], or type selection number: 

Note: This seems to work with both Oracle’s JDK6 and JDK7. If you’d like to use JDK7 instead of the suggested prerequisite, you can follow the same steps above but instead install version 7 of the JDK:

$ sudo apt-get install oracle-java7-installer

Link libudev.so.1 to libudev.so.0

After all of these steps, you should be able to start up the Tizen IDE (which is a modified Eclipse IDE). However, I had a message that sdb failed to start. This will prevent you from starting an application on an emulator (you can still start on the web simulator for a web application). The sdb issue is caused by a newer libudev library version installed from Ubuntu 13.10’s repository. The fix (which you can also find on askubuntu) is to symlink the newer version of the shared library to the expected version:

$ # 32-bit:
$ sudo ln -sf /lib/i386-linux-gnu/libudev.so.1 /lib/i386-linux-gnu/libudev.so.0
$ # 64-bit:
$ sudo ln -sf /lib/x86_64-linux-gnu/libudev.so.1 /lib/x86_64-linux-gnu/libudev.so.0

Try Tizen SDK

The download of the Tizen SDK is available here.

Flattr this!

bash function: md.view

One of my favorite bash functions is md.view, which is available in my dotfiles repository.

# test markdown files, probably a better way to test for programs.
function md.view {
    local i=true
    type -p markdown &> /dev/null || i=false
    if $i ; then 
        local output="/tmp/md.view-$(date +%F).html";
        markdown $1 > "$output";
        google-chrome "$output"; # xdg-open would open default browser after next line executes
        rm "$output";
    else
        echo "markdown is not installed"
    fi
}

There’s not really anything particularly cool about this function. It checks for markdown and reminds me to install it if I haven’t yet. It doesn’t do this for Google Chrome because that’s the first application I install on new machines. Usually, you’d use xdg-open to choose a browser, but this can cause problems with the remove command at the end of the function (depending on how the browser creates its process).

Anyway, this is really useful when I’m creating README files using markdown for my github repositories. For instance, I was just updating a README to refer to *_32.png images. The problem is that * and _ are special tokens in markdown. I’d never attempted to escape one after the other, which is often broken in parsers. With this function in my extended .bash\functions, I can quickly check output with:

$ md.view README.md

I know, it’s such a simple thing to have as a favorite. Sometimes really simple things have a huge impact, especially if you type out README files regularly

Flattr this!

‘Upgrading’ couchdb on Ubuntu 12.04

If you’ve installed couchdb from the Ubuntu repositories, you’re likely running 1.0.1. If you decide to build and install couchdb from source, you may have some issues with the server starting after a reboot.

The trick, as outlined here is to perform the following:

sudo mv /etc/init.d/couchdb /etc/init.d/couchdb.1.0.1.donotuse
sudo ln -sf /usr/local/etc/init.d/couchdb /etc/init.d/couchdb
sudo update-rc.d couchdb defaults

This assumes you’ve configured the source with --prefix=/usr/local (which is the default).

Re-linking the init script should allow you to start/stop/restart the server as expected.

Flattr this!

Ubuntu: Open With… Wireshark

Reading through Practical Packet Analysis, I ran local captures to follow along until Chapter 5. Instead of trying to simulate an improperly-dissected packet, I downloaded the captures which accompany the book. In Ubuntu 11.10, I didn’t see any way to easily associate .pcap files with Wireshark. In previous versions of Ubuntu, the ‘add’ button was available in the ‘Open with…’ dialog, but in 11.10 it is grayed out. Clicking the option to choose another application or find an available application online didn’t work. Bummer.

To get wireshark to show up in the ‘Open With…’ dialog’s choice of applications, I followed the instructions on AskUbuntu.com. The instructions are for vim, so I’ve modified them for wireshark.

Save the following as ~/.local/share/applications/wireshark.desktop:

[Desktop Entry]
Encoding=UTF-8
Name=Wireshark
Comment=Wireshark packet capturing
Exec=wireshark %u
Terminal=false
Type=Application
Icon=/usr/share/pixmaps/hi48-app-wireshark.png
Categories=Application;Utility;
StartupNotify=true
MimeType=application/octet-stream
NoDisplay=false

For more information about .desktop files, check out the Desktop File Specification.

Flattr this!