prb.io

Moved Yet Again

2011-11-20T00:00:00-08:00

I had been meaning to move this blog outside of the mult.ifario.us domain for a while. I initially thought about getting perpubplat updated to take account for newer versions of GHC and Twitter’s switch to OAuth for API access and del.icio.us being deprecated (in favor of Pinboard) and maybe Akismet integration and…, but that’s a good bit of work for a project that is truly just for fun.

Instead, partially as an experiment for getting FasterXML content moved into Github pages and partially because the push-to-publish workflow is appealing, I decided to give Jekyll via Github pages a go as a replacement for perpubplat. It turns out that perpubplat’s post metadata is close enough to Jekyll’s YAML front matter that it’s easy (~50 total lines of Scala) to write a quick conversion. Permalinks that don’t end in an unsightly .html, comments past and future, and tag-based navigation and feeds are all casualties of the conversion, and I’m OK with that. (I’ve come around to the idea that your comment is your content, and if it’s important enough, you can publish it somewhere else yourself.)

Crunching Java Class Versions with `bash`-fu

2010-08-20T00:00:00-07:00

I recently needed to root out the JDK 6 classes lurking in an application that was supposed to run on JDK 5, and it turns out that it's not that difficult with a little bash-fu. After unpacking all of the constituent JAR files:

$ find . -name *.class | tee -a classes | xargs -n 1 head -n 1 | \
  cut -b 8 | xargs -IX printf '%d\n' "'X" | \
  paste -d ' ' - classes | grep '^50'

Et, voila! I have the culprit:

50 ./jlayer-1.0.1.jar/javazoom/jl/converter/Converter$PrintWriterProgressListener.class
50 ./jlayer-1.0.1.jar/javazoom/jl/converter/Converter$ProgressListener.class
[...]

A rebuild of the JLayer library, and all's well again.

Come Work for Me

2010-05-12T00:00:00-07:00

Multifarious has been running comfortably and profitably for the past couple of years with just me and the occasional subcontractor, but it's time to grow the business: I'm looking for an engineer to come and work for me full-time.

The basic "win" for the position is obvious — great pay, great environment, challenging work — but the bigger picture should be compelling as well. As I described last year, Multifarious is intended to be a springboard for an ongoing series of business experiments, and this is a place where the right candidate can gain knowledge and experience that they wouldn't otherwise have access to in a purely engineering context.

Splitting XML Well with XSLT 2

2009-09-30T00:00:00-07:00

I recently had the need to split up a result set from a Solr query into a collection of smaller groups of add requests for POSTing into a different core. There are some ways to make the split work with text processing tools (split and friends), but it's always an open question whether an ad hoc approach will trip over some markup — it's just better to use XML tooling. By no coincidence (based on features missing from ), XSLT 2 makes it easy to do the right thing.

First up is grouping in chunks of 2000 records:

<xsl:for-each-group select="/response/result/doc"
                    group-by="round(position() div 2000)">
...
</xsl:for-each-group>

Outputting each hunk to a file named for the index of the group is also a one-liner:

<xsl:result-document href="{current-grouping-key()}_out.xml">
  <add>
    <xsl:for-each select="current-group()">
      <doc>
        <xsl:apply-templates />
      </doc>
    </xsl:for-each>
  </add>
</xsl:result-document>

And that's it. The only trick is choosing an XSLT processor, and the superlative Saxon (from Saxonica) is my default choice.

Commandline Puzzler

2009-09-25T00:00:00-07:00

Suppose that you have to files that consist of records, one per line, and you want to ensure that none of the records in the second file appear in the first. How do you do it with only the text processing commandline tools commonly available on *nix systems?

NoNoSQL

2009-09-23T00:00:00-07:00

Ben Black, one of the organizers of no:sql(east). conference, tweeted, and I twote:

Chris Williams, another no:sql(east). organizer, has had similar sentiments, but what really needs to happen is for people to stop using the "NoSQL" term. I originally proposed "dbng" for next-generation database (and with an intended allusion to RELAX NG), but I'm warming up to Ben Black's suggestion of "altdb" for the hint of Usenet alt.* if nothing else.

I propose a new movement called the NoNoSQL movement. It is a movement for those interested in alternative and next-generation databases but not in the inaccurate "NoSQL" neologism.

Seems like some cool altdb schwag (t-shirts, mugs, etc.) is in order — "Not your daddy's database." or "Joiners need not apply." or...

Voldemort-Based Twitter Clone Talk at OSCON

2009-07-23T00:00:00-07:00

Dan and I just finished up our talk at OSCON. You can download the slides or view it on Slideshare. I'll probably take it down at some point in the near future, but the sample system from the presentation is up and running for the moment.

We got started on the material for the talk several months back with the Twitter one-to-many publishing problem as a motivating problem to play with various non-relational data stores, and after some dabbling with Cassandra and HBase, we ended up focusing on Voldemort as an initial backend for the system. It is very likely that we'll craft some additional backends, and I'd particularly like to get to a more forgiving model for storing lists. (I'm already part way there on Dynomite with Osmos as the storage engine.)

The system described in the talk uses a small (two nodes) Voldemort cluster and a small cluster of web nodes (JAX-RS with a jQuery front-end) to implement enough microblogging functionality to be interesting — users, follow/followed, publishing — along with a simple dashboard implemented with Cacti and rudimentary deployment automation. The source is out on GitHub if you want to take a look. (Feel free to fork with it...)

Dan's blog entry on the presentation is here.

If you have nothing to say, say nothing

2009-06-05T00:00:00-07:00

There is never a good reason to announce that you're going to make an announcement. This rule came to mind when I saw this tweet scroll by this morning:

This belongs in the same category of non-actions as a blog post to say you haven't been blogging, telling people about your "stealth" startup, or a statement like "with all due respect".

Speaking at OSCON 2009

2009-05-28T00:00:00-07:00

With Dan Diephouse, I'll be speaking at OSCON on July 23.

Taking the abstract literally, the talk looks like it is about building a Twitter clone with open source components, but it is not at all intended to be armchair quarterbacking about Twitter's early problems with availability. (We should all have these problems!) Rather, the talk is intended to be about some of the current crop of interesting open source distributed storage technologies — Cassandra, Voldemort, Redis (where the folks have already done some thinking about Twitter-like apps), CouchDB, HBase, Dynomite — as well as how to attack some of the operational problems (e.g., deployment, instrumentation, application updates) that come with using new tools in multi-node environments.

That's obviously quite a bit to fit into a relatively short speaking slot, but Dan and I plan to blog or otherwise publish material that won't fit.

Integrating Github and Redmine

2009-05-26T00:00:00-07:00

I've been a fan and user of Atlassian's excellent Jira since the company was founded back in 2002, but I needed the ability to set up some quick-hit bug/task/wiki sites for smaller consulting projects and neither the month-to-month hosted model nor the enterprise license made good economic sense. I opted for the an install of Redmine, and while it's no Jira, I've been reasonably happy with it. (The one big headache was getting SMTP over TLS working.)

Redmine supports integration with Git repositories on a per-project basis and will link commits to issues based on the presence of keywords and issue identifiers (e.g., "refs #123"). The way the integration is implemented works well if the Git repository is hosted on the same machine as the Remine instance, but I host all customer and internal work on github. Here's a quick recipe to bridge the gap.

First, add an ssh key for the redmine user to your github account.

Next, create a home for the following shell script, e.g., /opt/redmine_extras/bin and a home for Git repositories on the server, e.g., /var/redmine/git_repositories and ensure that the redmine user has write privileges for the repositories. Here's the pull_git script:

#!/bin/bash
export REPOS=/var/redmine/git_repositories
export REDMINE_HOME=/opt/redmine-0.8.2
export LOGFILE=/var/log/redmine_extras.log

function log_prefix {
        echo -n `date '+%Y/%m/%d %H:%M:%S'`" ["$$"] ${2}"
}

for i in `ls -d ${REPOS}/*.git`; do 
  cd $i;
  log_prefix && echo 'Processing git repository from '${i}'...';
  /usr/local/bin/git --bare fetch origin :master
done

cd ${REDMINE_HOME}
log_prefix && echo 'Updating Redmine...'
/usr/local/bin/ruby script/runner "Repository.fetch_changesets" -e production

Then (I'm logged in as root) add the command to the redmine user's crontab:

# echo '*/10 * * * *    /opt/redmine_extras/bin/pull_git 2>&1 >> /var/log/redmine_extras.log'\
 | crontab -u redmine -

Now, for each repository, say foo and your github user is bar, you will track from Redmine, do:

# cd /var/redmine/git_repositories
# sudo -u redmine -H git clone --bare git@github.com:bar/foo.git
# cd foo.git
# sudo -u redmine -H git --bare remote add origin git@github:bar/foo.git

Ensure that the Redmine project points to the local copy of the Git repository, and the revisions should start getting syncronized every ten minutes.