Programblings

Working with new people often influences the way you work. The influences can range from picking up simple tricks to seeing fundamental facts about your craft in a new light.This week when I began working with James I saw him run his tests directly from TextMate. Of course I knew it was possible to run Ruby from TM, including tests.For some inexplicable reason however I had never bothered to try it. This trick is very convenient for two reasons: TextMate cleans up the backtraces and resolves each level of the trace to a clickable link to your code file. So I decided to include this trick in my workflow.I encountered two problems with this however. Two gems I usually use in my tests don’t play well with running tests from TextMate.

redgreen

The redgreen gem highlights the . F and E (among other bits) in your test runs with green, red and yellow. When running my whole test suite it’s something I want to have. It’s not only visually pleasing, but it also lets me see at a glance whether any problems were encountered.When run from TextMate, tests using redgreen are displayed without having the console coloring stripped out, which gives something like this:

Riiiight.For the record, here’s the nice result when run at the console:

quietbacktrace

The other gem I can’t do without is James Golick and Dan Croak’s quietbacktrace. This gem lets you specify filters and silencers to clean up those huuuuuge Rails or Merb backtraces.Filters let you remove useless parts of a given line in the backtrace, such as the path leading to your gems. Silencers let you completely remove some lines from the backtrace, such as all the lines referring to what happened inside Rails, leading to your error. The most popular filters and silencers are already provided with quietbacktrace, as you’ll see below.So the result, of course is that messing with backtraces breaks TextMate’s ability to link a backtrace line with the corresponding file.

A simple snippet to fix this

Since I didn’t really want to do away with these tools, I included a bit of a hacky snippet in my test_helper.rb file to detect whether a test run was happening in TextMate or at the console.If you find yourself in the same fix as me, feel free to use the following.Among your requires:

IN_TM = !ENV['TM_DIRECTORY'].nil?
unless IN_TM
  require 'redgreen'
  require 'quietbacktrace'end

And then:

class Test::Unit::TestCase
  unless IN_TM
    self.backtrace_silencers << :rails_vendor
    self.backtrace_filters   << :rails_root
  end
  #...
end

Now I can have my testing niceties when running my tests from the console and they don’t break the TextMate integration :-)

Trying this out for the first time?

If like me you just decided to try this out for the first time, the shortcuts are easy to remember. Command-R runs any Ruby file (in this case, the whole test file) and Command-Shift-R runs the single test the cursor is in.There’s a little hiccup in sight, however. If you’re using Rails 2+, you have to apply a very small fix to TextMate before you’ll be able to run your tests. Read more about the fix on Marc-André Cournoyer’s blog.

A wise man once suggested that one of the steps in teaching yourself programming involved working with people better than you. (See Teach Yourself Programming in Ten Years)

Even though I feel I’m a pretty competent software developer, I’m very excited to join another developer for which I have a truly profound respect. I’ve only recently made the career move of working on the web full time, and this is a great opportunity for me to kickstart this phase of my career by working with a Rails / REST / jQuery guru.

Starting today, I’m working for GiraffeSoft, with James Golick. We’ve wanted to work together for a while, and the stars have finally aligned to make this possible.

I’m ready to bet that setting up and deleting remote branches is something you do rarely enough that you always find yourself looking up the documentation. Or maybe it’s just me.

Due to its roots, Git supports a wide array of usage scenarios for interacting with remote repositories, and we love it that way. It’s a big factor in the flexibility and power of the tool.

However in simple scenarios, there’s still a bunch of commands you must run to accomplish simple tasks. I believe the commands for the simple scenario can be simpler.

Last January, Carl Mercier created git-remote-branch. I’ve found this script very useful and, with his permission, I’ve decided to keep moving it forward and add a few features to it.

git_remote_branch

The first purpose of git_remote_branch is to encapsulate all the commands that need to be run to interact with remote branches in simple scenarios.

Its second purpose is to be a learning tool. git_remote_branch does two things to help you learn these commands:

• It clearly displays the commands it runs on your behalf, in a beautiful shade of red;
• It has an ‘explain’ meta-command that will simply display the list of commands instead of running them for you.

Examples

Help

Let’s start with the simplest of all:

$ grb

or

$ grb help
git_remote_branch version 0.2.2

  Usage:

  grb create branch_name [origin_server]

  grb delete branch_name [origin_server]

  grb track branch_name [origin_server]

  If origin_server is not specified, the name 'origin' is assumed (git's default)

  The explain meta-command: you can also prepend any command with the keyword 'explain'.
  Instead of executing the command, git_remote_branch will simply output the list of commands
  you need to run to accomplish that goal.
  Example:
    grb explain create
    grb explain create my_branch github

  All commands also have aliases:
  create: create, new
  delete: delete, destroy, kill, remove
  track: track, follow, grab, fetch

As you can see, the syntax for all commands is very regular: action, branch_name and optionally, origin_server.

To facilitate learning even more, aliases are also provided. So to take an example,

$ grb track his_branch

and

$ grb fetch his_branch

are perfectly equivalent.

create

Create lets you create a new branch both remotely and locally. Note that it’s not made to share an existing branch (that feature’s coming).

So what it does is to push your current branch as a new remote branch, then create it locally and track it, for easier pulling afterwards.

$ grb create some_branch
git_remote_branch version 0.2.2

git push origin master:refs/heads/some_branch
Total 0 (delta 0), reused 0 (delta 0)
To /path/to/repo/
* [new branch]      master -> some_branch

git fetch origin

git branch −−track some_branch origin/some_branch

git checkout some_branch
Switched to branch "some_branch"

delete

Presenting features with names that are too self-evident is boring. Let’s get to the point, already.

$ grb delete some_branch
git_remote_branch version 0.2.2

git push origin :refs/heads/some_branch
To /path/to/repo/
- [deleted]         some_branch

git checkout master
Switched to branch "master"

git branch -d some_branch

track

Track lets you easily track the changes that are made to an existing remote branch you’re not tracking already. Each time you pull from the remote repository, the local branch will be automatically merged with the remote branch.

$ grb track his_branch
git_remote_branch version 0.2.2

git fetch origin
From /path/to/repo/
* [new branch]      his_branch -> origin/his_branch

git branch −−track his_branch origin/his_branch

explain

Explain will spew out all commands necessary to accomplish one of the previous actions. There are two ways of using it. The simplest will give you dummy commands:

$ grb explain create
git_remote_branch version 0.2.2

List of operations to do to create a new remote branch and track it locally:

git push origin master:refs/heads/branch_to_create
git fetch origin
git branch −−track branch_to_create origin/branch_to_create
git checkout branch_to_create

Or you can have steps that are tailor-made for what you want to accomplish.

$ grb explain create my_branch github_origin
git_remote_branch version 0.2.2

List of operations to do to create a new remote branch and track it locally:

git push github_origin master:refs/heads/my_branch
git fetch github_origin
git branch −−track my_branch github_origin/my_branch
git checkout my_branch

get git_remote_branch

(Hey, this title has a nice ring to it)

sudo gem install webmat-git_remote_branch −−source=http://gems.github.com

Now with rubygems 1.2.0 out (don’t do it with a prior version), you can also add GitHub as a permanent source for your gems:

sudo gem sources -a http://gems.github.com

Now and ever after, you will be able to get anything from GitHub with a simple

sudo gem install webmat-git_remote_branch

If in your eagerness you’ve added github as a source before running

sudo gem update −−system

Please refer to What to do when gems.github.com breaks gems FOREVAR

Look ma, no tests!

Uhhhh, yeah, I know…

This tool is still extremely early in its life and - dare I say it - it’s only hand-tested for now. I’ve been using it personally for a while and it’s working very well for me, if that means anything :-)

So this is still a quick script, only now it has lipstick.

This software should be considered an early version of a pre-alpha. You’ve been warned!

If this makes you queasy, there’s always the ‘explain’ command that can act as your cheat sheet without actually having grb run the commands on your behalf.

For the extra queasy, well you can find the commands very easily without running grb. Just point your favorite editor to lib/git_remote_branch.rb. All commands are there, at the beginning of the file.

For the brave, try it out and tell me what you think. Improvements, bug reports and contributions are all welcome.

And remember, in case of an emergency, you can always refer to my illustrated guide to recovering lost commits with Git ;-)

Here’s what’s to come:

• lots of tests
• a ‘remotize’ functionality (for existing local branches)
• a much better resilience to use in faulty situations (e.g. deleting something that’s not present locally or remotely)
• the possibility to specify different branch names locally vs remotely
• slap an open source licence on it all
• and so much more!

I’m kind of jealous. This guy put out 2 freaking good blog posts in 3 days. These are apparently his first 2 blog posts ever, too :-)

You’ll never find a group who wears their ignorance of technology more proudly than the average business person. “I’m not a computer guy,” they’ll say with a big smile on their face. Well gee, the personal computer is only the most significant invention to come along in the past 100 years.

By Tales of an IT Director

Git is one hell of a powertool.

Like with any such tool, as soon as you get to know it enough, you start pushing the boundaries. Git gives you a lot of control over your repository:

• trivial branching and merging (even with long lived branches);
• rebasing as a cleaner alternative to merging;
• stashing aside your changes for a quick fix elsewhere;
• extracting logical, distinct commits from a multi-hours coding spree;

The list goes on…

More traditional version control systems don’t give you as much power as Git by any stretch of the mind. They are like taking a walk in the woods with your parents, at age 14.

You’re probably gonna see and do neat stuff, but you sure ain’t gonna get lost or anything.

Using Git on the other hand is more akin to being handed a cool motocross to go play alone in the woods… Also at age 14.

We all know what’s bound to happen, right?

You’ll smash into a tree.

The source control equivalent to slamming into a tree is losing commits. Getting all of Git’s power and flexibility at once can be somewhat dangerous. You’ll find it so easy and helpful to branch and merge that you’ll start doing it way more often. On the other hand — especially in the beginning — you’ll misunderstand or plainly miss some important warnings, and make errors. Or you may just end up in weird merging situations you never thought of, and don’t necessarily understand. These situations can often result in losing commits or whole branches.

My goal with this article is to make sure you understand the situation you’re really in: you have temporarily lost commits or branches.

Disclaimer

This article assumes a basic knowledge of how git works, e.g. committing, branching and merging.

My first time

The first time I lost a commit was a good while ago. I can’t remember the details, but basically I got bit by the fact that under the covers, Git uses hard links liberally. Which means that copy / pasting your code directory as a recovery solution isn’t going to save your ass when you attempt a potentially damaging operation you don’t fully understand.

Note that compressing your code directory will, though.

So there I was, after attempting an operation I didn’t really understand. I knew I had failed what I attempted and I knew I had lost my last commit. Ironically, I still had Gitk open, displaying that very commit. As long as I didn’t refresh the Gitk view with F5 I could see the lost commit.

Here’s a fun fact: under OSX (not sure about Linux) you cannot select and copy text from Gitk’s interface, except for the SHA1 field [1]. I knew Git probably had a way to recover from that… But you know, I just wanted to get back to work and NOT search documentation and blog posts endlessly.

So I took screenshots, passed them real quick through GOCR, just to see how far it would get.

The result: GOCR doesn’t like the font Monaco :-)

How to (really) recover lost commits with Git

Recently I lost a commit again. This time however, Gitk was not up to date. I knew I’d just lost something I wouldn’t necessarily remember in its entirety. It was a commit an hour old, touching many files. And I have a crappy memory.

This time I had to do it the right way. I found out it’s really easy (once you figure it out), but I found no really clear explanation anywhere. So here goes.

Initial setup

If you wanna follow along — and I strongly recommend it — here’s the boring few steps to create a dummy repo and bring it up to speed with for the rest of this article. We’re going to beat the hell out of this repo and it’s going to be fun.

So just paste the following into a console:

mkdir recovery;cd recovery
git init
touch file
git add file
git commit -m "First commit"
echo "Hello World" > file
git add .
git commit -m "Greetings"git branch cool_branch
git checkout cool_branch
echo "What up world?" > cool_file
git add .
git commit -m "Now that was cool"
git checkout master
echo "What does that mean?" >> file

Ok, let’s look at where we’re at:

gitk ––all &

The ––all option lets you see all branches at the same time, as well as your stashes.

Click here to enlarge your picture!!1

We can see the cool_branch as well as some yet uncommitted changes over the master branch.

mathieu@ml recovery (master)$ ls -l
total 16
-rw-r--r--  1 mathieu  staff    15B  7 Jun 18:19 cool_file
-rw-r--r--  1 mathieu  staff    33B  7 Jun 18:19 file

Got my 2 files, I’m good to go.

Let’s make a mistake

Let’s say I decide I want to bring in these cool changes in master. I’ll do it with a rebase. I know there’s no big risk of conflicts so that’s a no-brainer.

mathieu@ml recovery (master)$ git rebase cool_branch
file: needs update

Now if you look carefully you’ll notice I wasn’t paying attention when Git gave me a feeble complaint about ‘file’.

Everything’s well, so I think “Ok, I don’t need cool_branch anymore”.

mathieu@ml recovery (master)$ git branch -d cool_branch
error: The branch 'cool_branch' is not an ancestor of your current HEAD.
If you are sure you want to delete it, run 'git branch -D cool_branch'.

Huh? Whatever you say, Linus. Let’s get on with it.

mathieu@ml recovery (master)$ git branch -D cool_branch
Deleted branch cool_branch.

Ahh, it feels good to be a Git ninja. Now let’s see where we’re at and refresh Gitk with F5.

Oops, my cool commit is gone! That thing can’t be right. Let’s panic:

mathieu@ml recovery (master)$ ls
file

mathieu@ml recovery (master)$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#
#    modified:   file
#
no changes added to commit (use "git add" and/or "git commit -a")

mathieu@ml recovery (master)$ git diff
diff --git a/file b/file
index 557db03..f2a8bf3 100644
--- a/file
+++ b/file
@@ -1 +1,2 @@
 Hello World
+What does that mean?

Oh sh!t

So the ‘file: needs update’ message back there meant that the rebase didn’t happen, because I had pending changes.

Helpful.

Recovering a lost commit

Since I don’t think my uncommitted work is complete, I’ll just stash it instead of committing it. Then I’ll hunt down my lost work.

mathieu@ml recovery (master)$ git stash save "Questioning the universe"
Saved working directory and index state "On master: Questioning the universe" HEAD is now at 6da726f... Greetings

In the name of paranoïa, let’s make sure this got in right:

Ok, let’s get on with our rescue mission:

mathieu@ml recovery (master)$ git fsck −−lost-found
dangling commit 93b0c51cfea8c731aa385109b8e99d19b38a55be

That sounds right, exactly one commit in the lost and found.

Let’s just make sure:

mathieu@ml recovery (master)$ git show 93b0c51cfea8c731aa385109b8e99d19b38a55be | mate

Bingo!

Different ways to recover the commit

There are a few different ways to recover that commit. Obviously we can just copy and paste that snippet, but in the case of a bigger commit, that approach will just amount to a lot of error-prone busywork.

I’ll reclaim my Git ninja status and try it a few different ways.

Recover it with rebase

Let’s just replay this change on top of master:

mathieu@ml recovery (master)$ git rebase 93b0c51cfea8c731aa385109b8e99d19b38a55be
First, rewinding head to replay your work on top of it...
HEAD is now at 93b0c51... Now that was cool
Fast-forwarded master to 93b0c51cfea8c731aa385109b8e99d19b38a55be.

Neat! Now I feel like a ninja worthy of the title again.

So let’s rewind one commit and try it another way.

mathieu@ml recovery (master)$ git reset --hard head^
HEAD is now at 6da726f... Greetings

Ok, the commit’s gone.

(Don’t tell anyone but my inner ninja is feeling queasy again.)

Recover it with merge

There are cases where rebase is not powerful enough. For example when you expect to face a lot of conflicts. In this case merge is a better solution:

mathieu@ml recovery (master)$ git merge 93b0c51cfea8c731aa385109b8e99d19b38a55be
Updating 6da726f..93b0c51
Fast forward
 cool_file |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 cool_file

Too easy… Rewind!

mathieu@ml recovery (master)$ git reset --hard head^
HEAD is now at 6da726f... Greetings

Recover it with cherry-pick

If instead you had a few commits one after another but you just want to pick the last one, rebase and merge won’t do. They would bring the whole branch back in master. That’s a situation for cherry-pick.

mathieu@ml recovery (master)$ git cherry-pick 93b0c51cfea8c731aa385109b8e99d19b38a55be
Finished one cherry-pick.
Created commit f443703: Now that was cool
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 cool_file

Insane!

This only leaves one open question: WHO’S YOUR DADDY NOW, GIT?

Now that we’ve established the answer to that question, let’s get back to work!

Let’s make a second mistake

mathieu@ml recovery (master)$ git stash clear

Or was it Git stash apply?

Oh jeez, there we go again…

mathieu@ml recovery (master)$ git fsck −−lost-found
dangling commit 24e3752f7a73ae98b361ce1c260e1f285d653447
dangling commit 93b0c51cfea8c731aa385109b8e99d19b38a55be

Ok, we still see the one we lost earlier, 93b0c51… Let’s look at the other one.

mathieu@ml recovery (master)$ git show 24e3752f7a73ae98b361ce1c260e1f285d653447
commit 24e3752f7a73ae98b361ce1c260e1f285d653447
Merge: 6da726f... c90f079...
Author: Mathieu Martin <webmat@gmail.com>
Date:   Sat Jun 7 16:02:57 2008 -0400

On master: Questioning the universe

diff --cc file
index 557db03,557db03..f2a8bf3
--- a/file
+++ b/file
@@@ -1,1 -1,1 +1,2 @@@
Hello World
++What does that mean?

Spot on. Let’s try something wild, while we’re here.

mathieu@ml recovery (master)$ git checkout 24e3752f7a73ae98b361ce1c260e1f285d653447
Note: moving to "24e3752f7a73ae98b361ce1c260e1f285d653447" which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
  git checkout -b <new_branch_name>
HEAD is now at 24e3752... On master: Questioning the universe

mathieu@ml recovery (24e3752...)$

As you may have noticed, my console always indicates which branch I’m in, so far [2]. But now I seem to be in some kind of twilight zone, which Gitk confirms.

Let’s follow Git’s suggestion and make that a branch.

mathieu@ml recovery (24e3752...)$ git checkout -b recovery
Switched to a new branch "recovery"

mathieu@ml recovery (recovery)$

Looks weird, like stashed items always do, but at least we have our commit.

After fiddling around with what’s been recovered from the stash, I recommend NOT keeping it as a commit.

If you try to replay the change in the recovery branch over master’s most recent commit, you lose the “Questioning the universe” commit. Probably because a stash is a weird kind of commit, or maybe because of a bug. I don’t know.

(Don’t follow this one in your console)

mathieu@ml recovery (recovery)$ git rebase master  #I said don't do this one
First, rewinding head to replay your work on top of it...
HEAD is now at 93b0c51... Now that was cool
Nothing to do.

If instead I checkout master and then rebase its last change over the ‘recovery’ branch it seems to work.

However since I just saw a commit disappear when rebasing the other way around, I get the feeling that this isn’t a normal commit and it may come back to haunt me later.

Recover it by applying a diff

Let’s just apply the diff to master. I’ll do as if it actually was a substantial commit, involving lots of modifications on lots of files, and apply it automatically with ‘git apply’.

First let’s visualize where we’re at, again:

A diff against master is not what we want since master includes a new (very cool) commit.

Instead we just want to see the changes introduced by the current commit. To do this we can compare it with the common ancestor between the master and recovery branches. So let’s start by finding it’s ID.

mathieu@ml recovery (recovery)$ git diff 6da726f37683c83947d54314cd32ca1ee9d490e0
diff --git a/file b/file
index 557db03..f2a8bf3 100644
--- a/file
+++ b/file
@@ -1 +1,2 @@
Hello World
+What does that mean?

Looks good. Now we throw that diff upstairs.

git diff 6da726f37683c83947d54314cd32ca1ee9d490e0 > ../recovery.diff

Then get apply it to our master branch.

mathieu@ml recovery (recovery)$ git checkout master
Switched to branch "master"

mathieu@ml recovery (master)$ git apply ../recovery.diff

And we finally confirm that everything’s under control.

mathieu@ml recovery (master)$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#
#    modified:   file
#
no changes added to commit (use "git add" and/or "git commit -a")

mathieu@ml recovery (master)$ git diff
diff --git a/file b/file
index 557db03..f2a8bf3 100644
--- a/file
+++ b/file
@@ -1 +1,2 @@
Hello World
+What does that mean?

This change was first stashed rather than committed because I felt it was not complete. Applying it with Git apply only introduces it as an unstaged change, which works perfectly for this situation. Now I can keep banging at the code until I feel this actually deserves to be committed.

mathieu@ml recovery (master)$ echo "I don't know" >> file

mathieu@ml recovery (master)$ git commit -a -m "Conversation of staggering depth"
Created commit 65a4794: Conversation of staggering depth
 1 files changed, 2 insertions(+), 0 deletions(-)

Cleaning up the crud

Ok, so now I still have this weird looking recovery branch.

Since it’s now useless we can get rid of it.

mathieu@ml recovery (master)$ git branch -d recovery
error: The branch 'recovery' is not an ancestor of your current HEAD.
If you are sure you want to delete it, run 'git branch -D recovery'.

Aha! This time everything’s committed correctly, so I know I can delete it for real. Git is complaining because that commit was not included through its normal merge or rebase commands. So it warns me that I may be about to lose something. However I know I got everything through the diff I made and re-applied.

mathieu@ml recovery (master)$ git branch -D recovery
Deleted branch recovery.

Now that I’m aware that commits are reachable even if they’re not in a branch anymore, I wonder about my repo’s size.

mathieu@ml recovery (master)$ git gc
Counting objects: 22, done.
Compressing objects: 100% (14/14), done.
Writing objects: 100% (22/22), done.
Total 22 (delta 7), reused 0 (delta 0)

mathieu@ml recovery (master)$ git prune

Fair enough. I would expect the unused commits to now be unreachable, but strangely enough:

mathieu@ml recovery (master)$ git fsck −−lost-found
dangling commit 49ed65cdea22443af3f1fd400754fe1517421b24
dangling commit 4b1bf4792cba929e88114379d7d5e86a2dc9990f
dangling commit 6cdf88318109dede7bd3c1a75be76c7255708ded
dangling commit 715a6b2cfe797383216d0f9b04fe8f50e90e779f
dangling commit f443703e5060d9f3b4d97504bda5f97e5a0b31e8

If anyone finds out what that’s all about, please let me know!

Maybe Git’s just refusing to do any work unless it’s going to actually save a considerable amount of space? I have no idea.

Conclusion

Once you know how to recover from bad mistakes, you’ll find that Git is not only a very powerful tool, but also a very forgiving one. As opposed to a motocross.

The following commands will help you figure you way out of most bad situations:

• git show
• git fsck −−lost-found
• git diff

And these ones will actually get out of these bad situations:

• git rebase
• git cherry-pick
• git merge
• git apply

As I think I demonstrated, Git gives you the ability to recover from most bad mistakes. The fact that any single commit can be cherry-picked, checked out, rebased or merged makes it really easy to recover from hairy situations.

The only case where you might actually lose information is when something has not been committed or stashed yet, which I think is perfectly reasonable.

So if you take only one thing away from this article, let it be this. Git is much safer than a motocross.

Footnotes

[1] At the time I didn’t know that just having the SHA1 id was enough to save me.

[2] See how to configure your console in the same manner and also get auto-completion for Git here.