R Analysis on Git Log Data

A few weeks ago at work one of the managers asked me an interesting question. They were preparing an end-of-year presentation and needed to some stats for a few slides. Paraphrased, 'How many commits did the team deploy in 2014 and is there anything else interesting in those numbers?'.

Please note: number of commits is a slimy metric to play with and should not (and is not at my workplace) be used for serious decisions.

Anyways, the number of commits is relatively easy to figure out. Git has a nice command 'git log' that will fetch commit messages and stats on a per-branch stats that also accepts filters to limit to certain users, timeframes, etc. The other part of the question is what struck me. Was there a way to analyze the data in the git log in a scientific manner to pull interesting statistics out? A way to use data science - R - on git logs? Well, of course there is.

The first step is to pull the data out of git. Like I mentioned earlier, 'git log' is a simple and easy way to produce a stream output on the commit history of a given branch. For example, here is a simple bash line to pull a nicely-formatted stream of history.

git log --date=local --no-merges --shortstat\
        --pretty_format:"%h,%an,%ad,%s"

Most of the params here are self-explanatory. I wanted the dates to be formatted based on my computer so I didn't have to translate between timezones, which R really stinks at doing, so I used the local setting. I didn't want any merges included in my stream. Also, I wanted to know how many files were affected, the number of lines inserted, and the number of lines deleted, which is what the shortstat line gives. The pretty_format is a bit more tricky… Each param is a placeholder. For more information on the pretty_format flag I recommend checking on the documentation on git log.

This works great for a quick view but I needed more. I needed something that I could parse in R. The author name, commit message, and shortstat fields each had the potential of including commas, so delimiting by that was useless. Plus git has an annoying way of dropping the stortstat to the next line, which further complicates parsing. I needed something more powerful.

git log --date=local --no-merges --shortstat\
        --pretty=format:"%h%x09%an%x09%ad%x09%sEOL" |\
        perl -pi -e 's/EOL\n/\t/g' > git.log

By using tabs as a delimiter '%x09', a token for the formatted end of line, and Perl to catch that end and crunch the shortstat onto the above line, I finally had something that was ready for analysis. Tab-delimited files are pretty easy for R to crunch through, even if comma-delimited makes more sense to us. Of course, tabs could be still embedded in author names and commit messages, but I wasn't worried about that case yet.

log <- read.delim(
    file = 'git.log',
    sep = "\t",
    col.names = c('commit', 'author', 'time', 'message', 'effect'),
    stringsAsFactors = FALSE
)

Loading the file into R is easy. Each column will be a character vector, which makes things a bit difficult to play with, and our last column is an annoying cluster of text. Some examples of the last column could include '1 file changed, 34 insertions(+), 3 deletions(-)' or '1 file changed, 303 deletions(-)'. Plus the time column is a flat string, no date information stored in it. This data needs to be cleaned up in R before moving forward.

#format column into date format
log$time <- strptime(log$time, format = '%a %b %e %H:%M:%S %Y')
#setup threw new columns for stat information
log$files_changed <- numeric(nrow(log))
log$insertions <- numeric(nrow(log))
log$deletions <- numeric(nrow(log))
#loop through stats to backfill stat information
for (row in 1:nrow(log)) {
    log$files_changed[row] <- regmatches(
        log$effect[row],
        regexec('([0-9]+) files? changed', log$effect[row])
    )[[1]][2];
    log$insertions[row] <- regmatches(
        log$effect[row],
        regexec('([0-9]+) insertions?', log$effect[row])
    )[[1]][2];
    log$deletions[row] <- regmatches(
        log$effect[row],
        regexec('([0-9]+) deletions?', log$effect[row])
    )[[1]][2];
}
#re-cast numeric columns as numeric
log$files_changed <- as.numeric(log$files_changed)
log$insertions <- as.numeric(log$insertions)
log$deletions <- as.numeric(log$deletions)

There. Now the data is nice and clean and ready to run analysis on. For example, you can easily figure out a breakdown of inserts and deletes by author like this:

#limit to 2014 data
log2014 <- log[
    format(log$time, '%Y') == '2014',
    c('author', 'insertions', 'deletions')
]
#output breakdown of inserts by author
aggregate(insertions ~ author, log2014, sum)
#output breakdown of deletes by author
aggregate(deletions ~ author, log2014, sum)

Or, for something more visual, you can graph the number of files affected per commit over the year. Here's an example of using ggplot that will make a pretty graph for the year's data, using one of Bigstock's internal repos as a data source.

#limit to 2014 data
log2014month <- log[
    format(log$time, '%Y') == '2014',
    c('commit', 'author', 'time')
]
#add column for month/year aggregation
log2014month$short_time <- strftime(
    log2014month$time,
    format = '%Y/%m'
)
#aggregate commits by author by month
log2014month <- aggregate(
    commit ~ short_time + author,
    log2014month,
    length
)
#constrain result to at least five commits per month
log2014month <- log2014month[
    log2014month$commit >= 5,
]
#graph the result using ggplot
ggplot(log2014month, aes(short_time, commit)) +
    geom_point(aes(color = author))

There is a lot more that one can do with this data, just with tweaking and playing with the fields that we brought in. Plus, we didn't even try to parse the commit messages. Or the merged branches. Either one of those facets could bring in a lot of different data and insights into codebase history and flow. Still, this isn't a bad start to playing with git history data.