Code review tool comparison

  To review or not to review?

  “Team work: most probably you will fail if you only want to work alone.”

  Sometimes a good review can open your eyes on how awesome you think your solution is. Authors need editors to track down and hunt mistakes. But what about us – software developers? We also write lines and it’s within our nature to carry out errors. It might be the reason we usually don’t work alone. The point is, software developers need the same assistance like the authors of novels, to achieve the goals of the software project, and organization overall. After all, “If debugging is the process of removing software bugs, then programming must be the process of putting them in.” (E. Dijkstra)

  The reasons for the reviews can be as many as the environments in which they are held, but the top ideas behind this are:

  • To create a software product as reliable as possible, and to ensure that it’s well documented;
  • To make sure that every new piece of code that is registered respects the enterprise’s coding standards;
  • To share knowledge between developers.

  The actual time spent on the review itself is well worth the previously mentioned benefits.

  But what is actually a code review? I am pretty much sure that most of the programmers will think about meetings where somebody is presenting a piece of software he’s been working on for a while, pointing out the key aspects of the changes, and explaining how all this works. Other guys can imagine the e-mail inbox full of review requests, equipped with diff files and additional explanations. No matter how it is done, it’s still a code review.

  Ideally, the reviewers should be completely disengaged from the project they are reviewing, as this maximizes their degree of objectiveness and minimizes the conflicts between the team members, that can result from the negative feedback they’re receiving. Actually, during these meetings, people also are pointing out positive aspects of the change, but the criticism is usually worth more than the compliments in the software development teams.

  All this sounds very good, but nevertheless, it has its downsides. Code reviews consume time, and this can grow exponentially, along with the size of the team.

  The second big drawback is the availability and the commitment of the reviewers: it’s usually a very difficult task to gather all the experienced programmers around a table at the same time, because they have some other work to do too.

  The third disadvantage is the fact that the code review is led by the person who’s actually written it. The truth is, we cannot be fully objective about ourselves, we tend to get sensible because of the criticism and this can lead to pointless discussions and defense during the code review meeting.

  So what is the solution to all these points?

  It might be a code review software (keep on reading to understand why I am not entirely sure about this).

  As you may know, a code review software is an application, that allows creating discussions on a specific topic. People commit a piece of code or upload a diff file, image or word document to a place, where other members can join and discuss it without an actual meeting. This sounds so good that we’ve decided to give it a try.

  But to better understand which tool to choose from the available myriad of applications, I present you the comparison overview of the code review tools available on the market and in the open-source community.

  Even though there are also other tools present on the market, I’ve decided to investigate the following options:

Atlassian® “Crucible”

  The reason for checking out this tool is that we already use another application created by Atlassian: “Jira”, which allows us to track issues and manage time within the company. “Crucible” seemed to me like a piece to the big puzzle of our workflow.

Advantages:

  • Works well with either SVN, Git or Mercurial;
  • Very simple installation process;
  • Intuitive interface: it even looks familiar to those who already make use of Jira.
  • Supports linking with other “Atlassian” applications.

Fig.1 Atlassian’s “Crucible” review

Fig.2 You can freely add comments to a specific line of code making the review a lot easier

Disadvantages:

  • The paycheck can get high for large companies ($2750 for teams of up to 50 users). Don’t get me wrong, every software costs money, that’s why we’re in this business anyway. But “Crucible” comes packed with “FishEye” – Atlassian’s repository browsing software and you cannot benefit from “Crucible” entirely, without buying a license for the second one (another $2750 per year), as you would not be able to browse the repository and see the information about the change-sets that have been committed. As a result, you have to pay not for one, but for 2 products.;
  • It lacks some functionality like LDAP support, that is nonetheless present in their other apps, like Jira. Good news is that you can link the applications and benefit from the required feature, like, for example, linking “Crucible” with Jira user’s server (another $3300 per year), but it’s an obvious attempt to make customers use as many tools from their company, as possible, in order to fully satisfy their needs.

GitLab®

Fig.3 Local server running “GitLab”

  “GitLab” is a web-based Git repository manager with wiki and issue tracking features. It is available in both free and paid versions (i.e. community and enterprise editions).

  The general idea in “GitLab” is that you get your own private GitHub server, which means that you get GitHub functionality on your own. And as you may know, the GitHub has the code review functionality out of the box: that’s integrated into the working process.

  Git itself, cannot really do this. It’s the git repository service that has a very neat feature called ‘Pull request’. This means that when somebody from the team wants to add his/her contribution to the project, (s)he can request a review (a pull request), define the reviewers and wait for the review.

  Once all the necessary changes have been discussed and approved, it can be merged into the master branch.

Here’s what you get using “GitLab”:

  • Free, private GitHub server;
  • Continuous integration service, that is able to run tests on the commits;
  • Issue management system;
  • Slack like messaging-app – “Mattermost”;
  • Inline code editor that allows committing code directly from the web interface.

As you may see, “GitLab” comes very well equipped even in free version, but you still have to pay for:

  • Only Git support;
  • LDAP support;
  • Possibility to have more than 10 projects in the community edition;
  • Pull requests: that’s right, code review is not available in the community edition. (for this crucial feature you have to pay an additional $39 per user every year).

Phabricator

Fig.4 Phabricator’s main page

  Phabricator is an all-in-one software engineering web application and platform. It has issue/bug tracking, code review, repository navigation, wikis, etc. It’s the kitchen sink for software lifecycle management. It supports Git, SVN and Mercurial. And, it’s open source and easy to extend. Sounded promising.

  Phabricator is essentially a collection of applications tied together by a single web interface. So, everything is readily available under one domain. It ships with a command line tool called Arcanist, which simplifies the use of Phabricator: you can basically use it to create reviews, tasks and run tests. The code review comes from the review tool – Differential. It also supports post-push review and has a lot of other tools like blogging-platform called Phame, wiki called Phriction and polling application called Slowvote. It has a nice interface, although it’s a subjective matter. I like the tools availability it has, but in the case you already have specific solutions integrated in your organization’s workflow, it means that you have to rethink which tools you are going to use: Phabricator can really solve a lot of issues, but haven’t you already solved them?

  It’s free up to 5 users. You have to pay additional $20 for each user, per month.

Pros:

  • Handles both Git & SVN;
  • Integrates loads of tools;
  • Has an official API;
  • Good command line support through Arcanist;
  • Includes project activity feeds that normal humans can use;
  • Search works very well, unlike “Gerrit”;
  • Supports open registration with a variety of methods.

Cons:

  • Used for single review of a feature branch. It would be indeed nice if it supported diff of each commit in the feature branch, but it only supports diff of the entire feature branch.
  • Doesn’t manage repositories – would have to be interfaced with another repository management tool, or such a tool would have to be built.

Gerrit

Fig.5 “Gerrit” – Side by Side checking of the project’s files (the comments can be added by double clicking on each separate line)

  “Gerrit” is a web application that can be used for software reviews. It uses Git as version control system and it is installed on a server that holds the repository of the projects that are being developed. Each developer has to download the code for the project from the main repository and once the code is checked out, it can be re-committed to it through a Pending Changes step, which “Gerrit” takes care of. So once the code that has some changes made to is committed to this temporary repository, the reviewers kick in and decide whether they are going to approve this change or not.

Pros:

  • Free;
  • Has plugins (e.g. “Gerrit” plugin for Jenkins that allows verifying that the code compiles before actually committing it);
  • Acts as a repository: when you push to it and it creates a review issue of your commit. As a result: No commit ever gets through without review as it is impossible to forget to create review issue;
  • No need to use additional command line tools to push the code;
  • LDAP integration. “Gerrit” supports this out of the box with minimal configuration;
  • Automatic merge on approve.

Cons:

  • Git support only;
  • Can’t handle post commit review (auditing) at all;
  • Architectural idea: “Gerrit” forces a one commit at a time workflow on developers and forces the use of “git commit –amend” as the only way to update patches. As a consequence, it is impractical to work on local branches with multiple commits to be merged – this is one of the reasons to use git at all;
  • Can’t compete with commercial solutions in term of user interface.

  There are also nice free code review tools that even have free demos set. You don’t have to install them, to have an insight on how they look and behave.

Kallithea

Fig.6 Kallithea’s main page

  “Kallithea” is a free software source code management system that Mercurial and Git, and has a web interface. It is free and has all the main features required for code review:

  • Code review: in-line comments and overall commit specific comments;
  • Repository browsing with searching capabilities;
  • Full changelog visualisation;
  • Neat design: although very subjective matter;
  • Has a demo page, which means that you don’t have to install it in order to try it.

Cons:

  • Does not support SVN.

  In my opinion, “Kallithea” has a very big potential in terms of code review software. Adding SVN support would be an un-doubtable must in order to be adopted by the organizations that have been using it.

Review board

Fig.7 Review board’s main page

  Thing that makes Review board different from the other tools described above is the fact that you don’t really review code from the repository. The key aspect of this tool is that it doesn’t care about what you’re reviewing: you just upload the file (image, document, diff file in case of the source code) and the review is ready. The most interesting part about this tool is that it has some interesting features (it even has LDAP support, which $5000 “Crucible” doesn’t).

Why should you consider using this tool?

  • Free;
  • Does not care which SCM software you use;
  • Simple to use;
  • Can review whatever you need;
  • LDAP support;
  • It has a demo page which allows you to try it out without actually installing it.

Cons:

  • You have to upload files manually;
  • It is decoupled from the SCM;
  • Obsolete design.

Conclusion

  Code review would seem like an unknown territory but if you look closer, you’ll see that there are plenty of tools to choose from. Nevertheless, the code review should come naturally. Ideally it has to be fully integrated in the working environment of the team, i.e. no additional tool has to be installed and yet, no change in the code should be done without the team being aware of it. This can be easier achieved if you use “Git” as a subversion system, although given every specific workflow of any company, you can surely find the solution that fits you. There is no “to review or not to review” question: do it and watch your code base getting better.

Share this article:

Stanislav
Java Developer