Category Archives: Programming

Piecemeal Feature Development and Merging

Posted on by
Reply

Branching and merging is a staple of team software development, since it’s very important that programmers working on the same code base don’t mess with each other’s changes. But while branching is fun (and hassle free in git), merging is quite often a big pain. The longer the programmer stays in his own branch the more his code diverges from master. A good rule of thumb is that branches should be synced with master every day so that the branches do not diverge too much from the master. If the whole feature can be developed in a single day, than merging it back into a master on the same day is not a problem. But if the feature is big and is developed over several days it cannot be merged into the master piecemeally without changing the development practices.

The most important change in development practices is to break down the feature into several smaller pieces where each piece can be merged into master (and preferably deployed to the production) independently. If any of the pieces interferes with the existing functionality, the existing code should be refactored and made more extensible. I’ve heard several programmers objecting to this approach with an argument that it will make development longer. In my experience, the initial development of a complex feature in a piecemeal approach really takes longer, but once the feature is finished it’s really done. On the other hand I’ve heard too many programmers uttering the famous phrase “I’ve finished the feature. I only need to merge it into master” followed by several days of frustrations when the “finished” feature is being shoved into the master.

Related articles

Faking Ruby blocks in Python

Posted on by
Reply

Ruby has a construct called “block” that enables a programmer to define a block of code and pass it to a method. This construct comes handy whenever you want to do something on a block of code but you don’t want to put it to a separate function. Consider, for example, how tedious it would be if you would have to define a separate function for every while, for, or if statement, instead of putting the code in a block immediately following the control statement, such as in the following example.

def block_of_code:
  <instructions>

if(condition, block_of_code)

The closest equivalent in Python to Ruby blocks are lambda functions, but with important distinction that lambda functions are (intentionally) limited to one line of code. But since blocks are merely nameless methods that you can pass to other methods, there exists a trick using decorators how to have blocks also in Python.

Take for example that for some reason you’d like to have an infinite_loop construct in Python. One way how to implement it in the style of Ruby blocks is as follows

@infinite_loop
def _():
  <your instructions>

where infinite_loop decorator is specified by

class infinite_loop:
  def __init__(self, function):
    while True:
      function()

Using decorators as an approximation for Ruby blocks in Python is possible because the __init__ constructor for the decorator is executed at the point of decoration of the function.

I googled for other solutions how to fake Ruby blocks in Python, but I couldn’t find any. If you know of one yourself, let me know in the comments. And if you like this construct, maybe we can petition Guido to include in the future extensions of Python the following construct

@decorator:
  <your instructions>

that would enable decorating also an anonymous block of code, not just of a named function.

Related articles

django-probe

Posted on by
2

Unittests are a very nice way to construct scaffolding around code. Such scaffolding enables a programmer more care free development with a reduced fear that a change in one part of the code base would cause unwanted consequences somewhere else. If a project is part of a larger system, unittests quite often gets exploited also to implement system tests that verify that changes made in one component don’t have unwanted effects in other parts of the system. Using unittests also for system tests have some very annoying side effects:

  1. Running unittests requires setting dependent services,
  2. Calling external services considerably increases running time of unittests,
  3. System tests are brittle and require constant maintenance,
  4. System tests are run only at deploy time, not with every system upgrade

To address these problems I’ve developed a python module django-probe whose aim is to separate system tests from unittests. I named an individual system test a probe, since a probe is meant to be run not only at deploy time but periodically and as part of a production system.  Similarly as Django organizes tests into tests modules, probes are organized in probes modules. All you need to do to set up a new probe, is to make either a file probes.py or module probes and populate it with the following code:

import unittest

class TestMyProbe(unittest.TestCase):

def test_my_probe(self):
 # usual unittest code
 # But don't forget. This unittests will be run in the production setting on live data!

Additionally, you should add django_probe to your list of INSTALLED_APPS and PROBE_RUNNER = ‘django_probe.simple.DjangoProbeSuiteRunner’ to your settings.py.

Probes should not be descendants of django.test.TestCase class but of unittest.TestCase class. Django’s TestCase replaces several services with mocks which could wreak havoc if run in production.

Probes are run in a similar fashion as unittests but with a different management command probe

./manage.py probe        # Run all probes in all applications

./manage.py probe app        # Run all probes in application app

./manage.py probe app.TestMyProbe        # Run all probes in class TestMyProbe

./manage.py probe app.TestMyProbe.test_my_probe        # Run a single probe

Django command probe is meant primarily to be executed at deploy time. In order to enable easy execution of probes at runtime django-probe enables triggering of probes through a web interface by accessing /probe url. To install this option add the following to your urls.py

(r'^probe/', include('django_probe.urls'))

The whole django-probe project is a result of a single day of hacking at Zemanta‘s summer hackday. As such is in a very preliminary phase and I don’t even know if the whole approach makes sense. Your comments would be greatly appreciated. Of course, forking the code on github.com/dusano/django-probe would be appreciated even more.

Related articles

A Unittest should take at most 10ms to Run

Posted on by
2

For a typical project of ours, it normally takes 10 minutes to run all 300+ unittests. No wonder that I overheard the following conversation in the office yesterday

Pančo: Why are you playing Manic Miner?
Tom: I’m waiting for unittests to finish.
Pančo: Oh. Carry on.

When you develop new code and corresponding unittests, one normally doesn’t put lot of attention to execution time of unittests. After all, a typical unittest takes imperceptible amount of time to execute. But once you multiply this imperceptible amount of time by the factor of 300 or more, the total running time of unittests becomes very apparent.

I think that total execution time of unittests should not excede 10 seconds. That’s probably the maximum time that you’re willing to just sit and observe unittests being run, without switching to some other context (like playing Manic Miner). Since our bigger projects are covered by more than 300 unittests, a logical consequence of limiting total unittests execution time to 10 seconds is that a single unittest should not take more than 30ms to execute. But since our test coverage should be at least tripled, I would prescribe that a single unittest shouldn’t take more then 10 miliseconds to execute.

With our current situation of unittests requiring 600 seconds to finish, we are still very far away from the goal of unittests not slowing us down. But as is the case with almost everything, patience and perseverance deliver results.

Related articles

Which jars is my java application using?

Posted on by
Reply

I’m pretty sure that this trick is well known among Linux users. But to me it was quite a revelation. If you issue command

$ ls -l /proc/<PID>/fd

it will list all the open files of a process identified by <PID>.

This comes especially handy when developing java application. A frequent source of problems in java applications is using incorrect version of jars. While situation with jars in java is not as bad as dll hell in Windows, it is not that far off. When debugging a java application, one of the first thing that you should do is to check if your application is using correct version of jars.

Here’s partial output for a running instance of Apache Lucene/Solr

$ ls -l /proc/20879/fd

0 -> /dev/null
1 -> /home/user/solr/apache-tomcat-5.5.28/logs/catalina.out
…
18 -> /home/user/solr/apache-tomcat-5.5.28/common/lib/commons-math3-3.0.jar
50 -> /home/user/solr/apache-tomcat-5.5.28/server/lib/catalina.jar
51 -> /home/user/solr/apache-tomcat-5.5.28/server/lib/tomcat-http.jar
53 -> /home/user/solr/apache-tomcat-5.5.28/server/lib/tomcat-coyote.jar
…

Besides the list of jars used by the application, you also get the location of catalina.out, which is the place where Solr by default logs all it outputs.

1054 unittests

Posted on by
Reply

Last week I checked out all 12 main projects that constitute Zemanta and ran their respective unittests. To my great delight I saw 1054 unittests being run with only 3 of them failing. This is quite amazing result since just a year ago the total number of unittests was much lower and number of failing tests much higher. Actually we had so many failing tests that we regularly committed code even if the tests did not pass through. But for the past year we have spent quite some time fixing failing unittests and required that new functionality has good code coverage.

All this came about as quite natural by-product of code reviews. Most of the new functionality that we develop is just extension of the existing functionality. In order to review the code implementing the new functionality you must understand the code. And the best way to understand the code is to execute it. Executing a lump of code buried deep down within a large system is not trivial and unittests are usually the best way to do it. Additionally, unittests are very good at exposing inputs and outputs of the new functionality, thus further helping in understanding of the new functionality.

If you haven’t yet introduced code reviews into your development process, you should do it immediately. Even if you’re a one-man shop, you should do code reviews. Send them to yourself and wait with a review for a day. I promise you that it will be quite often really hard to grasp the ignorance of the person who has sent you the review 🙂

Related articles

Comments are not Code (redux)

Posted on by
2

Yesterday, Tomaž pleasantly surprised me with the following tweet

Just a few days before I wrote in a code review (that Tomaž wasn’t part of)

It’s a good rule of thumb: if you need to comment something, quite probably there is something wrong with your code.

I fancy myself that Tomaž’s tweet might be influenced by my constant nagging about comments and code reorganization. Ever since we have introduced code reviews at Zemanta, I see tremendous change in the way we do and approach coding. While my coworkers have traditionally written the code mostly for the machines to understand, they are increasingly aware that their primary audience are their fellow coworkers. Especially in the case of Zemanta and other service providers, the need for comprehensible code is paramount for productive development. We develop our service in an incremental, piecemeal way, always fixing, optimizing, or improving the part of the system with the highest priority. Even if you are author of a particular subsystem, you’ll be completely lost in that code base after not seeing it for half a year or more. And once you do touch the code you’ll be very grateful to yourself for not writing that piece of code in brainfuck.

Related articles

Logging to /dev/null

Posted on by
5

For the last few days I’ve been debugging Zemanta‘s system a lot. Besides noticing how poor our logging practices really are, I’ve also noticed that our logs are full of warnings, errors, and exceptions that nobody cares about. We have a complex service running composed of many different parts. Each of this parts is keeping a separate log file that resides on a server somewhere. A comment by one of our Ops people is quite telling in describing the situation:

Oh, we have a log for this application.

I won’t reveal which application this is, but it is one of our more important applications. It seems that in the case of logging, the saying “out of sight, out of mind” quite accurately characterize usual state of affairs.

I’ve been thinking for quite some time now, how we could improve our logging situation. One obvious way is to go regularly over all logs and manually check for anomalies. But if you prefer to spend your time on more creative endeavors, the log checking process should be automated in either push or pull fashion. Unfortunately, log aggregation seems to be quite a neglected area. I only know Facebook Scribe as a push log aggregator, and Splunk or Loggly as pull log aggregators, but I don’t have any real life experience with any of them. Therefore, I’d really like to hear from you what’s your take on log aggregation and if you have some first hand experience with it yourself.

Related articles

Blinkenlights and the Church of Graphs

Posted on by
Reply

Last week Swizec wrote a brilliant (>30K views, >300 RTs) post for Zemanta’s FruitBlog about our internal status dashboard aka. Blinkenlights. I wanted to write a post about importance of measuring everything for a long time, but Swizec put it so eloquently that the only thing I have to add to his post are some implementations details that will hopefully show how trivial it is to build an effective visualization of your system.

Our dashboard is just a regular web application that uses divs for layouting charts and graphs on the screen. The dashboard is mainly supported by backend and Operations people, so complex JavaScript is beyond our capabilities. The charts and graphs themselves are not included through HTML, but through JavaScript on DOM ready event. Graphite graphs and other images are loaded into an img element and then periodically refreshed. Charts that require more complex rendering and special JS scripts are organized as independent web pages that are loaded into divs of the main page using jQuery load method. Finally, charts that are served from different domains, are included as iframes due to same origin policy of the browsers.

It took me a hackday and a weekend to build the whole dashboard (and special version of it that works on iPad) and I’m not particular versed in web technologies. I hope this will provide an encouragement for you to build your own dashboard and join “The Church of Graphs” also yourself.

Related articles

Code polish using Git Rebase

Posted on by
1

We are using git for more than half a year now and I just cannot believe that we could ever do anything using subversion. The difference between git and svn feels as big as a difference between svn and managing versions by zipping folders. In subversion it is really difficult to alter code once you committed it to the repository and consequently programmers are usually waiting with commits until the code is fully polished. In git, on the contrary, commits feel more like temporary snapshots with no pressure on programmer to polish them completely, since he has a full control over his local repository.

A big part of final code polish in git that I’ve started to use more frequently recently is interactive rebasing. Rebasing comes especially handy in preparing code for a review. For example, if you’ve included code of external libraries into the project, you can use rebasing to move commits of library code before the commits with your code changes. Doing so makes it much easier to pack code for a review. Similarly, if you decide to do some code reorganization in the midst of a programming spree, you can use rebasing to move commits of reorganized code before commits of new functionality. Another useful practice that it’s made really easy with git rebasing is squashing south migrations into a single migration before pushing code to the main repository.

I see preparing code before pushing it to the main repository and packing it for a code review, as the very first code review. The moment just before git push is a nice final check if your code makes sense and if it is polished enough to be presented to your coworkers in the form of a code review request.

Related articles