Introduction

Testing is essential to software quality. The success of service oriented companies heavily depends on the quality of their software. It is still not clear in what way to apply functional test automation in order to significantly improve software quality.

Here I will describe an approach for how to successfully implement functional test automation which in comparison to the traditional approach significantly reduces the cost of ownership of a test suite. The approach mitigates the risk of fast evolving software through the use of maintainable functional test automation.

This paper will identify and describe the problems of functional test automation and propose an approach for how to address the problems.

My contributions:

• My approach significantly reduces the total cost of ownership of an automated functional test suite.
• I have been experiencing in practice that domain experts are not involved in testing and, what is worse, domain specific tests are neither documented nor executed. I show how this approach involves the domain experts in the functional test automation.
• I analysed existing approaches that could support DSL Modelling. I show how to use the existing knowledge and leverage it in order to successfully jump-start a test automation initiative.
• I demonstrate how to implement this approach in Python.
• I use an up-to-date example throughout the whole paper to show the utilization of the various aspects of my approach.

The Problem

Software experts agree that testing is essential to software quality. As with software engineering, strategies vary on how to do it. When arguing about the right testing strategy the discussion circles around the following contexts:

• What level of application testing is appropriate? Should we focus on unit testing, functional testing, GUI testing, integration testing?
• How do we determine if the testing is finished? How do we decide if the product can be shipped/ released?
• How often do we test? Every release, every month, every week, every day, or even after each change to the source code?
• What tools do we use?
• What methodologies should we follow?
• What amount of tests should we automate?
• Who performs the testing? Who owns the testing discipline or process?

Test automation does not solve or even address all of the questions above but it should definitely be considered in your testing strategy. The biggest benefit of automated testing is that once the tests are implemented you can run them as often as your hardware allows at no additional cost. Also, automated tests are reproducible whereas manual tests, in many cases, are not. On the other hand, test automation itself is another expensive software development project. An automated test suite must be maintained to be effective, which is expensive, as well. For example GUI level test automation is typically created with the capture and replay method. If the GUI changes substantially the test cases need to be re-recorded. In an environment that evolves rapidly this can be hard to achieve. As an analogy to software-refactoring, in software development the test suite should be refactored to encourage reuse and the DRY (Don’t Repeat Yourself) pattern.

The following diagram shows the information flows in a typical software development process for service oriented industries such as telecommunications.

The diagram gives an overview of the information flows between the business units within a company. The technical business units provide services like software development, end-to-end-testing and systems operation. The business department understands the markets in which the company operates and defines product strategies. The people in the business department define the requirements for software development and are the domain experts. They work closely with analysts from the technical development unit who capture and analyse the requirements. Experienced analysts can also be seen as domain experts. Domain experts frequently do not define domain specific requirements because, by nature, these seem trivial or too intuitive to specify. An example for such implicit domain knowledge is what products are placed in what channels. Instead they focus entirely on specifying requirements for the computer system. The problem I want to point to is that testers may lack domain specific knowledge, which in turn impacts the test results. As a rule, testers do not get in touch with domain experts because they work for different business units within the company. The test analysts and testers base their test case documentation on analysis and development artefacts such as requirements, analysis and specifications. If requirements have been based on computer system requirements, as opposed to requirements based on domain knowledge, then these artefacts will not contain domain specific knowledge. This means domain specific tests are not performed. As a consequence this can only mean that you must involve the domain experts in test initiatives. Leaving the cultural issues out of the equation for a moment, this is still difficult because domain experts are often lacking application knowledge, test knowledge and technical background. To be successful in domain specific testing you must involve the domain experts and bridge the technological gap in order to capture the domain specific knowledge you need for the test cases.

Automated testing has a lot of potential and is very promising but currently is not delivering all possible results because it makes the maintenance and domain specific knowledge questions more difficult to answer.

If properly done an automated functional test suite can:

• Reduce cost and time of regression testing.
• Increase quality by freeing testers to focus on value added activities such as exploratory testing.
• Increase quality by faster feedback to developers.
• Capture and implement domain specific tests.
• Make requirements explicit by giving developers executable acceptance tests.

Domain Specific Languages (DSL) to the rescue

A Domain Specific Language (DSL), as opposed to a General Purpose Language (GPL), is a programming language tailored specifically to an application domain; rather than being for general purpose, it captures precisely the domain’s semantics. Popular examples for DSLs are HTML (used for document markup) and VHDL (used to describe electronic circuits). DSLs allow the concise description of an applications logic reducing the semantic distance between the problem and the program.

Testing tools and frameworks address a broad range of application domains (broad focus). This is natural because the tool vendor or developer wants to see his tool applied in a broad range of applications. For instance the Selenium test automation tool can be used to test a web shop application and the same tool can be used to test a social community web application. The tester or test automation developer is at the other extreme; he has a very narrow functional focus on the complexity of his application. He needs tooling that fits as well as possible in order to be able to succeed with the initiative and gain efficiency. He will probably start a new framework to exactly address his requirements. That is one of the reasons why there are so many frameworks available.

Besides writing a new framework there is another possibility to address the needs of the test automation developer. You can customize an existing framework so that it fits your needs. This can be done in two ways. The first and obvious way is to modify or extend the existing tool. Unfortunately this has its own problems, like the fact that the source code is not available for proprietary tools. Therefore updates, to new tool versions are problematic especially for customized proprietary solutions. The other alternative is to perform the customization within a layer on top of the existing tool. In this case you can totally shield away the complexity of the underlying testing tool. In the extreme, you can later if necessary exchange the underlying framework or tool you are using, without changing your test cases. In this variant the possibilities to extend functionality are limited. The positive aspects of the layering approach outweigh the negative therefore we will use this approach for this paper. The following diagram shows the different software layers within our testing approach.

DSLs are by definition special purpose languages:

• Concrete expressions of domain knowledge (captured in human readable form, not buried into system source code).
• Direct involvement of the domain experts. The DSL format can often be designed in a way that matches the format typically used by domain experts. This results in keeping experts in a very tight software life cycle loop.

As explained in the last section, domain experts are often lacking technical background, but it is necessary to involve them in order to get domain specific testing right. The DSL you and I are going to build has its focus on the application domain and hides the complexity of the testing framework away. Since no testing framework knowledge is necessary this brings test automation in reach of application domain experts. With a little help they will be able to review and comment on test cases and draft outlines of new test cases soon. The goal of a functional testing DSL is to be as human readable as possible and to abstract the complexity of the underlying testing tools. The following example test case for the openstreetmap should give you an intuitive idea of what we want to achieve.

browse http://www.openstreetmap.org

search Von-Gumppenberg-Strasse, Schmiechen

pan right

zoom in

Another test automation problem we want to address with the functional testing DSL is the expensive maintenance of the test suites. The DSL commands enforce reuse and reduce code duplication. The decomposition of the test cases into commands makes it easier to identify where to apply changes and simplify the changes as well. The DSL will enable maintenance of the test suite without the necessity of re-recording test cases after each change. This will significantly reduce maintenance costs and the cost of ownership of the automated test suite.

DSL Modelling

The term DSL modelling is rarely used in literature. Most of the scientific papers on DSL use the terms DSL design and development. Most authors focus on the topic of implementing a DSL. They discuss differences, pros and cons of internal and external DSLs and describe implementation related patterns in great detail. Unfortunately the practice oriented reader who wants to use a DSL to solve a concrete problem is left alone when it comes to DSL modelling. In order to address the tasks necessary to model a specific application domain I will use the term DSL Modelling to distinguish it from the tasks necessary to implement a DSL, which I will address as DSL Implementation (the next section).

DSL modelling consists of the analysis of the problem domain, refinement and design, and the methods used to drive this process. A prerequisite for the design of a DSL is a detailed analysis and structuring of the application domain. Graphical Feature Diagrams [3] are part of the FODA methodology and have been proposed to organize the dependencies between such features, and to document variabilities. In DSL modelling it is important to capture variabilities of your application domain because variability is the key factor in identifying complexity in your application domain. The following diagram shows such a feature diagram for the application domain of our example.

The feature diagram helps us to document and drive the DSL modelling. Potential sources of features are:

• Existing and potential stakeholders.
• Domain experts and domain literature.
• Existing systems and Documentation.
• Existing analysis and design models (Use-case models, Object-models, etc.).
• Models created during development (Entity-Relationship-Model, Class-Diagram, etc.).

For DSL modelling in the area of test automation we should use the following additional sources to look for features and vocabulary of domain specific knowledge:

• Product descriptions targeted to the customer.
• Any other sources that describe the behaviour of the system in the terms used by the domain expert (change requests, trouble tickets, bug reports).
• Descriptions of manual test cases of the area to be automated.

Strategies to identify features from the Czarnecki book [1]:

• Look for important domain terminology that implies variability, for example, checking account vs. savings account.
• Examine domain concepts for different sources of variability, for example, different stakeholders, client programs, settings, contexts, environments, aspects, and so on. In other words, investigate what different sets of requirements mean for different domain aspects.
• Use feature starter sets to start the analysis. A feature starter set consists of a set of aspects for modelling. Some combinations of aspects are more appropriate for a given domain than for another. For example, authentication, security, transactions, logging, etc.
• Look for features at any point in the development. As mentioned before, we have high-level system requirement features, architectural features, subsystem and component features, and implementation features. Thus, we have to maintain and update feature models during the entire development cycle. We may identify all kinds of features by investigating variability in use case, analysis, design and implementation models.
• Identify more features than you initially intend to implement. This is a useful strategy which allows us to create room for growth.

Steps in feature modelling from the Czarnecki book [1]:

1. Record similarities between instances, that is, common features, for example, all accounts have an account number.
2. Record differences between instances, that is, variability, for example, some accounts are checking accounts and others are savings accounts
3. Organize features in feature diagrams. Organize them into hierarchies and classify them as mandatory, alternative, optional.
4. Analyse feature combinations and interactions. We may find certain combinations to be invalid.
5. Record all the additional information regarding features such as short semantic descriptions, rationales for each feature, stakeholders and client programs interested in each feature, examples of systems with a given feature. Document constraints, default dependency rules, availability sites, binding sites, binding modes, open/closed attributes, and priorities.

Start with steps 1 and 2 in the form of a brainstorming session by writing down as many features as you can. Then try to cluster them and organize them into feature hierarchies while identifying the kinds of variability involved (i.e. alternative, optional etc.). Finally, refine the feature diagrams by checking different combinations of the variable features, adding new features, and writing down additional constraints. Maintain and update the initial feature models during the rest of the development cycle. You may also start new diagrams at any point during the development.

The feature diagram is very helpful during domain analysis. The DSL modelling for our openstreetmap example is not yet finished. In general it is unclear how to proceed once a feature diagram exists [3]. I think in software engineering there is no “formula” which can be generically applied on how to implement a piece of software from a design model. Like in software engineering there are engineering skills necessary to transfer a feature diagram into a DSL. One thing that definitely helps to bridge this gap is the decision on how to implement the DSL in the first place. How to implement a DSL in Python is covered in the next section.

DSL Implementation in Python

Two common variants exist for DSL implementation. A DSL can be implemented as internal DSL or external DSL. The terms internal DSL and external DSL have been coined by Martin Fowler [8]. An external DSL is a “independent” programming language. The external DSL is implemented like a general purpose programming language. Some kind of parser tool or framework is used to interpret or compile the external DSL based on a grammar to the target platform.

On the other hand, a DSL can be implemented as an internal DSL. This is also known as the piggyback pattern. The internal DSL is written like a normal program in the target programming language. The syntactic features of the target language are used to make the program more human readable. The programmer uses indentation and naming of the methods and variables to make the program read like sentences in a natural language. All the existing infrastructure of the target language like a parser, interpreter or compiler are used. It is also possible to extend or limit the features of the target language if necessary. The effort to create an internal DSL is usually smaller compared to creating an external DSL. Sometimes it also comes in handy to have the underlying power of the target language at hand. On the other hand the syntax of the internal DSL is limited by the syntax of the target language.

Internal DSLs are often implemented by use of Method Chaining. With method chaining it is easy to implement a DSL even in system programming languages like C++ and Java. The following code fragment shows the implementation of a test case as internal DSL in Python.

from osm_dsl import Browser

Browser("http://www.openstreetmap.org/") \
    .click_view() \
    .search("Von-Gumppenberg-Strasse, Schmiechen") \
    .pan_right() \
    .zoom_in()

Compared to natural English language this is strictly formatted and still has a lot of parentheses. Note that the verification steps for the automated test case will be built into the DSL commands and will not be visible in the test case description.

from selenium import selenium
import unittest, time, re

class Browser(unittest.TestCase):
    def __init__(self, website):
        self.setUp(website)

    def zoom_in(self, amount):
        for i in range(amount):
            self.selenium.click("OpenLayers.Control \
 .PanZoomBar_6_zoomin_innerImage")
            time.sleep(1)
        return self

    def pan_right(self, amount):
        for i in range(amount):
            self.selenium.click("OpenLayers.Control \
 .PanZoomBar_6_panright_innerImage")
            time.sleep(1)
        return self
    ...

The above code sample shows the implementation of the commands “zoom_in” and “pan_right” for the internal DSL. In order to follow the method chaining pattern all methods in the example return a reference to the class instance.

The implementation of the functional testing DSL as an internal DSL is a valid option but in my opinion the resulting language is still not at all like natural language, and therefore not suitable for interaction with the domain experts. Alternatively the implementation as external DSL means additional effort for modelling and maintaining the grammar of the external DSL. It is clear that, when modelling a DSL for functional test automation, the language would change a couple of times in the beginning in order to adapt to the application domain, and to make it as human readable as possible. Changes to the DSL grammar would mean additional overhead when implementing an external DSL. I could not find a way to improve the readability of the internal DSL implementation in Python to an acceptable level. While looking for a way to improve the readability of the internal DSL, I had an idea for implementing an external DSL in such a way that the benefits of the internal DSL can be captured while having all the syntactic possibilities of an external DSL at hand. In fact the implementations of commands for the external DSL and internal DSL are almost identical in this approach. This means no additional effort for the implementation of the commands. The best thing is that during the start phase when you work with three or four commands, these can be modified without the need to change the grammar of the external DSL.

When automating test cases I always have a look at the manual test cases first. Manual test cases should be described in a way that a relatively inexperienced tester, who has read the application user guide, can execute the test case and verify the results. I think DSLs for test automation usually start small with about three to five implemented commands. These commands are used to formulate a couple of test cases. I have to adjust the commands a couple of times during this process until they perfectly fit my needs. The language grows step by step along the project. During maintenance of the test suite it will often be necessary to adjust the implementation of the commands due to changes in the GUI of the system under test.

The following sample shows a test case formulated in the external DSL. The test case is formulated in natural language which allows us to involve domain experts in the test automation project. The test case describes two different scenarios for location search. The first scenario is the search in the View tab which does not require the user to log in. The second scenario tests the location search in the Edit tab which requires a login.

Story Search Location uses osm_dsl

Scenario Search Location in View tab:
    browse http://www.openstreetmap.org
    search Von-Gumppenberg-Strasse, Schmiechen
    pan right
    zoom in

Scenario Search Location in Edit tab:
    browse http://www.openstreetmap.org
    click Edit
    login user mark identified by test1234
    search Von-Gumppenberg-Strasse, Schmiechen
    pan right
    zoom in
    logout

Next we will take a look at the Selenium tools. For the recording of the raw tests I use the Selenium IDE. The Selenium IDE is a Firefox plugin used to record clicks on a webpage, to add wait conditions, and verification steps. We export a captured test case as a Python script. We can use this script later as a basis for the implementation of the DSL commands. We use another tool, the Selenium Server, for execution of the test cases. The Selenium Server provides the test profile for the web browser, starts and ends the web browser, and handles all the communication of our test suite with the web browser. I cover the usage of Selenium in more detail in an article “Functional testing of web applications with Selenium” on my website [5].

The following table shows a Selenium test case with all the technical details like wait conditions and verification steps.

In Selenium IDE it is possible to export the recorded test case into a Python script which looks like the following:

from selenium import selenium
import unittest, time, re

class NewOsmTest(unittest.TestCase):
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium("localhost", 4444, "*chrome",
            "http://www.openstreetmap.org/")
        self.selenium.start()

    def test_new_osm_test_case(self):
        sel = self.selenium
        sel.open("/")
        sel.type("query", "Von-Gumppenberg-Strasse, Schmiechen")
        sel.click("commit")
        for i in range(60):
            try:
                if sel.is_element_present("permalinkanchor"): break
            except: pass
            time.sleep(1)
        else: self.fail("time out")
        sel.click("OpenLayers.Control.PanZoomBar_6_panright_innerImage")
        sel.click("OpenLayers.Control.PanZoomBar_6_zoomin_innerImage")
        try: self.failUnless(sel.is_element_present("loginanchor"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        sel.click("loginanchor")
        for i in range(60):
            try:
                if sel.is_element_present("//form[@action='/login']"):
                    break
            except: pass
            time.sleep(1)
        else: self.fail("time out")
        try: self.failUnless(sel.is_element_present("loginanchor"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        try: self.failUnless(sel.is_element_present("user_email"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        try: self.failUnless(sel.is_element_present("user_password"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        sel.type("user_email", "markfink")
        sel.type("user_password", "test1234")
        try: self.failUnless(
            sel.is_element_present("//form[@action='/login']"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        sel.click("commit")
        for i in range(60):
            try:
                if sel.is_element_present("link=markfink"): break
            except: pass
            time.sleep(1)
        else: self.fail("time out")
        try: self.failUnless(sel.is_element_present("link=markfink"))
        except AssertionError, e: self.verificationErrors.append(str(e))
        sel.click("editanchor")
        sel.click("logoutanchor")

    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

It is relatively easy to identify which parts of the script implement certain aspects (search, zoom, login, …) of the test case. It also takes a little while to get used to reading the captured test cases. After a while you will slice the scripts into reusable commands in no time. Sometimes test cases that worked before, or which worked for another browser, will fail. This happens due to the asynchronous behavior of AJAX implementations. If this happens you must simply add another wait condition. It is better to add a waitForElement condition than a sleep() for a fixed time interval. Due to the fact that some browsers execute Javascript faster than others you must use waitFor conditions for synchronization to occur properly. After you slice the script into parts and form them into DSL commands, take some time to add additional verification steps. The DSL commands will be reused and you do not need to rework the verification steps of a command later if was done correctly the first time.

After some tweaking of the implementation of your external DSL commands would look something like that:

from selenium import selenium

@dsl('search (\w*)')
def search(query):
    sel.open("/")
    sel.type("query", query)
    sel.click("commit")
    for i in range(60):
        try:
            if sel.is_element_present("permalinkanchor"): break
        except: pass
        time.sleep(1)
    else: fail("time out")

@dsl('zoom in (\w*)')
def zoom_in(amount=1):
    for i in range(amount):
        sel.click("OpenLayers.Control.PanZoomBar_6_zoomin_innerImage")

@dsl('pan right (\w*)')
def pan_right(amount=1):
    for i in range(amount):
        sel.click("OpenLayers.Control.PanZoomBar_6_panright_innerImage")

@dsl('login user (\w+) identified by (\w+)')
def login(user_email, user_password):
    try: failUnless(sel.is_element_present("loginanchor"))
    except AssertionError, e: self.verificationErrors.append(str(e))
    sel.click("loginanchor")
    for i in range(60):
        try:
            if sel.is_element_present("//form[@action='/login']"): break
        except: pass
        time.sleep(1)
    else: fail("time out")
    try: failUnless(sel.is_element_present("loginanchor"))
    except AssertionError, e: verificationErrors.append(str(e))
    try: failUnless(sel.is_element_present("user_email"))
    except AssertionError, e: verificationErrors.append(str(e))
    try: failUnless(sel.is_element_present("user_password"))
    except AssertionError, e: verificationErrors.append(str(e))
    sel.type("user_email", user_email)
    sel.type("user_password", user_password)
    try: failUnless(sel.is_element_present("//form[@action='/login']"))
    except AssertionError, e: verificationErrors.append(str(e))
    sel.click("commit")
    for i in range(60):
        try:
            if sel.is_element_present("link="+user_email): break
        except: pass
        time.sleep(1)
    else: fail("time out")
    try: failUnless(sel.is_element_present("link="+user_email))
    except AssertionError, e: verificationErrors.append(str(e))
...

Note that the functions in the above source code block which form the DSL commands are modified by the @dsl decorator. The @dsl decorator links a regular expression statement to the function signature. This regular expression is used during execution of a test case scenario to identify which DSL command will be executed. The grammar an other infrastructure of the functional testing DSL is implemented as a Python nose unit test plugin (open source) [6].

I think this is really worth the effort. With a little bit of additional coding it is possible to break up the capture and replay tests into reusable DSL commands that can be easily reused to extend the test suite with more new test cases. The amount of work to fix timing issues and the time for writing additional verification steps pays back when you formulate new test cases based on these commands.

Now we have finished the implementation of our openstreetmap test case as an external DSL. The test case in this form is much more readable than the original Selenium test case (see the test case in the tabular form above). I would go so far as to say that domain experts, after a minimal amount of training, will be able to review test cases in this format. After a little practice, domain experts will probably be able to sketch new test cases that can than be automated incredibly quickly. Of course this depends heavily on the working environment. I would guess in a start-up-company environment domain experts would be motivated to support writing test cases in this way because of the high quality of the resulting software product.

Related Work

Test automation could play an important role in software engineering. That is the reason why so much literature exists in this area which covers different approaches to the topic. In this section I want to compare the approach of the functional testing DSL with other approaches. I also give credit to other work that inspired me. I also want to give the reader reference points for further research.

Fit and Fitnesse [7] are very interesting testing tools that target similar goals like the functional testing DSL. The test cases are documented in tabular format. The tabular format helps to collect the domain knowledge of the test cases. The complexity of the underlying test implementations are hidden in so called fixtures. It is also possible to maintain Selenium tests with these tools. I personally think that Selenium has its strength in functional testing. For GUI or in-browser tests, I prefer the approach discussed in this article because I think that it is more flexible.

In the field I have seen implementations of functional test cases in XML. To implement test cases in XML is orthogonal to implementing them in an internal or external DSL. In fact, the implementation in XML is a DSL as well. I think that XML markup gets in the way when you want to collect and document domain specific knowledge from domain experts. During the XML hype many DSLs have been implemented in XML. A popular example where they used XML but should have used a DSL is ANT. ANT is a popular Java build framework. To build a complex application you often have the need for the power of a programming language. That is why so many additional tasks exist and are in use. The result is that ANT build files are overly complex and difficult to maintain.

Behaviour Driven Development (or BDD) is an agile software development technique that encourages collaboration between developers, QA and domain experts in a software project. It was originally conceived by Dan North [9]. BDD evolved from Test Driven Development. The focus of BDD is the language and interactions used in the process of software development. Developers who follow BDD use their native language in combination with the ubiquitous language of Domain Driven Design to describe the purpose and benefit of their code [4]. I was inspired by Dan North JBehave implementation when I designed the test scenarios for the functional testing DSL.

Another approach is Model Driven Testing. For Model Driven Testing you use the same Analysis and Design models that you use for the implementation and generate tests from them. As a prerequisite these models have to be standardized and complete, which is a problem for complex systems. While this looks promising, only time will tell if this approach is successful.

U2TP is the UML Testing Profile Specification of the Object Management Group [2]. This approach uses UML models to standardize the description of test concepts like black-box testing, local behaviour, distributed behaviour, test data. Again this looks promising but it is a brand new methodology. This means it will probably change radically in the near future and it will take some time to demonstrate whether this approach is successful in the industry.

Over that last 10 months I used much of my spare times to do research in the area of Domain Analysis and Domain Engineering. Domain Specific Languages, or Little Languages, have been a popular topic for Computer Scientists and Software Engineers since the 1970s. I was drawn by the idea that there must be a wealth of different methods and tools available to support DSL modelling. So far I have identified the Feature Diagrams I used in the DSL Modelling section above. Feature Diagrams are the main part of the FODA (Feature-Oriented Domain Analysis) method. Domain Engineering aims to collect, organize and store information about past software developments, and to exploit it for new developments [1]. Czarnecki’s book on Generative Programming is a good starting point to familiarize yourself with the topic of Domain Analysis and Domain Engineering.

Conclusions and further work

The only drawback of this approach is that special knowledge is necessary to model the DSL. It requires you to do some analysis of your application domain. A Feature Diagram can help you with this. Read the documentation on incidents, tickets, bug reports, or whatever they are named in your organization in order to learn the language of your business. Talk to business analysts and read the description of manual test cases – if they exist – in the area of your application domain. Start small and iterate. This means literally select just enough DSL commands to implement one test case. Implement another test case with the same commands and add additional commands as you go along. Do not hesitate to change the language to make it as human readable as possible. Collect early feedback from your domain experts. Avoid the capture and replay of test cases as much as possible. They are not easy to maintain and collecting too many of them will render your test suite useless. Instead break up the captured tests into reusable DSL commands.

If you know of a open source Python application that needs functional testing please let me know. I would be interested in executing functional test automation in a community project.

[1] Krzysztof Czarnecki, Ulrich W. Eisenecker, 2000. Generative Programming – Methods, Tools, and Applications. Addison-Wesley.

[2] Z. R. Dai, 2006. An Approach to Model-Driven Testing – Functional and Real-Time Testing with UML 2.0, U2TP and TTCN-3. Fraunhofer Publications

[3] Arie van Deursen, Paul Klint, 2001. Domain-Specific Language Design Requires Feature Descriptions. Journal of Computing and Information Technology.

[4] Eric Evans, 2004. Domain-Driven Design – Tackling Complexity in the Heart of Software. Addison-Wesley.

[5] Mark Fink, 2008. Functional testing of web applications with Selenium. http://www.testing-software.org/cast/Webapp-Testing/Selenium.html.

[6] Mark Fink, 2009. tdsl – A Functional Testing DSL. http://bitbucket.org/markfink/tdsl/.

[7] Fitnesse acceptance testing framework. http://fitnesse.org/.

[8] Martin Fowler, 2008. Method Chaining. http://www.martinfowler.com/dslwip/.

[9] Dan North, 2006. Introducing BDD, http://dannorth.net/introducing-bdd/.

Selenium Core

Selenium Core is at heart a set of JavaScript code that executes tests which are contained in a HTMl table. Those tests life within the browser. The browser sees a test page which contains and drives your application. This is a simple concept but it is very effective and portable (to all the platforms mentioned above) at the same time.

Selenium IDE

Writing web application tests by hand without tool support is exhausting and almost impossible at lager scale. The tooling support you need is Selenium IDE. Selenium IDE is a Firefox extension for recording tests. It records all user interaction while browsing your application and you just need to add your verification steps while doing so.

The complete list of available selenium commands is available at Selenium Core TestRunner reference page:

http://www.openqa.org/selenium-core/testrunner.html

The following screenshot shows the Selenium IDE in action. A test case has been recorded. To resolve timing issues with execution of the JavaScript I had to insert a waitForElementPresent condition.

Selenium IDE saves the testcase into a html table:

It is really just a plain HTML file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head profile="http://selenium-ide.openqa.org/profiles/test-case">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="selenium.base" href="" />
<title>openstreetmap</title>
</head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
<thead>
<tr><td rowspan="1" colspan="3">openstreetmap</td></tr>
</thead><tbody>
<tr>
  <td>open</td>
  <td>/</td>
  <td></td>
</tr>
<tr>
  <td>click</td>
  <td>commit</td>
  <td></td>
</tr>
<tr>
  <td>type</td>
  <td>query</td>
  <td>Schmiechen</td>
</tr>
<tr>
  <td>click</td>
  <td>commit</td>
  <td></td>
</tr>
<tr>
  <td>waitForElementPresent</td>
  <td>link=Schmiechen, Bayern</td>
  <td></td>
</tr>
<tr>
  <td>click</td>
  <td>link=Schmiechen, Bayern</td>
  <td></td>
</tr>
<tr>
  <td>click</td>
  <td>link=Close</td>
  <td></td>
</tr>
</tbody></table>
</body>
</html>

In Selenium IDE select “File|Export Test Case as…|Python – Selenium RC” to export the test case to Python. You get the following Python script:

from selenium import selenium
import unittest, time, re

class NewTest(unittest.TestCase):
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium("localhost", 4444, "*firefox", "http://www.openstreetmap.org/")
        self.selenium.start()

    def test_new(self):
        sel = self.selenium
        sel.open("/")
        sel.click("query")
        sel.type("query", "Schmiechen")
        sel.click("commit")
        for i in range(60):
            try:
                if sel.is_element_present("link=Schmiechen, Bayern"): break
            except: pass
            time.sleep(1)
        else: self.fail("time out")
        sel.click("link=Schmiechen, Bayern")
        sel.click("link=Close")

    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

Selenium Remote Control (RC)

The Selenium RC provides a proxy server in order to treat the web application in a way that the browser thinks that the Selenium Core JavaScript files have the same origin as the web application. As a consequence the “same origin policy” of the browser is not violated.

Run the server part of Selenium RC with “java -jar selenium-server.jar”. This is often also called Selenium Server.

Now we can easily execute the test case:

It takes additional overhead (~25 sec) to prepare the profile and launch the browser.

Selenium Grid

Selenium Grid is a platform to support parallel execution of test cases. The grid consists of multiple machines each running its own Selenium Server. Through parallel execution the completion time of automated web application testing can be drastically reduced.

You will find more information on Selenium Grid and “Web Testing That Doesn’t Take Hours!”:

http://selenium-grid.openqa.org/

Installation

Selenium IDE is a Firefox plugin. Open Firefox and select “Tools|add-ons|Get Add-ons”. Search for Selenium IDE and select “Add to Firefox”.

Everything else you usually need is in Selenium RC. Get the latest snapshot from:

http://selenium-rc.openqa.org/download.html

At time of writing the latest snapshot resolved some issues with Firefox 3 on Ubuntu and OS X. Unpack the “selenium-python-client-driver” somewhere on your PYTHONPATH. You also need the selenium-server.jar to run the server part of Selenium RC.

Measuring Lines Of Code (LOC)

sudo apt-get install sloccount
cd sloccount openerp-server-5.0.0_rc3

SLOC        Directory       SLOC-by-Language (Sorted)
58241   addons-extra    python=50644,php=7434,sh=163
53168   addons          python=53126,php=42
22288   server          python=22287,sh=1
15599   client          python=15599
12334   web             python=12205,sh=129
72      top_dir         python=72

Totals grouped by language (dominant language first):
python:      153933 (95.20%)
php:           7476 (4.62%)
sh:             293 (0.18%)

Measuring complexity (McCabe)

Another very useful code metric in the situation described above is the McCabe complexity metric. This tool helps you to identify the most complex code. The complex areas need the most attention when it comes to quality assurance measures (documentation, testing, etc.). These areas usually contain the most defects, too.

#manually extract tarball
PyMetrics.py program.py

Understanding the code structure using Callgraphs

Almost every time I am approaching a code base unknown to me I am looking for a certain functionality which I am particularly interested in. As soon as I understand the functionality I go to the next one and so on until I understand everything I need to. To identify relevant parts in the code a call graph proved to be very handy. The graph “lists” all the relevant modules and functions and shows in which order they are called. I always use the call graph as a “map” that helps me to navigate the code base unknown to me.

apt-get install pycallgraph

pycallgraph run-recording-proxy -i scotch.* -e *.*

sudo apt-get install dia

Apache2 setup and installation

First of all you need Apache2 installed (for Ubuntu):

sudo apt-get install apache2

If you want to make it really pretty (what I usually want to do) put the following FQDN into your /etc/hosts configuration (you probably have to restart your browser to make this work):

127.0.2.1   my.sample.com

Any requests for the my.sample.com domain, which does not exit on the internet, will be sent to the 127.0.2.1 IP address (part of the loopback).

Start a new file in the /etc/apache/sites-available directory with the name of the site, for example: my.sample.com. Use this configuration as a starting point and change the IP Address, ServerName, ServerAdmin, DocumentRoot and Directory as appropriate.

<VirtualHost 127.0.2.1:443>
    ServerName my.sample.com
    ServerAdmin admin@sample.com

    DocumentRoot /home/mark/devel/tam/build
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /home/mark/devel/tam/build>
        Options Indexes FollowSymLinks Multiviews
        AllowOverride all
        Order allow,deny
        allow from all
    </Directory>

    ErrorLog /var/log/apache2/error.log

    LogLevel warn

    CustomLog /var/log/apache2/access.log combined
</VirtualHost>

Execute the following command to enable the new site:

sudo a2ensite my.sample.com

Restart Apache by executing the following command:

sudo /etc/init.d/apache2 restart

Test the new website:

Put a sample file (index.html or something like that) into the document root and check that it is visible using your browser.

Adding SSL to the site

In order to create certificates you need openssl and some configuration which is described in this section. If you want you can just install the provided sample certificates instead of creating your own (the certificates are contained in testing-software/ssh-sniffer). If you want to skip the certificate creation just jump to install certificates. Make sure you use this only as a local test scenario!

Install or upgrade openssl (in Ubuntu):

sudo apt-get install openssl

In order to use the steps provided below you have to configure openssl (/etc/ssl/openssl.cfg).

#
# OpenSSL example configuration file.
# This is mostly being used for generation of certificate requests.
#

# This definition stops the following lines choking if HOME isn't
# defined.
HOME            = .
RANDFILE        = $ENV::HOME/.rnd

# Extra OBJECT IDENTIFIER info:
#oid_file       = $ENV::HOME/.oid
oid_section     = new_oids

# To use this configuration file with the "-extfile" option of the
# "openssl x509" utility, name here the section containing the
# X.509v3 extensions to use:
# extensions    =
# (Alternatively, use a configuration file that has only
# X.509v3 extensions in its main [= default] section.)

[ new_oids ]

# We can add new OIDs in here for use by 'ca' and 'req'.
# Add a simple OID like this:
# testoid1=1.2.3.4
# Or use config file substitution like this:
# testoid2=${testoid1}.5.6

####################################################################
[ ca ]
default_ca      = sampleCA           # The default ca section

####################################################################
[ sampleCA ]

dir             = ./CA               # Where everything is kept
certs           = $dir/certs         # Where the issued certs are kept
crl_dir         = $dir/crl           # Where the issued crl are kept
database        = $dir/index.text     # database index file.
#unique_subject = no                 # Set to 'no' to allow creation of
                                     # several certificates with same subject.
new_certs_dir   = $dir               # default place for new certs.

certificate     = $dir/sampleCA.crt  # The CA certificate
serial          = $dir/serial        # The current serial number
crlnumber       = $dir/crlnumber     # the current crl number
                                     # must be commented out to leave a V1 CRL
crl             = $dir/sampleCA.crl  # The current CRL
private_key     = $dir/sampleCA.key  # The private key
RANDFILE        = $dir/private/.rand # private random number file

x509_extensions = usr_cert           # The extensions to add to the cert
...

mkdir ./sampleCA

mkdir ./sampleCA/CA

mkdir ./sampleCA/CA/newcerts

mkdir ./sampleCA/CA/certs

mkdir ./sampleCA/CA/crl

mkdir ./sampleCA/CA/private

touch ./sampleCA/CA/index.text

touch ./sampleCA/CA/serial

echo "01" > ./sampleCA/CA/serial

cd  ./sampleCA

openssl genrsa -out ./CA/sampleCA.key

openssl req -new -key ./CA/sampleCA.key -out ./CA/sampleCA.csr

openssl x509 -req -days 365 -in ./CA/sampleCA.csr -out ./CA/sampleCA.crt -signkey ./CA/sampleCA.key

openssl x509 -in ./CA/sampleCA.crt -text

openssl genrsa -des3 -out sampleWebServer.key

(pass phrase: test123)

openssl rsa -in sampleWebServer.key -out sampleWebServer.key

openssl req -new -key sampleWebServer.key -out sampleWebServer.csr

openssl ca -in sampleWebServer.csr -cert ./CA/sampleCA.crt -keyfile ./CA/sampleCA.key -out sampleWebServer.crt

openssl x509 -in sampleWebServer.crt -text

sudo cp sampleWebServer.key /etc/ssl/private/

sudo chmod 400 /etc/ssl/private/sampleWebServer.key

sudo cp sampleWebServer.crt /etc/ssl/certs/

sudo cp ./CA/sampleCA.crt /etc/ssl/certs/

sudo a2enmod ssl

<IfModule mod_ssl.c>
<VirtualHost 127.0.2.1:443>
    ServerName my.sample.com
    ServerAdmin admin@sample.com

    DocumentRoot /home/mark/devel/tam/build
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /home/mark/devel/tam/build>
        #SSLRequireSSL
        #SSLRequire %{SSL_CLIENT_S_DN_O}  eq "Internet Widgits Pty Ltd"
        Options Indexes FollowSymLinks Multiviews
        AllowOverride all
        Order allow,deny
        allow from all
   </Directory>

    ErrorLog /var/log/apache2/error.log

    LogLevel warn

    CustomLog /var/log/apache2/access.log combined

    # SSL
    SSLEngine on

    # Server Certificate
    SSLCertificateFile /etc/ssl/certs/sampleWebServer.crt
    SSLCertificateKeyFile /etc/ssl/private/sampleWebServer.key

    #SSLCACertificateFile /etc/ssl/certs/sampleCA.crt
    #SSLVerifyClient require
    #SSLVerifyDepth 2

    BrowserMatch ".*MSIE.*"\
        nokeepalive ssl-unclean-shutdown \
        downgrade-1.0 force-response-1.0
</VirtualHost>
</IfModule>

sudo /etc/init.d/apache2 restart

(you probably have to provide the pass phrase test123)

Create client certificate and enforce client authentication by the server

In order to use client authentication you need a client certificate to be installed with your browser. Please note that your CA certificate which is needed to verify the client certificate has already been installed on the server.

cd ./sampleCA

openssl genrsa -des3 -out client.key 1024

(pass phrase: test123)

openssl req -new -key client.key -out client.crs

(This time we enter for the common name (CN) our full name, for example "client")

openssl ca -in client.crs -cert ./CA/sampleCA.crt -keyfile ./CA/sampleCA.key -out client.crt

openssl pkcs12 -export -clcerts -in client.crt -inkey client.key -out client.p12

openssl pkcs12 -in client.p12 -out client.pem -nodes

SSLRequireSSL

SSLRequire %{SSL_CLIENT_S_DN_O}  eq "Internet Widgits Pty Ltd"

and:

SSLCACertificateFile /etc/ssl/certs/sampleCA.crt

SSLVerifyClient require

SSLVerifyDepth 2

sudo /etc/init.d/apache2 restart

(you probably have to provide the pass phrase test123)

Edit > Preferences > Advanced > View Certificates > Your Certificates -> Import -> Choose file

(I did not set any password)

Introduction

I order to optimize the heap configuration for your situation you have to try different settings. Usually in order to do so you need some kind of test environment and a load generator which is able to produce realistic load patterns in a repeatable way. With this setup you can try different heap configurations and monitor the heap utilization and garbage collector. Best practice is to make an educated guess on the configuration and start optimizing from there. In this article I will show you how to do this.

Java Memory Management

In order to being able to optimize the heap utilization of your application you need to understand how the Java heap and garbage collector work and how they are configured.

The heap is split up in Young-Generation (Eden-Space, and two Survivor-Spaces of identical size usually called From and To), Old Generation (Tenured) and Permanent Space.

The idea behind this organization of the heap is that most of the objects dye in New-Space because they have a very short life cycle:

• 80-98% of all new instantiated objects dye within the next million of instructions.
• 80-98% of all new instantiated objects dye before an additional Megabyte of storage is allocated.

To clean up the whole heap within one garbage collector run would take much longer than reducing the cleaning up to just the Young-Generation (Minor-GC). Only the Full-Gc works on the complete heap. Usually a Minor-Gc is much faster than a Full-Gc. The execution time of a Gc run is very important to you because most garbage collection strategies stop the world. This means that even if you have multiple processor cores, all execution is halted during GC run. The stop-the-world strategies have many negative effects on applications that optimize response times. Today most JVMs also provide an implementation of a mark-and-sweep strategy which eliminates the negative effects because they do not need long pauses.

Heap configuration parameters

The next step before you can start optimizing the Java heap and garbage collector configuration is to develop a thorough understanding of the configuration parameters.

-Xms

defines the minimal/ initial size of the heap

-Xmx

defines the maximum size of the heap

For server applications you should set both parameters identical because the resizing of the heap also initiates a Full-Gc. With an application under load this could have a severe impact on application performance from the very start. On most operating systems the regained memory is not returned to the operating system.

-XX:NewSize

initial size of the New Space. This includes Eden plus two Survivour Spaces (From and To).

-XX:MaxNewSize

usually we set -XX:MaxNewSize the same as -XX:NewSize because resizing would cause a full collection

-XX:PermSize

initial size of the Perm Space

-XX:MaxPermSize

usually we set -XX:MaxPermSize the same as -XX:PermSize because resizing would cause a full collection

-XX:SurvivorRatio

ratio Eden to Survivor-Space; I usually start with a survivor ratio of 8; in this case each of the survivor spaces (From and To) uses 1/10 of the size of the New-Space

-XX:NewRatio

ratio between old and new generation

-XX:MaxPermSize

size of the Permanent Generation

-XX:TargetSurvivorRatio=90

the survivor space can be filled by 90% until the objects are moved to old generation

-XX:MaxTenuringThreshold

number of from-to runs until object is moved to old generation

GC Algorithms

Over time Sun developed different strategies for the garbage collector. Different applications have different requirements regarding response times. In case you optimize for response times or you are having realtime requirements you are in a totally different situation than somebody who is optimizing for throughput. In this section I describe the different strategies for you so that you will be able to select the one that fits best to your needs. Different strategies and configuration options are available in the different JVM versions. Check the version of your JVM before you start working.

Analysis of the memory utilization

After you decided for the initial configuration for heap and garbage collector you will use a load tool to put your application under realistic load (depending on your application this takes usually 8-10 hours). In order to decide for improvements you need to analyse the memory utilization under load. Many monitoring tools also provide monitoring for the Java memory utilization. Call me old-fashioned but I do not trust them. Many of them have weird algorithms to calculate the values you are dependant on for your optimization. I have seen their inaccuracy so many times. Therefore I recommend to log the garbage collector information to a logfile (usually you will do this anyway). After your test run is complete you extract the data for your analysis from the logfile.

important: look at the frequency and time consumed by gc

Tools

This section gives a brief overview of some tools that assist you in analysing the heap utilization of your JVM. Depending on your environment a particular tool could be more suitable than others. Currently there are many hundreds of tools available and there is no way that this section could provide a comprehensive overview of tool situation. Nevertheless it is intended to give a brief overview of different approaches and to reference supportive tools.

export JVMSTAT_JAVA_HOME='/usr/lib/jvm/java-6-sun'
PATH=$PATH:/usr/local/bin/jvmstat/bin

#jps
9599 Jps
9590 Life

#visualgc 9590

-verbose:gc -XX:+PrintGCTimeStamps
-XX:-TraceClassUnloading -XX:+PrintHeapAtGC

# jps
16945 Life
16954 Jps
# jstat -gcnew 16945 250 5
    S0C     S1C    S0U     S1U TT MTT     DSS       EC       EU    YGC     YGCT
48000.0 48000.0    0.0 48000.0  1  15 24000.0 672000.0 109709.5      1    0.623
48000.0 48000.0    0.0 48000.0  1  15 24000.0 672000.0 123418.2      1    0.623
48000.0 48000.0    0.0 48000.0  1  15 24000.0 672000.0 137127.0      1    0.623
48000.0 48000.0    0.0 48000.0  1  15 24000.0 672000.0 150835.8      1    0.623
48000.0 48000.0    0.0 48000.0  1  15 24000.0 672000.0 164544.6      1    0.623

#jps
16633 Jps
16624 Life

#jconsole 16624

Howto tune the heap utilization of the JVM

This section describes in detail how to analyse and tune a JVM. As I explained above a optimized configuration can neither be calculated nor guessed because the heap utilization depends on the application and its usage. We usually start with a suboptimal configuration provided by an experienced developer. On a test environment we put realistic load on the application and measure/ analyse the behaviour. From the analysis results we derive a optimized JVM configuration. This configuration will be tested and analysed again and so forth until we get satisfying outcome with our optimized JVM configuration.

In order to demonstrate the optimization of the heap utilization I wrote a small Java application that simulates the creation of two different lifeforms (grass and sheep) at given ratios. This is not a real life simulation. In fact you would be surprised how simple it is. The applications only purpose is to instantiate a lot of small objects with a little lifetime and some bigger objects that have a much higher lifespan. You can find the sample application here: http://bitbucket.org/markfink/testing-software/src/tip/gcview/tuning_sample/.

SCENARIO=grass-sheep
DAYS=5000            # number of days to simulate
SLEEPTIME=10         # slow down the simulation in ms

GRASS_GROWTHRATE=250 # amount of new entities every day
GRASS_MAXAGE=25      # entities die after they reach maxage
GRASS_SIZE=1000      # memory footprint for each entity

SHEEP_GROWTHRATE=5   # amount of new entities every day
SHEEP_MAXAGE=900     # entities die after they reach maxage
SHEEP_SIZE=10000     # memory footprint for each entity

I visualized the behaviour of the simulation in the following diagram:

First approach towards running our sample application is without any configuration of the JVM. The execution failed with an OutOfMemoryError.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at Lifeform.<init>(Life.java:86)
        at Life.<init>(Life.java:61)
        at Life.main(Life.java:13)

If we analyse a little further we notice 49 full garbage collections within a few seconds. Note that there are no minor collections.

We realize that the heap is way to small and decide to configure its size. The application now finishes without an error. Analysing the garbage collections we see that only one Full-Gc is executed. In the GC-diagram we are now able to identify single minor garbage collector runs and see how the Survivor spaces (From and To) are utilized.

It will be tough to further optimize this heap configuration. In real world I would probably stick with this configuration but for the sake of demonstration I will try to further optimize the heap configuration.

I will start out with my standard heap configuration which turns out to reduce amount of minor-Gc runs by 80 percent. The time spent in GC is now down to 7 sec.

-Xms1500m -Xmx1500m
-XX:NewSize=256m -XX:MaxNewSize=256m
-XX:PermSize=128m -XX:MaxPermSize=128m
-XX:SurvivorRatio=8

Now I start optimizing the size of the survivor spaces. Please note that a bigger SurvivorRatio in the JVM configuration means smaller survivor spaces! I now set the SurvivorRatio to 12. All in all this results into a very tiny improvement. The only significant effect is that the Full-Gcs are now completely gone. The time spent in Gc is now down to 4.77 sec.

All of the above gc diagrams are created from the logfile with a Python script: http://bitbucket.org/markfink/testing-software/src/tip/gcview/

Additional Information

[1] Whitepaper on memory management by Sun, http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepaper.pdf

[2] General information in Java performance, http://java.sun.com/docs/performance

[3] Garbage Collection tuning guide, http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html

[4] Article on GC methods of the Java HotSpot VM, http://www.devx.com/Java/Article/21977/

[5] Joseph D. Mocker’s collection of JVM Options, http://blogs.sun.com/roller/resources/watt/jvm-options-list.html

[6] Concurrent Mark and Sweep, http://research.sun.com/techrep/2000/smli_tr-2000-88.pdf

[7] G1, http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf and http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf

Why does this matter

I think it was around 1998 when we first learned that the clock rate could not be increased and we where about to seeing the end of Moore’s law that the number of transistors that compounds a CPU will double every two years. The CPU clockrate at this time was slightly below 1 GHz. A few month later clever scientists discovered a way around that barrier and Moore’s law applied again. The computer industry was saved. I believe I was not the only one who did not take it seriously when in 2005 they announced that they really can not increase the clockrate any further. I was like “yeah, you will discover it within no time”. Now it is crystal clear that the only way to increase the computing power of a CPU is to give it more cores. Energy consumption/ cooling constrains the clockrate and we know today that energy efficient computing requires clock rates to drop again to below 1 GHz.

Today having more than one core on the CPU is yesterdays news because everybody already has at least a dual core CPU in his or her PC. The hardware is available but parallel computing this is still the hardest challenge we are today facing in the computer industry. At the moment our machines have multiple cores but we are not able to put them to effective use. Not being able to increase computing power in the PC is a serious thread for the industry because nobody will replace one unless it breaks. Besides the economic aspects there are a lot of topics desperately waiting for more computing power like speech, image recognition, and advanced computer interfaces.

What to do

What I think makes parallel computing really a hard problem is that there is currently and most likely there will be no general solution to the problem. Many computer scientists have spent their lives trying to find one. Among the most prominent is David Patterson who is Professor for Computer Architecture at Berkeley who worked on three separate projects on parallel computing. Many insights have been discovered but still the problem is far from being solved. On the other hand specific problems in this area have been addressed by today’s companies like Amazon, Facebook and Google. I think we can learn from them and look at the programming models used in their approaches.

I am convinced that there are at least to different kind of websites. There are sites that all people are using all the time like Amazon, Facebook and Google and on the other hand there are sites which are only use by some people some of the time. I am particularly interested in the later one so the rest of this piece is about the situation where you have less than 20.000 active connections at a time.

Programming with Thread vs. Eventoriented Programming

One of the strange things in Software Engineering is that people have strong opinions and you can get easily into a situation one starts a fight over like for example Emacs vs. VI, Windows vs. Mac, Ruby vs. Python. Same here in programming models. I believe that most software engineers think that threads are the way to go when using multicore CPUs. One the other hand there are strong indications that this is not always true. John Ousterhout who has been so far to one of the most influential Computer Scientist for my own career gave in 1996 a talk on “Why Threads are a bad idea” at the USENIX conference. When I found his slides I smiled since at that very second it occurred to me that I am on the right track. The slides (http://home.pacbell.net/ouster/threads.pdf) are freely available and you should definitely have a look. For this article I will only use his main argument that thread programming is extremely difficult to get right and a burden even for the best programmers.

Threads split the resources available to the operating system into pieces. Even if the resource consumption for a single thread is very small in total it sums up. For example a webserver which has 1GB available and splits it into 0.5 MB pieces for each thread is limited to 2.000 simultaneous requests. A recent example makes the issue very clear. The Apache webserver uses threads. Nginx is an Apache webserver clone where the threading mechanism was replaced by a event loop. It was successful and today Nginx by far outperforms Apache in terms of number of connections and memory utilization.

Eventoriented programming is a natural way and many programmers today prove to be comfortable using it.

Event oriented programming is often said to produce spaghetti code which is difficult to debug. I believe these are skill related issues and the event oriented programming model needs to be mastered like every other programming model. For example signal processing diagrams can help documenting programs following the event oriented model. Special debugging tools assign every callback a complete stack trace so that one can trace back the source of a given problem if he must.

Non-blocking I/O

Non-blocking, or asynchronous I/O allows the computation to continue while the input/ output operation is running. So it is for example possible to have multiple database queries running at a time. Input and output operations on the network or file can be extremely slow compared to processing of data in memory. Most of I/O operations involve disc access in some form and spinning of discs is orders of magnitude slower than read/ write operations on a electronic circuit.

There are different forms of non-blocking I/O like polling, signals and callbacks.

Greenlet

Greenlets or eventlets are an abstraction on top of an event loop. When using greenlets one is programming in a blocking way and the framework is using the event loop and a scheduler under the hood. The programming model is more like threads but avoiding the resource consumption.

Callbacks

A callback is a function, that is passed as an argument to I/O operation. Once the I/O operation is completed the part of the program which deals with the results of this operation is invoked. It is possible to use anonymous callbacks but this is considered a bad practice since it makes programs difficult to read. Therefore named callbacks are to be preferred over anonymous callbacks.

Event-loop

The event loop, also called message dispatcher is a programming construct that waits for and dispatches events or messages in a program. It works by polling some internal or external “event provider”, which generally blocks until an event has arrived, and then calls the relevant event handler (“dispatches the event”).

Ted Faison’s book “Event-Based Programming: Taking Events to the Limit” proved very helpful to me in understanding the ins and outs of event oriented programming. The book follows a structured approach and is very well written.

According to Fairson the event programming model is by far the superior programming model especially for large software systems. Dependencies between software parts make large software systems complex and difficult to maintain. Dependencies have the biggest impact on software quality since the most complex parts also have the most dependencies. The event oriented programming model helps you gain better modularization so development, testing and maintenance of parts of the system is easier. Fairson makes a point that by following the event oriented programming model complexity grows linear with the size of the system while you will experience exponential grows when following other models. If this proves to be true this will be the biggest discovery in software engineering for the last decade.

Message passing

An event loop is one of the methods used for implementing inter-process communication. Message passing is currently used in many software systems, including operation system kernels.

Actor model

The actor model is a programming model that treats “actors” as the universal primitive. The actor model is an abstraction on the message passing model where in response to a message that it receives, an actor can make local decisions, create more actors, send more messages, and determine how to respond to the next message received.

In programming network servers we try to avoid abstractions and rip out many of the layers of the current software stack so we can get done more on a single machine (green computing).

Implementations

As we have seen in this article the event loop is the superior programming model. Some remarkable implementations which support the event oriented programming model are available at the moment like Twisted and node.js.