Testing Framework for glideinWMS

Proposal

PyUnit? is the industry standard for managing and executing Unit Tests on Python. It is based on the long tradition of the xUnit testing methodology/patterns, which started in Smalltalk but has quickly expanded to most modern programming languages.

Conveniently enough, PyUnit? has been included in the Python Standard Library since 2.1; the module is called "unittest". While there are many unit testing frameworks out there for Python, PyUnit? has often been found as one of the preferred ones.

As stated in this report <http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-1-unittest.html>, here is a run-down of relevant Pros and Cons of PyUnit? :

unittest pros

• available in the Python standard library
• easy to use by people familiar with the xUnit frameworks
• flexibility in test execution via command-line arguments
• support for test fixture/state management via set-up/tear-down hooks
• strong support for test organization and reuse via test suites

unittest Cons

• xUnit flavor may be too strong for "pure" Pythonistas
• API can get in the way and can make the test code intent hard to understand
• tests can end up having a different look-and-feel from the code under test
• tests are executed in alphanumerical order
• assertions use custom syntax

I worked with xUnit in my past jobs and the framework is really an idea one for unit testing in general.

Attached is a quick unit test written up for xmlParse.py and xmlFormat.py, which fail intentionally to show how the output is formatted.

Traditionally, unit tests are stored in a directory in the root of the project, and checked into version control. Ideally, one file would be used per "unit" tested.

There are python frameworks available, such as nose, which work to manage and run series of tests and aggregate reports seamlessly.

Examples

Example 1

Sample testing code

import xmlFormat
import xmlParse
import os.path

import unittest

class TestSequenceFunctions(unittest.TestCase):
  
  # Is run before each test
  def setUp(self):
    
    self.test_file_path = "test.xml"
    self.test_xml_path = "test_xml.xml"
    
    self.test_dict = {"hello": "world", "okay": "go", "foo": "bar"}
    self.test_string = '\n   \n   \n   \n'
  
  
  #### TESTS ####
  
  def test_dict2string(self):
    # (I'm not 100% sure what dict_name or el_name should be)
    self.serialized_string = xmlFormat.dict2string(self.test_dict,"test_dict","key")
    
    # TEST: Assert that serialized_string is indeed a string
    self.assertTrue(isinstance(self.serialized_string,str))
    
    # TEST: Assert that each key and value pair is found in the string at least once
    for k, v in self.test_dict.items():
      self.assertTrue(self.serialized_string.find(k) > -1)
      self.assertTrue(self.serialized_string.find(v) > -1)
  
  def test_xmlstring2dict(self):
    # Input: Any valid, well-formed xml file representing a dict
    self.parsed_dict = xmlParse.xmlstring2dict(self.test_string)
    
    # TEST: Assert that parsed_dict is indeed a dict
    self.assertTrue(isinstance(self.parsed_dict,dict))
  
  def test_dict2file(self):
    self.f = open(self.test_file_path,"w")
    xmlFormat.dict2file(self.f,self.test_dict,"test_dict","key")
    self.f.close()
    
    # TEST: Assert that file exists
    self.assertTrue(os.path.isfile(self.test_file_path))
  
  def test_xmlfile2dict(self):
    self.parsed_dict = xmlParse.xmlfile2dict(self.test_xml_path)
    
    # TEST: Assert that parsed_dict is indeed a dict
    self.assertTrue(isinstance(self.parsed_dict,dict))
  
  
  
  # Testing serialization and parsing together
  
  def test_string_serialization(self):
    self.serialized_string = xmlFormat.dict2string(self.test_dict,"test_dict","key")
    self.parsed_dict = xmlParse.xmlstring2dict(self.serialized_string)
    
    # TEST: Assert that the original dict is equal to the parsed dict
    self.assertEqual(self.test_dict,self.parsed_dict)
    
    
  def test_xml_serialization(self):
    self.f = open(self.test_file_path,"w")
    xmlFormat.dict2file(self.f,self.test_dict,"test_dict","key")
    self.f.close()
    
    self.parsed_dict = xmlParse.xmlfile2dict(self.test_file_path)
    
    # TEST: Assert that the original dict is equal to the parsed dict
    self.assertEqual(self.test_dict,self.parsed_dict)

if __name__ == '__main__':
  unittest.main()

Sample Output

C:\Users\justin\Projects\Glidein\glideinWMS\lib>python xmlTest.py
..FF..
======================================================================
FAIL: test_string_serialization (__main__.TestSequenceFunctions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "xmlTest.py", line 63, in test_string_serialization
    self.assertEqual(self.test_dict,self.parsed_dict)
AssertionError: {'foo': 'bar', 'okay': 'go', 'hello': 'world'} != {u'key': {u'name': u'okay', u'val': u'go'}}

======================================================================
FAIL: test_xml_serialization (__main__.TestSequenceFunctions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "xmlTest.py", line 74, in test_xml_serialization
    self.assertEqual(self.test_dict,self.parsed_dict)
AssertionError: {'foo': 'bar', 'okay': 'go', 'hello': 'world'} != {u'key': {u'name': u'okay', u'val': u'go'}}

----------------------------------------------------------------------
Ran 6 tests in 0.169s

FAILED (failures=2)
#################### VERBOSE MODE ###################

C:\Users\justin\Projects\Glidein\glideinWMS\lib>python xmlTest.py -v
test_dict2file (__main__.TestSequenceFunctions) ... ok
test_dict2string (__main__.TestSequenceFunctions) ... ok
test_string_serialization (__main__.TestSequenceFunctions) ... FAIL
test_xml_serialization (__main__.TestSequenceFunctions) ... FAIL
test_xmlfile2dict (__main__.TestSequenceFunctions) ... ok
test_xmlstring2dict (__main__.TestSequenceFunctions) ... ok

======================================================================
FAIL: test_string_serialization (__main__.TestSequenceFunctions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "xmlTest.py", line 63, in test_string_serialization
    self.assertEqual(self.test_dict,self.parsed_dict)
AssertionError: {'foo': 'bar', 'okay': 'go', 'hello': 'world'} != {u'key': {u'name': u'okay', u'val': u'go'}}

======================================================================
FAIL: test_xml_serialization (__main__.TestSequenceFunctions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "xmlTest.py", line 74, in test_xml_serialization
    self.assertEqual(self.test_dict,self.parsed_dict)
AssertionError: {'foo': 'bar', 'okay': 'go', 'hello': 'world'} != {u'key': {u'name': u'okay', u'val': u'go'}}

----------------------------------------------------------------------
Ran 6 tests in 0.053s

FAILED (failures=2)