Modern Testing provides concise summaries of key points from the document in 3 sentences: The document discusses modern testing practices and frameworks for Python projects, emphasizing the importance of writing tests to ensure high quality software and outlining different types of tests like unit, integration, and acceptance tests. It presents examples of using the unittest and pytest frameworks for writing tests in Python as well as the Robot Framework for acceptance testing. The document concludes by advocating for adopting best practices like using tox for local test invocation and continuous integration to enforce testing standards.

1. Modern
Testing
Alexander Loechel
PloneConf 2017 - Barcelona

2. The Goal of Testing
→ Produce high quality / correct software
“Program testing can be used to show
the presence of bugs, but never show
their absence!”
Edsger Dijkstra

3. Testing is about
responsibility & sustainability
→ Path for Plone on Python 3

4. What to test
● Requirements
● Design
● Interfaces
● Code / Implementation
● Documentation
● Conventions

5. Test Design / Types
V-ModelUser Requirements Acceptance Test
Requirements Verification Functional Test
System Design Validation System Integration Test
Detailed Design Subsystem Integration Tests
Software Architecture (Interfaces) Unit-Tests
Coding / Implementation
Classical Software Development Process / Governmental Standard

6. Open Source Development
The development process in Open Source Projects work differently,
but a lot of the concepts of the V-Model are reflected in other ways.
Documentation
→ Reflects Requirements, Intentions and Design
Tests reflects the requirements
⇒ TTD - Test Driven Development

7. “Testing leads to failure,
and failure leads to understanding.”
Burt Rutan
Writing tests is mandatory for good software

8. Definitions

9. Test
A test is a specific set of assertions

10. Test Case
A test case is the smallest unit of testing.
It checks for a specific response to a particular set of inputs.

11. Test Fixture
A test fixture represents the preparation needed to perform
one or more tests, and any associate cleanup actions

12. Test Suite
A test suite is a collection of test cases, test suites, or both.
It is used to aggregate tests that should be executed together.

13. Test Layer
(plone.testing / plone.app.testing)
A test layer represents the baseline for a specific test
→ Reflects Test-Level in V-Model
● Unit tests
● Integration tests
● Functional tests
● Acceptance tests

14. Test Runner
A test runner is a component which orchestrates the execution of tests
and provides the outcome to the user.

15. Test Invocation Tool
A test invocation tool is a component which provides the
required infrastructure to the test runner to execute the test sets.

16. Mock
mock is a library for testing in Python.
It allows you to replace parts of your system under test with mock objects and
make assertions about how they have been used.
→ Test Isolation

17. Writing test / What to test
● Requirements → Acceptance tests
● Design → Functional tests
● Interfaces → Integration tests
● Code / Implementation → Unit tests
● Documentation → Test if your code examples actually works
● Conventions → Test if the convention of the project is followed
(e.g. Coding Conventions)

18. Test Frameworks

19. Unittest
The Python unit testing framework, sometimes referred to as “PyUnit,” is a Python language version of JUnit, by Kent Beck
and Erich Gamma. JUnit is, in turn, a Java version of Kent’s Smalltalk testing framework. Each is the de facto standard unit
testing framework for its respective language.
unittest supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections,
and independence of the tests from the reporting framework. The unittest module provides classes that make it easy to
support these qualities for a set of tests.
● Python Standard Library module
● Used in Plone for testing
● Specific BaseClasses, and assert methods necessary, setup and teardown
methods

20. Unitest example
(Plone context - Plone Training Documentation)
import unittest
class TalkIntegrationTest(unittest.TestCase):
layer = PLONECONF_SITE_INTEGRATION_TESTING
def setUp(self):
self.portal = self.layer['portal']
setRoles(self.portal, TEST_USER_ID, ['Manager'])
def test_fti(self):
fti = queryUtility(IDexterityFTI, name='talk')
self.assertTrue(fti)
def test_adding(self):
self.portal.invokeFactory('talk', 'talk')
self.assertTrue(self.portal.talk)
self.assertTrue(ITalk.providedBy(self.portal.talk))
…
suite = unittest.TestLoader().loadTestsFromTestCase(TalkIntegrationTest)
Assert Methods
Method Checks that
assertEqual(a, b) a == b
assertNotEqual(a, b) a != b
assertTrue(x) bool(x) is True
assertFalse(x) bool(x) is False
assertIs(a, b) a is b
assertIsNot(a, b) a is not b
assertIsNone(x) x is None7
assertIsNotNone(x) x is not None
assertIn(a, b) a in b
assertNotIn(a, b) a not in b
assertIsInstance(a, b) isinstance(a, b)
assertNotIsInstance(a, b) not isinstance(a, b)

21. “The pytest framework makes it easy to write small tests, yet scales to support
complex functional testing for applications and libraries.”
● De Facto Standard in the Python world
● Some magic to make writing tests simpler
● just assert
● Implicit test loader
● Plugable addon system

22. pytest example
(Plone context - RestrictedPython)
from tests import e_eval
import pytest
@pytest.mark.parametrize(*e_eval)
def test_Eq(e_eval):
assert e_eval('1 == 1') is True
@pytest.mark.parametrize(*e_eval)
def test_NotEq(e_eval):
assert e_eval('1 != 2') is True
@pytest.mark.parametrize(*e_eval)
def test_Gt(e_eval):
assert e_eval('2 > 1') is True
@pytest.mark.parametrize(*e_eval)
def test_Lt(e_eval):
assert e_eval('1 < 2')

23. unittest2 / nose / nose2
Forks of unitest that either enhance or backport functionality
Mostly outdated and not recommended anymore.

24. Robot Framework
Robot Framework is a generic test automation framework for acceptance testing
and acceptance test-driven development (ATDD).
● Focus: Web Applications
● Wrapper around Selenium

25. Robot tests example
(Plone context - Plone Training Documentation)
*** Settings ***********************************************
Resource plone/app/robotframework/selenium.robot
Resource plone/app/robotframework/keywords.robot
Library Remote ${PLONE_URL}/RobotRemote
Test Setup Open test browser
Test Teardown Close all browsers
*** Test Cases *********************************************
Scenario: As a site administrator I can add a Talk
Given a logged-in site administrator
and an add talk form
When I type 'My Talk' into the title field
and I type 'Awesome talk' into the details field
and I type 'Team Banzai' into the speakers field
and I type 'banzai@example.com' into the email field
and I submit the form
Then a talk with the title 'My Talk' has been created
Scenario: As a site administrator I can view a Talk
Given a logged-in site administrator
and a talk 'My Talk'
When I go to the talk view
Then I can see the talk title 'My Talk'
Scenario: As a visitor I can view the new talk list
When I go to the talk list view
Then I can see a talk about 'Diazo designs are great'
*** Keywords ****************************************
# --- Given ------------------------------------------
a logged-in site administrator
Enable autologin as Site Administrator
an add talk form
Go To ${PLONE_URL}/++add++talk
a talk 'My Talk'
Create content type=talk id=my-talk title=My Talk
# --- WHEN --------------------------------------------
I type '${title}' into the title field
Input Text name=form.widgets.IDublinCore.title ${title}
I type '${details}' into the details field
Select frame form-widgets-details_ifr
Input text tinymce ${details}
Unselect Frame
I type '${speaker}' into the speakers field
Input Text name=form.widgets.speaker ${speaker}
I type '${email}' into the email field
Input Text name=form.widgets.email ${email}
I submit the form
Click Button Save
I go to the talk view
Go To ${PLONE_URL}/my-talk
Wait until page contains Site Map
I go to the talk list view
Go To ${PLONE_URL}/demoview
Wait until page contains Site Map
# --- THEN ----------------------
a talk with the title '${title}' has been created
Wait until page contains Site Map
Page should contain ${title}
Page should contain Item created
I can see the talk title '${title}'
Wait until page contains Site Map
Page should contain ${title}
I can see a talk about '${topic}'
Wait until page contains Site Map
Page should contain ${topic}

26. “The first principle is that you must not
fool yourself - and you are the easiest
person to fool.”
Richard Feynman

27. Test runners
● unittest testrunner
● zope.testrunner
● pytest-testrunner
● gocept.pytestlayer
A test runners is a component which orchestrates the execution of tests and
provides the outcome to the user.
● collects tests
● presents outcome
● interact with other tools (e.g. coverage)

28. Test invocation tools
● Command line
● Continuous Integration Servers
● tox

29. Test invocation tools
● Command line
○ python setup.py tests
○ python -m unittest test_module.TestClass
○ python -m pytest tests
○ bin/test (zc.buildout script generated by zope.recipe.testrunner)

30. ● Continuous Integration Servers
(Linux/MacOS X - Machines)
perfect for pure Python tests
(Docker containers, Linux/MacOS X)
(Windows)
Test invocation tools

31. travis-ci example
(Plone context - RestrictedPython)
language: python
sudo: false
matrix:
include:
- python: "2.7"
env: TOXENV=docs,lint-py27
- python: "3.6"
env: TOXENV=docs,lint-py36
- python: "2.7"
env: TOXENV=py27
- python: "2.7"
env: TOXENV=py27-datetime
- python: "3.4"
env: TOXENV=py34
- python: "3.5"
env: TOXENV=py35
- python: "3.6"
env: TOXENV=py36
- python: "3.6"
env: TOXENV=py36-datetime
- python: "3.7-dev"
env: TOXENV=py37
- python: "pypy"
env: TOXENV=pypy
- python: "pypy3"
env: TOXENV=pypy
allow_failures:
- python: "3.7-dev"
env: TOXENV=py37
install:
- travis_retry pip install -U pip
setuptools
- travis_retry pip install -U -c
constraints.txt tox coveralls coverage
script:
- travis_retry tox
after_success:
- coverage combine
- coveralls
notifications:
email: false

32. Test invocation tools
● tox - Translate the concept of continuous Integration to local development
○ De facto standard as a local test invocation tool
○ Groups environments, allow to test different Python versions support
Fantastic if you use additional helpers:
● pyenv - having multiple Python version
● git-hooks - run scripts on git commands → pre-commit hook

33. tox example
(Plone context - RestrictedPython)
[tox]
envlist =
py{27,34,35,36},
docs,
lint-py27,
lint-py36,
coverage,
skip_missing_interpreters = False
[testenv]
usedevelop = True
extras =
develop
commands =
pytest --cov=src --cov-report=xml --html=reports/pytest/report-{envname}.html --doctest-glob=*.rst --self-contained-html
{posargs}
pytest --doctest-modules src/RestrictedPython/compile.py {posargs}
setenv =
COVERAGE_FILE=.coverage.{envname}
deps =
-cconstraints.txt
pytest-cov
pytest-remove-stale-bytecode
pytest-html
[testenv:coverage]
basepython = python2.7
skip_install = true
deps =
-cconstraints.txt
coverage
setenv =
COVERAGE_FILE=.coverage
commands =
coverage erase
coverage combine
coverage html
coverage xml
coverage report
[testenv:docs]
deps =
-cconstraints.txt
Sphinx
commands =
python -V
sphinx-build -b html -d build/docs/doctrees docs build/docs/html
sphinx-build -b doctest docs build/docs/doctrees

34. tox example - Linter - enforce Coding Conventions
(Plone context - RestrictedPython)
[lint]
skip_install = true
deps =
-cconstraints.txt
isort
flake8
# helper to generate HTML reports:
flake8-html
# Useful flake8 plugins that are Python and Plone specific:
flake8-coding flake8-debugger flake8-deprecated
Flake8-print flake8-pytest flake8-todo flake8-isort
mccabe
# Potential flake8 plugins that should be used:
Flake8-blind-except flake8-commas,flake8-docstrings
Flake8-pep3101 flake8-plone-hasattr, flake8-string-format
Flake8_strict flake8-quotes
commands =
mkdir -p {toxinidir}/reports/flake8
isort --check-only --recursive {toxinidir}/src {toxinidir}/tests setup.py
- flake8 --format=html --htmldir={toxinidir}/reports/flake8 --doctests src tests setup.py
flake8 src tests setup.py --doctests
whitelist_externals =
mkdir
[testenv:lint-py27]
basepython = python2.7
skip_install = true
deps = {[lint]deps}
commands = {[lint]commands}
whitelist_externals = {[lint]whitelist_externals}
[testenv:lint-py36]
basepython = python3.6
skip_install = true
deps = {[lint]deps}
commands = {[lint]commands}
whitelist_externals = {[lint]whitelist_externals}

35. The Zen of Python - PEP20
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
...
Lessons learned from Zope and Plone
→ we should embrace each tool that helps us to provide a fantastic products

36. My Wishes to better “Best Practices” for Plone
● Adopt tox on all packages
● Switch to a different package structure and enforce that → bobtemplates.plone
○ docs
○ src
○ tests
● detailed and enforced settings for conventions
○ .editorconf
○ setup.cfg
→ https://github.com/plone/plone_best_practices_discussion