a bit about me

twitter

@j19sch

web

testingcurve.wordpress.com

fourhourtester.net (with Helena Jeret-Mäe)

slides

j19sch.github.io

2014 Postman & Newman

some issues

version control

details in test report

setup & teardown

polling

2017 own framework

why Python?

used for other auto-tests

great language for automation

migration still ongoing

my own framework

requests

pytest

pytest-html

git

my own framework

read test data files (pyyaml and pytest)

pytest-logfest

default API credentials

response body parsing

glue well-established tools
and libraries together

what we'll be doing today

the API we will be using

books API (GitHub)


GET /books/{book-id}

{
    "author": "Gerald Weinberg", 
    "id": "9b30d321-d242-444f-b2db-884d04a4d806", 
    "pages": 182, 
    "publisher": "Dorset House Publishing", 
    "sub_title": null, 
    "title": "Perfect Software And Other Illusions About Testing", 
    "year": 2008
}

let's run some tests


$ pytest
================================= test session starts ==================================
platform linux -- Python 3.6.7, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
rootdir: /home/jss/workspace/building-an-api-testing-framework/ukstar2019, inifile:
plugins: metadata-1.8.0, html-1.20.0
collected 14 items                                                                     

test_books.py ..............                                                     [100%]

============================== 14 passed in 0.09 seconds ===============================

Run & Report

so what did we test?


$ pytest -v
================================= test session starts ==================================
platform linux -- Python 3.6.7, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
rootdir: /home/jss/workspace/building-an-api-testing-framework/ukstar2019, inifile:
plugins: metadata-1.8.0, html-1.20.0
collected 14 items                                                                     

test_books.py::test_get_books PASSED                                     [  7%]
test_books.py::test_get_single_book PASSED                               [ 14%]
test_books.py::test_get_single_book_invalid_uuid PASSED                  [ 21%]
test_books.py::test_add_book PASSED                                      [ 28%]
test_books.py::test_add_book_invalid_request PASSED                      [ 35%]

so what did we test?


test_books.py::test_delete_book PASSED                                   [ 42%]
test_books.py::test_delete_nonexisting_book PASSED                       [ 50%]
test_books.py::test_delete_book_no_auth PASSED                           [ 57%]
test_books.py::test_delete_book_invalid_auth PASSED                      [ 64%]
test_books.py::test_delete_book_invalid_uuid PASSED                      [ 71%]
test_books.py::test_update_book PASSED                                   [ 78%]
test_books.py::test_update_nonexisting_book PASSED                       [ 85%]
test_books.py::test_update_book_invalid_request PASSED                   [ 92%]
test_books.py::test_update_book_invalid_uuid PASSED                      [100%]

============================== 14 passed in 0.09 seconds ===============================

name your tests
to clarify intention

a test does one thing
and one thing only

let's look at the code


def test_add_book(books_api):
    new_book = {
        "author": "Neil Gaiman",
        "pages": 299,
        "publisher": "W.W. Norton & Company",
        "sub_title": None,
        "title": "Norse Mythology",
        "year": 2017
    }

    response = books_api.add_book(new_book)
    assert response.status_code == 201
    new_book['id'] = response.json()['id']

    response = books_api.get_one_book(new_book['id'])
    assert response.status_code == 200
    assert response.json() == new_book

separate test code
from interface code

let's look at more code


class BooksApi(requests.Session):
    def __init__(self):
        super().__init__()
        self.hooks['response'].append(self._log_details)
        self.url = f"{BASE_URL}/books"

    def get_one_book(self, book_id):
        return self.get(self.url + '/' + book_id)

    def add_book(self, new_book):
        return self.post(self.url, json=new_book)

use transparent abstractions
for your seams

readable tests
versus
clean code

Read

let's add a test


def test_add_book_invalid_request(books_api):
    new_book = {
        "pages": 299,
        "publisher": "W.W. Norton & Company",
        "sub_title": None,
        "title": "Norse Mythology",
        "year": 2017
    }

    response = books_api.add_book(new_book)
    assert response.status_code == 400
    assert response.json()['title'] == "Failed data validation"
    assert response.json()['description'] == "'author' is a required property"

programming languages
are domain-specific languages

Create

and then a test failed


$ pytest test_books.py::test_delete_book
====================================== test session starts =============================
platform linux -- Python 3.6.7, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
rootdir: /home/jss/workspace/building-an-api-testing-framework/ukstar2019, inifile:
plugins: metadata-1.8.0, html-1.20.0
collected 1 item                                                                                

test_books.py F                                                                   [100%]

======================================= FAILURES ========================================

and then a test failed


_______________________________________ test_delete_book ________________________________________

books_api = <ukstar2019.api_clients.BooksApi object at 0x7f78545794a8>
initial_books = [{'author': 'Gerald Weinberg', 'id': '9b30d321-d242-444f-b2db-884d04a4d806', 'pages': 182, 'publisher': 'Dorset House ...or': 'John Ousterhout', 'id': 'd917b78e-1dc2-4f3e-87ea-245c55c6fd52', 'pages': 178, 'publisher': 'Yaknyam Press', ...}]
creds = ('bob', 'FnbLmFORYenkROl')
new_book = {'author': 'Nicole Forsgren, Jez Humble, Gene Kim', 'id': '224ec88b-eac6-4509-8ab7-5983e764fcc6', 'pages': 257, 'publisher': 'IT Revolution', ...}

    def test_delete_book(books_api, initial_books, creds, new_book):
        book_to_delete = new_book['id']
        user, token = creds
    
        response = books_api.delete_book(book_to_delete, user, token)
>       assert response.status_code == 200
E       assert 204 == 200
E        +  where 204 = <Response [204]>.status_code

test_books.py:69: AssertionError

and then a test failed


-------------------------------------- Captured log setup ---------------------------------------
api_clients.py    15 INFO    POST: http://localhost:8001/token/bob
api_clients.py    16 INFO    headers: {'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '0'}
api_clients.py    20 INFO    response status: 201, elapsed: 0.001371s
api_clients.py    21 INFO    headers: {'Server': 'gunicorn/19.9.0', 'Date': 'Sun, 03 Mar 2019 20:07:44 GMT', 'Connection': 'close', 'content-type': 'application/json; charset=UTF-8', 'content-length': '28'}
api_clients.py    23 INFO    response body: {"token": "FnbLmFORYenkROl"}
api_clients.py    15 INFO    POST: http://localhost:8001/books
api_clients.py    16 INFO    headers: {'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '210', 'Content-Type': 'application/json'}
api_clients.py    18 INFO    request body: b'{"author": "Nicole Forsgren, Jez Humble, Gene Kim", "pages": 257, "publisher": "IT Revolution", "sub_title": "Building and scaling high performing technology organizations", "title": "Accelerate", "year": 2018}'
api_clients.py    20 INFO    response status: 201, elapsed: 0.001469s
api_clients.py    21 INFO    headers: {'Server': 'gunicorn/19.9.0', 'Date': 'Sun, 03 Mar 2019 20:07:44 GMT', 'Connection': 'close', 'content-type': 'application/json; charset=UTF-8', 'content-length': '46'}
api_clients.py    23 INFO    response body: {"id": "224ec88b-eac6-4509-8ab7-5983e764fcc6"}
--------------------------------------- Captured log call ---------------------------------------
api_clients.py    15 INFO    DELETE: http://localhost:8001/books/224ec88b-eac6-4509-8ab7-5983e764fcc6
api_clients.py    16 INFO    headers: {'User-Agent': 'python-requests/2.21.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'user': 'bob', 'token': 'FnbLmFORYenkROl', 'Content-Length': '0'}
api_clients.py    20 INFO    response status: 204, elapsed: 0.000801s
api_clients.py    21 INFO    headers: {'Server': 'gunicorn/19.9.0', 'Date': 'Sun, 03 Mar 2019 20:07:44 GMT', 'Connection': 'close'}
=================================== 1 failed in 0.04 seconds ====================================

there is no such thing as
too much information
if it's well-structured

never trust a test
you haven't seen fail

Run & Report

and then a test errored


_______________________________________ test_get_single_book _______________________________________

books_api = <ukstar2019.api_clients.BooksApi object at 0x7fe3ab605550>
initial_books = [{'author': 'Gerald Weinberg', 'id': '9b30d321-d242-444f-b2db-884d04a4d806', 'pages': 182, 'publisher': 'Dorset House ...or': 'John Ousterhout', 'id': 'd917b78e-1dc2-4f3e-87ea-245c55c6fd52', 'pages': 178, 'publisher': 'Yaknyam Press', ...}]

    def test_get_single_book(books_api, initial_books):
        expected = initial_books[0].copy()
    
>       response = books_api.get_one_book(expected['id'])

test_books.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

and then a test errored


api_clients.py:43: in get_one_book
    return self.get(self.url + '/' + book_id)
venv/lib/python3.6/site-packages/requests/sessions.py:546: in get
    return self.request('GET', url, **kwargs)
venv/lib/python3.6/site-packages/requests/sessions.py:533: in request
    resp = self.send(prep, **send_kwargs)
venv/lib/python3.6/site-packages/requests/sessions.py:653: in send
    r = dispatch_hook('response', hooks, r, **kwargs)
venv/lib/python3.6/site-packages/requests/hooks.py:31: in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)

and then a test errored


response = <Response [200]>, args = (), kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': None, ...}

    @staticmethod
    def _log_details(response, *args, **kwargs):
        logging.info("{}: {}".format(response.request.method, response.request.url))
        logging.info("headers: {}".format(response.request.headers))
        if response.request.body is not None:
            logging.info("request body: {}".format(response.request.body))
    
        logging.info("response status: {}, elapsed: {}s".format(response.status_code, response.elapsed.total_seconds()))
>       logging.info("headers: {}".format(response.header))
E       AttributeError: 'Response' object has no attribute 'header'

api_clients.py:20: AttributeError
================================== 1 failed in 0.05 seconds ==================================

let's do some debugging


$ pytest test_books.py::test_get_single_book --pdb
====================================== test session starts ======================================
platform linux -- Python 3.6.7, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
rootdir: /home/jss/workspace/building-an-api-testing-framework/ukstar2019, inifile:
plugins: metadata-1.8.0, html-1.20.0
collected 1 item

test_books.py F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
--- snip test output ---

let's do some debugging


>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[17] > /home/jss/workspace/building-an-api-testing-framework/ukstar2019/
	api_clients.py(20)_log_details()
-> logging.info("headers: {}".format(response.header))
   6 frames hidden (try 'help hidden_frames')
(Pdb++) dir(response)
['__attrs__', '__bool__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__',
'__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__',
'__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__',
'__le__', '__lt__', '__module__', '__ne__', '__new__', '__nonzero__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__',
'__subclasshook__', '__weakref__', '_content', '_content_consumed', '_next',
'apparent_encoding', 'close', 'connection', 'content', 'cookies', 'elapsed', 'encoding',
'headers', 'history', 'is_permanent_redirect', 'is_redirect', 'iter_content', 'iter_lines',
'json', 'links', 'next', 'ok', 'raise_for_status', 'raw', 'reason', 'request', 'status_code',
'text', 'url']

Debug

many different things can go wrong
make sure you can find out what

forced switching
between levels sucks

wrap-up

Activities

Create

Read

U

D

Debug

E

Run & report

Activities

Create

Read

Update

D

Debug

E

Run & report

Activities

Create

Read

Update

Delete

Debug

E

Run & report

Activities

Create

Read

Update

Delete

Debug

Explore

Run & report

Levels

Appliance

product

user

Tool

tests

power user

Machine

implementation

toolsmith

forget about the shiny

is your tool helping you
to do better testing?

image attributions

“lightbulb icon” by paomedia is part of the public domain

"nuts and bolts icon" by Chanut is Industries is licensed under CC BY 3.0

"toaster icon" by W.X. Chee is free for commercial use (include link to author's website)

"drill icon" by Chanut is Industries is licensed under CC BY 3.0

Thank you!

@j19sch
j19sch.github.io

a bit about me

twitter

@j19sch

web

testingcurve.wordpress.com

fourhourtester.net (with Helena Jeret-Mäe)

slides

j19sch.github.io

2014 Postman & Newman

some issues

version control

details in test report

setup & teardown

polling

2017 own framework

why Python?

used for other auto-tests

great language for automation

migration still ongoing

my own framework

requests

pytest

pytest-html

git

my own framework

read test data files (pyyaml and pytest)

pytest-logfest

default API credentials

response body parsing

glue well-established toolsand libraries together

what we'll be doing today

the API we will be using

books API (GitHub)

let's run some tests

Run & Report

so what did we test?

so what did we test?

name your teststo clarify intention

a test does one thingand one thing only

let's look at the code

separate test codefrom interface code

let's look at more code

use transparent abstractionsfor your seams

readable testsversusclean code

Read

let's add a test

programming languagesare domain-specific languages

Create

and then a test failed

and then a test failed

and then a test failed

there is no such thing astoo much informationif it's well-structured

never trust a testyou haven't seen fail

Run & Report

and then a test errored

and then a test errored

and then a test errored

let's do some debugging

let's do some debugging

Debug

many different things can go wrongmake sure you can find out what

forced switchingbetween levels sucks

wrap-up

Activities

Create

Read

U

D

Debug

E

Run & report

Activities

Create

Read

Update

D

Debug

E

Run & report

Activities

glue well-established tools
and libraries together

name your tests
to clarify intention

a test does one thing
and one thing only

separate test code
from interface code

use transparent abstractions
for your seams

readable tests
versus
clean code

programming languages
are domain-specific languages

there is no such thing as
too much information
if it's well-structured

never trust a test
you haven't seen fail

many different things can go wrong
make sure you can find out what

forced switching
between levels sucks

is your tool helping you
to do better testing?

@j19sch
j19sch.github.io