Part 5: RESTful Resources and Advanced Request Parsing

How To: Create a Flask API with JWT-Based Authentication (Part 5)

Photo by Tania Melnyczuk on Unsplash

Project Structure

The chart below shows the folder structure for this section of the tutorial. In this post, we will work on all files marked as NEW CODE. Files that contain code from previous sections but will not be modified in this post are marked as NO CHANGES.

. (project root folder) |- src | |- flask_api_tutorial | |- api | | |- auth | | | |- __init__.py | | | |- business.py | | | |- decorators.py | | | |- dto.py | | | |- endpoints.py | | | | | |- widgets | | | |- __init__.py | | | |- business.py | | | |- dto.py | | | |- endpoints.py | | | | | |- __init__.py | | | |- models | | |- __init__.py | | |- token_blacklist.py | | |- user.py | | |- widget.py | | | |- util | | |- __init__.py | | |- datetime_util.py | | |- result.py | | | |- __init__.py | |- config.py | |- tests | |- __init__.py | |- conftest.py | |- test_auth_login.py | |- test_auth_logout.py | |- test_auth_register.py | |- test_auth_user.py | |- test_config.py | |- test_user.py | |- util.py | |- .env |- .gitignore |- .pre-commit-config.yaml |- pyproject.toml |- pytest.ini |- README.md |- run.py |- setup.py |- tox.ini
KEY:
FOLDER
NEW CODE
NO CHANGES
EMPTY FILE

Introduction

In the previous section, we created four API endpoints to perform basic user registration and authentication functions. However, these API endpoints are not designed as RESTful resources (I explained my reasoning for this choice in Part 3).

In this section of the tutorial, we will create a resource that is REST-like (REST-faux? REST-adjacent?). I am deliberately not describing it as RESTful, because designing a truly RESTful system is HARD. Check out this blog post from Roy Fielding and the discussion in the comments to get an idea of what I mean.

The only features of this resource that I am willing to state are 100% bona fide REST-compliant are:

  • The naming convention of the resource and associated endpoints.
  • The HTTP methods supported by each endpoint that enables clients to perform CRUD operations.
  • Through the use of pagination and navigational links included in JSON sent by the server, clients can interact with the API purely through hypertext (i.e., clients NEVER need to manually construct URLs to interact with the API).

The resource we will create is a collection of widgets. I decided to model something generic rather than the cliche “to-do list” project that you encounter in every introductory programming tutorial. I feel safe assuming that you are not reading this because you have a burning desire to create the next, great API-driven to-do list.

The main purpose of this section is to learn more advanced techniques for request parsing and response marshalling. The Widget model will contain attributes that require creating custom input types for parsing request data. The Widget model also contains hybrid properties that will require rendering various data types in JSON. Whatever project you have in mind, the techniques demonstrated with the Widget model and associated RequestParser and API model instances can easily be adapted to any object in your domain.

widget_ns Endpoints

The proper way to name resources is one of the (many) hotly debated topics regarding RESTful web services. I recommend taking the time to read the articles below to understand the current, accepted best practices:

The accepted best practice for naming resources is to use plural nouns when constructing a URI for a resource (e.g. /widgets instead of /widget). Another widely accepted standard is to create two endpoints per resource — one for operations that apply to the entire collection (e.g., /api/v1/widgets) and one for operations that apply only to a single instance (e.g., /api/v1/widgets/<name>).

Another common architectural pattern is to expose methods which allow the client to perform CRUD operations on the resource. CRUD (Create, Retrieve, Update, Delete) is a term that usually refers to relational database systems, but it is also valid for any dataset that can be manipulated by a client, including RESTful resources.

Now, you might be wondering how this can be accomplished if we only expose two endpoints per resource, and CRUD defines (at leat) four different operations. Table 1 shows how we will structure the API for our Widget resource:

Table 1
Widget API endpoint specifications
Endpoint NameURIHTTP MethodCRUD OperationRequired Token
api.widget_list/api/v1/widgetsPOSTCreate a new widgetAdmin user
api.widget_list/api/v1/widgetsGETRetrieve a list of widgetsRegular user
api.widget/api/v1/widgets/<name>GETRetrieve a single widgetRegular user
api.widget/api/v1/widgets/<name>PUTUpdate an existing widgetAdmin user
api.widget/api/v1/widgets/<name>DELETEDelete a single widgetAdmin user

Each endpoint can be configured to respond to a unique set of HTTP method types. Per Table 1, the api.widget_list endpoint will support GET and POST requests, and the api.widget endpoint will suport GET, PUT, and DELETE requests. Each combination of endpoint and method type are mapped to a CRUD operation. The remainder of this section (and the entire next section) are devoted to implementing the Widget API.

Operations that create, modify or delete widgets are restricted to users with the administrator role. Regular (non-admin) users can only retrieve individual widgets and/or lists of widgets from the database.

flask add-user Command

Way back in Part 1, we discussed the Flask CLI and created the method that executes when the flask shell command is invoked. The Flask CLI is based on a project called Click which can be used to create powerful Python CLI applications, and is easy to get started with thanks to excellent documentation.

Currently, the api/v1/auth/register endpoint can only create regular (non-admin) users. We want to leave it this way since this endpoint is publically-accessible. However, we also need a way to create admin users since regular users cannot create, update or delete widget objects.

There are a few different methods we could use to create admin users. Using the flask shell command, we can execute arbitrary Python code to create admin users. Or, we could create a function and store it in a file in our project, then run the function through the command-line. However, both of these methods are cumbersome and would require documentation if anyone else needed to create an admin user.

My preferred solution is to expose a command in the Flask CLI that can create both regular and admin users. To do so, open run.py in the project root folder. First, update the import statements to include click (Line 4):

1
2
3
4
5
6
7
8
"""Flask CLI/Application entry point."""
import os

import click

from flask_api_tutorial import create_app, db
from flask_api_tutorial.models.token_blacklist import BlacklistedToken
from flask_api_tutorial.models.user import User

Then, add the content below and save the file:

18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
@app.cli.command("add-user", short_help="Add a new user")
@click.argument("email")
@click.option(
    "--admin", is_flag=True, default=False, help="New user has administrator role"
)
@click.password_option(help="Do not set password on the command line!")
def add_user(email, admin, password):
    """Add a new user to the database with email address = EMAIL."""
    if User.find_by_email(email):
        error = f"Error: {email} is already registered"
        click.secho(f"{error}\n", fg="red", bold=True)
        return 1
    new_user = User(email=email, password=password, admin=admin)
    db.session.add(new_user)
    db.session.commit()
    user_type = "admin user" if admin else "user"
    message = f"Successfully added new {user_type}:\n {new_user}"
    click.secho(message, fg="blue", bold=True)
    return 0

Explaining how to create a command with click is beyond the scope of this tutorial. Thankfully, the click documentation is exceptional. If you are interested, you can find everything you need to understand the add_user function in the links below:

Finally, the Flask documentation explains how to add custom commands to the Flask CLI. Coincidentally, the example given in the documentation is a command to create a new user.

You might be wondering how is this different than the idea I dismissed, "create a function and store it in a file in our project, then run the function through the command-line"? The main difference is discoverability — Flask automatically includes a CLI with all installations and extending the CLI with custom commands is one of the intended use cases.

Also, help documentation is automatically generated for CLI commands (via click). You can view the documentation for the add-user command by running flask add-user --help:

(flask-api-tutorial) flask-api-tutorial $ flask add-user --help
Usage: flask add-user [OPTIONS] EMAIL

  Add a new user to the database with email address = EMAIL.

Options:
  --admin          New user has administrator privileges
  --password TEXT  Do not set password on the command line!
  --help           Show this message and exit.

Users can view all Flask CLI commands, including custom commands by running flask:

(flask-api-tutorial) flask-api-tutorial $ flask
Usage: flask [OPTIONS] COMMAND [ARGS]...

  A general utility script for Flask applications.

  Provides commands from Flask, extensions, and the application. Loads the
  application defined in the FLASK_APP environment variable, or from a
  wsgi.py file. Setting the FLASK_ENV environment variable to 'development'
  will enable debug mode.

    $ export FLASK_APP=hello.py
    $ export FLASK_ENV=development
    $ flask run

Options:
  --version  Show the flask version
  --help     Show this message and exit.

Commands:
  add-user  add a new user
  db        Perform database migrations.
  routes    Show the routes for the app.
  run       Run a development server.
  shell     Run a shell in the app context.

This is my preferred solution because the process for creating admin users is now documented, and the command is accessible only to those with access to the server where the flask instance is running.

Finally, let’s demonstrate how to use the flask add-user command:

  • Create regular user

    (flask-api-tutorial) flask-api-tutorial $ flask add-user user@test.com
    Password:
    Repeat for confirmation:
    Successfully added new user:
     <User email=user@test.com, public_id=be5e164e-7d33-4919-8d37-a5d02ca27d47, admin=False>
  • Create admin user

    (flask-api-tutorial) flask-api-tutorial $ flask add-user admin@test.com --admin
    Password:
    Repeat for confirmation:
    Successfully added new admin user:
     <User email=admin@test.com, public_id=1e248b12-e08b-467f-86bf-80f547f20ce6, admin=True>
  • Error: Email already exists

    (flask-api-tutorial) flask-api-tutorial $ flask add-user user@test.com
    Password:
    Repeat for confirmation:
    Error: user@test.com is already registered

As you can see, after running the command, the user is immediately prompted to create a password and to confirm the password for the new user. Before proceeding, please create an admin user with the flask add-user command since creating, modifying and deleting widgets cannot be performed otherwise.

Widget DB Model

Before we can begin implementing the API endpoints in Table 1, we need to create a database table to store Widget instances. To do so, we extend db.Model (just like we did for the User and BlacklistedToken classes). Create a new file widget.py in src/flask_api_tutorial/models and add the content below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
"""Class definition for Widget model."""
from datetime import datetime, timezone, timedelta

from sqlalchemy.ext.hybrid import hybrid_property

from flask_api_tutorial import db
from flask_api_tutorial.util.datetime_util import (
    utc_now,
    format_timedelta_str,
    get_local_utcoffset,
    localized_dt_string,
    make_tzaware,
)


class Widget(db.Model):
    """Widget model for a generic resource in a REST API."""

    __tablename__ = "widget"

    id = db.Column(db.Integer, primary_key=True, autoincrement=True)
    name = db.Column(db.String(100), unique=True, nullable=False)
    info_url = db.Column(db.String(255))
    created_at = db.Column(db.DateTime, default=utc_now)
    deadline = db.Column(db.DateTime)

    owner_id = db.Column(db.Integer, db.ForeignKey("site_user.id"), nullable=False)
    owner = db.relationship("User", backref=db.backref("widgets"))

    def __repr__(self):
        return f"<Widget name={self.name}, info_url={self.info_url}>"

    @hybrid_property
    def created_at_str(self):
        created_at_utc = make_tzaware(
            self.created_at, use_tz=timezone.utc, localize=False
        )
        return localized_dt_string(created_at_utc, use_tz=get_local_utcoffset())

    @hybrid_property
    def deadline_str(self):
        deadline_utc = make_tzaware(self.deadline, use_tz=timezone.utc, localize=False)
        return localized_dt_string(deadline_utc, use_tz=get_local_utcoffset())

    @hybrid_property
    def deadline_passed(self):
        return datetime.now(timezone.utc) > self.deadline.replace(tzinfo=timezone.utc)

    @hybrid_property
    def time_remaining(self):
        time_remaining = self.deadline.replace(tzinfo=timezone.utc) - utc_now()
        return time_remaining if not self.deadline_passed else timedelta(0)

    @hybrid_property
    def time_remaining_str(self):
        timedelta_str = format_timedelta_str(self.time_remaining)
        return timedelta_str if not self.deadline_passed else "No time remaining"

    @classmethod
    def find_by_name(cls, name):
        return cls.query.filter_by(name=name).first()

To demonstrate the process of serializing a complex object to/from JSON, the Widget class includes attributes with as many different data types as possible. Additionally, the project requirements include various rules restricting which values are considered valid for the name and deadline attributes:

  • The widget model contains attributes with URL, datetime, timedelta and bool data types, along with normal text fields.
  • The widget model contains a "name" field which must be a string value containing only lowercase-letters, numbers and the "-" (hyphen character) or "_" (underscore character).
  • The widget model contains a "deadline" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.

Let’s take a look at how these attributes are defined and how they fulfill the various project requirements:

  • Line 22: Table 1 indicates that the value of the name attribute will be embedded in the URI for each Widget. Because of this, the value must be unique which is ensured by setting unique=True. Additionally, it would be ideal to prevent the user from creating a new widget if the name contains characters that are not URL-safe (/, +, & etc.). To accomplish this, we will design a custom RequestParser data type that considers a value to be valid if it contains ONLY lowercase-letters, numbers, underscore and/or hyphen characters.

  • Line 23: The purpose of the info_url attribute is to demonstrate how to implement input validation for URL values. Any values that are not recognized as a valid URL must be rejected without adding the widget to the database. The validation logic will be implemented using a built-in RequestParser data type for URL values.

  • Line 25: The purpose of the deadline attribute is to demonstrate how to implement input validation for datetime values. Additionally, only datetime values that are either the same as or greater than the current date are considered valid. Values not recognized as valid datetime values AND valid datetime values in the past must be rejected without adding the widget to the database.

  • Lines 27-28: We will also use the Widget class to demonstrate how relationships between database tables are defined and managed. We have defined a foreign key relationship between this table and the site_user table. The owner of each widget will be the User that created it (The User.id attribute will be stored when each Widget is created).

  • Lines 33-41: Both of these hybrid properties convert the datetime value stored in the database to the timezone of the machine executing the code and formats the datetime as a string value.

  • Lines 43-45: deadline_passed is a bool value, this has been included as part of the Widget model solely to increase the number of data types that are serialized when Widget objects are included in an HTTP response. This attribute should return True if the curent date is greater than the date stored in deadline, and should return False if the current date is less than or the same as the date stored in deadline.

  • Lines 47-50: time_remaining is a timedelta value that represents the time remaining until the deadline is passed. If the curent date is greater than the date stored in deadline, then this attribute should return timedelta(0).

  • Lines 52-55: time_remaining_str converts the timedelta object returned by time_remaining to a formatted string if the deadline has not been passed. If the deadline has passed, "No time remaining" is returned.

  • Lines 57-59: find_by_name is a class method just like the find_by_email and find_by_public_id methods we previously created in the User class. Since the name attribute must be unique, we can use it to retrieve a specific Widget.

Next, we need to update run.py in order for the Flask-Migrate extension to recognize the Widget class and create a migration script that adds the new table to the database (this is the same process we previously performed for the User class in Part 2 and for the BlacklistedToken class in Part 4).

Open run.py in the project root folder and update the import statements to include the Widget class (Line 9). Then add the Widget class to the dict object that is returned by the make_shell_context function (Line 16):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
"""Flask CLI/Application entry point."""
import os

import click

from flask_api_tutorial import create_app, db
from flask_api_tutorial.models.token_blacklist import BlacklistedToken
from flask_api_tutorial.models.user import User
from flask_api_tutorial.models.widget import Widget

app = create_app(os.getenv("FLASK_ENV", "development"))


@app.shell_context_processor
def shell():
    return {
        "db": db,
        "User": User,
        "BlacklistedToken": BlacklistedToken,
        "Widget": Widget,
    }

Next, run flask db migrate and add a message explaining the changes that will be made by executing this migration script:

(flask-api-tutorial) flask-api-tutorial $ flask db migrate --message "add widget model"
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added table 'widget'
  Generating /Users/aaronluna/Projects/flask-api-tutorial/migrations/versions/7c66df0e878f_add_widget_model.py ...  done

You can verify that the widget table was detected by the Flask Migrate extension from the output of the flask db migrate command. You should see a message similar to the example above (Detected added table 'widget'), followed by a statement indicating the migration script was successfully generated.

Next, run flask db upgrade to run the migration script on the database in your development environment:

(flask-api-tutorial) flask-api-tutorial $ flask db upgrade
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 079d26d45cc9 -> 7c66df0e878f, add widget model

After the widget table has been added to the database, we can begin implementing the API endpoints specified in Table 1.

Create Widget

So where should we begin? In my opinion, the endpoint that should be implemented first is the endpoint responsible for the create operation, since without Widget objects there’s nothing to be retrieved, updated or deleted. In Part 3 we followed the process below for each endpoint in the auth_ns namespace. We will follow the same process to implement the widget_ns endpoints in Table 1:

  1. Create request parsers/API models to validate request data and serialize response data.
  2. Define the business logic necessary to process the request if validation succeeds.
  3. Create a class that inherits from Resource and bind it to the API endpoint/URL route.
  4. Define the set of HTTP methods that the API endpoint will support and expose methods on the concrete Resource class for each. Methods named get, post, put, delete, patch, options or head will be called when the API endpoint receives a request of the same HTTP method type.

    If the API endpoint does not support the HTTP method, do not expose a method with the name of the HTTP method and the client will receive a response with status code 405 HTTPStatus.METHOD_NOT_ALLOWED

  5. Document the Resource class and methods which handle HTTP requests as explained in the Flask-RESTx docs. Most of the content on the Swagger UI page is generated by decorating the concrete Resource classes created in Step 3 and the handler methods in Step 4.
  6. Import the business logic functions created in Step 2 and call them within the HTTP method that corresponds to the operation performed by the business logic.
  7. Create unit tests to verify that the input validation provided by the request parsers/API models is working correctly, and verify the endpoint behaves as expected.

Step 1 says create request parsers/API models to validate request data and serialize response data. So let’s dive into it!

create_widget_reqparser Request Parser

When a client sends a request to create a new Widget, what data is required? Take a look at the attributes of the Widget class:

22
23
24
25
26
27
28
29
id = db.Column(db.Integer, primary_key=True, autoincrement=True)
name = db.Column(db.String(100), unique=True, nullable=False)
info_url = db.Column(db.String(255))
created_at = db.Column(db.DateTime, default=utc_now)
deadline = db.Column(db.DateTime)

owner_id = db.Column(db.Integer, db.ForeignKey("site_user.id"), nullable=False)
owner = db.relationship("User", backref=db.backref("widgets"))

The client must provide values for the three attributes highlighted above (name, info_url and deadline). What about the other attributes?

  • id: This is the database table's primary key. The value is automatically set when a Widget is committed to the database (starting at one, then incrementing by one each time a Widget is added).

  • created_at: When a Widget object is created, the expression specified in the default parameter is evaluated and stored as the value for created_at. utc_now returns the current date and time as a datetime value that is timezone-aware and localized to UTC.

  • owner_id: This value is a foreign-key, which is indicated by db.ForeignKey("site_user.id"). site_user is the name of the database table where User objects are stored, and site_user.id is the primary-key that the owner_id column is referencing.

  • owner: It is important to note that owner is NOT an instance of db.Column. This means that unlike the other Widget class attributes, owner is not a column that exists in the widget database table. Instead, this is a relationship object that demonstates one of the main features of the SQLAlchemy ORM (Click here for more information).

    When processing a request to create a new Widget, the business logic will set the value of the owner_id attribute to the id of the User who sent the request. After the Widget is created and committed to the database, the owner attribute will contain a User object that represents the User who created it.

    Another interesting feature is achieved by backref=db.backref("widgets"). This creates a new attribute on all User objects named widgets (without modifying the User class at all), which is a list of all Widget objects in the database where User.id is equal to owner_id.

Flask-RESTx includes helpful pre-defined types (e.g., email, URL, etc.) in the inputs module for validating request data. When we created the auth_reqparser in Part 3, we imported the email class from flask_restx.inputs to verify if a value provided by the client is a valid email address. You can also define custom input types if none of the pre-defined types are sufficient, and we will do so for both the name and deadline attributes.

Create a new file named dto.py in src/flask_api_tutorial/api/widgets and enter the content below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
"""Parsers and serializers for /widgets API endpoints."""
import re
from datetime import date, datetime, time, timezone

from dateutil import parser
from flask_restx.inputs import URL
from flask_restx.reqparse import RequestParser

from flask_api_tutorial.util.datetime_util import make_tzaware, DATE_MONTH_NAME


def widget_name(name):
    """Validation method for a string containing only letters, numbers, '-' and '_'."""
    if not re.compile(r"^[\w-]+$").match(name):
        raise ValueError(
            f"'{name}' contains one or more invalid characters. Widget name must "
            "contain only letters, numbers, hyphen and underscore characters."
        )
    return name


def future_date_from_string(date_str):
    """Validation method for a date in the future, formatted as a string."""
    try:
        parsed_date = parser.parse(date_str)
    except ValueError:
        raise ValueError(
            f"Failed to parse '{date_str}' as a valid date. You can use any format "
            "recognized by dateutil.parser. For example, all of the strings below "
            "are valid ways to represent the same date: '2018-5-13' -or- '05/13/2018' "
            "-or- 'May 13 2018'."
        )

    if parsed_date.date() < date.today():
        raise ValueError(
            f"Successfully parsed {date_str} as "
            f"{parsed_date.strftime(DATE_MONTH_NAME)}. However, this value must be a "
            f"date in the future and {parsed_date.strftime(DATE_MONTH_NAME)} is BEFORE "
            f"{datetime.now().strftime(DATE_MONTH_NAME)}"
        )
    deadline = datetime.combine(parsed_date.date(), time.max)
    deadline_utc = make_tzaware(deadline, use_tz=timezone.utc)
    return deadline_utc


create_widget_reqparser = RequestParser(bundle_errors=True)
create_widget_reqparser.add_argument(
    "name",
    type=widget_name,
    location="form",
    required=True,
    nullable=False,
    case_sensitive=True,
)
create_widget_reqparser.add_argument(
    "info_url",
    type=URL(schemes=["http", "https"]),
    location="form",
    required=True,
    nullable=False,
)
create_widget_reqparser.add_argument(
    "deadline",
    type=future_date_from_string,
    location="form",
    required=True,
    nullable=False,
)

Let’s break this down by looking at how each of the attributes are validated by the code above:

name Argument

None of the pre-defined types in the flask_restx.inputs module perform input validation that satisfies the requirements of the name attribute. Thankfully, Flask-RESTx provides a way to create custom types that can be used in the same way. The example of a custom type shown below is taken from the documentation:

def my_type(value):
    '''Parse my type'''
    if not condition:
        raise ValueError('This is not my type')
    return parse(value)

The term “type” is (IMO) misleading since we only need to create a function (not a class). The function must accept at least one parameter — the value provided by the client. If the value is successfully validated, the function must convert the value to the expected data type before returning it (in the case of the name attribute, the expected type is a string so no conversion is necessary). If the value provided by the client is invalid, the function must raise a ValueError.

The widget_name function is adapted directly from the example shown above to satisfy the project requirements:

12
13
14
15
16
17
18
19
def widget_name(name):
    """Validation method for a string containing only letters, numbers, '-' and '_'."""
    if not re.compile(r"^[\w-]+$").match(name):
        raise ValueError(
            f"'{name}' contains one or more invalid characters. Widget name must "
            "contain only letters, numbers, hyphen and underscore characters."
        )
    return name
  • Line 14: The simplist way to implement our custom type is with a regular expression. The regex ^[\w-]+$ will match any string that consists of ONLY alphanumeric characters (which includes the underscore character) and the hyphen character.

    The syntax of regular expressions is extremely dense. To make any regex easier to understand, we could compile it with the re.VERBOSE flag. This causes whitespace that is not within a character class to be ignored, allowing us to place comments within the regex to document the design and the effect of each component of the expression. For example, we could document our regex as shown below (I am only showing this for demonstration purposes, and have not modified the code in src/flask_api_tutorial/api/widgets/dto.py):

    NAME_REGEX = re.compile(r"""
        ^        # Matches the beginning of the string
        [\w-]    # Character class: \w matches all alphanumeric characters (including underscore), - matches the hyphen character
        +        # Match one or more instances of the preceding character class
        $        # Matches the end of the string
    """, re.VERBOSE)

    Teaching regular expressions is beyond the scope of this tutorial. However, if you are looking for a good introduction to the topic I recommend reading Regular Expression HOWTO from the official Python docs.

  • Line 15: If the value does not match the regex, a ValueError is raised with a message explaining why the value is not a valid widget_name.

  • Line 19: If the value passed to this function matches the regex, the value is a valid widget_name and is returned.

We can see how this function works by testing it in the flask shell:

(flask-api-tutorial) flask-api-tutorial $ flask shell
Python 3.7.6 (default, Jan 19 2020, 06:08:58)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
App: flask-api-tutorial [development]
Instance: /Users/aaronluna/Projects/flask_api_tutorial/instance
>>> from flask_api_tutorial.api.widgets.dto import widget_name
# 1. 'test' is a valid widget_name
>>> widget_name("test")
'test'
# 2. 'test_1-AZ' is a valid widget_name
>>> widget_name("test_1-AZ")
'test_1-AZ'
# 3. 'some widget' is NOT a valid widget_name
>>> widget_name("some widget")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/aaronluna/Projects/flask_api_tutorial/src/flask_api_tutorial/api/widgets/dto.py", line 18, in widget_name
    f"'{name}' contains one or more invalid characters. Widget name must contain "
ValueError: 'some widget' contains one or more invalid characters. Widget name must contain only letters, numbers, hyphen and/or underscore characters.
# 4. 't**&*&' is NOT a valid widget_name
>>> widget_name("t**&*&")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/aaronluna/Projects/flask_api_tutorial/src/flask_api_tutorial/api/widgets/dto.py", line 18, in widget_name
    f"'{name}' contains one or more invalid characters. Widget name must contain "
ValueError: 't**&*&' contains one or more invalid characters. Widget name must contain only letters, numbers, hyphen and/or underscore characters.

The first test passes since test consists of only letters. The second test passes because it contains one of each of the allowed character types (letter, number, hyphen, underscore) and no other characters. The third and fourth tests fail because they both contain one or more forbidden characters (space, asterisk, ampersand).

Wait, let’s back up. Didn’t the requirement for the name attribute say that only lowercase letters were allowed? Yep, you got me. I kinda sort-of lied about test_1-AZ being a valid widget_name. But there is a reason why I did this, which will be revealed by the configuration of the argument object for the name attribute:

47
48
49
50
51
52
53
54
create_widget_reqparser.add_argument(
    "name",
    type=widget_name,
    location="form",
    required=True,
    nullable=False,
    case_sensitive=True,
)
  • Line 49: All we need to do to use our custom type is set the value of the type parameter to the widget_name function.

  • Lines 50-52: The location, required and nullable parameters should be familiar since we explained their purpose in Part 3.

  • Line 53: This is the first time we are using the case_sensitive parameter. According to the documentation, by default, case_sensitive=True. If this value is False "this will convert all values to lowercase".

Configuring the name argument with type=widget_name and case_sensitive=False allows the client to provide a value for the widget name in any combination of upper and lowercase letters. When an HTTP request is received to create a widget, if the request contains a name attribute, the value of that attribute will be passed to the widget_name function.

If the widget name contains only valid characters, it will be converted to lowercase before being passed to the business logic. The business logic will query the database for widgets with the same name. If a widget is found with the same name, the request will be aborted and the widget will not be created. If no widget is found with the same name, a new widget object will be created and added to the database.

info_url Argument

55
56
57
58
59
60
61
create_widget_reqparser.add_argument(
    "info_url",
    type=URL(schemes=["http", "https"]),
    location="form",
    required=True,
    nullable=False,
)

The info_url attribute can be parsed using the pre-defined URL type from the inputs module, which can be configured in numerous ways. For example, it is possible to restrict the allowed values to a whitelist of domains and/or exclude a blacklist of domains. You can choose to perform a DNS lookup on the domain specified by the client and reject the value if the domain fails to resolve. Check out the documentation for the URL type for even more ways you can control the allowed URL range/format.

The only restriction on info_url that we will employ is that the URL scheme must be either http or https. This can be more useful than you may think. Imagine if the Widget class had git_url and ssh_url attributes in addition to info_url. By simply changing the value of the schemes parameter from schemes=['http', 'https'] to schemes=['git'] and schemes=['ssh'], we could easily and effectively prevent clients from inserting values into the database that would cause errors if an application blindly tried to use those URLs to access a git repository or login to a server.

The value of the schemes parameter must always be a list or tuple, even if you intend to limit the allowed URL values to a single scheme.

There really isn’t anything else to say about how the info_url attribute is parsed since we already covered using a pre-defined input type when we used the email type as part of the auth_reqparser in Part 3. So let’s move on to something a bit more interesting.

deadline Argument

deadline is the last piece of data that we must receive from the client in order to create a new widget. Utlimately, this value must be converted to a datetime since the Widget class has several hybrid_properties that perform comparisons or calculations that assume this is the case. The inputs module provides several pre-defined types that can convert request data to either a date or a datetime value. I’ve summarized the requirements for these types in Table 2 below:

Table 2
Pre-defined Input Types for date and datetime Values
Pre-defined Type 1Required Format 2ExampleReference
dateYYYY-MM-DD2019-10-02N/A
date_from_iso8601YYYY-MM-DD2019-10-02ISO 8601
datetime_from_iso8601YYYY-MM-DDThh:mm:ss(+/-)zh:zm2019-10-02T15:05:06-07:00ISO 8601
datetime_from_rfc822DN, DD MN YYYY hh:mm:ss (+/-)zhzmWed, 02 Oct 2019 15:05:06 -0700RFC 5322 3

It would absolutely be possible to satisfy the project requirements using the pre-defined types in Table 2, but (IMO) they all have one big limitation — there is only a single valid format for the value provided by the client. Dates are expressed in so many different ways via text, so my preference is to design a parser that can accommodate a variety of date formats.

There is also a problem that arises from the project requirements for the deadline attribute: the value must not be a date in the past. Since the pre-defined types only validate that the value provided by the client is in the proper format to be converted to a datetime value, we would have to perform the task of checking if the date provided by the client is in the past in the business logic.

This second issue is another matter of opinion, since you could easily object by pointing out that the name attribute has a requirement to be unique and the task of querying the database for widgets with the same name is not performed in the wiget_name function we just created.

I’ll explain the distinction between these two requirements with a hypothetical situation: A request is received to create a new widget with name="test". This is a valid widget name, so the request is passed to the business logic and since no widget already exists in the database with name="test", a new widget is created. Then, an identical request is received to create a widget with name="test". IMO, from the point-of-view of the create_widget_reqparser, test is a valid widget name, so the request is again passed to the business logic. However, since a widget already exists with name="test", the request is aborted with a message indicating that a widget already exists in the database with name="test".

OTOH, consider this scenario: A request is received to create a new widget with deadline="1923-03-28". This string is in a valid format but since the date is obviously in the past, the create_widget_reqparser raises a ValueError and the request is aborted. A deadline in the past is always invalid and a request to create a widget containing an invalid value for a required parameter must always be rejected. Hopefully my reasoning makes sense to you.

dateutil.parser

Since the pre-defined types are adequate but inflexible, what can we use to parse a date value from a string? If we wanted to restrict ourselves to the Python standard library, we would need to create a complicated function that uses the strptime method of either the date or datetime class. This would involve testing as many format strings as possible and would quickly become a nightmare.

Luckily, I see no reason to impose such a restriction for this project. IMO, the most robust and usable way to parse datetime values from a string is the parse function in the dateutil.parser module (this was listed as a requirement in the setup.py file, so it is already installed). Unlike strptime, this method does not require you to provide a format string.

dateutil.parser.parse recognizes a wide variety of date and datetime formats. The locale setting of the machine where the function is executed is taken into consideration, which is extremely helpful in situations where the order of the month and day values are transposed (e.g., US date format vs. European). You can find more information in the official documentation for the dateutil.parser module.

future_date_from_string

Since deadline must be either the current date or a date in the future, I decided to name the custom type function for this attribute future_date_from_string. This function is more complex than widget_name since two separate validations must be peformed — parsing the input string to a date value and checking that the parsed date is not in the past:

22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
def future_date_from_string(date_str):
    """Validation method for a date in the future, formatted as a string."""
    try:
        parsed_date = parser.parse(date_str)
    except ValueError:
        raise ValueError(
            f"Failed to parse '{date_str}' as a valid date. You can use any format "
            "recognized by dateutil.parser. For example, all of the strings below "
            "are valid ways to represent the same date: '2018-5-13' -or- '05/13/2018' "
            "-or- 'May 13 2018'."
        )

    if parsed_date.date() < date.today():
        raise ValueError(
            f"Successfully parsed {date_str} as "
            f"{parsed_date.strftime(DATE_MONTH_NAME)}. However, this value must be a "
            f"date in the future and {parsed_date.strftime(DATE_MONTH_NAME)} is BEFORE "
            f"{datetime.now().strftime(DATE_MONTH_NAME)}"
        )
    deadline = datetime.combine(parsed_date.date(), time.max)
    deadline_utc = make_tzaware(deadline, use_tz=timezone.utc)
    return deadline_utc

There are a few things about the future_date_from_string function that are worth pointing out:

  • Line 25: The value provided by the user is passed to the dateutil.parser.parse method. If the parser is able to convert the string value to a datetime value it is stored in the parsed_date variable.

  • Lines 26-32: If the dateutil.parser.parse method is unable to parse the value provided by the user, it raises a ValueError. Since the error message that is generated isn't very descriptive, the original error is suppressed and another value error is raised with an error message that clearly explains why the value provided by the user was rejected and provides examples of string values that would be considered valid.

  • Line 34: Per the project requirements, the date value that was provided by the user is compared to the current date, without considering the time component of either date when making the comparison. If the date provided by the user is less than (i.e., before) the current date, the value is rejected since this means the deadline has passed.

  • Lines 35-40: If the deadline has passed, raise a ValueError explaining why the value is invalid.

  • Line 41: Per the project requirements, the deadline value flips from "not passed" to "passed" at the stroke of midnight. To achieve this, we set the deadline by taking the date component of the value provided by the user and combining it with time.max(), which is a shortcut for a time value equal to 11:59:59 PM.

  • Lines 42-43: Since we need to ensure that any datetime value is timezone-aware before we add it to the database, we localize the value to UTC before it is returned and passed to the business logic.

We already know how to use a custom type, so the last thing we need to do is add an argument to our create_widget_reqparser with type=future_date_from_string;

62
63
64
65
66
67
68
create_widget_reqparser.add_argument(
    "deadline",
    type=future_date_from_string,
    location="form",
    required=True,
    nullable=False,
)

Remember, we are following the process defined earlier to create each endpoint. We can now move on to step #2: Define the business logic necessary to process the request if validation succeeds.

create_widget Method

Next, we need to define a function that performs the following actions:

  • Add a new widget to the database
  • Associate the widget with the user who created it
  • Construct an HTTP response including a “Location” field in the header equal to the URI of the widget

RFC7231 defines the semantics of HTTP/1.1 messages. Specifically, it specifies when the “Location” field should be included in the header of an HTTP response:

7.1.2. Location

The "Location" header field is used in some responses to refer to a specific resource in relation to the response. The type of relationship is defined by the combination of request method and status code semantics.

Location = URI-reference

...

For 201 (Created) responses, the Location value refers to the primary resource created by the request.

Create a new file named business.py in src/flask_api_tutorial/api/widgets and enter the content below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
"""Business logic for /widgets API endpoints."""
from http import HTTPStatus

from flask import jsonify, url_for
from flask_restx import abort

from flask_api_tutorial import db
from flask_api_tutorial.api.auth.decorators import admin_token_required
from flask_api_tutorial.models.user import User
from flask_api_tutorial.models.widget import Widget


@admin_token_required
def create_widget(widget_dict):
    name = widget_dict["name"]
    if Widget.find_by_name(name):
        error = f"Widget name: {name} already exists, must be unique."
        abort(HTTPStatus.CONFLICT, error, status="fail")
    widget = Widget(**widget_dict)
    owner = User.find_by_public_id(create_widget.public_id)
    widget.owner_id = owner.id
    db.session.add(widget)
    db.session.commit()
    response = jsonify(status="success", message=f"New widget added: {name}.")
    response.status_code = HTTPStatus.CREATED
    response.headers["Location"] = url_for("api.widget", name=name)
    return response

Let’s take a look at how the create_widget function performs the tasks listed above:

  • Line 13: Per the specification in Table 1, the ability to create a widget object is limited to users with the administrator role. To enforce this, we decorate the create_widget method with @admin_token_required. If you would like to review how this decorator is implemented, click here.

  • Line 14: After the request data has been parsed and validated, it is passed to the create_widget function as a dict object named widget_dict.

  • Lines 15-18: The function starts by checking if a widget already exists with the same name provided in the request. If this is true, the request is aborted. The first argument to the abort function is the HTTP status code to include in the response. In this case, the appropriate response code is 409 HTTPStatus.CONFLICT.

  • Line 19: ** is the dictionary unpacking operator, you can find more info on it and the related list unpacking operator (*) in PEP 448. It is a concise way to pass the name, info_url, and deadline values to the Widget constructor.

  • Lines 20-21: Thanks to the @admin_token_required decorator, the public_id of the user who sent the request is stored in create_widget.public_id (If you don't remember why this is is the case, review the last section where we broke down how the decorators are designed).

    We retrieve the User object that corresponds to the public_id of the user that sent the request and assign it to owner. Then, owner.id is set as the value of widget.owner_id.

    Why can't we just use public_id as the value of widget.owner_id, instead of going through the process of retrieving the User object and using the id attribute? Since widget.owner_id is defined as a foreign key referencing site_user.id (the primary key of the site_user table), we must store the value of the id attribute in order to correctly configure the relationship between the two tables.

  • Lines 22-23: The new widget is added to the database and the changes are committed..

  • Line 24: As we have seen previously, if we need to include a custom field in the header of a response, we must create the response object manually. One way to do this is with the flask.jsonify function.

  • Line 25: Since the request to create a new widget was successful, the correct HTTP status code for the response is 201 HTTPStatus.CREATED.

  • Line 27: After ensuring that the response is configured with all required header fields, it is sent to the client.

The call to the url_for function in Line 26 relies on the api.widget endpoint being implemented. We haven't implemented either of the two endpoints defined in Table 1 at this point, so this will result in an unhandled exception until both have have been fully implemented.

Next, we need to create the API endpoint for the create operation and incorporate it with the create_widget_reqparser and the create_widget function.

api.widget_list Endpoint (POST Request)

According to Table 1, the operation to create a widget is accessed by sending a POST request to the api.widget_list resource, located at /api/v1/widgets. To create this endpoint, create a new file endpoints.py in src/flask_api_tutorial/api/widgets and enter the content below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
"""API endpoint definitions for /widgets namespace."""
from http import HTTPStatus

from flask_restx import Namespace, Resource

from flask_api_tutorial.api.widgets.dto import create_widget_reqparser
from flask_api_tutorial.api.widgets.business import create_widget

widget_ns = Namespace(name="widgets", validate=True)


@widget_ns.route("", endpoint="widget_list")
@widget_ns.response(int(HTTPStatus.BAD_REQUEST), "Validation error.")
@widget_ns.response(int(HTTPStatus.UNAUTHORIZED), "Unauthorized.")
@widget_ns.response(int(HTTPStatus.INTERNAL_SERVER_ERROR), "Internal server error.")
class WidgetList(Resource):
    """Handles HTTP requests to URL: /widgets."""

    @widget_ns.doc(security="Bearer")
    @widget_ns.response(int(HTTPStatus.CREATED), "Added new widget.")
    @widget_ns.response(int(HTTPStatus.FORBIDDEN), "Administrator token required.")
    @widget_ns.response(int(HTTPStatus.CONFLICT), "Widget name already exists.")
    @widget_ns.expect(create_widget_reqparser)
    def post(self):
        """Create a widget."""
        widget_dict = create_widget_reqparser.parse_args()
        return create_widget(widget_dict)

There’s nothing in the code above that we haven’t already encountered and explained while implementing the auth_ns API endpoints. Click here if you need a refresher on Namespace objects, Resource objects, @doc, @response, or @expect decorators, etc.

The important part is Lines 26-27 where the parse_args method of create_widget_reqparser is used to validate the request data, which is then passed to the create_widget function which we just defined.

You may have noticed that the @response decorator is sometimes applied to the WidgetList Resource and sometimes applied to the post method. Why would you do this instead of solely documenting one or the other? Flask-RESTx documentation decorators applied to a class cascade and apply to all methods within the class.

In this case, HTTPStatus.BAD_REQUEST, HTTPStatus.UNAUTHORIZED, and HTTPStatus.INTERNAL_SERVER_ERROR are valid responses to any HTTP request made to this endpoint. Placing the @response decorators on the Resource prevents duplicating them on each method.

Add widget_ns Namespace to api

In order to register the widget_ns namespace with the api object, open src/flask_api_tutorial/api/__init__.py and add the highlighted lines (Line 6 and Line 27):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
"""API blueprint configuration."""
from flask import Blueprint
from flask_restx import Api

from flask_api_tutorial.api.auth.endpoints import auth_ns
from flask_api_tutorial.api.widgets.endpoints import widget_ns


api_bp = Blueprint("api", __name__, url_prefix="/api/v1")
authorizations = {
  "Bearer": {
    "type": "apiKey",
    "in": "header",
    "name": "Authorization"
  }
}

api = Api(
    api_bp,
    version="1.0",
    title="Flask API with JWT-Based Authentication",
    description="Welcome to the Swagger UI documentation for the Widget API",
    doc="/ui",
    authorizations=authorizations,
)

api.add_namespace(auth_ns, path="/auth")
api.add_namespace(widget_ns, path="/widgets")

We can verify that the endpoint was created by running flask routes:

flask-api-tutorial $ flask routes
Endpoint             Methods  Rule
-------------------  -------  --------------------------
api.auth_login       POST     /api/v1/auth/login
api.auth_logout      POST     /api/v1/auth/logout
api.auth_register    POST     /api/v1/auth/register
api.auth_user        GET      /api/v1/auth/user
api.doc              GET      /api/v1/ui
api.root             GET      /api/v1/
api.specs            GET      /api/v1/swagger.json
api.widget_list      POST     /api/v1/widgets
restplus_doc.static  GET      /swaggerui/<path:filename>
static               GET      /static/<path:filename>

As expected, a new endpoint has been created named api.widget_list that responds to requests sent to /api/v1/widgets. Currently, this endpoint only supports requests where the method type is POST.

Normally, we would create unit tests to verify the endpoint does create widget objects correctly. However, it was noted while explaining the design of the create_widget function that currently an unhandled exception occurs when running this function since the business logic for creating a new Widget depends on the api.widget endpoint existing. We will create unit tests for both endpoints in the widget_ns namespace when both have been fully implemented.

Checkpoint

Even though we only implemented one of the five CRUD operations specified in Table 1, we actually satisfied several of the remaining requirements. However, since we have not created any test coverage and confirmed that the process of creating a widget object is working correctly, these will be marked as only half-complete at this point:

User Management/JWT Authentication

New users can register by providing an email address and password

Existing users can obtain a JWT by providing their email address and password

JWT contains the following claims: time the token was issued, time the token expires, a value that identifies the user, and a flag that indicates if the user has administrator access

JWT is sent in access_token field of HTTP response after successful authentication with email/password

JWTs must expire after 1 hour (in production)

JWT is sent by client in Authorization field of request header

Requests must be rejected if JWT has been modified

Requests must be rejected if JWT is expired

If user logs out, their JWT is immediately invalid/expired

If JWT is expired, user must re-authenticate with email/password to obtain a new JWT

API Resource: Widget List

All users can retrieve a list of all widgets

All users can retrieve individual widgets by name

Users with administrator access can add new widgets to the database

Users with administrator access can edit existing widgets

Users with administrator access can delete widgets from the database

The widget model contains attributes with URL, datetime, timedelta and bool data types, along with normal text fields.

URL and datetime values must be validated before a new widget is added to the database (and when an existing widget is updated).

The widget model contains a "name" attribute which must be a string value containing only lowercase-letters, numbers and the "-" (hyphen character) or "_" (underscore character).

The widget model contains a "deadline" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.

Widget name must be validated before a new widget is added to the database (and when an existing widget is updated).

If input validation fails either when adding a new widget or editing an existing widget, the API response must include error messages indicating the name(s) of the fields that failed validation.

In the next section, we will finish implementing the Widget API. If you have any questions/feedback, please leave a comment!