[{"categories":"Hugo ; Javascript","content":"Introduction If you DuckDuckGo or bing! the phrase \u0026ldquo;add search to static site\u0026rdquo;, you will find hundreds of articles explaining how a client-side search product like Lunr.js can be integrated with Hugo, Eleventy or Gatsby. Given this fact, I feel compelled to explain why I created this blog post when it appears that this question has been sufficiently answered.\nI decided to document my Hugo/Lunr.js solution because unlike other guides I found while researching this subject, my implementation:\nUses Vanilla JS and does not require jQuery for DOM manipulation/JSON fetching functions. Generates the JSON page data file file whenever the site is built by creating a simple Hugo template, NOT by npm scripts you have to manually run or integrate with a CI process.  Includes unique features to enhance the search UX:\n Lunr.js gives each search result a \"score\" indicating the quality of the search hit. I use this score to dynamically alter the hue of the search result link font color: \"warm\" colors are applied to higher quality results while \"cool\" colors are applied to lower quality results, producing a gradient effect on the list of search results. This makes the list easier to visually digest since the relative quality of each result is communicated in a simple, intuitive way. IMO, the text below the search result link containing text from the page should use an algorithm to display text that is directly related to the search term entered by the user. Most implementations I found simply truncate the page content to the first N characters, which is lazy and may not be pertinent to what the user is searching for. Rather than navigating to a new page/route when a search is performed, the main content div is hidden and the search results are \"unhidden\" (shown) in their place. Prominent Clear Search Results buttons are shown and when clicked, the search results are hidden and replaced by the main content div. Why is this better? It reduces a tiny amount of friction from the search process, making the experience like that of a Single Page App.    The items in the list above make my implementation notably different and (IMO) markedly better than the examples I found when I started looking into this topic. Hopefully this makes my decision to create this post in spite of the fact that this topic has been covered extensively easier to understand.\nWhat is a Client-Side Search Tool? How is it possible to add a search feature to a static site? Let\u0026rsquo;s back up and instead ask, how is it possible to add any type of dynamic/interactive behavior to a static site? In general, the answer will fall under one of two categories: serverless functions (e.g., AWS Lambda, Azure Functions, etc.) or client-side javascript.\nLunr.js falls under the latter category, which means that all of the work done to produce search results takes place in the client\u0026rsquo;s browser. How does the client get the data required for this task? Since we are not running a server where a database is available to provide the data, we must generate a file that acts like a database when the site is built and make it accessible from a route/URL exposed by our site.\nThe file we generate will be in JSON format, containing the data for all posts that need to be searchable (this file will be referred to as the JSON page data file for the remainder of this guide). When a user performs a search, this JSON file will be fetched and Lunr.js will figure out what posts, if any, match what the user searched for. Most of the code that is unique to my implementation is responsible for the next and final step: rendering the search results as HTML and presenting them to the user.\nDetermine Your Search Strategy Before you implement Lunr.js or a similar package that provides client-side search capabilities, it is extremely important to understand the strengths and weaknesses of such a product. Lunr.js is a perfect fit for my website, since the total number of pages that are searchable is very small (24 pages at the time this was written). I am by no means a prolific blogger, so over time this number should increase very slowly.\nHowever, if your site has thousands of pages that need to be searchable, a client-side search solution could be a very poor choice. You must closely monitor the size of the JSON page data file since it will be transferred to the client when the user performs a search.\nIf you include the entire content of each page in the JSON, the file can become very large, very quickly. This will have an obvious impact on the response time of the search request, making your site appear sluggish.\nThis can be mitigated by omitting the page content from the page data file and including a description of the page instead. This would address the performance issue but would greatly lessen the power of the search feature.\nIf you have reached this page and your site has a much greater amount of content than mine, I would recommend a product like Algolia rather than Lunr.js. Algolia is a paid product, but there is a free version and it is much better suited to large sites than Lunr.js.\n  At the very least, generate the page data file in order to understand the size of the JSON for your site. If the file is more than 2 MB, the search UX will be sluggish for clients with a high-speed connection and unusable for clients with slower data rates.\nIf the file is between 1-2 MB, you need to make a decision based on the quality of the internet service for an average user of your site. At this file size, high speed users should only experience a minor delay when performing a search, but if your average user only has access to 3G, a search could take a minute to execute!\nIdeally, you want the JSON page data file to be less than 500 KB, however a high-speed connection will easily handle 1 MB, but this cannot be necessarily assumed for all users.\n  Install Lunr.js Lunr.js can be installed locally using npm:\n$ npm install --save-dev lunr This is the method I use in my project. I bundle the lunr.js script with the other javascript files in use on my website and provide the bundle to the client in a single script tag. Of course, you could also have clients download it directly from the unpkg CDN with the script tag below:\n\u0026lt;script src=\u0026#34;https://unpkg.com/lunr/lunr.js\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; How to Integrate Lunr.js with Hugo If you are still with me, let\u0026rsquo;s get started! There are quite a few things that we must accomplish in order to fully integrate Lunr.js with a Hugo site:\nUpdate config.toml to generate a JSON version of the home page. Create an output format template for the home page that generates the JSON page data file. Write javascript to fetch the JSON page data file and build the Lunr.js search index. Create a partial template that allows the user to input and submit a search query. Write javascript to retrieve the value entered by the user and generate a list of search results, if any. Create a partial template that renders the search results. Write javascript to render the search results on the page.  Codepen with Final Code You can inspect the final HTML, CSS and JavaScript with the CodePen below. Please note that the search results will be slightly different from the search form on this page since the CodePen only includes (roughly) the first 500 words of the content from each page:\nSee the Pen Lunr.js Static Site Search Example by Aaron Luna (@a-luna) on CodePen.  I recommend changing the zoom from 1x to 0.5x if you choose to view the code alongside the Result.\nGenerate JSON Page Data File One of Hugo\u0026rsquo;s most useful (and powerful) features is the ability to generate content in any number of formats. By default, Hugo will render an HTML version of each page, as well as an XML version based on an the default internal RSS template.\nServe index.json from Site Root There are a few other built-in formats that are not enabled by default, including JSON. In your config.toml file, add \u0026quot;JSON\u0026quot; to the home entry in the [outputs] section.\n[outputs] home = [\u0026#34;HTML\u0026#34;, \u0026#34;RSS\u0026#34;, \u0026#34;JSON\u0026#34;] page = [\u0026#34;HTML\u0026#34;]   If your config.toml file does not already have a section called [outputs], you can copy and paste the entire code block above. By default, Hugo generates HTML and XML versions of each page, so generating a JSON version of the homepage is the only change that will be made by applying the code above.\n  This tells Hugo to generate the following files in the site root folder: index.html, index.xml, and index.json. If you save config.toml and attempt to build the site, you will be warned that Hugo is unable to locate a layout template for the JSON output format for the home page:\n$ hugo server -D Building sites … WARN 2020/05/29 14:20:17 found no layout file for \"JSON\" for kind \"home\": You should create a template file which matches Hugo Layouts Lookup Rules for this combination. Since the JSON output template cannot be found, you will receive a 404 response if you attempt to access the URL /index.json. Obviously, our next step is to create this template with the goal of generating the page data file.\nCreate JSON Output Template Create a file in the root of the layouts folder named index.json and add the content below:\n1 2 3 4 5 6 7 8 9 10  {{-$.Scratch.Add\u0026#34;pagesIndex\u0026#34;slice-}} {{-range$index,$page:=.Site.Pages-}} {{-ifeq$page.Type\u0026#34;post\u0026#34;-}}  {{-ifgt(len$page.Content)0-}} {{-$pageData:=(dict\u0026#34;title\u0026#34;$page.Title\u0026#34;href\u0026#34;$page.Permalink\u0026#34;categories\u0026#34;(delimit$page.Params.categories\u0026#34; ; \u0026#34;)\u0026#34;content\u0026#34;$page.Plain)-}} {{-$.Scratch.Add\u0026#34;pagesIndex\u0026#34;$pageData-}} {{-end-}} {{-end-}} {{-end-}} {{-$.Scratch.Get\u0026#34;pagesIndex\u0026#34;|jsonify-}}   Most of this should be self-explanatory. The only line that I am calling attention to is the if statement in Line 3. For my site, I have pages of types other than post that I need to make searchable, so my template includes these other page types with the code below:\n{{-ifin(slice\u0026#34;post\u0026#34;\u0026#34;flask_api_tutorial\u0026#34;\u0026#34;projects\u0026#34;)$page.Type-}} Hopefully this helps if you also need to include other page types in your site search, the changes you make will depend on the structure of your content folder.\nWith the JSON output template in place, rebuild the site. When complete, if you navigate to /index.json you should find a JSON list containing all of the page data for your site:\n[ { \u0026#34;categories\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;href\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;title\u0026#34;: \u0026#34;\u0026#34; }, ... ] You can verify that the JSON page data file for my site exists by clicking here.\n  I do not use the tags taxonomy on my site, preferring to use categories only. If you only use tags or use both and you want the list of tags to be searchable, the code required is nearly identical to the code shown for categories.\nIf you do not use categories and want the list of tags for each page to be searchable, replace \"categories\" (delimit $page.Params.categories \" ; \") with \"tags\" (delimit $page.Params.tags \" ; \") in the $pageData dict object.\nIf you use both tags and categories and want both to be searchable, simply add \"tags\" (delimit $page.Params.tags \" ; \") to the $pageData dict object.\n  Build Lunr.js Search Index After creating the JSON output template and updating config.toml to generate a JSON version of the home page, rebuild your site. Verify that this succeeds without the error we encountered earlier and also verify that index.json exists in the root of the public folder.\nIf the format of index.json is incorrect, make any necessary changes to the template file and rebuild your site. When you are satisfied with the content and format, create a new file named search.js.\nThere are multiple valid ways to structure your site resources, so place this file with the rest of your javascript files (in my project this location is static/js/search.js). Add the content below and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19  let pagesIndex, searchIndex; async function initSearchIndex() { try { const response = await fetch(\u0026#34;/index.json\u0026#34;); pagesIndex = await response.json(); searchIndex = lunr(function () { this.field(\u0026#34;title\u0026#34;); this.field(\u0026#34;categories\u0026#34;); this.field(\u0026#34;content\u0026#34;); this.ref(\u0026#34;href\u0026#34;); pagesIndex.forEach((page) =\u0026gt; this.add(page)); }); } catch (e) { console.log(e); } } initSearchIndex();     What exactly is a Lunr search index? We designed the JSON output template to generate a list of page objects (one object for every page that is searchable) with the following fields: href, title, categories and content. We tell Lunr which fields contain data to be searched, and specify one field as an identifier that is returned in the list of search results.\nLunr takes the list of search fields, the identifier and the list of page objects and builds an index. The process of building a search index is complicated and way over my head, you can read the official docs for all the nerdy details.\n  The Lunr search index is pretty important, so let\u0026rsquo;s explain how it is built from the index.json file:\n Line 1: search.js will have two global variables: pagesIndex will contain the JSON page data read in from index.json, and searchIndex will contain the Lunr.js search index. Lines 5-6: The client asynchronously fetches index.json, and stores the list of page data in the pagesIndex global variable. Line 7: The lunr function is a convenience method that creates a new search index from a list of documents. The search index itself is immutable, all documents to be searched must be provided at this time. It is not possible to add or remove documents after the search index has been created. Lines 8-10: Within the lunr function, the first thing we do is define specify all fields that need to be searchable. The set of possible fields are those that exist on the JSON objects stored in pagesIndex. We specify that three of the four fields available on our page objects (title, categories and content) contain text that should be available to search. Line 11: We specify that the remaining field, href, will be used as the identifier. When a search is performed, the list of results will contain the href value, and this value can be used with pagesIndex to retrieve the page title and content to construct the list of search results and render them as HTML. Line 12: Finally, each page object is added to the Lunr search index. This is the only time that pages can be added to the index, since the Lunr search index object is immutable.   The order of the tasks performed within the lunr function is actually important. According to the official docs, \u0026ldquo;all fields should be added before adding documents to the index. Adding fields after a document has been indexed will have no effect on already indexed documents.\u0026rdquo;\n   Line 19: The initSearchIndex function is called automatically when the page loads. Most likely, index.json will by downloaded only once (assuming the user's browser is configured to cache it), and all subsequent searches will provide results immediately.   A search of all documents is performed by calling searchIndex.search. The search method requires a single parameter, a query string to run against the search index. Our next task is to create a form for users to enter the query string and submit a search request.\nCreate Search Form HTML Template The search input form I designed for my site is intended to appear simple and minimalist, revealing additional functionality when a search is attempted. For example, when a search does not return any results or the enter key is pressed and the text box is empty, an error message fades in and out directly overlapping the input element as shown below:\nSorry, your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!  Search Input Form (Error Messages)\n   My implementation is tied directly to the base theme I started with. All of the items in the sidebar of my site are \u0026ldquo;widgets\u0026rdquo; which can be hidden/shown and reordered by adjusting values in config.toml. The base theme does not include a \u0026ldquo;search\u0026rdquo; widget, but the theme makes creating custom widgets simple and straightforward.\nI have removed class names that are related to the base theme/widget system from the HTML below to make the markup more readable and generic (You can compare it to the actual template in the github repo for this site). For the purposes of this guide, I assume that you will adapt my template to suit your needs or create a new layout:\n\u0026lt;div id=\u0026#34;site-search\u0026#34;\u0026gt; \u0026lt;h4\u0026gt;\u0026lt;span class=\u0026#34;underline--magical\u0026#34;\u0026gt;Search\u0026lt;/span\u0026gt;\u0026lt;/h4\u0026gt; \u0026lt;div class=\u0026#34;search-flex\u0026#34;\u0026gt; \u0026lt;form id=\u0026#34;search-form\u0026#34; name=\u0026#34;searchForm\u0026#34; role=\u0026#34;form\u0026#34; onsubmit=\u0026#34;return false\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;search-wrapper\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;search-container\u0026#34;\u0026gt; \u0026lt;i class=\u0026#34;fa fa-search\u0026#34;\u0026gt;\u0026lt;/i\u0026gt; \u0026lt;input autocomplete=\u0026#34;off\u0026#34; placeholder=\u0026#34;\u0026#34; id=\u0026#34;search\u0026#34; class=\u0026#34;search-input\u0026#34; aria-label=\u0026#34;Search site\u0026#34; type=\u0026#34;text\u0026#34; name=\u0026#34;q\u0026#34;\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;search-error hide-element\u0026#34;\u0026gt;\u0026lt;span class=\u0026#34;search-error-message\u0026#34;\u0026gt;\u0026lt;/span\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/form\u0026gt; \u0026lt;button id=\u0026#34;clear-search-results-sidebar\u0026#34; class=\u0026#34;button clear-search-results hide-element\u0026#34;\u0026gt;Clear Search Results\u0026lt;/button\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; There are two form elements which are hidden when the page is initially loaded, and this is indicated by the presence of hide-element in their class list. As we will see shortly, these elements will be shown when they are needed by removing this class and other elements will be hidden by adding hide-element to their class list (via javascript).\n  You may be wondering why the search input form is so highly nested. The fade in/out behavior for error messages shown above is one reason, since this requires absolute/relative positioning of stacked elements.\nThe purple border effect that occurs when the text box is in focus is achieved by wrapping the input element and magnifying glass icon with the \u0026lt;div class=search-container\u0026gt; element.\nAlso, while the search form may appear simple when finally rendered, it contains multiple flex-containers. Finally, in an attempt to abide by HTML5 recommendations, the widget includes a form element which further increases the levels of nesting.\n  The CSS for the search input form is rather lengthy, obviously you can ignore the coloring/font size details and adapt the style to your own needs:\n.hide-element { display: none; } .search-flex { display: flex; flex-flow: column nowrap; margin: 8px 0 3px 0; } .search-wrapper { position: relative; z-index: 0; height: 31px; } .search-container { position: absolute; z-index: 1; width: 100%; display: flex; flex-flow: row nowrap; align-items: center; justify-content: space-evenly; line-height: 1.3; color: hsl(0, 0%, 77%); cursor: pointer; background-color: hsl(0, 0%, 16%); border: 1px solid hsl(0, 0%, 27%); border-radius: 0.5em; } .search-container:hover { border-color: hsl(0, 0%, 40%); } .search-container.focused { border-color: hsl(261, 100%, 45%); box-shadow: 0 0 1px 2px hsl(261, 100%, 45%); } #site-search a { color: hsl(0, 0%, 85%); text-decoration: none; } #site-search h4 { position: relative; font-family: raleway, sans-serif; font-weight: 500; line-height: 1.2; font-size: 1.45em; color: #e2e2e2; padding: 0 0 5px 0; margin: 0; } .underline--magical { background-image: linear-gradient( 120deg, hsl(261, 100%, 45%) 0%, hsl(261, 100%, 45%) 100% ); background-repeat: no-repeat; background-size: 100% 0.2em; background-position: 0 88%; transition: background-size 0.25s ease-in; } #site-search.expanded #search-form { margin: 0 0 15px 0; } #search-form .fa { margin: 0 auto; font-size: 0.9em; } #search { flex: 1 0; max-width: calc(100% - 35px); font-size: 0.8em; color: hsl(0, 0%, 77%); background-color: transparent; font-family: roboto, sans-serif; box-sizing: border-box; -moz-appearance: none; -webkit-appearance: none; cursor: pointer; border: none; border-radius: 4px; padding: 6px 0; margin: 0 5px 0 0; } #search:hover { background-color: hsl(0, 0%, 16%); } #search:focus { border-color: hsl(0, 0%, 16%); box-shadow: none; color: hsl(0, 0%, 77%); outline: none; } #clear-search-results-mobile { display: none; } .clear-search-results { font-size: 0.9em; color: hsl(0, 0%, 85%); background-color: hsl(261, 100%, 45%); border-radius: 4px; line-height: 16px; padding: 7px 10px; box-sizing: border-box; text-align: center; width: 100%; min-width: 170px; transition: all 0.4s linear; box-shadow: 0 3px 3px rgba(0, 0, 0, 0.2), 0 3px 20px rgba(0, 0, 0, 0.3); cursor: pointer; border: none; } .clear-search-results:hover, .clear-search-results:active, .clear-search-results:focus, .clear-search-results:active:hover { color: hsl(0, 0%, 95%); background-color: hsl(261, 100%, 35%); } #search-form .search-error.hide-element { display: none; } .search-error { position: absolute; z-index: 2; top: 4px; right: 7px; font-size: 0.75em; background-color: hsl(0, 0%, 16%); height: 24px; display: flex; transition: all 0.5s ease-out; width: 95%; } .search-error-message { font-style: italic; color: #e6c300; text-align: center; margin: auto; line-height: 1; transition: all 0.5s ease-out; } .fade { -webkit-animation: fade 4s; animation: fade 4s; -moz-animation: fade 4s; -o-animation: fade 4s; } @-webkit-keyframes fade { 0% { opacity: 0.2; } 50% { opacity: 1; } 100% { opacity: 0.2; } } @-moz-keyframes fade { 0% { opacity: 0.2; } 50% { opacity: 1; } 100% { opacity: 0; } } @keyframes fade { 0% { opacity: 0.2; } 50% { opacity: 1; } 100% { opacity: 0.2; } } @-o-keyframes fade { 0% { opacity: 0.2; } 50% { opacity: 1; } 100% { opacity: 0.2; } } Open search.js and add the code highlighted below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36  let pagesIndex, searchIndex; async function initSearchIndex() { try { const response = await fetch(\u0026#34;/index.json\u0026#34;); pagesIndex = await response.json(); searchIndex = lunr(function () { this.field(\u0026#34;title\u0026#34;); this.field(\u0026#34;categories\u0026#34;); this.field(\u0026#34;content\u0026#34;); this.ref(\u0026#34;href\u0026#34;); pagesIndex.forEach((page) =\u0026gt; this.add(page)); }); } catch (e) { console.log(e); } } function searchBoxFocused() { document.querySelector(\u0026#34;.search-container\u0026#34;).classList.add(\u0026#34;focused\u0026#34;); document .getElementById(\u0026#34;search\u0026#34;) .addEventListener(\u0026#34;focusout\u0026#34;, () =\u0026gt; searchBoxFocusOut()); } function searchBoxFocusOut() { document.querySelector(\u0026#34;.search-container\u0026#34;).classList.remove(\u0026#34;focused\u0026#34;); }  initSearchIndex(); document.addEventListener(\u0026#34;DOMContentLoaded\u0026#34;, function () { if (document.getElementById(\u0026#34;search-form\u0026#34;) != null) { const searchInput = document.getElementById(\u0026#34;search\u0026#34;); searchInput.addEventListener(\u0026#34;focus\u0026#34;, () =\u0026gt; searchBoxFocused()); } })   The code that was just added does not perform any search functionality, it simply adds and removes the purple border around the search box when it is focused/becomes out of focus.\nHandle User Search Request Now that we have our search input form, we need to retrieve the search query entered by the user. Since I did not create a \u0026ldquo;Submit\u0026rdquo; button for the search form, the user will submit their query by pressing the Enter key. I also added a click event handler to the magnifying glass icon, to provide another way to submit a search query:\nOpen search.js and add the lines highlighted below:\n30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45  initSearchIndex(); document.addEventListener(\u0026#34;DOMContentLoaded\u0026#34;, function () { if (document.getElementById(\u0026#34;search-form\u0026#34;) != null) { const searchInput = document.getElementById(\u0026#34;search\u0026#34;); searchInput.addEventListener(\u0026#34;focus\u0026#34;, () =\u0026gt; searchBoxFocused()); searchInput.addEventListener(\u0026#34;keydown\u0026#34;, (event) =\u0026gt; { if (event.keyCode == 13) handleSearchQuery(event) }); document .querySelector(\u0026#34;.search-error\u0026#34;) .addEventListener(\u0026#34;animationend\u0026#34;, removeAnimation); document .querySelector(\u0026#34;.fa-search\u0026#34;) .addEventListener(\u0026#34;click\u0026#34;, (event) =\u0026gt; handleSearchQuery(event));  } })   The code above creates three new event handlers:\nCall handleSearchQuery(event) when the Enter key is pressed and the search input text box is in focus. Call removeAnimation when the animationend event occurs on the \u0026lt;div class=\"search-error\"\u0026gt; element. Call handleSearchQuery(event) when the user clicks on the magnifying glass icon.  We need to create the functions that are called by these event handlers, add the content below to search.js and save the file:\n30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87  function handleSearchQuery(event) { event.preventDefault(); const query = document.getElementById(\u0026#34;search\u0026#34;).value.trim().toLowerCase(); if (!query) { displayErrorMessage(\u0026#34;Please enter a search term\u0026#34;); return; } const results = searchSite(query) if (!results.length) { displayErrorMessage(\u0026#34;Your search returned no results\u0026#34;) return } } function displayErrorMessage(message) { document.querySelector(\u0026#34;.search-error-message\u0026#34;).innerHTML = message; document.querySelector(\u0026#34;.search-container\u0026#34;).classList.remove(\u0026#34;focused\u0026#34;); document.querySelector(\u0026#34;.search-error\u0026#34;).classList.remove(\u0026#34;hide-element\u0026#34;); document.querySelector(\u0026#34;.search-error\u0026#34;).classList.add(\u0026#34;fade\u0026#34;); } function removeAnimation() { this.classList.remove(\u0026#34;fade\u0026#34;); this.classList.add(\u0026#34;hide-element\u0026#34;); document.querySelector(\u0026#34;.search-container\u0026#34;).classList.add(\u0026#34;focused\u0026#34;); } function searchSite(query) { const originalQuery = query; query = getLunrSearchQuery(query); let results = getSearchResults(query); return results.length ? results : query !== originalQuery ? getSearchResults(originalQuery) : []; } function getSearchResults(query) { return searchIndex.search(query).flatMap((hit) =\u0026gt; { if (hit.ref == \u0026#34;undefined\u0026#34;) return []; let pageMatch = pagesIndex.filter((page) =\u0026gt; page.href === hit.ref)[0]; pageMatch.score = hit.score; return [pageMatch]; }); } function getLunrSearchQuery(query) { const searchTerms = query.split(\u0026#34; \u0026#34;); if (searchTerms.length === 1) { return query; } query = \u0026#34;\u0026#34;; for (const term of searchTerms) { query += `+${term}`; } return query.trim(); }   There is a lot of information contained within the code above, so please bear with me and read the following explanations:\n Line 31: When the Enter key is pressed and the focus is on a form element, the default behavior is to submit the form, which causes the page to reload. Calling event.preventDefault prevents this and stops the event from propgating further. Line 32: When the query entered by the user is retrieved, it is converted to lowercase and any leading and trailing whitespace is removed. Lines 33-36: If nothing has been entered into the search text box, we display an error message to the user (Please enter a search term). Line 37: Since we have verified that an actual value has been entered, we call searchSite which returns results. Lines 38-41: If the total number of results is zero, we display an error message to the user (Your search returned no results). Lines 44-55: These two functions are responsible for fading in and fading out the error message directly on top of the search text box. Line 58: We store a copy of the search query entered by the user in originalQuery.  Lines 59, 77-87: Before generating the list of search results by calling searchIndex.search, the query string is modified by calling getLunrSearchQuery. In order to understand the purpose of this function, you must understand how the searchIndex.search method uses the query string.\nBefore conducting the search, the query string is parsed to a search query object. Lunr.js provides a special syntax for query strings, allowing the user to isolate search terms to individual fields, use wildcard characters, perform fuzzy matching, etc.\nThe default behavior when a search is performed with a query string that contains more than one word is explained below (from the official docs):\nBy default, Lunr combines multiple terms together in a search with a logical OR. That is, a search for “foo bar” will match documents that contain “foo” or contain “bar” or contain both. ... By default each term is optional in a matching document, though a document must have at least one matching term. IMO, requiring at least one search term to be found in a document rather than requiring all search terms to be found tends to produce low quality search results. For this reason, the getLunrSearchQuery function modifies the query string in the following manner:\nCall query.split(\" \") to produce a list of search terms. If the query only contains a single search term, it is returned unmodified. Otherwise, each search term is prefixed with the + character, which specifies that the term must be present in the document to be considered a match. The modified search term is concatenated to produce a modified query string. After all search terms have had a + prepended, trailing whitespace is trimmed from the modified query string. The modified query string is returned.  For example, if the user entered vanilla javascript, the getLunrSearchQuery function would produce +vanilla +javascript. How does modifying the query string in this way change the search behavior? Again, from the Lunr.js docs:\nTo indicate that a term must be present in matching documents the term should be prefixed with a plus (+) In this example, the effect of the getLunrSearchQuery function changes the search behavior from documents containing vanilla OR javascript to documents containing vanilla AND javascript.\n Lines 60-65: If the modified search query does not produce any results, we will re-try with the original version of the search query, since this will return results where ANY search term is present. Please note however, that query and originalQuery will be the same value if the user entered a single-word search query. In this case, we do not need to re-try the search and an empty list is returned.  Line 69: If we have generated the JSON page data file and built the Lunr.js search index correctly, calling searchIndex.search with a query string will produce a list of search results.\nThe search results as returned from searchIndex.search only contain the href field from the JSON page data file and a score value that indicates the quality of the search result. Since we will use the title and page content to construct the list of search results, we need to update the search results to include this information.\nI am using the Array.flatMap method to produce a list of search results containing the data from the JSON page data file and the search result score. I learned about this usage of flatMap from MDN, click here if you are interested in the reason why this produces the list of search results.\n Line 70: If the ref property of the Lunr.js search result object is undefined, we exclude it from the list of search results. Line 71: For each hit, we retrieve the matching page object from pagesIndex by filtering with the condition hit.ref == page.href. There can only be a single page that matches this condition, so we grab the first element returned by the filter method. Line 72: We update the page object returned from the filter method to include the score value of the search result.   I couldn\u0026rsquo;t find anything in the Lunr.js docs that describe exact score ranges and how those ranges correlate to high-quality, average-quality and low-quality search results. However, based on the results I receive from my site, anything above 3.0 is a high-quality result, 2-1.5 appears average, and less than 1.5 seems to be low-quality. This is just my completely amateurish approximation, so take it with a substantial grain of salt.\n    On my site, if I place a breakpoint at Line 38, submit the query \u0026ldquo;search\u0026rdquo; and debug this code, the list of search results that are returned from the searchSite function can be inspected from the console (obviously, the values for content have been truncated since they contain the entire page content):\n\u0026raquo; results \u0026#8594; (7) [{…}, {…}, {…}, {…}, {…}, {…}, {…}] 0: categories: \"Hugo ; Javascript\" content: \"Introduction If you DuckDuckGo or bing! the phrase\" href: \"http://172.20.10.2:1313/blog/add-search-to-static-site-lunrjs-hugo-vanillajs/\" score: 3.6189999999999998 title: \"Add Search to Your Static Site with Lunr.js (Hugo, Vanilla JS)\" __proto__: Object 1: categories: \"Virtualization\" content: \"The majority of my work consists of C#/.NET develo\" href: \"http://172.20.10.2:1313/blog/optimize-vm-performance-windows10-visual-studio-vmware-fusion/\" score: 1.985 title: \"How to Optimize Performance of Windows 10 and Visual Studio in VMWare Fusion\" __proto__: Object 2: categories: \"DevOps\" content: \"If you create Heroku apps, you know that the only\" href: \"http://172.20.10.2:1313/blog/continuously-deploy-heroku-azure-devops/\" score: 1.577 title: \"How to Continuously Deploy a Heroku App with Azure DevOps\" __proto__: Object 3: categories: \"Hugo ; JavaScript\" content: \"Hugo includes a built-in syntax-highlighter called\" href: \"http://172.20.10.2:1313/blog/add-copy-button-to-code-blocks-hugo-chroma/\" score: 1.537 title: \"Hugo: Add Copy-to-Clipboard Button to Code Blocks with Vanilla JS\" __proto__: Object 4: categories: \"Linux\" content: \"Why would you want to install NGINX from source co\" href: \"http://172.20.10.2:1313/blog/install-nginx-source-code-shell-script/\" score: 1.11 title: \"Create a Shell Script to Install NGINX from Source On Ubuntu\" __proto__: Object 5: categories: \"Flask ; Python ; Tutorial-Series\" content: \"Project Structure The chart below shows the folder\" href: \"http://172.20.10.2:1313/series/flask-api-tutorial/part-6/\" score: 0.808 title: \"How To: Create a Flask API with JWT-Based Authentication (Part 6)\" __proto__: Object 6: categories: \"Flask ; Python ; Tutorial-Series\" content: \"Introduction My goal for this tutorial is to provi\" href: \"http://172.20.10.2:1313/series/flask-api-tutorial/part-1/\" score: 0.717 title: \"How To: Create a Flask API with JWT-Based Authentication (Part 1)\" __proto__: Object length: 7 As you can see, the list of search results is ordered by score and each list item contains all of the information from the JSON page data file. This is everything we need to construct the list of search results and render them as HTML.\nCreate Search Results HTML Template Create a new partial template named search_results.html and add the content below:\n\u0026lt;section class=\u0026#34;search-results hide-element\u0026#34;\u0026gt; \u0026lt;div id=\u0026#34;search-header\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;search-query search-query-left\u0026#34;\u0026gt;search term: \u0026lt;span id=\u0026#34;query\u0026#34;\u0026gt;\u0026lt;/span\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;search-query search-query-right\u0026#34;\u0026gt;\u0026lt;span id=\u0026#34;results-count\u0026#34;\u0026gt;\u0026lt;/span\u0026gt;\u0026lt;span id=\u0026#34;results-count-text\u0026#34;\u0026gt;\u0026lt;/span\u0026gt;\u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;button id=\u0026#34;clear-search-results-mobile\u0026#34; class=\u0026#34;button clear-search-results\u0026#34;\u0026gt;Clear Search Results\u0026lt;/button\u0026gt; \u0026lt;ul\u0026gt;\u0026lt;/ul\u0026gt; \u0026lt;/section\u0026gt;   The partial HTML template and how it is implemented into the page should not be taken and applied blindly to your web site. The layout and CSS you see here are designed to work in concert with the layout templates I have adapted from the Mainroad base theme and there is very little chance that it will make sense on another site unless significant changes are made.\n  My site is based on a responsive, two-column layout. The two columns are the \u0026lt;div class=\u0026quot;primary\u0026quot;\u0026gt; and \u0026lt;aside class=\u0026quot;sidebar\u0026quot;\u0026gt; elements shown in the example below. The highlighted elements are the search input form which we just created (\u0026lt;div id=\u0026quot;site-search\u0026quot;/\u0026gt;, which is the first widget in the sidebar), and the other is the search results which we will implement now (i.e., \u0026lt;section class=\u0026quot;search-results hide-element\u0026quot;/\u0026gt;).\n\u0026lt;div class=\"wrapper flex\"\u0026gt; \u0026lt;div class=\"primary\"\u0026gt; ...page content \u0026lt;/div\u0026gt; \u0026lt;section class=\"search-results hide-element\"/\u0026gt; \u0026lt;aside class=\"sidebar\"\u0026gt; \u0026lt;div id=\"site-search\"/\u0026gt; ...other widgets \u0026lt;/aside\u0026gt; \u0026lt;/div\u0026gt; The important thing to take away from this is that the search results are hidden when the page loads. However, when the user submits a search query, the \u0026lt;div class=\u0026quot;primary\u0026quot;\u0026gt; element containing the page content will be hidden (by adding the hide-element class) and the search results will be shown in their place (by removing the hide-element class).\nThe search results will contain prominent buttons labeled Clear Search Results that will restore the original layout when clicked (i.e., the search results will be hidden by adding the hide-element class and the \u0026lt;div class=\u0026quot;primary\u0026quot;\u0026gt; element containing the page content will be shown again by removing the hide-element class).\nAgain, the CSS below for the search results HTML template is what I am using on this site, you may have to modify it substantially in order to use it. Alternatively, you could create your own layout and style for the search results:\n.wrapper { display: flex; padding: 40px; } .primary { flex: 1 0 68%; margin: 0; display: flex; flex-flow: column nowrap; } main { margin: 0 20px 0 0; } .content { font-size: 1em; } .sidebar { font-size: 1.1em; flex: 0 0 30%; margin: 0; padding: 0; } .sidebar \u0026gt; div { background-color: hsl(0, 0%, 13%); border-radius: 4px; margin: 0 0 20px; padding: 10px 15px; box-shadow: 0 3px 3px rgba(0, 0, 0, 0.2), 0 3px 20px rgba(0, 0, 0, 0.3); } .search-results { flex: 1 0 68%; margin: 0 0 20px 0; } .search-query, .search-detail { color: hsl(0, 0%, 70%); line-height: 1.2; } #results-count { font-size: 0.9em; margin: 0 6px 0 0; } .search-query { flex: 1 1 auto; font-size: 1.5em; } .search-query-left { align-self: flex-end; } .search-query-right { flex: 0 1 auto; text-align: right; white-space: nowrap; align-self: flex-end; } .search-result-page-title { font-size: 1.4em; line-height: 1.2; } .search-detail { font-size: 1.4em; } .search-results ul { list-style: none; margin: 0 2em 0 0; padding: 0; } .search-results a { text-decoration: none; } .search-results p { font-size: 0.95em; color: hsl(0, 0%, 70%); line-height: 1.45; margin: 10px 0 0 0; word-wrap: break-word; } .search-result-item { background-color: hsl(0, 0%, 13%); padding: 20px; margin: 0 0 20px 0; border-radius: 4px; box-shadow: 0 3px 3px rgba(0, 0, 0, 0.2), 0 3px 20px rgba(0, 0, 0, 0.3); } .search-result-item p strong { color: hsl(0, 0%, 95%); } #search-header { display: flex; flex-flow: row nowrap; justify-content: space-between; margin: 0 2em 10px 0; } #query { color: hsl(0, 0%, 95%); font-weight: 700; margin: 0 0 0 3px; } @media screen and (max-width: 767px) { .wrapper { display: block; } .primary { margin: 0; } .sidebar { float: none; width: 100%; margin: 0 0 40px 0; padding: 20px 0; border-top: 2px dotted hsl(0, 0%, 50%); } #search-header { font-size: 0.85em; margin: 0 0 20px 0; } #clear-search-results-mobile { display: block; font-size: 1.3em; padding: 10px 15px; margin: 0 0 20px 0; } .search-results ul { margin: 0; } } Render Search Results Fair warning! This section is going to be lengthy since this is where I have created features and behaviors that are (as far as I have seen) distinct from other Lunr.js/Hugo implementations:\nThe text blurb below the title of each search result is generated by an algorithm that finds sentences in the page content containing one or more search terms. All such sentences are added to the text blurb until the total word count of the blurb exceeds or is equal to a configurable maximum length. If the search term occurs sparingly, the text blurb will be short. There is no minimum length. Whenever a search term appears in the text blurb, it is displayed with bold font weight. The search result score is used to set the color of the search result page title. In the videos below, the first search result (i.e., the search result with the highest score) is displayed with a teal font color, and each subsequent page is displayed in a \"cooler\" color. The final search result (i.e., the lowest/worst score) is displayed in a mid-dark blue color.  You can see all of these features and compare the desktop and mobile user experience in the videos below.\n Sorry, your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!     Desktop Experience\n  Sorry, your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!    Mobile Experience\n   Create Search Result Blurb Let\u0026rsquo;s start with the code that generates the text blurb beneath each search result. Begin by adding Lines 2-4 to search.js:\n1 2 3 4  let pagesIndex, searchIndex const MAX_SUMMARY_LENGTH = 100 const SENTENCE_BOUNDARY_REGEX = /\\b\\.\\s/gm const WORD_REGEX = /\\b(\\w*)[\\W|\\s|\\b]?/gm     MAX_SUMMARY_LENGTH: Maximum length (in words) of each text blurb. You can change this value if you find that 100 is too short or too long for your taste.\n  SENTENCE_BOUNDARY_REGEX: Since the blurb is comprised of full sentences containing any search term, we need a way to identify where each sentence begins/ends. This regex will be used to produce a list of all sentences from the page content.\n  WORD_REGEX: This is a simple regex that produces a list of words from the text it is applied to. This will be used to check the number of total words in the blurb as it is being built.\n   Next, add the createSearchResultBlurb function to search.js:\n92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128  function createSearchResultBlurb(query, pageContent) { const searchQueryRegex = new RegExp(createQueryStringRegex(query), \u0026#34;gmi\u0026#34;); const searchQueryHits = Array.from( pageContent.matchAll(searchQueryRegex), (m) =\u0026gt; m.index ); const sentenceBoundaries = Array.from( pageContent.matchAll(SENTENCE_BOUNDARY_REGEX), (m) =\u0026gt; m.index ); let searchResultText = \u0026#34;\u0026#34;; let lastEndOfSentence = 0; for (const hitLocation of searchQueryHits) { if (hitLocation \u0026gt; lastEndOfSentence) { for (let i = 0; i \u0026lt; sentenceBoundaries.length; i++) { if (sentenceBoundaries[i] \u0026gt; hitLocation) { const startOfSentence = i \u0026gt; 0 ? sentenceBoundaries[i - 1] + 1 : 0; const endOfSentence = sentenceBoundaries[i]; lastEndOfSentence = endOfSentence; parsedSentence = pageContent.slice(startOfSentence, endOfSentence).trim(); searchResultText += `${parsedSentence}... `; break; } } } const searchResultWords = tokenize(searchResultText); const pageBreakers = searchResultWords.filter((word) =\u0026gt; word.length \u0026gt; 50); if (pageBreakers.length \u0026gt; 0) { searchResultText = fixPageBreakers(searchResultText, pageBreakers); } if (searchResultWords.length \u0026gt;= MAX_SUMMARY_LENGTH) break; } return ellipsize(searchResultText, MAX_SUMMARY_LENGTH).replace( searchQueryRegex, \u0026#34;\u0026lt;strong\u0026gt;$\u0026amp;\u0026lt;/strong\u0026gt;\u0026#34; ); }   There is a lot to breakdown in this function, so bear with me and stay focussed!\n  Line 92: The important thing to note about the function definition is that createSearchResultBlurb requires two parameters: query and pageContent. query is the value entered by the user into the search form text box, and pageContent is the full text of the page that was returned as a match for the user's query.\n  Line 93: The first thing that we do is call createQueryStringRegex, providing query as the only argument. Add the code below to search.js:\n130 131 132 133 134 135 136 137 138 139 140 141  function createQueryStringRegex(query) { const searchTerms = query.split(\u0026#34; \u0026#34;); if (searchTerms.length == 1) { return query; } query = \u0026#34;\u0026#34;; for (const term of searchTerms) { query += `${term}|`; } query = query.slice(0, -1); return `(${query})`; }   This function creates a regular expression from the user's search query which is designed to find all occurrences of any term in the search query within any given piece of text. This regular expression is created by performing the following steps:\nCall query.split(\" \") to produce a list of search terms. If the query only contains a single search term, it is returned unmodified. Otherwise, the function iterates through the search terms and appends the | character to each word, creating a new string by concatenating the modified search terms. After iterating through all search terms, the final | character is removed from the concatenated string. The concatenated string is surrounded by parentheses to create a capturing group and this string is used as the return value.  For example, if the user entered vanilla javascript, the createQueryStringRegex function would produce (vanilla|javascript). This is a very simple regular expression that finds either the word vanilla OR the word javascript.\nThe string value returned from createQueryStringRegex is provided to the RegExp constructor, along with the global, multi-line and case-insensitive flags to create a RegExp object that is stored in the searchQueryRegex variable.\n  Lines 94-97: pageContent.matchAll(searchQueryRegex) is called to produce an array containing the location of all occurrences of any search term within the page content. The list of search term locations is stored in the searchQueryHits variable.\n  Lines 98-101: pageContent.matchAll(SENTENCE_BOUNDARY_REGEX) is called to produce an array containing the location of all period (.) characters that are followed by a whitespace character within the page content. The list of sentence boundary locations is stored in the sentenceBoundaries variable.\n  Line 102: The variable searchResultText is declared as an empty string. Sentences from the page content containing search terms will be parsed and appended to this string, which will be used as the return value when all relevant sentences have been added or the maximum length for the search result text blurb has been reached.\n  Line 103: The variable lastEndOfSentence will be used to keep track of the location of the last sentence that was found containing a search term.\n  Line 104: We use a for..of loop to iterate through searchQueryHits, hitLocation is the current search term location.\n  Line 105: For each hitLocation, the first thing we do is make sure that the location of the search term occurs after the location of the last sentence boundary. If the location of the search term occurs before lastEndOfSentence, this means that the last sentence that was added to searchResultText contains multiple search terms. In that case, we move on to the next hitLocation.\n  Line 106: For each hitLocation, we iterate through the list of sentenceBoundaries using a traditional for loop. We use this type of loop because we need access to the index of the current item.\n  Line 107: We compare the hitLocation to the current sentenceBoundary. When the current sentenceBoundary is greater than hitLocation, we know that the current sentenceBoundary is the end of the sentence containing the hitLocation.\n  Lines 108-110: Since the current sentenceBoundary is the end of the sentence containing the hitLocation, the previous sentenceBoundary is the start of the sentence, these two locations are stored in the variables endOfSentence and startOfSentence, respectively. Also, we update lastEndOfSentence to be equal to the value of endOfSentence.\n  Lines 111: Using the locations startOfSentence and endOfSentence, we create a substring from pageContent and trim any leading/trailing whitespace, storing the string as parsedSentence.\n  Line 112: We append parsedSentence and an ellipsis (...) to the end of searchResultText.\n  Line 113: After parsing the sentence containing the hitLocation and updating searchResultText to include this sentence, we do not need to iterate through the list of sentenceBoundaries any longer, so we break out of the for loop.\n  Line 117: After we break out of the inner for loop, the first thing we do is call the tokenize function, providing searchResultText as the only argument. The code for this function is given below, add it to search.js:\n143 144 145 146 147 148 149 150 151  function tokenize(input) { const wordMatches = Array.from(input.matchAll(WORD_REGEX), (m) =\u0026gt; m); return wordMatches.map((m) =\u0026gt; ({ word: m[0], start: m.index, end: m.index + m[0].length, length: m[0].length, })); }   This function takes an input string and uses the WORD_REGEX regular expression to capture the individual words from the string. The return value is an array of objects containing (for each word) the word itself, the start/end indices where the word is located within the input string, and the length of the word.\nThe tokenize function is called with the current value of searchResultText, storing the tokenized list of words in the searchResultWords variable.\n  Line 118: Next, we filter searchResultWords for all words that are longer than 50 characters and store the result in a variable named pageBreakers. The name should be a clue as to why we are filtering for these words, since extremely long words will fail to cause a line-break leading to layout issues with the page.\n  Lines 119-121: If searchResultText contains any words longer than 50 characters, they need to be broken into smaller pieces. This task is performed by calling the fixPageBreakers function. Add the code below to search.js:\n153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177  function fixPageBreakers(input, largeWords) { largeWords.forEach((word) =\u0026gt; { const chunked = chunkify(word.word, 20); input = input.replace(word.word, chunked); }); return input; } function chunkify(input, chunkSize) { let output = \u0026#34;\u0026#34;; let totalChunks = (input.length / chunkSize) | 0; let lastChunkIsUneven = input.length % chunkSize \u0026gt; 0; if (lastChunkIsUneven) { totalChunks += 1; } for (let i = 0; i \u0026lt; totalChunks; i++) { let start = i * chunkSize; let end = start + chunkSize; if (lastChunkIsUneven \u0026amp;\u0026amp; i === totalChunks - 1) { end = input.length; } output += input.slice(start, end) + \u0026#34; \u0026#34;; } return output; }   I won't explain the fixPageBreakers and chunkify functions in great detail since they are fairly self-explanatory. The largeWords argument contains a list of extremely long words and we iterate through this list calling chunkify for each word.\nchunkify accepts two arguments, a word that needs to be broken into chunks (input) and chunkSize which is the length of each chunk. The most important thing to note about chunkify is that the return value is a single string, not a list of \"chunks\". This string is constructed by concatenating the chunks into a single string, separated by space characters.\nThe string returned from chunkify is used to replace the original really long word in the input string. After doing so for all words in largeWords, the input string (which no longer has any really long words) is returned.\n  Line 122: At this point we are still within the main for loop. Before beginning the next iteration, we check the length of searchResultText (in words). If the number of words is greater than or equal to the value MAX_SUMMARY_LENGTH, we break out of the for loop.\n  Line 124: After constructing searchResultText, the first thing we do is call ellipsize. The code for this function is given below, please add it to search.js:\n179 180 181 182 183 184 185  function ellipsize(input, maxLength) { const words = tokenize(input); if (words.length \u0026lt;= maxLength) { return input; } return input.slice(0, words[maxLength].end) + \u0026#34;...\u0026#34;; }   This function is very simple. It requires two parameters: an input string and a number (maxLength) which is the maximum number of words allowed in the input string. First, the tokenize function is called, which produces a list with an object for every word in the input string. If the number of words in input is less than or equal to maxLength, the original string is returned, unmodified.\nHowever, if the number of words is greater than maxLength, the string is truncated in an intelligent way. Using the words list produced by the tokenize function, we know that the last word that is allowed is words[maxLength]. The end property is the index of the last character of this word within the input string. This allows us to create a substring from the start of input to the location of the end of the last allowed word. The return value is the truncated string with an ellipsis appended (...).\n  Line 124-127: The last thing we do to searchResultText is call the replace method using the searchQueryRegex created earlier in Line 85 as the first argument (this regular expression is designed to capture any search term entered by the user). Any matches in searchResultText for this regular expression will be replaced by the second argument, '\u0026lt;strong\u0026gt;$\u0026\u0026lt;/strong\u0026gt;'. The special syntax $\u0026 represents the substring that matched the regular expression (i.e., the search term).\nSummarizing the above, calling the replace method finds all occurrences of any search term entered by the user within searchResultText and wraps the search term in a \u0026lt;strong\u0026gt; element. This will make all search terms appear in bold font weight when rendered as HTML.\n   If you are still here and you are still awake, that is seriously impressive! The createSearchResultBlurb function takes care of items #1 and #2 in the list of features provided earlier, hopefully that makes the insane level of detail provided understandable.\nPolyfill String.matchAll Before moving on to the next implementation detail, add the code below to search.js:\n204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220  if (!String.prototype.matchAll) { String.prototype.matchAll = function (regex) { \u0026#34;use strict\u0026#34;; function ensureFlag(flags, flag) { return flags.includes(flag) ? flags : flags + flag; } function* matchAll(str, regex) { const localCopy = new RegExp(regex, ensureFlag(regex.flags, \u0026#34;g\u0026#34;)); let match; while ((match = localCopy.exec(str))) { match.index = localCopy.lastIndex - match[0].length; yield match; } } return matchAll(this, regex); }; }   This is a polyfill for the String.matchAll method, which is available in most browsers but notably absent from mobile Safari 12, which is still used by a lot of devices. Adding this polyfill will make this method available to all browsers.\nSet Font Color Based on Search Result Score The only item that has not been implemented is probably the most impactful feature, at least visually. This feature uses the search result score to set the color of the search result page title, producing a gradient effect of warm to cool color to communicate which results are determined to be higher- and lower-quality.\nAdd the code below to search.js:\n187 188 189 190 191 192 193 194 195 196 197 198  function getColorForSearchResult(score) { const highQualityHue = 171; const lowQualityHue = 212; return adjustHue(highQualityHue, lowQualityHue, score); } function adjustHue(hue1, hue2, score) { if (score \u0026gt; 3) return `hsl(${hue1}, 100%, 50%)`; const hueAdjust = (parseFloat(score) / 3) * (hue1 - hue2); const newHue = hue2 + Math.floor(hueAdjust); return `hsl(${newHue}, 100%, 50%)`; }   The getColorForSearchResult function requires a single argument, score, which is the Lunr.js score for a single search result. If you remember way back when we created the searchSite function, this is a numeric value that describes the quality of the search result.\nAlso within getColorForSearchResult, we define two values: warmColorHue and coolColorHue. These values are used for the hue component of a HSL color value. If you are unfamiliar with HSL colors, I wholeheartedly recommend using them instead of RBG colors since the individual components are much easier to understand and manipulate through code. A quick definition of these components is given below:\n  Hue: Think of a color wheel. Around 0° and 360° are reds. 120° is where greens are and 240° are blues. Use anything in between 0-360. Values above and below will be modulus 360. Saturation: 0% is completely desaturated (grayscale). 100% is fully saturated (full color). Lightness: 0% is completely dark (black). 100% is completely light (white). 50% is average lightness.   Given the definitions above, we can easily modify a color simply by adjusting the hue component. The adjustHue function leaves both saturation and lightness components untouched at 100% and 50%, respectively, in order to produce fully saturated colors with average lightness.\nFrom my observations of the search results produced by the Lunr.js implementation on this site, the vast majority of search result scores are in the range 0-3. For this reason, if a search result score is greater than 3, we do not perform any calculations to adjust the hue and simply return a HSL color using the warmColorHue value.\nHowever, if the score is less than 3, we will adjust the hue and produce an HSL color that gradually becomes \u0026ldquo;cooler\u0026rdquo;. As the score approaches zero, the color becomes closer to the coolColorHue value.\nTieing it All Together You may be wondering, where is the code that calls the createSearchResultBlurb and getColorForSearchResult functions? IMO, it is much easier to understand how these functions are designed when they are presented on their own. Now, we will close the loop and create the code that uses these functions.\nFirst, add the highlighted line below to the handleSearchQuery function:\n33 34 35 36 37 38 39 40 41 42 43 44 45 46  function handleSearchQuery(event) { event.preventDefault() const query = document.getElementById(\u0026#34;search\u0026#34;).value.trim().toLowerCase() if (!query) { displayErrorMessage(\u0026#34;Please enter a search term\u0026#34;) return } const results = searchSite(query) if (!results.length) { displayErrorMessage(\u0026#34;Your search returned no results\u0026#34;) return } renderSearchResults(query, results) }   The handleSearchQuery function is called when the user submits a search query. If the query returned any results, the renderSearchResults function is called using the query string entered by the user and the list of search results.\nObviously, we need to implement the renderSearchResults function. Next, add the following functions to search.js (In other words, add the code that is highlighted below):\n renderSearchResults clearSearchResults updateSearchResults showSearchResults scrollToTop  93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250  function renderSearchResults(query, results) { clearSearchResults(); updateSearchResults(query, results); showSearchResults(); scrollToTop(); } function clearSearchResults() { const results = document.querySelector(\u0026#34;.search-results ul\u0026#34;); while (results.firstChild) results.removeChild(results.firstChild); } function updateSearchResults(query, results) { document.getElementById(\u0026#34;query\u0026#34;).innerHTML = query; document.querySelector(\u0026#34;.search-results ul\u0026#34;).innerHTML = results .map( (hit) =\u0026gt; ` \u0026lt;li class=\u0026#34;search-result-item\u0026#34; data-score=\u0026#34;${hit.score.toFixed(2)}\u0026#34;\u0026gt; \u0026lt;a href=\u0026#34;${hit.href}\u0026#34; class=\u0026#34;search-result-page-title\u0026#34;\u0026gt;${hit.title}\u0026lt;/a\u0026gt; \u0026lt;p\u0026gt;${createSearchResultBlurb(query, hit.content)}\u0026lt;/p\u0026gt; \u0026lt;/li\u0026gt; ` ) .join(\u0026#34;\u0026#34;); const searchResultListItems = document.querySelectorAll(\u0026#34;.search-results ul li\u0026#34;); document.getElementById(\u0026#34;results-count\u0026#34;).innerHTML = searchResultListItems.length; document.getElementById(\u0026#34;results-count-text\u0026#34;).innerHTML = searchResultListItems.length \u0026gt; 1 ? \u0026#34;results\u0026#34; : \u0026#34;result\u0026#34;; searchResultListItems.forEach( (li) =\u0026gt; (li.firstElementChild.style.color = getColorForSearchResult(li.dataset.score)) ); }  function createSearchResultBlurb(query, pageContent) { const searchQueryRegex = new RegExp(createQueryStringRegex(query), \u0026#34;gmi\u0026#34;); const searchQueryHits = Array.from( pageContent.matchAll(searchQueryRegex), (m) =\u0026gt; m.index ); const sentenceBoundaries = Array.from( pageContent.matchAll(SENTENCE_BOUNDARY_REGEX), (m) =\u0026gt; m.index ); let searchResultText = \u0026#34;\u0026#34;; let lastEndOfSentence = 0; for (const hitLocation of searchQueryHits) { if (hitLocation \u0026gt; lastEndOfSentence) { for (let i = 0; i \u0026lt; sentenceBoundaries.length; i++) { if (sentenceBoundaries[i] \u0026gt; hitLocation) { const startOfSentence = i \u0026gt; 0 ? sentenceBoundaries[i - 1] + 1 : 0; const endOfSentence = sentenceBoundaries[i]; lastEndOfSentence = endOfSentence; parsedSentence = pageContent.slice(startOfSentence, endOfSentence).trim(); searchResultText += `${parsedSentence}... `; break; } } } const searchResultWords = tokenize(searchResultText); const pageBreakers = searchResultWords.filter((word) =\u0026gt; word.length \u0026gt; 50); if (pageBreakers.length \u0026gt; 0) { searchResultText = fixPageBreakers(searchResultText, pageBreakers); } if (searchResultWords.length \u0026gt;= MAX_SUMMARY_LENGTH) break; } return ellipsize(searchResultText, MAX_SUMMARY_LENGTH).replace( searchQueryRegex, \u0026#34;\u0026lt;strong\u0026gt;$\u0026amp;\u0026lt;/strong\u0026gt;\u0026#34; ); } function createQueryStringRegex(query) { const searchTerms = query.split(\u0026#34; \u0026#34;); if (searchTerms.length == 1) { return query; } query = \u0026#34;\u0026#34;; for (const term of searchTerms) { query += `${term}|`; } query = query.slice(0, -1); return `(${query})`; } function tokenize(input) { const wordMatches = Array.from(input.matchAll(WORD_REGEX), (m) =\u0026gt; m); return wordMatches.map((m) =\u0026gt; ({ word: m[0], start: m.index, end: m.index + m[0].length, length: m[0].length, })); } function fixPageBreakers(input, largeWords) { largeWords.forEach((word) =\u0026gt; { const chunked = chunkify(word.word, 20); input = input.replace(word.word, chunked); }); return input; } function chunkify(input, chunkSize) { let output = \u0026#34;\u0026#34;; let totalChunks = (input.length / chunkSize) | 0; let lastChunkIsUneven = input.length % chunkSize \u0026gt; 0; if (lastChunkIsUneven) { totalChunks += 1; } for (let i = 0; i \u0026lt; totalChunks; i++) { let start = i * chunkSize; let end = start + chunkSize; if (lastChunkIsUneven \u0026amp;\u0026amp; i === totalChunks - 1) { end = input.length; } output += input.slice(start, end) + \u0026#34; \u0026#34;; } return output; } function ellipsize(input, maxLength) { const words = tokenize(input); if (words.length \u0026lt;= maxLength) { return input; } return input.slice(0, words[maxLength].end) + \u0026#34;...\u0026#34;; } function showSearchResults() { document.querySelector(\u0026#34;.primary\u0026#34;).classList.add(\u0026#34;hide-element\u0026#34;); document.querySelector(\u0026#34;.search-results\u0026#34;).classList.remove(\u0026#34;hide-element\u0026#34;); document.getElementById(\u0026#34;site-search\u0026#34;).classList.add(\u0026#34;expanded\u0026#34;); document.getElementById(\u0026#34;clear-search-results-sidebar\u0026#34;).classList.remove(\u0026#34;hide-element\u0026#34;); } function scrollToTop() { const toTopInterval = setInterval(function () { const supportedScrollTop = document.body.scrollTop \u0026gt; 0 ? document.body : document.documentElement; if (supportedScrollTop.scrollTop \u0026gt; 0) { supportedScrollTop.scrollTop = supportedScrollTop.scrollTop - 50; } if (supportedScrollTop.scrollTop \u0026lt; 1) { clearInterval(toTopInterval); } }, 10); }  function getColorForSearchResult(score) { const warmColorHue = 171; const coolColorHue = 212; return adjustHue(warmColorHue, coolColorHue, score); } function adjustHue(hue1, hue2, score) { if (score \u0026gt; 3) return `hsl(${hue1}, 100%, 50%)`; const hueAdjust = (parseFloat(score) / 3) * (hue1 - hue2); const newHue = hue2 + Math.floor(hueAdjust); return `hsl(${newHue}, 100%, 50%)`; }   I included the code for the createSearchResultBlurb and getColorForSearchResult functions (and all functions they rely on), in case there was any confusion about the layout/structure I am using for search.js.\n  Lines 94-97: The algorithm of the renderSearchResults function can be quickly understood just by reading the names of the functions that are called. The steps performed in order to render the results are given below:\nFirst, results from the previous search request are cleared. Second, the list of search results is updated with the results from the current search request. Next, the \u0026lt;div class=\"primary\"\u0026gt; element containing the page content will be hidden (by adding the hide-element class) and the search results will be shown in their place (by removing the hide-element class). Finally, the page is automatically scrolled to the top of the page (this is done for mobile devices since the search form is placed at the bottom of the page).    Lines 100-103: Since the search results are rendered as li elements, we clear the list by querying for the ul element they belong to. All li elements are removed from the ul element using a while loop.\n  Line 106: The first thing we do in the updateSearchResults function is populate a span element above the list of search results to display the search query entered by the user (click here to see the search results HTML template, the query element is in the search-header div element).\n  Lines 107-108: We populate the list by first querying for the ul element within the search results HTML template. Then, we set the innerHTML value of this element to the text string returned from the results.map method call.\n  Lines 109-110: hit is the current search result that the map method is acting upon in order to construct an HTML representation of a search result. Each search result is represented as a li element that contains two child elements, the title of the page as a clickable link and a paragraph of text generated by the createSearchResultBlurb function.   PLEASE NOTE that we are defining a attribute on each li element named data-score and setting the value equal to hit.score, which is the Lunr.js score for the quality of the search result.\n    Line 111: In order to create the page title as a clickable link, first we set the href attribute to the value of hit.href. Next, by using hit.title as the innerHTML value of the a element, the title of the page will be used as the link text.\n  Line 112: The second child element is created simply by wrapping the text returned from the createSearchResultBlurb function inside a p element.\n  Line 116: After the map method has iterated through all search results, it returns a list of strings where each list item is the raw HTML for a li element. The join method of this list is called to convert the list into a single string of raw HTML which is set as the innerHTML value of the search results ul element.\n  Line 117: After generating raw HTML for the search results, we query for all li elements that are children of the search results ul element and store the result in the totalSearchResults variable.\n  Lines 118-119: In the search results HTML template, we created placeholder span elements to display the number of results found above the list of search results. To populate these with the correct values, we find these elements by ID and set the value for each using the totalSearchResults variable.\n  Lines 120-122: The last thing we do in the updateSearchResults function is the most important \u0026mdash; we finally change the color of the search result page title using the getColorForSearchResult function.\nTo do so, we iterate through the li elements in searchResultListItems. For each li element, the page title can be accessed with li.firstElementChild. Also, because we created the data-score attribute on each li element, we can access this value from the dataset object as li.dataset.score.\n  If you are unfamiliar with data attributes, they are easy to use and extremely useful. Click here for an MDN article explaining what they are and how to use them.\n    Lines 221-224: After constructing the list of search results in HTML, we need to actually display them since the entire \u0026lt;section class=\"search-results\"\u0026gt; element is hidden when the page initially loads due to the hide-element class.\nThe showSearchResults function adds the hide-element class to the \u0026lt;div class=\"primary\"\u0026gt; element (which contains the regular page content) and removes the hide-element class from \u0026lt;section class=\"search-results\"\u0026gt;, displaying them in place of the regular page content.\nThe Clear Search Results buttons below the search form text box and above the list of search results are also displayed, which allows the user to restore the page to normal.\n  Lines 227-237: Finally, after the regular page content has been hidden and the list of search results is displayed, the page automatically scrolls to the top. This is done solely for mobile users, since the search form is displayed at the bottom of the page if the screen width is less than 767px.\nWithout the scrollToTop function, when a mobile user submits a search query, they would have to manually scroll to the top in order to see the first (i.e., the highest-quality) search result. You can see the scroll-to-top behavior in the video demonstrating the mobile search UX from the beginning of this section.\n   Clear Search Results Finally, we need a way to clear the search results and display the page content when the user clicks the Clear Search Results button. To do so, add the lines highlighted below to search.js:\n252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285  function handleClearSearchButtonClicked() { hideSearchResults(); clearSearchResults(); document.getElementById(\u0026#34;search\u0026#34;).value = \u0026#34;\u0026#34;; } function hideSearchResults() { document.getElementById(\u0026#34;clear-search-results-sidebar\u0026#34;).classList.add(\u0026#34;hide-element\u0026#34;); document.getElementById(\u0026#34;site-search\u0026#34;).classList.remove(\u0026#34;expanded\u0026#34;); document.querySelector(\u0026#34;.search-results\u0026#34;).classList.add(\u0026#34;hide-element\u0026#34;); document.querySelector(\u0026#34;.primary\u0026#34;).classList.remove(\u0026#34;hide-element\u0026#34;); }  initSearchIndex() document.addEventListener(\u0026#34;DOMContentLoaded\u0026#34;, function () { if (document.getElementById(\u0026#34;search-form\u0026#34;) != null) { const searchInput = document.getElementById(\u0026#34;search\u0026#34;) searchInput.addEventListener(\u0026#34;focus\u0026#34;, () =\u0026gt; searchBoxFocused()) searchInput.addEventListener(\u0026#34;keydown\u0026#34;, (event) =\u0026gt; { if (event.keyCode == 13) handleSearchQuery(event) }) document .querySelector(\u0026#34;.search-error\u0026#34;) .addEventListener(\u0026#34;animationend\u0026#34;, removeAnimation) document .querySelector(\u0026#34;.fa-search\u0026#34;) .addEventListener(\u0026#34;click\u0026#34;, (event) =\u0026gt; handleSearchQuery(event)) } document .querySelectorAll(\u0026#34;.clear-search-results\u0026#34;) .forEach((button) =\u0026gt; button.addEventListener(\u0026#34;click\u0026#34;, () =\u0026gt; handleClearSearchButtonClicked()) ) })     Lines 252-256: The handleClearSearchButtonClicked function is called when the user clicks one of the Clear Search Results buttons. When this occurs, the hideSearchResults function is called first, which hides the search results and un-hides the regular page content (restoring the page to the normal state). Then, the clearSearchResults function is called (this was already defined and explained), and finally any value in the search input text box is cleared.\n  Lines 258-263: The hideSearchResults function performs the exact opposite set of steps that the showSearchResults function performs. The Clear Search Results buttons are hidden, the \u0026lt;section class=\"search-results\"\u0026gt; element is hidden, and the hide-element class is removed from the \u0026lt;div class=\"primary\"\u0026gt; element (which contains the regular page content).\n  Lines 280-284: Since there are two Clear Search Results buttons, we can add an event listener for the click event to each by querying for all elements with the clear-search-results class, iterating over the result and assigning the handleClearSearchButtonClicked function as the event handler.\n   Conclusion I hope this post was helpful to you, if you have any questions please leave a comment. Remember, you can view the full, finished CSS and JS using the CodePen I embedded earlier.\n","href":"/blog/add-search-to-static-site-lunrjs-hugo-vanillajs/","title":"Add Search to Your Static Site with Lunr.js (Hugo, Vanilla JS)"},{"categories":"Python ; NodeJS ; my-projects","content":"Project Documentation vigorish is a hybrid Python/Node.js application that scrapes data from mlb.com, brooksbaseball.net and baseball-reference.com, primarily using Nightmare/Electron. My original goal was to only capture PitchFX data, but the scope increased to include play-by-play data, player biographical data, aggregate batting/pitching stats and various sets of metadata such as weather conditions, stadium information, etc.\nSelect the document you wish to view from the options below:\nInstallation User Guide  ","href":"/projects/vigorish/","title":"Vigorish: Hybrid Python/Node.Js Web Scraper"},{"categories":"Python","content":"Introduction Decorators can be a daunting topic when first encountered. While The Zen of Python states \u0026ldquo;There should be one\u0026ndash; and preferably only one \u0026ndash;obvious way to do it\u0026rdquo;, there are many, equally valid ways to implement the same decorator. These different methods can be categorized as either function-based, class-based, or a hybrid of both. In this post I will explain the design and behavior of Python decorators and provide examples of decorators that I frequently use in my own code.\nWhat is a Decorator? In Python, absolutely everything is an object, including functions. Since functions are objects, they can be passed as arguments to another function, they can be the return value of a function, and they can be assigned to a variable. If you understand these concepts then you have everything you need to understand decorators.\nA decorator is any callable object that takes a function as an input parameter. I specifically said \u0026ldquo;callable object\u0026rdquo; rather than \u0026ldquo;function\u0026rdquo; since Python allows you to create other types of callable objects. This interesting language feature is what allows us to create class-based decorators, as we will see shortly.\nA Simple Decorator Decorators are wrappers that allow you to execute code before and after the \u0026ldquo;wrapped\u0026rdquo; (or \u0026ldquo;decorated\u0026rdquo;) function is executed. By manually constructing a decorator function this \u0026ldquo;wrapping\u0026rdquo; effect can easily be demonstrated. Consider the decorators.basic module given below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14  \u0026#34;\u0026#34;\u0026#34;decorators.basic\u0026#34;\u0026#34;\u0026#34; def simple_decorator(function_to_decorate): def function_wrapper(): print(f\u0026#34;Preparing to execute: {function_to_decorate.__name__}\u0026#34;) function_to_decorate() print(f\u0026#34;Finished executing: {function_to_decorate.__name__}\u0026#34;) return function_wrapper def undecorated_function(): print(\u0026#34;I FORBID you from modifying how this function behaves!\u0026#34;)     Line 2: simple_decorator accepts a function as a parameter, making it a decorator.\n  Line 3: function_wrapper (defined within simple_decorator) allows us to execute code before and after executing the wrapped function.\n  Line 5: This print statement will be executed before the wrapped function.\n  Line 6: The wrapped function is executed within function_wrapper.\n  Line 7: This print statement will be executed after the wrapped function\n  Line 9: This is probably the most confusing part. function_wrapper is the return value of the decorator function (simple_decorator). At this point, the wrapped function (function_to_decorate) HAS NOT been executed.\n  Line 12: We will use this function to demonstrate how simple_decorator works.\n   Open an interactive Python shell and execute undecorated_function() to see the behavior before applying any decorator. Next, we pass undecorated_function as a parameter to simple_decorator, and store the return value in decorated_function:\n(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.  from decorators.basic import *   undecorated_function() I FORBID you from modifying how this function behaves!  decorated_function = simple_decorator(undecorated_function)   Please note the difference between calling a function (undecorated_function()) and passing a function as a parameter to another function (simple_decorator(undecorated_function)). If we had executed simple_decorator(undecorated_function()), we would not see the effect of the wrapper function. The behavior would be the same as when we executed undecorated_function().\n  Finally, we execute decorated_function():\n decorated_function() Preparing to execute: undecorated_function I FORBID you from modifying how this function behaves! Finished executing: undecorated_function This is the result we expect after applying simple_decorator. However, wouldn\u0026rsquo;t it be better if we could permanently alter the behavior of undecorated_function? We can easily do this if we replace the reference to undecorated_function with the function returned by simple_decorator:\n undecorated_function = simple_decorator(undecorated_function)  undecorated_function() Preparing to execute: undecorated_function I FORBID you from modifying how this function behaves! Finished executing: undecorated_function This is exactly what happens when you decorate a function using Python\u0026rsquo;s @decorator syntax. To reinforce this point, we can modify our code to use the normal decorator syntax. Note that we are applying the decorator to undecorated_function in Line 12:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15  \u0026#34;\u0026#34;\u0026#34;decorators.basic\u0026#34;\u0026#34;\u0026#34; def simple_decorator(function_to_decorate): def function_wrapper(): print(f\u0026#34;Preparing to execute: {function_to_decorate.__name__}\u0026#34;) function_to_decorate() print(f\u0026#34;Finished executing: {function_to_decorate.__name__}\u0026#34;) return function_wrapper @simple_decorator def undecorated_function(): print(\u0026#34;I FORBID you from modifying how this function behaves!\u0026#34;)   Let\u0026rsquo;s verify that the behavior of the function has been modified:\n(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.  from decorators.basic import *   undecorated_function() Preparing to execute: undecorated_function I FORBID you from modifying how this function behaves! Finished executing: undecorated_function As you can see, @simple_decorator is just syntactic sugar for undecorated_function = simple_decorator(undecorated_function).\nThis type of decorator is nice, but what if our original function has one or more input parameters? It wouldn\u0026rsquo;t be possible to provide values for these parameters with the current implementation of simple_decorator. What can we do to fix this?\nPassing Arguments to the Wrapped Function Fixing this is very easy. Remember, when we execute the \u0026ldquo;wrapped\u0026rdquo; function, we are really executing the function returned by the decorator function (function_wrapper in the previous example). Any values we provide to this function can be passed on to the wrapped function.\nHowever, we need to accommodate all possible combinations of input parameters. We can do this by modifying function_wrapper to accept *args, **kwargs and passing them on to function_to_decorate. This has been implemented in a new module, decorators.better:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20  \u0026#34;\u0026#34;\u0026#34;decorators.better\u0026#34;\u0026#34;\u0026#34; def a_better_decorator(function_to_decorate): def function_wrapper(*args, **kwargs): print(f\u0026#34;Preparing to execute: {function_to_decorate.__name__}\u0026#34;) function_to_decorate(*args, **kwargs) print(f\u0026#34;Finished executing: {function_to_decorate.__name__}\u0026#34;) return function_wrapper @a_better_decorator def greeting_anonymous(): print(\u0026#34;Hello anonymous website viewer!\u0026#34;) @a_better_decorator def greeting_personal(name=\u0026#34;Aaron\u0026#34;): print(f\u0026#34;Hello {name}!\u0026#34;)   Let\u0026rsquo;s test this version with the two decorated functions, greeting_anonymous and greeting_personal:\n(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.  from decorators.better import *   greeting_anonymous() Preparing to execute: greeting_anonymous Hello anonymous website viewer! Finished executing: greeting_anonymous  greeting_personal() Preparing to execute: greeting_personal Hello Aaron! Finished executing: greeting_personal  greeting_personal(name=\"Jerry\") Preparing to execute: greeting_personal Hello Jerry! Finished executing: greeting_personal This works exactly the way we need it to, the value we passed to greeting_personal is used because it was passed from function_wrapper to function_to_decorate. Also, we did not break any existing functionality since the function that does not take any input parameters (greeting_anonymous) is also decorated and behaves as expected.\nPassing Arguments to the Decorator The decorators we have created so far are certainly useful for many different scenarios, but they are also limited since there is no way to pass arguments to the decorator itself. How is this different than the last decorator we created? A decorator that accepts arguments would have the form shown below:\n@foo(baz, fiz=buz) def bar(): ... If you remember back to the beginning of this post, I mentioned that, in general, there are two different ways to implement Python decorators: function-based and class-based. The examples we have seen so far have all been function-based. Unless you need to create a decorator that accepts arguments, you should use these function-based designs since they are more readable and require less nesting/indentation than the equivalent class-based design.\nHowever, when you need to pass arguments to a decorator, there isn\u0026rsquo;t an advantage to using either the function-based or class-based design. Let\u0026rsquo;s take a look at the function-based design first since it is a natural progression from the decorators we have already examined.\nFunction-based Design The generic form of a function-based decorator that accepts arguments is given below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18  \u0026#34;\u0026#34;\u0026#34;decorators.function_based\u0026#34;\u0026#34;\u0026#34; def decorator_factory(arg1, arg2): def decorator_with_args(function_to_decorate): def function_wrapper(*args, **kwargs): print(f\u0026#34;First argument provided to decorator: {arg1}\u0026#34;) function_to_decorate(*args, **kwargs) print(f\u0026#34;Second argument provided to decorator: {arg2}\u0026#34;) return function_wrapper return decorator_with_args @decorator_factory(\u0026#34;foo\u0026#34;, \u0026#34;bar\u0026#34;) def special_greeting(name=\u0026#34;Dennis\u0026#34;): print(f\u0026#34;Allow me to give a very special welcome to {name}!\u0026#34;)   In order to pass arguments to the decorator, we add another wrapper function, called decorator_factory (Line 2). I am calling it a factory because the return value is the actual function decorator (decorator_with_args is where function_to_decorate is passed in as an argument).\nIn order to understand how this works you need to realize that Line 15 (where we are applying the decorator to a function) is actually a function call to decorator_factory(\u0026quot;foo\u0026quot;, \u0026quot;bar\u0026quot;) which returns decorator_with_args. decorator_with_args is the actual decorator but in order to pass the params \u0026ldquo;foo\u0026rdquo; and \u0026ldquo;bar\u0026rdquo; to the decorator, we had to call the factory method that creates the actual decorator which accepts the function special_greeting as an argument and wraps it.\nContrast this with the other decorators we have created: when a function is decorated with @simple_decorator or @a_better_decorator these are not function calls (notice that there are no parentheses after the decorator name).\nLet\u0026rsquo;s fire up the REPL and test this decorator. We will also verify that we are still able to use the decorated function as expected by passing arguments that modify its behavior:\n(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.  from decorators.function_based import *   special_greeting() First argument provided to decorator: foo Allow me to give a very special welcome to Dennis! Second argument provided to decorator: bar  special_greeting(name=\"Harold\") First argument provided to decorator: foo Allow me to give a very special welcome to Harold! Second argument provided to decorator: bar As you can see, the arguments that were passed into the decorator (\u0026quot;foo\u0026quot; and \u0026quot;bar\u0026quot;) were used by the wrapper function, and the value passed into the decorated function (\u0026quot;Dennis\u0026quot;/\u0026quot;Harold\u0026quot;) is also used when the wrapper function is executed.\nFunctions, Classes and Callables Are you familiar with the built-in callable function? callable accepts a single argument and returns a bool value: True if the object provided appears to be callable and False if it does not. If you think that functions are the only \u0026ldquo;callable\u0026rdquo; that exists you might consider this function rather useless or unnecessary. However, classes are also callable since this is how new instances are created (e.g., object = MyObject()).\nThe call syntax, (...), can call functions or create class instances as we have just seen. But Python has a unique feature that objects other than functions can also be called. Adding the __call__ method to any class will make instances of that class callable. This allows us to create decorators that are implemented using classes.\nClass-based Design The same decorator can be implemented using a callable class instance:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20  \u0026#34;\u0026#34;\u0026#34;decorators.class_based\u0026#34;\u0026#34;\u0026#34; class DecoratorFactory: def __init__(self, arg1, arg2): self.arg1 = arg1 self.arg2 = arg2 def __call__(self, function_to_decorate): def function_wrapper(*args, **kwargs): print(f\u0026#34;First argument provided to decorator: {self.arg1}\u0026#34;) function_to_decorate(*args, **kwargs) print(f\u0026#34;Second argument provided to decorator: {self.arg2}\u0026#34;) return function_wrapper @DecoratorFactory(\u0026#34;foo\u0026#34;, \u0026#34;bar\u0026#34;) def special_greeting(name=\u0026#34;Dennis\u0026#34;): print(f\u0026#34;Allow me to give a very special welcome to {name}!\u0026#34;)   The main difference between the function-based and class-based designs is how the arguments passed to the decorator are handled. In the function-based approach, the arguments are available to function_wrapper as local variables. In the class-based design, the arguments are provided to the __init__ method and assigned to instance variables which can be accessed from function_wrapper.\nWe can confirm that this decorator behaves in exactly the same way as the function-based version:\n(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.  from decorators.class_based import *   special_greeting() First argument provided to decorator: foo Allow me to give a very special welcome to Dennis! Second argument provided to decorator: bar  special_greeting(name=\"Hank\") First argument provided to decorator: foo Allow me to give a very special welcome to Hank! Second argument provided to decorator: bar Which Design Is Better? Is there any advantage to using either decorator design? In my opinion, the class-based design is flatter and easier to read, making it the more Pythonic choice. However, I acknowledge that the function-based design is more conventional since the idea of a callable object that is an instance of a class (rather than a function) is not what most people expect when they encounter the concept of decorators.\nOther than that, there are no obvious benefits to choosing one design over the other. You should use the design that makes the most sense to you.\nAlways Use functools.wraps I have intentionally left out something very important from these decorator examples. You should ALWAYS decorate function_wrapper (or whatever name is used in your application) with the functools.wraps decorator (located in the standard library\u0026rsquo;s functools module). Why is this important? Consider the example below:\n(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.  from decorators.class_based import *   import inspect  special_greeting.__name__ 'function_wrapper'  inspect.signature(special_greeting) \u0026lt;Signature (*args, **kwargs)\u0026gt; When we inspect the name and signature of the special_greeting function, we instead receive the name and signature of the decorator that was applied to it. While confusing, it becomes even more of a headache if you need to debug this code. This is easily fixed with the functools.wraps decorator (Lines 2,11):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24  \u0026#34;\u0026#34;\u0026#34;decorators.class_based\u0026#34;\u0026#34;\u0026#34; from functools import wraps  class DecoratorFactory: def __init__(self, arg1, arg2): self.arg1 = arg1 self.arg2 = arg2 def __call__(self, function_to_decorate): @wraps(function_to_decorate)  def function_wrapper(*args, **kwargs): print(f\u0026#34;First argument provided to decorator: {self.arg1}\u0026#34;) function_to_decorate(*args, **kwargs) print(f\u0026#34;Second argument provided to decorator: {self.arg2}\u0026#34;) return function_wrapper @DecoratorFactory(\u0026#34;foo\u0026#34;, \u0026#34;bar\u0026#34;) def special_greeting(name=\u0026#34;Dennis\u0026#34;): print(f\u0026#34;Allow me to give a very special welcome to {name}!\u0026#34;)   Now, if we inspect special_greeting we will see the correct name and signature:\n(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.  from decorators.class_based import *   import inspect  special_greeting.__name__ 'special_greeting'  inspect.signature(special_greeting) \u0026lt;Signature (name='Dennis')\u0026gt;   Why did I not include the functools.wraps decorator in the previous examples? Since the point was to explain how decorators work, seeing a random decorator in the middle of everything would have been confusing and would have drawn attention away from the core concepts being demonstrated.\n  Hopefully you have a better understanding of how decorators are designed and how they behave in Python. The remainder of this post will contain examples of decorators that I frequently use in my Python projects along with pytest functions that demonstrate the intended usage.\nExample: Function Timeout If you have ever written code that interacts with an outside service, you have probably encountered a situation where your program becomes stuck waiting for a response with no way to abort the function call. One way to get un-stuck is with the @timeout decorator:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26  \u0026#34;\u0026#34;\u0026#34;decorators.timeout\u0026#34;\u0026#34;\u0026#34; from functools import wraps from signal import signal, alarm, SIGALRM def timeout(*, seconds=3, error_message=\u0026#34;Call to function timed out!\u0026#34;): \u0026#34;\u0026#34;\u0026#34;Abort the wrapped function call after the specified number of seconds have elapsed.\u0026#34;\u0026#34;\u0026#34; def _handle_timeout(signum, frame): raise TimeoutError(error_message) def decorator(func): @wraps(func) def wrapper(*args, **kwargs): signal(SIGALRM, _handle_timeout) alarm(seconds) try: result = func(*args, **kwargs) finally: alarm(0) return result return wrapper return decorator     The syntax used in Line 6 above prevents the user from specifying positional arguments with the @timeout decorator. This is done to ensure that the purpose of the decorator is clear to the reader based on its usage. Without the asterisk (*) in the function definition, the user could decorate a function with @timeout(3), rather than @timeout(seconds=3). The latter usage communicates the meaning of the value 3 in the context of the @timeout decorator. If you are unfamiliar with this syntax, you can find the background and justification for this feature in PEP 3102.\n  It\u0026rsquo;s very easy to use. Simply decorate any function with @timeout(seconds=X) and specify the number of seconds to wait before aborting the function call as shown below (Line 7). If the function completes before the specified number of seconds have elapsed, your program will continue executing. However if the function has not completed after the specified number of seconds have elapsed, a TimeoutError will be raised, aborting the function call.\nIn the simple test scenario below, the sleep function waits for two seconds when called. Since it is decorated with @timeout(seconds=1) a TimeoutError will be raised one second after it is called. The test_timeout function verifies that the correct error is raised (Lines 13-14).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15  \u0026#34;\u0026#34;\u0026#34;tests.test_timeout\u0026#34;\u0026#34;\u0026#34; import time import pytest from decorators.timeout import timeout @timeout(seconds=1) def sleep(): time.sleep(2) def test_timeout(): with pytest.raises(TimeoutError): sleep()   Example: Retry Function The next example is similar to the @timeout decorator since both are designed to handle functions that are unreliable. The @retry decorator adds retry logic to the decorated function.\nTo use it, you specify a set of exceptions that can trigger a failed attempt, the number of failed attempts that can occur before aborting the function call (max_attempts), the number of seconds to delay after each failed attempt before trying again (delay) and an optional handler method to be called whenever an exception is raised (for logging, etc).\nThe actual decorator definition begins on Line 26 below. Before that, the custom Exception RetryLimitExceededError is defined (Lines 6-13). This is the exception raised after max_attempts to call the function have failed. The handle_failed_attempt function (Lines 16-23) is provided as an example of what could be provided to the @retry decorator\u0026rsquo;s on_failure parameter.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48  \u0026#34;\u0026#34;\u0026#34;decorators.retry\u0026#34;\u0026#34;\u0026#34; from functools import wraps from time import sleep class RetryLimitExceededError(Exception): \u0026#34;\u0026#34;\u0026#34;Custom error raised by retry decorator when max_attempts have failed.\u0026#34;\u0026#34;\u0026#34; def __init__(self, func, max_attempts): message = ( f\u0026#34;Retry limit exceeded! (function: {func.__name__}, max attempts: {max_attempts})\u0026#34; ) super().__init__(message) def handle_failed_attempt(func, remaining, ex, delay): \u0026#34;\u0026#34;\u0026#34;Example function that could be supplied to on_failure attribute of retry decorator.\u0026#34;\u0026#34;\u0026#34; message = ( f\u0026#34;Function name: {func.__name__}\\n\u0026#34; f\u0026#34;Error: {repr(ex)}\\n\u0026#34; f\u0026#34;{remaining} attempts remaining, retrying in {delay} seconds...\u0026#34; ) print(message) def retry(*, max_attempts=2, delay=1, exceptions=(Exception,), on_failure=None): \u0026#34;\u0026#34;\u0026#34;Retry the wrapped function when an exception is raised until max_attempts have failed.\u0026#34;\u0026#34;\u0026#34; def decorator(func): @wraps(func) def wrapper(*args, **kwargs): for remaining in reversed(range(max_attempts)): try: return func(*args, **kwargs) except exceptions as ex: if remaining \u0026gt; 0: if on_failure: on_failure(func, remaining, ex, delay) sleep(delay) else: raise RetryLimitExceededError(func, max_attempts) from ex else: break return wrapper return decorator   We can use the @timeout decorator to demonstrate and test the @retry decorator. Since we need to know the type of Exception that we expect to occur when we call the decorated function, we specify exceptions=(TimeoutError,) since this is the error raised by the @timeout decorator.\nWith the code below, we will attempt to call retry_with_timeout a maximum of two times. Since the only thing this function does is wait for two seconds and we have decorated it with @timeout(seconds=1), calling it will always raise a TimeoutError. Therefore, after the second failed attempt, the @retry decorator will raise a RetryLimitExceededError.\nFinally, the test_retry_with_timeout function verifies that the RetryLimitExceededError is in fact raised after calling the retry_with_timeout function.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19  \u0026#34;\u0026#34;\u0026#34;tests.test_retry\u0026#34;\u0026#34;\u0026#34; import time import pytest from decorators.retry import retry, RetryLimitExceededError from decorators.timeout import timeout @retry(max_attempts=2, delay=1, exceptions=(TimeoutError,)) @timeout(seconds=1) def retry_with_timeout(): time.sleep(2) def test_retry_with_timeout(): with pytest.raises(RetryLimitExceededError): retry_with_timeout()   Example: Log Call Signature and Execution Time The most common application of decorators might be logging. It\u0026rsquo;s easy to see why, having the ability to run code immediately before and after a function is called allows you to report information and capture metrics.\nThis decorator uses the class-based design, and allows the user to provide a custom logger. If none is provided, a logger will be created based on the module that contains the wrapped function.\nWhenever the function is called, the @LogCall() decorator adds an info level entry the the log with the following data:\n Timestamp when the function was called function name and values of all arguments provided to the function (i.e. the call signature) Time elapsed while executing the function  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58  \u0026#34;\u0026#34;\u0026#34;decorators.log_call\u0026#34;\u0026#34;\u0026#34; import inspect import logging from datetime import datetime from functools import wraps DT_NAIVE = \u0026#34;%Y-%m-%d%I:%M:%S %p\u0026#34; class LogCall: \u0026#34;\u0026#34;\u0026#34;Log call signature and execution time of decorated function.\u0026#34;\u0026#34;\u0026#34; def __init__(self, logger=None): self.logger = logger def __call__(self, func): if not self.logger: logging.basicConfig() self.logger = logging.getLogger(func.__module__) self.logger.setLevel(logging.INFO) @wraps(func) def wrapper(*args, **kwargs): func_call_args = get_function_call_args(func, *args, **kwargs) exec_start = datetime.now() result = func(*args, **kwargs) exec_finish = datetime.now() exec_time = format_timedelta_str(exec_finish - exec_start) exec_start_str = exec_start.strftime(DT_NAIVE) self.logger.info(f\u0026#34;{exec_start_str} | {func_call_args} | {exec_time}\u0026#34;) return result def get_function_call_args(func, *args, **kwargs): \u0026#34;\u0026#34;\u0026#34;Return a string containing function name and list of all argument names/values.\u0026#34;\u0026#34;\u0026#34; func_args = inspect.signature(func).bind(*args, **kwargs) func_args.apply_defaults() func_args_str = \u0026#34;, \u0026#34;.join(f\u0026#34;{arg}={val}\u0026#34; for arg, val in func_args.arguments.items()) return f\u0026#34;{func.__name__}({func_args_str})\u0026#34; def format_timedelta_str(td): \u0026#34;\u0026#34;\u0026#34;Convert timedelta to an easy-to-read string value.\u0026#34;\u0026#34;\u0026#34; (milliseconds, microseconds) = divmod(td.microseconds, 1000) (minutes, seconds) = divmod(td.seconds, 60) (hours, minutes) = divmod(minutes, 60) if td.days \u0026gt; 0: return f\u0026#34;{td.days}d {hours:.0f}h {minutes:.0f}m {seconds}s\u0026#34; if hours \u0026gt; 0: return f\u0026#34;{hours:.0f}h {minutes:.0f}m {seconds}s\u0026#34; if minutes \u0026gt; 0: return f\u0026#34;{minutes:.0f}m {seconds}s\u0026#34; if td.seconds \u0026gt; 0: return f\u0026#34;{td.seconds}s {milliseconds:.0f}ms\u0026#34; if milliseconds \u0026gt; 0: return f\u0026#34;{milliseconds}ms\u0026#34; return f\u0026#34;{td.microseconds}us\u0026#34; return wrapper   The code below tests the @LogCall() decorator with a custom logger and with the default logger. With the default logger, we expect the name of the logger to be the name of the module containing the decorated function, tests.test_log_call (Line 30). When a custom logger is provided, we expect the name to match the value we specified when the logger was created, custom_log (Lines 10, 38).\nA nice feature of this decorator is that the function call signature contains the names and values of all keyword arguments, even if a default value was used or if the name was not given when the call occurred. For example, the call to the decorated function in Line 28 is save_values(\u0026quot;Aaron\u0026quot;, \u0026quot;Charlie\u0026quot;, \u0026quot;Ollie\u0026quot;), but the call signature that is logged contains the names of all three arguments, save_values(a=Aaron, b=Charlie, c=Ollie) (Line 32).\nSimilarly, the call to the decorated function in Line 36 is rand_time(max=4, add_random=True), which only provides two arguments. The call signature that is logged includes the default value of the missing argument, rand_time(min=1, max=4, add_random=True) (Line 40).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41  \u0026#34;\u0026#34;\u0026#34;tests.test_log_call\u0026#34;\u0026#34;\u0026#34; import time import logging from random import randint from decorators.log_call import LogCall logging.basicConfig() log = logging.getLogger(\u0026#34;custom_log\u0026#34;) log.setLevel(logging.INFO) log.info(\u0026#34;logging started\u0026#34;) @LogCall() def save_values(a, b, c): pass @LogCall(log) def rand_time(min=1, max=3, add_random=False): time.sleep(randint(min, max)) if add_random: time.sleep(randint(100, 500) / 1000.0) def test_default_logger(caplog): save_values(\u0026#34;Aaron\u0026#34;, \u0026#34;Charlie\u0026#34;, \u0026#34;Ollie\u0026#34;) logger, level, message = caplog.record_tuples[-1] assert logger == \u0026#34;tests.test_log_call\u0026#34; assert level == logging.INFO assert \u0026#34;save_values(a=Aaron, b=Charlie, c=Ollie)\u0026#34; in message def test_custom_logger(caplog): rand_time(max=4, add_random=True) logger, level, message = caplog.record_tuples[-1] assert logger == \u0026#34;custom_log\u0026#34; assert level == logging.INFO assert \u0026#34;rand_time(min=1, max=4, add_random=True)\u0026#34; in message   If we run pytest for these decorator examples, all of the tests pass:\n(venv) decorators $ pytest tests/test_* ================================================================= test session starts ================================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Desktop/vigorish/venv/bin/python cachedir: .pytest_cache rootdir: /Users/aaronluna/Desktop/vigorish, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, mypy-0.5.0, dotenv-0.4.0, flake8-1.0.4, cov-2.8.1 collected 10 items tests/test_log_call.py::FLAKE8 PASSED [ 10%] tests/test_log_call.py::BLACK PASSED [ 20%] tests/test_log_call.py::test_default_logger PASSED [ 30%] tests/test_log_call.py::test_custom_logger PASSED [ 40%] tests/test_retry.py::FLAKE8 PASSED [ 50%] tests/test_retry.py::BLACK PASSED [ 60%] tests/test_retry.py::test_retry_with_timeout PASSED [ 70%] tests/test_timeout.py::FLAKE8 PASSED [ 80%] tests/test_timeout.py::BLACK PASSED [ 90%] tests/test_timeout.py::test_timeout PASSED [100%] ================================================================== 10 passed in 6.63s ==================================================================  Summary If you would like to download all or some of the code from this post, you can easily do so from the Github gist linked below:\nPython Decorators  I hope this introduction to decorators in Python was helpful and easy to understand. If you have any questions, criticism or feedback please leave a comment. Thanks!\n","href":"/blog/intro-to-python-decorators/","title":"An Introduction to Decorators in Python"},{"categories":"svelte ; Javascript ; my-projects","content":"I am currently teaching myself svelte and I have enjoyed pretty much everything about it. I decided to create a simple application that encodes ASCII text or hex strings to base64, and vice-versa. In order to take advantage of svelte\u0026rsquo;s strengths, the application contains reactive UI components to help illustrate the encoding process.\nLaunch App View Github Repo  I deployed this project to Netlify (click the Launch App button above to open the application in a new tab/window). I also included a button to view/fork the project with codesandbox (below), please feel free to log a github issue if you find any bugs. Thank you!\n  Base64 Encoder/Decoder  Input/output strings displayed in hex, decimal, binary, and base64 to demonstrate how input bytes are encoded to base64. Mouseover/touch any part of the Hex/Base64 output to highlight all related bit groups and the matching base64/ASCII characters in the Lookup Tables. Hex strings must contain only numbers and/or upper and lowercase hex digits (a-f, A-F, 0-9). Hex strings can be prefixed by \u0026ldquo;0x\u0026rdquo;, but this is not required.  e.g., 0xFE and FE both represent the value 254 and both will produce the same output when encoded to base64   Encoded strings must be valid base64 (standard) or base64url (url/filename safe) values If a decoded string is composed entirely of ASCII-printable characters, it will be rendered as text. If a decoded string contains a single byte that is outside of the range of ASCII-printable characters (0x20 - 0x7E), the entire string will be rendered as hex digits.  CSS Preprocessing \u0026amp; Rollup Config  Svelte 3 + Svelma integrated via svelte-preprocess.  Svelma is a set of UI components for Svelte based on the Bulma CSS framework.   Bulma/FontAwesome 5 integrated via node-sass SASS/SCSS files are bundled and copied to public/build folder (node-sass, postcss/cssnano). FontAwesome font files are copied from from node_modules folder to public/webfonts folder. When building for production, bundled CSS and JS files are minified. resolve, commonjs, and terser rollup plugins configured.  Tests  Cypress E2E tests created for basic encode/decode scenarios. 5 test cases are each executed with 4 different screen types/orientations. Input and expected output for both ASCII and Hex strings taken directly from the Examples and Illustrations and Test Vectors sections of RFC4648 which is the original specification for Base64 and other print-safe binary encodings. Github action configured to run all Cypress tests on each push to master branch, build is deployed only if all tests pass.  Sorry, your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!  Automated Cypress Tests (4 Tests on 2 Screen Sizes Shown)\n   ","href":"/projects/svelte-base64/","title":"My First Svelte Application: Base64 Encoder/Decoder"},{"categories":"Hugo ; JavaScript","content":"Hugo includes a built-in syntax-highlighter called Chroma. Chroma is extremely fast since it is written in pure Go (like Hugo) and supports every language I can think of. Chroma\u0026rsquo;s speed is especially important since syntax highlighters are notorious for causing slow page loads. However, it lacks one vital feature — an easy way to copy a code block to the clipboard. I decided to document my implementation using only vanilla JS (every blog post I found for this issue relied on jquery to parse the DOM, which is completely unnecessary at this point).\nThe finished product can be seen/modified with the codepen below:\nSee the Pen Add Copy Button to Chroma (Hugo) Code Blocks by Aaron Luna (@a-luna) on CodePen.  A quick search led me to this post on Danny Guo\u0026rsquo;s blog. I used his example as my starting point but made several changes:\n The \"copy\" button is placed within the code block rather than outside it. Instead of polyfilling the Clipboard API, my implementation falls back to using document.execCommand(\"copy\") if it is unsupported. \"Copy\" buttons are only added to code elements that are generated by Chroma.  The Hugo highlight shortcode accepts a line-nos parameter. If line-nos is not specified or line-nos=inline, the rendered HTML has this structure:\n\u0026lt;div class=\u0026#34;highlight\u0026#34;\u0026gt; \u0026lt;pre class=\u0026#34;chroma\u0026#34;\u0026gt; \u0026lt;code class=\u0026#34;language-xxxx\u0026#34;\u0026gt; (the code we wish to copy) \u0026lt;/code\u0026gt; \u0026lt;/pre\u0026gt; \u0026lt;/div\u0026gt; If line-nos=table, the HTML is slightly more complicated:\n\u0026lt;div class=\u0026#34;highlight\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;chroma\u0026#34;\u0026gt; \u0026lt;table class=\u0026#34;lntable\u0026#34;\u0026gt; \u0026lt;tbody\u0026gt; \u0026lt;tr\u0026gt; \u0026lt;td class=\u0026#34;lntd\u0026#34;\u0026gt; (line numbers are rendered here) \u0026lt;/td\u0026gt; \u0026lt;td class=\u0026#34;lntd\u0026#34;\u0026gt; \u0026lt;pre class=\u0026#34;chroma\u0026#34;\u0026gt; \u0026lt;code class=\u0026#34;language-xxxx\u0026#34;\u0026gt; (the code we wish to copy) \u0026lt;/code\u0026gt; \u0026lt;/pre\u0026gt; \u0026lt;/td\u0026gt; \u0026lt;/tr\u0026gt; \u0026lt;/tbody\u0026gt; \u0026lt;/table\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; I use the version with line numbers much more often than the version without, so it is important to me to support both. I decided to place the button inside the \u0026lt;div class=\u0026ldquo;highlight\u0026rdquo;\u0026gt; element. Stacking elements on top of one another requires positioning and assigning z-index values, which you can see below along with the styling for the \u0026ldquo;copy\u0026rdquo; button:\n.highlight-wrapper { display: block; } .highlight { position: relative; z-index: 0; padding: 0; margin: 0; border-radius: 4px; } .highlight \u0026gt; .chroma { color: #d0d0d0; background-color: #212121; position: static; z-index: 1; border-radius: 4px; padding: 10px; } .chroma .lntd:first-child { padding: 7px 7px 7px 10px; margin: 0; } .chroma .lntd:last-child { padding: 7px 10px 7px 7px; margin: 0; } .copy-code-button { position: absolute; z-index: 2; right: 0; top: 0; font-size: 13px; font-weight: 700; line-height: 14px; letter-spacing: 0.5px; width: 65px; color: #232326; background-color: #7f7f7f; border: 1.25px solid #232326; border-top-left-radius: 0; border-top-right-radius: 4px; border-bottom-right-radius: 0; border-bottom-left-radius: 4px; white-space: nowrap; padding: 4px 4px 5px 4px; margin: 0 0 0 1px; cursor: pointer; opacity: 0.6; } .copy-code-button:hover, .copy-code-button:focus, .copy-code-button:active, .copy-code-button:active:hover { color: #222225; background-color: #b3b3b3; opacity: 0.8; } .copyable-text-area { position: absolute; height: 0; z-index: -1; opacity: .01; } Did you notice that the CSS includes a selector for a highlight-wrapper class that is not present in the HTML structure generated by Chroma? We will create this element and append the positioned elements as a child node, then insert the wrapper into the DOM in place of the \u0026lt;div class=\u0026ldquo;highlight\u0026rdquo;\u0026gt; element.\nSimilarly, the copyable-text-area class will be applied to a textarea element that will only exist if the Clipboard API is not available. This element will be added to the DOM and have it\u0026rsquo;s value set to the innerText value of the code we wish to copy. After copying the text, the textarea element will be removed fom the DOM. The height: 0 and opacity: .01 stylings make it virtually invisible, and z-index: -1 places it behind the code block.\nWith that in mind, let\u0026rsquo;s take a look at the JavaScript:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61  function createCopyButton(highlightDiv) { const button = document.createElement(\u0026#34;button\u0026#34;); button.className = \u0026#34;copy-code-button\u0026#34;; button.type = \u0026#34;button\u0026#34;; button.innerText = \u0026#34;Copy\u0026#34;; button.addEventListener(\u0026#34;click\u0026#34;, () =\u0026gt; copyCodeToClipboard(button, highlightDiv)); addCopyButtonToDom(button, highlightDiv); } async function copyCodeToClipboard(button, highlightDiv) { const codeToCopy = highlightDiv.querySelector(\u0026#34;:last-child \u0026gt; .chroma \u0026gt; code\u0026#34;).innerText; try { result = await navigator.permissions.query({ name: \u0026#34;clipboard-write\u0026#34; }); if (result.state == \u0026#34;granted\u0026#34; || result.state == \u0026#34;prompt\u0026#34;) { await navigator.clipboard.writeText(codeToCopy); } else { copyCodeBlockExecCommand(codeToCopy, highlightDiv); } } catch (_) { copyCodeBlockExecCommand(codeToCopy, highlightDiv); } finally { codeWasCopied(button); } } function copyCodeBlockExecCommand(codeToCopy, highlightDiv) { const textArea = document.createElement(\u0026#34;textArea\u0026#34;); textArea.contentEditable = \u0026#39;true\u0026#39; textArea.readOnly = \u0026#39;false\u0026#39; textArea.className = \u0026#34;copyable-text-area\u0026#34;; textArea.value = codeToCopy; highlightDiv.insertBefore(textArea, highlightDiv.firstChild); const range = document.createRange() range.selectNodeContents(textArea) const sel = window.getSelection() sel.removeAllRanges() sel.addRange(range) textArea.setSelectionRange(0, 999999) document.execCommand(\u0026#34;copy\u0026#34;); highlightDiv.removeChild(textArea); } function codeWasCopied(button) { button.blur(); button.innerText = \u0026#34;Copied!\u0026#34;; setTimeout(function() { button.innerText = \u0026#34;Copy\u0026#34;; }, 2000); } function addCopyButtonToDom(button, highlightDiv) { highlightDiv.insertBefore(button, highlightDiv.firstChild); const wrapper = document.createElement(\u0026#34;div\u0026#34;); wrapper.className = \u0026#34;highlight-wrapper\u0026#34;; highlightDiv.parentNode.insertBefore(wrapper, highlightDiv); wrapper.appendChild(highlightDiv); } document.querySelectorAll(\u0026#34;.highlight\u0026#34;) .forEach(highlightDiv =\u0026gt; createCopyButton(highlightDiv));   So what is happening here? Whenever a page is loaded, all \u0026lt;div class=\u0026ldquo;highlight\u0026rdquo;\u0026gt; elements are located and a \u0026ldquo;Copy\u0026rdquo; button is created for each. Then, the copyCodeToClipboard function is assigned as the event handler for the button\u0026rsquo;s click event (Lines 2-6). Finally, some DOM manipulation is performed by calling addCopyButtonToDom. Let\u0026rsquo;s examine how this function works:\n  Line 53: First, the \"Copy\" button is inserted into the DOM as the first child of the \u0026lt;div class=\"highlight\"\u0026gt; element.\n  Line 54-55: We create a div element to act as a wrapper for the \u0026lt;div class=\"highlight\"\u0026gt; element and assign the appropriate styling.\n  Line 56-57: Finally, the wrapper element is inserted into the DOM in the same location as the \u0026lt;div class=\"highlight\"\u0026gt; element, and the \u0026lt;div class=\"highlight\"\u0026gt; element is \"wrapped\" by calling appendChild.\n   When the user clicks a \u0026ldquo;Copy\u0026rdquo; button, the copyCodeToClipboard function is called. Since the logic that determines which copy function to use may not seem intuitive, let\u0026rsquo;s go through it together:\n  Lines 13-14: Within the try/catch block, we first check if the clipboard-write permission has been granted.\n  Line 15: If the browser supports the Clipboard API and the clipboard-write permission has been granted, the text within the code block is copied to the clipboard by calling navigator.clipboard.writeText.\n  Line 17: If the browser supports the Clipboard API but the clipboard-write permission has not been granted, we call copyCodeBlockExecCommand.\n  Line 20: If the browser does not support the Clipboard API, an error will be raised and caught. Since this is an expected failure, the error is not re-thrown, and the same action that is performed when the browser supports the Clipboard API but the clipboard-write permission has not been granted is executed \u0026mdash; within the catch block we call copyCodeBlockExecCommand.\n  Line 23: Code within a finally block is called after either try/catch, regardless of result. Since the \"copy\" operation was invoked in either try/catch block, we call codeWasCopied which changes the button text to \"Copied!\". After two seconds the button text is changed back to \"Copy\".\n  Line 28-31: When the Clipboard API is unsupported/permission is not granted, we create a textarea element and assign the appropriate styling to make it hidden from the user but still available programmatically.\n  Line 32: We set the value of the textarea element to be equal the text the user wishes to copy.\n  Line 33: The textarea element is temporarily aded to the DOM next to the copy button.\n  Line 34-39: While testing my code, I found that it worked correctly on all browsers on desktop. However, on my iPhone the text wasn't being copied. I researched this issue and found the steps performed in these lines are needed.\n  Line 40-41: The textarea element is selected before calling document.execCommand(\"copy\"), which copies the text we assigned to the textarea element to the clipboard. After doing so, the textarea element is removed from the DOM.\n   On this site, the JavaScript in the code block above is bundled with other js files and minified. If you\u0026rsquo;d like, you can verify the code and debug it using your browser\u0026rsquo;s dev tools on any page that contains a code block (the easiest way would be to inspect the copy button, find the event listeners attached to it and add breakpoints to the method attached to the click handler). I hope this is helpful to you if you use Hugo and have run into the same problem, please leave any feedback/questions in the comments below!\n","href":"/blog/add-copy-button-to-code-blocks-hugo-chroma/","title":"Hugo: Add Copy-to-Clipboard Button to Code Blocks with Vanilla JS"},{"categories":"Python","content":"In a previous post, I presented a C# Result class that represents the outcome of an operation. This class is intended to be used for error handling as an alternative to throwing and handling exceptions. I was introduced to this concept by a post from the Enterprise Craftsmanship blog. I recommend reading the entire post, which is part of a series examining how principles from Functional Programming can be applied to C#.\nI thought it would be interesting to implement the Result class in Python, and since Python is dynamically-typed this ended up being much simpler than the C# implementation which required the use of generic types. The entire implementation is given below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49  \u0026#34;\u0026#34;\u0026#34;app.util.result\u0026#34;\u0026#34;\u0026#34; class Result(): \u0026#34;\u0026#34;\u0026#34;Represents the outcome of an operation. Attributes ---------- success : bool A flag that is set to True if the operation was successful, False if the operation failed. value : object The result of the operation if successful, value is None if operation failed or if the operation has no return value. error : str Error message detailing why the operation failed, value is None if operation was successful. \u0026#34;\u0026#34;\u0026#34; def __init__(self, success, value, error): self.success = success self.error = error self.value = value @property def failure(self): \u0026#34;\u0026#34;\u0026#34;True if operation failed, False if successful (read-only).\u0026#34;\u0026#34;\u0026#34; return not self.success def __str__(self): if self.success: return f\u0026#39;[Success]\u0026#39; else: return f\u0026#39;[Failure] \u0026#34;{self.error}\u0026#34;\u0026#39; def __repr__(self): if self.success: return f\u0026#34;\u0026lt;Result success={self.success}\u0026gt;\u0026#34; else: return f\u0026#39;\u0026lt;Result success={self.success}, message=\u0026#34;{self.error}\u0026#34;\u0026gt;\u0026#39; @classmethod def Fail(cls, error): \u0026#34;\u0026#34;\u0026#34;Create a Result object for a failed operation.\u0026#34;\u0026#34;\u0026#34; return cls(False, value=None, error=error) @classmethod def Ok(cls, value=None): \u0026#34;\u0026#34;\u0026#34;Create a Result object for a successful operation.\u0026#34;\u0026#34;\u0026#34; return cls(True, value=value, error=None)   The Result class encapsulates all information relevant to the outcome of an operation. For example, let\u0026rsquo;s say that we have a Result instance named result. If the operation which result represents failed, result.success will be False and result.error will contain a string detailing why the operation failed. If the operation succeeded, result.success will be True (result.error will be None). If the operation produced any output, this will be stored in result.value (an operation is not required to produce an output). Result objects are not intended to replace exception handling in all scenarios, and the author of the EC blog provides a simple rule to determine when each should be used:\n Use a Result object for expected failures that you know how to handle. Throw an exception when an unexpected error occurs.  To demonstrate how the Result class should be used, the function decode_auth_token in module app.util.auth validates an access token in JWT format. Please note the highlighted line numbers:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29  \u0026#34;\u0026#34;\u0026#34;app.util.auth\u0026#34;\u0026#34;\u0026#34; import jwt from app.config import key from app.models.blacklist_token import BlacklistToken from app.util.result import Result def decode_auth_token(access_token): \u0026#34;\u0026#34;\u0026#34;Decode an access token in JWT format.\u0026#34;\u0026#34;\u0026#34; result = check_blacklist(access_token) if result.failure: return result try: payload = jwt.decode(access_token, key) return Result.Ok(payload[\u0026#39;sub\u0026#39;]) except jwt.ExpiredSignatureError: error = \u0026#39;Access token expired. Please log in again.\u0026#39; return Result.Fail(error) except jwt.InvalidTokenError: error = \u0026#39;Invalid token. Please log in again.\u0026#39; return Result.Fail(error) def check_blacklist(access_token): exists = BlacklistToken.query.filter_by(token=str(access_token)).first() if exists: error = \u0026#39;Token blacklisted. Please log in again.\u0026#39; return Result.Fail(error) return Result.Ok()    Lines 11-13: If you call a function that returns a Result object, you should check the value of result.failure (or result.success). I prefer checking result.failure to reduce unnecessary indentation.  If the operation failed, you should handle the failure immediately or return the result object upstream until you reach an appropriate place to handle and/or report the failure. If the operation was successful and you expect the function to return a value, you can retrieve it by calling result.value. If no value is expected, (as is the case for the check_blacklist function) you simply keep executing your current function.   Lines 16 and 29: To indicate that a function (operation) was successful, the function should return Result.Ok(). You may have noticed in the Result class that providing a value as a parameter is optional. If the successful operation produces a result (e.g. payload['sub']) the client can retrieve it from result.value. Lines 19, 22, 28 In the case of decoding a json web token, we expect exceptions jwt.ExpiredSignatureError and jwt.InvalidTokenError to occur and we know how to handle them (Deny the user from performing the requested action and prompt them to re-authenticate). This is the exact use case we defined for the Result class earlier in this post. To indicate that a function has failed, return Result.Fail(error) (error should be a message explaining why the operation failed).  The Python REPL code below demonstrates how the decode_auth_token function behaves and how to interact with the Result objects that the function returns:\n access_token = request.headers.get('Authorization')  result = decode_auth_token(access_token)  result Result\u0026lt;success=True\u0026gt;  result.success True  result.value '570eb73b-b4b4-4c86-b35d-390b47d99bf6'  result.failure False  result.error  print(result) [Success]  exit()  auth_token_bad = request.headers.get('Authorization')  result = decode_auth_token(auth_token_bad)  result Result\u0026lt;success=False, message=\"Invalid token. Please log in again.\"\u0026gt;  result.success False  result.value  result.failure True  result.error 'Invalid token. Please log in again.'  print(result) [Failure] \"Invalid token. Please log in again.\"  exit()  auth_token_expired = request.headers.get('Authorization')  result = decode_auth_token(auth_token_expired)  result Result\u0026lt;success=False, message=\"Access token expired. Please log in again.\"\u0026gt;  result.success False  result.value  result.failure True  result.error 'Access token expired. Please log in again.'  print(result) [Failure] \"Access token expired. Please log in again.\"  exit() I find that code becomes easier to read and digest visually when the Result class is incorporated. It becomes easier to discern what happens when a failure occurs and how the failure is handled, in contrast to a design that favors exception handling as the primary method of error handling.\nI have taken the time to explain the Python version of the Result class because it will be referenced frequently in upcoming posts. As always, please give me your feedback or questions in the comments!\n","href":"/blog/error-handling-python-result-class/","title":"Error Handling in Python: Result Class"},{"categories":"DevOps","content":"If you create Heroku apps, you know that the only built-in options for deployment are 1) the Heroku CLI or 2) integration with a github repo. If you don\u0026rsquo;t want your application\u0026rsquo;s code to be publicly accessible, this results in a tedious routine of manually pushing your changes to Heroku. With Azure DevOps, you can create flexible build pipelines to automate various tasks: running tests, creating binaries, publishing artifacts, etc. In this post, I will go through the steps to create a simple build pipeline that automatically pushes the latest code to Heroku when changes are committed to a Azure DevOps repo.\nCreating a New Build Pipeline Creating a project in Azure DevOps gives you more than a private git repository. With Build Pipelines, you can quickly create a CICD process. Navigate to your project\u0026rsquo;s build piplelines by clicking on the blue rocketship icon. If you have never created a build pipeline for this project, you will be directed to create one as shown in Figure 1. Otherwise, select New -\u0026gt; New build pipeline as shown in Figure 2:\n Figure 1 - First build pipeline\n    Figure 2 - New build pipeline\n    First, you must select the repository that contains your heroku app. Leave Azure Repos Git selected and choose the repo from the dropdown list. Then, click Continue (Figure 3):\n Figure 3 - Select a source repository\n  Next, you are asked to select a template. You can find templates for many technologies/tools, but since we are creating a simple CD process, scroll to the bottom and select Empty pipeline (Figure 4):\n Figure 4 - Select a Template\n  You should see the empty pipeline shown in Figure 5. Notice that you are currently viewing the Tasks section of the pipeline. Change the name to \u0026ldquo;Deploy to Heroku\u0026rdquo; or anything else you like. You can leave the value for Agent pool as Hosted VS2017:\n Figure 5 - Empty pipeline\n  Enabling Continous Integration Currently, this pipeline is not configured to run in response to any trigger. In fact, it will never run unless you setup a schedule or enable continuous integration. To do so, Click Triggers and select the checkbox for Enable continuous integration as shown in Figure 6.\n Figure 6 - Trigger settings\n  Now this pipeline will run whenever a commit is made to the master branch of your repo. You can make this behavior more fine-grained if you wish, by selecting a different branch or adding a path filter in the Branch Filters section.\nObtaining a Heroku API Token In order to make code changes to your Heroku app, you must authenticate yourself. Normally this is handled for you by the Heroku CLI. When you first run heroku login an API token is generated and stored in a ~/.netrc file on your machine. To see the value of your API token, you can run heroku auth:token or cat ~/.netrc. This default token is silently used for all further CLI interactions with Heroku, and it can also be used to push changes from our build pipleline.\nHowever, there are two reasons why this may not be the best token to use for our CD process. 1) The default token expires one year after being generated. 2) The default token will be immediately expired if you run heroku logout. Obviously we do not want our build pipeline to fail unexpectedly when the default token expires. To avoid this, we can create a token which never expires with the heroku authorizations:create command. For more information on Heroku API tokens, please read this support article.\nAs shown below, creating a API token is simple. You can also add a description to remind yourself of the intended use. For information on the possible options/values with the heroku authorizations:create command, please see the official docs.\n$ heroku authorizations:create --description=\"azure cd token\" --short 787f6805-b74a-45a0-bc58-f6a187bacff2 $ heroku authorizations Heroku CLI 5b570a3c-3d2a-4b64-af97-7c1c69fd8f2e global azure cd token 787f6805-b74a-45a0-bc58-f6a187bacff2 global Generate a new API token and have the value ready before moving on to the next step.\nCreating a Pipeline Variable Ok, back at the Azure Build Pipeline, click Variables and create a new Pipeline variable as shown in Figure 7. I recommend using a simple, short name for any build variables, so I named mine pat (personal access token).\n  Since exposing the API token would be no different than exposing the password to your Heroku account, make sure to click the lock icon to make the variable secret as shown in Figures 7 and 8. This will prevent the API token from appearing in build logs.\n   Figure 7 - Create pipeline variable for API token\n   Figure 8 - Make pipeline variable secret\n  Configuring the Build Task It might seem like we\u0026rsquo;ve done a lot of work up to this point, but right now our pipeline doesn\u0026rsquo;t actually do anything. Click on Tasks to return to the empty build template. Click on the  symbol to add a task to Agent job 1 as shown in Figure 9:\n Figure 9 - Add a build task to Agent job 1\n  The list of tasks is pretty huge, so filter it by typing \u0026ldquo;power\u0026rdquo; or \u0026ldquo;powershell\u0026rdquo; into the search box as shown in Figure 10. Then, click the Add button to add a blank Powershell task to Agent job 1:\n Figure 10 - Add a Powershell task\n  This task requires either a filepath to a Powershell script or an inline Powershell script that executes when the build is triggered. Click the radio button for Inline as shown in Figure 11:\n Figure 11 - Select inline for script type\n  What do we need our Powershell script to do, exactly? In order to push our updated code to Heroku, our script needs to do three things:\n Checkout the branch which was just updated Add a remote for our heroku git repo Push the current version of the updated branch to the heroku remote (authenticating our account with the API token we added as a pipeline variable).  Using Predefined Build Variables Azure DevOps provides a set of predefined variables which can be referenced in build pipelines. For our Powershell script, we need to reference the name of the branch which was updated, and luckily there is a build variable which contains that exact information.\nBuild.SourceBranchName is the name of the branch the build was queued for ... The last path segment in the ref. For example, in refs/heads/master this value is master.  We can insert build or pipeline variables into our Powershell script by wrapping the name of the variable in $().\nFinalizing the Powershell Script We have everything we need to create a Powershell script that pushes our latest changes to Heroku. If the name of your Heroku app is test_app and your pipeline variable containing your Heroku API token is named pat, you would enter the following three lines into the Powershell task as shown in Figure 12 (obviously, modify the script to match the name of your Heroku app and pipeline variable):\ngit checkout $(Build.SourceBranchName) git remote add heroku https://heroku:$(pat)@git.heroku.com/test_app.git git push heroku $(Build.SourceBranchName)   Figure 12 - Inline Powershell script to push changes to Heroku\n  It\u0026rsquo;s a good idea to doublecheck your work at this point. When you are confident that everyting is configured correctly, Save \u0026amp; queue your build as shown in Figure 13:\n Figure 13 - Save pipeline and queue build\n  Verify Deployment After the build is queued, click the link that appears, This will take you to build logs which update live as the deployment is running. If everything was configured correctly, you should eventually see that every step of the build pupeline succeeded as shown in Figure 14:\n Figure 14 - Successful build logs\n  To see detailed logs for the Heroku deployment process, click on Powershell Script. Scroll to the bottom of the log in the popup that appears and you should see something similar to Figure 15:\n Figure 15 - Heroku deployment logs\n  Note the revision number (v60) and commit ID (3cced9be) in the Powershell logs. You can confirm these same details in the Activity stream for your Heroku app (Figure 16):\n Figure 16 - Heroku activity stream\n  Conclusion To sum up, Azure DevOps provides a simple way to create a continuous deployment process for your Heroku app. The most important points to remember are:\nEnable continuous integration in the Triggers section of your build pipeline. If you forget to do this, your build process will fail to trigger when a new commit is made. Create a Heroku API token that is only used for authenticating automated deployment tasks. Add the API token as a secret variable to your build pipeline to prevent the token from being displayed in log messages. Inspect the log messages for your build if any errors occur.  I hope this post was helpful to you, let me know if you have any questions or suggestions in the comments! ","href":"/blog/continuously-deploy-heroku-azure-devops/","title":"How to Continuously Deploy a Heroku App with Azure DevOps"},{"categories":"AWS ; DevOps","content":"Build automation is a foundational concept within the culture of Continuous Delivey and DevOps. Implementing a successful build automation strategy can dramatically improve product quality and produce time savings in several areas. Packer is a powerful build automation tool that is relatively simple to integrate with your team\u0026rsquo;s workflow. The official website describes packer as follows:\nPacker is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.\n Installing packer is simple, visit this page for instructions as well as download links for all platforms. The packer template shown in this post builds an Amazon EC2 image, so create an account if you do not already have one. The image will use a t2.micro instance, which qualifies for the free-tier.\n  If your account is not elibigle for the free-tier or you have used the maximun number of EC2 hours for the current month, you can incur actual costs as a result of using these examples (The accumulated costs should be relatively minor, less than a dollar or a few dollars if you launch an instance and leave it running for a month).\n  I am warning you with peace and love, I am not responsible for any charges that are generated. If you are not comfortable with setting up an AWS account (credit card is required, even for the free-tier), I will be doing a follow-up post that uses Vagrant and Ansible to build VirtualBox images.\nPacker Template Format Before we begin, let\u0026rsquo;s define the components that make up a packer template in order to understand the role each component performs in the process of creating a machine image (The definitions below are copied from https://www.packer.io/docs/templates/index.html):\nA template is a JSON object that has a set of keys configuring various components of Packer. The available keys within a template are listed below. Along with each key, it is noted whether it is required or not.\n builders (required) is an array of one or more objects that defines the builders that will be used to create machine images for, and configures each of those builders. variables (optional) is an object of one or more key/value strings that defines user variables contained in the template. If it is not specified, then no variables are defined. provisioners (optional) is an array of one or more objects that defines the provisioners that will be used to install and configure software for the machines created by each of the builders. If it is not specified, then no provisioners will be run. post-processors (optional) is an array of one or more objects that defines the various post-processing steps to take with the built images. If not specified, then no post-processing will be done.  Github Repository If you would like to download the source code for the packer examples shown in this post, you can do so from the github repository linked below:\na-luna/packer-examples [github.com] In this post I will go through each section of the packer template file nginx_ubuntu_from_source.json, showing the expected console output from each component individually. I will not cover the shell scripts which build NGINX from source code and configure the installation, since this has already been documented in this post.\nHere is the complete template file, the line numbers will stay consistent throughout this post:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82  { \u0026#34;variables\u0026#34;: { \u0026#34;nginx_ver\u0026#34;: \u0026#34;1.15.6\u0026#34;, \u0026#34;pcre_ver\u0026#34;: \u0026#34;8.42\u0026#34;, \u0026#34;zlib_ver\u0026#34;: \u0026#34;1.2.11\u0026#34;, \u0026#34;openssl_ver\u0026#34;: \u0026#34;1.1.0h\u0026#34;, \u0026#34;geoip_ver\u0026#34;: \u0026#34;20180403\u0026#34;, \u0026#34;working_dir\u0026#34;: \u0026#34;/opt\u0026#34;, \u0026#34;src_folder\u0026#34;: \u0026#34;src_files\u0026#34;, \u0026#34;deb_pkg_folder\u0026#34;: \u0026#34;deb_pkg\u0026#34;, \u0026#34;log_folder\u0026#34;: \u0026#34;log\u0026#34;, \u0026#34;log_file\u0026#34;: \u0026#34;install_source.log\u0026#34; }, \u0026#34;builders\u0026#34;: [{ \u0026#34;type\u0026#34;: \u0026#34;amazon-ebs\u0026#34;, \u0026#34;access_key\u0026#34;: \u0026#34;{{user `aws_access_key`}}\u0026#34;, \u0026#34;secret_key\u0026#34;: \u0026#34;{{user `aws_secret_key`}}\u0026#34;, \u0026#34;region\u0026#34;: \u0026#34;us-west-1\u0026#34;, \u0026#34;source_ami_filter\u0026#34;: { \u0026#34;filters\u0026#34;: { \u0026#34;virtualization-type\u0026#34;: \u0026#34;hvm\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*\u0026#34;, \u0026#34;root-device-type\u0026#34;: \u0026#34;ebs\u0026#34; }, \u0026#34;owners\u0026#34;: [\u0026#34;099720109477\u0026#34;], \u0026#34;most_recent\u0026#34;: true }, \u0026#34;instance_type\u0026#34;: \u0026#34;t2.micro\u0026#34;, \u0026#34;ssh_username\u0026#34;: \u0026#34;ubuntu\u0026#34;, \u0026#34;ami_name\u0026#34;: \u0026#34;custom_nginx_ubuntu_{{timestamp}}\u0026#34;, \u0026#34;vpc_id\u0026#34;: \u0026#34;vpc-xxxxxxxx\u0026#34;, \u0026#34;subnet_id\u0026#34;: \u0026#34;subnet-xxxxxxxx\u0026#34;, \u0026#34;associate_public_ip_address\u0026#34;: \u0026#34;true\u0026#34; }], \u0026#34;provisioners\u0026#34;: [{ \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;inline\u0026#34;: [\u0026#34;sudo mkdir -p {{user `working_dir`}}/{{user `src_folder`}}\u0026#34;, \u0026#34;sudo chown ubuntu:ubuntu {{user `working_dir`}}/{{user `src_folder`}}\u0026#34;, \u0026#34;sudo mkdir -p {{user `working_dir`}}/{{user `deb_pkg_folder`}}/\u0026#34;, \u0026#34;sudo chown ubuntu:ubuntu {{user `working_dir`}}/{{user `deb_pkg_folder`}}\u0026#34;] },{ \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;./upload/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `deb_pkg_folder`}}\u0026#34; },{ \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;scripts\u0026#34;: [\u0026#34;./bash-scripts/00--nginx-prep_install.sh\u0026#34;, \u0026#34;./bash-scripts/01a-nginx-install_from_source.sh\u0026#34;, \u0026#34;./bash-scripts/02--nginx-configure_post_install.sh\u0026#34;], \u0026#34;expect_disconnect\u0026#34;: true, \u0026#34;environment_vars\u0026#34;: [ \u0026#34;NGINX_VER={{user `nginx_ver`}}\u0026#34;, \u0026#34;PCRE_VER={{user `pcre_ver`}}\u0026#34;, \u0026#34;ZLIB_VER={{user `zlib_ver`}}\u0026#34;, \u0026#34;OPENSSL_VER={{user `openssl_ver`}}\u0026#34;, \u0026#34;GEOIP_VER={{user `geoip_ver`}}\u0026#34;, \u0026#34;WORKING_DIR={{user `working_dir`}}\u0026#34;, \u0026#34;SRC_FOLDER={{user `src_folder`}}\u0026#34;, \u0026#34;DEB_PKG_FOLDER={{user `deb_pkg_folder`}}\u0026#34;, \u0026#34;LOG_FOLDER={{user `log_folder`}}\u0026#34;, \u0026#34;LOG_FILE={{user `log_file`}}\u0026#34;] },{ \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;script\u0026#34;: \u0026#34;./bash-scripts/03--nginx-verify_install.sh\u0026#34;, \u0026#34;pause_before\u0026#34;: \u0026#34;10s\u0026#34;, \u0026#34;environment_vars\u0026#34;: [ \u0026#34;WORKING_DIR={{user `working_dir`}}\u0026#34;, \u0026#34;LOG_FOLDER={{user `log_folder`}}\u0026#34;, \u0026#34;LOG_FILE={{user `log_file`}}\u0026#34;] },{ \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `deb_pkg_folder`}}/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;./download\u0026#34;, \u0026#34;direction\u0026#34;: \u0026#34;download\u0026#34; },{ \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `log_folder`}}/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;./download\u0026#34;, \u0026#34;direction\u0026#34;: \u0026#34;download\u0026#34; }] }   The github repository also contains a second template file, nginx_ubuntu_from_deb.json. I will briefly cover the differences between the two templates at the end of this post.\nUser Variables The first section of the template contains user variables which are passed to shell scripts in the provisioners section.\n2 3 4 5 6 7 8 9 10 11 12 13 14  \u0026#34;variables\u0026#34;: { \u0026#34;nginx_ver\u0026#34;: \u0026#34;1.15.6\u0026#34;, \u0026#34;pcre_ver\u0026#34;: \u0026#34;8.42\u0026#34;, \u0026#34;zlib_ver\u0026#34;: \u0026#34;1.2.11\u0026#34;, \u0026#34;openssl_ver\u0026#34;: \u0026#34;1.1.0h\u0026#34;, \u0026#34;geoip_ver\u0026#34;: \u0026#34;20181204\u0026#34;, \u0026#34;working_dir\u0026#34;: \u0026#34;/opt\u0026#34;, \u0026#34;src_folder\u0026#34;: \u0026#34;src_files\u0026#34;, \u0026#34;deb_pkg_folder\u0026#34;: \u0026#34;deb_pkg\u0026#34;, \u0026#34;log_folder\u0026#34;: \u0026#34;log\u0026#34;, \u0026#34;log_file\u0026#34;: \u0026#34;install_source.log\u0026#34; },   Please keep the version numbers for NGINX, PCRE, zlip, OpenSSL and GeoIp libraries up-to-date. You can find the latest release for each product at the links below:\n NGINX PCRE zlib OpenSSL GeoIP (GeoLite2)  The other variables are directory and file names used within the shell scripts, the purpose for each is explained below:\n working_dir root directory for nginx install files src_folder sub-directory for downloaded source files deb_pkg_folder sub-directory for .deb package file and .zip containing source files log_folder sub-directory for installation log files log_file name of the installation log file  In order to use the value of a user variable within the templete, call the {{user}} function in the form of {{user `variable`}}. This function can be used in any value except type within the template: in builders, provisioners, anywhere outside the variables section. User variables are available globally within the rest of the template.\nPacker CLI Packer is controlled through a simple command-line interface, you can find documentation for the CLI here. For this example, we will only be using two commands: validate and build.\nIt\u0026rsquo;s a good idea to validate the template after making any changes to the JSON file by running packer validate with the template file. This command validates the syntax and configuration values. If the template is not valid, any errors will be output to the console. For our template file, the output should look similar to below since the template is valid:\n$ packer validate nginx_ubuntu_from_deb.json Template validated successfully. After validating your template file, you are ready to build your machine image. This is done by calling packer build with the template file. Before doing so, there are a few values within the builders section which you need to modify according to your AWS EC2/VPC configuration.\nSpecifying Amazon Credentials In order to build the machine image, you must provide your AWS authentication credentials to the packer template. Packer will check for your credentials in a defined order, for more info read the official documentation. The order is:\n  Static credentials AWS access key and secret key are defined in the variables section of the packer template (lines 3-13 where our version numbers, etc are defined) or provided with the packer build command as shown below:\n$ packer build \\  -var 'aws_access_key=YOUR ACCESS KEY' \\  -var 'aws_secret_key=YOUR SECRET KEY' \\  nginx_ubuntu_from_source.json   Environment variables User provides credentials through the command line, as in the example below:\n$ export AWS_ACCESS_KEY_ID=\"YOUR ACCESS KEY\" $ export AWS_SECRET_ACCESS_KEY=\"YOUR SECRET KEY\" $ export AWS_DEFAULT_REGION=\"us-west-2\" $ packer build nginx_ubuntu_from_source.json   If credentials are provided through the command line, anyone who can see your environment variables (including programs you run) can see the credentials, and anyone that can view your processes can see the command line used to run them, including the parameters.\n    Shared credentials file You can use an AWS credentials file to specify your credentials. The default location is $HOME/.aws/credentials on Linux and MacOS, or %USERPROFILE%.aws\\credentials for Windows users. You can optionally specify a different location for packer to look for the configuration file by setting the value of the AWS_SHARED_CREDENTIALS_FILE environment variable.\n  EC2 Role This is the preferred method since you avoid hardcoding any credentials. For more info, read the documentation.\n  I use a credentials file to avoid the potential insecurity of methods #1 and #2. You can create the credentials file manually or generate it using the AWS CLI, here is the expected format:\n Figure 1 - AWS Credentials File\n  The Amazon-EBS Builder The builders section contains an array of objects that define the platform, operating system, etc. of each machine image that will be built from the template. Builders are responsible for creating virtual machines and generating images from them for various platforms. For example, there are separate builders for EC2, VMware, VirtualBox, etc. Packer comes with many builders by default, and can also be extended to add new builders.\nThe builder for our template is shown below:\n15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35  \u0026#34;builders\u0026#34;: [{ \u0026#34;type\u0026#34;: \u0026#34;amazon-ebs\u0026#34;, \u0026#34;access_key\u0026#34;: \u0026#34;{{user `aws_access_key`}}\u0026#34;, \u0026#34;secret_key\u0026#34;: \u0026#34;{{user `aws_secret_key`}}\u0026#34;, \u0026#34;region\u0026#34;: \u0026#34;us-west-1\u0026#34;, \u0026#34;source_ami_filter\u0026#34;: { \u0026#34;filters\u0026#34;: { \u0026#34;virtualization-type\u0026#34;: \u0026#34;hvm\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*\u0026#34;, \u0026#34;root-device-type\u0026#34;: \u0026#34;ebs\u0026#34; }, \u0026#34;owners\u0026#34;: [\u0026#34;099720109477\u0026#34;], \u0026#34;most_recent\u0026#34;: true }, \u0026#34;instance_type\u0026#34;: \u0026#34;t2.micro\u0026#34;, \u0026#34;ssh_username\u0026#34;: \u0026#34;ubuntu\u0026#34;, \u0026#34;ami_name\u0026#34;: \u0026#34;custom_nginx_ubuntu_{{timestamp}}\u0026#34;, \u0026#34;vpc_id\u0026#34;: \u0026#34;vpc-xxxxxxxx\u0026#34;, \u0026#34;subnet_id\u0026#34;: \u0026#34;subnet-xxxxxxxx\u0026#34;, \u0026#34;associate_public_ip_address\u0026#34;: \u0026#34;true\u0026#34; }],   The amazon-ebs builder builds an AMI by launching an EC2 instance from a source AMI, provisioning that running machine, and then creating an AMI from that machine. The simplest way to choose the source AMI is through the source_ami configuration setting. For example, if you want to base your AMI off of the most recent, official Ubuntu 16.04 HVM EBS AMI, you could find the AMI ID inside the AWS console as shown below:\n Figure 2 - AMI Filter Settings\n  To use this as your source AMI, you would add \u0026quot;source-ami\u0026quot;: \u0026quot;ami-925144f2\u0026quot; to the JSON above and remove the entire source_ami_filter object.\nHowever, if you plan on re-using this template and would like to always use the most recent Ubuntu 16.04 HVM EBS AMI from Canonical, you can do so and avoid hardcoding the AMI ID into the template with the source_ami_filter object. This is done in our example by setting the values below:\n virtualization-type The virtualization type. name The name of the AMI. root-device-type The type of the root device volume. owners An array of strings that filters the list of AMIs by Amazon account IDs. Used in our example to select only AMIs owned by Canonical. most_recent Boolean value that selects the newest created image when true.  The remaining configuration settings in the amazon-ebs builder deal with authenticating your AWS credentials and various aspects of the machine image:\n region (required) The name of the region, such as us-east-1, where the source AMI EC2 instance will be launched, and where the provisioned AMI will be stored. instance_type (required) The EC2 instance type to use while building the AMI, such as t2.micro. ssh_username (optional) Username to use for SSH, modify accordingly if you wish to use a different Linux distribution (e.g., use ec2-user for Amazon Linux AMI) ami_name (required) The name of the provisioned AMI that will appear in the EC2 console with other AMIs you own. Use a function like timestamp to ensure the AMI name is unique. vpc_id, subnet_id (optional) If you have not defined a default VPC or you would like to use a VPC other than your default, you must provide these values. You can remove these if you wish to use your account\u0026rsquo;s default VPC. associate_public_ip_address (optional) If using a non-default VPC, public IP addresses are not provided by default. If this is toggled, your new instance will get a Public IP.  When you call packer build with our template file, the portion of the output which is generated solely by the builder is shown below:\n$ packer build nginx_ubuntu_from_source.json amazon-ebs output will be in this color. == amazon-ebs: Prevalidating AMI Name: custom_nginx_ubuntu_1524988180 amazon-ebs: Found Image ID: ami-925144f2 == amazon-ebs: Creating temporary keypair: packer_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx == amazon-ebs: Creating temporary security group for this instance: packer_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx == amazon-ebs: Authorizing access to port 22 from 0.0.0.0/0 in the temporary security group... == amazon-ebs: Launching a source AWS instance... == amazon-ebs: Adding tags to source instance amazon-ebs: Adding tag: \"Name\": \"Packer Builder\" amazon-ebs: Instance ID: i-xxxxxxxxxxxxxxxxx == amazon-ebs: Waiting for instance (i-xxxxxxxxxxxxxxxxx) to become ready... == amazon-ebs: Waiting for SSH to become available... The output above shows that the source AMI used to launch our EC2 instance (ami-925144f2) is the most recent official Ubuntu 16.04 HVM EBS AMI, as we desired. The rest of the output is a result of packer creating a temporary keypair and security group in order to access the EC2 instance. After these are in place, the instance is launched and packer waits for the SSH service to become available in order to perform the actions in the provisioners object.\nProvisioners Provisioners perform configuration or installation procedures, and can be extremely useful since each provisioner is run for every builder defined. So if we had two builders defined in our template (e.g. Amazon EC2 and VMWare), then the complete set of provisioners would run as part of both builds. It is possible to only run specific provisioners for one builder, but it will not be covered in this basic example (you can find more information in the official documentation)\nIn order to install the latest version of NGINX and perform customizations such as enabling modules disabled by default, the packer template utilizes the shell and file provisioners. Please read the official documentation for more information about these and the other provisioners that are included with packer.\nShell Provisioner (Inline Script) The first provisioner in our template is a shell provisioner. This provisioner requires that one of the following configuration options be present: inline, script or scripts. In this instance, we are using the inline option to define a sequence of shell commands to execute on the EC2 instance. The commands are run with the user defined by the ssh_username setting.\n36 37 38 39 40 41 42  \u0026#34;provisioners\u0026#34;: [{ \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;inline\u0026#34;: [\u0026#34;sudo mkdir -p {{user `working_dir`}}/{{user `src_folder`}}\u0026#34;, \u0026#34;sudo chown ubuntu:ubuntu {{user `working_dir`}}/{{user `src_folder`}}\u0026#34;, \u0026#34;sudo mkdir -p {{user `working_dir`}}/{{user `deb_pkg_folder`}}/\u0026#34;, \u0026#34;sudo chown ubuntu:ubuntu {{user `working_dir`}}/{{user `deb_pkg_folder`}}\u0026#34;] },   These commands create two directories and change their owner to the SSH user, this is needed to use the file provisioner which is the next action performed on our EC2 instance. The output from the packer build command which is generated solely from this shell provisioner is shown below:\n== amazon-ebs: Connected to SSH! == amazon-ebs: Provisioning with shell script: /var/folders/7t/g16tw4qd2x532lsb_1lm0vyh0000gn/T/packer-shell348591307 As you can see, packer creates a temporary script file from our shell commands and executes this script on our EC2 instance.\nFile Provisioner (Upload to EC2) The file provisioner transfers files and folders between the local machine and the EC2 instance. The default transfer direction is upload (from the local machine to the EC2 instance). This is the default behavior, so you do not need to provide the direction setting in the file provisioner if this is the desired behavior. We will encounter the download transfer direction in a later step:\n43 44 45 46  \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;./upload/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `deb_pkg_folder`}}\u0026#34; },   This provisioner transfers the contents of the upload directory to the location on the EC2 instance defined by the values of the user variables working_dir and deb_pkg_folder (I chose to use /opt/deb_pkg). The uploaded files are used to configure the firewall settings and provide the systemd unit file for the NGINX service. The output from the packer build command which is generated solely from this file provisioner is shown below:\n== amazon-ebs: Uploading ./upload/ = /opt/deb_pkg Shell Provisioner (Local File) The next provisioner is where the bulk of the configuration process is performed. This is another shell provisioner but this time we are using the scripts configuration option to execute multiple bash script files. All of the user variables are used by this provisioner in the environment_vars object which injects the user variables into the shell scripts.\nAlso, please note the use of the configuration setting \u0026quot;expect_disconnect\u0026quot;: true. This is necessary since the last shell script reboots the server to verify the NGINX service was configured successfully. If this option was not set, the build process would produce an error when the server disconnects and terminate the EC2 instance:\n47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63  \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;scripts\u0026#34;: [\u0026#34;./bash-scripts/00--nginx-prep_install.sh\u0026#34;, \u0026#34;./bash-scripts/01a-nginx-install_from_source.sh\u0026#34;, \u0026#34;./bash-scripts/02--nginx-configure_post_install.sh\u0026#34;], \u0026#34;expect_disconnect\u0026#34;: true, \u0026#34;environment_vars\u0026#34;: [ \u0026#34;NGINX_VER={{user `nginx_ver`}}\u0026#34;, \u0026#34;PCRE_VER={{user `pcre_ver`}}\u0026#34;, \u0026#34;ZLIB_VER={{user `zlib_ver`}}\u0026#34;, \u0026#34;OPENSSL_VER={{user `openssl_ver`}}\u0026#34;, \u0026#34;GEOIP_VER={{user `geoip_ver`}}\u0026#34;, \u0026#34;WORKING_DIR={{user `working_dir`}}\u0026#34;, \u0026#34;SRC_FOLDER={{user `src_folder`}}\u0026#34;, \u0026#34;DEB_PKG_FOLDER={{user `deb_pkg_folder`}}\u0026#34;, \u0026#34;LOG_FOLDER={{user `log_folder`}}\u0026#34;, \u0026#34;LOG_FILE={{user `log_file`}}\u0026#34;] },   These scripts perform the following sequence of actions:\n Prepare the EC2 instance for the NGINX installation Download all source files needed to build the latest NGINX release Build NGINX and create a .deb package which can install/uninstall our custom build Install NGINX from the .deb package Verify NGINX install was successful Configure NGINX for multiple virtual hosts Configure NGINX firewall settings and systemd unit file Remove downloaded source files Start NGINX service Reboot EC2 instance and verify NGINX service starts automatically  The output from the packer build command which is generated solely from this shell provisioner is shown below:\n== amazon-ebs: Provisioning with shell script: ./bash-scripts/00--nginx-prep_install.sh amazon-ebs: 29-Apr-2018-07-50-22 | Updating system... amazon-ebs: 29-Apr-2018-07-51-07 | Downloading prerequisites... == amazon-ebs: Provisioning with shell script: ./bash-scripts/01a-nginx-install_from_source.sh amazon-ebs: 29-Apr-2018-07-51-23 | Downloading source files... amazon-ebs: 29-Apr-2018-07-51-28 | Building NGINX from source... amazon-ebs: amazon-ebs: 29-Apr-2018-07-54-26 | Installing NGINX from .deb package... amazon-ebs: amazon-ebs: Selecting previously unselected package nginx. amazon-ebs: (Reading database ... 81442 files and directories currently installed.) amazon-ebs: Preparing to unpack nginx_1.15.6-1_amd64.deb ... amazon-ebs: Unpacking nginx (1.15.6-1) ... amazon-ebs: Setting up nginx (1.15.6-1) ... amazon-ebs: 29-Apr-2018-07-54-26 | Install completed successfully, creating archive of source files... == amazon-ebs: Provisioning with shell script: ./bash-scripts/02--nginx-configure_post_install.sh amazon-ebs: amazon-ebs: 29-Apr-2018-07-54-41 | Verify configuration file syntax is correct and test is successful: amazon-ebs: amazon-ebs: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok amazon-ebs: nginx: configuration file /etc/nginx/nginx.conf test is successful amazon-ebs: amazon-ebs: 29-Apr-2018-07-54-41 | Verify NGINX version and configure arguments match your selections: amazon-ebs: amazon-ebs: nginx version: nginx/1.15.6 (Ubuntu) amazon-ebs: built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) amazon-ebs: built with OpenSSL 1.1.0h 27 Mar 2018 amazon-ebs: TLS SNI support enabled amazon-ebs: configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --user=www-data --group=www-data --build=Ubuntu --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-openssl=/opt/src_files/openssl-1.1.0h --with-openssl-opt=enable-ec_nistp_64_gcc_128 --with-openssl-opt=no-nextprotoneg --with-openssl-opt=no-weak-ssl-ciphers --with-openssl-opt=no-ssl3 --with-pcre=/opt/src_files/pcre-8.42 --with-pcre-jit --with-zlib=/opt/src_files/zlib-1.2.11 --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_slice_module --with-http_ssl_module --with-http_sub_module --with-http_stub_status_module --with-http_v2_module --with-http_secure_link_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-debug --add-module=../ngx_http_geoip2_module --add-module=../ngx_cache_purge --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed' amazon-ebs: amazon-ebs: 29-Apr-2018-07-54-41 | Creating folders for nginx virtual hosts... amazon-ebs: amazon-ebs: 29-Apr-2018-07-54-41 | Downloading GeoIP2 database files... amazon-ebs: 29-Apr-2018-07-54-43 | Configuring firewall app profile... amazon-ebs: 29-Apr-2018-07-54-43 | Configuring systemd unit file... amazon-ebs: 29-Apr-2018-07-54-43 | Setting permissions for NGINX user account... amazon-ebs: 29-Apr-2018-07-54-43 | Installation and configuration is complete, removing source files... amazon-ebs: amazon-ebs: 29-Apr-2018-07-54-43 | Starting NGINX, verify service is active: amazon-ebs: amazon-ebs: ● nginx.service - A high performance web server and a reverse proxy server amazon-ebs: Loaded: loaded (/etc/systemd/system/nginx.service; enabled; vendor preset: enabled) amazon-ebs: Active: active (running) since Sun 2018-04-29 07:54:43 UTC; 149ms ago amazon-ebs: Main PID: 8595 (nginx) amazon-ebs: CGroup: /system.slice/nginx.service amazon-ebs: ├─8595 nginx: master process /usr/sbin/nginx -g daemon on; master_process on amazon-ebs: └─8597 nginx: worker process amazon-ebs: amazon-ebs: Apr 29 07:54:43 ip-172-31-12-194 systemd[1]: Starting A high performance web server and a reverse proxy server... amazon-ebs: Apr 29 07:54:43 ip-172-31-12-194 systemd[1]: Started A high performance web server and a reverse proxy server. amazon-ebs: amazon-ebs: 29-Apr-2018-07-54-43 | NGINX is successfully installed and configured amazon-ebs: 29-Apr-2018-07-54-43 | Rebooting server to verify NGINX starts automatically... The next provisioner executes a single shell script after the EC2 instance has rebooted. Following a reboot, it is recommended to use the pause_before option when executing a shell script because Packer is able to detect and SSH into the instance as soon as SSH is available. However, Ubuntu actually doesn\u0026rsquo;t get proper amounts of time to initialize. The pause makes sure that the OS properly initializes:\n64 65 66 67 68 69 70 71  \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;script\u0026#34;: \u0026#34;./bash-scripts/03--nginx-verify_install.sh\u0026#34;, \u0026#34;pause_before\u0026#34;: \u0026#34;10s\u0026#34;, \u0026#34;environment_vars\u0026#34;: [ \u0026#34;WORKING_DIR={{user `working_dir`}}\u0026#34;, \u0026#34;LOG_FOLDER={{user `log_folder`}}\u0026#34;, \u0026#34;LOG_FILE={{user `log_file`}}\u0026#34;] },   The output from the packer build command which is generated solely from this shell provisioner is shown below:\n== amazon-ebs: Pausing 10s before the next provisioner... == amazon-ebs: Provisioning with shell script: ./bash-scripts/03--nginx-verify_install.sh amazon-ebs: amazon-ebs: 29-Apr-2018-07-55-37 | Reboot completed, NGINX service should be running: amazon-ebs: amazon-ebs: ● nginx.service - A high performance web server and a reverse proxy server amazon-ebs: Loaded: loaded (/etc/systemd/system/nginx.service; enabled; vendor preset: enabled) amazon-ebs: Active: active (running) since Sun 2018-04-29 07:55:18 UTC; 18s ago amazon-ebs: Process: 1164 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) amazon-ebs: Process: 1124 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) amazon-ebs: Main PID: 1171 (nginx) amazon-ebs: Tasks: 2 amazon-ebs: Memory: 4.7M amazon-ebs: CPU: 9ms amazon-ebs: CGroup: /system.slice/nginx.service amazon-ebs: ├─1171 nginx: master process /usr/sbin/nginx -g daemon on; master_process on amazon-ebs: └─1174 nginx: worker process amazon-ebs: amazon-ebs: Apr 29 07:55:18 ip-172-31-12-194 systemd[1]: Starting A high performance web server and a reverse proxy server... amazon-ebs: Apr 29 07:55:18 ip-172-31-12-194 systemd[1]: Started A high performance web server and a reverse proxy server. amazon-ebs: amazon-ebs: 29-Apr-2018-07-55-37 | Verify 3 NGINX UFW app profiles are listed below (Full, HTTP and HTTPS): amazon-ebs: WARN: uid is 0 but '/etc/ufw/applications.d/nginx' is owned by 1000 amazon-ebs: Available applications: amazon-ebs: Nginx Full amazon-ebs: Nginx HTTP amazon-ebs: Nginx HTTPS amazon-ebs: OpenSSH File Provisioner (Download from EC2) The final two provisioners in our template are file provisioners that download files from the EC2 instance to our local machine. In order to do so, we must use the configuration setting \u0026quot;direction\u0026quot;: \u0026quot;download\u0026quot; as shown below:\n72 73 74 75 76 77 78 79 80 81  \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `deb_pkg_folder`}}/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;./download\u0026#34;, \u0026#34;direction\u0026#34;: \u0026#34;download\u0026#34; },{ \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `log_folder`}}/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;./download\u0026#34;, \u0026#34;direction\u0026#34;: \u0026#34;download\u0026#34; }]   These provisioners download the contents of the directories defined by the values of the user variables working_dir/deb_pkg_folder and working_dir/log_folder to the download folder which must exist within the directory where the packer template is located. The first provisioner downloads the .deb package which can install/uninstall the custom NGINX build and the .tar.gz file containing the source files needed to perform the installation. The second provisioner downloads the log of the NGINX installation process.\nThe output from the packer build command which is generated solely from these file provisioners is shown below:\n== amazon-ebs: Downloading /opt/deb_pkg/ = ./download == amazon-ebs: Downloading /opt/log/ = ./download Finishing the Build After the provisioners have completed, packer stops the EC2 instance and creates an AMI that is associated with your account. You can use this AMI to launch any number of t2.micro instances that are running the latest version of NGINX, configured with your desired set of builtin and third-party modules.\nThe output from the packer build command which is generated after the provisioners have completed is shown below:\n== amazon-ebs: Stopping the source instance... amazon-ebs: Stopping instance, attempt 1 == amazon-ebs: Waiting for the instance to stop... == amazon-ebs: Creating the AMI: custom_nginx_ubuntu_1524988180 amazon-ebs: AMI: ami-xxxxxxxx == amazon-ebs: Waiting for AMI to become ready... == amazon-ebs: Terminating the source AWS instance... == amazon-ebs: Cleaning up any extra volumes... == amazon-ebs: No volumes to clean up, skipping == amazon-ebs: Deleting temporary security group... == amazon-ebs: Deleting temporary keypair... Build 'amazon-ebs' finished. == Builds finished. The artifacts of successful builds are: -- amazon-ebs: AMIs were created: us-west-1: ami-xxxxxxxx   Unless you want to be charged for storing the AMI that has been created, you should deregister it and delete the snapshot associated with the AMI immediately.\n  First, login to the AWS Console and navigate to EC2 -\u0026gt; AMIs. The AMI created by the packer template will have the name specified in line 31 of the JSON file:\n Figure 3 - AMI Built from Packer Template\n  Select the AMI and click Deregister from the Actions menu:\n Figure 4 - Deregister the AMI to Avoid Incurring Costs\n  Next, navigate to EC2 -\u0026gt; Snapshots:\n Figure 5 - Snapshot Associated with Packer AMI\n  Select the snapshot and click Delete from the Actions menu\n Figure 6 - Delete the AMI Snapshot to Avoid Incurring Costs\n  Installing NGINX from .deb File The final provisioner downloaded two files from the deb_pkg folder on the EC2 instance to the local machine:\n$ ls -l total 194512 -rwxr-xr-x 1 aaronluna staff 97927787 Apr 29 03:09 nginx-1.15.6-src_files.tar.gz -rwxr-xr-x 1 aaronluna staff 1655608 Apr 29 00:58 nginx_1.15.6-1_amd64.deb The .deb package can be used to uninstall NGINX from an EC2 instance launched from an AMI produced by the nginx_ubuntu_from_source.json template file, and it can also be used (along with the .tar.gz file containing the necessary source files) to install NGINX on an EC2 instance. To do so, you only need to call packer build with the other template file in the github repository: nginx_ubuntu_from_deb.json.\nThe two templates are very similar. In fact, the shell scripts have been modularized so they can be used in both versions. Here is the full JSON for nginx_ubuntu_from_deb.json:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81  { \u0026#34;variables\u0026#34;: { \u0026#34;nginx_ver\u0026#34;: \u0026#34;1.15.6\u0026#34;, \u0026#34;pcre_ver\u0026#34;: \u0026#34;8.42\u0026#34;, \u0026#34;zlib_ver\u0026#34;: \u0026#34;1.2.11\u0026#34;, \u0026#34;openssl_ver\u0026#34;: \u0026#34;1.1.0h\u0026#34;, \u0026#34;geoip_ver\u0026#34;: \u0026#34;20180403\u0026#34;, \u0026#34;working_dir\u0026#34;: \u0026#34;/opt\u0026#34;, \u0026#34;src_folder\u0026#34;: \u0026#34;src_files\u0026#34;, \u0026#34;deb_pkg_folder\u0026#34;: \u0026#34;deb_pkg\u0026#34;, \u0026#34;log_folder\u0026#34;: \u0026#34;log\u0026#34;, \u0026#34;log_file\u0026#34;: \u0026#34;install_deb.log\u0026#34; }, \u0026#34;builders\u0026#34;: [{ \u0026#34;type\u0026#34;: \u0026#34;amazon-ebs\u0026#34;, \u0026#34;access_key\u0026#34;: \u0026#34;{{user `aws_access_key`}}\u0026#34;, \u0026#34;secret_key\u0026#34;: \u0026#34;{{user `aws_secret_key`}}\u0026#34;, \u0026#34;region\u0026#34;: \u0026#34;us-west-1\u0026#34;, \u0026#34;source_ami_filter\u0026#34;: { \u0026#34;filters\u0026#34;: { \u0026#34;virtualization-type\u0026#34;: \u0026#34;hvm\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*\u0026#34;, \u0026#34;root-device-type\u0026#34;: \u0026#34;ebs\u0026#34; }, \u0026#34;owners\u0026#34;: [\u0026#34;099720109477\u0026#34;], \u0026#34;most_recent\u0026#34;: true }, \u0026#34;instance_type\u0026#34;: \u0026#34;t2.micro\u0026#34;, \u0026#34;ssh_username\u0026#34;: \u0026#34;ubuntu\u0026#34;, \u0026#34;ami_name\u0026#34;: \u0026#34;custom_nginx_ubuntu_{{timestamp}}\u0026#34;, \u0026#34;vpc_id\u0026#34;: \u0026#34;vpc-xxxxxxxx\u0026#34;, \u0026#34;subnet_id\u0026#34;: \u0026#34;subnet-xxxxxxxx\u0026#34;, \u0026#34;associate_public_ip_address\u0026#34;: \u0026#34;true\u0026#34; }], \u0026#34;provisioners\u0026#34;: [{ \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;inline\u0026#34;: [\u0026#34;sudo mkdir -p {{user `working_dir`}}/{{user `src_folder`}}\u0026#34;, \u0026#34;sudo chown ubuntu:ubuntu {{user `working_dir`}}/{{user `src_folder`}}\u0026#34;, \u0026#34;sudo mkdir -p {{user `working_dir`}}/{{user `deb_pkg_folder`}}/\u0026#34;, \u0026#34;sudo chown ubuntu:ubuntu {{user `working_dir`}}/{{user `deb_pkg_folder`}}\u0026#34;] },{ \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;./upload/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `deb_pkg_folder`}}\u0026#34; },{ \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;./download/{{user `deb_pkg_folder`}}/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `src_folder`}}\u0026#34; },{ \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;scripts\u0026#34;: [\u0026#34;./bash-scripts/00--nginx-prep_install.sh\u0026#34;, \u0026#34;./bash-scripts/01b-nginx-install_from_deb.sh\u0026#34;, \u0026#34;./bash-scripts/02--nginx-configure_post_install.sh\u0026#34;], \u0026#34;expect_disconnect\u0026#34;: true, \u0026#34;environment_vars\u0026#34;: [ \u0026#34;NGINX_VER={{user `nginx_ver`}}\u0026#34;, \u0026#34;PCRE_VER={{user `pcre_ver`}}\u0026#34;, \u0026#34;ZLIB_VER={{user `zlib_ver`}}\u0026#34;, \u0026#34;OPENSSL_VER={{user `openssl_ver`}}\u0026#34;, \u0026#34;GEOIP_VER={{user `geoip_ver`}}\u0026#34;, \u0026#34;WORKING_DIR={{user `working_dir`}}\u0026#34;, \u0026#34;SRC_FOLDER={{user `src_folder`}}\u0026#34;, \u0026#34;DEB_PKG_FOLDER={{user `deb_pkg_folder`}}\u0026#34;, \u0026#34;LOG_FOLDER={{user `log_folder`}}\u0026#34;, \u0026#34;LOG_FILE={{user `log_file`}}\u0026#34;] },{ \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;script\u0026#34;: \u0026#34;./bash-scripts/03--nginx-verify_install.sh\u0026#34;, \u0026#34;pause_before\u0026#34;: \u0026#34;10s\u0026#34;, \u0026#34;environment_vars\u0026#34;: [ \u0026#34;WORKING_DIR={{user `working_dir`}}\u0026#34;, \u0026#34;LOG_FOLDER={{user `log_folder`}}\u0026#34;, \u0026#34;LOG_FILE={{user `log_file`}}\u0026#34;] },{ \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `log_folder`}}/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;./download\u0026#34;, \u0026#34;direction\u0026#34;: \u0026#34;download\u0026#34; }] }   The two files are identical through lines 1-46. The template which installs NGINX from the .deb package contains an additional file provisioner (lines 47-50) which uploads the two files in the deb_pkg folder to the EC2 instance:\n47 48 49 50  \u0026#34;type\u0026#34;: \u0026#34;file\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;./download/{{user `deb_pkg_folder`}}/\u0026#34;, \u0026#34;destination\u0026#34;: \u0026#34;{{user `working_dir`}}/{{user `src_folder`}}\u0026#34; },   The next provisioner which executes the array of shell scripts to install NGINX is also nearly identical in both files. The only difference is in line 53 which executes a different installation script that installs NGINX from the .deb package rather than downloading and building NGINX from source:\n51 52 53 54 55  \u0026#34;type\u0026#34;: \u0026#34;shell\u0026#34;, \u0026#34;scripts\u0026#34;: [\u0026#34;./bash-scripts/00--nginx-prep_install.sh\u0026#34;, \u0026#34;./bash-scripts/01b-nginx-install_from_deb.sh\u0026#34;, \u0026#34;./bash-scripts/02--nginx-configure_post_install.sh\u0026#34;], \u0026#34;expect_disconnect\u0026#34;: true,   The entire output from the packer build command for this template file is shown below:\n$ packer build nginx_ubuntu_from_deb.json == amazon-ebs: Prevalidating AMI Name: custom_nginx_ubuntu_1525002641 amazon-ebs: Found Image ID: ami-925144f2 == amazon-ebs: Creating temporary keypair: packer_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx == amazon-ebs: Creating temporary security group for this instance: packer_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx == amazon-ebs: Authorizing access to port 22 from 0.0.0.0/0 in the temporary security group... == amazon-ebs: Launching a source AWS instance... == amazon-ebs: Adding tags to source instance amazon-ebs: Adding tag: \"Name\": \"Packer Builder\" amazon-ebs: Instance ID: i-xxxxxxxxxxxxxxxxx == amazon-ebs: Waiting for instance (i-xxxxxxxxxxxxxxxxx) to become ready... == amazon-ebs: Waiting for SSH to become available... == amazon-ebs: Connected to SSH! == amazon-ebs: Provisioning with shell script: /var/folders/7t/g16tw4qd2x532lsb_1lm0vyh0000gn/T/packer-shell264972135 == amazon-ebs: Uploading ./upload/ = /opt/deb_pkg == amazon-ebs: Uploading ./download/deb_pkg/ = /opt/src_files == amazon-ebs: Provisioning with shell script: ./bash-scripts/00--nginx-prep_install.sh amazon-ebs: 29-Apr-2018-14-00-32 | Updating system... amazon-ebs: 29-Apr-2018-14-01-22 | Downloading prerequisites... == amazon-ebs: Provisioning with shell script: ./bash-scripts/01b-nginx-install_from_deb.sh amazon-ebs: 29-Apr-2018-14-01-48 | Extracting source files from archive... amazon-ebs: amazon-ebs: 29-Apr-2018-14-01-51 | Installing NGINX from .deb package... amazon-ebs: amazon-ebs: Selecting previously unselected package nginx. amazon-ebs: (Reading database ... 81442 files and directories currently installed.) amazon-ebs: Preparing to unpack nginx_1.15.6-1_amd64.deb ... amazon-ebs: Unpacking nginx (1.15.6-1) ... amazon-ebs: Setting up nginx (1.15.6-1) ... == amazon-ebs: Provisioning with shell script: ./bash-scripts/02--nginx-configure_post_install.sh amazon-ebs: amazon-ebs: 29-Apr-2018-14-01-59 | Verify configuration file syntax is correct and test is successful: amazon-ebs: amazon-ebs: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok amazon-ebs: nginx: configuration file /etc/nginx/nginx.conf test is successful amazon-ebs: amazon-ebs: 29-Apr-2018-14-01-59 | Verify NGINX version and configure arguments match your selections: amazon-ebs: amazon-ebs: nginx version: nginx/1.15.6 (Ubuntu) amazon-ebs: built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) amazon-ebs: built with OpenSSL 1.1.0h 27 Mar 2018 amazon-ebs: TLS SNI support enabled amazon-ebs: configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --user=www-data --group=www-data --build=Ubuntu --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-openssl=/opt/src_files/openssl-1.1.0h --with-openssl-opt=enable-ec_nistp_64_gcc_128 --with-openssl-opt=no-nextprotoneg --with-openssl-opt=no-weak-ssl-ciphers --with-openssl-opt=no-ssl3 --with-pcre=/opt/src_files/pcre-8.42 --with-pcre-jit --with-zlib=/opt/src_files/zlib-1.2.11 --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_slice_module --with-http_ssl_module --with-http_sub_module --with-http_stub_status_module --with-http_v2_module --with-http_secure_link_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-debug --add-module=../ngx_http_geoip2_module --add-module=../ngx_cache_purge --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed' amazon-ebs: amazon-ebs: 29-Apr-2018-14-01-59 | Creating folders for nginx virtual hosts... amazon-ebs: amazon-ebs: 29-Apr-2018-14-01-59 | Downloading GeoIP2 database files... amazon-ebs: 29-Apr-2018-14-02-01 | Configuring firewall app profile... amazon-ebs: 29-Apr-2018-14-02-01 | Configuring systemd unit file... amazon-ebs: 29-Apr-2018-14-02-01 | Setting permissions for NGINX user account... amazon-ebs: 29-Apr-2018-14-02-01 | Installation and configuration is complete, removing source files... amazon-ebs: amazon-ebs: 29-Apr-2018-14-02-01 | Starting NGINX, verify service is active: amazon-ebs: amazon-ebs: ● nginx.service - A high performance web server and a reverse proxy server amazon-ebs: Loaded: loaded (/etc/systemd/system/nginx.service; enabled; vendor preset: enabled) amazon-ebs: Active: active (running) since Sun 2018-04-29 14:02:01 UTC; 127ms ago amazon-ebs: Main PID: 31508 (nginx) amazon-ebs: CGroup: /system.slice/nginx.service amazon-ebs: ├─31508 nginx: master process /usr/sbin/nginx -g daemon on; master_process on amazon-ebs: └─31509 nginx: worker process amazon-ebs: amazon-ebs: Apr 29 14:02:01 ip-172-31-14-86 systemd[1]: Starting A high performance web server and a reverse proxy server... amazon-ebs: Apr 29 14:02:01 ip-172-31-14-86 systemd[1]: Started A high performance web server and a reverse proxy server. amazon-ebs: amazon-ebs: 29-Apr-2018-14-02-01 | NGINX is successfully installed and configured amazon-ebs: 29-Apr-2018-14-02-01 | Rebooting server to verify NGINX starts automatically... == amazon-ebs: Pausing 10s before the next provisioner... == amazon-ebs: Provisioning with shell script: ./bash-scripts/03--nginx-verify_install.sh amazon-ebs: amazon-ebs: 29-Apr-2018-14-02-56 | Reboot completed, NGINX service should be running: amazon-ebs: amazon-ebs: ● nginx.service - A high performance web server and a reverse proxy server amazon-ebs: Loaded: loaded (/etc/systemd/system/nginx.service; enabled; vendor preset: enabled) amazon-ebs: Active: active (running) since Sun 2018-04-29 14:02:34 UTC; 22s ago amazon-ebs: Process: 1173 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) amazon-ebs: Process: 1119 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) amazon-ebs: Main PID: 1175 (nginx) amazon-ebs: Tasks: 2 amazon-ebs: Memory: 4.5M amazon-ebs: CPU: 9ms amazon-ebs: CGroup: /system.slice/nginx.service amazon-ebs: ├─1175 nginx: master process /usr/sbin/nginx -g daemon on; master_process on amazon-ebs: └─1177 nginx: worker process amazon-ebs: amazon-ebs: Apr 29 14:02:33 ip-172-31-14-86 systemd[1]: Starting A high performance web server and a reverse proxy server... amazon-ebs: Apr 29 14:02:34 ip-172-31-14-86 systemd[1]: nginx.service: Failed to read PID from file /run/nginx.pid: Invalid argument amazon-ebs: Apr 29 14:02:34 ip-172-31-14-86 systemd[1]: Started A high performance web server and a reverse proxy server. amazon-ebs: amazon-ebs: 29-Apr-2018-14-02-56 | Verify 3 NGINX UFW app profiles are listed below (Full, HTTP and HTTPS): amazon-ebs: WARN: uid is 0 but '/etc/ufw/applications.d/nginx' is owned by 1000 amazon-ebs: Available applications: amazon-ebs: Nginx Full amazon-ebs: Nginx HTTP amazon-ebs: Nginx HTTPS amazon-ebs: OpenSSH == amazon-ebs: Downloading /opt/log/ = ./download == amazon-ebs: Stopping the source instance... amazon-ebs: Stopping instance, attempt 1 == amazon-ebs: Waiting for the instance to stop... == amazon-ebs: Creating the AMI: custom_nginx_ubuntu_1525002641 amazon-ebs: AMI: ami-xxxxxxxx == amazon-ebs: Waiting for AMI to become ready... == amazon-ebs: Terminating the source AWS instance... == amazon-ebs: Cleaning up any extra volumes... == amazon-ebs: No volumes to clean up, skipping == amazon-ebs: Deleting temporary security group... == amazon-ebs: Deleting temporary keypair... Build 'amazon-ebs' finished. == Builds finished. The artifacts of successful builds are: -- amazon-ebs: AMIs were created: us-west-1: ami-xxxxxxxx Installation Log Files Each packer template downloads a log file of the installation process containing the console output from the EC2 instance to the local machine:\n$ ls -l total 2520 -rw-r--r-- 1 aaronluna staff 92355 Apr 29 07:03 install_deb.log -rw-r--r--@ 1 aaronluna staff 1195654 Apr 29 03:11 install_source.log The install_source.log file is much larger than install_deb.log, since the output of the make command which builds NGINX is extremely verbose. If you are interested in the fine details of each process, you can find it in these logs.\nConclusion I hope this post helps you understand the basics of packer and how this tool can be used in many situations to create machine images of various types with identical configurations. The example presented here of a custom NGINX install is just the beginning of what you can do with a packer template and simple shell scripts. You could install many more applications and services and do so with other configuration management tools such as Ansible and Vagrant. I will present a more advanced packer template using these tools in a future post.\n","href":"/blog/packer-template-aws-ec2-ubuntu-nginx/","title":"Packer Template: Amazon EC2 Ubuntu AMI with Latest NGINX"},{"categories":"DevOps","content":"When I began learning about DevOps software and processes, it seemed like there were numerous tools that accomplished the same task, or nearly the same task without a clear consensus opinion on the best toolchain and workflow. I quickly discovered that there isn\u0026rsquo;t a single, correct answer, mostly due to the nature of open-source software. The Periodic Table of DevOps Tools is a crowdsourced website that nicely illustrates the current landscape:\n Figure 1 - Periodic Table of DevOps Tools\n  Anyone can cast votes for their favorite set of tools or recommend a tool that isn\u0026rsquo;t listed in the table, and the table is regularly updated to reflect changes in the vote total. There are several interesting discussions in the comments below the table regarding the website\u0026rsquo;s methodology, which helped me understand how various tools are related to one another and which are highly-regarded within each category. Also, each tool is labeled according to it\u0026rsquo;s pricing model (Open Source, Free, Fremium, Paid, Enterprise). It\u0026rsquo;s hard to miss the fact that the majority of DevOps tools are Open Source or Free software.\n","href":"/blog/periodic-table-of-devops-tools/","title":"The Periodic Table of DevOps Tools"},{"categories":"dotnet","content":"Let\u0026rsquo;s finish off the NetworkUtilities class from the previous post with a set of methods that retrieve the private (local) and public (external) IP addresses of the local machine. All source files can be downloaded individually as .cs files or as a single zip file containing the entire class at the gist link below:\nNetwork Utilities: IP Parsing/Retrieving (C\u0026#35; .NET Core 2.0) [gist.github.com]  Retrieve Local IP Address (Requires Internet) The method below, GetLocalIPv4AddressRequiresInternet, retrieves the local IP address, and does so in a fairly clever way (IMHO). Internet access is required because the method uses a System.Net.Socket object to connect to Google\u0026rsquo;s Public DNS service. When the connection is made, the Socket\u0026rsquo;s LocalEndPoint property contains the local IP address of our machine. Our goal of retrieving the local IP address is handled, essesntially, with just 2 lines of code:\nnamespace AaronLuna.Common.Network { using System.Net; using System.Net.Sockets; public static partial class NetworkUtilities { public static IPAddress GetLocalIPv4AddressRequiresInternet() { var localIp = IPAddress.None; try { using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, 0)) { // Connect socket to Google\u0026#39;s Public DNS service  socket.Connect(\u0026#34;8.8.8.8\u0026#34;, 65530); if (!(socket.LocalEndPoint is IPEndPoint endPoint)) { throw new InvalidOperationException($\u0026#34;Error occurred casting {socket.LocalEndPoint} to IPEndPoint\u0026#34;); } localIp = endPoint.Address; } } catch (SocketException ex) { // Handle exception  } return localIp; } } } Retrieve Local IP Address Without Internet We have to do a bit more work if we decide that relying on an external service is not an option. Obviously, if there is only a single network adapter on our machine the task is easier but this is often not the case.\nSo, where do we start in this scenario? The System.Net.NetworkInformation namespace contains the static method NetworkInterface.GetAllNetworkInterfaces which returns an array of objects describing every network interface available on the current computer. Then, with this information and a LINQ query, I create a list containing all local IPv4 addresses.\nYou may have noticed that GetLocalIpAddressNoInternet requires a string parameter named localNetworkCidrIp. This is the LAN configuration in CIDR notation. If you are unfamiliar with CIDR notation or basic networking concepts, please read this helpful article from DigitalOcean.\nIn the previous post I explicitly walked through the process of checking if an IP address exists within the address space defined by a CIDR block. This process is implemented by the IpAddressIsInCidrRange method.\nEach local IP address is checked to see if it exists within the LAN specified by the cidrMask, and the first matching address is returned as the local IP address:\nnamespace AaronLuna.Common.Network { using System.Collections.Generic; using System.Linq; using System.Net; using System.Net.NetworkInformation; using System.Net.Sockets; public static partial class NetworkUtilities { // This method uses the address range defined by localNetworkCidrIp to  // determine which local IP should be used. The CIDR IP must be in correct  // format: a.b.c.d/n. For example, if your LAN is setup to allow 254 hosts  // and the router\u0026#39;s address is 192.168.2.1, the correct value for  // localNetworkCidrIp would be \u0026#34;192.168.2.0/24\u0026#34;  public static IPAddress GetLocalIpAddressNoInternet(string localNetworkCidrIp) { var localIps = GetLocalIPv4AddressList(); if (localIps.Count == 1) { return localIps[0]; } foreach (var ip in localIps) { if (IpAddressIsInRange(ip, localNetworkCidrIp)) { return ip; } } // If no IP addresses match CIDR IP or none were found, return 255.255.255.255  return IPAddress.None; } public static List\u0026lt;IPAddress\u0026gt; GetLocalIPv4AddressList() { var localIps = new List\u0026lt;IPAddress\u0026gt;(); foreach (var nic in NetworkInterface.GetAllNetworkInterfaces()) { var ips = nic.GetIPProperties().UnicastAddresses .Select(uni =\u0026gt; uni.Address) .Where(ip =\u0026gt; ip.AddressFamily == AddressFamily.InterNetwork).ToList(); localIps.AddRange(ips); } return localIps; } } } Retrieve Public IP Address (Requires Internet) I was unable to determine any way to get the public IP without relying on an external system, and this makes sense since the public IP is really the address of the router and not the local machine. I thought that it might be possible to get the public IP from the router itself through telnet, but the process would vary greatly for each router and I couldn\u0026rsquo;t even determine if it was possible to telnet into my router without installing a custom firmware.\nThe method below, GetPublicIPv4AddressAsync, is extremely reliable and fast. The website which tells us our public IP address does not serve up an HTML doc that requires parsing, only a string containing our public IP:\nnamespace AaronLuna.Common.Network { using System.Net; using System.Net.Http; using System.Threading.Tasks; public static partial class NetworkUtilities { public static async Task\u0026lt;IPAddress\u0026gt; GetPublicIPv4AddressAsync() { var urlContent = await GetUrlContentAsStringAsync(\u0026#34;http://ipv4.icanhazip.com/\u0026#34;).ConfigureAwait(false); return ParseSingleIPv4Address(urlContent); } public static async Task\u0026lt;string\u0026gt; GetUrlContentAsStringAsync(string url) { var urlContent = string.Empty; try { using (var httpClient = new HttpClient()) using (var httpResonse = await httpClient.GetAsync(url).ConfigureAwait(false)) { urlContent = await httpResonse.Content.ReadAsStringAsync().ConfigureAwait(false); } } catch (HttpRequestException ex) { // Handle exception  } return urlContent; } } } Conclusion I hope these methods are helpful to you, please download the source code from github and let me know if you have any suggestions or improvements in the comments.\n","href":"/blog/csharp-retrieve-local-public-ip-address/","title":"Network Utility Methods Part 2: Retrieving Local and Public IP Addresses"},{"categories":"dotnet","content":"While creating the TPL Socket extension methods, I ended up with a library of networking functions and IP address parsing/retrieving methods. If you would like to download individual .cs files or a zip file containing the entire library, you can do so at the github link below:\nNetwork Utilities: IP Parsing/Retrieving (C\u0026#35; .NET Core 2.0) [gist.github.com] This library targets the .NET Core 2.0 framework and therefore can be used on Windows, MacOS and any Linux system where the framework is installed. All methods are explicitly designed for IPv4 addresses and return (or accept as parameters) System.Net.IPAddress objects.\nIPv4 Parsing The first method, ParseSingleIPv4Address is less useful than the rest of the library since it requires the input string to already be in proper IPv4 format, \u0026ldquo;a.b.c.d\u0026rdquo;. I am presenting it first because other methods in the library which process unformatted text eventually call ParseSingleIPv4Address to produce a System.Net.IPAddress object.\nThis method calls string.Split on the input string to produce 4 substrings containing the IP address bytes, each byte is validated to ensure it is within the allowed range (0,255). Conveniently, the IPAddress class contains a constructor that accepts a byte array. This constructor is used to produce the IPAddress object represented by the input string.\nnamespace AaronLuna.Common.Network { using System; using System.Linq; using System.Net; public static partial class NetworkUtilities { public static IPAddress ParseSingleIPv4Address(string input) { if (string.IsNullOrEmpty(input)) { throw new ArgumentException(\u0026#34;Input string must not be null\u0026#34;, input); } var addressBytesSplit = input.Trim().Split(\u0026#39;.\u0026#39;).ToList(); if (addressBytesSplit.Count != 4) { throw new ArgumentException(\u0026#34;Input string was not in valid IPV4 format \\\u0026#34;a.b.c.d\\\u0026#34;\u0026#34;, input); } var addressBytes = new byte[4]; foreach (var i in Enumerable.Range(0, addressBytesSplit.Count)) { if (!int.TryParse(addressBytesSplit[i], out var parsedInt)) { throw new FormatException($\u0026#34;Unable to parse integer from {addressBytesSplit[i]}\u0026#34;); } if (0 \u0026gt; parsedInt || parsedInt \u0026gt; 255) { throw new ArgumentOutOfRangeException($\u0026#34;{parsedInt} not within required IP address range [0,255]\u0026#34;); } addressBytes[i] = (byte)parsedInt; } return new IPAddress(addressBytes); } } } The next method parses all IPv4 addresses from a block of text using a regular expression. Each match is then fed into ParseSingleIPv4Address to produce a list of IPAddress objects. This method is much more useful than the previous since it accepts any text and produces a list of IPAddress objects.\nnamespace AaronLuna.Common.Network { using System; using System.Collections.Generic; using System.Net; using System.Text.RegularExpressions; public static partial class NetworkUtilities { public static List\u0026lt;IPAddress\u0026gt; ParseIPv4Addresses(string input) { const string ipV4Pattern = @\u0026#34;(?:(?:1\\d\\d|2[0-5][0-5]|2[0-4]\\d|0?[1-9]\\d|0?0?\\d)\\.){3}(?:1\\d\\d|2[0-5][0-5]|2[0-4]\\d|0?[1-9]\\d|0?0?\\d)\u0026#34;; if (string.IsNullOrEmpty(input)) { throw new ArgumentException(\u0026#34;Input string must not be null\u0026#34;, input); } var ips = new List\u0026lt;IPAddress\u0026gt;(); try { var regex = new Regex(ipV4Pattern); foreach (Match match in regex.Matches(input)) { var ip = ParseSingleIPv4Address(match.Value); ips.Add(ip); } } catch (RegexMatchTimeoutException ex) { // Handle exeption  } return ips; } } } Verify IP Address In CIDR Range Another common scenario is checking if an IP address exists within a range defined in CIDR format. If you are unfamiliar with CIDR notation or networking concepts such as netmasking, read this helpful primer. One way to check if an IP address exists within a CIDR range is described below:\n Perform a bitwise AND operation with the IP address in question and the netmask defined in the CIDR notation Perform another bitwise AND operation with the CIDR address and netmask If the results of 1 and 2 are the same value, the IP address is within the CIDR range  This can be easily demonstrated. Let\u0026rsquo;s use the CIDR range 192.168.2.0/24, which defines a network value 192.168.2.0 and associated netmask 255.255.255.0. Let\u0026rsquo;s check if two IP addresses are part of this address range. First, let\u0026rsquo;s check 192.168.0.15:\n192 168 0 15 192 168 2 0 ip : 1100 0000 - 1010 1000 - 0000 0000 - 0000 1111 cidr : 1100 0000 - 1010 1000 - 0000 0010 - 0000 0000 255 255 255 0 255 255 255 0 mask : 1111 1111 - 1111 1111 - 1111 1111 - 0000 0000 mask : 1111 1111 - 1111 1111 - 1111 1111 - 0000 0000 ----------------------------------------------------- ----------------------------------------------------- 192 168 0 0 192 168 2 0 1100 0000 - 1010 1000 - 0000 0000 - 0000 0000 1100 0000 - 1010 1000 - 0000 0010 - 0000 0000 The results do not match, 192.168.0.0 != 192.168.2.0 which means 192.168.0.15 is not within the CIDR range defined by 192.168.2.0/24.\nNext, let\u0026rsquo;s check 192.168.2.2 using the same process:\n192 168 2 2 192 168 2 0 ip : 1100 0000 - 1010 1000 - 0000 0010 - 0000 0010 cidr : 1100 0000 - 1010 1000 - 0000 0010 - 0000 0000 255 255 255 0 255 255 255 0 mask : 1111 1111 - 1111 1111 - 1111 1111 - 0000 0000 mask : 1111 1111 - 1111 1111 - 1111 1111 - 0000 0000 ----------------------------------------------------- ----------------------------------------------------- 192 168 2 0 192 168 2 0 1100 0000 - 1010 1000 - 0000 0010 - 0000 0000 1100 0000 - 1010 1000 - 0000 0010 - 0000 0000 This time the values match, meaning 192.168.2.2 is within the CIDR range defined by 192.168.2.0/24.\nI created a method to perform the same process, IpAddressIsInCidrRange. This function uses the ParseIPv4Addresses method to parse both the IP address to check and the network address from the CIDR mask. The number of bits that correspond to the network routing is also parsed from the CIDR mask and all values are validated to ensure they are within their allowed range.\nnamespace AaronLuna.Common.Network { using System; using System.Net; public static partial class NetworkUtilities { // true if ipAddress falls inside the range defined by cidrIp, example:  // bool result = IsInCidrRange(\u0026#34;192.168.2.3\u0026#34;, \u0026#34;192.168.2.0/24\u0026#34;); // result = true  public static bool IpAddressIsInRange(string checkIp, string cidrIp) { if (string.IsNullOrEmpty(checkIp)) { throw new ArgumentException(\u0026#34;Input string must not be null\u0026#34;, checkIp); } var ipAddress = ParseIPv4Addresses(checkIp)[0]; return IpAddressIsInRange(ipAddress, cidrIp); } public static bool IpAddressIsInRange(IPAddress checkIp, string cidrIp) { if (string.IsNullOrEmpty(cidrIp)) { throw new ArgumentException(\u0026#34;Input string must not be null\u0026#34;, cidrIp); } var cidrAddress = ParseIPv4Addresses(cidrIp)[0]; var parts = cidrIp.Split(\u0026#39;/\u0026#39;); if (parts.Length != 2) { throw new FormatException($\u0026#34;cidrMask was not in the correct format:\\nExpected: a.b.c.d/n\\nActual: {cidrIp}\u0026#34;); } if (!Int32.TryParse(parts[1], out var netmaskBitCount)) { throw new FormatException($\u0026#34;Unable to parse netmask bit count from {cidrIp}\u0026#34;); } if (0 \u0026gt; netmaskBitCount || netmaskBitCount \u0026gt; 32) { throw new ArgumentOutOfRangeException($\u0026#34;Netmask bit count value of {netmaskBitCount} is invalid, must be in range 0-32\u0026#34;); } var ipAddressBytes = BitConverter.ToInt32(checkIp.GetAddressBytes(), 0); var cidrAddressBytes = BitConverter.ToInt32(cidrAddress.GetAddressBytes(), 0); var cidrMaskBytes = IPAddress.HostToNetworkOrder(-1 \u0026lt;\u0026lt; (32 - netmaskBitCount)); var ipIsInRange = (ipAddressBytes \u0026amp;#38; cidrMaskBytes) == (cidrAddressBytes \u0026amp;#38; cidrMaskBytes); return ipIsInRange; } } } This function is used primarily within my code to check if an IP address is within one of the CIDR blocks reserved for private networks, i.e. check if a given address is a public or private IP address. I created a method named IpAddressIsInPrivateAddressSpace with overloads that accept either a string value or IPAddress object:\nnamespace AaronLuna.Common.Network { using System.Net; public static partial class NetworkUtilities { public const string CidrPrivateAddressBlockA = \u0026#34;10.0.0.0/8\u0026#34;; public const string CidrPrivateAddressBlockB = \u0026#34;172.16.0.0/12\u0026#34;; public const string CidrPrivateAddressBlockC = \u0026#34;192.168.0.0/16\u0026#34;; public static bool IpAddressIsWithinPrivateAddressSpace(string ipv4string) { var checkIp = ParseIPv4Addresses(ipv4string)[0]; return IpAddressIsInPrivateAddressSpace(checkIp); } public static bool IpAddressIsInPrivateAddressSpace(IPAddress ipAddress) { var inPrivateBlockA = IpAddressIsInCidrRange(ipAddress, CidrPrivateAddressBlockA); var inPrivateBlockB = IpAddressIsInCidrRange(ipAddress, CidrPrivateAddressBlockB); var inPrivateBlockC = IpAddressIsInCidrRange(ipAddress, CidrPrivateAddressBlockC); return inPrivateBlockA || inPrivateBlockB || inPrivateBlockC; } } } Conclusion I hope these methods are helpful to you, please download the source code from github and let me know if you have any suggestions or improvements in the comments. In Part 2 I will provide additional Network functions for retrieving public and local IP addresses associated with the machine executing the code, including methods which require and do not require internet access.\n","href":"/blog/csharp-parse-ip-address-check-cidr-range/","title":"Network Utility Methods Part 1: Parsing IP Addresses, Handling CIDR Ranges and Netmasks"},{"categories":"dotnet","content":"The Task-based Asynchronous Pattern (TAP) is the recommended way to write asynchronous code for .NET applications. As I explained in my last post, TAP is intended to replace the Asynchronous Programming Model (APM) and the Event-based Asynchronous Pattern (EAP), however many classes in the .NET framework still use these older patterns. Fortunately, these can be turned into TAP-style \u0026ldquo;awaitable\u0026rdquo; methods with relative ease. By doing so, you reap the benefits that come from working with Task (and Task) objects. In this post, I will convert a set of APM-style methods from the System.Net.Sockets namespace to TAP methods and provide an end-to-end example of how to use them in a generic TCP socket server.\nAll of the code examples in this post can be downloaded as individual .cs files at my gist link below:\nTask-based Socket Extension Methods (C\u0026#35; TPL) [gist.github.com] I\u0026rsquo;ll go through the various Socket methods in the order they would occur for a simple scenario: Connect to a remote host and send a short text message that is received and read by the server over TCP. This code also uses a Result object for all return values. The code for the Result class will also be covered.\nConnect We begin with the assumption that a remote end point exists, defined by an IP address and port number, that is bound and listening for incoming connetions with a TCP socket. We will refer to this side of our example as the server. The other side which attempts to connect to this end point by calling ConnectWithTimeoutAsync will be refered to as the client. ConnectWithTimeoutAsync is an extension method that wraps the BeginConnect/EndConnect methods of the Socket class and returns a Task object: namespace AaronLuna.TplSockets { using System; using System.Net.Sockets; using System.Threading.Tasks; using AaronLuna.Common.Result; public static partial class TplSocketExtensions { public static async Task\u0026lt;Result\u0026gt; ConnectWithTimeoutAsync( this Socket socket, string remoteIpAddress, int port, int timeoutMs) { try { var connectTask = Task.Factory.FromAsync( socket.BeginConnect, socket.EndConnect, remoteIpAddress, port, null); if (connectTask == await Task.WhenAny(connectTask, Task.Delay(timeoutMs)).ConfigureAwait(false)) { await connectTask.ConfigureAwait(false); } else { throw new TimeoutException(); } } catch (SocketException ex) { return Result.Fail($\u0026#34;{ex.Message} ({ex.GetType()})\u0026#34;); } catch (TimeoutException ex) { return Result.Fail($\u0026#34;{ex.Message} ({ex.GetType()})\u0026#34;); } return Result.Ok(); } } }\nNote that this method requires a timeout value. The timeout behavior is achieved through use of Task.WhenAny. Since the same timeout behavior has been applied to all of the socket methods (where appropriate) let\u0026rsquo;s take a closer look at how this is done: if (connectTask == await Task.WhenAny(connectTask, Task.Delay(timeoutMs)).ConfigureAwait(false)) { await connectTask.ConfigureAwait(false); } else { throw new TimeoutException(); }\nWe pass 2 tasks into Task.WhenAny: connectTask and Task.Delay(timeoutMs). Whichever one of these tasks completes first will be returned. Let\u0026rsquo;s say timeoutMs = 5000, which is a timeout value of five seconds. If the task returned from Task.WhenAny is connectTask, the connection was successful and we await the result. However if connectTask is not returned, that means our socket was unable to connect to the endpoint after trying for five seconds and a TimeoutException is thrown. The exception is handled by calling Result.Fail(string error), a static method which returns an instance of Result with Success = false and a string containing the Message property of the exception.\nResult Class, Explained The Result class encapsulates all information pertaining to the outcome of a task, conveniently providing an error message in case the task failed and an object instance in case the task succeeded.\nI was introduced to the Result class by this blog post by Vladimir Khorikov, part of a series that applies Functional programming techniques to C#. He has made the code available on github, but I am still using the basic version below: namespace AaronLuna.Common.Result { public class Result { protected Result(bool success, string error) { Success = success; Error = error; } public bool Success { get; } public string Error { get; } public bool Failure =\u0026gt; !Success; public static Result Fail(string message) { return new Result(false, message); } public static Result\u0026lt;T\u0026gt; Fail\u0026lt;T\u0026gt;(string message) { return new Result\u0026lt;T\u0026gt;(default(T), false, message); } public static Result Ok() { return new Result(true, string.Empty); } public static Result\u0026lt;T\u0026gt; Ok\u0026lt;T\u0026gt;(T value) { return new Result\u0026lt;T\u0026gt;(value, true, string.Empty); } public static Result Combine(params Result[] results) { foreach (Result result in results) { if (result.Failure) { return result; } } return Ok(); } } public class Result\u0026lt;T\u0026gt; : Result { protected internal Result(T value, bool success, string error) : base(success, error) { Value = value; } public T Value { get; } } }\nAccept Ok, back to our socket example. On the other side of our connectTask, the server is waiting for incoming connections inside the AcceptAsync method, which wraps the BeginAccept/EndAcceppt method pair: namespace AaronLuna.TplSockets { using System; using System.Net.Sockets; using System.Threading.Tasks; using AaronLuna.Common.Result; public static partial class TplSocketExtensions { public static async Task\u0026lt;Result\u0026lt;Socket\u0026gt;\u0026gt; AcceptAsync(this Socket socket) { Socket transferSocket; try { var acceptTask = Task\u0026lt;Socket\u0026gt;.Factory.FromAsync(socket.BeginAccept, socket.EndAccept, null); transferSocket = await acceptTask.ConfigureAwait(false); } catch (SocketException ex) { return Result.Fail\u0026lt;Socket\u0026gt;($\u0026#34;{ex.Message} ({ex.GetType()})\u0026#34;); } catch (InvalidOperationException ex) { return Result.Fail\u0026lt;Socket\u0026gt;($\u0026#34;{ex.Message} ({ex.GetType()})\u0026#34;); } return Result.Ok(transferSocket); } } }\nReceive After ConnectWithTimeoutAsync and AcceptAsync have successfully ran to completion, the server will have a new Socket instance, which is named transferSocket in the code example above. The server will use this socket to receive data from the client. I created two diferent ReceiveAsync methods, one with the same timeout behavior as ConnectWithTimeoutAsync and one without: namespace AaronLuna.TplSockets { using System; using System.Net.Sockets; using System.Threading.Tasks; using AaronLuna.Common.Result; public static partial class TplSocketExtensions { public static async Task\u0026lt;Result\u0026lt;int\u0026gt;\u0026gt; ReceiveWithTimeoutAsync( this Socket socket, byte[] buffer, int offset, int size, SocketFlags socketFlags, int timeoutMs) { int bytesReceived; try { var asyncResult = socket.BeginReceive(buffer, offset, size, socketFlags, null, null); var receiveTask = Task\u0026lt;int\u0026gt;.Factory.FromAsync(asyncResult, _ =\u0026gt; socket.EndReceive(asyncResult)); if (receiveTask == await Task.WhenAny(receiveTask, Task.Delay(timeoutMs)).ConfigureAwait(false)) { bytesReceived = await receiveTask.ConfigureAwait(false); } else { throw new TimeoutException(); } } catch (SocketException ex) { return Result.Fail\u0026lt;int\u0026gt;($\u0026#34;{ex.Message} ({ex.GetType()})\u0026#34;); } catch (TimeoutException ex) { return Result.Fail\u0026lt;int\u0026gt;($\u0026#34;{ex.Message} ({ex.GetType()})\u0026#34;); } return Result.Ok(bytesReceived); } public static async Task\u0026lt;Result\u0026lt;int\u0026gt;\u0026gt; ReceiveAsync( this Socket socket, byte[] buffer, int offset, int size, SocketFlags socketFlags) { int bytesReceived; try { var asyncResult = socket.BeginReceive(buffer, offset, size, socketFlags, null, null); bytesReceived = await Task\u0026lt;int\u0026gt;.Factory.FromAsync(asyncResult, _ =\u0026gt; socket.EndReceive(asyncResult)); } catch (SocketException ex) { return Result.Fail\u0026lt;int\u0026gt;($\u0026#34;{ex.Message} ({ex.GetType()})\u0026#34;); } return Result.Ok(bytesReceived); } } }\nSend On the client side, I created an equivalent SendWithTimeoutAsync method, but did not create the version without the timeout. My reasoning for this is that a Send operation is an active process while a Receive operation is a passive one. There are situations where a socket may be waiting for a Send operation on the other side to begin, without any way to predict how long it\u0026rsquo;s going to take until receiving data. In those cases a timeout would not be helpful. This is obviously not the case with a Send operation: namespace AaronLuna.TplSockets { using System; using System.Net.Sockets; using System.Threading.Tasks; using AaronLuna.Common.Result; public static partial class TplSocketExtensions { public static async Task\u0026lt;Result\u0026lt;int\u0026gt;\u0026gt; SendWithTimeoutAsync( this Socket socket, byte[] buffer, int offset, int size, SocketFlags socketFlags, int timeoutMs) { int bytesSent; try { var asyncResult = socket.BeginSend(buffer, offset, size, socketFlags, null, null); var sendBytesTask = Task\u0026lt;int\u0026gt;.Factory.FromAsync(asyncResult, _ =\u0026gt; socket.EndSend(asyncResult)); if (sendBytesTask != await Task.WhenAny(sendBytesTask, Task.Delay(timeoutMs)).ConfigureAwait(false)) { throw new TimeoutException(); } bytesSent = await sendBytesTask; } catch (SocketException ex) { return Result.Fail\u0026lt;int\u0026gt;($\u0026#34;{ex.Message} ({ex.GetType()})\u0026#34;); } catch (TimeoutException ex) { return Result.Fail\u0026lt;int\u0026gt;($\u0026#34;{ex.Message} ({ex.GetType()})\u0026#34;); } return Result.Ok(bytesSent); } } }\nClient/Server Example It\u0026rsquo;s time to put these pieces together and demonstrate how to use these methods and the Result objects:\nnamespace AaronLuna.TplSockets { using System.Linq; using System.Net; using System.Net.Sockets; using System.Text; using System.Threading.Tasks; using AaronLuna.Common.Result; public class TplSocketExample { const int BufferSize = 8 * 1024; const int ConnectTimeoutMs = 3000; const int ReceiveTimeoutMs = 3000; const int SendTimeoutMs = 3000; Socket _listenSocket; Socket _clientSocket; Socket _transferSocket; public async Task\u0026lt;Result\u0026gt; SendAndReceiveTextMesageAsync() { _listenSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp); _clientSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp); _transferSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp); var serverPort = 7003; var ipHostInfo = Dns.GetHostEntry(Dns.GetHostName()); var ipAddress = ipHostInfo.AddressList.Select(ip =\u0026gt; ip) .FirstOrDefault(ip =\u0026gt; ip.AddressFamily == AddressFamily.InterNetwork); var ipEndPoint = new IPEndPoint(ipAddress, serverPort); // Step 1: Bind a socket to a local TCP port and Listen for incoming connections  _listenSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true); _listenSocket.Bind(ipEndPoint); _listenSocket.Listen(5); // Step 2: Create a Task and accept the next incoming connection (ServerAcceptTask)  // NOTE: The call to AcceptConnectionTask is not awaited, therefore this method  // continues executing  var acceptTask = Task.Run(AcceptConnectionTask); // Step 3: With another socket, connect to the bound socket and await the result (ClientConnectTask)  var connectResult = await _clientSocket.ConnectWithTimeoutAsync( ipAddress.ToString(), serverPort, ConnectTimeoutMs).ConfigureAwait(false); // Step 4: Await the result of the ServerAcceptTask  var acceptResult = await acceptTask.ConfigureAwait(false); // If either ServerAcceptTask or ClientConnectTask did not complete successfully,stop execution and report the error  if (Result.Combine(acceptResult, connectResult).Failure) { return Result.Fail(\u0026#34;There was an error connecting to the server/accepting connection from the client\u0026#34;); } // Step 5: Store the transfer socket if ServerAcceptTask was successful  _transferSocket = acceptResult.Value; // Step 6: Create a Task and recieve data from the transfer socket (ServerReceiveBytesTask)  // NOTE: The call to ReceiveMessageAsync is not awaited, therefore this method  // continues executing  var receiveTask = Task.Run(ReceiveMessageAsync); // Step 7: Encode a string message before sending it to the server  var messageToSend = \u0026#34;this is a text message from a socket\u0026#34;; var messageData = Encoding.ASCII.GetBytes(messageToSend); // Step 8: Send the message data to the server and await the result (ClientSendBytesTask)  var sendResult = await _clientSocket.SendWithTimeoutAsync( messageData, 0, messageData.Length, 0, SendTimeoutMs).ConfigureAwait(false); // Step 9: Await the result of ServerReceiveBytesTask  var receiveResult = await receiveTask.ConfigureAwait(false); // Step 10: If either ServerReceiveBytesTask or ClientSendBytesTask did not complete successfully,stop execution and report the error  if (Result.Combine(sendResult, receiveResult).Failure) { return Result.Fail(\u0026#34;There was an error sending/receiving data from the client\u0026#34;); } // Step 11: Compare the string that was received to what was sent, report an error if not matching  var messageReceived = receiveResult.Value; if (messageToSend != messageReceived) { return Result.Fail(\u0026#34;Error: Message received from client did not match what was sent\u0026#34;); } // Step 12: Report the entire task was successful since all subtasks were successful  return Result.Ok(); } async Task\u0026lt;Result\u0026lt;Socket\u0026gt;\u0026gt; AcceptConnectionTask() { return await _listenSocket.AcceptAsync().ConfigureAwait(false); } async Task\u0026lt;Result\u0026lt;string\u0026gt;\u0026gt; ReceiveMessageAsync() { var message = string.Empty; var buffer = new byte[BufferSize]; var receiveResult = await _transferSocket.ReceiveWithTimeoutAsync( buffer, 0, BufferSize, 0, ReceiveTimeoutMs).ConfigureAwait(false); var bytesReceived = receiveResult.Value; if (bytesReceived == 0) { return Result.Fail\u0026lt;string\u0026gt;(\u0026#34;Error reading message from client, no data was received\u0026#34;); } message = Encoding.ASCII.GetString(buffer, 0, bytesReceived); return Result.Ok(message); } } } Summary As stated at the beginning of this post, all of the source code for these extension methods and the example can be found at the gist embedded below:\nTask-based Socket Extension Methods (C\u0026#35; TPL) [gist.github.com] ","href":"/blog/csharp-tpl-socket-methods/","title":"Task-based Socket Extension Methods (C# TPL)"},{"categories":"dotnet","content":"C# classes contain a mixture of three asynchronous patterns:\n Asynchronous Programming Model (APM) uses the IAsyncResult interface and requires async methods to be defined as BeginProcess and EndProcess methods (e.g., \u0026lt;code\u0026gt;BeginSend\u0026lt;/code\u0026gt;/\u0026lt;code\u0026gt;EndSend\u0026lt;/code\u0026gt; methods for asynchronous Socket send operations). Event-based Asynchronous Pattern (EAP) was introduced with .NET Framework 2.0 and requires that asynchronous method names end with \u0026quot;Async\u0026quot; and uses event types, delegates, and custom EventArgs classes. Task-based Asynchronous pattern (TAP) was introduced in .NET Framework 4.0 and Microsoft recommends that you use TAP for new projects. TAP uses Task-objects and only requires a single asynchronous method which can be \u0026ldquo;awaited\u0026rdquo;, in contrast to APM and EAP which require more than one method to achieve asynchrony. Like EAP, you should end your TAP method names with \u0026ldquo;Async\u0026rdquo;.  However, if you must use APM instead of TAP, do not mix async/await code with code which calls Task.Result and Task.Wait(), your code must be .Result/.Wait() all the way down (or async/await all the way down). Combining the two makes it extremely likely that you will encounter a deadlock at some point.\nThe sole exception to this rule used to be the static main entry method for a console app, which would not compile if defined as static async Task\u0026lt;int\u0026gt; Main(string[] args). If your main program relied on an async method call, you had to workaround this issue with the solution below:\nstatic int Main(string[] args) { DoAsyncWork().GetAwaiter().GetResult(); Console.WriteLine(\u0026#34;\\nPress ENTER to continue...\u0026#34;); Console.Read(); return 0; } static async Task DoAsyncWork() { await ExpensiveComputationAsync(); } However, with the release of C# 7.1 your Main method can now be async:\nstatic async Task\u0026lt;int\u0026gt; Main(string[] args) { await ExpensiveComputationAsync(); Console.WriteLine(\u0026#34;\\nPress ENTER to continue...\u0026#34;); Console.Read(); return 0; } If your main method does not return a value for the exit code, you can also define a main method that returns a Task object:\nstatic async Task Main(string[] args) { await AnotherAsyncMethod(); } I will give an example of a console app that uses this new language feature in a future post. Since TAP is the recommended pattern, let\u0026rsquo;s focus on best practices for async code that relies on the Task Parallelism Library (TPL).\nTPL Best Practices Avoid using Task.Factory.StartNew in almost every scenario in favor of Task.Run The reasons for this are explained in this blog post by Stephen Cleary. The major reason to avoid StartNew is because it does not understand async delegates. Rather than the Task returned by StartNew representing the async delegate, it represents only the beginning of the delegate. Please read his post for more detail and examples of the pitfalls introduced by indiscriminate use of StartNew.\nCleary also argues that the options available with StartNew such as LongRunning and PreferFairness should only be used after an application has been profiled to ensure these options are actually going to have a significant impact. Typically, using Task.Run will provide nearly the same efficiency.\nAvoid async void in every scenario besides event handlers In an article from MSDN Magazine, Cleary gives three guidelines for using async/await. async methods can only have three return types: Task, Task\u0026lt;T\u0026gt; and void. The only reason void is allowed is to enable asynchronous event handlers. async event handlers are necessary but can be dangerous because exceptions thrown from async void methods can\u0026rsquo;t be caught in the normal way with a try/catch block. In C#, these exceptions can be caught by using the catch-all AppDomain.UnhandledException\nAnother reason to avoid async void is that, unlike Task and Task\u0026lt;T\u0026gt; objects, methods returning void cannot be awaited or used with methods like Task.WhenAny and Task.WhenAll. This makes it difficult to determine the status of the method and whether it has completed. Because of these issues with exception handling and status monitoring, async void methods are difficult to unit test. Please see the article for detailed examples.\nawait the result of ConfigureAwait(false) whenever you can Doing so is especially important when you are creating a library that will be used by client code. Consider the method below which retrieves the text of a webpage:\npublic async Task\u0026lt;string\u0026gt; GetUrlContentAsString() { using (var httpClient = new HttpClient()) using (var httpResonse = await httpClient.GetAsync(\u0026#34;https://aaronluna.dev\u0026#34;)) { return await httpResonse.Content.ReadAsStringAsync(); } } If every client would call our method like this: await GetUrlContentAsString();, everything would be perfect. Of course, this will not always be the case. Now, consider what would happen if a client were to do the following:\npublic void GetUrlContentButton_Clicked(object sender, RoutedEventArgs e) { var urlContents = GetUrlContentAsString().Result; } GUI applications have a SynchronizationContext that permits only one chunk of code to run at a time. When the await completes, it attempts to execute the remainder of the async method within the captured context. But that context already has a thread in it, which is (synchronously) waiting for the async method to complete. They’re each waiting for the other, causing a deadlock.\nThis can be fixed by adding ConfigureAwait(false) to our original method wherever await is used:\npublic async Task\u0026lt;string\u0026gt; GetUrlContentAsString() { using (var httpClient = new HttpClient()) using (var httpResonse = await httpClient.GetAsync(\u0026#34;https://aaronluna.dev\u0026#34;).ConfigureAwait(false)) { return await httpResonse.Content.ReadAsStringAsync().ConfigureAwait(false); } } To summarize this third guideline, you should use Configure­Await(false) when possible. Context-free code has better performance for GUI applications and is a useful technique for avoiding deadlocks when working with a partially async codebase. If you\u0026rsquo;re creating a library that’s potentially shared with desktop applications, consider using ConfigureAwait(false) in the library code.\n","href":"/blog/parallel-async-csharp-best-practices-tpl/","title":"Asynchronous Programming Best Practices in C#"},{"categories":"Virtualization","content":"The majority of my work consists of C#/.NET development, and Visual Studio+ReSharper is my preferred IDE. Since I\u0026rsquo;m a Mac user, I run VS on a Windows 10 instance with VMWare Fusion. For the first month or so everything ran smoothly. However, the performance of the VM degraded significantly over time, especially with multiple Visual Studio projects open in the VM and resource-hungry programs running on the Mac simultaneously.\nThankfully, after changing a few settings and clearing unnecessary files I was able to improve the speed and responsiveness of both my Mac and the Windows 10 instance. I\u0026rsquo;ve documented these steps below to help others who find themselves in a similar situation.\nSettings in VMWare Fusion I found that a combination of 2 processor cores with 4096 MB of memory works best for my needs (Note: My MacBook Pro has a 2.8GHz Intel quad core i7 with 8 GB RAM and Intel Iris GPU with 1536 MB). Also, select \u0026ldquo;Enable hypervisor applications in this virtual machine\u0026rdquo; as shown in Figure 1.\n Figure 1 - Processors \u0026amp; Memory Settings\n  I also saw a performance boost by changing the driver for the hard drive bus type to SCSI. On the Hard Disk Settings page, expand the \u0026ldquo;Advanced options\u0026rdquo; menu and change the Bus type to SCSI. Also, select \u0026ldquo;Pre-allocate disk space\u0026rdquo; as shown in Figure 2:\n Figure 2 - Hard Disk Settings\n  Settings in the .VMX File VMWare only exposes a small number of settings through the UI. There are hundreds of additional configuration options which can only be modified by editing the .vmx file.\n  You should always make a backup of the .vmx file before modifying it. You can easily make changes that render the VM unusable or unable to boot!\n  Before changing the .vmx file itself, launch TextEdit and open the Preferences menu. Make sure Smart quotes is unchecked (See Figure 3 below). If this option is checked and you save the .vmx file, you will not be able to boot your VM.\n Figure 3 - TextEdit Preferences menu showing location of Smart quotes setting\n  The easiest way to access the .vmx file is from the list of virtual machines in the main VMWare Fusion window. If the list of virtual machines is not visible, go to Window -\u0026gt; Virtual Machine Library.\n Figure 4 - The virtual machine library can be accessed from the Window menu\n  Next, shutdown the VM. Ensure that it is completely shutdown and not in a suspended state.\nRight click on the VM as shown in Figure 5. If you press and hold the ⌥ (ALT) key, Show in Finder will change to Open Config File in Editor.\n Figure 5 - Menu options available with the ALT key pressed\n  There are other hidden options within this menu, try holding the ^ (CTRL) key or ⌘ (CMD) key and the menu item will change to Show Config File in Finder and Open Latest Log File, respectively.\nRight-click on your VM while holding the ⌥ (ALT) key and select Open Config File in Editor. This will launch TextEdit. Make the following changes to the .vmx file:\n  Find ethernet0.virtualDev = “e1000e” and change it to ethernet0.virtualDev = “vmxnet3”.\nE1000E is the default value and is recognized by Windows as an Intel 82574L Gigabit NIC. VMXNET3 is a virtual NIC which is optimized for use in a VM and is not based on a physical part (Figure 6 shows how the two NICs are displayed in Windows) Changing this value should drastically improve network performance. If you would like to read a detailed comparison of these two NICs and others which can be used with VMWare, check out this page.\n Figure 6 - Network adapter properties for E1000E and VMXNET3 within Windows\n    Add the following lines to the .vmx file:\n  Table 1   Configuration settings to add to .vmx file1   Add to .VMX File Notes     scsi0:0.virtualSSD = 1 Optimizes disk I/O for SSD. Only use this if your Mac has an SSD!   prefvmx.useRecommendedLockedMemSize = “TRUE” Speed up I/O at the cost of increased memory usage in the host OS   sched.mem.pshare.enable = “FALSE” Disables page sharing   MemAllowAutoScaleDown = “FALSE” Disables scale down of memory allocation     1Source: http://artykul8.com/2012/06/vmware-performance-enhancing/       Figure 7 - .VMX file with additional settings added\n    Windows Update Cleanup One of the main causes of slower performance in my Windows VM turned out to be caused by Windows Update. I noticed that I was constantly running out of hard drive space, and would allocate more space from the host OS to improve performance. It turns out that Windows keeps copies of all updates, and performing a normal Disk Cleanup does not remove these files even when they are completely redundant.\n Figure 8 - Disk Cleanup Tool\n  The Disk Cleanup tool is located at Control Panel -\u0026gt; System and Security -\u0026gt; Administrative Tools. Note that in Figure 8, performing the cleanup will free up less than 10 MB. You need to click the \u0026ldquo;Clean up system files\u0026rdquo; button and select the checkbox for \u0026ldquo;Windows Update Cleanup\u0026rdquo; as shown in Figure 9:\n Figure 9 - Disk Cleanup Tool with Windows Update Cleanup option selected\n  Now, the Disk Cleanup tool shows that 3 GB of space will be reclaimed. Running this tool once a month should improve the performance of your Windows 10 VM dramatically. The cleanup process took approximately 45 minutes to complete when I ran it for the first time.\n Figure 10 - Disk Cleanup in progress\n  Virtual Memory Settings in Windows 10 Another way to realize performance gains is by adjusting the size of the paging file. Right-click the Start button and select \u0026ldquo;Settings\u0026rdquo;. Enter \u0026ldquo;performance\u0026rdquo; in the search box and select \u0026ldquo;Adjust the appearance and performance of Windows\u0026rdquo; as shown in Figure 11:\n Figure 11 - Windows Settings Search Results\n  This will open Performance Options. Navigate to the Advanced tab and click the Change… button. Uncheck the \u0026ldquo;Automatically manage paging file size for all drives\u0026rdquo; checkbox and select the radio button for \u0026ldquo;Custom Size\u0026rdquo;. For both Initial and Maximum Size enter twice the amount of RAM that the VM has allocated. For my setup, this equals 2 * 4096 MB = 8192 MB. See Figure 12:\n Figure 12 - Changes to paging file size within Performance Options\n  Click the Set button and reboot the VM in order to apply the changes.\nRemove AppData Folders If Visual Studio is still sluggish, you may be experiencing the issue described in this blog post. Using the extremely useful Process Monitor (the link in the blog post is broken and outdated), the author found that Visual Studio was accessing the location C:\\Users...\\AppData\\Local\\Microsoft\\WebSiteCache on nearly every IDE operation. The author zipped the folder contents and moved them elsewhere, but it appears from the comments that the files are not needed. Moving/deleting the contents of this directory completely fixed the sluggishness in Visual Studio.\nHere is a link to Sysinternals Suite which is a collection of free troubleshooting utilties from Microsoft (Process Monitor is part of the suite). Again, I am linking this because the link in the original blog post is broken and outdated.\n","href":"/blog/optimize-vm-performance-windows10-visual-studio-vmware-fusion/","title":"How to Optimize Performance of Windows 10 and Visual Studio in VMWare Fusion"},{"categories":"Linux","content":"Why would you want to install NGINX from source code rather than a pre-built package? The most important reason is that the libraries which NGINX depends on (PCRE, zlib, OpenSSL) are part of the pre-built package, and building from source allows you to use the latest versions which may contain vital security patches.\nFor instance, you will find that the version of NGINX available through your package manager is several major versions behind the current mainline. Note, for Ubuntu systems, you can find pre-built packages which are much closer to the current mainline from Launchpad\u0026rsquo;s Personal Package Archive service.\nAnother reason to install from source is that NGINX can be configured in many different ways, and the only way to choose how it is configured is to install from source. When you install NGINX from a pre-built package, you are stuck with whatever set of modules are enabled. In this post, I will show how to include third-party modules and how to enable modules which are disabled by default.\nIf you just came here for the script, you can download the completed shell script from the gist linked below. This script installs the latest mainline NGINX version along with the latest versions of PCRE, zlib and OpenSSL libraries, and includes two useful third-party modules:\nShell Script: Build and Install NGINX from Source (Ubuntu) [github.com] The rest of this post will explain in detail how to install NGINX from source. It does not match my shell script exactly since the script is designed to run non-interactively and uses variables. The steps below require user input and no variables are used.\nExpected Results These steps produce a .deb package which can be used to uninstall this version of NGINX via apt-get (the .deb package can also be used to install this customized version of NGINX on another system with the same architecture). You can find information on the checkinstall program which creates the .deb package here.\nAfter NGINX is built and installed, all source code files are added to a .tar.gz archive file. This allows you to avoid downloading the source code again if you use the .deb package to install this version of NGINX on another system.\nInstall Prerequisites and Dependancies This guide is written for Ubuntu, but the steps only need to be changed in minor ways to apply to other Linux distributions.\nThe steps also assume that you are logged in with a user account that has sudo access. If you are unsure what this means, please perform the steps in this guide from DigitalOcean.\nIf you are using a different Linux distribution the steps to create a user with sudo access will be similar, try a google search with your distro\u0026rsquo;s name and \u0026ldquo;root privileges\u0026rdquo;.\n  To begin, add the Maxmind PPA to your list of sources. This PPA has much newer versions of the GeoIP libraries and databases than the default apt source:\n~$ sudo add-apt-repository ppa:maxmind/ppa -y gpg: keyring `/tmp/tmpmku7ishg/secring.gpg' created gpg: keyring `/tmp/tmpmku7ishg/pubring.gpg' created gpg: requesting key DE742AFA from hkp server keyserver.ubuntu.com gpg: /tmp/tmpmku7ishg/trustdb.gpg: trustdb created gpg: key DE742AFA: public key \"Launchpad PPA for MaxMind\" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) OK   Update your system (Depending on your Ubuntu version and configuration, libgeoip1 may be updated from the PPA added in the previous step):\n~$ sudo apt update \u0026\u0026 sudo apt upgrade -y #removed messages detailing update process The following packages will be upgraded: libgeoip1 1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 76.7 kB of archives. After this operation, 3,072 B disk space will be freed. #removed messages detailing upgrade process   Remove any libraries which are no longer needed:\n~$ sudo apt autoremove -y Reading package lists... Done Building dependency tree Reading state information... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.   NGINX is written in C, so we need to install GCC, make and other tools:\n~$ sudo apt install build-essential -y #removed messages detailing install process   We are going to install a third-party GeoIP module, but in order to do so we must first install these libraries:\n~$ sudo apt install libmaxminddb0 libmaxminddb-dev mmdb-bin -y #removed output messages detailing install process   Install checkinstall:\n~$ sudo apt install checkinstall -y #removed output messages detailing install process   Install Uncomplicated Firewall (UFW) which makes it easier to manage rules for ports used by NGINX:\n~$ sudo apt install ufw -y #removed output messages detailing install process   Prepare for Install  Next, create a directory to store source files:\n~$ sudo mkdir -p /opt/src_files \u0026\u0026 cd /opt/src_files /opt/src_files $   PLEASE DO NOT USE A FOLDER IN YOUR USER\u0026rsquo;S HOME DIRECTORY. If you plan on using the .deb file to install your custom version of NGINX on another system, the source files must be located in the same path on each system. If you store the source files in a directory in this user\u0026rsquo;s home directory, you will only be able to use the .deb package on another system with a user who\u0026rsquo;s name is exactly the same.\n    Download and extract the source code for the latest version of NGINX:\n/opt/src_files $ sudo wget http://nginx.org/download/nginx-1.15.6.tar.gz \u0026\u0026 \\  sudo tar xzf nginx-1.15.6.tar.gz #removed messages detailing download progress 2018-11-26 16:36:19 (2.98 MB/s) - ‘nginx-1.15.6.tar.gz’ saved [1025761/1025761]   Download dependencies (PCRE, zlib and OpenSSL) and extract the source code:\n/opt/src_files $ sudo wget https://downloads.sourceforge.net/project/pcre/pcre/8.42/pcre-8.42.tar.gz \u0026\u0026 \\  sudo tar xzf pcre-8.42.tar.gz #removed messages detailing download progress 2018-11-26 16:40:14 (5.18 MB/s) - ‘pcre-8.42.tar.gz’ saved [2081413/2081413] /opt/src_files $ sudo wget http://zlib.net/zlib-1.2.11.tar.gz \u0026\u0026 \\  sudo tar xzf zlib-1.2.11.tar.gz #removed messages detailing download progress 2018-11-26 16:41:56 (2.80 MB/s) - ‘zlib-1.2.11.tar.gz’ saved [607698/607698] /opt/src_files $ sudo wget https://www.openssl.org/source/openssl-1.1.1a.tar.gz \u0026\u0026 \\  sudo tar xzf openssl-1.1.1a.tar.gz #removed messages detailing download progress 2018-11-26 16:42:56 (6.66 MB/s) - ‘openssl-1.1.1a.tar.gz’ saved [8350547/8350547] You can find the latest version for each library at the following locations:\n  https://www.pcre.org   NGINX requires the original PCRE library, NOT PCRE2\n    http://zlib.net   https://www.openssl.org/source/   NGINX can be built with any 1.x.x version of OpenSSL, but using the latest (1.1.1) is strongly recommended since this version includes suport for TLS 1.3, the latest and most secure version of the encryption protocol.\n      Download third-party NGINX modules: cache purge and GeoIP2:\nFeel free to omit this step or substitute different third-party modules, depending on your website's requirements. In that case, you will need to modify the configure command in Step 13\nIf you do not want to install any third-party modules, remove the two lines which start with \u0026#8211;\u0026#8211;add\u0026#8211;module\u0026#61; (lines 49-50). If you are installing different modules, modify these lines to point to folders containing module source code.\n/opt/src_files $ sudo git clone --recursive https://github.com/FRiCKLE/ngx_cache_purge.git Cloning into 'ngx_cache_purge'... #removed messages detailing download progress /opt/src_files $ sudo git clone --recursive https://github.com/leev/ngx_http_geoip2_module.git Cloning into 'ngx_http_geoip2_module'... #removed messages detailing download progress The GeoIP module included with NGINX only works with v1 MaxMind database files. V2 database provide better info than v1, to use v2 files on NGINX you will need to install this module.\n  Build Configuration  We are ready to begin building NGINX. Delete all .tar.gz files. After doing so, your directory should contain folders for each program/library downloaded in the previous steps. After confirming this is the case, navigate to the folder containing the NGINX source code:\n/opt/src_files $ sudo rm -rf *.tar.gz /opt/src_files $ ls -al total 32 drwxr-xr-x 8 root root 4096 Nov 26 17:23 . drwxr-xr-x 3 root root 4096 Nov 26 16:34 .. drwxr-xr-x 8 1001 1001 4096 Nov 6 13:32 nginx-1.15.6 drwxr-xr-x 4 root root 4096 Nov 26 17:20 ngx_cache_purge drwxr-xr-x 3 root root 4096 Nov 26 17:20 ngx_http_geoip2_module drwxr-xr-x 19 root root 4096 Nov 20 13:35 openssl-1.1.1a drwxr-xr-x 7 1169 1169 4096 Mar 20 2018 pcre-8.42 drwxr-xr-x 14 501 staff 4096 Jan 15 2017 zlib-1.2.11 /opt/src_files $ cd nginx-1.15.6 /opt/src_files/nginx-1.15.6 $   This is the most important step. With this ./configure command, you enable/disable modules and manage all configuration settings:\n  The values below for OpenSSL, PCRE and zlib library folder paths (Line #18, 21 and 23, respectively) are valid for the actions performed in Step 10. If you saved the source code somewhere besides /opt/src_files, modify the config arguments to use the correct location.\n  1 /opt/src_files/nginx-1.15.6 $ sudo ./configure \\ 2  --prefix=/usr/share/nginx \\ 3  --sbin-path=/usr/sbin/nginx \\ 4  --modules-path=/usr/lib/nginx/modules \\ 5  --conf-path=/etc/nginx/nginx.conf \\ 6  --error-log-path=/var/log/nginx/error.log \\ 7  --http-log-path=/var/log/nginx/access.log \\ 8  --pid-path=/var/run/nginx.pid \\ 9  --lock-path=/var/lock/nginx.lock \\ 10  --user=www-data \\ 11  --group=www-data \\ 12  --build=Ubuntu \\ 13  --http-client-body-temp-path=/var/lib/nginx/body \\ 14  --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \\ 15  --http-proxy-temp-path=/var/lib/nginx/proxy \\ 16  --http-scgi-temp-path=/var/lib/nginx/scgi \\ 17  --http-uwsgi-temp-path=/var/lib/nginx/uwsgi \\ 18  --with-openssl=/opt/src_files/openssl-1.1.1a \\ 19  --with-openssl-opt=enable-ec_nistp_64_gcc_128 \\ 20  --with-openssl-opt=no-weak-ssl-ciphers \\ 21  --with-pcre=/opt/src_files/pcre-8.42 \\ 22  --with-pcre-jit \\ 23  --with-zlib=/opt/src_files/zlib-1.2.11 \\ 24  --with-compat \\ 25  --with-file-aio \\ 26  --with-threads \\ 27  --28  --with-http_auth_request_module \\ 29  --with-http_dav_module \\ 30  --with-http_flv_module \\ 31  --with-http_gunzip_module \\ 32  --with-http_gzip_static_module \\ 33  --with-http_mp4_module \\ 34  --with-http_random_index_module \\ 35  --with-http_realip_module \\ 36  --with-http_slice_module \\ 37  --with-http_ssl_module \\ 38  --with-http_sub_module \\ 39  --with-http_stub_status_module \\ 40  --with-http_v2_module \\ 41  --with-http_secure_link_module \\ 42  --with-mail \\ 43  --with-mail_ssl_module \\ 44  --with-stream \\ 45  --with-stream_realip_module \\ 46  --with-stream_ssl_module \\ 47  --with-stream_ssl_preread_module \\ 48  --with-debug \\ 49  --add-module=../ngx_http_geoip2_module \\ 50  --add-module=../ngx_cache_purge \\ If you need to disable a module which is enabled by default, please read these instructions. All arguments of the form \u0026#8211;\u0026#8211;with*_module enable NGINX modules which are by default disabled. If you need to do the opposite (i.e. disable a module which by default is enabled when installing NGINX) (e.g. the fastcgi module), you would add \u0026#8211;\u0026#8211;without\u0026#8211;http_fastcgi_module to the list of arguments.\nFor more information on how you can customize your install with arguments supplied to the configure command, see the links below:\nhttp://nginx.org/en/docs/configure.html (https://www.nginx.com/resources/wiki/start/topics/tutorials/installoptions/)    When you are 100% confident that every requirement of your NGINX installation is satisfied, run the ./configure command. When the configuration is complete, you should see a report of the configuration settings that will be used to build NGINX.\n1 /opt/src_files/nginx-1.15.6 $ sudo ./configure  2 #removed configuration arguments 3 #removed verification statements 4 configuring additional modules 5 adding module in ../ngx_http_geoip2_module 6 checking for MaxmindDB library ... found 7 + ngx_geoip2_module was configured 8 adding module in ../ngx_cache_purge 9 + ngx_http_cache_purge_module was configure 10 creating objs/Makefile 11 12 Configuration summary 13 + using threads 14 + using PCRE library: /opt/src_files/pcre-8.42 15 + using OpenSSL library: /opt/src_files/openssl-1.1.1a 16 + using zlib library: /opt/src_files/zlib-1.2.11 17 18 nginx path prefix: \"/usr/share/nginx\" 19 nginx binary file: \"/usr/sbin/nginx\" 20 nginx modules path: \"/usr/lib/nginx/modules\" 21 nginx configuration prefix: \"/etc/nginx\" 22 nginx configuration file: \"/etc/nginx/nginx.conf\" 23 nginx pid file: \"/var/run/nginx.pid\" 24 nginx error log file: \"/var/log/nginx/error.log\" 25 nginx http access log file: \"/var/log/nginx/access.log\" You should be able to confirm several of the configuration settings have been correctly defined:\n Lines 4-9 confirm that the third-party modules downloaded in Step 11 (GeoIP2 and cache purge) are configured correctly (also, the MaxMind libraries installed in Step 5 were successfully located). Lines 12-16 confirm that the versions of PCRE, zlib and OpenSSL downloaded in Step 10 will be used to build NGINX Lines 18-25 confirm various settings such as where to store the NGINX binary, location of NGINX configuration files, location of log files, etc.    After finalizing your configuration, build NGINX:\n/opt/src_files/nginx-1.15.6 $ sudo make #removed thousands of build statements #eventually, you should see a message like this: sed -e \"s|%%PREFIX%%|/usr/share/nginx|\" \\ -e \"s|%%PID_PATH%%|/var/run/nginx.pid|\" \\ -e \"s|%%CONF_PATH%%|/etc/nginx/nginx.conf|\" \\ -e \"s|%%ERROR_LOG_PATH%%|/var/log/nginx/error.log|\" \\ objs/nginx.8 make[1]: Leaving directory '/opt/src_files/nginx-1.15.6' At this point, you are probably thinking that the next step is to run sudo make install. Doing so would produce a working install of NGINX, finely calibrated to our unique demands, but would deprive us of benefits that we have come to expect when we install a program using apt or any other package management product (pip, npm, brew, etc). We can achieve a few of these benefits by utilizing the checkinstall program that we installed in Step 6.\ncheckinstall is a simple program which monitors the installation of files, and creates a Debian package from them. There are two primary benefits to using checkinstall instead of running make install:\n You can easily remove the package with one step. You can install the resulting package upon multiple machines.  After building the binary, you are ready to use checkinstall to create a .deb package for your custom NGINX configuration.\n  Create Debian Install Package  Run checkinstall and specify values for two flags:\n --install=no will cause ONLY a .deb package to be created, the default behavior is to install the binary and create the .deb package. -y will create the .deb package non-interactively, accepting the default value for all prompts that are normally presented to the user.  /opt/src_files/nginx-1.15.6 $ sudo checkinstall --install=no -y #removed installation messages #eventually, you should see a message like this: ********************************************************************** Done. The new package has been saved to\n/opt/src_files/nginx-1.15.6/nginx_1.15.6-1_amd64.deb You can install it in your system anytime using:\n dpkg -i nginx_1.15.6-1_amd64.deb  **********************************************************************\n  You can examine the contents of the .deb package to see the target location of all files that will be copied to your system during installation:\n1 /opt/src_files/nginx-1.15.6 $ dpkg --contents nginx_1.15.6-1_amd64.deb 2 drwxr-xr-x root/root 0 2018-11-27 12:24 ./ 3 drwxr-xr-x root/root 0 2018-11-27 12:24 ./etc/ 4 drwxr-xr-x root/root 0 2018-11-27 12:24 ./etc/nginx/ 5 #removed listing of files in nginx config folder 6 drwxr-xr-x root/root 0 2018-11-14 21:43 ./usr/ 7 drwxr-xr-x root/root 0 2018-11-27 12:24 ./usr/share/ 8 drwxr-xr-x root/root 0 2018-11-27 12:24 ./usr/share/nginx/ 9 drwxr-xr-x root/root 0 2018-11-27 12:24 ./usr/share/nginx/html/ 10 #removed listing of files in default website root 11 drwxr-xr-x root/root 0 2018-11-27 12:24 ./usr/share/doc/ 12 drwxr-xr-x root/root 0 2018-11-27 12:24 ./usr/share/doc/nginx/ 13 #removed listing of files in nginx documentation folder 14 drwxr-xr-x root/root 0 2018-11-27 12:24 ./usr/sbin/ 15 -rwxr-xr-x root/root 4820568 2018-11-27 12:24 ./usr/sbin/nginx The folder paths listed in the .deb package are taken from values provided to the ./configure command in Step 13:\n /etc/nginx is the root path for NGINX configuration files, this location is taken from the value --conf-path=/etc/nginx/nginx.conf (Step 13, Line #5) /usr/share/nginx is the folder used to serve html content in the default configuration, this location is taken from the value --prefix=/usr/share/nginx (Step 13, Line #2) /usr/sbin is the location of the NGINX binary and **/usr/sbin/nginx** is the binary file, this location (and the name of the binary itself) is taken from the value --sbin-path=/usr/sbin/nginx (Step 13, Line #3)  The location used to store NGINX documentation (/usr/share/doc/nginx) is NOT controlled by the ./configure command. This directory was created because we ran checkinstall with the -y flag, answering \"yes\" to the prompt asking if you want to create the documentation directory. There are several values you can customize when creating the .deb package, please see the official documentation for more details.\n  Install and Configure NGINX  We are finally ready to install NGINX using the .deb package:\n/opt/src_files/nginx-1.15.6 $ sudo dpkg -i nginx_1.15.6-1_amd64.deb (Reading database ... 56508 files and directories currently installed.) Preparing to unpack nginx_1.15.6-1_amd64.deb ... Unpacking nginx (1.15.6-1) over (1.15.6-1) ... Setting up nginx (1.15.6-1) ... Installing with the .deb package (rather than the usual sudo make install) allows our custom version of NGINX to be cleanly uninstalled with either command below:\n sudo apt-get remove nginx sudo dpkg \u0026#8211;\u0026#8211;remove nginx  Now that the installation is complete, we are going to create an archive file from the source code we used to build NGINX.\n  Move the .deb package to a new folder since we want to keep it separate from the source code files. Navigate to the location of the .deb file after you have moved it:\n  I created a new folder, /opt/save, and moved the .deb file to this location. Feel free to use any folder other than the directory containing the source files.\n  /opt/src_files/nginx-1.15.6 $ sudo mkdir -p /opt/save /opt/src_files/nginx-1.15.6 $ sudo mv nginx_1.15.6-1_amd64.deb /opt/save /opt/src_files/nginx-1.15.6 $ cd /opt/save /opt/save $   Create a tar file from the directory containing the source code files. Modify the command accordingly if you used a different directory:\n/opt/save $ sudo tar -zcf nginx_1.15.6-src_files.tar.gz /opt/src_files tar: Removing leading `/' from member names   The current folder contain two files: the .deb package and a .tar.gz file containing the downloaded source code. Make both files executable by all users:\n/opt/save $ sudo chmod 755 nginx*.* /opt/save $ ls -al total 186452 drwxr-xr-x 2 root root 4096 Nov 28 17:43 . drwxr-xr-x 4 root root 4096 Nov 28 17:37 .. -rwxr-xr-x 1 root root 1841212 Nov 27 12:24 nginx_1.15.6-1_amd64.deb -rwxr-xr-x 1 root root 189067874 Nov 28 17:47 nginx_1.15.6-src_files.tar.gz   For some reason NGINX does not create a folder at /var/lib/nginx even though this location is specified several times in our list of configure arguments in Step 13. Create this directory to ensure the config file passes verification in the next step:\n/opt/save $ sudo mkdir -p /var/lib/nginx   The nginx -t command checks the nginx.conf file for syntax errors and performs serveral other helpful tests as well. It looks at all included config files and reports if there are any issues with permissions, file-not-found, etc. I recommend making a habit of running this command everytime you modify your web server's configuration to check for errors you may have introduced.\nRunning the command at this point should report that the config file syntax is error-free and passes the verification tests:\n/opt/save $ sudo nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Another usefu command is nginx -V. This will report the version number of NGINX as well as the version of OpenSSL that was used to build the binary. The values that were provided to configure the build (Step 13) are also reported:\n/opt/save $ sudo nginx -V nginx version: nginx/1.15.6 (Ubuntu) built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) built with OpenSSL 1.1.1a 20 Nov 2018 TLS SNI support enabled configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --user=www-data --group=www-data --build=Ubuntu --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-openssl=/opt/src_files/openssl-1.1.1a --with-openssl-opt=enable-ec_nistp_64_gcc_128 --with-openssl-opt=no-weak-ssl-ciphers --with-pcre=/opt/src_files/pcre-8.42 --with-pcre-jit --with-zlib=/opt/src_files/zlib-1.2.11 --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_slice_module --with-http_ssl_module --with-http_sub_module --with-http_stub_status_module --with-http_v2_module --with-http_secure_link_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-debug --add-module=../ngx_http_geoip2_module --add-module=../ngx_cache_purge --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed'   If your config file passes validation, create two folders as shown below which will contain config settings for the website(s) which this server is hosting:\n/opt/save $ sudo mkdir /etc/nginx/sites-available /opt/save $ sudo mkdir /etc/nginx/sites-enabled   To easily manage the ports NGINX will use for HTTP/HTTPS traffic, create a UFW profile (feel free to use a different text editor than nano):\n/opt/save $ sudo nano /etc/ufw/applications.d/nginx Copy the text below into the file:\n[Nginx HTTP] title=Web Server (Nginx, HTTP) description=Small, but very powerful and efficient web server ports=80/tcp [Nginx HTTPS] title=Web Server (Nginx, HTTPS) description=Small, but very powerful and efficient web server ports=443/tcp [Nginx Full] title=Web Server (Nginx, HTTP + HTTPS) description=Small, but very powerful and efficient web server ports=80,443/tcp  Save and close the file.\n  Verify the UFW profiles for NGINX are available:\n/opt/save $ sudo ufw app list Available applications: Nginx Full Nginx HTTP Nginx HTTPS OpenSSH   Create a systemd unit file for NGINX:\n/opt/save $ sudo nano /etc/systemd/system/nginx.service Copy the text below into the file. Verify the location of the PIDFile and NGINX binaries, these should match the settings specified in Step 13:\n[Unit] Description=A high performance web server and a reverse proxy server After=network.target [Service] Type=forking PIDFile=/run/nginx.pid ExecStartPre=/usr/sbin/nginx -t -q -g \u0026#39;daemon on; master_process on;\u0026#39; ExecStart=/usr/sbin/nginx -g \u0026#39;daemon on; master_process on;\u0026#39; ExecReload=/usr/sbin/nginx -g \u0026#39;daemon on; master_process on;\u0026#39; -s reload ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid TimeoutStopSec=5 KillMode=mixed [Install] WantedBy=multi-user.target  Save and close the file.\n  Remove all source code files:\n/opt/save $ sudo rm -rf /opt/src_files   Start and enable NGINX\n/opt/save $ sudo systemctl start nginx.service \u0026\u0026 sudo systemctl enable nginx.service   Verify NGINX is running:\n/opt/save $ sudo systemctl status nginx.service ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/etc/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2018-11-29 15:08:48 UTC; 9s ago Main PID: 3118 (nginx) CGroup: /system.slice/nginx.service ├─3118 nginx: master process /usr/sbin/nginx -g daemon on; master_process on └─3119 nginx: worker process Nov 29 15:08:48 ip-10-10-0-225 systemd[1]: Starting A high performance web server and a reverse proxy server\u0026hellip; Nov 29 15:08:48 ip-10-10-0-225 systemd[1]: Started A high performance web server and a reverse proxy server.\n  Reboot your server to check NGINX starts automatically:\n/opt/save $ sudo shutdown -r now   Log in after the reboot is complete, and verify NGINX is running:\n~$ sudo systemctl status nginx.service ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/etc/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2018-11-29 15:10:57 UTC; 1min 53s ago Process: 1150 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 1083 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 1159 (nginx) Tasks: 2 Memory: 4.7M CPU: 31ms CGroup: /system.slice/nginx.service ├─1159 nginx: master process /usr/sbin/nginx -g daemon on; master_process on └─1162 nginx: worker process Nov 29 15:10:56 ip-10-10-0-225 systemd[1]: Starting A high performance web server and a reverse proxy server\u0026hellip; Nov 29 15:10:57 ip-10-10-0-225 systemd[1]: Started A high performance web server and a reverse proxy server.\n  Conclusion Congratulations! You are now running the latest version of NGINX which includes the latest versions of the PCRE, zlib and OpenSSL libraries. Pre-built NGINX packages often include older versions of these libraries, which may contain high-severity bugs. By building NGINX from source, your web server contains the most up-to-date features of these libraries along with any bug fixes and security patches that have been released.\n","href":"/blog/install-nginx-source-code-shell-script/","title":"Create a Shell Script to Install NGINX from Source On Ubuntu"},{"categories":"dotnet","content":"Type casting or type converting is a common occurrance in C#, and as the language has evolved new methods and operators have been introduced that handle type conversion in varying ways.\nOver the years, I have built up my own set of best practices and decided to get them out of my head with the goal of soliciting feedback and hopefully refining them further. Before laying them out, I\u0026rsquo;ll cover the basics of type casting and type conversion in C#.\n  Type Casting vs Type Conversion\n \u0026ldquo;Casting\u0026rdquo; is only valid for reference types, value types are converted The cast operator attempts to cast an object to a specific type, and throws an exception if it fails Although (long)some_integer has the same syntax as a cast operation, the integer is actually being converted to a long    Upcasting vs Downcasting\n Upcasting = Derived Class =\u0026gt; Base Class  Upcasting is implicit (cast operator is not required) Animal animal = dog;\n   Downcasting = Base Class =\u0026gt; Derived Class  Downcasting requires a cast operator (explicit) Dog dog = (Dog)animal;\n      is vs as Operators\n The is operator checks the compatibility of an object with a given type and returns the result as a Boolean (True Or False) The as operator attempts to cast an object to a specific type, and returns null if it fails    Casting/Type Conversion Best Practices If \u0026lsquo;randomObject\u0026rsquo; really \u0026lsquo;should\u0026rsquo; be an instance \u0026lsquo;TargetType\u0026rsquo;, i.e. if it\u0026rsquo;s not, that means there\u0026rsquo;s a bug, then casting is the right solution. Throwing an exception immediately means that no more work is done under incorrect assumptions, and the exception correctly shows the type of bug. (cast only)\n// This will throw an exception if randomObject is non-null // and refers to an object of an incompatible type. //\u0026#34;Cast only\u0026#34; is the best method if that\u0026#39;s the behavior you want. TargetType convertedRandomObject = (TargetType) randomObject; If randomObject might be an instance of TargetType and TargetType is a reference type, then use as-and-null-check:\nTargetType convertedRandomObject = randomObject as TargetType; if (convertedRandomObject != null) { // Do stuff with convertedRandomObject } If randomObject might be an instance TargetType and TargetType is a value type, then we can\u0026rsquo;t use as with TargetType itself, but we can use a nullable type:\nTargetType? convertedRandomObject = randomObject as TargetType?; if (convertedRandomObject != null) { // Do stuff with convertedRandomObject.Value } If you really don\u0026rsquo;t need the converted value, but you just need to know whether randomObject is an instance of TargetType, then use the is operator . In this case it doesn\u0026rsquo;t matter whether TargetType is a reference type or a value type.\nDon\u0026rsquo;t do this:\n// Bad code - checks type twice for no reason if (randomObject is TargetType) { TargetType foo = (TargetType) randomObject; // Do something with foo } is-and-cast (or is-and-as) are both potentially unsafe, as the type of randomObject may change due to another thread between the test and the cast (e.g., randomObject is a field or property rather than a local variable)\nas-and-null-check gives a better separation of concerns. We have one statement which attempts a conversion, and then one statement which uses the result. The is-and-cast or is-and-as performs a test and then another attempt to convert the value.\nHowever, all of the above guidelines are moot with C# 7.0, where pattern matching has largely replaced the as operator. It is now possible to write:\nif (randomObject is TargetType tt) { // Use tt here } This new construct enables cleaner syntax and eliminates the possibility of randomObject\u0026rsquo;s type being changed by another thread and resolves the issue where randomObject\u0026rsquo;s type is needlessly checked twice.\n","href":"/blog/casting-type-conversion-best-practices-c-sharp/","title":"Casting and Type Conversion Best Practices in C#"},{"categories":"AWS","content":"In previous posts, I provided an overview of what the AWS Free Usage Tier contains and some basic guidelines for avoiding overage charges while using the free tier to host a small website.\nIn this post, I will explore the different pricing models that Amazon offers for EC2 and RDS instances and estimate the cost to host four different website configurations for one year. I decided to gather this data in order to answer the following questions:\nWhat amount should I expect to pay to host my website with the same AWS instances when I'm no longer eligible for the free usage tier? Given the light specs of the t2.micro instance (1CPU/1GiB RAM), what would it cost to host a site that requires more capacity? Or a large web application that handles thousands of simultaneous connections? How much more would it cost to host any of these websites with a Windows/MSSQL toolchain versus the Linux/MySQL I'm currently using? How much cheaper is the Reserved Instance Rate (pay for one year of usage upfront) rather than the On-Demand Instance rate?  Thankfully, Amazon created a tool we can use to answer these questions: the AWS Simple Monthly Calculator. Calling it \"simple\" must be an attempt at modesty because the calculator contains a massive amount of configuration options. To begin, let\u0026rsquo;s take a look at the pricing models for EC2 and RDS instances.\nEC2 and RDS Pricing Models Amazon has four different pricing models for EC2 and RDS instances:\n On-Demand rates offer the most flexibility since you can ramp up, scale down and re-locate your resources if your estimates and actual usage differ significantly. However, you\u0026rsquo;ll pay for the convenience. This is the pricing model used when you exceed your monthly limit of EC2 usage included with the free tier. Reserved Instance rates require you to sign a one or three year contract for usage of a single instance type. This means that you cannot change to a more powerful instance type if you underestimated the amount of compute power or memory necessary for your website, or vice-versa (you cannot change to a less expensive instance type). You can pay for the entire year, pay for a partial amount or pay nothing upfront. Naturally, paying the full amount gives the largest discount, but signing a contract and paying nothing upfront is still cheaper than the On-Demand rate for the same instance type. Spot Instances offer much cheaper rates than On-Demand instances, but they are not applicable to our scenario of a public-facing website which requires constant availability. If you need to train your new machine learning algorithm on a massive data set or you need to render a large 3d graphics project, spot instances allow you to access enormous CPU and GPU resources for a huge discount compared to On-Demand rates. Dedicated Hosts/Instances will not be discussed further since their benefits are unnecessary for most websites, especially a personal blog site.  Along with the pricing model and the OS/database system you utilize, the cost of an EC2 instance varies based on the region of the world where the instance is located. All of the cost estimates I gathered are for the US East region.\nDefining Our Data Set My goal is to understand how the cost of hosting a website with AWS changes based on three different criteria:\n OS and database type Pricing model Size/capacity of the website  To do so, I decided to calculate estimated costs for four OS/database variations and 3 pricing models:\n  OS/Database Combinations \u0026nbsp; Pricing Models     Linux/MySQL \u0026nbsp; On-Demand   Windows/MySQL \u0026nbsp; 1 Year Reserved Instance (Partial Payment Upfront)   Windows/MS SQL Bring Your Own License (BYOL) \u0026nbsp; 1 Year Reserved Instance (Full Payment Upfront)   Windows/MS SQL Standard Edition (SE) \u0026nbsp; \u0026nbsp;      In order to show how the cost savings change as the website size/capacity is increased, I created four different configurations of EC2/RDS instances, SSD storage, monthly data transfer amounts, etc. I\u0026rsquo;ve summarized the different website configurations in Table 1:\n Table 1   Configuration Details for Example Websites   \u0026nbsp; Linux/MySQL Windows/MySQL Windows/MS SQL BYOL Windows/MS SQL SE     Free Usage Website \u0026#40;1\u0026#41; Linux EC2 t2.micro instance \u0026#40;1\u0026#41; 10GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.t2.micro MySQL instance   \u0026#40;1\u0026#41; Windows EC2 t2.micro instance \u0026#40;1\u0026#41; 10GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.t2.micro MySQL instance   \u0026#40;1\u0026#41; Windows EC2 t2.micro instance \u0026#40;1\u0026#41; 10GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.t2.small MS SQL BYOL instance   \u0026#40;1\u0026#41; Windows EC2 t2.micro instance \u0026#40;1\u0026#41; 10GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.m1.small MS SQL Standard instance     Small Website \u0026#40;1\u0026#41; Linux EC2 t2.medium instance \u0026#40;1\u0026#41; 100GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.m1.small MySQL instance   \u0026#40;1\u0026#41; Windows EC2 t2.medium instance \u0026#40;1\u0026#41; 100GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.m1.small MySQL instance   \u0026#40;1\u0026#41; Windows EC2 t2.medium instance \u0026#40;1\u0026#41; 100GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.m1.small MS SQL BYOL instance   \u0026#40;1\u0026#41; Windows EC2 t2.medium instance \u0026#40;1\u0026#41; 100GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.m1.small MS SQL Standard instance     Mid-Size Website \u0026#40;2\u0026#41; Linux EC2 m1.small instances \u0026#40;1\u0026#41; 300GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.m1.small MySQL instance   \u0026#40;2\u0026#41; Windows EC2 m1.small instances \u0026#40;1\u0026#41; 300GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.m1.small MySQL instance   \u0026#40;2\u0026#41; Windows EC2 m1.small instances \u0026#40;1\u0026#41; 300GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.m1.small MS SQL BYOL instance   \u0026#40;2\u0026#41; Windows EC2 m1.small instances \u0026#40;1\u0026#41; 300GB SSD EBS volume \u0026#40;1\u0026#41; 20GB RDS db.m1.small MS SQL Standard instance     Large Web Site \u0026#40;4\u0026#41; Linux EC2 m1.small instances \u0026#40;4\u0026#41; 300GB SSD EBS volumes \u0026#40;1\u0026#41; 20GB RDS db.m1.large MySQL instance   \u0026#40;4\u0026#41; Windows EC2 m1.small instances \u0026#40;4\u0026#41; 300GB SSD EBS volumes \u0026#40;1\u0026#41; 20GB RDS db.m1.large MySQL instance   \u0026#40;4\u0026#41; Windows EC2 m1.small instances \u0026#40;4\u0026#41; 300GB SSD EBS volumes \u0026#40;1\u0026#41; 20GB RDS db.m1.large MS SQL BYOL instance   \u0026#40;4\u0026#41; Windows EC2 m1.small instances \u0026#40;4\u0026#41; 300GB SSD EBS volumes \u0026#40;1\u0026#41; 20GB RDS db.m1.large MS SQL Standard instance        Cost Estimates Each cell in Table 2 contains a link to a screenshot showing configuration settings from the AWS Simple Cost Calculator. If you enter the same settings shown in the EC2 and RDS links, as well as the settings from the Common column, it should match the estimated monthly bill shown in the Total Cost link. The settings found in the Common column do not change based on the pricing model:\n Table 2   AWS Simple Cost Calculator Settings1   \u0026nbsp; \u0026nbsp; Linux/MySQL Win/MySQL Win/MS SQL BYOL Win/MS SQL SE Common     Free Usage Website On-Demand Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS S3 Route 53   \u0026nbsp;   1 Year Upfront (Partial) Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS   1 Year Upfront (Full) Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS   Small Website On-Demand Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS S3 Route 53   \u0026nbsp;   1 Year Upfront (Partial) Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS   1 Year Upfront (Full) Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS   Mid-Size Website On-Demand Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS S3 Route 53   Cloudfront DynamoDB     1 Year Upfront (Partial) Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS   1 Year Upfront (Full) Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS   Large Website On-Demand Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS S3 Route 53   Cloudfront DynamoDB     1 Year Upfront (Partial) Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS   1 Year Upfront (Full) Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS Total Cost EC2 RDS     1All cost estimates are for EC2 and RDS Instances in the US East (Virginia) region, and all RDS Instances are configured for Multi-Availability Zone deployment      The screenshot in each Total Cost link contains both a monthly cost and a one-time upfront cost. From these, you can calculate the hosting cost for one year. This is the value shown in blue text in Table 3 and will be the basis for all cost comparisons:\n Table 3   Cost Estimates for One Year of AWS Web Hosting1   \u0026nbsp; Linux/MySQL Win/MySQL Win/MS SQL BYOL Win/MS SQL SE   \u0026nbsp; Upfront 1 Month 1 Year Upfront 1 Month 1 Year Upfront 1 Month 1 Year Upfront 1 Month 1 Year     Free Usage Website On-Demand $0 $40 $480 $0 $44 $528 $0 $83 $996 $0 $956 $11,472   1 Year Upfront (Partial) $132 $18 $348 $152 $20 $392 $330 $34 $738 $2,890 $375 $7,390   1 Year Upfront (Full) $262 $7 $346 $302 $7 $386 $649 $7 $733 $7,158 $7 $7,242   Small Website On-Demand $0 $133 $1,586 $0 $146 $1,752 $0 $197 $2,364 $0 $1,002 $12,024   1 Year Upfront (Partial) $366 $51 $978 $445 $58 $1,141 $529 $89 $1,597 $3,039 $398 $7,815   1 Year Upfront (Full) $751 $18 $967 $909 $18 $1,125 $1,352 $18 $1,568 $7,452 $18 $7,668   Mid-Size Website On-Demand $0 $272 $3,264 $0 $317 $3,804 $0 $368 $4,416 $0 $1,173 $14,076   1 Year Upfront (Partial) $492 $165 $2,472 $576 $194 $2,904 $660 $225 $3,360 $3,170 $534 $9,578   1 Year Upfront (Full) $928 $127 $2,452 $1,354 $127 $2,878 $1,797 $127 $3,321 $7,897 $127 $9,421   Large Website On-Demand $0 $927 $11,124 $0 $1,017 $12,204 $0 $1,200 $14,400 $0 $2,093 $25,116   1 Year Upfront (Partial) $1,478 $595 $8,618 $1,646 $654 $9,494 $1,980 $770 $11,220 $4,638 $1,089 $17,706   1 Year Upfront (Full) $3,027 $461 $8,559 $3,879 $461 $9,411 $5,580 $461 $11,112 $11,928 $461 $17,460     11 Year = Upfront + (1 Month * 12)      With this set of cost estimates, let\u0026rsquo;s take a look at the questions from the beginning of this post:\nWhat amount should I expect to pay to host this site with the same AWS instances when I'm no longer eligible for the free usage tier? Since this site is running on a LEMP stack, the hosting costs without the benefits of the free usage tier would be:\n  Pricing Model Upfront Cost Monthly Cost One Year Total Monthly Avg.    On-Demand $0 $40 $480 $40.00  1 Year Upfront (Partial) $132 $18 $348 $29.00  1 Year Upfront (Full) $262 $7 $346 $28.83       Given the light specs of the t2.micro instance (1CPU/1GiB RAM), what would it cost to host a site that requires more capacity? Or a large web application that handles thousands of simultaneous connections? It would cost approximately $1600/year to host the example \"Small Website\" using Linux/MySQL instances with On-Demand pricing, but less than $1000/year with a 1 year contract and $366 paid upfront. This would result in a monthly hosting bill of $51, which sounds very reasonable to me.\n How much more would it cost to host any of these websites with a Windows/MSSQL toolchain versus the Linux/MySQL I'm currently using? How much cheaper is it to pay partially or for an entire year of service upfront rather than the On-Demand Instance rate? The general trends from Table 3 are obvious: Committing to a year of service reduces your costs substantially, while choosing Windows or MS SQL increases costs. We want to dive further into these trends, however. Let's start by examining the savings from using Linux/MySQL instances compared to the three Windows flavors.\n Cost Savings from Linux/MySQL It\u0026rsquo;s simple to see from Table 3 that hosting your website with a Linux/MySQL server is the best way to go in terms of cost. This is entirely expected since Linux and MySQL are free, open-source software while Windows and MS SQL are paid, licensed software.\nBefore capturing the data, I underestimated how much a MS SQL license would cost and overestimated the cost of a Windows license. I was surprised by the small increase in price between Linux/MySQL and Windows/MySQL, and similarly surprised by the large increase between Windows/MS SQL BYOL and Windows/MS SQL SE. I had assumed that bringing your own license (Windows/MS SQL BYOL) would end up with a similar price to using MySQL (Windows/MySQL).\nI was expecting the Windows/MS SQL Standard Edition (SE) hosting costs to be the most expensive, but approximately $11,500 for 1 year at On-Demand rates ($615/month when including the partial upfront payment at the Reserved Instance rate) was far more than I would have guessed. Keep in mind, this is for the t2.micro instance types which are included with the Free Usage Tier.\nI wanted to visualize the money saved by choosing Linux/MySQL comparted to the Windows options. I started by noting the money saved for every Windows estimate in Table 3 as compared to the Linux/MySQL estimate for the same website size and pricing model, and then calculated the savings as a percentage of the yearly cost of the Windows plan. These calculations ($ Saved and % Saved, in blue and green text, respectively) are shown in Table 4:\n Table 4   1 Year Cost Savings with Linux/MySQL Toolchain   \u0026nbsp; Free Tier Website Small Website Mid-Size Website Large Website   \u0026nbsp; 1 Year $ Saved % Saved 1 Year $ Saved % Saved 1 Year $ Saved % Saved 1 Year $ Saved % Saved     On-Demand Linux/MySQL vs $480 \u0026nbsp; \u0026nbsp; $1,586 \u0026nbsp; \u0026nbsp; $3,264 \u0026nbsp; \u0026nbsp; $11,124 \u0026nbsp; \u0026nbsp;   Win/MySQL $528 $48 9% $1,752 $166 10% $3,804 $540 14% $12,204 $1,080 9%   Win/MS SQL BYOL $996 $516 52% $2,364 $778 33% $4,416 $1,152 26% $14,400 $3,276 23%   Win/MS SQL SE $11,472 $10,992 96% $12,024 $10,438 87% $14,076 $10,812 77% $25,116 $13,992 56%   1 Year (Partial Upfront) Linux/MySQL vs $348 \u0026nbsp; \u0026nbsp; $978 \u0026nbsp; \u0026nbsp; $2,472 \u0026nbsp; \u0026nbsp; $8,618 \u0026nbsp; \u0026nbsp;   Win/MySQL $392 $44 11% $1,141 $163 14% $2,904 $432 15% $9,494 $876 9%   Win/MS SQL BYOL $738 $390 53% $1,597 $619 39% $3,360 $888 26% $11,220 $2,602 23%   Win/MS SQL SE $7,390 $7,042 95% $7,815 $6,837 88% $9,578 $7,106 74% $17,706 $9,088 51%   1 Year (All Upfront) Linux/MySQL vs $346 \u0026nbsp; \u0026nbsp; $967 \u0026nbsp; \u0026nbsp; $2,562 \u0026nbsp; \u0026nbsp; $8,559 \u0026nbsp; \u0026nbsp;   Win/MySQL $386 $40 10% $1,125 $158 14% $2,878 $426 15% $9,411 $852 9%   Win/MS SQL BYOL $733 $387 53% $1,568 $601 38% $3,321 $869 26% $11,112 $2,553 23%   Win/MS SQL SE $7,242 $6,896 95% $7,668 $6,701 87% $9,421 $6,969 74% $17,460 $8,901 51%      Table 4 shows that choosing Linux and MySQL to host my website is $516 cheaper each year than Windows/MS SQL BYOL. The savings are nearly $11,000 if I do not have a license for MS SQL Standard Edition.\nNext, I wanted to know if these savings changed as website size increased. To visualize this, for each website configuration, I took the average of the % Saved for all pricing models and plotted the data for each OS/database configuration. See Chart 1:\n Figure 1 - Web Host Cost Savings with Linux/MySQL\n  It turns out that the relative cost savings decrease as the size of your website increases. For the instance types included with the free tier, using Linux/MySQL is 95% cheaper than using MS SQL SE, but for the large website, the Linux/MySQL is only 53% cheaper.\nThis makes sense to me, since the large website is using EC2 instances with much greater CPU and memory capacity, as well as larger SSD volumes. These resources would represent a much larger portion of the monthly costs than the EC2 instances included with the free tier, and thus the licensing costs would be a smaller portion.\nIt is also interesting that the cost savings compared to Windows/MySQL do not behave in the same way as the pricing for MS SQL. I believe this shows that Amazon charges (roughly) a flat 10-15% surcharge for Windows, regardless of the instance type.\nCost Savings from Windows/MySQL After realizing I had underestimated how much MS SQL would increase hosting costs, I decided to calculate the cost savings for Windows/MySQL compared to the MS SQL options in the same way I had done for Linux/MySQL:\n Table 5   1 Year Cost Savings with Windows/MySQL Toolchain   \u0026nbsp; Free Tier Website Small Website Mid-Size Website Large Website   \u0026nbsp; 1 Year $ Saved % Saved 1 Year $ Saved % Saved 1 Year $ Saved % Saved 1 Year $ Saved % Saved     On-Demand Win/MySQL vs $528 \u0026nbsp; \u0026nbsp; $1,752 \u0026nbsp; \u0026nbsp; $3,804 \u0026nbsp; \u0026nbsp; $12,204 \u0026nbsp; \u0026nbsp;   Win/MS SQL BYOL $996 $468 47% $2,364 $612 26% $4,416 $612 14% $14,400 $2,196 15%   Win/MS SQL SE $11,472 $10,944 95% $12,024 $10,272 85% $14,076 $10,272 73% $25,116 $12,912 51%   1 Year (Partial Upfront) Win/MySQL vs $392 \u0026nbsp; \u0026nbsp; $1,141 \u0026nbsp; \u0026nbsp; $2,904 \u0026nbsp; \u0026nbsp; $9,494 \u0026nbsp; \u0026nbsp;   Win/MS SQL BYOL $738 $346 47% $1,597 $456 29% $3,360 $456 14% $11,220 $1,726 15%   Win/MS SQL SE $7,390 $6,998 95% $7,815 $6,674 85% $9,578 $6,674 70% $17,706 $8,212 46%   1 Year (All Upfront) Win/MySQL vs $386 \u0026nbsp; \u0026nbsp; $1,125 \u0026nbsp; \u0026nbsp; $2,878 \u0026nbsp; \u0026nbsp; $9,411 \u0026nbsp; \u0026nbsp;   Win/MS SQL BYOL $733 $347 47% $1,568 $443 28% $3,321 $443 13% $11,112 $1,701 15%   Win/MS SQL SE $7,242 $6,856 95% $7,668 $6,543 85% $9,421 $6,543 69% $17,460 $8,049 46%       Figure 2 - Web Host Cost Savings with Windows/MySQL\n  The trend for these savings looks exactly like the trend for Linux/MySQL, only shifted down by 8-10%, which makes sense since this shift would represent the flat \u0026ldquo;surcharge\u0026rdquo; I noted at the end of the previous section.\nCost Savings from Reserved Instance Rates There is only one question remaining from my original list: How much cheaper is it to pay partially or for an entire year of service upfront rather than the On-Demand Instance rate? Looking at Table 3, I immediately noticed there is almost no reason to pay the full amount upfront. Across the board, the estimated cost is only $5-10 cheaper than paying the partial amount.\nYou can see the % Savings for each pricing model and website configuration in Table 6:\n Table 6   Cost Savings When Paying Reserved Instance Rates   \u0026nbsp; Linux/MySQL Win/MySQL Win/MS SQL BYOL Win/MS SQL SE   \u0026nbsp; 1 Year $ Saved % Saved 1 Year $ Saved % Saved 1 Year $ Saved % Saved 1 Year $ Saved % Saved     Free Usage Website On-Demand $480 \u0026nbsp; \u0026nbsp; $528 \u0026nbsp; \u0026nbsp; $996 \u0026nbsp; \u0026nbsp; $11,472 \u0026nbsp; \u0026nbsp;   1 Year Upfront (Partial) $348 $132 28% $392 $136 26% $738 $258 26% $7,390 $4,082 36%   1 Year Upfront (Full) $346 $134 28% $386 $142 27% $733 $263 26% $7,242 $4,230 37%   Small Website On-Demand $1,586 \u0026nbsp; \u0026nbsp; $1,752 \u0026nbsp; \u0026nbsp; $2,364 \u0026nbsp; \u0026nbsp; $12,024 \u0026nbsp; \u0026nbsp;   1 Year Upfront (Partial) $978 $608 38% $1,141 $611 35% $1,597 $767 32% $7,815 $4,209 35%   1 Year Upfront (Full) $967 $619 39% $1,125 $627 36% $1,568 $796 34% $7,668 $4,356 36%   Mid-Size Website On-Demand $3,264 \u0026nbsp; \u0026nbsp; $3,804 \u0026nbsp; \u0026nbsp; $4,416 \u0026nbsp; \u0026nbsp; $14,076 \u0026nbsp; \u0026nbsp;   1 Year Upfront (Partial) $2,472 $792 24% $2,904 $900 24% $3,360 $1,056 24% $9,578 $4,498 32%   1 Year Upfront (Full) $2,452 $6812 25% $2,878 $926 24% $3,321 $1,095 25% $9,421 $4,655 33%   Large Website On-Demand $11,124 \u0026nbsp; \u0026nbsp; $12,204 \u0026nbsp; \u0026nbsp; $14,400 \u0026nbsp; \u0026nbsp; $25,116 \u0026nbsp; \u0026nbsp;   1 Year Upfront (Partial) $8,618 $2,506 23% $9,494 $2,710 23% $11,220 $3,180 22% $17,706 $7,410 30%   1 Year Upfront (Full) $8,559 $2,565 23% $9,411 $2,793 23% $11,112 $3,288 23% $17,460 $7,656 31%     11 Year = Upfront + (1 Month * 12)      For some reason, Amazon does not give a larger incentive for paying the entire amount upfront. This is the case even when full upfront cost is thousands of dollars greater than the partial amount.\nI again created a graph to show this by taking the average of the % Savings for all website configurations, for each OS/database combination:\n Figure 3 - Cost Savings with Reserved Instance Pricing\n  Summary I hope this post helps you understand how the cost of hosting a website with AWS is impacted by the choice of OS, database type and pricing model. I know that my example websites may not be applicable to your application, causing a significant difference in the hosting costs.\nIf you have any feedback for how to improve these estimates, please let me know in the comments!\n","href":"/blog/aws-hosting-cost-examples-linux-windows-ec2-rds/","title":"Cost Estimates for Hosting Small, Medium and Large Websites with AWS"},{"categories":"AWS","content":"With Amazon's Free Usage Tier, it's very easy to generate overage charges even with the best intentions of operating entirely within the offer\u0026rsquo;s limits (this is why Amazon requires a credit card when you create your account). For each individual service (EC2, RDS, S2, etc.), your usage for the month is tabulated in extremely fine detail. At first, the sheer number of items on your invoice can be intimidating. In this post, I will show what a typical bill contains for running a small Wordpress site and how to avoid overage charges and other unnecesary costs.\nMonitor Running Instances in the EC2 Dashboard The easiest way to ensure you won\u0026rsquo;t generate overrage costs with Amazon's Free Usage Tier is to keep track of how many Linux and Windows EC2 instances are running under your account.\nThe free tier includes 750 hours per month of Linux EC2 usage and another 750 hours per month of Windows EC2 usage (both must be either t1 or t2.micro instances). Similarly, you are allowed 750 hrs/month for an Elastic Load Balancer and 750 hrs/month for an RDS (database) instance. Together, these components will allow you to create a web application and run it continuously, free of charge.\nYou can see how many EC2 instances are currently running via the EC2 Dashboard within the AWS Console. Navigate to: Services -\u0026gt; Compute -\u0026gt; EC2:\n Figure 1 - Services menu showing link to EC2 Dashboard in AWS Console\n  The dashboard provides a summary of various EC2 resources which are allocated to your account, including the number of running instances. In the screenshot below, you can see that I have one running server instance, one elastic IP, one volume (storage disk), one SSH key pair and three network security groups which are allocated to my account.\n Figure 2 - EC2 Dashboard showing one running instance\n  If you are continuously running two server instances (must be one Linux instance and one Windows instance), you will not be charged while your account is eligible for the free tier. Also, if you are continuously running only a single instance (either Linux or Windows) you will not be charged.\nHowever, if you are continuously using two instances of Linux or two instances of Windows, you will eventually incur charges at the rate for your specific region, instance type and OS type when you have exceeded 750 total hours of use. Two Linux (or Windows) servers running 24 hours a day would exceed this limit after 15 days.\nThankfully, the instance types that are availabe at no cost (t1 and t2.micro) are the cheapest to operate (but also the lowest in terms of CPU/RAM). The table below shows how much you would be charged for running these instances after exceeding the free tier limit:\n  Table 1   Overage Pricing for EC2 Instance Types Included with Free Tier   Instance Type Hourly Rate1 Daily Rate2 Monthly Rate3    Linux t1.micro $0.0250 $0.60 $18.25  Linux t2.micro $0.0138 $0.33  $10.07  Windows t1.micro $0.0350 $0.84 $25.55  Windows t2.micro $0.0184 $0.44 $13.43     Note: Hourly rates are for On-Demand EC2 instances in US West (N. California) Region   1Source: https://aws.amazon.com/ec2/pricing/on-demand/   2Daily Rate = Hourly Rate * 24   3Monthly Rate = (Daily Rate * 365) / 12      Monitor Current Charges in the Billing Dashboard As mentioned in my previous post, the amount of AWS products that offer a free version is staggering. To prevent unexpected charges from your month-long spree of creating VR environments and pseudo-sentient chatbots, you should make a habit of frequently checking your current bill in the AWS Console.\nYou can open the Billing \u0026amp; Cost Management Dashboard by clicking the Billing link under the Services menu. This page contains various tables and graphs that show how your usage is tracking agianst the limits of the free tier.\nTop Free Tier Services by Usage is a list of the services you have used the most this month and a projection of how much of the free allotment you are on track to consume:\n Figure 3 - Top Free Tier Services by Usage table in AWS Billing Dashboard\n  Unfortunately, I am on track to use more than the 750 hours of EC2 Linux usage that is included with the free tier. By my calculations, I will be responsible for roughly 250 excess hours of t2.micro On-Demand usage, which comes out to a whopping $3.45. Thankfully, I am on track to stay within the free tier limits for everything else since I am projected to use 82.6% percent of the alloted RDS - Storage free tier usage, and less than that for all other services.\nMonth-to-Date Spend by Service contains a summary of all charges, including services which are not part of the free tier. The $0.51 charges for Route 53 are related to DNS record sets and lookups for this domain, which was registered through Route 53.\n Figure 4 - Month-to-Date Spend by Service in AWS Billing Dashboard, includes products which are not part of the free tier\n  Clicking the Bill Details button generates a complete report of your usage for all AWS products in all regions. This is the best place to look if you want to know how each service is metered and how much your usage will cost.\nFor instance, here is how I managed to incur $0.91 in charges even though I intended to stay within the limits of the free tier:\n Figure 5 - Bill Details in AWS Billing Dashboard\n  I\u0026rsquo;ve highlighted three areas which encompass the total charges. The Route 53 charges of $0.51 can be ignored since this service is not part of the free tier, $0.50 is the monthly charge for managing the record sets for this domain amd $0.01 is for handling DNS queries for the same.\nThe Elastic IP charges of $0.36 demonstrate a usage scenario that is easy to avoid. The description states that these charges are for \u0026quot;Elastic IP address not attached to a running instance\u0026quot;. I must have stopped or terminated an EC2 instance that had a public IP address attached to it without releasing the IP address from my account.\nThe last charge of $0.04 appears to be related to data transfer between my EC2 instances. I'm not sure how this could have been avoided and will investigate further. Here is the detail for this charge:\n Figure 6 - Data transfer charges\n  Release Unused Elastic IPs to Avoid Free Tier Charges As mentioned earlier, you can easily avoid being charged for unused IP addresses. The number of running instances and the number of elastic IP addresses allocated to your account are listed in the EC2 Dashboard:\n Figure 7 - EC2 Dashboard showing one Elastic IP allocated to my account\n  If you have any IP addresses which are not attached to a running EC2 instance, you will be charged as shown in Figure 5. By clicking 1 Elastic IPs (or, Elastic IPs under Network \u0026amp; Security in the left-hand menu), you can manage the IP addresses allocated to your account:\n Figure 8 - An unused Elastic IP address (no value in the Instance column)\n  If you have an unassociated IP address as shown in Figure 8, simply click Actions -\u0026gt; Release addresses to remove it from your account:\n Figure 9 - Click Actions -\u0026gt; Release addresses to release the IP address\n  You will be prompted to confirm that you really do want to remove the IP from your account:\n Figure 10 - Prompt confirms your decision to release the IP address\n  Summary There are a few easy guidelines you can follow in order to avoid unnecessary charges when using the AWS free tier:\n  Monitor the number of and type of running EC2 instances\n You can run continuously 1 Linux instance and 1 Windows instance (but NOT 2 Linux or 2 Windows) without incurring any charges    Only launch t1.micro or t2.micro EC2 instances\n All other instance types are NOT included with the Free Tier and can generate substantial charges if left running.    Only create a single db.t2.micro RDS instance\n The free tier includes 750 hrs/month of RDS, same as the EC2 Linux and Windows limit    Release any Elastic IP addresses that are not attached to a running instance\n You will incur an hourly charge for any IP address that is not attached to a running EC2 instance.    Delete any snapshots or AMIs that are absolutely not vital\n This was not covered in this article, but charges will incur for having these under your account. These resources can be monitored in the EC2 Dashboard.    ","href":"/blog/how-to-monitor-aws-free-tier-usage/","title":"AWS Free Tier: How to Monitor Usage and Avoid Overage Charges"},{"categories":"AWS","content":"For the past 2 months, I have been learning Linux server administration, PHP/JavaScript and other skills such as DevOps tools/processes. By far, the most valuable resource I have used to develop these skills is the Amazon Web Services (AWS) Free Usage Tier that is offered to anyone for a period of one year. Well, anyone with a credit card to be more precise.\n Figure 1: List of AWS product categories, vast and varied\n  AWS comprises an overwhelming number of products and services. Take a look at the list here. Besides the expected categories like \u0026ldquo;compute\u0026rdquo;, \u0026ldquo;database\u0026rdquo;, or \u0026ldquo;developer tools\u0026rdquo;, the free tier also provides access to many services that I was unaware of:\n Voice and Chat Textbot Builders Deep learning-based facial image recognition Virtual Reality (VR) and Augmented Reality (AR) development platform A variety of AI products such as natural language processing, language translation, text-to-speech and vice versa  However, \u0026quot;free\u0026quot; in this sense does not mean unlimited. Each service has a defined limit which when exceeded will begin billing at the service's normal rate (the limit for most services resets each month). The overage is invoiced monthly and charged to the credit card you provided at signup.\nMany of the products remain \u0026quot;free\u0026quot; after the 1 year term is over. The most useful of which (to me) is the CodeCommit service. I built and deployed a Gogs EC2 instance which I planned to use as a private code repository for my personal projects. I quickly took it down when I learned that AWS offers the service completely free-to-use, forever, for up to 5 user accounts. There are monthly limits on the size of your repositories and frequency of git requests, but these will never be reached by my usage.\nAmazon provides this free tier to allow users to gain experience with the AWS platform, with the obvious goal of encouraging adoption of their products in large-scale applications and organizations the user belongs to. On each AWS product page, Amazon boasts of such customers: the WaPo utilizes the Comprehend service, NASA has leveraged Lex and Zillow uses Lambda.\nJust as impresively, I used multiple Amazon services to host this website. I registered my domain and configured the associated record sets through Route 53 (This product does not have a free offering). I installed the Wordpress core on a t2.micro EC2 instance and deployed the Wordpress database on an RDS instance. I store site assets such as image files in a S3 bucket. My RDS instance can only communicate with the Wordpress EC2 server and is isolated from any outside web traffic thanks to custom subnet configurationsmade in my Virtual Private Cloud (VPC).\nIn my next post, I will give recommendations on how to manage billing for your AWS account and how to avoid going over the free tier limits when hosting a website.\n","href":"/blog/aws-free-usage-tier/","title":"AWS Free Usage Tier: An Incredibly Useful and Generous Offering"},{"categories":"Flask ; Python ; Tutorial-Series","content":"This tutorial series provides step-by-step instructions and in-depth explanations to guide you through the process of creating a robust, production-quality REST API. The toolstack consists of Flask, Flask-RESTx, SQLAlchemy, pyjwt, tox and other packages. Code quality is a major focus, with considerable time dedicated to testing (using pytest), logging and tools such as coverage, flake8 and mypy. The tutorial concludes by creating a process that continuously integrates (with tox, travis/circle CI, coveralls) and deploys the API (with either Github or Azure DevOps to Heroku).\n   Part 1: Project Setup and Environment Configuration    In Part 1, the core concepts of REST and JWTs are introduced, project dependencies are described and installed, and the project is fully configured for prod/dev environments. The flask server and CLI are demonstrated to ensure the setup was performed correctly before moving on to Part 2.\n Code Diff  .zip .tar.gz       Part 2: Database Models, Migrations and JWT Setup    Part 2 covers the basics of SQLAlchemy, the Flask-SQLAlchemy extension, and the Flask-Migrate extension. The process of creating a new database table to store user information by defining a class and \u0026ldquo;registering\u0026rdquo; it with SQLAlchemy is demonstrated. Next, setting up a system that manages changes to a database schema is thoroughly explained and demonstrated. After initializing the database, functions to encode and decode JSON Web Tokens (JWTs) are created. This section concludes with an introduction to pytest fixtures and examples of several test cases that verify the ability to encode/decode JWTs issued by our API.\n Code Diff  .zip .tar.gz       Part 3: API Configuration and User Registration    Part 3 explains how to initialize the Flask-RESTx extension and how API routes/endpoints are defined. In order to create an endpoint for new user registration, the modules and classes available in Flask-RESTx for parsing and validating request data are explored and demonstrated. Additionally, the process for serializing Python objects in order to send them in an HTTP response is covered. After implementing the user registration API endpoint, test cases are created and executed to verify the registration process is working correctly.\n Code Diff  .zip .tar.gz       Part 4: JWT Authentication, Decorators and Blacklisting Tokens    Part 4 completes the user authorization API by implementing login, logout and user verification API endpoints. The process to create a custom decorator that only allows access to users with a valid JWT is covered in-depth. How to send an HTTP request for a protected resource that includes a JWT is demonstrated with both Swagger UI and command-line tools. A new class/database model is introduced to create a token blacklist, to ensure that JWTs cannot be used after the user has logged out. Test cases are created and executed for all API endpoints covering successful and failed attempts to login/logout/retrieve user info.\n Code Diff  .zip .tar.gz       Part 5: RESTful Resources and Advanced Request Parsing    Part 5 is dedicated to implementing a RESTful resource (a collection of \u0026lsquo;widgets\u0026rsquo;) with API endpoints that allow users to perform CRUD actions. In order to allow users to create a new widget, several advanced techniques for parsing request data are covered. Specifically, two custom \u0026lsquo;types\u0026rsquo; are created since the built-in types provided by Flask-RESTx cannot fulfill our requirements. Finally, the custom request parser is integrated with the collection endpoint for the Widget API.\n Code Diff  .zip .tar.gz       Part 6: Pagination, HATEOAS and Parameterized Testing    Part 6 completes the implementation of the Widget API. Since one of the requirements is to allow users to retrieve a paginated list of widgets, advanced techniques for serializing objects to JSON are demonstrated. Both the widget API model and the pagination API model are complex, requiring the use of several new classes from the Flask-RESTx fields module. The update and delete processes are implemented next and unit tests for all endpoint/request type combination are created and executed.\n Code Diff  .zip .tar.gz     ","href":"/series/flask-api-tutorial/overview/","title":"How To: Create a Flask API with JWT-Based Authentication (Overview)"},{"categories":"Flask ; Python ; Tutorial-Series","content":"Introduction My goal for this tutorial is to provide a detailed guide to designing and creating a Flask API that uses JSON Web Tokens (JWT) to authenticate HTTP requests. There are many different Flask extensions and Python packages that can be used to create a web service that satisfies these requirements. The toolchain that this product utilizes includes Flask-RESTx, SQLAlchemy, PyJWT, pytest and tox (this is simply my personal preference).\nThis is NOT a full-stack tutorial, creating a front-end that consumes the API is not covered. However, Flask-RESTx will automatically generate a Swagger UI webpage that allows anyone to send requests and inspect responses from the API.\nIn addition to the user management and authentication functions, the API will contain a RESTful resource that registered users can manipulate with CRUD actions — a list of \u0026ldquo;widgets\u0026rdquo;. Why did I decide to create widgets and not to-do items, or something real? Using a generic resource reinforces the idea that this code is boilerplate and could be easily adapted for use in a real-world API.\nPerforming CRUD actions and restricting access based on a user\u0026rsquo;s assigned role/permissions are extremely common requirements, and the code to do so is the same for a widget, blog post or anything else that you expose to clients via HTTP.\nThe feature specification for the API is given below. I hope that the various methodologies and \u0026ldquo;best practices\u0026rdquo; that I present are well-founded and justified by the arguments I present for them. Any and all comments/criticism are appreciated, please feel free to log issues in the github repository for suggested improvements and/or any bugs that I missed.\nAt the end of each section, any requirements that have been completely implemented will be marked as complete ():\nUser Management/JWT Authentication\nNew users can register by providing an email address and password\nExisting users can obtain a JWT by providing their email address and password\nJWT contains the following claims: time the token was issued, time the token expires, a value that identifies the user, and a flag that indicates if the user has administrator access\nJWT is sent in access_token field of HTTP response after successful authentication with email/password\nJWTs must expire after 1 hour (in production)\nJWT is sent by client in Authorization field of request header\nRequests must be rejected if JWT has been modified\nRequests must be rejected if JWT is expired\nIf user logs out, their JWT is immediately invalid/expired\nIf JWT is expired, user must re-authenticate with email/password to obtain a new JWT\n API Resource: Widget List\nAll users can retrieve a list of all widgets\nAll users can retrieve individual widgets by name\nUsers with administrator access can add new widgets to the database\nUsers with administrator access can edit existing widgets\nUsers with administrator access can delete widgets from the database\nThe widget model contains attributes with URL, datetime, timedelta and bool data types, along with normal text fields.\nURL and datetime values must be validated before a new widget is added to the database (and when an existing widget is updated).\nThe widget model contains a \"name\" attribute which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nThe widget model contains a \"deadline\" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.\nWidget name must be validated before a new widget is added to the database (and when an existing widget is updated).\nIf input validation fails either when adding a new widget or editing an existing widget, the API response must include error messages indicating the name(s) of the fields that failed validation.\n  Core Concepts It is important to understand the history and actual meaning of the term REST, as well as the structure and purpose of JSON Web Tokens. Let\u0026rsquo;s review these topics before we begin working on the application.\nStatelessness I have made the conscious decision NOT to refer to this series as a REST API tutorial. Seemingly every API and every how-to article on API design written in the last few years proclaims itself RESTful. This trend is a disservice to the depth and complexity that Roy Fielding laid out in his doctoral thesis introducing and defining REST. I will go into further detail on this subject in Part 3 when we begin configuring the API.\nHowever, I think it is important to point out where I am attempting to adhere to the requirements/constraints of REST. One of these constaints is statelessness. Statelessness is an essential characteristic of a RESTful system, but it can be a confusing concept at first.\nObviously both the client and server in any hypothetical system keep state; they just keep different types of state. For example, a web browser keeps track of each web page visited as well as the current page; in a RESTful system, this is called application state. If the website is a banking application, the server hosting the website keeps track of which bank accounts have been accessed or modified; this is called resource state. \u0026ldquo;Statelessness\u0026rdquo; is meant to convey that the server doesn\u0026rsquo;t care about the client\u0026rsquo;s application state, and therefore no data about the client\u0026rsquo;s application state should be stored by the server.\nThis has obvious implications for authentication scenarios since in a RESTful system the server does not store any information about which users are currently logged in. Therefore, in order to access a protected resource a client must must include authentication information with every request. In order to avoid including the client\u0026rsquo;s password with every request, a common practice is for the server to generate an access token when user credentials have been verified. Now, when the client sends a request for access to a protected resource, the token is included in the request header and verified by the server. The most common format for authorization tokens is the JSON Web Token, which we will take a look at in the next section.\nJSON Web Tokens JSON Web Token (JWT) is an open IETF standard that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. JWTs are made up of three parts: header, payload and signature. These are converted to a URL-safe base64-encoded string and concatenated together. Each part is separated by \u0026ldquo;.\u0026rdquo; (the full-stop or period character).\nThe header will identify the object as a JWT and also identify the algorithm used to generate the signature (e.g., {\u0026quot;typ\u0026quot;: \u0026quot;JWT\u0026quot;, \u0026quot;alg\u0026quot;: \u0026quot;HS256\u0026quot;}). The conversion to URL-safe base64-encoded string is shown below:\nASCII: {\"t yp\" :\"J WT\" ,\"a lg\" :\"H S25 6\"} URLSAFE-B64: eyJ0 eXAi OiJK V1Qi LCJh bGci OiJS UzI1 NiJ9 The payload is made up of various claims, which are key/value pairs containing information about the user and the key itself. There are many claims which are predefined (called registered claims), but you are free to create your own as well.\nUsually, the payload contains the time when the token was issued and the time when the token expires. These are registered claims and are identified by iat and exp, respectively. Datetime values must be expressed as \u0026ldquo;seconds since the epoch\u0026rdquo;, and python contains built-in functions for converting datetime objects to this numeric format. However, the PyJWT package will take care of this conversion for you when creating a token.\nAnother registered claim is sub (subject) which is meant to represent the entity that the token was issued to. When a user registers with the API, a random UUID value is generated and stored in the database which will be used as the value for sub.\nAn example payload containing these three claims would be: {\u0026quot;sub\u0026quot;: \u0026quot;570eb73b-b4b4-4c86-b35d-390b47d99bf6\u0026quot;, \u0026quot;exp\u0026quot;: 1555873759, \u0026quot;iat\u0026quot;: 1555872854}. The conversion to URL-safe base64-encoded string is shown below:\nASCII: {\"s ub\" :\"5 70e b73 b-b 4b4 -4c 86- b35 d-3 90b 47d 99b f6\" ,\"e xp\" :15 558 737 59, \"ia t\": 155 587 285 4} URLSAFE-B64: eyJz dWIi OiI1 NzBl Yjcz Yi1i NGI0 LTRj ODYt YjM1 ZC0z OTBi NDdk OTli ZjYi LCJl eHAi OjE1 NTU4 NzM3 NTks Imlh dCI6 MTU1 NTg3 Mjg1 NH0= The cryptographic signature is calculated from the header and payload which ensures that the information in both parts has not been modified. The conversion to URL-safe base64-encoded string is shown below:\nHEX: c88b51 cb57fc 521fff 0baf19 162dba b7d3e6 c2395b 90512b 1f1847 4f3ec5 672e URLSAFE-B64: yItR y1f8 Uh__ C68Z Fi26 t9Pm wjlb kFEr HxhH Tz7F Zy4= Combining these into a JWT would result in the following token:\neyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiI1NzBlYjczYi1iNGI0LTRjODYtYjM1ZC0zOTBiNDdkOTliZjYiLCJleHAiOjE1NTU4NzM3NTksImlhdCI6MTU1NTg3Mjg1NH0.yItRy1f8Uh__C68ZFi26t9PmwjlbkFErHxhHTz7FZy4 The PyJWT package trims all padding characters (\u0026quot;=\u0026quot;) from the JWT components. The payload and signature each originally had one such character that is not present in the final version shown above.\n  Base64-encoded strings may look like gibberish, but DO NOT make the mistake of assuming that the payload data has been encrypted. NEVER include any sensitive data (e.g., user password, payment info) in the JWT payload since it can be easily decoded by anyone.\n  Project Dependencies My favorite thing about Python is that for any type of application or library you could possibly need, it\u0026rsquo;s already been created and made available via pip. When it comes to tools for creating REST APIs and JWTs, there is a dizzying array of possibilities. I\u0026rsquo;d like to give a brief overview of the most important packages and Flask extensions that we will be using in this project.\nPyJWT PyJWT is the package we will use to generate and decode JSON Web Tokens (JWTs).\nFlask-RESTx Flask-RESTx is a Flask extension that makes creating APIs simple (in fact, most of the configuration can be done with decorators). This extension provides helpful tools for marshalling data from custom Python objects to an appropriate format for sending as a HTTP response. As you would expect, there are also tools for parsing data from HTTP requests into basic and custom Python datatypes. However, my favorite feature is the visual, interactive documentation that is automatically generated for you using Swagger UI.\nOpenAPI/Swagger UI The OpenAPI Initiative (OAI) is an organization that aims to curate a single format for documenting API services. The OpenAPI format was originally known as the Swagger Specification. Swagger UI is an extremely useful tool that generates a webpage from an OpenAPI/Swagger spec, providing visual documentation for your API that allows anybody to test your API methods, construct requests, inspect responses, etc.\nFlask-CORS Flask-CORS is a Flask extension for handling Cross Origin Resource Sharing (CORS), making cross-origin AJAX possible. Using this extension to enable CORS for all routes (as is the case in this project) is extremely simple. As you will see shortly, the entire process involves initializing the extension with the Flask application instance with default values.\nFlask-SQLAlchemy Flask-SQLAlchemy is a Flask extension that adds support for SQLAlchemy and makes integrating the ORM with your Flask application simple. If you are unfamiliar with SQLAlchemy, the description below from the official documentation is a perfect summation:\nThe SQLAlchemy Object Relational Mapper presents a method of associating user-defined Python classes with database tables, and instances of those classes (objects) with rows in their corresponding tables. It includes a system that transparently synchronizes all changes in state between objects and their related rows, called a unit of work, as well as a system for expressing database queries in terms of the user defined classes and their defined relationships between each other.\n I know, it sounds like magic. Another key feature of SQLALchemy is that the type of database you use (MySQL, SQLite, PostgreSQL, etc.) is almost completely irrelevent (it comes into play if you need to use a feature that is only supported by a specific backend). For example, you could have your API configured to use a PostgreSQL database in production, and use a simple SQLite file as the backend in your test and development environments. There would be no need to change any code to support each configuration, which, again sounds like magic.\nFlask-Migrate (Alembic) Alembic is a database migrations tool created by the author of SQLAlchemy, and Flask-Migrate is a Flask extension that adds Alembic\u0026rsquo;s operations to the Flask CLI. A database migration is a set of changes to a database schema (e.g., add new table, update foreign key relationships, etc.), similar to a commit in a version-control system. With Flask-Migrate, each migration is represented as a script of SQL statements, allowing you to \u0026ldquo;upgrade\u0026rdquo; a database to apply the schema changes or \u0026ldquo;downgrade\u0026rdquo; and undo the changes. This makes the process of deploying database changes to a production environment safe and easy; simply create a migration script when your changes have been tested and verified, then run the migration script in the production environment to apply the changes.\nDevelopment Dependencies The installation script for our application will allow the user to install dependencies that are only needed to run the test set and/or contribute to the development of the app. This is an extremely common option for a Python application, and in fact this is how we will install the application to ensure that we are executing our test cases against the code as it would be installed by an end-user.\nFor a good description of the process we will use to enable this installation option, please read this section from An Introduction to Python Packaging.\nThe [dev] installation option for our project will install a code formatter, a linter, the unit testing framework, some pytest plugins and the pre-commit package which will automatically run the code formatter on all changed files. Next, for each of these tools, I will give a brief explanation of why I chose that specific tool/package.\nPytest I have a strong preference for pytest as my testing framework. Compared to the built-in unittest library (or other frameworks like nose), pytest requires almost no boilerplate code (e.g., inheriting from TestCase) and relies solely on the built-in assert statement for verifying expected behavior. In contrast, with unittest you have to learn a new API with several different methods in order to \u0026ldquo;assert\u0026rdquo; the same expression (i.e., self.assertEqual, self.assertFalse, self.assertIsNotNone, etc.).\nThe other feature that sets pytest apart is fixtures. Fixtures can be extremely complex but in the simplest case a fixture is just a function that constructs and returns a test object (e.g., a function named db that returns a mock database object). A fixture is created by decorating the function with @pytest.fixture:\n@pytest.fixture def db(): return MockDatabase() If we wish to use the mock database object in a test case, we simply add a parameter with the same name as the fixture (i.e., db) to the test case function as shown below:\ndef test_new_user(db, email, password): new_user = User(email=email, password=password) db.session.add(new_user) db.session.commit() user_exists = User.query.filter_by(email=email).first() assert user_exists When this test executes, pytest will discover and call the fixture named db, making the mock database object available within the test case. This method of decoupling test code from the objects needed to execute the test code is an example of dependency injection.\nBlack Black is my preferred code formatter. Compared to YAPF or autopep8, black is deliberately opinionated and provides very few configuration options. With the other formatting tools, you have to spend time tweaking the configuration until it produces your desired format. With black, the only setting I tweak is the maximum line length (I increase it from 79 to 89).\nThis has an additional benefit if you are collaborating with others on a code base, since enforcing consistent style/format is difficult when everyone is using different customized autopep8 settings. Having a consistent style throughout a project will make your team more productive since less time will be spent conforming to style and the code will become easier to digest visually.\nFlake8 Flake8 is my preferred code linter. While black reformats your code, it doesn\u0026rsquo;t modify the behavior in any way (black verifies that the AST is not modified before applying any changes). Flake8 is actually a wrapper for three different static-analysis tools: pydocstyle (checks for compliance with PEP8 formatting rules, like black but stricter), PyFlakes (checks for programming errors that would only be caught at run-time) and mccabe (checks cyclomatic complexity).\nFlake8 can be configured in a multitude of ways, so getting the most out of it requires a bit of an investment. Applied correctly, flake8 will make your code easier to read, less bug-prone and more maintainable. We will explore my preferred flake8 settings later in this tutorial.\nTox Tox is a very powerful tool that can be used as a single entry point for various build, test and release activities. The most common use case for tox is validating the installation process for a project and running arbitrary commands (such as unit tests) within isolated virtual environments. This is extremely important if you need to support multiple Python versions, and extremely helpful since tox automates what would otherwise be a tedious, involved process.\nProject Structure The location of your test code in relation to your app code is very important. There are multiple valid ways to layout your project, and the pros and cons of various project layouts are collected as a helpful set of best practices on the pytest documentation site. The specific recommendations that I have applied to this project are given below:\n  Place a setup.py file in your project's root folder. We will cover the contents of this file shortly, having this file allows you to install your application with pip. Place your test code in a separate folder outside of your application code.  This allows you to run your tests against an installed version of your application after executing either pip install . or pip install -e . The -e flag installs the application in editable mode, which allows you to run your tests against your local development instance of your code. This saves you from having to re-install your application whenever a change is made, since your tests will be executed against the code that you modified.     In addition to these requirements, I am using a project structure with an isolated src folder for this project. The important thing about the src folder is that it is not a Python package (i.e., it does not contain a __init__.py file). The src folder is located at the root of the project and contains only a single folder named flask_api_tutorial. This folder is a Python package and will contain all of our application code.\nThe project root will also contain the tests folder, the setup.py file to install the application, configuration files, license, README, etc. Here\u0026rsquo;s a visual to help if you\u0026rsquo;re confused by my description:\n. (project root folder) |- src | |- flask_api_tutorial | |- __init__.py | |- ... | |- tests | |- __init__.py | |- ... | |- setup.py |- README.md |- ... There are numerous benefits that result from structuring your project in this manner. The most obvious is that you are forced to install your application through pip in order to run your tests. This ensures that your setup.py script correctly deploys the application, allowing any issues to be detected and fixed immediately.\nPython Packaging by Ionel Cristian Mărieș provides an excellent argument in favor of the src layout. Similarly, Testing \u0026amp; Packaging by Hynek Schlawack is a more recent article arguing in favor of the src layout. I strongly recommend reading both blog posts in their entirety.\nCreate Initial Folders \u0026amp; Files With those guidelines in mind, let\u0026rsquo;s start by creating the folder layout for our application (and also create empty __init__.py files for each Python package).\nYou can name your root folder whatever you like (represented by the top-level \u0026ldquo;.\u0026rdquo; node below), or you can be just like me and use flask_api_tutorial. In most projects using the src-folder structure, the root folder and the folder containing the application code within the src-folder will have the same name.\nIn this section, we will work on everything marked as NEW CODE in the chart below (all files will be empty at this point):\n. (project root folder) |- src | |- flask_api_tutorial | |- api | | |- auth | | | |- __init__.py | | | | | |- widgets | | | |- __init__.py | | | | | |- __init__.py | | | |- models | | |- __init__.py | | | |- util | | |- __init__.py | | |- datetime_util.py | | |- result.py | | | |- __init__.py | |- config.py | |- tests | |- __init__.py | |- test_config.py | |- .env |- .gitignore |- .pre-commit-config.yaml |- pyproject.toml |- pytest.ini |- README.md |- run.py |- setup.py |- tox.ini KEY: FOLDER NEW CODE EMPTY FILE   Feel free to create the project structure manually or through the command line as shown below:\n~ $ mkdir flask_api_tutorial \u0026\u0026 cd flask_api_tutorial flask-api-tutorial $ mkdir src \u0026\u0026 cd src flask-api-tutorial/src $ mkdir flask_api_tutorial \u0026\u0026 cd flask_api_tutorial \u0026\u0026 touch __init__.py flask-api-tutorial/src/flask_api_tutorial $ mkdir api \u0026\u0026 cd api \u0026\u0026 touch __init__.py flask-api-tutorial/src/flask_api_tutorial/api $ mkdir auth \u0026\u0026 cd auth \u0026\u0026 touch __init__.py flask-api-tutorial/src/flask_api_tutorial/api/auth $ cd .. flask-api-tutorial/src/flask_api_tutorial/api $ mkdir widgets \u0026\u0026 cd widgets \u0026\u0026 touch __init__.py flask-api-tutorial/src/flask_api_tutorial/api/widgets $ cd ../.. flask-api-tutorial/src/flask_api_tutorial $ mkdir models \u0026\u0026 cd models \u0026\u0026 touch __init__.py flask-api-tutorial/src/flask_api_tutorial/models $ cd .. flask-api-tutorial/src/flask_api_tutorial $ mkdir util \u0026\u0026 cd util \u0026\u0026 touch __init__.py flask-api-tutorial/src/flask_api_tutorial/util $ cd ../../.. flask-api-tutorial $ mkdir tests \u0026\u0026 cd tests \u0026\u0026 touch __init__.py flask-api-tutorial/tests $ cd .. flask-api-tutorial $ Create Virtual Environment Next, create a new virtual environment by whatever method you prefer (this project requires Python 3.6+). I use pyenv to manage multiple installations of Python since various projects must support and be tested against different or multiple versions. For a quick and easy guide to setting up and using pyenv, check out this article from Hacker Noon.\nEven if you do not use pyenv, the process to create and activate a virtual environment will be similar to the steps below:\nflask-api-tutorial $ python --version Python 2.7.14 flask-api-tutorial $ pyenv local 3.7.6 flask-api-tutorial $ python --version Python 3.7.5 flask-api-tutorial $ python -m venv venv --prompt flask-api-tutorial flask-api-tutorial $ source venv/bin/activate (flask-api-tutorial) flask-api-tutorial $ After activating the new virtual environment, upgrade pip, setuptools and wheel:\n(flask-api-tutorial) flask-api-tutorial $ pip install --upgrade pip setuptools wheel # removed package upgrade messages... Successfully installed pip-20.0.2 setuptools-45.2.0 wheel-0.34.2 Finally, initialize a new git repository for our project:\n(flask-api-tutorial) flask-api-tutorial $ git init Initialized empty Git repository in /Users/aaronluna/Projects/flask-api-tutorial Configuration Files If you are familiar with the Python ecosystem, you have probably noticed that the root folder for any project that is more complex than a to-do list is filled with various configuration files, a README.md, a license file, requirements.txt, etc. Unfortunately, this project will be no different. Let\u0026rsquo;s get these out of the way right now.\nAt this point, I recommend switching to your IDE of choice for Python development. I am a huge fan of VSCode, so that is what I will be using.\nREADME.md and .gitignore Create two empty files in the project root folder: one named README.md and the other named .gitignore. Feel free to copy the versions in the github repository for this project. I am not providing an example to copy \u0026amp; paste since people tend to very opinionated about what files/folders they include in their .gitignore. The version I am using is customized from this example .gitignore file for Python projects from the official github repository.\n.env File Create a file named .env in the project root folder and add the values below. Save the file:\nFLASK_APP=run.py FLASK_ENV=development SECRET_KEY=\u0026#34;please change me\u0026#34; FLASK_APP is the filepath (or module import-path) where the Flask application object is located (More info on FLASK_APP).\nFLASK_ENV only has two valid values: development and production. Setting FLASK_ENV=development enables the interactive debugger and automatic file reloader (More info on FLASK_ENV).\nThe SECRET_KEY will be used to sign our JSON authorization tokens. The value you choose for this key should be a long, random string of bytes. It is absolutely vital that this value remains secret since anyone who knows the value can generate authorization keys for your API. The recommended way to generate a SECRET_KEY is to use the Python interpreter:\n(flask-api-tutorial) flask-api-tutorial $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.  import os  os.urandom(24) b'\\x1ah\\xe9\\x00\\x04\\x1d\\x00\\x14($\\x17\\x90\\x1f?~?\\xdc\\xe9\\x91U\\xd2\\xb5\\xd7' Update your .env file to use the random value you generated. Save the changes:\nFLASK_APP=run.py FLASK_ENV=development SECRET_KEY=\u0026#34;\\x1ah\\xe9\\x00\\x04\\x1d\u0026gt;\\x00\\x14($\\x17\\x90\\x1f?~?\\xdc\\xe9\\x91U\\xd2\\xb5\\xd7\u0026#34; Black Configuration Before we write any app code, let\u0026rsquo;s customize the rules used by black. Create a file named pyproject.toml in the project root folder and add the following content:\n[tool.black] line-length = 89 target-version = [\u0026#39;py37\u0026#39;] include = \u0026#39;\\.pyi?$\u0026#39; exclude = \u0026#39;\u0026#39;\u0026#39; /( \\.eggs | \\.git | \\.hg | \\.mypy_cache | \\.pytest_cache | \\.tox | \\.vscode | __pycache__ | _build | buck-out | build | dist | venv )/ \u0026#39;\u0026#39;\u0026#39; I prefer to increase the maximum line length to 89. The black maintainers recommend a line-length of roughly 90, but you should use whatever line length works best for you. target-version controls which Python versions Black-formatted code should target. include and exclude are both regular expressions that match files and folders to format with black and exclude from formatting.\nPre-commit Configuration Wouldn\u0026rsquo;t it be helpful if there was a way to make sure that all of the code within a commit had been formatted with black before the changes were pushed to the remote server? That way, the project will never contain code with different formatting styles, making everything nice and uniform.\nThere is a really easy way to do this thanks to git hooks. There are many different hooks available but the one we are interested in is the pre-commit hook. We can use this hook by providing a script that determines whether the commit is rejected or allowed to proceed.\nThankfully, all of the work has been done for us by the folks behind the pre-commit package that we will install shortly. In order to run black on all files being committed, create a new file named .pre-commit-config.yaml in the project root folder and enter the content below:\nrepos:- repo:https://github.com/psf/blackrev:stablehooks:- id:blacklanguage_version:python3.7The workflow that results from running this script is as follows: if any file in the commit is not correctly formatted, the commit will be rejected and black will apply any necessary changes. Then, you simply update your commit to include the formatting changes, submit it again and this time the commit will be made.\nPytest Configuration Before we begin writing any test code, let\u0026rsquo;s configure how pytest reports test results and setup some of the plugins that we installed. Create a file named pytest.ini in the project root folder and enter the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21  [pytest] addopts =# generate report with details of all (non-pass) test results -ra # show local variables in tracebacks --showlocals # report formatting changes suggested by black --black # report linting issues with flake8 --flake8 # verbose output --verbose norecursedirs =.git .pytest_cache .vscode migrations venv flake8-max-line-length = 89 flake8-ignore = E203, E266, E501, W503 flake8-max-complexity = 18   We are obviously making many configuration decisions in this file. Please note the following:\n Line 2: The addopts config option adds the specified options to the set of command line arguments whenever pytest is executed by the user. In other words, if addopts = -ra --showlocals, executing the command pytest test_config.py would actually execute pytest -ra --showlocals test_config.py. Line 4: The -r flag generates a \"short test summary info\" section at the end of the test session making it easier to see all the non-pass test results. The -a flag means \"all except passes\". Line 6: The --showlocals flag adds all local variable values to the traceback for all test failures. Line 8: The --black flag reports formatting changes that are suggested by black. This option is only available because we will install the pytest-black plugin as a dev requirement. Line 10: The --flake8 flag reports linting changes that are suggested by flake8. This option is only available because we will install the pytest-flake8 plugin as a dev requirement. Line 12: This option should be self-explanatory, I prefer to enable verbose output for all test results. Lines 13-18: The norecursedirs config option tells pytest to not look for test code in the specified list of folders. This makes pytest run much faster since the total number of locations to search is greatly reduced. Line 19: This and all config options that begin with flake8 only apply to the pytest-flake8 plugin. flake8-max-line-length is set to 89 to enforce the same style rule I have customized in my black configuration. Line 21: flake8-ignore tells pytest-flake8 to never report instances of the specified rule violations. This list is copied from the flake8 config settings in black, which supressses these errors since the rules they enfore violate PEP8. Line 22: flake8-max-complexity sets the allowed threshold for cyclomatic complexity. If any function is more complex than the specified value, a flake8 error will be reported in the test results.   Tox Configuration The last configuration file we need is for Tox. The main reason we are using tox is because it allows us to test our src-folder project structure in the proper manner. Right now, the configuration we use will be very simple. Create a new file named tox.ini in the project root folder and add the content below:\n[tox] envlist = py37 [testenv] deps =black flake8 pydocstyle pytest pytest-black pytest-clarity pytest-dotenv pytest-flake8 pytest-flask commands = pytest This file tells tox to install the packages listed under deps (as well as our application) in a new, isolated virtual environment running Python 3.7 and run a single command: pytest.\nThat\u0026rsquo;s all of the configuration files we need! We still need to create another file in the project root folder, though.\nInstallation Script Next, create a new file named setup.py in the project root folder and add the content below. Then, save and close the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66  \u0026#34;\u0026#34;\u0026#34;Installation script for flask-api-tutorial application.\u0026#34;\u0026#34;\u0026#34; from pathlib import Path from setuptools import setup, find_packages DESCRIPTION = ( \u0026#34;Boilerplate Flask API with Flask-RESTx, SQLAlchemy, pytest, flake8, \u0026#34; \u0026#34;tox configured\u0026#34; ) APP_ROOT = Path(__file__).parent README = (APP_ROOT / \u0026#34;README.md\u0026#34;).read_text() AUTHOR = \u0026#34;Aaron Luna\u0026#34; AUTHOR_EMAIL = \u0026#34;contact@aaronluna.dev\u0026#34; PROJECT_URLS = { \u0026#34;Documentation\u0026#34;: \u0026#34;https://aaronluna.dev/series/flask-api-tutorial/\u0026#34;, \u0026#34;Bug Tracker\u0026#34;: \u0026#34;https://github.com/a-luna/flask-api-tutorial/issues\u0026#34;, \u0026#34;Source Code\u0026#34;: \u0026#34;https://github.com/a-luna/flask-api-tutorial\u0026#34;, } INSTALL_REQUIRES = [ \u0026#34;Flask\u0026#34;, \u0026#34;Flask-Bcrypt\u0026#34;, \u0026#34;Flask-Cors\u0026#34;, \u0026#34;Flask-Migrate\u0026#34;, \u0026#34;flask-restx\u0026#34;, \u0026#34;Flask-SQLAlchemy\u0026#34;, \u0026#34;PyJWT\u0026#34;, \u0026#34;python-dateutil\u0026#34;, \u0026#34;python-dotenv\u0026#34;, \u0026#34;requests\u0026#34;, \u0026#34;urllib3\u0026#34;, \u0026#34;werkzeug==0.16.1\u0026#34;, ] EXTRAS_REQUIRE = { \u0026#34;dev\u0026#34;: [ \u0026#34;black\u0026#34;, \u0026#34;flake8\u0026#34;, \u0026#34;pre-commit\u0026#34;, \u0026#34;pydocstyle\u0026#34;, \u0026#34;pytest\u0026#34;, \u0026#34;pytest-black\u0026#34;, \u0026#34;pytest-clarity\u0026#34;, \u0026#34;pytest-dotenv\u0026#34;, \u0026#34;pytest-flake8\u0026#34;, \u0026#34;pytest-flask\u0026#34;, \u0026#34;tox\u0026#34;, ] } setup( name=\u0026#34;flask-api-tutorial\u0026#34;, description=DESCRIPTION, long_description=README, long_description_content_type=\u0026#34;text/markdown\u0026#34;, version=\u0026#34;0.1\u0026#34;, author=AUTHOR, author_email=AUTHOR_EMAIL, maintainer=AUTHOR, maintainer_email=AUTHOR_EMAIL, license=\u0026#34;MIT\u0026#34;, url=\u0026#34;https://github.com/a-luna/flask-api-tutorial\u0026#34;, project_urls=PROJECT_URLS, packages=find_packages(where=\u0026#34;src\u0026#34;), package_dir={\u0026#34;\u0026#34;: \u0026#34;src\u0026#34;}, python_requires=\u0026#34;\u0026gt;=3.6\u0026#34;, install_requires=INSTALL_REQUIRES, extras_require=EXTRAS_REQUIRE, )     Installing the latest version of werkzeug (v1.0.0) breaks Flask-RESTx. However, this is due to an import error and should be fixed soon. Currently, werkzeug is pinned to the last version which does not break Flask-RESTx, and I will update this once this issue has been resolved.\n  If you are unfamiliar with the structure or operation of the setup.py file, I recommend bookmarking this example from the PyPA which is fully documented with comments explaining every keyword-argument that the setup function supports, which kwargs are required or optional, etc.\nInstall flask-api-tutorial Finally, install the flask-api-tutorial application in editable mode:\n(flask-api-tutorial) flask-api-tutorial $ pip install -e .[dev] # removed package install messages... Installing collected packages: PyJWT, six, pycparser, cffi, bcrypt, itsdangerous, MarkupSafe, Jinja2, click, werkzeug, Flask, Flask-Bcrypt, python-dotenv, SQLAlchemy, Flask-SQLAlchemy, python-editor, python-dateutil, Mako, alembic, Flask-Migrate, pytz, pyrsistent, attrs, zipp, importlib-metadata, jsonschema, aniso8601, flask-restx, urllib3, certifi, chardet, idna, requests, Flask-Cors, more-itertools, py, wcwidth, pyparsing, packaging, pluggy, pytest, snowballstemmer, pydocstyle, pytest-dotenv, toml, pathspec, typed-ast, regex, appdirs, black, pytest-flask, distlib, filelock, virtualenv, tox, pyyaml, identify, cfgv, nodeenv, pre-commit, termcolor, pytest-clarity, pytest-black, mccabe, pyflakes, entrypoints, pycodestyle, flake8, pytest-flake8, flask-api-tutorial Running setup.py develop for flask-api-tutorial Successfully installed Flask-1.1.1 Flask-Bcrypt-0.7.1 Flask-Cors-3.0.8 Flask-Migrate-2.5.2 Flask-SQLAlchemy-2.4.1 Jinja2-2.11.1 Mako-1.1.1 MarkupSafe-1.1.1 PyJWT-1.7.1 SQLAlchemy-1.3.13 alembic-1.4.0 aniso8601-8.0.0 appdirs-1.4.3 attrs-19.3.0 bcrypt-3.1.7 black-19.10b0 certifi-2019.11.28 cffi-1.14.0 cfgv-3.1.0 chardet-3.0.4 click-7.0 distlib-0.3.0 entrypoints-0.3 filelock-3.0.12 flake8-3.7.9 flask-api-tutorial flask-restx-0.1.1 identify-1.4.11 idna-2.9 importlib-metadata-1.5.0 itsdangerous-1.1.0 jsonschema-3.2.0 mccabe-0.6.1 more-itertools-8.2.0 nodeenv-1.3.5 packaging-20.1 pathspec-0.7.0 pluggy-0.13.1 pre-commit-2.1.1 py-1.8.1 pycodestyle-2.5.0 pycparser-2.19 pydocstyle-5.0.2 pyflakes-2.1.1 pyparsing-2.4.6 pyrsistent-0.15.7 pytest-5.3.5 pytest-black-0.3.8 pytest-clarity-0.3.0a0 pytest-dotenv-0.4.0 pytest-flake8-1.0.4 pytest-flask-0.15.1 python-dateutil-2.8.1 python-dotenv-0.11.0 python-editor-1.0.4 pytz-2019.3 pyyaml-5.3 regex-2020.2.20 requests-2.23.0 six-1.14.0 snowballstemmer-2.0.0 termcolor-1.1.0 toml-0.10.0 tox-3.14.5 typed-ast-1.4.1 urllib3-1.25.8 virtualenv-20.0.7 wcwidth-0.1.8 werkzeug-0.16.1 zipp-3.0.0   pip install -e . installs the flask-api-tutorial application in the virtual environment in editable mode. This allows our tests to be executed with the folder layout discussed previously, and also allows any changes made to app code or test code to be tested without needing to re-install the flask-api-tutorial application.\n  Next, run the pre-commit install command to actually add the hooks to the local .git folder:\n(flask-api-tutorial) flask-api-tutorial $ pre-commit install pre-commit installed at .git/hooks/pre-commit flask_api_tutorial.util Package The flask_api_tutorial.util package contains general-purpose utlity classes and functions that we will use throughout this project. They are not related to the main topics of this tutorial, so let\u0026rsquo;s get them out of the way before we begin working on the actual project.\nResult Class In a previous post, I demonstrated and explained the merits of incorporating principles from functional programming, with the Result class as a useful example. We will use this class frequently, so please read the linked post. When you finish that, create a new file in src/flask_api_tutorial/util named result.py and add the content below:\n\u0026#34;\u0026#34;\u0026#34;The Result class represents the outcome of an operation.\u0026#34;\u0026#34;\u0026#34; class Result: \u0026#34;\u0026#34;\u0026#34;Represent the outcome of an operation.\u0026#34;\u0026#34;\u0026#34; def __init__(self, success, value, error): \u0026#34;\u0026#34;\u0026#34;Represent the outcome of an operation.\u0026#34;\u0026#34;\u0026#34; self.success = success self.error = error self.value = value def __str__(self): \u0026#34;\u0026#34;\u0026#34;Informal string representation of a result.\u0026#34;\u0026#34;\u0026#34; if self.success: return \u0026#34;[Success]\u0026#34; else: return f\u0026#34;[Failure] {self.error}\u0026#34; def __repr__(self): \u0026#34;\u0026#34;\u0026#34;Official string representation of a result.\u0026#34;\u0026#34;\u0026#34; if self.success: return f\u0026#34;\u0026lt;Result success={self.success}\u0026gt;\u0026#34; else: return f\u0026#39;\u0026lt;Result success={self.success}, message=\u0026#34;{self.error}\u0026#34;\u0026gt;\u0026#39; @property def failure(self): \u0026#34;\u0026#34;\u0026#34;Flag that indicates if the operation failed.\u0026#34;\u0026#34;\u0026#34; return not self.success def on_success(self, func, *args, **kwargs): \u0026#34;\u0026#34;\u0026#34;Pass result of successful operation (if any) to subsequent function.\u0026#34;\u0026#34;\u0026#34; if self.failure: return self if self.value: return func(self.value, *args, **kwargs) return func(*args, **kwargs) def on_failure(self, func, *args, **kwargs): \u0026#34;\u0026#34;\u0026#34;Pass error message from failed operation to subsequent function.\u0026#34;\u0026#34;\u0026#34; if self.success: return self.value if self.value else None if self.error: return func(self.error, *args, **kwargs) return func(*args, **kwargs) def on_both(self, func, *args, **kwargs): \u0026#34;\u0026#34;\u0026#34;Pass result (either succeeded/failed) to subsequent function.\u0026#34;\u0026#34;\u0026#34; if self.value: return func(self.value, *args, **kwargs) return func(*args, **kwargs) @staticmethod def Fail(error_message): \u0026#34;\u0026#34;\u0026#34;Create a Result object for a failed operation.\u0026#34;\u0026#34;\u0026#34; return Result(False, value=None, error=error_message) @staticmethod def Ok(value=None): \u0026#34;\u0026#34;\u0026#34;Create a Result object for a successful operation.\u0026#34;\u0026#34;\u0026#34; return Result(True, value=value, error=None) @staticmethod def Combine(results): \u0026#34;\u0026#34;\u0026#34;Return a Result object based on the outcome of a list of Results.\u0026#34;\u0026#34;\u0026#34; if all(result.success for result in results): return Result.Ok() errors = [result.error for result in results if result.failure] return Result.Fail(\u0026#34;\\n\u0026#34;.join(errors)) datetime_util Module If you\u0026rsquo;ve spent anytime programming in Python, there is a 100% chance that you have encountered an annoying issue with datetime, timezone and/or timedelta objects. The datetime_util module contains helper functions for converting datetime objects from naive to timezone-aware, formatting datetime and timedelta objects as strings and a namedtuple named timespan that represents the difference between two datetime values but provides more data than the set of attributes provided by the timedelta class.\nCreate a new file in src/flask_api_tutorial/util named datetime_util.py and add the content below:\n\u0026#34;\u0026#34;\u0026#34;Helper functions for datetime, timezone and timedelta objects.\u0026#34;\u0026#34;\u0026#34; import time from collections import namedtuple from datetime import datetime, timedelta, timezone DT_AWARE = \u0026#34;%m/%d/%y %I:%M:%S %p %Z\u0026#34; DT_NAIVE = \u0026#34;%m/%d/%y %I:%M:%S %p\u0026#34; DATE_MONTH_NAME = \u0026#34;%b %d%Y\u0026#34; ONE_DAY_IN_SECONDS = 86400 timespan = namedtuple( \u0026#34;timespan\u0026#34;, [ \u0026#34;days\u0026#34;, \u0026#34;hours\u0026#34;, \u0026#34;minutes\u0026#34;, \u0026#34;seconds\u0026#34;, \u0026#34;milliseconds\u0026#34;, \u0026#34;microseconds\u0026#34;, \u0026#34;total_seconds\u0026#34;, \u0026#34;total_milliseconds\u0026#34;, \u0026#34;total_microseconds\u0026#34;, ], ) def utc_now(): \u0026#34;\u0026#34;\u0026#34;Current UTC date and time with the microsecond value normalized to zero.\u0026#34;\u0026#34;\u0026#34; return datetime.now(timezone.utc).replace(microsecond=0) def localized_dt_string(dt, use_tz=None): \u0026#34;\u0026#34;\u0026#34;Convert datetime value to a string, localized for the specified timezone.\u0026#34;\u0026#34;\u0026#34; if not dt.tzinfo and not use_tz: return dt.strftime(DT_NAIVE) if not dt.tzinfo: return dt.replace(tzinfo=use_tz).strftime(DT_AWARE) return dt.astimezone(use_tz).strftime(DT_AWARE) if use_tz else dt.strftime(DT_AWARE) def get_local_utcoffset(): \u0026#34;\u0026#34;\u0026#34;Get UTC offset from local system and return as timezone object.\u0026#34;\u0026#34;\u0026#34; utc_offset = timedelta(seconds=time.localtime().tm_gmtoff) return timezone(offset=utc_offset) def make_tzaware(dt, use_tz=None, localize=True): \u0026#34;\u0026#34;\u0026#34;Make a naive datetime object timezone-aware.\u0026#34;\u0026#34;\u0026#34; if not use_tz: use_tz = get_local_utcoffset() return dt.astimezone(use_tz) if localize else dt.replace(tzinfo=use_tz) def dtaware_fromtimestamp(timestamp, use_tz=None): \u0026#34;\u0026#34;\u0026#34;Time-zone aware datetime object from UNIX timestamp.\u0026#34;\u0026#34;\u0026#34; timestamp_naive = datetime.fromtimestamp(timestamp) timestamp_aware = timestamp_naive.replace(tzinfo=get_local_utcoffset()) return timestamp_aware.astimezone(use_tz) if use_tz else timestamp_aware def remaining_fromtimestamp(timestamp): \u0026#34;\u0026#34;\u0026#34;Calculate time remaining from now until UNIX timestamp value.\u0026#34;\u0026#34;\u0026#34; now = datetime.now(timezone.utc) dt_aware = dtaware_fromtimestamp(timestamp, use_tz=timezone.utc) if dt_aware \u0026lt; now: return timespan(0, 0, 0, 0, 0, 0, 0, 0, 0) return get_timespan(dt_aware - now) def format_timespan_digits(ts): \u0026#34;\u0026#34;\u0026#34;Format a timespan namedtuple as a string resembling a digital display.\u0026#34;\u0026#34;\u0026#34; if ts.days: day_or_days = \u0026#34;days\u0026#34; if ts.days \u0026gt; 1 else \u0026#34;day\u0026#34; return ( f\u0026#34;{ts.days} {day_or_days}, \u0026#34; f\u0026#34;{ts.hours:02d}:{ts.minutes:02d}:{ts.seconds:02d}\u0026#34; ) if ts.seconds: return f\u0026#34;{ts.hours:02d}:{ts.minutes:02d}:{ts.seconds:02d}\u0026#34; return f\u0026#34;00:00:00.{ts.total_microseconds}\u0026#34; def format_timedelta_digits(td): \u0026#34;\u0026#34;\u0026#34;Format a timedelta object as a string resembling a digital display.\u0026#34;\u0026#34;\u0026#34; return format_timespan_digits(get_timespan(td)) def format_timespan_str(ts): \u0026#34;\u0026#34;\u0026#34;Format a timespan namedtuple as a readable string.\u0026#34;\u0026#34;\u0026#34; if ts.days: day_or_days = \u0026#34;days\u0026#34; if ts.days \u0026gt; 1 else \u0026#34;day\u0026#34; return ( f\u0026#34;{ts.days} {day_or_days} \u0026#34; f\u0026#34;{ts.hours:.0f} hours {ts.minutes:.0f} minutes {ts.seconds} seconds\u0026#34; ) if ts.hours: return f\u0026#34;{ts.hours:.0f} hours {ts.minutes:.0f} minutes {ts.seconds} seconds\u0026#34; if ts.minutes: return f\u0026#34;{ts.minutes:.0f} minutes {ts.seconds} seconds\u0026#34; if ts.seconds: return f\u0026#34;{ts.seconds} seconds {ts.milliseconds:.0f} milliseconds\u0026#34; return f\u0026#34;{ts.total_microseconds} mircoseconds\u0026#34; def format_timedelta_str(td): \u0026#34;\u0026#34;\u0026#34;Format a timedelta object as a readable string.\u0026#34;\u0026#34;\u0026#34; return format_timespan_str(get_timespan(td)) def get_timespan(td): \u0026#34;\u0026#34;\u0026#34;Convert timedelta object to timespan namedtuple.\u0026#34;\u0026#34;\u0026#34; (milliseconds, microseconds) = divmod(td.microseconds, 1000) (minutes, seconds) = divmod(td.seconds, 60) (hours, minutes) = divmod(minutes, 60) total_seconds = td.seconds + (td.days * ONE_DAY_IN_SECONDS) return timespan( td.days, hours, minutes, seconds, milliseconds, microseconds, total_seconds, (total_seconds * 1000 + milliseconds), (total_seconds * 1000 * 1000 + milliseconds * 1000 + microseconds), ) Environment Configuration Next, create a file named config.py in the src/flask_api_tutorial folder and add the content below. Save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56  \u0026#34;\u0026#34;\u0026#34;Config settings for for development, testing and production environments.\u0026#34;\u0026#34;\u0026#34; import os from pathlib import Path HERE = Path(__file__).parent SQLITE_DEV = \u0026#34;sqlite:///\u0026#34; + str(HERE / \u0026#34;flask_api_tutorial_dev.db\u0026#34;) SQLITE_TEST = \u0026#34;sqlite:///\u0026#34; + str(HERE / \u0026#34;flask_api_tutorial_test.db\u0026#34;) SQLITE_PROD = \u0026#34;sqlite:///\u0026#34; + str(HERE / \u0026#34;flask_api_tutorial_prod.db\u0026#34;) class Config: \u0026#34;\u0026#34;\u0026#34;Base configuration.\u0026#34;\u0026#34;\u0026#34; SECRET_KEY = os.getenv(\u0026#34;SECRET_KEY\u0026#34;, \u0026#34;open sesame\u0026#34;) BCRYPT_LOG_ROUNDS = 4 TOKEN_EXPIRE_HOURS = 0 TOKEN_EXPIRE_MINUTES = 0 SQLALCHEMY_TRACK_MODIFICATIONS = False PRESERVE_CONTEXT_ON_EXCEPTION = False SWAGGER_UI_DOC_EXPANSION = \u0026#34;list\u0026#34; RESTX_MASK_SWAGGER = False JSON_SORT_KEYS = False class TestingConfig(Config): \u0026#34;\u0026#34;\u0026#34;Testing configuration.\u0026#34;\u0026#34;\u0026#34; TESTING = True SQLALCHEMY_DATABASE_URI = SQLITE_TEST class DevelopmentConfig(Config): \u0026#34;\u0026#34;\u0026#34;Development configuration.\u0026#34;\u0026#34;\u0026#34; TOKEN_EXPIRE_MINUTES = 15 SQLALCHEMY_DATABASE_URI = os.getenv(\u0026#34;DATABASE_URL\u0026#34;, SQLITE_DEV) class ProductionConfig(Config): \u0026#34;\u0026#34;\u0026#34;Production configuration.\u0026#34;\u0026#34;\u0026#34; TOKEN_EXPIRE_HOURS = 1 BCRYPT_LOG_ROUNDS = 13 SQLALCHEMY_DATABASE_URI = os.getenv(\u0026#34;DATABASE_URL\u0026#34;, SQLITE_PROD) PRESERVE_CONTEXT_ON_EXCEPTION = True ENV_CONFIG_DICT = dict( development=DevelopmentConfig, testing=TestingConfig, production=ProductionConfig ) def get_config(config_name): \u0026#34;\u0026#34;\u0026#34;Retrieve environment configuration settings.\u0026#34;\u0026#34;\u0026#34; return ENV_CONFIG_DICT.get(config_name, ProductionConfig)   There are several important things to note about the Config class:\n  Line 15: In the Config base class, we store the value of the SECRET_KEY environment variable. Since this value must be set to something, a default value of \"open sesame\" is used if the SECRET_KEY variable has not been set. This value (and all others) that are set in the base class are available in the Config subclasses (TestingConfig, DevelopmentConfig, ProductionConfig).   Line 17-18, 35, 42: This is where start using our subclasses to make unique configurations for each environment. The amount of time until a token expires is determined by these two values:\nToken expires after: TOKEN_EXPIRE_HOURS + TOKEN_EXPIRE_MINUTES + 5 seconds (the hard-coded 5 second addition allows us to write test cases where the access token has expired).\n TestingConfig: 5 seconds DevelopmentConfig: 15 minutes, 5 seconds ProductionConfig: 1 hour, 5 seconds    Line 29, 36, 44: The value of SQLALCHEMY_DATABASE_URI is set to a different value for each environment. By default, separate SQLite database files will be used for each environment. However, if you add an environment variable named DATABASE_URL to the .env file that contains a URL to a database instance (e.g., PostgreSQL, MSSQL, etc) that value will be used for either the development or production environment based on the value of FLASK_ENV.\n  Line 53: The get_config function retrieves the configuration settings for each environment. This will be used by the create_app method which instantiates the Flask application.    python-dotenv In order for our Config classes to work correctly, the environment variables defined in .env must be set. Rather than setting the value for each environment variable at the command-line every time we open a new terminal, we will use the python-dotenv package to set the values automatically. python-dotenv was installed from setup.py and requires no configuration after being installed.\nAs long as python-dotenv is installed, when the flask command is run any environment variables defined in .env will be set. This allows the os.getenv method to retrieve the values defined in .env and use them in our Flask application.\n  Never commit your .env file to your project\u0026rsquo;s git repository. Doing so publicly exposes the SECRET_KEY, allowing anyone to issue authorization tokens for the API.\n  The Application Factory Pattern In the src/flask_api_tutorial folder\u0026rsquo;s __init__.py file, add the following content and save the file:\n\u0026#34;\u0026#34;\u0026#34;Flask app initialization via factory pattern.\u0026#34;\u0026#34;\u0026#34; from flask import Flask from flask_bcrypt import Bcrypt from flask_cors import CORS from flask_migrate import Migrate from flask_sqlalchemy import SQLAlchemy from flask_api_tutorial.config import get_config cors = CORS() db = SQLAlchemy() migrate = Migrate() bcrypt = Bcrypt() def create_app(config_name): app = Flask(\u0026#34;flask-api-tutorial\u0026#34;) app.config.from_object(get_config(config_name)) cors.init_app(app) db.init_app(app) migrate.init_app(app, db) bcrypt.init_app(app) return app We are using the application factory pattern to create instances of our app. This makes creating different versions of our application with different settings simple, just provide the type of environment you wish to use as the config_name parameter to the create_app function. This value retrieves the configuration settings using the get_config function we created in config.py.\nAfter creating the Flask app and applying the config settings, we initialize the extension objects (cors, db, migrate, bcrypt) by calling each object\u0026rsquo;s init_app method and passing in the newly-created Flask app to the method. By doing this, no application-specific state is stored on the extension object, so one extension object can be used for multiple apps.\nYou should recognize all of the Flask extensions from the first section of this post, except for Flask-Bcrypt. This extension provides bcrypt hashing utitlities which will be used to securely store and verify user passwords.\n  By initializing the Flask-CORS extension as shown, CORS support for all domains and for all origins for all routes has been enabled.\n  Unit Tests: test_config.py Let\u0026rsquo;s verify that the config classes work as expected when we create an instance of our application and specify the environment configuration by name. Create a file named test_config.py in the tests folder, add the following content and save the file:\n\u0026#34;\u0026#34;\u0026#34;Unit tests for environment config settings.\u0026#34;\u0026#34;\u0026#34; import os from flask_api_tutorial import create_app from flask_api_tutorial.config import SQLITE_DEV, SQLITE_PROD, SQLITE_TEST def test_config_development(): app = create_app(\u0026#34;development\u0026#34;) assert app.config[\u0026#34;SECRET_KEY\u0026#34;] != \u0026#34;open sesame\u0026#34; assert not app.config[\u0026#34;TESTING\u0026#34;] assert app.config[\u0026#34;SQLALCHEMY_DATABASE_URI\u0026#34;] == os.getenv(\u0026#34;DATABASE_URL\u0026#34;, SQLITE_DEV) assert app.config[\u0026#34;TOKEN_EXPIRE_HOURS\u0026#34;] == 0 assert app.config[\u0026#34;TOKEN_EXPIRE_MINUTES\u0026#34;] == 15 def test_config_testing(): app = create_app(\u0026#34;testing\u0026#34;) assert app.config[\u0026#34;SECRET_KEY\u0026#34;] != \u0026#34;open sesame\u0026#34; assert app.config[\u0026#34;TESTING\u0026#34;] assert app.config[\u0026#34;SQLALCHEMY_DATABASE_URI\u0026#34;] == SQLITE_TEST assert app.config[\u0026#34;TOKEN_EXPIRE_HOURS\u0026#34;] == 0 assert app.config[\u0026#34;TOKEN_EXPIRE_MINUTES\u0026#34;] == 0 def test_config_production(): app = create_app(\u0026#34;production\u0026#34;) assert app.config[\u0026#34;SECRET_KEY\u0026#34;] != \u0026#34;open sesame\u0026#34; assert not app.config[\u0026#34;TESTING\u0026#34;] assert app.config[\u0026#34;SQLALCHEMY_DATABASE_URI\u0026#34;] == os.getenv( \u0026#34;DATABASE_URL\u0026#34;, SQLITE_PROD ) assert app.config[\u0026#34;TOKEN_EXPIRE_HOURS\u0026#34;] == 1 assert app.config[\u0026#34;TOKEN_EXPIRE_MINUTES\u0026#34;] == 0   In order for the pytest runner to discover the tests, each test class and test case method must begin with \u0026ldquo;test\u0026rdquo; (or \u0026ldquo;Test\u0026rdquo;).\n  In the first line of each test case, the create_app function is called to create a flask application object with the desired configuration settings. We pass the name of the environment to the create_app function, which will retrieve the desired config settings from the get_config function.\nFor each configuration, we verify that the value of SECRET_KEY is not equal to the default value, which verifies that the value from the .env file was successfully retieved. We also check that each database URL is set correctly and that the TOKEN_EXPIRE_HOURS and TOKEN_EXPIRE_MINUTES settings are correct for each environment.\n  If you are using a different database for any environment and you retrieved the URL from the .env file, make sure you update the test case to verify that this value is retrieved correctly.\n  We can run these tests (and our static-analysis tools) with the tox command. This has the added benefit of verifying that the setup.py file correctly installs our application:\n(flask-api-tutorial) flask-api-tutorial $ tox GLOB sdist-make: /Users/aaronluna/Projects/flask-api-tutorial/setup.py py37 create: /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37 py37 installdeps: black, flake8, pydocstyle, pytest, pytest-black, pytest-clarity, pytest-dotenv, pytest-flake8, pytest-flask py37 inst: /Users/aaronluna/Projects/flask-api-tutorial/.tox/.tmp/package/1/flask-api-tutorial-0.1.zip py37 installed: alembic==1.4.0,aniso8601==8.0.0,appdirs==1.4.3,attrs==19.3.0,bcrypt==3.1.7,black==19.10b0,certifi==2019.11.28,cffi==1.14.0,chardet==3.0.4,Click==7.0,entrypoints==0.3,flake8==3.7.9,Flask==1.1.1,flask-api-tutorial==0.1,Flask-Bcrypt==0.7.1,Flask-Cors==3.0.8,Flask-Migrate==2.5.2,flask-restx==0.1.1,Flask-SQLAlchemy==2.4.1,idna==2.9,importlib-metadata==1.5.0,itsdangerous==1.1.0,Jinja2==2.11.1,jsonschema==3.2.0,Mako==1.1.1,MarkupSafe==1.1.1,mccabe==0.6.1,more-itertools==8.2.0,packaging==20.1,pathspec==0.7.0,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pycparser==2.19,pydocstyle==5.0.2,pyflakes==2.1.1,PyJWT==1.7.1,pyparsing==2.4.6,pyrsistent==0.15.7,pytest==5.3.5,pytest-black==0.3.8,pytest-clarity==0.3.0a0,pytest-dotenv==0.4.0,pytest-flake8==1.0.4,pytest-flask==0.15.1,python-dateutil==2.8.1,python-dotenv==0.11.0,python-editor==1.0.4,pytz==2019.3,regex==2020.2.20,requests==2.23.0,six==1.14.0,snowballstemmer==2.0.0,SQLAlchemy==1.3.13,termcolor==1.1.0,toml==0.10.0,typed-ast==1.4.1,urllib3==1.25.8,wcwidth==0.1.8,Werkzeug==0.16.1,zipp==3.0.0 py37 run-test-pre: PYTHONHASHSEED='3249524107' py37 run-test: commands[0] | pytest ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/bin/python cachedir: .tox/py37/.pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 27 items setup.py::FLAKE8 PASSED [ 3%] setup.py::BLACK PASSED [ 7%] src/flask_api_tutorial/__init__.py::FLAKE8 PASSED [ 11%] src/flask_api_tutorial/__init__.py::BLACK PASSED [ 14%] src/flask_api_tutorial/config.py::FLAKE8 PASSED [ 18%] src/flask_api_tutorial/config.py::BLACK PASSED [ 22%] src/flask_api_tutorial/api/__init__.py::FLAKE8 PASSED [ 25%] src/flask_api_tutorial/api/__init__.py::BLACK PASSED [ 29%] src/flask_api_tutorial/api/auth/__init__.py::FLAKE8 PASSED [ 33%] src/flask_api_tutorial/api/auth/__init__.py::BLACK PASSED [ 37%] src/flask_api_tutorial/api/widgets/__init__.py::FLAKE8 PASSED [ 40%] src/flask_api_tutorial/api/widgets/__init__.py::BLACK PASSED [ 44%] src/flask_api_tutorial/models/__init__.py::FLAKE8 PASSED [ 48%] src/flask_api_tutorial/models/__init__.py::BLACK PASSED [ 51%] src/flask_api_tutorial/util/__init__.py::FLAKE8 PASSED [ 55%] src/flask_api_tutorial/util/__init__.py::BLACK PASSED [ 59%] src/flask_api_tutorial/util/datetime_util.py::FLAKE8 PASSED [ 62%] src/flask_api_tutorial/util/datetime_util.py::BLACK PASSED [ 66%] src/flask_api_tutorial/util/result.py::FLAKE8 PASSED [ 70%] src/flask_api_tutorial/util/result.py::BLACK PASSED [ 74%] tests/__init__.py::FLAKE8 PASSED [ 77%] tests/__init__.py::BLACK PASSED [ 81%] tests/test_config.py::FLAKE8 PASSED [ 85%] tests/test_config.py::BLACK PASSED [ 88%] tests/test_config.py::test_config_development PASSED [ 92%] tests/test_config.py::test_config_testing PASSED [ 96%] tests/test_config.py::test_config_production PASSED [100%] ================================================== 27 passed in 7.30s ================================================== _______________________________________________________ summary ________________________________________________________ py37: commands succeeded congratulations :) Flask CLI/Application Entry Point One of the many neat features of Flask is that it comes with a built-in Command-Line Interface (CLI) that is powered by click. In order to use the CLI, Flask needs to be able to find an application instance, which is accomplished with the FLASK_APP environment variable. FLASK_APP must be set to a file path or an import path of a module containing a Flask application (Read this for more info).\n  Make sure you have activated your virtual environment, you will not be able to use the Flask CLI otherwise.\n  You may remember that FLASK_APP was one of the values we defined in our .env file (FLASK_APP=run.py). This tells Flask to look within run.py for an object named app (or a factory method named create_app). Currently, this file is does not exist. If you attempt to run the Flask CLI with the flask command, an exception is thrown:\n(flask-api-tutorial) flask-api-tutorial $ flask Traceback (most recent call last): File \"/Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask/cli.py\", line 240, in locate_app __import__(module_name) ModuleNotFoundError: No module named 'run' During handling of the above exception, another exception occurred: Traceback (most recent call last): File \"/Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask/cli.py\", line 556, in list_commands rv.update(info.load_app().cli.list_commands(ctx)) File \"/Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask/cli.py\", line 388, in load_app app = locate_app(self, import_name, name) File \"/Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask/cli.py\", line 250, in locate_app raise NoAppException('Could not import \"{name}\".'.format(name=module_name)) flask.cli.NoAppException: Could not import \"run\". Usage: flask [OPTIONS] COMMAND [ARGS]... A general utility script for Flask applications. Provides commands from Flask, extensions, and the application. Loads the application defined in the FLASK_APP environment variable, or from a wsgi.py file. Setting the FLASK_ENV environment variable to 'development' will enable debug mode. $ export FLASK_APP=hello.py $ export FLASK_ENV=development $ flask run Options: --version Show the flask version --help Show this message and exit. Commands: db Perform database migrations. routes Show the routes for the app. run Run a development server. shell Run a shell in the app context.   Because we installed python-dotenv, the environment variables defined in .env are read from the file every time the flask command is executed (they can be accessed with the os.getenv method). Without this, we would need to set the value of FLASK_APP manually whenever we open a new terminal window.\n  Create a new file named run.py in the project root folder and add the following content:\n1 2 3 4 5 6 7 8 9 10 11 12  \u0026#34;\u0026#34;\u0026#34;Flask CLI/Application entry point.\u0026#34;\u0026#34;\u0026#34; import os from flask_api_tutorial import create_app, db app = create_app(os.getenv(\u0026#34;FLASK_ENV\u0026#34;, \u0026#34;development\u0026#34;)) @app.shell_context_processor def shell(): return {\u0026#34;db\u0026#34;: db}   Please note the following about run.py (a.k.a the application entry point):\n Line 6: This is the Flask application object which must exist in the run module in order for the flask command to execute without throwing an exception. Line 9: The @app.shell_context_processor decorator makes the decorated method execute when the flask shell command is run. Line 11: The flask shell command will automatically import the objects which are defined in the dictionary which is returned from the shell function.   The shell method in the run.py file is decorated with @app.shell_context_processor. This is the method that executes when we run flask shell. According to the flask --help documentation this command \u0026ldquo;Runs a shell in the app context.\u0026rdquo; If you are not sure what this means, consider the examples below:\n(flask-api-tutorial) flask-api-tutorial $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.  app Traceback (most recent call last): File \"\", line 1, in  NameError: name 'app' is not defined  db Traceback (most recent call last): File \"\", line 1, in  NameError: name 'db' is not defined  from run import app  from flask_api_tutorial import db  app \u0026#60;Flask 'app'\u0026#62;  db \u0026#60;SQLAlchemy engine=sqlite:////Users/aaronluna/Projects/flask_api_tutorial/flask_api_tutorial_dev.db\u0026#62;  exit() (flask-api-tutorial) flask-api-tutorial $ flask shell Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin App: app [development] Instance: /Users/aaronluna/Projects/flask_api_tutorial/instance  app \u0026#60;Flask 'app'\u0026#62;  db \u0026#60;SQLAlchemy engine=sqlite:////Users/aaronluna/Projects/flask_api_tutorial/flask_api_tutorial_dev.db\u0026#62;  exit() In the regular Python interpreter, the app and db objects are not recognized unless explicitly imported. But when we start the interpreter using flask shell the app object is imported automatically. This is nice, but what makes flask shell really useful is the ability to import a dictionary of objects that will be automatically imported in the Python interpreter. We configure this dictionary as the return value of the shell_context_processor function:\n@app.shell_context_processor def shell(): return {\u0026#34;db\u0026#34;: db} In Part 2, as we add model classes for each database table, we will add the models to this dictionary so they will be available to us in the shell context without importing them manually. The shell_context_processor function must return a dictionary and not a list. This allows you to control the names used in the shell, since the dictionary key for each object will be used as the name.\nLet\u0026rsquo;s make sure that the error we saw earlier has been fixed, Run flask:\n(flask-api-tutorial) flask-api-tutorial $ flask Usage: flask [OPTIONS] COMMAND [ARGS]... A general utility script for Flask applications. Provides commands from Flask, extensions, and the application. Loads the application defined in the FLASK_APP environment variable, or from a wsgi.py file. Setting the FLASK_ENV environment variable to 'development' will enable debug mode. $ export FLASK_APP=hello.py $ export FLASK_ENV=development $ flask run Options: --version Show the flask version --help Show this message and exit. Commands: db Perform database migrations. routes Show the routes for the app. run Run a development server. shell Run a shell in the app context. Checkpoint Most of the work done in this section wasn\u0026rsquo;t related to any specific project requirements, but I think we can claim at least partial credit on one (the ProductionConfig settings define the token age as one hour and will be used when creating JWTs). The JWTs must expire after 1 hour (in production) item has been marked as partially complete ():\nUser Management/JWT Authentication\nNew users can register by providing an email address and password\nExisting users can obtain a JWT by providing their email address and password\nJWT contains the following claims: time the token was issued, time the token expires, a value that identifies the user, and a flag that indicates if the user has administrator access\nJWT is sent in access_token field of HTTP response after successful authentication with email/password\nJWTs must expire after 1 hour (in production)\nJWT is sent by client in Authorization field of request header\nRequests must be rejected if JWT has been modified\nRequests must be rejected if JWT is expired\nIf user logs out, their JWT is immediately invalid/expired\nIf JWT is expired, user must re-authenticate with email/password to obtain a new JWT\n API Resource: Widget List\nAll users can retrieve a list of all widgets\nAll users can retrieve individual widgets by name\nUsers with administrator access can add new widgets to the database\nUsers with administrator access can edit existing widgets\nUsers with administrator access can delete widgets from the database\nThe widget model contains a \"name\" attribute which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nThe widget model contains a \"deadline\" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.\nURL and datetime values must be validated before a new widget is added to the database (and when an existing widget is updated).\nThe widget model contains a \"name\" field which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nWidget name must be validated before a new widget is added to the database (and when an existing widget is updated).\nIf input validation fails either when adding a new widget or editing an existing widget, the API response must include error messages indicating the name(s) of the fields that failed validation.\n  If you have any questions (or suggestions/complaints), please let me know in the comments.\n","href":"/series/flask-api-tutorial/part-1/","title":"How To: Create a Flask API with JWT-Based Authentication (Part 1)"},{"categories":"Flask ; Python ; Tutorial-Series","content":"Project Structure The chart below shows the folder structure that was created in Part 1. In this post, we will work on all files marked as NEW CODE. Files that contain code from Part 1 but will not be modified in this post are marked as NO CHANGES.\n. (project root folder) |- src | |- flask_api_tutorial | |- api | | |- auth | | | |- __init__.py | | | | | |- widgets | | | |- __init__.py | | | | | |- __init__.py | | | |- models | | |- __init__.py | | |- user.py | | | |- util | | |- __init__.py | | |- datetime_util.py | | |- result.py | | | |- __init__.py | |- config.py | |- tests | |- __init__.py | |- conftest.py | |- test_config.py | |- test_user.py | |- util.py | |- .env |- .gitignore |- .pre-commit-config.yaml |- pyproject.toml |- pytest.ini |- README.md |- run.py |- setup.py |- tox.ini KEY: FOLDER NEW CODE NO CHANGES EMPTY FILE   Database Models and Migrations with Flask-SQLAlchemy If you have never used SQLAlchemy (or any ORM) before, the basic concept is simple: ORMs allow you to interact with data stored in a database as high-level abstractions such as classes, instances of classes and methods rather than writing raw SQL (the ORM translates your application code into SQL commands and queries).\nA common task that I rarely see covered in tutorials like this is how to manage changes that are made to a database. Changes to the database schema will usually require changing data that is already stored in the database, or migrating the existing data. We will use the Flask-Migrate extension to handle this task and explain how to setup a migration system and how to create and apply migrations.\nUser DB Model In Part 1, we created an instance of the Flask-SQLAlchemy extension with the name db in the src/flask_api_tutorial/__init__.py file and initialized it in the create_app method. The db object contains functions and classes from sqlalchemy and sqlalchemy.orm.\nWhenever we need to declare a new database model (i.e., create a new database table), we create a class that subclasses db.Model. Since we are creating an API that performs user authentication, our first SQLAlchemy model will be a User class that stores login credentials and metadata for registered users.\nCreate a new file user.py in the src/flask_api_tutorial/models folder and add the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61  \u0026#34;\u0026#34;\u0026#34;Class definition for User model.\u0026#34;\u0026#34;\u0026#34; from datetime import datetime, timezone from uuid import uuid4 from flask import current_app from sqlalchemy.ext.hybrid import hybrid_property from flask_api_tutorial import db, bcrypt from flask_api_tutorial.util.datetime_util import ( utc_now, get_local_utcoffset, make_tzaware, localized_dt_string, ) class User(db.Model): \u0026#34;\u0026#34;\u0026#34;User model for storing logon credentials and other details.\u0026#34;\u0026#34;\u0026#34; __tablename__ = \u0026#34;site_user\u0026#34; id = db.Column(db.Integer, primary_key=True, autoincrement=True) email = db.Column(db.String(255), unique=True, nullable=False) password_hash = db.Column(db.String(100), nullable=False) registered_on = db.Column(db.DateTime, default=utc_now) admin = db.Column(db.Boolean, default=False) public_id = db.Column(db.String(36), unique=True, default=lambda: str(uuid4())) def __repr__(self): return ( f\u0026#34;\u0026lt;User email={self.email}, public_id={self.public_id}, admin={self.admin}\u0026gt;\u0026#34; ) @hybrid_property def registered_on_str(self): registered_on_utc = make_tzaware( self.registered_on, use_tz=timezone.utc, localize=False ) return localized_dt_string(registered_on_utc, use_tz=get_local_utcoffset()) @property def password(self): raise AttributeError(\u0026#34;password: write-only field\u0026#34;) @password.setter def password(self, password): log_rounds = current_app.config.get(\u0026#34;BCRYPT_LOG_ROUNDS\u0026#34;) hash_bytes = bcrypt.generate_password_hash(password, log_rounds) self.password_hash = hash_bytes.decode(\u0026#34;utf-8\u0026#34;) def check_password(self, password): return bcrypt.check_password_hash(self.password_hash, password) @classmethod def find_by_email(cls, email): return cls.query.filter_by(email=email).first() @classmethod def find_by_public_id(cls, public_id): return cls.query.filter_by(public_id=public_id).first()   The User class demonstrates several important concepts for creating database models in SQLAlchemy:\n  Line 17: User is defined as a subclass of db.Model. Subclassing db.Model \"registers\" the model with SQLAlchemy, allowing the ORM to create a database table based on the column definitions in Lines 18-23.\n  Line 20: Flask-SQLAlchemy will automatically set the name of the database table by converting the class name (User) to lowercase. However, user is a reserved word in multiple SQL implementations (e.g., PostgreSQL, MySQL, MSSQL), and using any reserved word as a table name is a bad idea. You can override this default value by setting the __tablename__ class attribute.\n  Python class names in CamelCase will also create tablenames by converting to lowercase, with underscores inserted between each word (e.g., Python class CamelCase =\u0026gt; Database table camel_case).\n    Lines 22-27: Use db.Column to define a column. Dy default, the column name will be the same as the name of the attribute you assign it to. The first argument to db.Column is the data type (i.e., db.Integer, db.String(size), db.Boolean). There are many different generic data types as well as vendor-specific data types available. Our site_user table will have the following columns:\n  id: This is the primary key for our table, specified by primary_key=True. We will use the db.Integer data type, but you could use another data type or even specify multiple columns as a primary key. Also, note that we have setup \"autoincrement\" behavior by specifying autoincrement=True. This is possible only with integer data types.\n  email: Like most sites, we will use an email address to identify our users. Notice that we have set a max length of 255 characters for this column with db.String(255). Obviously, this is a mandatory value for all users, which we specify with nullable=False. To ensure that an email address cannot be registered by more than one user, we specify unique=True.\n  password_hash: Storing passwords in a database is poor practice. Instead, we will compute a hash of the password using Flask-Bcrypt and store the hashed value in this column. When a user attempts to authenticate, we will again use Flask-Bcrypt to compare the password provided by the user to the hashed value. This will be explained in detail later on in this post.\n  registered_on: This column will contain the date and time when the user account was created. SQLAlchemy provides many ways to store datetime values, but the simplist method is to use db.DateTime. Notice that we have specified a default value for this column, default=utc_now. This is a function in the app.util.datetime_util module that returns the current UTC date and time as an \"aware\" datetime object. When a new User is added to the database, the current UTC time will be evaluated and stored as the value for registered_on.\n  In this project, all datetime values are assumed to be in UTC when written to the database.\n    admin: This is a flag that indicates whether a user has administrator access. Use the db.Boolean data type to create a column containing only TRUE/FALSE values. By default, users should not have administrator access. We specify default=False to ensure this behavior.\n  public_id: This column will contain UUID (Universally Unique IDentifier) values. This column is defined in the same way as the email column since we are storing a string value that must be unique for all users. However, since this is a random value (i.e., not user-provided), we populate the column similarly to registered_on, with the result of a lambda function that is called when a new User is added to the database.\n  You may be wondering why we are using default=lambda:str(uuid4()), rather than default=uuid4. Calling uuid.uuid4() returns a UUID object, which must be converted to a string before it can be written to the database.\n      Lines 34-39: The @hybrid_property decorator is another SQLAlchemy feature that is capable of much more than what I am demonstrating here. Most often, this decorator is used to create \"computed\" or \"virtual\" columns whose value is computed from the values of one or more columns. In this instance, the registered_on_str column converts the datetime value stored in registered_on to a formatted string.\n  I am using several of the functions from the app.util.datetime_util module here. The registered_on value (and all datetime values) is always converted to the UTC timezone when the value is written to the database. The registered_on_str value converts this value to the timezone of the machine executing this code and formats it as a string value.\n    Lines 41-43: This is part of the password-hashing implementation. The @property decorator exposes a password attribute on our User class. However, this is designed as a write-only value so when a client attempts to call user.password and retrieve the value, an AttributeError is raised.\n  Lines 45-49: This is the setter function for the password @property which calculates the value stored in the password_hash column. This design only stores the hashed value and discards the actual password. Also, hashing the same password multiple times always produces a different value, so it is impossible to compare password_hash values to determine if multiple users have the same password.\n  We are using a value from the Config class, BCRYPT_LOG_ROUNDS. Since we have created our app object using the factory pattern, we must access the Flask application instance through the proxy object current_app (Read this for more info).\n    Lines 51-52: This check_password function is used when the user is attempting to login. The password argument passed into the function is the value provided by the user, and this is provided to the bcrypt.check_password_hash function along with the password_hash value that was created when the user registered their account. The function returns True if the password provided by the user matches the hash, or False otherwise.\n  Lines 54-60: These two class methods are convenience methods that provide a clean, easy-to-read way to retrieve User accounts based on the values stored in the email or public_id columns. Since these values must be unique for all Users, we know that only one or zero Users can be returned from either method.\n   Did you notice that we never defined a __init__ method? That’s because SQLAlchemy adds an implicit constructor to all model classes which accepts keyword arguments for all its columns and relationships. If you decide to override the constructor for any reason, make sure to keep accepting **kwargs and call the super constructor with those **kwargs to preserve this behavior:\nclass User(db.Model): # ... def __init__(self, **kwargs): super(User, self).__init__(**kwargs) # do custom stuff Creating The First Migration Whenever we make a change to our database schema (e.g., add new table, change column name, change foreign key dependencies, etc.) we need to create a migration with the Flask-Migrate extension. Each migration is stored in the migration database (which is actually just a folder named migrations), which also keeps track of the order that the migrations occurred. This allows us to either upgrade or downgrade our database all the way back to the database\u0026rsquo;s initial state.\nThe Flask-Migrate extension adds a new set of commands to the Flask CLI grouped under flask db. In order to create the migration database, we must run flask db init:\n(flask-api-tutorial) flask-api-tutorial $ flask db init Creating directory /Users/aaronluna/Projects/flask-api-tutorial/migrations ... done Creating directory /Users/aaronluna/Projects/flask-api-tutorial/migrations/versions ... done Generating /Users/aaronluna/Projects/flask-api-tutorial/migrations/script.py.mako ... done Generating /Users/aaronluna/Projects/flask-api-tutorial/migrations/env.py ... done Generating /Users/aaronluna/Projects/flask-api-tutorial/migrations/README ... done Generating /Users/aaronluna/Projects/flask-api-tutorial/migrations/alembic.ini ... done Please edit configuration/connection/logging settings in '/Users/aaronluna/Projects/flask-api-tutorial/migrations/alembic.ini' before proceeding. In order for Flask-Migrate to detect the User model, we must import it in the run.py module. Open run.py in the project root folder and make the changes highlighted below:\n1 2 3 4 5 6 7 8 9 10 11 12  \u0026#34;\u0026#34;\u0026#34;Flask CLI/Application entry point.\u0026#34;\u0026#34;\u0026#34; import os from flask_api_tutorial import create_app, db from flask_api_tutorial.models.user import User  app = create_app(os.getenv(\u0026#34;FLASK_ENV\u0026#34;, \u0026#34;development\u0026#34;)) @app.shell_context_processor def shell(): return {\u0026#34;db\u0026#34;: db, \u0026#34;User\u0026#34;: User}     Line 5: The User class will only be detected by the Flask-Migrate extension as a new database table if this import statement exists.\n  Line 12: We have added the User object to the dictionary that is imported by the flask shell command. This makes this class available in the shell context without needing to be explicitly imported.\n     The changes we just made to run.py will be repeated whenever a new model is added. In general, whenever you add a new database model class to your project, you need to update your application entry point (in our case the run.py file to import the new model class before running the flask db migrate command.\n  Ok, after making the changes to run.py we are ready to create our first migration. To do so, we use the flask db migrate command. Also, I recommend adding a message describing the schema changes that will be made, as shown below:\n(flask-api-tutorial) flask-api-tutorial $ flask db migrate --message \"add User model\" INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.autogenerate.compare] Detected added table 'site_user' Generating /Users/aaronluna/Projects/flask-api-tutorial/migrations/versions/eb6c2faa0708_add_user_model.py ... done The flask db migrate command creates the migration script but does not apply the changes to the database. To upgrade the database and execute the migration script you must run the flask db upgrade command:\n(flask-api-tutorial) flask-api-tutorial $ flask db upgrade INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade - eb6c2faa0708, add User model   Each time the database schema changes, repeat the flask db migrate and flask db upgrade steps demonstrated above. Remember to add a message describing the schema changes when a new migration is created with flask db migrate.\n  We can verify that the site_user table has been created using flask shell and the sqlite3 module:\n(flask-api-tutorial) flask-api-tutorial $ flask shell Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin App: app [development] Instance: /Users/aaronluna/Projects/flask_api_tutorial/instance  import sqlite3  from pathlib import Path  DATABASE_URL = app.config.get(\"SQLALCHEMY_DATABASE_URI\")  db = sqlite3.connect(Path(DATABASE_URL).name)  cursor = db.cursor()  results = cursor.execute(\"SELECT name FROM sqlite_master WHERE type='table';\")  for row in results: ... print(row) ... ('alembic_version',) ('site_user',)  exit() The query we executed retrieves all rows from the sqlite_master table where the type column contains table. This returned two rows: alembic_version and site_user. The latter confirms that running the upgrade successfully created the table we specified in user.py.\nThe alembic_version table contains a single column named version_num and a single row containing the version number of the most recent migration that has been executed on this database. The version number can be traced to the migration script files in the migrations folder if you are interested in discovering the low-level details of how database migrations are created.\nJSON Web Token Authentication The user authentication process that we are implementing utilizes JSON Web Tokens (JWT), which were defined in detail in Part 1. The general workflow is given below:\n New User Registration  Client sends a request with email address and password. Server verifies that email address has not already been registered. If this is true the server creates a new User object and sends a response containing a JWT in access_token field. Client stores access_token.   Existing User Login  Client sends a request with email address and password. Server retrieves the User object with the email address provided by the client. Server verifies the password is valid for the User object and sends a response containing a JWT in access_token field if the password is valid. Client stores access_token.   After Successful Login/Registration  Client sends a request with JWT in the Authorization field of the request header. If requested resource requires authorization, server decodes access token and allows client to access requested resource if token is valid. The decoded token contains an admin flag. If the requested resource requires administrator access, the server only allows the client to access the resource if admin = True.     Depending on the environment, the access token will expire after a set amount of time. When this happens, the client must login with their email and password to obtain a new token.\nencode_access_token Function Let\u0026rsquo;s start implementing our workflow by creating the method that will be used to generate access tokens. Since the token will be generated using attributes from a User instance, we will create the method as a member of the User class.\nFirst, update the import statements in user.py to include datetime.timedelta and the jwt package (Line 2 and Line 5 below):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15  \u0026#34;\u0026#34;\u0026#34;Class definition for User model.\u0026#34;\u0026#34;\u0026#34; from datetime import datetime, timedelta, timezone from uuid import uuid4 import jwt from flask import current_app from sqlalchemy.ext.hybrid import hybrid_property from flask_api_tutorial import db, bcrypt from flask_api_tutorial.util.datetime_util import ( utc_now, get_local_utcoffset, make_tzaware, localized_dt_string, )   Then, add the encode_access_token method to user.py:\n55 56 57 58 59 60 61 62 63 64  def encode_access_token(self): now = datetime.now(timezone.utc) token_age_h = current_app.config.get(\u0026#34;TOKEN_EXPIRE_HOURS\u0026#34;) token_age_m = current_app.config.get(\u0026#34;TOKEN_EXPIRE_MINUTES\u0026#34;) expire = now + timedelta(hours=token_age_h, minutes=token_age_m) if current_app.config[\u0026#34;TESTING\u0026#34;]: expire = now + timedelta(seconds=5) payload = dict(exp=expire, iat=now, sub=self.public_id, admin=self.admin) key = current_app.config.get(\u0026#34;SECRET_KEY\u0026#34;) return jwt.encode(payload, key, algorithm=\u0026#34;HS256\u0026#34;)   Let\u0026rsquo;s breakdown how this method generates the access token:\n  Lines 57-58: Using the curent_app proxy object, we retrieve the config settings TOKEN_EXPIRE_HOURS and TOKEN_EXPIRE_MINUTES. Remember, we defined different values for these settings for each environment (development, testing, production).\n  Line 59: We calculate the time when the token will expire based on the config settings and the current time.\n  Lines 60-61: All tokens generated with the testing config settings will expire after five seconds, allowing us to write and execute test cases where the tokens actually expire so we can verify the expected behavior.\n  Line 62: The payload object is where data about the token and the user is stored. The payload contains a set of key/value pairs known as \"claims\" (refer to Part 1 for more info on claims). Our token will contain the following registered claims:\n exp: Date/time when the token will expire iat: Date/time when the token was generated sub: The subject of the token (i.e., the user that the token was generated for)  Our token also contains one private claim:\n admin: True/False value indicating whether the User has administrator access.    Line 63: In order to calculate the token's signature, we must retrieve the SECRET_KEY config setting. We will use this same value to decode the token and ensure that the contents have not been modified.\n  Line 64: The jwt.encode() function accepts three arguments. The first two of which we have just described: the payload and the secret key. The third argument is the signing algorithm. Most applications use the HS256 algorithm, which is short for HMAC-SHA256. The signing algorithm is what protects the payload of the JWT against tampering.\n   Global Test Fixtures: conftest.py In order to test the encode_access_token method, we will need a User object (which requires a database connection, which requires a Flask application instance, etc). In the unittest framework, the correct way to do this would involve creating a base test class with a setup method that creates the application instance and initializes the database. We would then create classes that inherit from the base test class, making the setup method available without duplicating the same code in each test class.\nThere\u0026rsquo;s nothing wrong with this approach, but it\u0026rsquo;s not (IMO) the best or the simplest solution. With pytest, we can eliminate all of the boilerplate code required for class inheritance \u0026ndash; we can even eliminate classes entirely. How? With fixtures, of course!\nFixtures are functions that construct any type of object needed by a test. We can utilize them by declaring our test functions with a parameter whose name is the same as a fixture. When the name of a parameter and the name of a fixture coincide, pytest will execute the fixture function and pass the result to the test function.\nFirst, create a file named util.py in the tests folder and add the content below:\n\u0026#34;\u0026#34;\u0026#34;Shared functions and constants for unit tests.\u0026#34;\u0026#34;\u0026#34; EMAIL = \u0026#34;new_user@email.com\u0026#34; PASSWORD = \u0026#34;test1234\u0026#34; It may seem silly to create a file for just these two string values, but as we develop more test cases you will see why I chose to do so.\nNext, create a new file conftest.py in the tests folder and add the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34  \u0026#34;\u0026#34;\u0026#34;Global pytest fixtures.\u0026#34;\u0026#34;\u0026#34; import pytest from flask_api_tutorial import create_app from flask_api_tutorial import db as database from flask_api_tutorial.models.user import User from tests.util import EMAIL, PASSWORD @pytest.fixture def app(): app = create_app(\u0026#34;testing\u0026#34;) return app @pytest.fixture def db(app, client, request): database.drop_all() database.create_all() database.session.commit() def fin(): database.session.remove() request.addfinalizer(fin) return database @pytest.fixture def user(db): user = User(email=EMAIL, password=PASSWORD) db.session.add(user) db.session.commit() return user   conftest.py is a special filename that pytest automatically looks for and loads test fixtures from, making the fixtures available to all functions in the same folder (and sub-folders) where conftest.py is located. You do not need to add an import statement to any test function in order to use the app or db fixtures we have defined.\nAlso, the app fixture is a special case. The pytest-flask extension looks for the app fixture and automatically creates the client fixture which returns an instance of the Flask test client that we will use to test API calls. You can read more about pytest-flask in the official docs.\nHere a few more things to note about the fixtures we defined in conftest.py:\n  Line 5: The db object imported from the app module (which is the Flask-SQLAlchemy extension object) is renamed to database since we will use the name db for the test fixture that injects the database object into our test functions.\n  Line 17: The db fixture is using the client fixture from pytest-flask. The request parameter is another special pytest feature that can be used as a parameter in any fixture function. The request object gives access to the test context where the fixture was requested.\n  Do not confuse the pytest request object and the global Flask request object. The former is used by a fixture to perform any teardown/destruct process on the test object created by the fixture. The latter represents an HTTP request received by the Flask application and contains the HTML body, headers, etc. sent by the client.\n    Line 18-20: This is my preferred way to teardown/create the database used for testing. You may notice that I do not remove the database after each test run as is common practice, rather I drop all tables before beginning a new test case. This allows me to inspect the database after a failing test run since the data is still present.\n  Line 22-25: The fin function is registered with the addFinalizer method of the pytest request object. The fin function will execute after the test function concludes and in this case removes the database session. You will see this pattern repeated in all fixtures where some sort of teardown process is needed to free resources allocated by the fixture function.\n  Line 30: As demonstrated here, fixtures can incorporate other fixtures. The user fixture relies on the db fixture which relies on the client (i.e., app) fixture. It's fixtures all the way down!\nThis fixture creates a new, regular (non-admin) User instance and commits the instance to the database. The User object is returned to the test function.\n   Unit Test: test_encode_access_token We are finally ready to write test code that verifies the encode_access_token method. Create a new file in the tests folder named test_user.py and add the following content (ensure that test_user.py and conftest.py are in the same folder):\n\u0026#34;\u0026#34;\u0026#34;Unit tests for User model class.\u0026#34;\u0026#34;\u0026#34; def test_encode_access_token(user): access_token = user.encode_access_token() assert isinstance(access_token, bytes) Run the tox command and verify that all tests pass:\n(flask-api-tutorial) flask-api-tutorial $ tox GLOB sdist-make: /Users/aaronluna/Projects/flask-api-tutorial/setup.py py37 create: /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37 py37 installdeps: black, flake8, pydocstyle, pytest, pytest-black, pytest-clarity, pytest-dotenv, pytest-flake8, pytest-flask py37 inst: /Users/aaronluna/Projects/flask-api-tutorial/.tox/.tmp/package/1/flask-api-tutorial-0.1.zip py37 installed: alembic==1.4.0,aniso8601==8.0.0,appdirs==1.4.3,attrs==19.3.0,bcrypt==3.1.7,black==19.10b0,certifi==2019.11.28,cffi==1.14.0,chardet==3.0.4,Click==7.0,entrypoints==0.3,flake8==3.7.9,Flask==1.1.1,flask-api-tutorial==0.1,Flask-Bcrypt==0.7.1,Flask-Cors==3.0.8,Flask-Migrate==2.5.2,flask-restx==0.1.1,Flask-SQLAlchemy==2.4.1,idna==2.9,importlib-metadata==1.5.0,itsdangerous==1.1.0,Jinja2==2.11.1,jsonschema==3.2.0,Mako==1.1.1,MarkupSafe==1.1.1,mccabe==0.6.1,more-itertools==8.2.0,packaging==20.1,pathspec==0.7.0,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pycparser==2.19,pydocstyle==5.0.2,pyflakes==2.1.1,PyJWT==1.7.1,pyparsing==2.4.6,pyrsistent==0.15.7,pytest==5.3.5,pytest-black==0.3.8,pytest-clarity==0.3.0a0,pytest-dotenv==0.4.0,pytest-flake8==1.0.4,pytest-flask==0.15.1,python-dateutil==2.8.1,python-dotenv==0.11.0,python-editor==1.0.4,pytz==2019.3,regex==2020.2.20,requests==2.23.0,six==1.14.0,snowballstemmer==2.0.0,SQLAlchemy==1.3.13,termcolor==1.1.0,toml==0.10.0,typed-ast==1.4.1,urllib3==1.25.8,wcwidth==0.1.8,Werkzeug==0.16.1,zipp==3.0.0 py37 run-test-pre: PYTHONHASHSEED='1533942126' py37 run-test: commands[0] | pytest ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/bin/python cachedir: .tox/py37/.pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 38 items run.py::FLAKE8 PASSED [ 2%] run.py::BLACK PASSED [ 5%] setup.py::FLAKE8 PASSED [ 7%] setup.py::BLACK PASSED [ 10%] src/flask_api_tutorial/__init__.py::FLAKE8 PASSED [ 13%] src/flask_api_tutorial/__init__.py::BLACK PASSED [ 15%] src/flask_api_tutorial/config.py::FLAKE8 PASSED [ 18%] src/flask_api_tutorial/config.py::BLACK PASSED [ 21%] src/flask_api_tutorial/api/__init__.py::FLAKE8 PASSED [ 23%] src/flask_api_tutorial/api/__init__.py::BLACK PASSED [ 26%] src/flask_api_tutorial/api/auth/__init__.py::FLAKE8 PASSED [ 28%] src/flask_api_tutorial/api/auth/__init__.py::BLACK PASSED [ 31%] src/flask_api_tutorial/api/widgets/__init__.py::FLAKE8 PASSED [ 34%] src/flask_api_tutorial/api/widgets/__init__.py::BLACK PASSED [ 36%] src/flask_api_tutorial/models/__init__.py::FLAKE8 PASSED [ 39%] src/flask_api_tutorial/models/__init__.py::BLACK PASSED [ 42%] src/flask_api_tutorial/models/user.py::FLAKE8 PASSED [ 44%] src/flask_api_tutorial/models/user.py::BLACK PASSED [ 47%] src/flask_api_tutorial/util/__init__.py::FLAKE8 PASSED [ 50%] src/flask_api_tutorial/util/__init__.py::BLACK PASSED [ 52%] src/flask_api_tutorial/util/datetime_util.py::FLAKE8 PASSED [ 55%] src/flask_api_tutorial/util/datetime_util.py::BLACK PASSED [ 57%] src/flask_api_tutorial/util/result.py::FLAKE8 PASSED [ 60%] src/flask_api_tutorial/util/result.py::BLACK PASSED [ 63%] tests/__init__.py::FLAKE8 PASSED [ 65%] tests/__init__.py::BLACK PASSED [ 68%] tests/conftest.py::FLAKE8 PASSED [ 71%] tests/conftest.py::BLACK PASSED [ 73%] tests/test_config.py::FLAKE8 PASSED [ 76%] tests/test_config.py::BLACK PASSED [ 78%] tests/test_config.py::test_config_development PASSED [ 81%] tests/test_config.py::test_config_testing PASSED [ 84%] tests/test_config.py::test_config_production PASSED [ 86%] tests/test_user.py::FLAKE8 PASSED [ 89%] tests/test_user.py::BLACK PASSED [ 92%] tests/test_user.py::test_encode_access_token PASSED [ 94%] tests/util.py::FLAKE8 PASSED [ 97%] tests/util.py::BLACK PASSED [100%] ================================================== 38 passed in 6.33s ================================================== _______________________________________________________ summary ________________________________________________________ py37: commands succeeded congratulations :) decode_access_token Function Let\u0026rsquo;s move on to the obvious next step in our authorization workflow: decoding tokens. We need to update the import statements in user.py to include the Result class (Line 16 below):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16  \u0026#34;\u0026#34;\u0026#34;Class definition for User model.\u0026#34;\u0026#34;\u0026#34; from datetime import datetime, timedelta, timezone from uuid import uuid4 import jwt from flask import current_app from sqlalchemy.ext.hybrid import hybrid_property from flask_api_tutorial import db, bcrypt from flask_api_tutorial.util.datetime_util import ( utc_now, get_local_utcoffset, make_tzaware, localized_dt_string, ) from flask_api_tutorial.util.result import Result   Next, add the decode_access_token method to the user.py file:\n67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90  @staticmethod def decode_access_token(access_token): if isinstance(access_token, bytes): access_token = access_token.decode(\u0026#34;ascii\u0026#34;) if access_token.startswith(\u0026#34;Bearer \u0026#34;): split = access_token.split(\u0026#34;Bearer\u0026#34;) access_token = split[1].strip() try: key = current_app.config.get(\u0026#34;SECRET_KEY\u0026#34;) payload = jwt.decode(access_token, key, algorithms=[\u0026#34;HS256\u0026#34;]) except jwt.ExpiredSignatureError: error = \u0026#34;Access token expired. Please log in again.\u0026#34; return Result.Fail(error) except jwt.InvalidTokenError: error = \u0026#34;Invalid token. Please log in again.\u0026#34; return Result.Fail(error) user_dict = dict( public_id=payload[\u0026#34;sub\u0026#34;], admin=payload[\u0026#34;admin\u0026#34;], token=access_token, expires_at=payload[\u0026#34;exp\u0026#34;], ) return Result.Ok(user_dict)   There are several important things to note about this method:\n  Line 67: The @staticmethod decorator indicates that this method is not bound to either a User instance or to the User class. This means that there is no practical difference between having the decode_access_token method defined inside the User class as a static method and having it defined as a regular function inside a module.\nThe only reason to define a method as static is to group related behaviors under a shared namespace. It is simply my opinion that since the encode_access_token method must be bound to a User instance, it is both logical and more aesthetically pleasing to define decode_access_token as a member of the User class.\n  Lines 69-70: Depending on how the access_token was passed to the decode_access_token function, it could either be a byte-array or a string. Before proceeding, we convert access_token to a string if necessary.\n  Lines 71-73: We could add this validation step later, since it will be needed when we define our API routes and incorporate Flask-RESTx. I'm adding it now and telling you: Sometimes, the Authorization field of a request header will be prefixed with \"Bearer\" and sometimes it won't. We need to handle both situations when decoding access tokens.\n  Line 75: Since the token signature was calculated with the SECRET_KEY, we must use the same value to decode the token.   Line 76: The jwt.decode function takes three arguments: the access token, the secret key, and a list of signature algorithms which the application accepts. If the access token is valid, has not been tampered with and has not expired, then the return value of the jwt.decode function (payload) is the dictionary containing the create time and expire time of the token, the user's public ID and a bool indicating if the user has administrator access.\n  Line 77-79: This code will only execute if the token is expired. Note that we do not have to perform any of the work to verify whether the token is expired or not, that is handled by the jwt.decode function. If the token is expired, a jwt.ExpiredSignatureError is raised. Since this is an expected failure, we catch it and create a Result object with an error message describing the failure and return the Result.\n  Line 80-82: This code will only execute if the signature validation process fails. This would occur if the token was tampered or modified in any way, and we will create unit tests to verify this works as expected. Again, we do not have to perform any of the work to determine if the token has been tampered with, that process is handled by the jwt.decode function. If the signature is invalid, a jwt.InvalidTokenError is raised. Since this is an expected failure, we catch it and create a Result object with an error message describing the failure and return the Result.\n  Line 84-90: This code will only execute if the token passed all validation criteria: token format is valid, signature is valid (i.e., token has not been modified/tampered) and token is not expired. In that case, we construct a dict object containing the validated access_token, the timestamp when the token expires, the user's public_id and administrator flag from the payload object. We return the dict within a Result object indicating the operation was successful.\n   Unit Tests: Decode Access Token We want to verify the three possible results of the decode_access_token method: token is valid, token is expired and token is invalid. Before we start writing any tests, place the import statements below at the top of test_user.py:\n1 2 3 4 5 6  \u0026#34;\u0026#34;\u0026#34;Unit tests for User model class.\u0026#34;\u0026#34;\u0026#34; import json import time from base64 import urlsafe_b64encode, urlsafe_b64decode from flask_api_tutorial.models.user import User   The first test will verify the expected behavior for a valid access token. Add the test_decode_access_token_success method to test_user.py:\n14 15 16 17 18 19 20  def test_decode_access_token_success(user): access_token = user.encode_access_token() result = User.decode_access_token(access_token) assert result.success user_dict = result.value assert user.public_id == user_dict[\u0026#34;public_id\u0026#34;] assert user.admin == user_dict[\u0026#34;admin\u0026#34;]   Please note the following verifications we are performing in this method:\n  Line 17: The decode_access_token method returns a Result object, so we first check that result.success is True. If this is the case, we know that access_token is valid and was successfully decoded.\n  Line 18: Since access_token is valid, we know that result.value contains the user_dict object.\n  Lines 19-20: The public_id and admin values in user_dict should match the user that created the access_token.\n   Next, let\u0026rsquo;s create a test that verifies what happens when we attempt to decode an expired access token. Add the test_decode_access_token_expired method to test_user.py:\n23 24 25 26 27 28  def test_decode_access_token_expired(user): access_token = user.encode_access_token() time.sleep(6) result = User.decode_access_token(access_token) assert not result.success assert result.error == \u0026#34;Access token expired. Please log in again.\u0026#34;   Let\u0026rsquo;s go through this test and explain how we achieved the desired result:\n  Line 25: As stated multiple times, when the TestConfig settings are in use, authorization tokens will expire after five seconds. Immediately after generating access_token, we call time.sleep(6) which waits for six seconds before attempting to decode the access token.\n  Line 27: Since we expect decode_access_token to return a Result object indicating access_token could not be decoded, we expect the value of result.success to be False.\n  Line 28: Since access_token was not decoded successfully, we verify that result.error indicates the reason for the failure is because the token is expired.\n   The last test is by far the most interesting of the three. It is trivially easy to modify part of a JWT and send it in place of the token that was generated by the server. For example, what if a user changed the admin claim in their token from False to True? Would they be able to access resources that require administrator access, even though they have not been granted the required access? Let\u0026rsquo;s try it out!\nAdd the test_decode_access_token_invalid method to test_user.py:\n31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48  def test_decode_access_token_invalid(user): access_token = user.encode_access_token() split = access_token.split(b\u0026#34;.\u0026#34;) payload_base64 = split[1] pad_len = 4 - (len(payload_base64) % 4) payload_base64 += pad_len * b\u0026#34;=\u0026#34; payload_str = urlsafe_b64decode(payload_base64) payload = json.loads(payload_str) assert not payload[\u0026#34;admin\u0026#34;] payload[\u0026#34;admin\u0026#34;] = True payload_mod = json.dumps(payload) payload_mod_base64 = urlsafe_b64encode(payload_mod.encode()) split[1] = payload_mod_base64.strip(b\u0026#34;=\u0026#34;) access_token_mod = b\u0026#34;.\u0026#34;.join(split) assert not access_token == access_token_mod result = User.decode_access_token(access_token_mod) assert not result.success assert result.error == \u0026#34;Invalid token. Please log in again.\u0026#34;   Rather than explain this test case line-by-line as done previously, I think it\u0026rsquo;s easier to execute the test in the flask shell interpreter and print out the value of several important variables:\n(flask-api-tutorial) flask-api-tutorial $ flask shell Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin App: flask-api-tutorial [development] Instance: /Users/aaronluna/Projects/flask_api_tutorial  import json  from base64 import urlsafe_b64encode, urlsafe_b64decode  from flask_api_tutorial import db  from flask_api_tutorial.models.user import User  USER_EMAIL = \"new_user@email.com\"  USER_PASSWORD = \"test1234\"  user = User(email=USER_EMAIL, password=USER_PASSWORD)  db.session.add(user)  db.session.commit()  access_token = user.encode_access_token()  split = access_token.split(b'.')  print(f'access_token (original):\\n header: {split[0]}\\n payload: {split[1]}\\n signature: {split[2]}') access_token (original): header: b'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9' payload: b'eyJleHAiOjE1NTc5NDA3MDAsImlhdCI6MTU1NzkzOTc5NSwic3ViIjoiMzhjMDMwYTAtNTdhNC00NmRjLWFjOWYtZTcwZDA0OWUzMDE2IiwiYWRtaW4iOmZhbHNlfQ' signature: b'EvNgTtgbgxpEmJedAOwLuxf6YSq09N2GCRmQOxF2REs' The highlighted value above is the base64-encoded payload from the original access token that was generated by the server. Next, we will decode the payload to reveal the values for the user claims.\n payload_base64 = split[1]  pad_len = 4 - (len(payload_base64) % 4)  payload_base64 += pad_len * b'='  payload_str = urlsafe_b64decode(payload_base64)  payload = json.loads(payload_str)  print(f'payload (original):\\n {json.dumps(payload, indent=2)}') payload (original): { \"exp\": 1557940700, \"iat\": 1557939795, \"sub\": \"38c030a0-57a4-46dc-ac9f-e70d049e3016\", \"admin\": false } Please note that anybody can decode the access token\u0026rsquo;s payload with the urlsafe_b64decode function as shown above. This is why you must never store sensitive user data (especially passwords) in a JWT.\nThe values stored in this token are common to most applications: the time when the token was created, the time when the token expires, a value used to identify the user, and a flag indicating the role/access level of the user.\nSince anybody can edit the value of the payload before sending it back to the server, we want to ensure that the token is rejected if it has been modified. It is especially important to ensure that if the \u0026ldquo;admin\u0026rdquo; value is flipped to \u0026ldquo;True\u0026rdquo;, the user is not given access to protected resources.\n payload['admin'] = True  print(f'payload (modified):\\n {json.dumps(payload, indent=2)}') payload (modified): { \"exp\": 1557940700, \"iat\": 1557939795, \"sub\": \"38c030a0-57a4-46dc-ac9f-e70d049e3016\", \"admin\": true } We have changed the \u0026ldquo;admin\u0026rdquo; value to True and printed the modified payload value to show that the value is now flipped.\n payload_mod = json.dumps(payload)  payload_mod_base64 = urlsafe_b64encode(payload_mod.encode())  split[1] = payload_mod_base64.strip(b'=')  print(f'access_token (modified):\\n header: {split[0]}\\n payload: {split[1]}\\n signature: {split[2]}') access_token (modified): header: b'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9' payload: b'eyJleHAiOiAxNTU3OTQwNzAwLCAiaWF0IjogMTU1NzkzOTc5NSwgInN1YiI6ICIzOGMwMzBhMC01N2E0LTQ2ZGMtYWM5Zi1lNzBkMDQ5ZTMwMTYiLCAiYWRtaW4iOiB0cnVlfQ' signature: b'EvNgTtgbgxpEmJedAOwLuxf6YSq09N2GCRmQOxF2REs' We use the urlsafe_b64encode function to encode the modified payload and then we replace the original encoded payload with this new value. Note that the header and signature portions are the same as the original access token that was generated by the server.\n access_token_mod = b'.'.join(split)  result = User.decode_access_token(access_token_mod)  result.success False  result.error 'Invalid token. Please log in again.'  exit() As expected, the modified access token is not decoded successfully and the error message indicates that the token is invalid, which is expected if the contents of the token have been modified in any way. Therefore, our hypothetical malicious user would not be able to give himself admin access by modifying the JWT. Crisis averted!\nLet\u0026rsquo;s run tox and make sure that all test cases pass:\n(flask-api-tutorial) flask-api-tutorial $ tox GLOB sdist-make: /Users/aaronluna/Projects/flask-api-tutorial/setup.py py37 create: /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37 py37 installdeps: black, flake8, pydocstyle, pytest, pytest-black, pytest-clarity, pytest-dotenv, pytest-flake8, pytest-flask py37 inst: /Users/aaronluna/Projects/flask-api-tutorial/.tox/.tmp/package/1/flask-api-tutorial-0.1.zip py37 installed: alembic==1.4.0,aniso8601==8.0.0,appdirs==1.4.3,attrs==19.3.0,bcrypt==3.1.7,black==19.10b0,certifi==2019.11.28,cffi==1.14.0,chardet==3.0.4,Click==7.0,entrypoints==0.3,flake8==3.7.9,Flask==1.1.1,flask-api-tutorial==0.1,Flask-Bcrypt==0.7.1,Flask-Cors==3.0.8,Flask-Migrate==2.5.2,flask-restx==0.1.1,Flask-SQLAlchemy==2.4.1,idna==2.9,importlib-metadata==1.5.0,itsdangerous==1.1.0,Jinja2==2.11.1,jsonschema==3.2.0,Mako==1.1.1,MarkupSafe==1.1.1,mccabe==0.6.1,more-itertools==8.2.0,packaging==20.1,pathspec==0.7.0,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pycparser==2.19,pydocstyle==5.0.2,pyflakes==2.1.1,PyJWT==1.7.1,pyparsing==2.4.6,pyrsistent==0.15.7,pytest==5.3.5,pytest-black==0.3.8,pytest-clarity==0.3.0a0,pytest-dotenv==0.4.0,pytest-flake8==1.0.4,pytest-flask==0.15.1,python-dateutil==2.8.1,python-dotenv==0.11.0,python-editor==1.0.4,pytz==2019.3,regex==2020.2.20,requests==2.23.0,six==1.14.0,snowballstemmer==2.0.0,SQLAlchemy==1.3.13,termcolor==1.1.0,toml==0.10.0,typed-ast==1.4.1,urllib3==1.25.8,wcwidth==0.1.8,Werkzeug==0.16.1,zipp==3.0.0 py37 run-test-pre: PYTHONHASHSEED='2592492654' py37 run-test: commands[0] | pytest ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/bin/python cachedir: .tox/py37/.pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 41 items run.py::FLAKE8 PASSED [ 2%] run.py::BLACK PASSED [ 4%] setup.py::FLAKE8 PASSED [ 7%] setup.py::BLACK PASSED [ 9%] src/flask_api_tutorial/__init__.py::FLAKE8 PASSED [ 12%] src/flask_api_tutorial/__init__.py::BLACK PASSED [ 14%] src/flask_api_tutorial/config.py::FLAKE8 PASSED [ 17%] src/flask_api_tutorial/config.py::BLACK PASSED [ 19%] src/flask_api_tutorial/api/__init__.py::FLAKE8 PASSED [ 21%] src/flask_api_tutorial/api/__init__.py::BLACK PASSED [ 24%] src/flask_api_tutorial/api/auth/__init__.py::FLAKE8 PASSED [ 26%] src/flask_api_tutorial/api/auth/__init__.py::BLACK PASSED [ 29%] src/flask_api_tutorial/api/widgets/__init__.py::FLAKE8 PASSED [ 31%] src/flask_api_tutorial/api/widgets/__init__.py::BLACK PASSED [ 34%] src/flask_api_tutorial/models/__init__.py::FLAKE8 PASSED [ 36%] src/flask_api_tutorial/models/__init__.py::BLACK PASSED [ 39%] src/flask_api_tutorial/models/user.py::FLAKE8 PASSED [ 41%] src/flask_api_tutorial/models/user.py::BLACK PASSED [ 43%] src/flask_api_tutorial/util/__init__.py::FLAKE8 PASSED [ 46%] src/flask_api_tutorial/util/__init__.py::BLACK PASSED [ 48%] src/flask_api_tutorial/util/datetime_util.py::FLAKE8 PASSED [ 51%] src/flask_api_tutorial/util/datetime_util.py::BLACK PASSED [ 53%] src/flask_api_tutorial/util/result.py::FLAKE8 PASSED [ 56%] src/flask_api_tutorial/util/result.py::BLACK PASSED [ 58%] tests/__init__.py::FLAKE8 PASSED [ 60%] tests/__init__.py::BLACK PASSED [ 63%] tests/conftest.py::FLAKE8 PASSED [ 65%] tests/conftest.py::BLACK PASSED [ 68%] tests/test_config.py::FLAKE8 PASSED [ 70%] tests/test_config.py::BLACK PASSED [ 73%] tests/test_config.py::test_config_development PASSED [ 75%] tests/test_config.py::test_config_testing PASSED [ 78%] tests/test_config.py::test_config_production PASSED [ 80%] tests/test_user.py::FLAKE8 PASSED [ 82%] tests/test_user.py::BLACK PASSED [ 85%] tests/test_user.py::test_encode_access_token PASSED [ 87%] tests/test_user.py::test_decode_access_token_success PASSED [ 90%] tests/test_user.py::test_decode_access_token_expired PASSED [ 92%] tests/test_user.py::test_decode_access_token_invalid PASSED [ 95%] tests/util.py::FLAKE8 PASSED [ 97%] tests/util.py::BLACK PASSED [100%] ================================================= 41 passed in 12.27s ================================================== _______________________________________________________ summary ________________________________________________________ py37: commands succeeded congratulations :) Checkpoint I promise that the pace will pick up, since we have again made very little progress on the API requirements. I believe it\u0026rsquo;s fair to say that one item is completely implemented: JWT contains the following claims: time the token was issued, time the token expires, a value that identifies the user, and a flag that indicates if the user has administrator access. I think we can also claim partial credit on two items: Requests must be rejected if JWT has been modified and Requests must be rejected if JWT is expired.\nUser Management/JWT Authentication\nNew users can register by providing an email address and password\nExisting users can obtain a JWT by providing their email address and password\nJWT contains the following claims: time the token was issued, time the token expires, a value that identifies the user, and a flag that indicates if the user has administrator access\nJWT is sent in access_token field of HTTP response after successful authentication with email/password\nJWTs must expire after 1 hour (in production)\nJWT is sent by client in Authorization field of request header\nRequests must be rejected if JWT has been modified\nRequests must be rejected if JWT is expired\nIf user logs out, their JWT is immediately invalid/expired\nIf JWT is expired, user must re-authenticate with email/password to obtain a new JWT\n API Resource: Widget List\nAll users can retrieve a list of all widgets\nAll users can retrieve individual widgets by name\nUsers with administrator access can add new widgets to the database\nUsers with administrator access can edit existing widgets\nUsers with administrator access can delete widgets from the database\nThe widget model contains a \"name\" attribute which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nThe widget model contains a \"deadline\" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.\nURL and datetime values must be validated before a new widget is added to the database (and when an existing widget is updated).\nThe widget model contains a \"name\" field which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nWidget name must be validated before a new widget is added to the database (and when an existing widget is updated).\nIf input validation fails either when adding a new widget or editing an existing widget, the API response must include error messages indicating the name(s) of the fields that failed validation.\n  As always, leave your feedback and questions in the comments!\n","href":"/series/flask-api-tutorial/part-2/","title":"How To: Create a Flask API with JWT-Based Authentication (Part 2)"},{"categories":"Flask ; Python ; Tutorial-Series","content":"Project Structure The chart below shows the folder structure for this section of the tutorial. In this post, we will work on all files marked as NEW CODE. Files that contain code from previous sections but will not be modified in this post are marked as NO CHANGES.\n. (project root folder) |- src | |- flask_api_tutorial | |- api | | |- auth | | | |- __init__.py | | | |- business.py | | | |- dto.py | | | |- endpoints.py | | | | | |- widgets | | | |- __init__.py | | | | | |- __init__.py | | | |- models | | |- __init__.py | | |- user.py | | | |- util | | |- __init__.py | | |- datetime_util.py | | |- result.py | | | |- __init__.py | |- config.py | |- tests | |- __init__.py | |- conftest.py | |- test_auth_register.py | |- test_config.py | |- test_user.py | |- util.py | |- .env |- .gitignore |- .pre-commit-config.yaml |- pyproject.toml |- pytest.ini |- README.md |- run.py |- setup.py |- tox.ini KEY: FOLDER NEW CODE NO CHANGES EMPTY FILE   Introduction It\u0026rsquo;s finally time to start configuring the API! Keep in mind that the URL routes and the business logic that takes place when a user sends a GET, POST, PUT or DELETE request could all be accomplished using the functions, classes and decorators provided in Flask (that is, without using the Flask-RESTx extension). However, doing so would require considerably more code and would not give us the Swagger UI page to document and test the API.\nBefore we begin, let\u0026rsquo;s discuss what makes a REST API RESTful and make a decision to abide (or not to abide) by the requirements and constraints of REST.\nUnderstanding REST The term \u0026ldquo;REST\u0026rdquo; was introduced in 2000 by Roy Fielding in his doctoral thesis, titled Architectural Styles and the Design of Network-based Software Architectures. I strongly recommend trying to fully digest Fielding\u0026rsquo;s thesis. At the very least you should read Chapter 5 Representational State Transfer (REST).\nWhat makes an API RESTful is much more than creating a set of URIs and implementing the response to GET, POST, PUT, etc. requests. First of all, REST does not require HTTP \u0026ndash; REST is protocol-agnostic. HTTP just happens to be very popular and well-suited to REST systems. REST is truly about resource state, and how hypermedia defines the actions available to these resources. REST is also about the media types that the system uses to represent the resources.\nI say these things because most so-called REST APIs and articles explaining how to design and construct a REST API are not truly RESTful. In this section of the tutorial we will implement the user registration and authentication processes. What is the correct way to design a REST API to perform these actions? Are these actions even appropriate for a REST API? The answer is not as straightforward or obvious as you might think.\nUser Authentication in a RESTful System The following questions/topics are from stackoverflow:\n Understanding REST: Verbs, error codes, and authentication   RESTfully design /login or /register resources?   Which HTTP method should Login and Logout Actions use in a “RESTful” setup   How to design a restful url for login?   Do sessions really violate RESTfulness?   RESTful Authentication   I recommend skimming these discussions. What I took away from them is that there is nothing close to a consensus on the topic of the \u0026ldquo;correct\u0026rdquo; way to design a RESTful authentication system. The design that we will implement ensures that the client\u0026rsquo;s application state is never stored by the server, which adheres to the statelessness constraint of REST. However, the design I chose for the endpoint names clearly violates the naming requirements for RESTful URIs.\nI believe that you should follow the tenets of REST as long as it makes sense for your application. For an authentication API, I believe the best design is to use routes with the verbs /register, /login, etc. A RESTful design where all routes strictly refer to a resource such as /session is far less intuitive, IMO.\nBearer Token Authentication RFC6750 is the specification document that defines an authorization process where clients utilize tokens issued by a resource server to access protected resources. The tokens are exchanged via HTTP response fields and header fields. RFC6750 defines the required format and contents of these fields, as well as recommended best practices for mitigating possible security threats.\nRFC6750 is a concrete implementation of the OAuth 2.0 Authorization Framework. OAuth 2.0 has its own specification document, RFC6749. To avoid duplicating work and to make maintenance of these reference docs easier, quite often terminology referred to in the Bearer Token Authentication spec references RFC6749 for the full definition/explanation. Because of this, I will refer to both documents and reproduce text whenever I am implementing something in order to fulfull a requirement for Bearer Token Authentication.\nMy intent is to adhere to the requirements in every possible case. Please let me know if you believe I have implemented anything incorrectly.\nAPI Versioning After you develop and release an API, it is important to remember that even minor changes such as a value in the request or response data changing from a string to an int could be a breaking change for clients. The best way to avoid causing frustration is to apply strict versioning with the promise that no breaking changes will be introduced within a major version.\nThere are several approaches to API versioning, but the one I prefer is the most explicit \u0026ndash; embed the version number in the URL route. For example, the API route that registers a new user account will be /api/v1/auth/register. The /api/v1 prefix will apply to all API routes, and clients will expect that any tools or proceses that integrate with our API will continue to function as long as they use the same URI.\nThis post on the SparkNotes blog provides a nice summary of the types of changes that are breaking changes and the types that are not:\nBreaking Changes (Very Bad)\nA new required parameter A new required key in POST bodies Removal of an existing endpoint Removal of an existing endpoint request method A materially different internal behavior of an API call – such as a change to the default behavior.   NOT Breaking Changes (Good)\nA new resource or API endpoint A new optional parameter A change to a non-public API endpoint A new optional key in the JSON POST body A new key returned in the JSON response body     API Configuration with Flask-RESTx Just like every other extension, Flask-RESTx can be initialized with a Flask application object (i.e., api.init_app(app) \u0026ndash; doing so would place the API at the website root). However, in most applications we would rather have the API routes configured with a prefix such as /api/v1 to enforce our versioning system.\nThe best way to accomplish this is with Flask blueprints. Typically, blueprints are used to factor a large, monolithic Flask application into logical groupings based on the functional areas of your website. Using a blueprint to isolate the API will allow us to define a url_prefix for the API endpoints.\napi_bp Blueprint In the src/flask_api_tutorial/api/__init__.py file, add the following content:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15  \u0026#34;\u0026#34;\u0026#34;API blueprint configuration.\u0026#34;\u0026#34;\u0026#34; from flask import Blueprint from flask_restx import Api api_bp = Blueprint(\u0026#34;api\u0026#34;, __name__, url_prefix=\u0026#34;/api/v1\u0026#34;) authorizations = {\u0026#34;Bearer\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;apiKey\u0026#34;, \u0026#34;in\u0026#34;: \u0026#34;header\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Authorization\u0026#34;}} api = Api( api_bp, version=\u0026#34;1.0\u0026#34;, title=\u0026#34;Flask API with JWT-Based Authentication\u0026#34;, description=\u0026#34;Welcome to the Swagger UI documentation site!\u0026#34;, doc=\u0026#34;/ui\u0026#34;, authorizations=authorizations, )   There are a few important things to note about how we are configuring the api and api_bp objects:\n  Line 5: This is where we create the Flask blueprint object for our API. The first parameter, \"api\", is the name of the blueprint. All API endpoint names will be prefixed with this value (e.g., api.func_name). The url_prefix value makes all API routes begin with /api/v1 (e.g., api/v1/auth/login).\n  Lines 6, 14: The API will implement Bearer token authentication. Providing a value for authorizations to the Flask-RESTx Api constructor will allow the user to add a JWT to the header of all requests sent through the Swagger UI. Specifically, the Swagger UI will contain a button labeled \"Authorize\" that opens a modal dialog prompting the user for the value of the Bearer access token.\n  Currently, Flask-RESTx only supports OpenAPI 2.0, which lacks sufficient configuration settings to accurately describe Bearer token authentication as a security scheme object. This is not the case in OpenAPI 3.0. Defining an apiKey named Bearer which is located in the Authorization field of the request header achieves nearly the same behavior as Bearer Token Authentication, and provides a dialog window on the Swagger UI page to send requests with the access token in the header.\n    Line 9: Passing the api_bp blueprint object to the Flask-RESTx Api constructor links the two objects and is how all API routes become prefixed with the url_prefix value from api_bp. Later, we will import the api_bp object in the run module and register the blueprint with the Flask application object to complete the process of configuring the API.\n  Lines 10-12: All of these string values are displayed in the Swagger UI.\n  Line 13: The doc value controls the URL path of the Swagger UI. With this value, the Swagger UI path is /api/v1/ui.\n   The next step in configuring the API is registering the api_bp blueprint with our Flask application. The correct place to do this is within the create_app method in the src/flask_api_tutorial/__init__.py file. Open this file and add the highlighted lines below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28  \u0026#34;\u0026#34;\u0026#34;Flask app initialization via factory pattern.\u0026#34;\u0026#34;\u0026#34; from flask import Flask from flask_bcrypt import Bcrypt from flask_cors import CORS from flask_migrate import Migrate from flask_sqlalchemy import SQLAlchemy from flask_api_tutorial.config import get_config cors = CORS() db = SQLAlchemy() migrate = Migrate() bcrypt = Bcrypt() def create_app(config_name): app = Flask(\u0026#34;flask-api-tutorial\u0026#34;) app.config.from_object(get_config(config_name)) from flask_api_tutorial.api import api_bp app.register_blueprint(api_bp)  cors.init_app(app) db.init_app(app) migrate.init_app(app, db) bcrypt.init_app(app) return app     The placement of the import statement is deliberate. To avoid a circular import, we do not want the app.api package to be imported unless the create_app method is invoked.\n  It\u0026rsquo;s a good idea to make sure that everything still works and we have not broken anything, so run the unit tests with tox. They should all pass. Then, run flask routes to see the new URL routes that have been registered in our application:\n(flask-api-tutorial) flask-api-tutorial $ flask routes Endpoint Methods Rule ------------------- ------- -------------------------- api.doc GET /api/v1/ui api.root GET /api/v1/ api.specs GET /api/v1/swagger.json restplus_doc.static GET /swaggerui/\u0026lt;path:filename\u0026gt; static GET /static/\u0026lt;path:filename\u0026gt; The first four routes in the list were added by registering the api_bp blueprint with our application. Next, run flask run to start the development server and point your browser to http://localhost:5000/api/v1/ui (if you are using a different port or hostname on your dev machine, adjust accordingly).\nYou should see something similar to the screenshot below. Note that the URL path, API version, title and description are taken directly from values we provided to the Api constructor in the src/flask_api_tutorial/api/__init__.py file.\n Figure 1 - Swagger UI without API Routes\n  API Namespaces In the same way that we can organize our Flask project with blueprints, we can organize our Flask-RESTx API with namespace objects. Our API will contain two namespaces: auth_ns and widget_ns, which correspond to the flask_api_tutorial.api.auth and flask_api_tutorial.api.widgets packages, respectively. For now, we will focus on auth_ns, since this is the namespace that handles authentication requests.\nCurrently, the src/flask_api_tutorial/api/auth folder only contains the __init__.py file. We need to create 3 new files in the auth folder: business.py, dto.py and endpoints.py. Run the command below from the project root folder to create the files (or create them yourself however you wish):\n(flask-api-tutorial) flask-api-tutorial $ cd src/flask_api_tutorial/api/auth \u0026\u0026 touch business.py dto.py endpoints.py (flask-api-tutorial) flask-api-tutorial/src/flask_api_tutorial/api/auth $ ls -al total 8 drwxr-xr-x 7 aaronluna staff 224 Dec 30 01:20 . drwxr-xr-x 5 aaronluna staff 160 Dec 27 02:47 .. -rw-r--r-- 1 aaronluna staff 0 Dec 29 13:05 __init__.py -rw-r--r-- 1 aaronluna staff 0 Dec 30 01:20 business.py -rw-r--r-- 1 aaronluna staff 0 Dec 30 01:20 dto.py -rw-r--r-- 1 aaronluna staff 0 Dec 30 01:20 endpoints.py All of these files are standard for any Flask-RESTx API namespace package that I create. These files perform specific roles that are common to request handling and response formatting:\n  business.py: This file contains the business logic that executes when an API endpoint in this namespace receives a valid request. (e.g., process registration request, process login request, etc.)\n  dto.py: DTO stands for data transfer object. This file will contain custom objects that parse and validate request data, and API model classes that will serialize our database model classes to JSON objects before sending them in an HTTP response.\n  endpoints.py: This file will contain Flask-RESTx Resource classes. Resources are the most important individual parts of a REST API. Each resource is an API endpoint, and the methods we add to each Resource class control the HTTP methods the endpoint responds to . The following method names are automatically mapped to the corresponding HTTP methods: get, post, put, delete, patch, options and head.\n   Before we begin implementing the API routes and business logic for the authentication processes, we need to understand how to parse the information sent by the client in a HTTP request.\nRequest Parsing and Response Marshalling Flask-RESTx provides two different approaches for parsing and validating request data. Deciding which method to use will depend on the complexity of the data and the interface you provide to the client.\nIn many cases the source of an HTTP POST request is a form submission from a page. Another common scenario is an HTTP GET request that occurs when a client accesses a URL with a query string containing pertinent request data. In these cases, you should use the RequestParser class provided in the reqparse module.\nIf your API endpoint expects a complex object (e.g., a blog post containing full content and metadata) and the source of the reqeust data is NOT a web form (i.e., client is accessing the API programmatically), then you should use the fields module to document the format of the object and instruct the client to send the object as serialized JSON. The expected JSON format is defined as an API model, and is used both for validating request data and documenting the output format for objects returned from HTTP GET requests.\nDocumenting the expected format of request and response data has an additional benefit \u0026ndash; the Swagger UI automatically documents the expected format for any API routes we specify.\nRequest Parser Configuration Flask-RESTx provides the RequestParser class as a way to parse data from the Flask request object. For each value to be parsed, we add an instance of the Argument class to the RequestParser. The Argument class is very flexible and is configured by the parameters listed below:\n name: The name of the argument to parse from the request. default: The value to use if the argument is not found in the request, default value is None. type: The type to convert the parsed argument to. This can be any primitive (e.g., int, str, etc.), but Flask-RESTx also provides more advanced types in the inputs module (e.g., email address, URL, etc.). You can also define your own custom data types. required: By default, arguments that are added to a RequestParser that are not found in the request are set to the default value. If required=True, any request where the argument is not found will be aborted with HTTP exception 400 HTTPStatus.BAD_REQUEST.  location: Where to look on the Flask.request object for the argument (can be args, form, headers, json, values or files). The default behavior is to parse values from values and json. values is actually a dictionary that conbines args and form. Also, you can specify multiple locations with a list (e.g., [\"form\", \"args\"]), the last location in the list takes precedence in the result set.\nI'm sure that this explanation is confusing. I think that you should explicitly define the location on the request object where you expect the data to be found. Since this value is documented on the Swagger UI, it is beneficial to users of the API to know how their requests should be constructed.\n help: If you provide a value, it will be pre-pended to any error message that is raised while parsing the argument. nullable: Whether a null or None value is allowed. By default, this is True.   I recommend reading and fully understanding the documentation explaining how to specify input validation for advanced and custom data types. Also, please review the API documentation for the inputs module which contains a list of advanced data types that are available before you create your own custom validator.\nDefining API Models For POST and PUT requests that create a new resource or update an existing resource in a collection, you should instruct the client to send the resource as a JSON object in the request body. You can define the expected API model by creating a dict object where the keys are the names of the attributes on the JSON object and the values are a class that will validate and convert the attribute to the required data type.\nIn the same way that the inputs module provides primitive data types and a set of predefined data formats to specify the type of each RequestParser Argument, the fields module fulfills the same role for model objects. You can find a list of pre-defined fields in the API documentation. You can also easily create your own custom field by subclassing fileds.Raw, as shown in the Flask-RESTx docs.\nTo avoid duplicating code, if you need to define two models which represent the same ORM object but expose slightly different sets of attributes, you can inherit a model rather than defining the same set of fields twice.\nResponse Marshalling API models can be used to document the output of an API operation as well as the expected request format. This is most often used to return a representation of an ORM object with a subset of the attributes defined by the actual ORM model. For example, the User class includes id and password_hash attributes which store the primary key for the database table and password hash used to verify a user\u0026rsquo;s password during login. There\u0026rsquo;s very little security risk to exposing these two values, but at the bare minimum they expose implementation details which the client has no need for.\nWe will see this shortly in the auth/user API route which inspects the access token of the current user and returns a representation of the User object as the HTTP response. The API model we define as the expected output of this API route omits the id and password_hash attributes from the response.\nauth_ns Endpoints Within the auth_ns namespace, we will create the four API endpoints listed in the table below:\n   Endpoint Name URL Path HTTP Method Authentication Process     api.auth_register /api/v1/auth/register POST Register new user   api.auth_login /api/v1/auth/login POST Authenticate user   api.auth_user /api/v1/auth/user GET Get logged-in user info   api.auth_logout /api/v1/auth/logout POST Blacklist access token      We will implement each endpoint in the same way, following the steps listed below:\n Create request parsers/API models to validate request data and serialize response data. Define the business logic necessary to process the request if validation succeeds. Create a class that inherits from Resource and bind it to the API endpoint/URL route. Define the set of HTTP methods that the API endpoint will support and expose methods on the concrete Resource class for each. Methods named get, post, put, delete, patch, options or head will be called when the API endpoint receives a request of the same HTTP method type.   If the API endpoint does not support the HTTP method, do not expose a method with the name of the HTTP method and the client will receive a response with status code 405 HTTPStatus.METHOD_NOT_ALLOWED.\n   Document the Resource class and all methods as explained in the Flask-RESTx docs. Most of the content on the Swagger UI page is generated by decorating your concrete Resource classes and their methods. Utilize the business logic created in Step 2 within the approprate HTTP methods to process the request. Create unit tests to verify that the input validation provided by the request parsers/API models is working correctly, and verify the endpoint behaves as expected.   This process will result in an updated Swagger UI page containing a documented version of the API endpoint which allows clients to construct data in the expected format, send HTTP requests and inspect responses from the server.\napi.auth_register Endpoint The first resource we create will handle the process of registering a new user account. If this were a full-stack tutorial, we would probably create a registration form that calls this API endpoint when a user clicks the Submit button. I\u0026rsquo;m leaving all decisions regarding the front-end in your hands, therefore designing such a form is your task to conquer.\nauth_reqparser Request Parser When a new user attempts to register, what data is required? The way our User model is defined, the value for email must be unique (i.e., two users cannot register with the same email address). The only other value which is provided by the user is their password, which is not stored in the database (the actual password is ony needed to create the password_hash value and to authenticate a user attempting to login). Open src/flask_api_tutorial/api/auth/dto.py, add the content below and save the file:\n\u0026#34;\u0026#34;\u0026#34;Parsers and serializers for /auth API endpoints.\u0026#34;\u0026#34;\u0026#34; from flask_restx.inputs import email from flask_restx.reqparse import RequestParser auth_reqparser = RequestParser(bundle_errors=True) auth_reqparser.add_argument( name=\u0026#34;email\u0026#34;, type=email(), location=\u0026#34;form\u0026#34;, required=True, nullable=False ) auth_reqparser.add_argument( name=\u0026#34;password\u0026#34;, type=str, location=\u0026#34;form\u0026#34;, required=True, nullable=False ) The first thing to note here is the parameter bundle_errors=True when we instantiate auth_reqparser. This value is false by default, which means that only a single error is reported whenever the request data fails validation. I prefer to have all error messages reported for all arguments in our request parser.\nNext, notice that we have specified type=email() for the email argument. This is a pre-defined type provided by Flask-RESTx that verifies that the value sent in the request is a valid email address. If a request includes a value of \u0026ldquo;213323 kjljk\u0026rdquo; for email, we expect that the user will not be registered and the response will include a status code indicating that a validation error occurred and a message explaining that the value for email is not valid.\nThe remaining parameters are the same for both arguments: location=\u0026quot;form\u0026quot;, required=True, nullable=False. The purpose of each parameter was explained previously, and should answer any questions you have about these settings.\nProcess Registration Request Next, we need to create a function that performs the following actions:\n Add a new user to the database Issue an access token for the new user Construct an HTTP response including the access token and send the response the client  For any response containing sensitive information (e.g., access tokens, credentials, etc), RFC6749 (OAuth 2.0) defines the required and optional fields in both the response body and header:\n5.1 Successful Response The authorization server issues an access token and optional refresh token, and constructs the response by adding the following parameters to the entity-body of the HTTP response: access_token\nREQUIRED. The access token issued by the authorization server.\ntoken_type\nREQUIRED. The type of the token issued as described in Section 7.1. Value is case insensitive.\nexpires_in\nRECOMMENDED. The lifetime in seconds of the access token. For example, the value \"3600\" denotes that the access token will expire in one hour from the time the response was generated. If omitted, the authorization server SHOULD provide the expiration time via other means or document the default value.\nrefresh_token\nOPTIONAL. The refresh token, which can be used to obtain new access tokens using the same authorization grant as described in Section 6.\nscope\nOPTIONAL, if identical to the scope requested by the client; otherwise, REQUIRED. The scope of the access token as described by Section 3.3.\nThe parameters are included in the entity-body of the HTTP response using the \"application/json\" media type as defined by [RFC4627]. The parameters are serialized into a JavaScript Object Notation (JSON) structure by adding each parameter at the highest structure level. Parameter names and string values are included as JSON strings. Numerical values are included as JSON numbers. The order of parameters does not matter and can vary.\nThe authorization server MUST include the HTTP \"Cache-Control\" response header field [RFC2616] with a value of \"no-store\" in any response containing tokens, credentials, or other sensitive information, as well as the \"Pragma\" response header field [RFC2616] with a value of \"no-cache\".\nFor example:\nHTTP/1.1 200 OK\nContent-Type: application/json;charset=UTF-8\nCache-Control: no-store\nPragma: no-cache\n{\n\u0026nbsp;\u0026nbsp;\"access_token\": \"2YotnFZFEjr1zCsicMWpAA\",\n\u0026nbsp;\u0026nbsp;\"token_type\": \"example\",\n\u0026nbsp;\u0026nbsp;\"expires_in\": 3600,\n\u0026nbsp;\u0026nbsp;\"refresh_token\": \"tGzv3JOkF0XG5Qx2TlKWIA\",\n\u0026nbsp;\u0026nbsp;\"example_parameter\": \"example_value\"\n}\n Open src/flask_api_tutorial/api/auth/business.py, add the content below and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35  \u0026#34;\u0026#34;\u0026#34;Business logic for /auth API endpoints.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask import current_app, jsonify from flask_restx import abort from flask_api_tutorial import db from flask_api_tutorial.models.user import User def process_registration_request(email, password): if User.find_by_email(email): abort(HTTPStatus.CONFLICT, f\u0026#34;{email} is already registered\u0026#34;, status=\u0026#34;fail\u0026#34;) new_user = User(email=email, password=password) db.session.add(new_user) db.session.commit() access_token = new_user.encode_access_token() response = jsonify( status=\u0026#34;success\u0026#34;, message=\u0026#34;successfully registered\u0026#34;, access_token=access_token.decode(), token_type=\u0026#34;bearer\u0026#34;, expires_in=_get_token_expire_time(), ) response.status_code = HTTPStatus.CREATED response.headers[\u0026#34;Cache-Control\u0026#34;] = \u0026#34;no-store\u0026#34; response.headers[\u0026#34;Pragma\u0026#34;] = \u0026#34;no-cache\u0026#34; return response def _get_token_expire_time(): token_age_h = current_app.config.get(\u0026#34;TOKEN_EXPIRE_HOURS\u0026#34;) token_age_m = current_app.config.get(\u0026#34;TOKEN_EXPIRE_MINUTES\u0026#34;) expires_in_seconds = token_age_h * 3600 + token_age_m * 60 return expires_in_seconds if not current_app.config[\u0026#34;TESTING\u0026#34;] else 5     Lines 12-13: The first thing we do is verify that the email address provided by the user has not been registered. If a User already exists with the same email address, the request is aborted.\n  The abort function is provided by Flask-RESTx and is the correct way to abort a request received by an API endpoint. The first argument is the HTTP status code to include in the response. In this case, the appropriate response code is 409 HTTPStatus.CONFLICT. The remaining arguments are included in the response body.\n    Line 14-16: If the email address has not been registered, we proceed to create a User object with the provided email and password values, and then commit the new user to the database.\n  Line 17: The response needs to include an access token and we can issue one by calling encode_access_token on the user object.\n  Line 18: The specification cited above requires that any response containing an access token must have the Cache-Control and Pragma fields present in the header. The only way to add the necessary headers is by constructing the response object ourselves.\nFlask provides the jsonify function which takes either a dict, a list of arguments, or a list of keyword-arguments and converts the data to a JSON object (similar to calling json.dumps on an object). Finally, jsonify returns a response object with the JSON object as the response body.\n  Line 21: Per the specification, the access_token attribute is included as a parameter of the serialized JSON in the response body.\n  Line 22: Per the specification, the token_type attribute is included as a parameter of the serialized JSON in the response body.\n  Line 23: Per the specification, the expires_in attribute is included as a parameter of the serialized JSON in the response body.\nWe calculate the lifespan of the access_token from the app.config values TOKEN_EXPIRE_HOURS and TOKEN_EXPIRE_MINUTES. If the app.config[\"TESTING\"] flag is set, then five seconds is used as the lifespan of the token. Otherwise, the lifespan in seconds is calculated with TOKEN_EXPIRE_HOURS * 3600 + TOKEN_EXPIRE_MINUTES * 60.\n  Line 25: The most appropriate HTTP status code for a response indicating we have created a new resource is 201 HTTPStatus.CREATED.\n  Line 26-27: The final requirement in the specification cited above is that the response must include the HTTP Cache-Control response header field with a value of no-store, and the Pragma response header field with a value of no-cache\n  Line 28: After ensuring that all required response body and header elements have been created and populated correctly, we send the HTTP response containing the newly issued access_token to the client.\n   Next, we need to create the API endpoint and incorporate it with the auth_reqparser and process_registration_request function.\nRegisterUser Resource If you look back at all of the material covered in this tutorial, it\u0026rsquo;s amazing (IMO) that we haven\u0026rsquo;t written one line of code that that makes our Flask application perform one of the basic functions of a web server: URL routing. It\u0026rsquo;s time to fix that.\nIn an application that adheres to the principles of REST, each API endpoint (IOW, each URL) is a representation of a resource. Clients interact with resources by sending HTTP requests. The method type of the client\u0026rsquo;s request (e.g., GET, PUT, POST) is used to perform different operations, and the nature of the operation should be related to the. For this reason, when we need to add a URL route to the API, we define a class that inherits from the flask_restx.Resource base class.\nAccording to the table defining the API endpoints for the auth_ns namespace, users can register for a new account by sending a POST request to /api/v1/auth/register. To create this API endpoint, open src/flask_api_tutorial/api/auth/endpoints.py, add the content below and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26  \u0026#34;\u0026#34;\u0026#34;API endpoint definitions for /auth namespace.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_restx import Namespace, Resource from flask_api_tutorial.api.auth.dto import auth_reqparser from flask_api_tutorial.api.auth.business import process_registration_request auth_ns = Namespace(name=\u0026#34;auth\u0026#34;, validate=True) @auth_ns.route(\u0026#34;/register\u0026#34;, endpoint=\u0026#34;auth_register\u0026#34;) class RegisterUser(Resource): \u0026#34;\u0026#34;\u0026#34;Handles HTTP requests to URL: /api/v1/auth/register.\u0026#34;\u0026#34;\u0026#34; @auth_ns.expect(auth_reqparser) @auth_ns.response(int(HTTPStatus.CREATED), \u0026#34;New user was successfully created.\u0026#34;) @auth_ns.response(int(HTTPStatus.CONFLICT), \u0026#34;Email address is already registered.\u0026#34;) @auth_ns.response(int(HTTPStatus.BAD_REQUEST), \u0026#34;Validation error.\u0026#34;) @auth_ns.response(int(HTTPStatus.INTERNAL_SERVER_ERROR), \u0026#34;Internal server error.\u0026#34;) def post(self): \u0026#34;\u0026#34;\u0026#34;Register a new user and return an access token.\u0026#34;\u0026#34;\u0026#34; request_data = auth_reqparser.parse_args() email = request_data.get(\u0026#34;email\u0026#34;) password = request_data.get(\u0026#34;password\u0026#34;) return process_registration_request(email, password)     Line 9: Flask-RESTx Namespace objects are used to group a related set of API endpoints in the same way that Flask Blueprint objects can be used to group related URL routes.\nIn this file, we will use the auth_ns object repeatedly as a decorator. Most of these have an effect on the behavior of the class or method they decorate, some do not. All of these decorators have one thing in common \u0026mdash; they all produce some sort of documentation on the Swagger UI page.\nThese decorators can inform clients about the expected format of request and response data or the set of possible HTTP status codes that the client can expect to receive from the server in a response. Also, docstrings for HTTP methods are rendered on the Swagger UI page and should be used to provide a short description of the method's purpose.\nCheck out the Flask-RESTx docs for examples of using decorators to document the Swagger UI page (if you need even more info it's probably in the API documentation).\n  Line 12: The route decorator is used to decorate a class that inherits from Resource. Here, the @auth_ns.route decorator registers the RegisterUser resource with the auth_ns namespace.\nThe first argument (\"/register\") is the URL route to register. The endpoint parameter overrides the default value for the endpoint name. I like to specify this value to enforce a consistent naming scheme for all endpoints within the same namespace.\n  Line 13: The RegisterUser class, which adds the \"/register\" endpoint to the auth_ns namespace, inherits from the Resource base class.\n  Line 16: The expect decorator is used to specify the data that the server expects the client to send in the HTTP request. The first argument can be either a request parser or an API model that defines the expected input model. The optional second argument is a bool value named validate. If validate=True, the request data will be checked to make sure it matches the expected input model.\nYou can also control validation behavior for an entire namespace, which we did when the auth_ns namespace was created in Line 9. You can also define this behavior for the entire API when instantiating the api object, or by setting the value of the app configuration setting RESTPLUS_VALIDATE. You can override the validation behavior for each method using the expect decorator.\nWe are using the auth_reqparser we created in src/flask_api_tutorial/api/auth/dto.py. In the Swagger UI, this renders a form with textboxes for the email and password values and also enforces the rules we configured for each argument. If we had used an API model, the Swagger UI instead renders a single textbox and an example of the expected JSON.\n  Line 17-20: The response decorator is solely for documentation purposes, removing these lines would have no impact on the behavior of this API endpoint. Still, you should document all of the response codes that can possibly be received from this endpoint. The second argument is a string value explaining why the client's request resulted in the response code that was sent, and is included in the Swagger UI page.\n  Line 21: Since the only supported HTTP method for this endpoint is POST, the only method exposed by the RegisterUser class is named post.\n  Line 22: This docstring will be rendered on the Swagger UI page.\n  Line 23-25: In order to access the email and password values provided by the user, we call the parse_args method on the auth_reqparer object. This method returns a dict object containing the validated arguments.\n  Line 26: Finally, we call the method we created to process a registration request and provide the email and password provided by the user.\n   Add auth_ns Namespace to api In order to register the auth_ns namespace with the api object, open src/flask_api_tutorial/api/__init__.py and add the highlighted lines (Line 5 and Line 19):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19  \u0026#34;\u0026#34;\u0026#34;API blueprint configuration.\u0026#34;\u0026#34;\u0026#34; from flask import Blueprint from flask_restx import Api from flask_api_tutorial.api.auth.endpoints import auth_ns  api_bp = Blueprint(\u0026#34;api\u0026#34;, __name__, url_prefix=\u0026#34;/api/v1\u0026#34;) authorizations = {\u0026#34;Bearer\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;apiKey\u0026#34;, \u0026#34;in\u0026#34;: \u0026#34;header\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Authorization\u0026#34;}} api = Api( api_bp, version=\u0026#34;1.0\u0026#34;, title=\u0026#34;Flask API with JWT-Based Authentication\u0026#34;, description=\u0026#34;Welcome to the Swagger UI documentation for the Widget API\u0026#34;, doc=\u0026#34;/ui\u0026#34;, authorizations=authorizations, ) api.add_namespace(auth_ns, path=\u0026#34;/auth\u0026#34;)   The path parameter in the add_namespace method sets the prefix for all endpoints in the auth_ns namepsace. This, along with the url_prefix value in Line 8, is why all URL routes in the auth_ns namespace begin with /api/v1/auth.\nWe can verify that our route has been correctly registered by running flask routes:\nflask-api-tutorial $ flask routes Endpoint Methods Rule ------------------- ------- -------------------------- api.auth_register POST /api/v1/auth/register api.doc GET /api/v1/ui api.root GET /api/v1/ api.specs GET /api/v1/swagger.json restplus_doc.static GET /swaggerui/\u0026lt;path:filename\u0026gt; static GET /static/\u0026lt;path:filename\u0026gt; The presence of the api.auth_register endpoint in the list of routes confirms a number of things:\n The RegisterUser resource supports HTTP POST requests, and no other method types. RegisterUser is within the auth_ns namespace. Because the api_bp Blueprint object is linked with the api object AND registered with the app Flask application object, adding the auth_ns Namespace object to the api object with the add_namespace method automatically registers all routes within the namespace with the Flask application object.   Start the development server by running flask run and point your browser to http://localhost:5000/api/v1/ui to check out the Swagger UI:\n Figure 2 - Swagger UI with api.auth_register endpoint\n  You can click anywhere on the green bar to expand the component. It might not seem like a huge deal, but everything you see was automatically generated by Flask-RESTx (from the api object, auth_ns object, auth_reqparser, RegisterUser, etc):\n Figure 3 - api.auth_register endpoint expanded\n  If you\u0026rsquo;d like to send a request, click the Try It Out button. Then, enter any valid email address and any value for password and click Execute:\n Figure 4 - api.auth_register endpoint ready to test\n  You should receive a response with status code 201 HTTPStatus.CREATED if the email address is formatted correctly (this is the only validation process being performed by auth_reqparser):\n Figure 5 - New user successfully registered (Swagger UI)\n    The Swagger UI helpfully provides a Curl textbox that contains the exact request that was submitted to the server based on the values you provided. cURL is a ubiquitous tool and you can copy and paste the contents of the textbox into any terminal if you would like to test your API from the command-line.\n  If you attempt to register with an email address that already exists in the database, you should receive a response with status code 409 HTTPStatus.CONFLICT. You can also test the API with a command-line tool (e.g., httpie, curl, wget, etc):\nflask-api-tutorial $ http -f :5000/api/v1/auth/register email=user@test.com password=123456 POST /api/v1/auth/login HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Connection: keep-alive Content-Length: 37 Content-Type: application/x-www-form-urlencoded; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/2.0.0 email=user%40test.com\u0026password=123456 HTTP/1.0 409 CONFLICT Access-Control-Allow-Origin: * Content-Length: 79 Content-Type: application/json Date: Sat, 03 Aug 2019 23:20:29 GMT Server: Werkzeug/0.16.1 Python/3.7.6 { \"message\": \"user@test.com is already registered\", \"status\": \"fail\", }   The CLI examples I provide in this tutorial will NOT use cURL I prefer httpie because the syntax is much cleaner and more intuitive. The options for styling and formatting the output are a huge plus as well.\n  Here\u0026rsquo;s an example of a successful request using httpie. Note that on the command-line or Swagger UI the response from the server is always formatted as JSON:\nflask-api-tutorial $ http -f :5000/api/v1/auth/register email=user2@test.com password=123456 POST /api/v1/auth/login HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Connection: keep-alive Content-Length: 37 Content-Type: application/x-www-form-urlencoded; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/2.0.0 email=user2%40test.com\u0026password=123456 HTTP/1.0 201 CREATED Access-Control-Allow-Origin: * Content-Length: 79 Content-Type: application/json Date: Sat, 03 Aug 2019 23:20:29 GMT Server: Werkzeug/0.16.1 Python/3.7.6 { \"access_token\": \"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1ODI5MzQwMzMsImlhdCI6MTU4MjkzMzEzMywic3ViIjoiNjcwOTVlZDUtZjdhYS00MGE3LTgzZGUtNzQ1YmMzYjA5NDFmIiwiYWRtaW4iOmZhbHNlfQ.ylvNfoWwhI-NRU2WS65t4ti6sTbOEDQcJYIQC6ua0Do\", \"expires_in\": 900, \"message\": \"successfully registered\", \"status\": \"success\", \"token_type\": \"bearer\" } Everything appears to be working correctly for the /register endpoint. Next, we will figure out how to create unit tests that interact with the API.\nUnit Tests: test_auth_register.py Before we start writing test cases for the newly-created endpoint, we need to add a function to tests/util.py. Open the file and add the lines highlighted below (Lines 2, 8-13):\n1 2 3 4 5 6 7 8 9 10 11 12 13  \u0026#34;\u0026#34;\u0026#34;Shared functions and constants for unit tests.\u0026#34;\u0026#34;\u0026#34; from flask import url_for  EMAIL = \u0026#34;new_user@email.com\u0026#34; PASSWORD = \u0026#34;test1234\u0026#34; def register_user(test_client, email=EMAIL, password=PASSWORD): return test_client.post( url_for(\u0026#34;api.auth_register\u0026#34;), data=f\u0026#34;email={email}\u0026amp;password={password}\u0026#34;, content_type=\u0026#34;application/x-www-form-urlencoded\u0026#34;, )   Let\u0026rsquo;s take a moment and discuss the register_user function. Understanding how we are using this method to test our API is absolutely vital to successfully completing this project:\n  Line 8: This function is not a test case (since the name does not begin with test_). register_user takes a Flask test client instance, and values for email and password as parameters. The test client instance must always be passed when using this function, but email and password will use the default values if none are specified.\n  Line 9: The Flask test client allows us to make HTTP requests. In order to register a new user, we must send a POST request to the api.auth_register endpoint. To do so, we call the test client's post method. The test client is capable of sending requests for all HTTP method types: get, post, put, delete, patch, options, head and trace.\n  Line 10: The first argument to the post method is the URL is the target of our request. Since the target URL is within our Flask application, we can dynamically construct the URL using the url_for function. This is really useful because it allows us to create links within our application without hardcoding any part of the path. All we need to do to use the url_for function is provide the name of the API endpoint, and voila, the URL is dynamically generated and provided to the post method.\n  Line 11: For a POST request, the server expects the data to be sent in the request body. Since we are simulating a form submission, we must format the data as a series of name/value pairs, each pair separated by an ampersand (\u0026), and for each pair, the name is separated from the value by an equals (=) sign.\n  Line 12: This is how we specify the value of the Content-Type HTTP header. The value of this header is very important because it tells the server what type of data is being sent. The value application/x-www-form-urlencoded tells the server that the request contains form data encoded as URL parameters.\n   Wow, that required a lot more detail than I originally anticipated. I hope it helps you understand how the test client is used within our test cases to interact with the API. Next, create a new file test_auth_register.py in the tests folder, add the content below and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.auth_register API endpoint.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_api_tutorial.models.user import User from tests.util import EMAIL, register_user SUCCESS = \u0026#34;successfully registered\u0026#34; def test_auth_register(client, db): response = register_user(client) assert response.status_code == HTTPStatus.CREATED assert \u0026#34;status\u0026#34; in response.json and response.json[\u0026#34;status\u0026#34;] == \u0026#34;success\u0026#34; assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == SUCCESS assert \u0026#34;token_type\u0026#34; in response.json and response.json[\u0026#34;token_type\u0026#34;] == \u0026#34;bearer\u0026#34; assert \u0026#34;expires_in\u0026#34; in response.json and response.json[\u0026#34;expires_in\u0026#34;] == 5 assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] result = User.decode_access_token(access_token) assert result.success user_dict = result.value assert not user_dict[\u0026#34;admin\u0026#34;] user = User.find_by_public_id(user_dict[\u0026#34;public_id\u0026#34;]) assert user and user.email == EMAIL   In the previous post, I explained the meaning and purpose of the app and client test fixtures. In order to send and receive HTTP requests from our API, the test function must include client as a parameter.\nLet\u0026rsquo;s go through the test_auth_register function and explain what is being tested:\n  Line 5: The `register_user` function which we just dissected and documented is imported.\n  Line 7: This string value will occur multiple times in the test cases we created in this test file, but it will not occur in any other test cases, so we do not need to refactor it and move it to the `tests/util.py` file.\n  Line 10: test_auth_register is a test case, and client and db are test fixtures defined in conftest.py. The reason for invoking the client fixture is obvious \u0026mdash; we need it to test the API. However, the reason for invoking db is not so obvious since it isn't actually being called in the test function. This fixture initializes the database by creating tables for each database model class (the only model class at this point is User).\nIn this test case, we are sending a request to register a new user and expecting the request to succeed. This will only work if the database has been initialized and the site_user table exists in the database since the SQLAlchemy extension will attempt to execute a INSERT INTO site_user... SQL statement.\nBOTTOM LINE\u0026nbsp;\u0026nbsp;Invoking the db fixture is necessary for any test cases that add or modify database objects.\n  Line 11: We start off the test case by submitting the registration request with the default values. This is really the only action performed in this test case, the rest of the code just verifies the server response to the registration request.\n  Line 12: Next, we verify that the HTTP status code of the server response is 201 HTTPStatus.CREATED which indicates that a new user was created in the database.\n  Line 13-14: These two lines verify that the status and message attributes exist in the response JSON and that the values indicate that the user was successfully registered.\n  Line 15: This assert statement verifies that the token_type attribute exists in the response JSON and that the value is bearer.\n  Line 16: This assert statement verifies that the expires_in attribute exists in the response JSON and that the value is 5.\n  Line 17-18: Next, we verify that the access_token exists and retrieve the access_token.\n  Line 19-22: Next, we call User.decode_access_token and verify the operation was successful. Then, we retrieve the user_dict and verify that the token (for the user that we just registered) does not have administrator priveleges.\n  Line 23-24: The next thing we do is call User.find_by_public_id with the public_id value decoded from access_token. This verifies that the user we registered actually exists within the database. Using the object from the database, we verify that the email address for the user matches the value submitted in the original HTTP request.\n   test_auth_register verifies the \u0026ldquo;happy path\u0026rdquo; for the api.auth_register endpoint. We obviously need to test scenarios where a registration request is not successful, as well. Before we create the next test case, update test_auth_register.py to import the PASSWORD value from tests/util.py (Line 5) and define a new string value (Line 8):\n1 2 3 4 5 6 7 8  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.auth_register API endpoint.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_api_tutorial.models.user import User from tests.util import EMAIL, PASSWORD, register_user  SUCCESS = \u0026#34;successfully registered\u0026#34; EMAIL_ALREADY_EXISTS = f\u0026#34;{EMAIL} is already registered\u0026#34;   To verify the expected behavior when a registration request is sent for an email address that has already been registered, add the content below and save the file:\n28 29 30 31 32 33 34 35 36 37 38 39  def test_auth_register_email_already_registered(client, db): user = User(email=EMAIL, password=PASSWORD) db.session.add(user) db.session.commit() response = register_user(client) assert response.status_code == HTTPStatus.CONFLICT assert ( \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == EMAIL_ALREADY_EXISTS ) assert \u0026#34;token_type\u0026#34; not in response.json assert \u0026#34;expires_in\u0026#34; not in response.json assert \u0026#34;access_token\u0026#34; not in response.json     Line 29-32: The first thing we do in this test case is manually create a User instance and add it to the database. Then, we send the same registration request that was sent in the previous test case.\n  Line 33: Since a User already exists in the database with the same email address that is sent in the registration request, the response code 409 (HTTPStatus.CONFLICT) indicates that the request could not be completed, but the user might be able to resolve the source of the conflict and resubmit the request.\n  Line 34-37: Next, we verify that \"status\" and \"message\" attributes exist in the JSON object sent in the response body and the value for each indicates that the registration request was not successful.\n  Line 38-40: The final three lines verify that the \"token_type\", \"expires_in\" and \"access_token\" attributes are not present in the JSON object sent in the response body.\n   The last test case we will cover at this point is where the client submits an email address that is not in the correct format (the final version of the API will have more test cases). Let\u0026rsquo;s take a look at the actual response that is sent if the email value in a registration request is not a valid email (the example below uses httpie):\nflask-api-tutorial $ http -f :5000/api/v1/auth/register email=\"first last\" password=123456 POST /api/v1/auth/register HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Connection: keep-alive Content-Length: 32 Content-Type: application/x-www-form-urlencoded; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/2.0.0 email=first+last\u0026password=123456 HTTP/1.0 400 BAD REQUEST Access-Control-Allow-Origin: * Content-Length: 127 Content-Type: application/json Date: Fri, 02 Aug 2019 17:45:40 GMT Server: Werkzeug/0.16.1 Python/3.7.6 { \"errors\": { \"email\": \"first last is not a valid email\" }, \"message\": \"Input payload validation failed\" } You might notice that none of the code we wrote for the api.auth_register endpoint generated the response above. That is because this response was automatically generated by Flask-RESTx based on the auth_reqparser we configured in src/flask_api_tutorial/api/auth/dto.py.\n  The response above DOES NOT have an attribute named status, because Flask-RESTx generated the response rather than any of the code that was written for this tutorial.\n  Whenever a request is rejected because of one or more RequestParser arguments failed validation, the format of the response will always contain a message attribute equal to \u0026ldquo;Input payload validation failed\u0026rdquo; and a an errors attribute with the value being another embedded list. The embedded list contains an entry for each argument in the parser that failed validation, with the name of the argument as the attribute name and the value equal to a message describing the failure that occurred.\nThe information above should make writing the test case pretty easy. Since the error message for an invalid parameter will occur in nearly every set of test cases we create, we need to add it to tests/util.py (Line 6):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14  \u0026#34;\u0026#34;\u0026#34;Shared functions and constants for unit tests.\u0026#34;\u0026#34;\u0026#34; from flask import url_for EMAIL = \u0026#34;new_user@email.com\u0026#34; PASSWORD = \u0026#34;test1234\u0026#34; BAD_REQUEST = \u0026#34;Input payload validation failed\u0026#34;  def register_user(test_client, email=EMAIL, password=PASSWORD): return test_client.post( url_for(\u0026#34;api.auth_register\u0026#34;), data=f\u0026#34;email={email}\u0026amp;password={password}\u0026#34;, content_type=\u0026#34;application/x-www-form-urlencoded\u0026#34;, )   We need to import this value into test_auth_register.py (Line 5):\n1 2 3 4 5 6 7 8  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.auth_register API endpoint.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_api_tutorial.models.user import User from tests.util import EMAIL, PASSWORD, BAD_REQUEST, register_user  SUCCESS = \u0026#34;successfully registered\u0026#34; EMAIL_ALREADY_EXISTS = f\u0026#34;{EMAIL} is already registered\u0026#34;   Next, add the content below to test_auth_register.py and save the file:\n42 43 44 45 46 47 48 49 50 51 52 53  def test_auth_register_invalid_email(client): invalid_email = \u0026#34;first last\u0026#34; response = register_user(client, email=invalid_email) assert response.status_code == HTTPStatus.BAD_REQUEST assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == BAD_REQUEST assert \u0026#34;token_type\u0026#34; not in response.json assert \u0026#34;expires_in\u0026#34; not in response.json assert \u0026#34;access_token\u0026#34; not in response.json assert \u0026#34;errors\u0026#34; in response.json assert \u0026#34;password\u0026#34; not in response.json[\u0026#34;errors\u0026#34;] assert \u0026#34;email\u0026#34; in response.json[\u0026#34;errors\u0026#34;] assert response.json[\u0026#34;errors\u0026#34;][\u0026#34;email\u0026#34;] == f\u0026#34;{invalid_email} is not a valid email\u0026#34;   I don\u0026rsquo;t think there\u0026rsquo;s anything that needs to be explained since most of it is the same as the previous test case, and the difference in the response JSON was explained thoroughly.\nThere are quite a few more test cases that we need to create for the api.auth_register endpoint. I will not go through any more at this point, since you can find the full set in the github repo. Also, trying to define the necessary test coverage yourself is a very worthwhile exercise.\nYou should run tox to make sure the new test cases all pass and that nothing else broke because of the changes:\n(flask-api-tutorial) flask-api-tutorial $ tox GLOB sdist-make: /Users/aaronluna/Projects/flask-api-tutorial/setup.py py37 create: /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37 py37 installdeps: black, flake8, pydocstyle, pytest, pytest-black, pytest-clarity, pytest-dotenv, pytest-flake8, pytest-flask py37 inst: /Users/aaronluna/Projects/flask-api-tutorial/.tox/.tmp/package/1/flask-api-tutorial-0.1.zip py37 installed: alembic==1.4.0,aniso8601==8.0.0,appdirs==1.4.3,attrs==19.3.0,bcrypt==3.1.7,black==19.10b0,certifi==2019.11.28,cffi==1.14.0,chardet==3.0.4,Click==7.0,entrypoints==0.3,flake8==3.7.9,Flask==1.1.1,flask-api-tutorial==0.1,Flask-Bcrypt==0.7.1,Flask-Cors==3.0.8,Flask-Migrate==2.5.2,flask-restx==0.1.1,Flask-SQLAlchemy==2.4.1,idna==2.9,importlib-metadata==1.5.0,itsdangerous==1.1.0,Jinja2==2.11.1,jsonschema==3.2.0,Mako==1.1.1,MarkupSafe==1.1.1,mccabe==0.6.1,more-itertools==8.2.0,packaging==20.1,pathspec==0.7.0,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pycparser==2.19,pydocstyle==5.0.2,pyflakes==2.1.1,PyJWT==1.7.1,pyparsing==2.4.6,pyrsistent==0.15.7,pytest==5.3.5,pytest-black==0.3.8,pytest-clarity==0.3.0a0,pytest-dotenv==0.4.0,pytest-flake8==1.0.4,pytest-flask==0.15.1,python-dateutil==2.8.1,python-dotenv==0.12.0,python-editor==1.0.4,pytz==2019.3,regex==2020.2.20,requests==2.23.0,six==1.14.0,snowballstemmer==2.0.0,SQLAlchemy==1.3.13,termcolor==1.1.0,toml==0.10.0,typed-ast==1.4.1,urllib3==1.25.8,wcwidth==0.1.8,Werkzeug==0.16.1,zipp==3.0.0 py37 run-test-pre: PYTHONHASHSEED='1825844209' py37 run-test: commands[0] | pytest ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/bin/python cachedir: .tox/py37/.pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 52 items run.py::FLAKE8 PASSED [ 1%] run.py::BLACK PASSED [ 3%] setup.py::FLAKE8 PASSED [ 5%] setup.py::BLACK PASSED [ 7%] src/flask_api_tutorial/__init__.py::FLAKE8 PASSED [ 9%] src/flask_api_tutorial/__init__.py::BLACK PASSED [ 11%] src/flask_api_tutorial/config.py::FLAKE8 PASSED [ 13%] src/flask_api_tutorial/config.py::BLACK PASSED [ 15%] src/flask_api_tutorial/api/__init__.py::FLAKE8 PASSED [ 17%] src/flask_api_tutorial/api/__init__.py::BLACK PASSED [ 19%] src/flask_api_tutorial/api/auth/__init__.py::FLAKE8 PASSED [ 21%] src/flask_api_tutorial/api/auth/__init__.py::BLACK PASSED [ 23%] src/flask_api_tutorial/api/auth/business.py::FLAKE8 PASSED [ 25%] src/flask_api_tutorial/api/auth/business.py::BLACK PASSED [ 26%] src/flask_api_tutorial/api/auth/dto.py::FLAKE8 PASSED [ 28%] src/flask_api_tutorial/api/auth/dto.py::BLACK PASSED [ 30%] src/flask_api_tutorial/api/auth/endpoints.py::FLAKE8 PASSED [ 32%] src/flask_api_tutorial/api/auth/endpoints.py::BLACK PASSED [ 34%] src/flask_api_tutorial/api/widgets/__init__.py::FLAKE8 PASSED [ 36%] src/flask_api_tutorial/api/widgets/__init__.py::BLACK PASSED [ 38%] src/flask_api_tutorial/models/__init__.py::FLAKE8 PASSED [ 40%] src/flask_api_tutorial/models/__init__.py::BLACK PASSED [ 42%] src/flask_api_tutorial/models/user.py::FLAKE8 PASSED [ 44%] src/flask_api_tutorial/models/user.py::BLACK PASSED [ 46%] src/flask_api_tutorial/util/__init__.py::FLAKE8 PASSED [ 48%] src/flask_api_tutorial/util/__init__.py::BLACK PASSED [ 50%] src/flask_api_tutorial/util/datetime_util.py::FLAKE8 PASSED [ 51%] src/flask_api_tutorial/util/datetime_util.py::BLACK PASSED [ 53%] src/flask_api_tutorial/util/result.py::FLAKE8 PASSED [ 55%] src/flask_api_tutorial/util/result.py::BLACK PASSED [ 57%] tests/__init__.py::FLAKE8 PASSED [ 59%] tests/__init__.py::BLACK PASSED [ 61%] tests/conftest.py::FLAKE8 PASSED [ 63%] tests/conftest.py::BLACK PASSED [ 65%] tests/test_auth_register.py::FLAKE8 PASSED [ 67%] tests/test_auth_register.py::BLACK PASSED [ 69%] tests/test_auth_register.py::test_auth_register PASSED [ 71%] tests/test_auth_register.py::test_auth_register_email_already_registered PASSED [ 73%] tests/test_auth_register.py::test_auth_register_invalid_email PASSED [ 75%] tests/test_config.py::FLAKE8 PASSED [ 76%] tests/test_config.py::BLACK PASSED [ 78%] tests/test_config.py::test_config_development PASSED [ 80%] tests/test_config.py::test_config_testing PASSED [ 82%] tests/test_config.py::test_config_production PASSED [ 84%] tests/test_user.py::FLAKE8 PASSED [ 86%] tests/test_user.py::BLACK PASSED [ 88%] tests/test_user.py::test_encode_access_token PASSED [ 90%] tests/test_user.py::test_decode_access_token_success PASSED [ 92%] tests/test_user.py::test_decode_access_token_expired PASSED [ 94%] tests/test_user.py::test_decode_access_token_invalid PASSED [ 96%] tests/util.py::FLAKE8 PASSED [ 98%] tests/util.py::BLACK PASSED [100%] =================================================== warnings summary =================================================== src/flask_api_tutorial/api/auth/business.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/model.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, MutableMapping src/flask_api_tutorial/api/auth/business.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/api.py:28: DeprecationWarning: The import 'werkzeug.cached_property' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.utils import cached_property' instead. from werkzeug import cached_property src/flask_api_tutorial/api/auth/business.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/swagger.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, Hashable -- Docs: https://docs.pytest.org/en/latest/warnings.html =========================================== 52 passed, 3 warnings in 14.47s ============================================ _______________________________________________________ summary ________________________________________________________ py37: commands succeeded congratulations :) The warning that is generated from Flask-RESTx is a very minor issue with the manner in which one of their modules is importing a type from the standard library. This has no effect on the operation of the API and will be fixed very soon in an upcoming release. I will update this test result when it has been fixed.\nCheckpoint Once again, we only implemented a small number of features from the requirements list. It\u0026rsquo;s not too surprising if you realize we only created one of the four auth_ns endpoints in this section. I think we have fully satisfied one requirement: New users can register by providing an email address and password, and partially satisfied one more since the JWT is sent in the registration and login response: JWT is sent in access_token field of HTTP response after successful authentication with email/password.\nUser Management/JWT Authentication\nNew users can register by providing an email address and password\nExisting users can obtain a JWT by providing their email address and password\nJWT contains the following claims: time the token was issued, time the token expires, a value that identifies the user, and a flag that indicates if the user has administrator access\nJWT is sent in access_token field of HTTP response after successful authentication with email/password\nJWTs must expire after 1 hour (in production)\nJWT is sent by client in Authorization field of request header\nRequests must be rejected if JWT has been modified\nRequests must be rejected if JWT is expired\nIf user logs out, their JWT is immediately invalid/expired\nIf JWT is expired, user must re-authenticate with email/password to obtain a new JWT\n API Resource: Widget List\nAll users can retrieve a list of all widgets\nAll users can retrieve individual widgets by name\nUsers with administrator access can add new widgets to the database\nUsers with administrator access can edit existing widgets\nUsers with administrator access can delete widgets from the database\nThe widget model contains a \"name\" attribute which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nThe widget model contains a \"deadline\" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.\nURL and datetime values must be validated before a new widget is added to the database (and when an existing widget is updated).\nThe widget model contains a \"name\" field which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nWidget name must be validated before a new widget is added to the database (and when an existing widget is updated).\nIf input validation fails either when adding a new widget or editing an existing widget, the API response must include error messages indicating the name(s) of the fields that failed validation.\n  In the next section we will create the remaining three auth_ns endpoints, so the entire set of JWT authentication requirements should be complete at the next checkpoint. Questions/comments are appreciated!\n","href":"/series/flask-api-tutorial/part-3/","title":"How To: Create a Flask API with JWT-Based Authentication (Part 3)"},{"categories":"Flask ; Python ; Tutorial-Series","content":"Project Structure The chart below shows the folder structure for this section of the tutorial. In this post, we will work on all files marked as NEW CODE. Files that contain code from previous sections but will not be modified in this post are marked as NO CHANGES.\n. (project root folder) |- src | |- flask_api_tutorial | |- api | | |- auth | | | |- __init__.py | | | |- business.py | | | |- decorators.py | | | |- dto.py | | | |- endpoints.py | | | | | |- widgets | | | |- __init__.py | | | | | |- __init__.py | | |- exceptions.py | | | |- models | | |- __init__.py | | |- token_blacklist.py | | |- user.py | | | |- util | | |- __init__.py | | |- datetime_util.py | | |- result.py | | | |- __init__.py | |- config.py | |- tests | |- __init__.py | |- conftest.py | |- test_auth_login.py | |- test_auth_logout.py | |- test_auth_register.py | |- test_auth_user.py | |- test_config.py | |- test_user.py | |- util.py | |- .env |- .gitignore |- .pre-commit-config.yaml |- pyproject.toml |- pytest.ini |- README.md |- run.py |- setup.py |- tox.ini KEY: FOLDER NEW CODE NO CHANGES EMPTY FILE   api.auth_login Endpoint That last section ended up being a lot longer than I anticipated, but I don\u0026rsquo;t think the rest of the auth_ns endpoints will require the same amount of explanation since there\u0026rsquo;s no need to repeat the same information. Right off the bat, we do not need to create a RequestParser or API model since the api.auth_login endpoint can use the auth_reqparser to validate request data.\nWhy is this the case? The data required to register a new user or authenticate an existing user is the same: email address and password. With that out of the way, we can move on to defining the business logic needed to authenticate an existing user.\nprocess_login_request Function When a user sends a login request and their credentials are successfully validated, the server must return an HTTP response that includes an access token. As we saw when we implemented the registration process, any response that includes sensitive information (e.g., an access token) must satisfy all OAuth 2.0 requirements, which we thoroughly documented and implemented in Part 3. The implementation for the response to a successful login request will be nearly identical.\nIn observance of DRY, I decided to refactor the code that constructs the HTTP response out of the process_registration_request function (extracted to new method _create_auth_successful_response), in order to avoid repeating nearly the same code in process_login_request. I\u0026rsquo;ve provided the entire code for the updated version of src/flask_api_tutorial/api/auth/business.py below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56  \u0026#34;\u0026#34;\u0026#34;Business logic for /auth API endpoints.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask import current_app, jsonify from flask_restx import abort from flask_api_tutorial import db from flask_api_tutorial.models.user import User def process_registration_request(email, password): if User.find_by_email(email): abort(HTTPStatus.CONFLICT, f\u0026#34;{email} is already registered\u0026#34;, status=\u0026#34;fail\u0026#34;) new_user = User(email=email, password=password) db.session.add(new_user) db.session.commit() access_token = new_user.encode_access_token() return _create_auth_successful_response( token=access_token.decode(), status_code=HTTPStatus.CREATED, message=\u0026#34;successfully registered\u0026#34;, ) def process_login_request(email, password): user = User.find_by_email(email) if not user or not user.check_password(password): abort(HTTPStatus.UNAUTHORIZED, \u0026#34;email or password does not match\u0026#34;, status=\u0026#34;fail\u0026#34;) access_token = user.encode_access_token() return _create_auth_successful_response( token=access_token.decode(), status_code=HTTPStatus.OK, message=\u0026#34;successfully logged in\u0026#34;, ) def _create_auth_successful_response(token, status_code, message): response = jsonify( status=\u0026#34;success\u0026#34;, message=message, access_token=token, token_type=\u0026#34;bearer\u0026#34;, expires_in=_get_token_expire_time(), ) response.status_code = status_code response.headers[\u0026#34;Cache-Control\u0026#34;] = \u0026#34;no-store\u0026#34; response.headers[\u0026#34;Pragma\u0026#34;] = \u0026#34;no-cache\u0026#34; return response def _get_token_expire_time(): token_age_h = current_app.config.get(\u0026#34;TOKEN_EXPIRE_HOURS\u0026#34;) token_age_m = current_app.config.get(\u0026#34;TOKEN_EXPIRE_MINUTES\u0026#34;) expires_in_seconds = token_age_h * 3600 + token_age_m * 60 return expires_in_seconds if not current_app.config[\u0026#34;TESTING\u0026#34;] else 5   With that taken care of, let\u0026rsquo;s take a look at the process_login_request function:\n25 26 27 28 29 30 31 32 33 34  def process_login_request(email, password): user = User.find_by_email(email) if not user or not user.check_password(password): abort(HTTPStatus.UNAUTHORIZED, \u0026#34;email or password does not match\u0026#34;, status=\u0026#34;fail\u0026#34;) access_token = user.encode_access_token() return _create_auth_successful_response( token=access_token.decode(), status_code=HTTPStatus.OK, message=\u0026#34;successfully logged in\u0026#34;, )   The first thing we do in this function is call User.find_by_email with the email address provided by the user. If no user exists with this email address, the current request is aborted with a response including 401 HTTPStatus.UNAUTHORIZED.\nIf a user matching the provided email address was found in the database, we call check_password on the user instance, which verifies that the password provided by the user matches the password_hash value stored in the database. If the password does not match, the current request is aborted with a response including 401 HTTPStatus.UNAUTHORIZED.\nIf the password is verified, then the response is almost exactly the same as a successful response to a registration request — we create an access token for the user and include the token in the response. Also, we adhere to the requirements from RFC6749 which were fully explained earlier. The only difference is the status code (200 HTTPStatus.OK instead of 201 HTTPStatus.CREATED) and the message (\u0026ldquo;successfully logged in\u0026rdquo; instead of \u0026ldquo;successfully registered\u0026rdquo;).\nLoginUser Resource The API resource that processes login requests will be very similar to the RegisterUser resource. First, update the import statements in src/flask_api_tutorial/api/auth/endpoints.py to include the process_login_request function that we just created (Line 9):\n1 2 3 4 5 6 7 8 9 10  \u0026#34;\u0026#34;\u0026#34;API endpoint definitions for /auth namespace.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_restx import Namespace, Resource from flask_api_tutorial.api.auth.dto import auth_reqparser from flask_api_tutorial.api.auth.business import ( process_registration_request, process_login_request, )   Next, add the content below and save the file:\n32 33 34 35 36 37 38 39 40 41 42 43 44 45 46  @auth_ns.route(\u0026#34;/login\u0026#34;, endpoint=\u0026#34;auth_login\u0026#34;) class LoginUser(Resource): \u0026#34;\u0026#34;\u0026#34;Handles HTTP requests to URL: /api/v1/auth/login.\u0026#34;\u0026#34;\u0026#34; @auth_ns.expect(auth_reqparser) @auth_ns.response(int(HTTPStatus.OK), \u0026#34;Login succeeded.\u0026#34;) @auth_ns.response(int(HTTPStatus.UNAUTHORIZED), \u0026#34;email or password does not match\u0026#34;) @auth_ns.response(int(HTTPStatus.BAD_REQUEST), \u0026#34;Validation error.\u0026#34;) @auth_ns.response(int(HTTPStatus.INTERNAL_SERVER_ERROR), \u0026#34;Internal server error.\u0026#34;) def post(self): \u0026#34;\u0026#34;\u0026#34;Authenticate an existing user and return an access token.\u0026#34;\u0026#34;\u0026#34; request_data = auth_reqparser.parse_args() email = request_data.get(\u0026#34;email\u0026#34;) password = request_data.get(\u0026#34;password\u0026#34;) return process_login_request(email, password)   There are two minor differences in the implementation of the LoginUser resource and the RegisterUser resource:\n  Line 32: The @auth_ns.route decorator binds this resource to the /api/v1/auth/login URL route.\n  Line 37: The HTTP status codes for successfully registering a new user and successfully authenticating an existing user are 201 HTTPStatus.CREATED and 200 HTTPStatus.OK , respectively.\n   We can verify that the new route was correctly registered by running flask routes:\nflask-api-tutorial $ flask routes Endpoint Methods Rule ------------------- ------- -------------------------- api.auth_login POST /api/v1/auth/login api.auth_register POST /api/v1/auth/register api.doc GET /api/v1/ui api.root GET /api/v1/ api.specs GET /api/v1/swagger.json restplus_doc.static GET /swaggerui/\u0026lt;path:filename\u0026gt; static GET /static/\u0026lt;path:filename\u0026gt; The Swagger UI should also be updated to include the new API endpoint:\n Figure 1 - Swagger UI with /auth/login endpoint\n  Finally, let\u0026rsquo;s create unit tests to verify the login process is working correctly.\nUnit Tests: test_auth_login.py First, we need to create a function that sends a post request to the new endpoint we just created. Since this function will be used by nearly all of our test cases, we place it in the tests/util.py file:\n17 18 19 20 21 22  def login_user(test_client, email=EMAIL, password=PASSWORD): return test_client.post( url_for(\u0026#34;api.auth_login\u0026#34;), data=f\u0026#34;email={email}\u0026amp;password={password}\u0026#34;, content_type=\u0026#34;application/x-www-form-urlencoded\u0026#34;, )   Hopefully this looks familiar to you since it is nearly the same as the register_user function. Next, create a new file named test_auth_login.py in the tests folder, add the content below and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.auth_login API endpoint.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_api_tutorial.models.user import User from tests.util import EMAIL, register_user, login_user SUCCESS = \u0026#34;successfully logged in\u0026#34; UNAUTHORIZED = \u0026#34;email or password does not match\u0026#34; def test_login(client, db): register_user(client) response = login_user(client) assert response.status_code == HTTPStatus.OK assert \u0026#34;status\u0026#34; in response.json and response.json[\u0026#34;status\u0026#34;] == \u0026#34;success\u0026#34; assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == SUCCESS assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] result = User.decode_access_token(access_token) assert result.success token_payload = result.value assert not token_payload[\u0026#34;admin\u0026#34;] user = User.find_by_public_id(token_payload[\u0026#34;public_id\u0026#34;]) assert user and user.email == EMAIL def test_login_email_does_not_exist(client, db): response = login_user(client) assert response.status_code == HTTPStatus.UNAUTHORIZED assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == UNAUTHORIZED assert \u0026#34;access_token\u0026#34; not in response.json   Everything in this file should be simple to understand since it is so similar to the test set we just created. Run tox and make sure no failures occur.\nAccessing Protected Resources The two remaining API routes in the auth_ns namespace and all endpoints in the widget_ns namespace are protected resources, so called because any request sent by a client must include an access_token in JWT format in the request header’s Authorization field.\nIn Part 3, we discussed the required format for the server’s response to an authorization request (according to RFC6749 and RFC6750). The specification documents also define the required format for the server’s response to a request for a protected resource.\nThe server must validate the access_token, and if the token is valid/not expired and the user has the necessary access rights for the protected resource (e.g., administrator privileges), the request is considered successful. In this case, the format and content of the server’s response will depend on the resource and the HTTP method type requested by the client.\nHowever, if the token is invalid/expired or there is some other issue preventing the request from succeeding, the server’s response will be the same regardless of the resource:\n  If a client sends a request for a protected resource without an access_token, the server must reject the request and must not provide any explanation for why the request was rejected in the response header’s WWW-Authenticate field. The status code of the response must be 401 (HTTPStatus.UNAUTHORIZED)\nPer Section 3 of RFC6750:\nIf the protected resource request does not include authentication credentials or does not contain an access token that enables access to the protected resource, the resource server MUST include the HTTP \"WWW-Authenticate\" response header field ... If the request lacks any authentication information (e.g., the client was unaware that authentication is necessary or attempted using an unsupported authentication method), the resource server SHOULD NOT include an error code or other error information.\nFor example:\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"example\"\n RFC6750 also states that WWW-Authenticate: Bearer \"MUST be followed by one or more auth-param values\" and goes on to suggest that \"a realm attribute MAY be included to indicate the scope of protection\". Section 3.2.1 of RFC2167 (HTTP Authentication: Basic and Digest Access Authentication) defines the realm attribute:\nrealm\nA string to be displayed to users so they know which username and password to use. This string should contain at least the name of the host performing the authentication and might additionally indicate the collection of users who might have access. An example might be \"registered_users@gotham.news.com\".\n If a request is received without an access token in the request header, the realm attribute in the WWW-Authenticate field of the response header will communicate the access level necessary for the requested resource \u0026mdash; either registered_users@mydomain.com or admin_users@mydomain.com.\n  If the token is invalid/expired, the server must reject the request and provide an error and/or error_description in the response header’s WWW-Authenticate field explaining why the token is invalid. The status code of the response must be 401 (HTTPStatus.UNAUTHORIZED)\nPer section 3.1 of RFC6750:\nIf the protected resource request included an access token and failed authentication, the resource server SHOULD include the \"error\" attribute to provide the client with the reason why the access request was declined ... In addition, the resource server MAY include the \"error_description\" attribute to provide developers a human-readable explanation that is not meant to be displayed to end-users.\nFor example, ... in response to a protected resource request with an authentication attempt using an expired access token:\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"example\", error=\"invalid_token\", error_description=\"The access token expired\"\n  If a client sends a request for a protected resource and the token is valid/not expired, but it has been blacklisted, the server must reject the request and provide a message prompting the client to log in again. The status code of the response must be 401 (HTTPStatus.UNAUTHORIZED)\n  If a client sends a request to a protected resource that requires administrator privileges, and the token is valid/not expired but the user does not have administrator privileges, the server's response will be nearly the same as the response when the token is invalid/expired. The only difference is that the error code must be insufficient_scope, and the status code of the response must be 403 (HTTPStatus.FORBIDDEN)\nPer section 3.1 of RFC6750:\ninsufficient_scope The request requires higher privileges than provided by the access token. The resource server SHOULD respond with the HTTP 403 (Forbidden) status code and MAY include the \"scope\" attribute with the scope necessary to access the protected resource.\n    @token_required and @admin_token_required Decorators We can implement the responses required when a request for a protected resource must be rejected with a pair of function decorators. It is important to understand how these decorators are designed and how this design is driven by the need to obey the specifications from RFC6750.\n  Understanding how decorators work and how to create them can be a daunting topic. I recommend reading at least one of the following articles:\nPrimer on Python Decorators (Real Python) Python Decorator Tutorial with Example (Apcelent Tech Blog) How to make a chain of function decorators? (Stack Overflow)    Create a new file named decorators.py in src/flask_api_tutorial/api/auth and add the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50  \u0026#34;\u0026#34;\u0026#34;Decorators that decode and verify authorization tokens.\u0026#34;\u0026#34;\u0026#34; from functools import wraps from flask import request from flask_api_tutorial.api.exceptions import ApiUnauthorized, ApiForbidden from flask_api_tutorial.models.user import User def token_required(f): \u0026#34;\u0026#34;\u0026#34;Execute function if request contains valid access token.\u0026#34;\u0026#34;\u0026#34; @wraps(f) def decorated(*args, **kwargs): token_payload = _check_access_token(admin_only=False) for name, val in token_payload.items(): setattr(decorated, name, val) return f(*args, **kwargs) return decorated def admin_token_required(f): \u0026#34;\u0026#34;\u0026#34;Execute function if request contains valid access token AND user is admin.\u0026#34;\u0026#34;\u0026#34; @wraps(f) def decorated(*args, **kwargs): token_payload = _check_access_token(admin_only=True) if not token_payload[\u0026#34;admin\u0026#34;]: raise ApiForbidden() for name, val in token_payload.items(): setattr(decorated, name, val) return f(*args, **kwargs) return decorated def _check_access_token(admin_only): token = request.headers.get(\u0026#34;Authorization\u0026#34;) if not token: raise ApiUnauthorized(description=\u0026#34;Unauthorized\u0026#34;, admin_only=admin_only) result = User.decode_access_token(token) if result.failure: raise ApiUnauthorized( description=result.error, admin_only=admin_only, error=\u0026#34;invalid_token\u0026#34;, error_description=result.error, ) return result.value     Line 6: The ApiUnauthorized and ApiForbidden classes which are imported here have not been created yet, we will cover these next.\n  Lines 10, 23: This module exposes two decorators, @token_required and @admin_token_required. If access to a method/function needs to be restricted to users who have a valid access_token, simply decorate it with @token_required. If access needs to be restricted solely to users who have a valid access_token AND administrator privileges, decorate the method with @admin_token_required instead.\n  Lines 15, 28: The first thing both decorators do is call the _check_access_token function. This function returns a token_payload object if an access_token was sent in the request header AND the token was successfully decoded. If no access_token was sent or the token is invalid/expired, the current request is aborted.\n  Line 39: Within the _check_access_token function, request is the global flask.request object (imported in Line 4) which allows us to access the headers from the current request, among other things. For more info, check out the Flask docs.\n  Lines 40-41:  If no token is found in the header’s Authorization field, the current request is aborted to prevent access to the requested resource.\n  Line 42: If the request header does contain an access_token, we attempt to verify and decode it. Remember, if the token is invalid/expired, then the value of result.failure will be True and result.error will contain a string value that explains why the validation failed.\n  Lines 43-49: If the token is invalid/expired, we abort the current request. We will explain how the ApiUnauthorized class is used shortly, but you probably noticed that we are providing values for more attributes in this instance than we did when we aborted the request in Line 41. What is the difference?\nIn Line 41, we aborted the current request because the header’s Authorization field did not contain an access_token. Per the specification, the server’s response should not contain an error code or other error information when this is the case.\nHowever, in Line 44, the client did send a token in the request header, but it was invalid/expired. In this case, the server’s response should include information explaining the reason why the request was denied. This information is contained in result.error, and will be included in the server’s response since we are providing it to the ApiUnauthorized __init__ function.\n  Line 50: If the access_token was successfully validated and decoded, then the JWT payload (which is stored in result.value) is returned to the decorator function.\n  The token payload is a JSON object containing data identifying the user that the token was issued for, and other information. Click here to review the data that is included in the JWT payload as well as the processes used to encode and decode the token.\n    Lines 29-30: Back in the admin_token_required decorator, after successfully decoding the token payload, the first thing we do is check the value of token_payload[\"admin\"], which tells us if the user has administrator privileges. If this value is False, then the request is rejected by calling ApiForbidden.\n  Lines 16-17, 31-32: Both decorators pass the contents of token_payload to the decorated function in the same way \u0026mdash; by iterating over the dictionary items and (for each item) creating a new attribute on the decorated function (attribute name = dict item key, attribute value = dict item value). This allows the decorated function to access the user’s public_id, the access_token string value, etc.\nIf that explanation was confusing, it should make more sense after we apply the decorator to a function and step through the code, which will happen very shortly.\n  Lines 20, 35: After the token has been decoded and passed to the wrapped function, the wrapped function is executed and returned to the code that originally called it.\n   ApiUnauthorized and ApiForbidden Exceptions Next, let’s take a look at the custom HTTP exceptions that are used in our decorator functions. Create a new file named exceptions.py in src/flask_api_tutorial/api and add the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53  \u0026#34;\u0026#34;\u0026#34;Custom HTTPException classes that extend werkzeug.exceptions.\u0026#34;\u0026#34;\u0026#34; from werkzeug.exceptions import Unauthorized, Forbidden _REALM_REGULAR_USERS = \u0026#34;registered_users@mydomain.com\u0026#34; _REALM_ADMIN_USERS = \u0026#34;admin_users@mydomain.com\u0026#34; class ApiUnauthorized(Unauthorized): \u0026#34;\u0026#34;\u0026#34;Raise status code 401 with customizable WWW-Authenticate header.\u0026#34;\u0026#34;\u0026#34; def __init__( self, description=\u0026#34;Unauthorized\u0026#34;, admin_only=False, error=None, error_description=None, ): self.description = description self.www_auth_value = self.__get_www_auth_value( admin_only, error, error_description ) Unauthorized.__init__( self, description=description, response=None, www_authenticate=None ) def get_headers(self, environ): return [(\u0026#34;Content-Type\u0026#34;, \u0026#34;text/html\u0026#34;), (\u0026#34;WWW-Authenticate\u0026#34;, self.www_auth_value)] def __get_www_auth_value(self, admin_only, error, error_description): realm = _REALM_ADMIN_USERS if admin_only else _REALM_REGULAR_USERS www_auth_value = f\u0026#39;Bearer realm=\u0026#34;{realm}\u0026#34;\u0026#39; if error: www_auth_value += f\u0026#39;, error=\u0026#34;{error}\u0026#34;\u0026#39; if error_description: www_auth_value += f\u0026#39;, error_description=\u0026#34;{error_description}\u0026#34;\u0026#39; return www_auth_value class ApiForbidden(Forbidden): \u0026#34;\u0026#34;\u0026#34;Raise status code 403 with WWW-Authenticate header.\u0026#34;\u0026#34;\u0026#34; description = \u0026#34;You are not an administrator\u0026#34; def get_headers(self, environ): return [ (\u0026#34;Content-Type\u0026#34;, \u0026#34;text/html\u0026#34;), ( \u0026#34;WWW-Authenticate\u0026#34;, f\u0026#39;Bearer realm=\u0026#34;{_REALM_ADMIN_USERS}\u0026#34;, \u0026#39; \u0026#39;error=\u0026#34;insufficient_scope\u0026#34;, \u0026#39; \u0026#39;error_description=\u0026#34;You are not an administrator\u0026#34;\u0026#39;, ), ]   There are a few things to point out regarding the code above:\n  Line 2: The werkzeug.exceptions module contains Python exceptions that you can raise from application code to trigger standard non-200 responses. Both werkzeug.exceptions.Unauthorized and werkzeug.exceptions.Forbidden are subclasses of werkzeug.exceptions.HTTPException with the value of code defned as 401 and 403, respectively.\nBoth functions inherit the get_headers method from the HTTPException base class, which returns a hard-coded value of [('Content-Type', 'text/html')]. We know that any response for a protected resource that is not successful MUST include the HTTP WWW-Authenticate response header field, per RFC6750.\nWe can add the WWW-Authenticate field to the response header by subclassing the werkzeug Unauthorized and Forbidden classes and overwriting the get_headers method.\n  Lines 4-5: These are hard-coded values that are used for the value of the realm attribute. If the resource that was requested requires administrator privileges, the value will be \"admin_users@mydomain.com\", if the resource only requires a valid token, the value will be \"registered_users@mydomain.com\".\nThe values provided in this project are completely generic examples that should be customized to reflect your domain before deploying this API in the real-world.\n  Lines 8-36: Refer back to the decorators.py file to see how we are raising the ApiUnauthorized exception. When the token fails validation, the reason for the failure is provided to both the description and error_description parameters, and these are used to construct the value of the WWW-Authenticate header field.\n  Line 39-53: The ApiForbidden implementation is much simpler since the value of the WWW-Authenticate header field does not change and can therefore be hardcoded to the value you see here.\n   Let\u0026rsquo;s see how we can apply the @token_required decorator to the remaining API endpoints since both rely upon the access_token being sent with the HTTP request.\napi.auth_user Endpoint The purpose of this endpoint is to verify that the access_token issued to the logged-in user is currently valid, and if so, to return a representation of the current user containing the email, public_id, admin and registered_on attributes from the User model class.\nThe way we implement this endpoint will demonstrate a few new concepts:\n How to create an API model and use it to serialize a database object to JSON, in order to send the database object in a HTTP response. How to use the @token_required decorator How to send requests from Swagger UI and httpie that include the access_token in the request header.  user_model API Model The first thing we need to do is create an API model for the User class. In src/flask_api_tutorial/api/auth/dto.py, update the import statements to include the Model class from flask_restx and the String and Boolean classes from the flask_restx.fields module (Lines 2-3):\n1 2 3 4 5  \u0026#34;\u0026#34;\u0026#34;Parsers and serializers for /auth API endpoints.\u0026#34;\u0026#34;\u0026#34; from flask_restx import Model from flask_restx.fields import String, Boolean from flask_restx.inputs import email from flask_restx.reqparse import RequestParser   Next, add the content below and save the file:\n16 17 18 19 20 21 22 23 24 25  user_model = Model( \u0026#34;User\u0026#34;, { \u0026#34;email\u0026#34;: String, \u0026#34;public_id\u0026#34;: String, \u0026#34;admin\u0026#34;: Boolean, \u0026#34;registered_on\u0026#34;: String(attribute=\u0026#34;registered_on_str\u0026#34;), \u0026#34;token_expires_in\u0026#34;: String, }, )   \u0026quot;User\u0026quot; is the name of the API Model, and this value will be used to identify the JSON object in the Swagger UI page. Please read the Flask-RESTx documentation for detailed examples of creating API models. Basically, an API model is a dictionary where the keys are the names of attributes on the object that we need to serialize, and the values are a class from the fields module that formats the value of the attibute on the object to ensure that it can be safely included in the HTTP response.\nAny other attributes of the object are considered private and will not be included in the JSON. If the name of the attribute on the object is different than the name that you wish to use in the JSON, specify the name of the attribute on the object using the attribute parameter, which is what we are doing for registered_on in the code above (Line 22).\nYou may have noticed that the User class has attributes named registered_on and registered_on_str, a datetime and str value, respectively. registered_on_str is the datetime value formatted as a concise, easy-to-read string. We want to use the string version in our JSON, but would rather use registered_on as the name, rather than registered_on_str. Specifying attribute=\u0026quot;registered_on_str\u0026quot; in the fields.String constructor achieves this.\nThe last attribute in the User API model is named token_expires_in, but the User db model does not contain an attribute that matches this in any way. So why would we define an API model with a value that doesn\u0026rsquo;t exist on the object being modeled?\nObjects in Python are quite permissive due to the language\u0026rsquo;s dynamic nature. For example, you are free to create new attributes of your choosing on any object, and we will modify the User object to include an attribute named token_expires_in before we marshal the object to JSON and send it to the client. The value for this attribute will be a formatted string representing the timedelta until the token expires, which is available from the payload of the user\u0026rsquo;s access token after validating that the token is valid.\nThe Flask-RESTx docs contain a full list of the classes available in the `fields' module as well as instructions for creating a custom formatter.\nget_logged_in_user Function Our next task is to create the business logic for the api.auth_user endpoint. The first thing we need to do is verify that the access token included in the request is valid. Sounds like a job for the @token_required decorator!\nOpen src/flask_api_tutorial/api/auth/business.py and update the import statements to include the @token_required decorator and a few helper functions from the datetime_util module (Line 8 and Lines 10-13).\n1 2 3 4 5 6 7 8 9 10 11 12 13  \u0026#34;\u0026#34;\u0026#34;Business logic for /auth API endpoints.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask import current_app, jsonify from flask_restx import abort from flask_api_tutorial import db from flask_api_tutorial.api.auth.decorators import token_required from flask_api_tutorial.models.user import User from flask_api_tutorial.util.datetime_util import ( remaining_fromtimestamp, format_timespan_digits, )   Then, add the decorated function:\n42 43 44 45 46 47 48  @token_required def get_logged_in_user(): public_id = get_logged_in_user.public_id user = User.find_by_public_id(public_id) expires_at = get_logged_in_user.expires_at user.token_expires_in = format_timespan_digits(remaining_fromtimestamp(expires_at)) return user   Thanks to the @token_required decorator, if the request header does not contain an access token or the access token was sent but is invalid/expired, the get_logged_in_user function is never actually executed (unless a valid token is found, the current request is aborted before calling the wrapped function). So what\u0026rsquo;s the deal with Line 56 above? It isn\u0026rsquo;t very obvious, so let\u0026rsquo;s break it down line-by-line. First, look at the code for @token_required:\n10 11 12 13 14 15 16 17 18 19 20  def token_required(f): \u0026#34;\u0026#34;\u0026#34;Allow access to the wrapped function if the request contains a valid access token.\u0026#34;\u0026#34;\u0026#34; @wraps(f) def decorated(*args, **kwargs): token_payload = _check_access_token(admin_only=False) for name, val in token_payload.items(): setattr(decorated, name, val) return f(*args, **kwargs) return decorated   After successfully decoding the token in Line 15, the function iterates over the items in token_payload. Each item\u0026rsquo;s name and value are then used to call setattr on decorated (remember, decorated is the wrapped function, in this case get_logged_in_user).\nThe code below shows the value of all local variables while iterating over the dictionary items within token_payload:\n# f = \u0026lt;function get_logged_in_user at 0x1080a1ef0\u0026gt; @wraps(f) def decorated(*args, **kwargs): # decorated = \u0026lt;function get_logged_in_user at 0x1080a1f80\u0026gt; token_payload = _check_access_token(admin_only=False) # token_payload = {\u0026#39;public_id\u0026#39;: \u0026#39;77e8570c-5432-4a5a-9a5d-71915604a0db\u0026#39;, \u0026#39;admin\u0026#39;: False, \u0026#39;token\u0026#39;: \u0026#39;eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE...HNlfQ.wNvDzYMiY70wWm4xmt698G1WYgPPup6OH1NrcsfaXy0\u0026#39;, \u0026#39;expires_at\u0026#39;: 1565075425} # len(token_payload) = 4 for name, val in token_payload.items(): # name = \u0026#39;public_id\u0026#39; # val = \u0026#39;77e8570c-5432-4a5a-9a5d-71915604a0db\u0026#39; setattr(decorated, name, val) # decorated.public_id = \u0026#39;77e8570c-5432-4a5a-9a5d-71915604a0db\u0026#39; for name, val in token_payload.items(): # name = \u0026#39;admin\u0026#39; # val = False setattr(decorated, name, val) # decorated.admin = False for name, val in token_payload.items(): # name = \u0026#39;token\u0026#39; # val = \u0026#39;eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE...HNlfQ.wNvDzYMiY70wWm4xmt698G1WYgPPup6OH1NrcsfaXy0\u0026#39; setattr(decorated, name, val) # decorated.token = \u0026#39;eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE...HNlfQ.wNvDzYMiY70wWm4xmt698G1WYgPPup6OH1NrcsfaXy0\u0026#39; for name, val in token_payload.items(): # name = \u0026#39;expires_at\u0026#39; # val = 1565075425 setattr(decorated, name, val) # decorated.expires_at = 1565075425 return f(*args, **kwargs) The important thing to grasp is that we are creating attributes on the get_logged_in_user function for each item in token_payload. I\u0026rsquo;ve isolated the lines from above that only report the value of the decorated object (which is the get_logged_in_user function):\n# decorated = \u0026lt;function get_logged_in_user at 0x1080a1f80\u0026gt; # decorated.public_id = \u0026#39;77e8570c-5432-4a5a-9a5d-71915604a0db # decorated.admin = False # decorated.token = \u0026#39;eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE...HNlfQ.wNvDzYMiY70wWm4xmt698G1WYgPPup6OH1NrcsfaXy0\u0026#39; # decorated.expires_at = 1565075425 Let\u0026rsquo;s again look at the code for get_logged_in_user:\n54 55 56 57 58 59 60  @token_required def get_logged_in_user(): public_id = get_logged_in_user.public_id user = User.find_by_public_id(public_id) expires_at = get_logged_in_user.expires_at user.token_expires_in = format_timespan_digits(remaining_fromtimestamp(expires_at)) return user     Line 56: When this line is executed, get_logged_in_user.public_id contains the value of token_payload[\"public_id\"], which was decoded from the access_token sent in the request header.\n  Line 57: Retrieve the User object from the database that matches the public_id decoded from the token.\n  Lines 58: get_logged_in_user.expires_at contains the value of token_payload['expires_at'], which is a timestamp stored as an integer. This value determines when the token is considered to be expired.\n  Lines 59: We can use the remaining_fromtimestamp function to calculate the time remaining until a token expires. Since this function returns a timespan object (this is a custom namedtuple defined in the datetime_util module), we format it as a string value using format_timespan_digits.\nFinally, we store the formatted string in a new attribute named token_expires_in, which we defined as a field in the user_model API model that we created in the flask_api_tutorial.api.auth.dto module.\n   Whew! That was a lot of detail for a simple function. The next step is to define the concrete Resource class for the api.auth_user endpoint.\nGetUser Resource Next, open src/flask_api_tutorial/api/auth/endpoints.py and update the import statements to include the user_model we created in the flask_api_tutorial.api.auth.dto module (Line 6) and the get_logged_in_user function we created in flask_api_tutorial.api.auth.business (Line 10). We also need to register user_model with the auth_ns namespace (Line 14):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14  \u0026#34;\u0026#34;\u0026#34;API endpoint definitions for /auth namespace.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_restx import Namespace, Resource from flask_api_tutorial.api.auth.dto import auth_reqparser, user_model from flask_api_tutorial.api.auth.business import ( process_registration_request, process_login_request, get_logged_in_user, ) auth_ns = Namespace(name=\u0026#34;auth\u0026#34;, validate=True) auth_ns.models[user_model.name] = user_model   Next, add the content below and save the file (this is the same file, src/flask_api_tutorial/api/auth/endpoints.py):\n51 52 53 54 55 56 57 58 59 60 61 62  @auth_ns.route(\u0026#34;/user\u0026#34;, endpoint=\u0026#34;auth_user\u0026#34;) class GetUser(Resource): \u0026#34;\u0026#34;\u0026#34;Handles HTTP requests to URL: /api/v1/auth/user.\u0026#34;\u0026#34;\u0026#34; @auth_ns.doc(security=\u0026#34;Bearer\u0026#34;) @auth_ns.response(int(HTTPStatus.OK), \u0026#34;Token is currently valid.\u0026#34;, user_model) @auth_ns.response(int(HTTPStatus.BAD_REQUEST), \u0026#34;Validation error.\u0026#34;) @auth_ns.response(int(HTTPStatus.UNAUTHORIZED), \u0026#34;Token is invalid or expired.\u0026#34;) @auth_ns.marshal_with(user_model) def get(self): \u0026#34;\u0026#34;\u0026#34;Validate access token and return user info.\u0026#34;\u0026#34;\u0026#34; return get_logged_in_user()   There are a few new concepts to note:\n  Line 55: When we configured the api object, we specified that the Bearer authentication scheme would be used. In order to mark an HTTP method as a protected resource that requires authorization, we decorate the HTTP method with @auth_ns.doc and set the value of the security parameter to \"Bearer\".\n  Line 56: The @auth_ns.response decorator can be configured with an API model as an optional third argument. This has no effect on the resource's behavior, but the API model is displayed on the Swagger UI page as an example response body for requests that produce a status code 200 HTTPStatus.OK.\n  Line 59: The @auth_ns.marshal_with decorator is how we tell Flask-RESTx to filter the data returned from this method against the provided API model (user_model), and validate the data against the set of fields configured in the API model.\n  Line 62: Remember, get_logged_in_user returns a User object. Without the @marshal_with decorator, this would produce a server error since we are not returning an HTTP response object as expected. The @marshal_with decorator creates the JSON using the user_model and assigns status code 200 HTTPStatus.OK before returning the response.\n   We can verify that the new route was correctly registered by running flask routes:\nflask-api-tutorial $ flask routes Endpoint Methods Rule ------------------- ------- -------------------------- api.auth_login POST /api/v1/auth/login api.auth_register POST /api/v1/auth/register api.auth_user GET /api/v1/auth/user api.doc GET /api/v1/ui api.root GET /api/v1/ api.specs GET /api/v1/swagger.json restplus_doc.static GET /swaggerui/\u0026lt;path:filename\u0026gt; static GET /static/\u0026lt;path:filename\u0026gt; By now, you should know what\u0026rsquo;s next: unit tests for the api.auth_user endpoint.\nUnit Tests: test_auth_user.py As with the two previous API endpoints, before we write any test cases we need to update the tests/util.py file with any error/success message string values and functions that will be re-used across the test set. Add WWW_AUTH_NO_TOKEN on Line 7:\n1 2 3 4 5 6 7  \u0026#34;\u0026#34;\u0026#34;Shared functions and constants for unit tests.\u0026#34;\u0026#34;\u0026#34; from flask import url_for EMAIL = \u0026#34;new_user@email.com\u0026#34; PASSWORD = \u0026#34;test1234\u0026#34; BAD_REQUEST = \u0026#34;Input payload validation failed\u0026#34; WWW_AUTH_NO_TOKEN = \u0026#39;Bearer realm=\u0026#34;registered_users@mydomain.com\u0026#34;\u0026#39;   Then, create a function to send a GET request to the api.auth_user endpoint. Open the tests/util.py file and add the function below:\n26 27 28 29  def get_user(test_client, access_token): return test_client.get( url_for(\u0026#34;api.auth_user\u0026#34;), headers={\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {access_token}\u0026#34;} )   Up to this point, we have used the test client to only send POST requests, however the api.auth_user endpoint only responds to GET requests. To accomplish this, we simply use the test client\u0026rsquo;s get method rather than the post method.\nThe api.auth_user endpoint requires a valid access token to be included in the request header\u0026rsquo;s Authorization field. We have already used dict objects to construct response headers, and we can also use a dict as the value of the headers argument. The access_token is provided as a parameter to the get_user function, and the format of the Authorization field is specified in Section 2.1 of RFC6750: \u0026ldquo;Bearer\u0026rdquo; plus a single whitespace followed by the access token in url-safe base64 encoding.\nNext, create a new file named test_auth_user.py in the tests folder, add the content below and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.auth_user API endpoint.\u0026#34;\u0026#34;\u0026#34; import time from http import HTTPStatus from flask import url_for from tests.util import EMAIL, register_user, login_user, get_user def test_auth_user(client, db): register_user(client) response = login_user(client) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = get_user(client, access_token) assert response.status_code == HTTPStatus.OK assert \u0026#34;email\u0026#34; in response.json and response.json[\u0026#34;email\u0026#34;] == EMAIL assert \u0026#34;admin\u0026#34; in response.json and not response.json[\u0026#34;admin\u0026#34;]   test_auth_user is the \u0026ldquo;happy path\u0026rdquo; for the api.auth_user endpoint, so we start by registering a new user and logging in. The response from the login request contains an access token, so we retrieve it and send it with the request to get the logged-in user info.\nAfter verifying that the response from the api.auth_user endpoint indicates the request was successful, we check that the response includes the correct user info. Since the response includes the user email and the admin flag, we verify that the email matches the email address we used to register the user and the admin flag should be False.\nI\u0026rsquo;m sure by now you\u0026rsquo;re getting used to the pattern I use to write test cases. Since we created the test case for the happy path, it\u0026rsquo;s time to think about all the ways we could send a request that does not succeed. Since api.auth_user is a protected resource, we should get an error if we send a GET request without an access token in the request header.\nFirst, update the import statements to include WWW_AUTH_NO_TOKEN from tests.util (Line 6). This is the value we expect to receive in the WWW-Authenticate header field if a request is sent to a protected resource without an access token.\n1 2 3 4 5 6  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.auth_user API endpoint.\u0026#34;\u0026#34;\u0026#34; import time from http import HTTPStatus from flask import url_for from tests.util import EMAIL, WWW_AUTH_NO_TOKEN, register_user, login_user, get_user   Copy the test case below and add it to test_auth_user.py:\n20 21 22 23 24 25  def test_auth_user_no_token(client, db): response = client.get(url_for(\u0026#34;api.auth_user\u0026#34;)) assert response.status_code == HTTPStatus.UNAUTHORIZED assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == \u0026#34;Unauthorized\u0026#34; assert \u0026#34;WWW-Authenticate\u0026#34; in response.headers assert response.headers[\u0026#34;WWW-Authenticate\u0026#34;] == WWW_AUTH_NO_TOKEN   There are a few things to note about this test case:\n  Line 21: The first thing we do is send a GET request using the test client to the api.auth_user endpoint without any headers.\n  Line 22: The expected response code when a request is sent to a protected resource without an access token is 401 HTTPStatus.UNAUTHORIZED.\n  Lines 23-24: These two lines verify that the status and message attributes exist in the response JSON and that the values indicate that the request did not succeed because the requested resource requires authorization which was not included in the request.\n  Lines 25: We previously explained the different values that the WWW-Authenticate header must include based on whether or not the request was successfully authorized, so please refer back to the Decorators section if you are unclear why this is the expected value.\nWhen a request for a protected resource does not include an access token, WWW-Authenticate must only include the realm attribute and must not contain any error information.\n   Let\u0026rsquo;s do one more. In Part 2, when we created test cases for the User class we used the time.sleep method to cause an access token to expire. If we send a request to the api.auth_user endpoint with an expired token, we should receive an error indicating that the token is expired.\nFirst, add the highlighted string values to test_auth_user.py after the import statements (Lines 8-13):\n1 2 3 4 5 6 7 8 9 10 11 12 13  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.auth_user API endpoint.\u0026#34;\u0026#34;\u0026#34; import time from http import HTTPStatus from flask import url_for from tests.util import EMAIL, WWW_AUTH_NO_TOKEN, register_user, login_user, get_user TOKEN_EXPIRED = \u0026#34;Access token expired. Please log in again.\u0026#34; WWW_AUTH_EXPIRED_TOKEN = ( f\u0026#34;{WWW_AUTH_NO_TOKEN}, \u0026#34; \u0026#39;error=\u0026#34;invalid_token\u0026#34;, \u0026#39; f\u0026#39;error_description=\u0026#34;{TOKEN_EXPIRED}\u0026#34;\u0026#39; )   Then, add the content below:\n35 36 37 38 39 40 41 42 43 44 45  def test_auth_user_expired_token(client, db): register_user(client) response = login_user(client) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] time.sleep(6) response = get_user(client, access_token) assert response.status_code == HTTPStatus.UNAUTHORIZED assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == TOKEN_EXPIRED assert \u0026#34;WWW-Authenticate\u0026#34; in response.headers assert response.headers[\u0026#34;WWW-Authenticate\u0026#34;] == WWW_AUTH_EXPIRED_TOKEN   As always, please note the following:\n  Lines 36-39: The first four lines of this test case are the same as the happy path test case test_auth_user. We register a new user, login and retrieve the access_token from the sucessful login response.\n  Lines 40: We suspend execution of the test case for six seconds to ensure the access token is expired.\n  Line 42: The expected response code when a request is sent to a protected resource with an expired access token is 401 HTTPStatus.UNAUTHORIZED.\n  Lines 43-44: These two lines verify that the status and message attributes exist in the response JSON and that the values indicate that the request did not succeed because the access token sent by the client is expired.\n  Lines 45: We verify that the WWW-Authenticate header contains the realm attribute as well as the error attribute and the error_description attribute.\n   How To: Request a Protected Resource We just demonstrated how to make a request for a protected resource in code (using the Flask test client). But, for ad hoc testing it would be nice to be able to send requests and include an access token using Swagger UI or a command-line tool like httpie. Let\u0026rsquo;s go through the process for both tools.\nSwagger UI Did you check out the Swagger UI page after implementing the /auth/user endpoint? Fire up the development server with flask run and navigate to http://localhost:5000/api/v1/ui:\n Figure 2 - Swagger UI with /auth/user endpoint\n  You may have already seen this and wondered, why is the GET /auth/user component the only one with a lock icon (), or more accurately, a unlocked lock icon? And does it have anything to do with that button labeled Authorize that also has a lock icon?\nThe two are in fact related. The lock icon indicates that the API endpoint requires authorization, and the GET /auth/user component is the only one with a lock icon because the GetUser.get method is the only resource method that we decorated with @doc(security=\u0026quot;Bearer\u0026quot;). \u0026ldquo;Bearer\u0026rdquo; is the name of the security object that we created in src/flask_api_tutorial/api/__init__.py and provided to the Api constructor, which causes the Authorize button to appear.\n  The unlocked lock icon indicates two things: the API method requires authorization, and the access token is currently not being sent in the request header. After clicking the Authorize button and entering the access token, all unlocked icons will become locked icons. When the icon is locked, this indicates that the access token will be sent in the header of any request.\n    There's one more thing to note about the Swagger UI page, the User model is shown at the bottom of the page (also on the /auth/user component under Responses). Any API model that you register with the API or an API namespcace will be rendered in this location (we registered user_model with the auth_ns namespace in src/flask_api_tutorial/api/auth/endpoints.py).\n  Let\u0026rsquo;s see what happens if we attempt to send a request to /auth/user as the component is currently configured. First, expand the component by clicking anywhere on the blue bar, click Try it out, then click Execute:\n Figure 3 - Endpoint requires authorization (Swagger UI)\n  We already knew that this would be the response since we created a test case (test_auth_user_no_token) to verify that sending a request to the api.auth_user endpoint without an access token would not succeed. So how do we get an access token and how do we send it in the request header with Swagger UI?\nGetting an access token is easy, just register a new user OR login with an existing user and the response will include a token in the access_token field. Copy the access token from the Response body text box:\n Figure 4 - Retrieve access token from response body (Swagger UI)\n  Next, click the Authorize button above the API routes (1). A dialog box will appear titled Avaialable authorizations. Paste the access token that was copied from the Response body text box into the Value text box in the dialog (2). Click Authorize (3):\n Figure 5 - Configure Swagger UI access token\n  After clicking Authorize, the button text changes to Logout, and the Value text box is replaced by a label of asterisk characters (see below). Click Close to dismiss the dialog and return to the Swagger UI:\n Figure 6 - Access token configuration complete\n  Notice that with the access token successfully configured, the lock icons have changed from being unlocked () to locked ():\n Figure 7 - Authorization required icons are locked after configuring access token\n  Let\u0026rsquo;s send a request to the api.auth_user endpoint again, now that the access token will be sent in the request header:\n Figure 8 - Request for /auth/user successful (Swagger UI)\n  As you can see, the access token is sent in the Authorization field of the request header, as required by the specification doc for Bearer Token Authentication (RFC6750).\nWith the development environment configuration settings, all access tokens expire fifteen minutes after being issued. If a request is sent with an expired access token, both the response body and header should contain error messages explaining why the request was not succesful:\n Figure 9 - Request failed (Token expired)\n  Similarly, try changing any part of an access token (even just a single character) and updating the value in the Available authorizations dialog box. The request will be rejected and the response will contain a different error message than the message for an expired token or for not sending an access token at all:\n Figure 10 - Request failed (Invalid token)\n  That\u0026rsquo;s all you need to do to automatically include the access token when a request is made to a protected resource. Let\u0026rsquo;s figure out how to do the same with httpie.\nhttpie Rather than using screenshots to illustrate the process of requesting a protected resoure with httpie, I will reproduce the text of the CLI commands and output from httpie.\nIf you enter a URL without a domain, for example :5000/api/v1/auth/user, the URL is automatically expanded to http://localhost:5000/api/v1/auth/user. With that in mind, the command below is a GET request to the api.auth_user endpont that does not include an access token (this is the same request/response pictured in Figure 3):\nflask-api-tutorial $ http :5000/api/v1/auth/user GET /api/v1/auth/user HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Connection: keep-alive Host: localhost:5000 User-Agent: HTTPie/2.0.0 HTTP/1.0 401 UNAUTHORIZED Access-Control-Allow-Origin: * Content-Length: 53 Content-Type: application/json Date: Sat, 29 Feb 2020 08:02:40 GMT Server: Werkzeug/0.16.1 Python/3.7.6 WWW-Authenticate: Bearer realm=\"registered_users@mydomain.com\" { \"message\": \"Unauthorized\" }   The command above is assumed to be a GET request since that is the default used by httpie if the only argument in the command is a URL.\n  We need to obtain an access token, so let\u0026rsquo;s login and retrieve the token generated by the server from the response (this is the same request/response pictured in Figure 4):\nflask-api-tutorial $ http -f :5000/api/v1/auth/login email=user@test.com password=123456 POST /api/v1/auth/login HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Connection: keep-alive Content-Length: 37 Content-Type: application/x-www-form-urlencoded; charset=utf-8 Host: localhost:5000 User-Agent: HTTPie/2.0.0 email=user%40test.com\u0026password=123456 HTTP/1.0 200 OK Cache-Control: no-store Access-Control-Allow-Origin: * Content-Length: 345 Content-Type: application/json Date: Sat, 29 Feb 2020 08:06:39 GMT Pragma: no-cache Server: Werkzeug/0.16.1 Python/3.7.6 { \"access_token\": \"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1NjQ4NzUzMjksImlhdCI6MTU2NDg3NDQyOSwic3ViIjoiNzdiNGYzYjctNzg2NC00ZmM0LWE4MzQtZjJhNjQ5OWYxNzJhIiwiYWRtaW4iOmZhbHNlfQ.LBYrCr5-8FqCKIF_1WEpk8ake235cB9hZNL01oQjPvw\", \"expires_in\": 900, \"message\": \"successfully logged in\", \"status\": \"success\", \"token_type\": \"bearer\" }   The -f option is short for --form and tells httpie that this command is submitting a form. When this option is used, POST is used as the method type, the data fields are serialized as URL parameters and the Content-Type is set to application/x-www-form-urlencoded; charset=utf-8.\n  Now that we have an access token, the only thing we need to do is send it in the Authorization field of the request header. httpie makes this really simple, any arguments in the form {name}:{value} will be added as headers (this is the same request/response pictured in Figure 8):\nflask-api-tutorial $ http :5000/api/v1/auth/user Authorization:\"Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1NjQ4NzUzMjksImlhdCI6MTU2NDg3NDQyOSwic3ViIjoiNzdiNGYzYjctNzg2NC00ZmM0LWE4MzQtZjJhNjQ5OWYxNzJhIiwiYWRtaW4iOmZhbHNlfQ.LBYrCr5-8FqCKIF_1WEpk8ake235cB9hZNL01oQjPvw\" GET /api/v1/auth/user HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1NjQ4NzUzMjksImlhdCI6MTU2NDg3NDQyOSwic3ViIjoiNzdiNGYzYjctNzg2NC00ZmM0LWE4MzQtZjJhNjQ5OWYxNzJhIiwiYWRtaW4iOmZhbHNlfQ.LBYrCr5-8FqCKIF_1WEpk8ake235cB9hZNL01oQjPvw Connection: keep-alive Host: localhost:5000 User-Agent: HTTPie/2.0.0 HTTP/1.0 200 OK Access-Control-Allow-Origin: * Content-Length: 159 Content-Type: application/json Date: Sat, 29 Feb 2020 08:23:46 GMT Server: Werkzeug/0.16.1 Python/3.7.6 { \"admin\": false, \"email\": \"user@test.com\", \"public_id\": \"77b4f3b7-7864-4fc4-a834-f2a6499f172a\", \"registered_on\": \"02/28/20 03:38:53 PM UTC-08:00\", \"token_expires_in\": \"00:14:42\" } api.auth_logout Endpoint The \u0026ldquo;logout\u0026rdquo; process for the API is really simple since we don\u0026rsquo;t actually implement any session handling. The api.auth_logout endpoint will use the @token_required decorator, just like the api.auth_user endpoint. Therefore, if a request is received without an access token or with an invalid/expired token, it will be aborted without executing any of the business logic we define for the logout process.\nOne of the requirements states: If user logs out, their JWT is immediately invalid/expired. In order to satisfy this requirement, when a user logs out and their access token is NOT invalid/expired, we must add the token to a blacklist and ensure that any subsequent request for a protected resource that includes the token is unsuccessful.\nEven though access tokens are typically configured to expire less than a day after being issued, the blacklist should be persistent (i.e., stored in a database, NOT in RAM). We can create a database table to store blacklisted tokens, which is what we will do next.\nBlacklistedToken DB Model Create a new file token_blacklist.py in src/flask_api_tutorial/models and add the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28  \u0026#34;\u0026#34;\u0026#34;Class definition for BlacklistedToken.\u0026#34;\u0026#34;\u0026#34; from datetime import timezone from flask_api_tutorial import db from flask_api_tutorial.util.datetime_util import utc_now, dtaware_fromtimestamp class BlacklistedToken(db.Model): \u0026#34;\u0026#34;\u0026#34;BlacklistedToken Model for storing JWT tokens.\u0026#34;\u0026#34;\u0026#34; __tablename__ = \u0026#34;token_blacklist\u0026#34; id = db.Column(db.Integer, primary_key=True, autoincrement=True) token = db.Column(db.String(500), unique=True, nullable=False) blacklisted_on = db.Column(db.DateTime, default=utc_now) expires_at = db.Column(db.DateTime, nullable=False) def __init__(self, token, expires_at): self.token = token self.expires_at = dtaware_fromtimestamp(expires_at, use_tz=timezone.utc) def __repr__(self): return f\u0026#34;\u0026lt;BlacklistToken token={self.token}\u0026gt;\u0026#34; @classmethod def check_blacklist(cls, token): exists = cls.query.filter_by(token=token).first() return True if exists else False   The BlacklistedToken class is pretty simple, but please note the following:\n  Line 14: token is the string value of the access token.\n  Line 15: Notice that we are capturing the current time with utc_now, which is a function from the flask_api_tutorial.util.datetime_util module. What is the difference between using this function which returns a timezone-aware datetime object and datetime.utcnow which returns a naive datetime? Consider the REPL commands below:\n from flask_api_tutorial.util.datetime_util import utc_now  from datetime import datetime  utc_now() datetime.datetime(2019, 8, 8, 9, 52, 4, tzinfo=datetime.timezone.utc)  datetime.utcnow() datetime.datetime(2019, 8, 8, 9, 52, 6, 793105) Working with datetime objects can be a source of insidious bugs that are very difficult to diagnose. For a thorough explanation of best practices that will prevent such issues, read this post by Travis Mick. The TL;DR version boils down to two guidelines:\nAlways ensure that your code produces and handles datetime objects that are timezone-aware. Always ensure that the datetime objects produced and utilized by your code are localized to the UTC timezone when written to the database (i.e., tzinfo=datetime.timezone.utc).  A simple example of why you should always use timezone \"aware\" datetime values is given below. When we create a BlacklistedToken object, the required expires_at parameter is a UNIX timestamp (i.e., an integer value) which can be converted to a datetime object using the fromtimestamp method:\n expires_at = 1565257955  datetime.fromtimestamp(expires_at).astimezone(timezone.utc) datetime.datetime(2019, 8, 8, 9, 52, 35, tzinfo=datetime.timezone.utc)  datetime.fromtimestamp(expires_at) datetime.datetime(2019, 8, 8, 2, 52, 35)  exit() Python assumes that the datetime value produced from datetime.fromtimestamp(expires_at) should be converted to the local time zone (in my case, PST or -700 UTC at the time this was written). You will quickly encounter bugs that are extremely difficult to diagnose unless you ensure that you are always dealing with \"aware\" datetime objects in the same timezone.\n  Lines 25-28: Whenever we need to check a user's access token, we will call BlacklistedToken.check_blacklist which will return a bool value based on whether or not the token has been added to the blacklist.\n   In order to register the BlacklistedToken model with the Flask application, we need to make a couple changes to run.py in the project root folder:\n1 2 3 4 5 6 7 8 9 10 11 12 13  \u0026#34;\u0026#34;\u0026#34;Flask CLI/Application entry point.\u0026#34;\u0026#34;\u0026#34; import os from flask_api_tutorial import create_app, db from flask_api_tutorial.models.token_blacklist import BlacklistedToken from flask_api_tutorial.models.user import User app = create_app(os.getenv(\u0026#34;FLASK_ENV\u0026#34;, \u0026#34;development\u0026#34;)) @app.shell_context_processor def shell(): return {\u0026#34;db\u0026#34;: db, \u0026#34;User\u0026#34;: User, \u0026#34;BlacklistedToken\u0026#34;: BlacklistedToken}     Line 5: We need to import the new model class as shown here.\n  Line 13: We should add the new model class here to make it available in the flask shell without needing to import it explicitly.\n   Finally, we need to create a new migration script and upgrade the database to create the actual token_blacklist table. We already went through this in Part 2 when we created the User model class, but let\u0026rsquo;s demonstrate the process once more.\nFirst, run flask db migrate and add a message explaining the changes that will be made by running this migration:\n(flask-api-tutorial) flask-api-tutorial $ flask db migrate --message \"add BlacklistedToken model\" INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.autogenerate.compare] Detected added table 'token_blacklist' Generating /Users/aaronluna/Projects/flask-api-tutorial/migrations/versions/079d26d45cc9_add_blacklistedtoken_model.py ... done Next, run flask db upgrade to run the migration script and add the new table to the database:\n(flask-api-tutorial) flask-api-tutorial $ flask db upgrade INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade eb6c2faa0708 - 079d26d45cc9, add BlacklistedToken model With the BlacklistedToken class fully implemented, we have everything we need to create the business logic for the api.auth_logout endpoint.\nprocess_logout_request Function After receiving a logout request containing a valid, unexpired token, the server then proceeds to create a BlasklistedToken object, add it to the database and commit the changes. Then, the server sends an HTTP response indicating that the logout request succeeded.\nThe function that performs this process will be defined insrc/flask_api_tutorial/api/auth/business.py. Open that file and update the import statements to include the BlacklistedToken class: (Line 9)\n1 2 3 4 5 6 7 8 9 10 11 12 13 14  \u0026#34;\u0026#34;\u0026#34;Business logic for /auth API endpoints.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask import current_app, jsonify from flask_restx import abort from flask_api_tutorial import db from flask_api_tutorial.api.auth.decorators import token_required from flask_api_tutorial.models.token_blacklist import BlacklistedToken from flask_api_tutorial.models.user import User from flask_api_tutorial.util.datetime_util import ( remaining_fromtimestamp, format_timespan_digits, )   Next, add the process_logout_request function and save the file:\n52 53 54 55 56 57 58 59 60  @token_required def process_logout_request(): access_token = process_logout_request.token expires_at = process_logout_request.expires_at blacklisted_token = BlacklistedToken(access_token, expires_at) db.session.add(blacklisted_token) db.session.commit() response_dict = dict(status=\u0026#34;success\u0026#34;, message=\u0026#34;successfully logged out\u0026#34;) return response_dict, HTTPStatus.OK   As explained in the Decorators section of this post, the access_token and expires_at attributes on process_logout_request come from decoding the access token\u0026rsquo;s payload, which takes place whenever the @token_required decorator is used (these values are required to create the BlacklistedToken object). Then, a HTTP response indicating the operation succeeded is sent to the client.\nUpdate decode_access_token Method There\u0026rsquo;s one more process we need to update in order to make the blacklist fully functional. Currently, when verifying an access token, it is only rejected by the server if the token is invalid or expired. Now, the server must also check if the token has been blacklisted before processing the client\u0026rsquo;s request.\nOpen src/flask_api_tutorial/models/user.py and update the import statements to include the BlacklistedToken class: (Line 10)\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17  \u0026#34;\u0026#34;\u0026#34;Class definition for User model.\u0026#34;\u0026#34;\u0026#34; from datetime import datetime, timedelta, timezone from uuid import uuid4 import jwt from flask import current_app from sqlalchemy.ext.hybrid import hybrid_property from flask_api_tutorial import db, bcrypt from flask_api_tutorial.models.token_blacklist import BlacklistedToken from flask_api_tutorial.util.datetime_util import ( utc_now, get_local_utcoffset, make_tzaware, localized_dt_string, ) from flask_api_tutorial.util.result import Result   We need to modify the decode_access_token method to reurn a Result object indicating the token has been blacklisted if that is the case. Add Lines 76-78 and save the changes:\n68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94  @staticmethod def decode_access_token(access_token): if isinstance(access_token, bytes): access_token = access_token.decode(\u0026#34;ascii\u0026#34;) if access_token.startswith(\u0026#34;Bearer \u0026#34;): split = access_token.split(\u0026#34;Bearer\u0026#34;) access_token = split[1].strip() try: key = current_app.config.get(\u0026#34;SECRET_KEY\u0026#34;) payload = jwt.decode(access_token, key, algorithms=[\u0026#34;HS256\u0026#34;]) except jwt.ExpiredSignatureError: error = \u0026#34;Access token expired. Please log in again.\u0026#34; return Result.Fail(error) except jwt.InvalidTokenError: error = \u0026#34;Invalid token. Please log in again.\u0026#34; return Result.Fail(error) if BlacklistedToken.check_blacklist(access_token): error = \u0026#34;Token blacklisted. Please log in again.\u0026#34; return Result.Fail(error)  token_payload = dict( public_id=payload[\u0026#34;sub\u0026#34;], admin=payload[\u0026#34;admin\u0026#34;], token=access_token, expires_at=payload[\u0026#34;exp\u0026#34;], ) return Result.Ok(token_payload)   With that out of the way, we can create the concrete Resource class for the api.auth_logout endpoint.\nLogoutUser Resource Open src/flask_api_tutorial/api/auth/endpoints.py and update the import statements to include the process_logout_request function we just created (Line 11):\n1 2 3 4 5 6 7 8 9 10 11 12  \u0026#34;\u0026#34;\u0026#34;API endpoint definitions for /auth namespace.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_restx import Namespace, Resource from flask_api_tutorial.api.auth.dto import auth_reqparser, user_model from flask_api_tutorial.api.auth.business import ( process_registration_request, process_login_request, get_logged_in_user, process_logout_request, )   Next, add the content below and save the file:\n66 67 68 69 70 71 72 73 74 75 76 77  @auth_ns.route(\u0026#34;/logout\u0026#34;, endpoint=\u0026#34;auth_logout\u0026#34;) class LogoutUser(Resource): \u0026#34;\u0026#34;\u0026#34;Handles HTTP requests to URL: /auth/logout.\u0026#34;\u0026#34;\u0026#34; @auth_ns.doc(security=\u0026#34;Bearer\u0026#34;) @auth_ns.response(int(HTTPStatus.OK), \u0026#34;Log out succeeded, token is no longer valid.\u0026#34;) @auth_ns.response(int(HTTPStatus.BAD_REQUEST), \u0026#34;Validation error.\u0026#34;) @auth_ns.response(int(HTTPStatus.UNAUTHORIZED), \u0026#34;Token is invalid or expired.\u0026#34;) @auth_ns.response(int(HTTPStatus.INTERNAL_SERVER_ERROR), \u0026#34;Internal server error.\u0026#34;) def post(self): \u0026#34;\u0026#34;\u0026#34;Add token to blacklist, deauthenticating the current user.\u0026#34;\u0026#34;\u0026#34; return process_logout_request()   This should all look very familiar, the only difference between the LogoutUser and GetUser is the HTTP method name (post vs get). Also, GetUser returned a JSON object which we defined in user_model and LogoutUser returns a regular HTTP response.\nUnit Tests: test_auth_logout.py Before we create any test cases, we need a way to send a POST request to the api.auth_logout endpoint. Open tests/util.py and add the logout_user function:\n32 33 34 35  def logout_user(test_client, access_token): return test_client.post( url_for(\u0026#34;api.auth_logout\u0026#34;), headers={\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {access_token}\u0026#34;} )   The \u0026ldquo;happy path\u0026rdquo; test case shown below simply registers a new user, logs in and then logs out. Create a new file /test/test_auth_logout.py and add the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.auth_logout API endpoint.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_api_tutorial.models.token_blacklist import BlacklistedToken from tests.util import register_user, login_user, logout_user SUCCESS = \u0026#34;successfully logged out\u0026#34; def test_logout(client, db): register_user(client) response = login_user(client) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] blacklist = BlacklistedToken.query.all() assert len(blacklist) == 0 response = logout_user(client, access_token) assert response.status_code == HTTPStatus.OK assert \u0026#34;status\u0026#34; in response.json and response.json[\u0026#34;status\u0026#34;] == \u0026#34;success\u0026#34; assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == SUCCESS blacklist = BlacklistedToken.query.all() assert len(blacklist) == 1 assert access_token == blacklist[0].token   There are a few things in this test case that we are seeing for the fist time, please note:\n  Line 15-16: After retrieving the access_token, we verify that the blacklist is currently empty (i.e., no tokens have been blacklisted at this point).\n  Line 17-20: We submit a request to the api.auth_logout endpoint and verify that the response indicates that the request succeeded.\n  Line 21-22: If the logout request succeeded, the token must have been added to the blacklist. The first way we verify this is by checking that the blacklist now contains one token.\n  Line 23: Finally, we verify that the blacklisted token is the same access_token that was submitted with the logout request.\n   We should definitely ensure that any requests for a protected resource using a blacklisted token does not succeed, and that the response indicates that the reason the request failed is due to the token being blacklisted.\nFirst, update the import statements to include the WWW_AUTH_NO_TOKEN string from tests.util (Line 5), and then add the highlighted lines to test_auth_logout.py\n1 2 3 4 5 6 7 8 9 10 11 12 13  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.auth_logout API endpoint.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_api_tutorial.models.token_blacklist import BlacklistedToken from tests.util import WWW_AUTH_NO_TOKEN, register_user, login_user, logout_user  SUCCESS = \u0026#34;successfully logged out\u0026#34; TOKEN_BLACKLISTED = \u0026#34;Token blacklisted. Please log in again.\u0026#34; WWW_AUTH_BLACKLISTED_TOKEN = ( f\u0026#34;{WWW_AUTH_NO_TOKEN}, \u0026#34; \u0026#39;error=\u0026#34;invalid_token\u0026#34;, \u0026#39; f\u0026#39;error_description=\u0026#34;{TOKEN_BLACKLISTED}\u0026#34;\u0026#39; )   This is the scenario contained in test_logout_token_blacklisted, below. Add the function to test_auth_logout.py:\n32 33 34 35 36 37 38 39 40 41 42 43  def test_logout_token_blacklisted(client, db): register_user(client) response = login_user(client) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = logout_user(client, access_token) assert response.status_code == HTTPStatus.OK response = logout_user(client, access_token) assert response.status_code == HTTPStatus.UNAUTHORIZED assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == TOKEN_BLACKLISTED assert \u0026#34;WWW-Authenticate\u0026#34; in response.headers assert response.headers[\u0026#34;WWW-Authenticate\u0026#34;] == WWW_AUTH_BLACKLISTED_TOKEN      Line 33-38: We begin by performing the same actions as the previous test case, registering a new user, logging in and finally logging out.\n  Line 39-44: The second time we call POST /api/v1/auth/logout with the same access_token, the status code of the HTTP response is 401 HTTPStatus.UNAUTHORIZED. This is the expected behavior since the token has not been tampered with, has not yet expired BUT has been added to the token_blacklist.\n   There are plenty of necessary test cases that are missing from the current set. You should try to identify as many different scenarios as you can think of and create test cases. You should compare your set to the final version in the github repository.\nYou should run tox to make sure the new test case passes and that nothing else broke because of the changes:\n(flask-api-tutorial) flask-api-tutorial $ tox GLOB sdist-make: /Users/aaronluna/Projects/flask-api-tutorial/setup.py py37 create: /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37 py37 installdeps: black, flake8, pydocstyle, pytest, pytest-black, pytest-clarity, pytest-dotenv, pytest-flake8, pytest-flask py37 inst: /Users/aaronluna/Projects/flask-api-tutorial/.tox/.tmp/package/1/flask-api-tutorial-0.1.zip py37 installed: alembic==1.4.0,aniso8601==8.0.0,appdirs==1.4.3,attrs==19.3.0,bcrypt==3.1.7,black==19.10b0,certifi==2019.11.28,cffi==1.14.0,chardet==3.0.4,Click==7.0,entrypoints==0.3,flake8==3.7.9,Flask==1.1.1,flask-api-tutorial==0.1,Flask-Bcrypt==0.7.1,Flask-Cors==3.0.8,Flask-Migrate==2.5.2,flask-restx==0.1.1,Flask-SQLAlchemy==2.4.1,idna==2.9,importlib-metadata==1.5.0,itsdangerous==1.1.0,Jinja2==2.11.1,jsonschema==3.2.0,Mako==1.1.1,MarkupSafe==1.1.1,mccabe==0.6.1,more-itertools==8.2.0,packaging==20.1,pathspec==0.7.0,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pycparser==2.19,pydocstyle==5.0.2,pyflakes==2.1.1,PyJWT==1.7.1,pyparsing==2.4.6,pyrsistent==0.15.7,pytest==5.3.5,pytest-black==0.3.8,pytest-clarity==0.3.0a0,pytest-dotenv==0.4.0,pytest-flake8==1.0.4,pytest-flask==0.15.1,python-dateutil==2.8.1,python-dotenv==0.12.0,python-editor==1.0.4,pytz==2019.3,regex==2020.2.20,requests==2.23.0,six==1.14.0,snowballstemmer==2.0.0,SQLAlchemy==1.3.13,termcolor==1.1.0,toml==0.10.0,typed-ast==1.4.1,urllib3==1.25.8,wcwidth==0.1.8,Werkzeug==0.16.1,zipp==3.0.0 py37 run-test-pre: PYTHONHASHSEED='3206075645' py37 run-test: commands[0] | pytest ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/bin/python cachedir: .tox/py37/.pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 71 items run.py::FLAKE8 PASSED [ 1%] run.py::BLACK PASSED [ 2%] setup.py::FLAKE8 PASSED [ 4%] setup.py::BLACK PASSED [ 5%] src/flask_api_tutorial/__init__.py::FLAKE8 PASSED [ 7%] src/flask_api_tutorial/__init__.py::BLACK PASSED [ 8%] src/flask_api_tutorial/config.py::FLAKE8 PASSED [ 9%] src/flask_api_tutorial/config.py::BLACK PASSED [ 11%] src/flask_api_tutorial/api/__init__.py::FLAKE8 PASSED [ 12%] src/flask_api_tutorial/api/__init__.py::BLACK PASSED [ 14%] src/flask_api_tutorial/api/exceptions.py::FLAKE8 PASSED [ 15%] src/flask_api_tutorial/api/exceptions.py::BLACK PASSED [ 16%] src/flask_api_tutorial/api/auth/__init__.py::FLAKE8 PASSED [ 18%] src/flask_api_tutorial/api/auth/__init__.py::BLACK PASSED [ 19%] src/flask_api_tutorial/api/auth/business.py::FLAKE8 PASSED [ 21%] src/flask_api_tutorial/api/auth/business.py::BLACK PASSED [ 22%] src/flask_api_tutorial/api/auth/decorators.py::FLAKE8 PASSED [ 23%] src/flask_api_tutorial/api/auth/decorators.py::BLACK PASSED [ 25%] src/flask_api_tutorial/api/auth/dto.py::FLAKE8 PASSED [ 26%] src/flask_api_tutorial/api/auth/dto.py::BLACK PASSED [ 28%] src/flask_api_tutorial/api/auth/endpoints.py::FLAKE8 PASSED [ 29%] src/flask_api_tutorial/api/auth/endpoints.py::BLACK PASSED [ 30%] src/flask_api_tutorial/api/widgets/__init__.py::FLAKE8 PASSED [ 32%] src/flask_api_tutorial/api/widgets/__init__.py::BLACK PASSED [ 33%] src/flask_api_tutorial/models/__init__.py::FLAKE8 PASSED [ 35%] src/flask_api_tutorial/models/__init__.py::BLACK PASSED [ 36%] src/flask_api_tutorial/models/token_blacklist.py::FLAKE8 PASSED [ 38%] src/flask_api_tutorial/models/token_blacklist.py::BLACK PASSED [ 39%] src/flask_api_tutorial/models/user.py::FLAKE8 PASSED [ 40%] src/flask_api_tutorial/models/user.py::BLACK PASSED [ 42%] src/flask_api_tutorial/util/__init__.py::FLAKE8 PASSED [ 43%] src/flask_api_tutorial/util/__init__.py::BLACK PASSED [ 45%] src/flask_api_tutorial/util/datetime_util.py::FLAKE8 PASSED [ 46%] src/flask_api_tutorial/util/datetime_util.py::BLACK PASSED [ 47%] src/flask_api_tutorial/util/result.py::FLAKE8 PASSED [ 49%] src/flask_api_tutorial/util/result.py::BLACK PASSED [ 50%] tests/__init__.py::FLAKE8 PASSED [ 52%] tests/__init__.py::BLACK PASSED [ 53%] tests/conftest.py::FLAKE8 PASSED [ 54%] tests/conftest.py::BLACK PASSED [ 56%] tests/test_auth_login.py::FLAKE8 PASSED [ 57%] tests/test_auth_login.py::BLACK PASSED [ 59%] tests/test_auth_login.py::test_login PASSED [ 60%] tests/test_auth_login.py::test_login_email_does_not_exist PASSED [ 61%] tests/test_auth_logout.py::FLAKE8 PASSED [ 63%] tests/test_auth_logout.py::BLACK PASSED [ 64%] tests/test_auth_logout.py::test_logout PASSED [ 66%] tests/test_auth_logout.py::test_logout_token_blacklisted PASSED [ 67%] tests/test_auth_register.py::FLAKE8 PASSED [ 69%] tests/test_auth_register.py::BLACK PASSED [ 70%] tests/test_auth_register.py::test_auth_register PASSED [ 71%] tests/test_auth_register.py::test_auth_register_email_already_registered PASSED [ 73%] tests/test_auth_register.py::test_auth_register_invalid_email PASSED [ 74%] tests/test_auth_user.py::FLAKE8 PASSED [ 76%] tests/test_auth_user.py::BLACK PASSED [ 77%] tests/test_auth_user.py::test_auth_user PASSED [ 78%] tests/test_auth_user.py::test_auth_user_no_token PASSED [ 80%] tests/test_auth_user.py::test_auth_user_expired_token PASSED [ 81%] tests/test_config.py::FLAKE8 PASSED [ 83%] tests/test_config.py::BLACK PASSED [ 84%] tests/test_config.py::test_config_development PASSED [ 85%] tests/test_config.py::test_config_testing PASSED [ 87%] tests/test_config.py::test_config_production PASSED [ 88%] tests/test_user.py::FLAKE8 PASSED [ 90%] tests/test_user.py::BLACK PASSED [ 91%] tests/test_user.py::test_encode_access_token PASSED [ 92%] tests/test_user.py::test_decode_access_token_success PASSED [ 94%] tests/test_user.py::test_decode_access_token_expired PASSED [ 95%] tests/test_user.py::test_decode_access_token_invalid PASSED [ 97%] tests/util.py::FLAKE8 PASSED [ 98%] tests/util.py::BLACK PASSED [100%] =================================================== warnings summary =================================================== src/flask_api_tutorial/api/exceptions.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/model.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, MutableMapping src/flask_api_tutorial/api/exceptions.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/api.py:28: DeprecationWarning: The import 'werkzeug.cached_property' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.utils import cached_property' instead. from werkzeug import cached_property src/flask_api_tutorial/api/exceptions.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/swagger.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, Hashable -- Docs: https://docs.pytest.org/en/latest/warnings.html =========================================== 71 passed, 3 warnings in 29.32s ============================================ _______________________________________________________ summary ________________________________________________________ py37: commands succeeded congratulations :) Checkpoint As promised, we have implemented all of the required features in the User Management/JWT Authentication section of the requirements list:\nUser Management/JWT Authentication\nNew users can register by providing an email address and password\nExisting users can obtain a JWT by providing their email address and password\nJWT contains the following claims: time the token was issued, time the token expires, a value that identifies the user, and a flag that indicates if the user has administrator access\nJWT is sent in access_token field of HTTP response after successful authentication with email/password\nJWTs must expire after 1 hour (in production)\nJWT is sent by client in Authorization field of request header\nRequests must be rejected if JWT has been modified\nRequests must be rejected if JWT is expired\nIf user logs out, their JWT is immediately invalid/expired\nIf JWT is expired, user must re-authenticate with email/password to obtain a new JWT\n API Resource: Widget List\nAll users can retrieve a list of all widgets\nAll users can retrieve individual widgets by name\nUsers with administrator access can add new widgets to the database\nUsers with administrator access can edit existing widgets\nUsers with administrator access can delete widgets from the database\nThe widget model contains a \"name\" attribute which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nThe widget model contains a \"deadline\" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.\nURL and datetime values must be validated before a new widget is added to the database (and when an existing widget is updated).\nThe widget model contains a \"name\" field which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nWidget name must be validated before a new widget is added to the database (and when an existing widget is updated).\nIf input validation fails either when adding a new widget or editing an existing widget, the API response must include error messages indicating the name(s) of the fields that failed validation.\n  Creating the Widget API will build upon the concepts introduced while the Auth API was being implemented. We will use most of these concepts and encounter many new ones in order to build the Widget API. If you have any questions/feedback, please leave a comment!\n","href":"/series/flask-api-tutorial/part-4/","title":"How To: Create a Flask API with JWT-Based Authentication (Part 4)"},{"categories":"Flask ; Python ; Tutorial-Series","content":"Project Structure The chart below shows the folder structure for this section of the tutorial. In this post, we will work on all files marked as NEW CODE. Files that contain code from previous sections but will not be modified in this post are marked as NO CHANGES.\n. (project root folder) |- src | |- flask_api_tutorial | |- api | | |- auth | | | |- __init__.py | | | |- business.py | | | |- decorators.py | | | |- dto.py | | | |- endpoints.py | | | | | |- widgets | | | |- __init__.py | | | |- business.py | | | |- dto.py | | | |- endpoints.py | | | | | |- __init__.py | | | |- models | | |- __init__.py | | |- token_blacklist.py | | |- user.py | | |- widget.py | | | |- util | | |- __init__.py | | |- datetime_util.py | | |- result.py | | | |- __init__.py | |- config.py | |- tests | |- __init__.py | |- conftest.py | |- test_auth_login.py | |- test_auth_logout.py | |- test_auth_register.py | |- test_auth_user.py | |- test_config.py | |- test_user.py | |- util.py | |- .env |- .gitignore |- .pre-commit-config.yaml |- pyproject.toml |- pytest.ini |- README.md |- run.py |- setup.py |- tox.ini KEY: FOLDER NEW CODE NO CHANGES EMPTY FILE   Introduction In the previous section, we created four API endpoints to perform basic user registration and authentication functions. However, these API endpoints are not designed as RESTful resources (I explained my reasoning for this choice in Part 3).\nIn this section of the tutorial, we will create a resource that is REST-like (REST-faux? REST-adjacent?). I am deliberately not describing it as RESTful, because designing a truly RESTful system is HARD. Check out this blog post from Roy Fielding and the discussion in the comments to get an idea of what I mean.\nThe only features of this resource that I am willing to state are 100% bona fide REST-compliant are:\n The naming convention of the resource and associated endpoints. The HTTP methods supported by each endpoint that enables clients to perform CRUD operations. Through the use of pagination and navigational links included in JSON sent by the server, clients can interact with the API purely through hypertext (i.e., clients NEVER need to manually construct URLs to interact with the API).  The resource we will create is a collection of widgets. I decided to model something generic rather than the cliche \u0026ldquo;to-do list\u0026rdquo; project that you encounter in every introductory programming tutorial. I feel safe assuming that you are not reading this because you have a burning desire to create the next, great API-driven to-do list.\nThe main purpose of this section is to learn more advanced techniques for request parsing and response marshalling. The Widget model will contain attributes that require creating custom input types for parsing request data. The Widget model also contains hybrid properties that will require rendering various data types in JSON. Whatever project you have in mind, the techniques demonstrated with the Widget model and associated RequestParser and API model instances can easily be adapted to any object in your domain.\nwidget_ns Endpoints The proper way to name resources is one of the (many) hotly debated topics regarding RESTful web services. I recommend taking the time to read the articles below to understand the current, accepted best practices:\n Resource Naming (RestApiTutorial.com)   REST API Design - Resource Modeling (ThoughtWorks)   RESTful API Design. Best Practices in a Nutshell (Philipp Hauer's Blog)   The accepted best practice for naming resources is to use plural nouns when constructing a URI for a resource (e.g. /widgets instead of /widget). Another widely accepted standard is to create two endpoints per resource — one for operations that apply to the entire collection (e.g., /api/v1/widgets) and one for operations that apply only to a single instance (e.g., /api/v1/widgets/\u0026lt;name\u0026gt;).\nAnother common architectural pattern is to expose methods which allow the client to perform CRUD operations on the resource. CRUD (Create, Retrieve, Update, Delete) is a term that usually refers to relational database systems, but it is also valid for any dataset that can be manipulated by a client, including RESTful resources.\nNow, you might be wondering how this can be accomplished if we only expose two endpoints per resource, and CRUD defines (at leat) four different operations. Table 1 shows how we will structure the API for our Widget resource:\n  Table 1   Widget API endpoint specifications   Endpoint Name URI HTTP Method CRUD Operation Required Token     api.widget_list /api/v1/widgets POST Create a new widget Admin user   api.widget_list /api/v1/widgets GET Retrieve a list of widgets Regular user   api.widget /api/v1/widgets/\u0026lt;name\u0026gt; GET Retrieve a single widget Regular user   api.widget /api/v1/widgets/\u0026lt;name\u0026gt; PUT Update an existing widget Admin user   api.widget /api/v1/widgets/\u0026lt;name\u0026gt; DELETE Delete a single widget Admin user      Each endpoint can be configured to respond to a unique set of HTTP method types. Per Table 1, the api.widget_list endpoint will support GET and POST requests, and the api.widget endpoint will suport GET, PUT, and DELETE requests. Each combination of endpoint and method type are mapped to a CRUD operation. The remainder of this section (and the entire next section) are devoted to implementing the Widget API.\n  Operations that create, modify or delete widgets are restricted to users with the administrator role. Regular (non-admin) users can only retrieve individual widgets and/or lists of widgets from the database.\n  flask add-user Command Way back in Part 1, we discussed the Flask CLI and created the method that executes when the flask shell command is invoked. The Flask CLI is based on a project called Click which can be used to create powerful Python CLI applications, and is easy to get started with thanks to excellent documentation.\nCurrently, the api/v1/auth/register endpoint can only create regular (non-admin) users. We want to leave it this way since this endpoint is publically-accessible. However, we also need a way to create admin users since regular users cannot create, update or delete widget objects.\nThere are a few different methods we could use to create admin users. Using the flask shell command, we can execute arbitrary Python code to create admin users. Or, we could create a function and store it in a file in our project, then run the function through the command-line. However, both of these methods are cumbersome and would require documentation if anyone else needed to create an admin user.\nMy preferred solution is to expose a command in the Flask CLI that can create both regular and admin users. To do so, open run.py in the project root folder. First, update the import statements to include click (Line 4):\n1 2 3 4 5 6 7 8  \u0026#34;\u0026#34;\u0026#34;Flask CLI/Application entry point.\u0026#34;\u0026#34;\u0026#34; import os import click  from flask_api_tutorial import create_app, db from flask_api_tutorial.models.token_blacklist import BlacklistedToken from flask_api_tutorial.models.user import User   Then, add the content below and save the file:\n18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36  @app.cli.command(\u0026#34;add-user\u0026#34;, short_help=\u0026#34;Add a new user\u0026#34;) @click.argument(\u0026#34;email\u0026#34;) @click.option( \u0026#34;--admin\u0026#34;, is_flag=True, default=False, help=\u0026#34;New user has administrator role\u0026#34; ) @click.password_option(help=\u0026#34;Do not set password on the command line!\u0026#34;) def add_user(email, admin, password): \u0026#34;\u0026#34;\u0026#34;Add a new user to the database with email address = EMAIL.\u0026#34;\u0026#34;\u0026#34; if User.find_by_email(email): error = f\u0026#34;Error: {email} is already registered\u0026#34; click.secho(f\u0026#34;{error}\\n\u0026#34;, fg=\u0026#34;red\u0026#34;, bold=True) return 1 new_user = User(email=email, password=password, admin=admin) db.session.add(new_user) db.session.commit() user_type = \u0026#34;admin user\u0026#34; if admin else \u0026#34;user\u0026#34; message = f\u0026#34;Successfully added new {user_type}:\\n{new_user}\u0026#34; click.secho(message, fg=\u0026#34;blue\u0026#34;, bold=True) return 0     Explaining how to create a command with click is beyond the scope of this tutorial. Thankfully, the click documentation is exceptional. If you are interested, you can find everything you need to understand the add_user function in the links below:\n Basic Concepts - Creating a Command Echoing Adding Parameters Basic Arguments Boolean Flags Password Prompts Help Texts Documenting Arguments Command Short Help ANSI Colors  Finally, the Flask documentation explains how to add custom commands to the Flask CLI. Coincidentally, the example given in the documentation is a command to create a new user.\n  You might be wondering how is this different than the idea I dismissed, \u0026ldquo;create a function and store it in a file in our project, then run the function through the command-line\u0026rdquo;? The main difference is discoverability — Flask automatically includes a CLI with all installations and extending the CLI with custom commands is one of the intended use cases.\nAlso, help documentation is automatically generated for CLI commands (via click). You can view the documentation for the add-user command by running flask add-user --help:\n(flask-api-tutorial) flask-api-tutorial $ flask add-user --help Usage: flask add-user [OPTIONS] EMAIL Add a new user to the database with email address = EMAIL. Options: --admin New user has administrator privileges --password TEXT Do not set password on the command line! --help Show this message and exit. Users can view all Flask CLI commands, including custom commands by running flask:\n(flask-api-tutorial) flask-api-tutorial $ flask Usage: flask [OPTIONS] COMMAND [ARGS]... A general utility script for Flask applications. Provides commands from Flask, extensions, and the application. Loads the application defined in the FLASK_APP environment variable, or from a wsgi.py file. Setting the FLASK_ENV environment variable to 'development' will enable debug mode. $ export FLASK_APP=hello.py $ export FLASK_ENV=development $ flask run Options: --version Show the flask version --help Show this message and exit. Commands: add-user add a new user db Perform database migrations. routes Show the routes for the app. run Run a development server. shell Run a shell in the app context. This is my preferred solution because the process for creating admin users is now documented, and the command is accessible only to those with access to the server where the flask instance is running.\nFinally, let\u0026rsquo;s demonstrate how to use the flask add-user command:\n Create regular user\n(flask-api-tutorial) flask-api-tutorial $ flask add-user user@test.com Password: Repeat for confirmation: Successfully added new user: \u0026lt;User email=user@test.com, public_id=be5e164e-7d33-4919-8d37-a5d02ca27d47, admin=False\u0026gt;   Create admin user\n(flask-api-tutorial) flask-api-tutorial $ flask add-user admin@test.com --admin Password: Repeat for confirmation: Successfully added new admin user: \u0026lt;User email=admin@test.com, public_id=1e248b12-e08b-467f-86bf-80f547f20ce6, admin=True\u0026gt;   Error: Email already exists\n(flask-api-tutorial) flask-api-tutorial $ flask add-user user@test.com Password: Repeat for confirmation: Error: user@test.com is already registered   As you can see, after running the command, the user is immediately prompted to create a password and to confirm the password for the new user. Before proceeding, please create an admin user with the flask add-user command since creating, modifying and deleting widgets cannot be performed otherwise.\nWidget DB Model Before we can begin implementing the API endpoints in Table 1, we need to create a database table to store Widget instances. To do so, we extend db.Model (just like we did for the User and BlacklistedToken classes). Create a new file widget.py in src/flask_api_tutorial/models and add the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61  \u0026#34;\u0026#34;\u0026#34;Class definition for Widget model.\u0026#34;\u0026#34;\u0026#34; from datetime import datetime, timezone, timedelta from sqlalchemy.ext.hybrid import hybrid_property from flask_api_tutorial import db from flask_api_tutorial.util.datetime_util import ( utc_now, format_timedelta_str, get_local_utcoffset, localized_dt_string, make_tzaware, ) class Widget(db.Model): \u0026#34;\u0026#34;\u0026#34;Widget model for a generic resource in a REST API.\u0026#34;\u0026#34;\u0026#34; __tablename__ = \u0026#34;widget\u0026#34; id = db.Column(db.Integer, primary_key=True, autoincrement=True) name = db.Column(db.String(100), unique=True, nullable=False) info_url = db.Column(db.String(255)) created_at = db.Column(db.DateTime, default=utc_now) deadline = db.Column(db.DateTime) owner_id = db.Column(db.Integer, db.ForeignKey(\u0026#34;site_user.id\u0026#34;), nullable=False) owner = db.relationship(\u0026#34;User\u0026#34;, backref=db.backref(\u0026#34;widgets\u0026#34;)) def __repr__(self): return f\u0026#34;\u0026lt;Widget name={self.name}, info_url={self.info_url}\u0026gt;\u0026#34; @hybrid_property def created_at_str(self): created_at_utc = make_tzaware( self.created_at, use_tz=timezone.utc, localize=False ) return localized_dt_string(created_at_utc, use_tz=get_local_utcoffset()) @hybrid_property def deadline_str(self): deadline_utc = make_tzaware(self.deadline, use_tz=timezone.utc, localize=False) return localized_dt_string(deadline_utc, use_tz=get_local_utcoffset()) @hybrid_property def deadline_passed(self): return datetime.now(timezone.utc) \u0026gt; self.deadline.replace(tzinfo=timezone.utc) @hybrid_property def time_remaining(self): time_remaining = self.deadline.replace(tzinfo=timezone.utc) - utc_now() return time_remaining if not self.deadline_passed else timedelta(0) @hybrid_property def time_remaining_str(self): timedelta_str = format_timedelta_str(self.time_remaining) return timedelta_str if not self.deadline_passed else \u0026#34;No time remaining\u0026#34; @classmethod def find_by_name(cls, name): return cls.query.filter_by(name=name).first()   To demonstrate the process of serializing a complex object to/from JSON, the Widget class includes attributes with as many different data types as possible. Additionally, the project requirements include various rules restricting which values are considered valid for the name and deadline attributes:\nThe widget model contains attributes with URL, datetime, timedelta and bool data types, along with normal text fields. The widget model contains a \"name\" field which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character). The widget model contains a \"deadline\" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.  Let\u0026rsquo;s take a look at how these attributes are defined and how they fulfill the various project requirements:\n  Line 22: Table 1 indicates that the value of the name attribute will be embedded in the URI for each Widget. Because of this, the value must be unique which is ensured by setting unique=True. Additionally, it would be ideal to prevent the user from creating a new widget if the name contains characters that are not URL-safe (/, +, \u0026 etc.). To accomplish this, we will design a custom RequestParser data type that considers a value to be valid if it contains ONLY lowercase-letters, numbers, underscore and/or hyphen characters.\n  Line 23: The purpose of the info_url attribute is to demonstrate how to implement input validation for URL values. Any values that are not recognized as a valid URL must be rejected without adding the widget to the database. The validation logic will be implemented using a built-in RequestParser data type for URL values.\n  Line 25: The purpose of the deadline attribute is to demonstrate how to implement input validation for datetime values. Additionally, only datetime values that are either the same as or greater than the current date are considered valid. Values not recognized as valid datetime values AND valid datetime values in the past must be rejected without adding the widget to the database.\n  Lines 27-28: We will also use the Widget class to demonstrate how relationships between database tables are defined and managed. We have defined a foreign key relationship between this table and the site_user table. The owner of each widget will be the User that created it (The User.id attribute will be stored when each Widget is created).\n  Lines 33-41: Both of these hybrid properties convert the datetime value stored in the database to the timezone of the machine executing the code and formats the datetime as a string value.\n  Lines 43-45: deadline_passed is a bool value, this has been included as part of the Widget model solely to increase the number of data types that are serialized when Widget objects are included in an HTTP response. This attribute should return True if the curent date is greater than the date stored in deadline, and should return False if the current date is less than or the same as the date stored in deadline.\n  Lines 47-50: time_remaining is a timedelta value that represents the time remaining until the deadline is passed. If the curent date is greater than the date stored in deadline, then this attribute should return timedelta(0).\n  Lines 52-55: time_remaining_str converts the timedelta object returned by time_remaining to a formatted string if the deadline has not been passed. If the deadline has passed, \"No time remaining\" is returned.\n  Lines 57-59: find_by_name is a class method just like the find_by_email and find_by_public_id methods we previously created in the User class. Since the name attribute must be unique, we can use it to retrieve a specific Widget.\n   Next, we need to update run.py in order for the Flask-Migrate extension to recognize the Widget class and create a migration script that adds the new table to the database (this is the same process we previously performed for the User class in Part 2 and for the BlacklistedToken class in Part 4).\nOpen run.py in the project root folder and update the import statements to include the Widget class (Line 9). Then add the Widget class to the dict object that is returned by the make_shell_context function (Line 16):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21  \u0026#34;\u0026#34;\u0026#34;Flask CLI/Application entry point.\u0026#34;\u0026#34;\u0026#34; import os import click from flask_api_tutorial import create_app, db from flask_api_tutorial.models.token_blacklist import BlacklistedToken from flask_api_tutorial.models.user import User from flask_api_tutorial.models.widget import Widget  app = create_app(os.getenv(\u0026#34;FLASK_ENV\u0026#34;, \u0026#34;development\u0026#34;)) @app.shell_context_processor def shell(): return { \u0026#34;db\u0026#34;: db, \u0026#34;User\u0026#34;: User, \u0026#34;BlacklistedToken\u0026#34;: BlacklistedToken, \u0026#34;Widget\u0026#34;: Widget,  }   Next, run flask db migrate and add a message explaining the changes that will be made by executing this migration script:\n(flask-api-tutorial) flask-api-tutorial $ flask db migrate --message \"add widget model\" INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.autogenerate.compare] Detected added table 'widget' Generating /Users/aaronluna/Projects/flask-api-tutorial/migrations/versions/7c66df0e878f_add_widget_model.py ... done   You can verify that the widget table was detected by the Flask Migrate extension from the output of the flask db migrate command. You should see a message similar to the example above (Detected added table 'widget'), followed by a statement indicating the migration script was successfully generated.\n  Next, run flask db upgrade to run the migration script on the database in your development environment:\n(flask-api-tutorial) flask-api-tutorial $ flask db upgrade INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade 079d26d45cc9 - 7c66df0e878f, add widget model After the widget table has been added to the database, we can begin implementing the API endpoints specified in Table 1.\nCreate Widget So where should we begin? In my opinion, the endpoint that should be implemented first is the endpoint responsible for the create operation, since without Widget objects there\u0026rsquo;s nothing to be retrieved, updated or deleted. In Part 3 we followed the process below for each endpoint in the auth_ns namespace. We will follow the same process to implement the widget_ns endpoints in Table 1:\n Create request parsers/API models to validate request data and serialize response data. Define the business logic necessary to process the request if validation succeeds. Create a class that inherits from Resource and bind it to the API endpoint/URL route. Define the set of HTTP methods that the API endpoint will support and expose methods on the concrete Resource class for each. Methods named get, post, put, delete, patch, options or head will be called when the API endpoint receives a request of the same HTTP method type.   If the API endpoint does not support the HTTP method, do not expose a method with the name of the HTTP method and the client will receive a response with status code 405 HTTPStatus.METHOD_NOT_ALLOWED\n   Document the Resource class and methods which handle HTTP requests as explained in the Flask-RESTx docs. Most of the content on the Swagger UI page is generated by decorating the concrete Resource classes created in Step 3 and the handler methods in Step 4. Import the business logic functions created in Step 2 and call them within the HTTP method that corresponds to the operation performed by the business logic. Create unit tests to verify that the input validation provided by the request parsers/API models is working correctly, and verify the endpoint behaves as expected.   Step 1 says create request parsers/API models to validate request data and serialize response data. So let\u0026rsquo;s dive into it!\ncreate_widget_reqparser Request Parser When a client sends a request to create a new Widget, what data is required? Take a look at the attributes of the Widget class:\n22 23 24 25 26 27 28 29  id = db.Column(db.Integer, primary_key=True, autoincrement=True) name = db.Column(db.String(100), unique=True, nullable=False) info_url = db.Column(db.String(255)) created_at = db.Column(db.DateTime, default=utc_now) deadline = db.Column(db.DateTime)  owner_id = db.Column(db.Integer, db.ForeignKey(\u0026#34;site_user.id\u0026#34;), nullable=False) owner = db.relationship(\u0026#34;User\u0026#34;, backref=db.backref(\u0026#34;widgets\u0026#34;))   The client must provide values for the three attributes highlighted above (name, info_url and deadline). What about the other attributes?\n  id: This is the database table's primary key. The value is automatically set when a Widget is committed to the database (starting at one, then incrementing by one each time a Widget is added).\n  created_at: When a Widget object is created, the expression specified in the default parameter is evaluated and stored as the value for created_at. utc_now returns the current date and time as a datetime value that is timezone-aware and localized to UTC.\n  owner_id: This value is a foreign-key, which is indicated by db.ForeignKey(\"site_user.id\"). site_user is the name of the database table where User objects are stored, and site_user.id is the primary-key that the owner_id column is referencing.\n  owner: It is important to note that owner is NOT an instance of db.Column. This means that unlike the other Widget class attributes, owner is not a column that exists in the widget database table. Instead, this is a relationship object that demonstates one of the main features of the SQLAlchemy ORM (Click here for more information).\nWhen processing a request to create a new Widget, the business logic will set the value of the owner_id attribute to the id of the User who sent the request. After the Widget is created and committed to the database, the owner attribute will contain a User object that represents the User who created it.\nAnother interesting feature is achieved by backref=db.backref(\"widgets\"). This creates a new attribute on all User objects named widgets (without modifying the User class at all), which is a list of all Widget objects in the database where User.id is equal to owner_id.\n   Flask-RESTx includes helpful pre-defined types (e.g., email, URL, etc.) in the inputs module for validating request data. When we created the auth_reqparser in Part 3, we imported the email class from flask_restx.inputs to verify if a value provided by the client is a valid email address. You can also define custom input types if none of the pre-defined types are sufficient, and we will do so for both the name and deadline attributes.\nCreate a new file named dto.py in src/flask_api_tutorial/api/widgets and enter the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68  \u0026#34;\u0026#34;\u0026#34;Parsers and serializers for /widgets API endpoints.\u0026#34;\u0026#34;\u0026#34; import re from datetime import date, datetime, time, timezone from dateutil import parser from flask_restx.inputs import URL from flask_restx.reqparse import RequestParser from flask_api_tutorial.util.datetime_util import make_tzaware, DATE_MONTH_NAME def widget_name(name): \u0026#34;\u0026#34;\u0026#34;Validation method for a string containing only letters, numbers, \u0026#39;-\u0026#39; and \u0026#39;_\u0026#39;.\u0026#34;\u0026#34;\u0026#34; if not re.compile(r\u0026#34;^[\\w-]+$\u0026#34;).match(name): raise ValueError( f\u0026#34;\u0026#39;{name}\u0026#39; contains one or more invalid characters. Widget name must \u0026#34; \u0026#34;contain only letters, numbers, hyphen and underscore characters.\u0026#34; ) return name def future_date_from_string(date_str): \u0026#34;\u0026#34;\u0026#34;Validation method for a date in the future, formatted as a string.\u0026#34;\u0026#34;\u0026#34; try: parsed_date = parser.parse(date_str) except ValueError: raise ValueError( f\u0026#34;Failed to parse \u0026#39;{date_str}\u0026#39; as a valid date. You can use any format \u0026#34; \u0026#34;recognized by dateutil.parser. For example, all of the strings below \u0026#34; \u0026#34;are valid ways to represent the same date: \u0026#39;2018-5-13\u0026#39; -or- \u0026#39;05/13/2018\u0026#39; \u0026#34; \u0026#34;-or- \u0026#39;May 13 2018\u0026#39;.\u0026#34; ) if parsed_date.date() \u0026lt; date.today(): raise ValueError( f\u0026#34;Successfully parsed {date_str} as \u0026#34; f\u0026#34;{parsed_date.strftime(DATE_MONTH_NAME)}. However, this value must be a \u0026#34; f\u0026#34;date in the future and {parsed_date.strftime(DATE_MONTH_NAME)} is BEFORE \u0026#34; f\u0026#34;{datetime.now().strftime(DATE_MONTH_NAME)}\u0026#34; ) deadline = datetime.combine(parsed_date.date(), time.max) deadline_utc = make_tzaware(deadline, use_tz=timezone.utc) return deadline_utc create_widget_reqparser = RequestParser(bundle_errors=True) create_widget_reqparser.add_argument( \u0026#34;name\u0026#34;, type=widget_name, location=\u0026#34;form\u0026#34;, required=True, nullable=False, case_sensitive=True, ) create_widget_reqparser.add_argument( \u0026#34;info_url\u0026#34;, type=URL(schemes=[\u0026#34;http\u0026#34;, \u0026#34;https\u0026#34;]), location=\u0026#34;form\u0026#34;, required=True, nullable=False, ) create_widget_reqparser.add_argument( \u0026#34;deadline\u0026#34;, type=future_date_from_string, location=\u0026#34;form\u0026#34;, required=True, nullable=False, )   Let\u0026rsquo;s break this down by looking at how each of the attributes are validated by the code above:\nname Argument None of the pre-defined types in the flask_restx.inputs module perform input validation that satisfies the requirements of the name attribute. Thankfully, Flask-RESTx provides a way to create custom types that can be used in the same way. The example of a custom type shown below is taken from the documentation:\ndef my_type(value): \u0026#39;\u0026#39;\u0026#39;Parse my type\u0026#39;\u0026#39;\u0026#39; if not condition: raise ValueError(\u0026#39;This is not my type\u0026#39;) return parse(value) The term \u0026ldquo;type\u0026rdquo; is (IMO) misleading since we only need to create a function (not a class). The function must accept at least one parameter — the value provided by the client. If the value is successfully validated, the function must convert the value to the expected data type before returning it (in the case of the name attribute, the expected type is a string so no conversion is necessary). If the value provided by the client is invalid, the function must raise a ValueError.\nThe widget_name function is adapted directly from the example shown above to satisfy the project requirements:\n12 13 14 15 16 17 18 19  def widget_name(name): \u0026#34;\u0026#34;\u0026#34;Validation method for a string containing only letters, numbers, \u0026#39;-\u0026#39; and \u0026#39;_\u0026#39;.\u0026#34;\u0026#34;\u0026#34; if not re.compile(r\u0026#34;^[\\w-]+$\u0026#34;).match(name): raise ValueError( f\u0026#34;\u0026#39;{name}\u0026#39; contains one or more invalid characters. Widget name must \u0026#34; \u0026#34;contain only letters, numbers, hyphen and underscore characters.\u0026#34; ) return name     Line 14: The simplist way to implement our custom type is with a regular expression. The regex ^[\\w-]+$ will match any string that consists of ONLY alphanumeric characters (which includes the underscore character) and the hyphen character.\nThe syntax of regular expressions is extremely dense. To make any regex easier to understand, we could compile it with the re.VERBOSE flag. This causes whitespace that is not within a character class to be ignored, allowing us to place comments within the regex to document the design and the effect of each component of the expression. For example, we could document our regex as shown below (I am only showing this for demonstration purposes, and have not modified the code in src/flask_api_tutorial/api/widgets/dto.py):\nNAME_REGEX = re.compile(r\"\"\" ^ # Matches the beginning of the string [\\w-] # Character class: \\w matches all alphanumeric characters (including underscore), - matches the hyphen character + # Match one or more instances of the preceding character class $ # Matches the end of the string \"\"\", re.VERBOSE)   Teaching regular expressions is beyond the scope of this tutorial. However, if you are looking for a good introduction to the topic I recommend reading Regular Expression HOWTO from the official Python docs.\n    Line 15: If the value does not match the regex, a ValueError is raised with a message explaining why the value is not a valid widget_name.\n  Line 19: If the value passed to this function matches the regex, the value is a valid widget_name and is returned.\n   We can see how this function works by testing it in the flask shell:\n(flask-api-tutorial) flask-api-tutorial $ flask shell Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin App: flask-api-tutorial [development] Instance: /Users/aaronluna/Projects/flask_api_tutorial/instance  from flask_api_tutorial.api.widgets.dto import widget_name # 1. 'test' is a valid widget_name  widget_name(\"test\") 'test' # 2. 'test_1-AZ' is a valid widget_name  widget_name(\"test_1-AZ\") 'test_1-AZ' # 3. 'some widget' is NOT a valid widget_name  widget_name(\"some widget\") Traceback (most recent call last): File \"\u0026lt;console\u0026gt;\", line 1, in \u0026lt;module\u0026gt; File \"/Users/aaronluna/Projects/flask_api_tutorial/src/flask_api_tutorial/api/widgets/dto.py\", line 18, in widget_name f\"'{name}' contains one or more invalid characters. Widget name must contain \" ValueError: 'some widget' contains one or more invalid characters. Widget name must contain only letters, numbers, hyphen and/or underscore characters. # 4. 't**\u0026*\u0026' is NOT a valid widget_name  widget_name(\"t**\u0026*\u0026\") Traceback (most recent call last): File \"\u0026lt;console\u0026gt;\", line 1, in \u0026lt;module\u0026gt; File \"/Users/aaronluna/Projects/flask_api_tutorial/src/flask_api_tutorial/api/widgets/dto.py\", line 18, in widget_name f\"'{name}' contains one or more invalid characters. Widget name must contain \" ValueError: 't**\u0026*\u0026' contains one or more invalid characters. Widget name must contain only letters, numbers, hyphen and/or underscore characters. The first test passes since test consists of only letters. The second test passes because it contains one of each of the allowed character types (letter, number, hyphen, underscore) and no other characters. The third and fourth tests fail because they both contain one or more forbidden characters (space, asterisk, ampersand).\nWait, let\u0026rsquo;s back up. Didn\u0026rsquo;t the requirement for the name attribute say that only lowercase letters were allowed? Yep, you got me. I kinda sort-of lied about test_1-AZ being a valid widget_name. But there is a reason why I did this, which will be revealed by the configuration of the argument object for the name attribute:\n47 48 49 50 51 52 53 54  create_widget_reqparser.add_argument( \u0026#34;name\u0026#34;, type=widget_name, location=\u0026#34;form\u0026#34;, required=True, nullable=False, case_sensitive=True, )     Line 49: All we need to do to use our custom type is set the value of the type parameter to the widget_name function.\n  Lines 50-52: The location, required and nullable parameters should be familiar since we explained their purpose in Part 3.\n  Line 53: This is the first time we are using the case_sensitive parameter. According to the documentation, by default, case_sensitive=True. If this value is False \"this will convert all values to lowercase\".\n   Configuring the name argument with type=widget_name and case_sensitive=False allows the client to provide a value for the widget name in any combination of upper and lowercase letters. When an HTTP request is received to create a widget, if the request contains a name attribute, the value of that attribute will be passed to the widget_name function.\nIf the widget name contains only valid characters, it will be converted to lowercase before being passed to the business logic. The business logic will query the database for widgets with the same name. If a widget is found with the same name, the request will be aborted and the widget will not be created. If no widget is found with the same name, a new widget object will be created and added to the database.\ninfo_url Argument 55 56 57 58 59 60 61  create_widget_reqparser.add_argument( \u0026#34;info_url\u0026#34;, type=URL(schemes=[\u0026#34;http\u0026#34;, \u0026#34;https\u0026#34;]), location=\u0026#34;form\u0026#34;, required=True, nullable=False, )   The info_url attribute can be parsed using the pre-defined URL type from the inputs module, which can be configured in numerous ways. For example, it is possible to restrict the allowed values to a whitelist of domains and/or exclude a blacklist of domains. You can choose to perform a DNS lookup on the domain specified by the client and reject the value if the domain fails to resolve. Check out the documentation for the URL type for even more ways you can control the allowed URL range/format.\nThe only restriction on info_url that we will employ is that the URL scheme must be either http or https. This can be more useful than you may think. Imagine if the Widget class had git_url and ssh_url attributes in addition to info_url. By simply changing the value of the schemes parameter from schemes=['http', 'https'] to schemes=['git'] and schemes=['ssh'], we could easily and effectively prevent clients from inserting values into the database that would cause errors if an application blindly tried to use those URLs to access a git repository or login to a server.\n  The value of the schemes parameter must always be a list or tuple, even if you intend to limit the allowed URL values to a single scheme.\n  There really isn\u0026rsquo;t anything else to say about how the info_url attribute is parsed since we already covered using a pre-defined input type when we used the email type as part of the auth_reqparser in Part 3. So let\u0026rsquo;s move on to something a bit more interesting.\ndeadline Argument deadline is the last piece of data that we must receive from the client in order to create a new widget. Utlimately, this value must be converted to a datetime since the Widget class has several hybrid_properties that perform comparisons or calculations that assume this is the case. The inputs module provides several pre-defined types that can convert request data to either a date or a datetime value. I\u0026rsquo;ve summarized the requirements for these types in Table 2 below:\n  Table 2   Pre-defined Input Types for date and datetime Values   Pre-defined Type 1 Required Format 2 Example Reference     date YYYY-MM-DD 2019-10-02 N/A   date_from_iso8601 YYYY-MM-DD 2019-10-02 ISO 8601   datetime_from_iso8601 YYYY-MM-DDThh:mm:ss(+/-)zh:zm 2019-10-02T15:05:06-07:00 ISO 8601   datetime_from_rfc822 DN, DD MN YYYY hh:mm:ss (+/-)zhzm Wed, 02 Oct 2019 15:05:06 -0700 RFC 5322 3     1 Pre-defined types are located in the flask_restx.inputs module.   2 YYYY = 4-digit year, MM = 2-digit month, DD = 2-digit day,\nhh = 2-digit hour, mm = 2-digit minutes, ss = 2-digit seconds,\nzh = 2-digit UTC offset hours, zm = 2-digit UTC offset minutes,\nDN (Day Name) = \"Mon\" / \"Tue\" / \"Wed\" / \"Thu\" / \"Fri\" / \"Sat\" / \"Sun\",\nMN (Month Name) = \"Jan\" / \"Feb\" / \"Mar\" / \"Apr\" / \"May\" / \"Jun\" / \"Jul\" / \"Aug\" / \"Sep\" / \"Oct\" / \"Nov\" / \"Dec\"\n   3 RFC 5322 is a revision of RFC 2822, which itself obsoleted RFC 822      It would absolutely be possible to satisfy the project requirements using the pre-defined types in Table 2, but (IMO) they all have one big limitation — there is only a single valid format for the value provided by the client. Dates are expressed in so many different ways via text, so my preference is to design a parser that can accommodate a variety of date formats.\nThere is also a problem that arises from the project requirements for the deadline attribute: the value must not be a date in the past. Since the pre-defined types only validate that the value provided by the client is in the proper format to be converted to a datetime value, we would have to perform the task of checking if the date provided by the client is in the past in the business logic.\nThis second issue is another matter of opinion, since you could easily object by pointing out that the name attribute has a requirement to be unique and the task of querying the database for widgets with the same name is not performed in the wiget_name function we just created.\nI\u0026rsquo;ll explain the distinction between these two requirements with a hypothetical situation: A request is received to create a new widget with name=\u0026quot;test\u0026quot;. This is a valid widget name, so the request is passed to the business logic and since no widget already exists in the database with name=\u0026quot;test\u0026quot;, a new widget is created. Then, an identical request is received to create a widget with name=\u0026quot;test\u0026quot;. IMO, from the point-of-view of the create_widget_reqparser, test is a valid widget name, so the request is again passed to the business logic. However, since a widget already exists with name=\u0026quot;test\u0026quot;, the request is aborted with a message indicating that a widget already exists in the database with name=\u0026quot;test\u0026quot;.\nOTOH, consider this scenario: A request is received to create a new widget with deadline=\u0026quot;1923-03-28\u0026quot;. This string is in a valid format but since the date is obviously in the past, the create_widget_reqparser raises a ValueError and the request is aborted. A deadline in the past is always invalid and a request to create a widget containing an invalid value for a required parameter must always be rejected. Hopefully my reasoning makes sense to you.\ndateutil.parser Since the pre-defined types are adequate but inflexible, what can we use to parse a date value from a string? If we wanted to restrict ourselves to the Python standard library, we would need to create a complicated function that uses the strptime method of either the date or datetime class. This would involve testing as many format strings as possible and would quickly become a nightmare.\nLuckily, I see no reason to impose such a restriction for this project. IMO, the most robust and usable way to parse datetime values from a string is the parse function in the dateutil.parser module (this was listed as a requirement in the setup.py file, so it is already installed). Unlike strptime, this method does not require you to provide a format string.\ndateutil.parser.parse recognizes a wide variety of date and datetime formats. The locale setting of the machine where the function is executed is taken into consideration, which is extremely helpful in situations where the order of the month and day values are transposed (e.g., US date format vs. European). You can find more information in the official documentation for the dateutil.parser module.\nfuture_date_from_string Since deadline must be either the current date or a date in the future, I decided to name the custom type function for this attribute future_date_from_string. This function is more complex than widget_name since two separate validations must be peformed — parsing the input string to a date value and checking that the parsed date is not in the past:\n22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43  def future_date_from_string(date_str): \u0026#34;\u0026#34;\u0026#34;Validation method for a date in the future, formatted as a string.\u0026#34;\u0026#34;\u0026#34; try: parsed_date = parser.parse(date_str) except ValueError: raise ValueError( f\u0026#34;Failed to parse \u0026#39;{date_str}\u0026#39; as a valid date. You can use any format \u0026#34; \u0026#34;recognized by dateutil.parser. For example, all of the strings below \u0026#34; \u0026#34;are valid ways to represent the same date: \u0026#39;2018-5-13\u0026#39; -or- \u0026#39;05/13/2018\u0026#39; \u0026#34; \u0026#34;-or- \u0026#39;May 13 2018\u0026#39;.\u0026#34; ) if parsed_date.date() \u0026lt; date.today(): raise ValueError( f\u0026#34;Successfully parsed {date_str} as \u0026#34; f\u0026#34;{parsed_date.strftime(DATE_MONTH_NAME)}. However, this value must be a \u0026#34; f\u0026#34;date in the future and {parsed_date.strftime(DATE_MONTH_NAME)} is BEFORE \u0026#34; f\u0026#34;{datetime.now().strftime(DATE_MONTH_NAME)}\u0026#34; ) deadline = datetime.combine(parsed_date.date(), time.max) deadline_utc = make_tzaware(deadline, use_tz=timezone.utc) return deadline_utc   There are a few things about the future_date_from_string function that are worth pointing out:\n  Line 25: The value provided by the user is passed to the dateutil.parser.parse method. If the parser is able to convert the string value to a datetime value it is stored in the parsed_date variable.\n  Lines 26-32: If the dateutil.parser.parse method is unable to parse the value provided by the user, it raises a ValueError. Since the error message that is generated isn't very descriptive, the original error is suppressed and another value error is raised with an error message that clearly explains why the value provided by the user was rejected and provides examples of string values that would be considered valid.\n  Line 34: Per the project requirements, the date value that was provided by the user is compared to the current date, without considering the time component of either date when making the comparison. If the date provided by the user is less than (i.e., before) the current date, the value is rejected since this means the deadline has passed.\n  Lines 35-40: If the deadline has passed, raise a ValueError explaining why the value is invalid.\n  Line 41: Per the project requirements, the deadline value flips from \"not passed\" to \"passed\" at the stroke of midnight. To achieve this, we set the deadline by taking the date component of the value provided by the user and combining it with time.max(), which is a shortcut for a time value equal to 11:59:59 PM.\n  Lines 42-43: Since we need to ensure that any datetime value is timezone-aware before we add it to the database, we localize the value to UTC before it is returned and passed to the business logic.\n   We already know how to use a custom type, so the last thing we need to do is add an argument to our create_widget_reqparser with type=future_date_from_string;\n62 63 64 65 66 67 68  create_widget_reqparser.add_argument( \u0026#34;deadline\u0026#34;, type=future_date_from_string, location=\u0026#34;form\u0026#34;, required=True, nullable=False, )   Remember, we are following the process defined earlier to create each endpoint. We can now move on to step #2: Define the business logic necessary to process the request if validation succeeds.\ncreate_widget Method Next, we need to define a function that performs the following actions:\n Add a new widget to the database Associate the widget with the user who created it Construct an HTTP response including a \u0026ldquo;Location\u0026rdquo; field in the header equal to the URI of the widget  RFC7231 defines the semantics of HTTP/1.1 messages. Specifically, it specifies when the \u0026ldquo;Location\u0026rdquo; field should be included in the header of an HTTP response:\n7.1.2. Location The \"Location\" header field is used in some responses to refer to a specific resource in relation to the response. The type of relationship is defined by the combination of request method and status code semantics.\nLocation = URI-reference\n...\nFor 201 (Created) responses, the Location value refers to the primary resource created by the request.\n Create a new file named business.py in src/flask_api_tutorial/api/widgets and enter the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27  \u0026#34;\u0026#34;\u0026#34;Business logic for /widgets API endpoints.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask import jsonify, url_for from flask_restx import abort from flask_api_tutorial import db from flask_api_tutorial.api.auth.decorators import admin_token_required from flask_api_tutorial.models.user import User from flask_api_tutorial.models.widget import Widget @admin_token_required def create_widget(widget_dict): name = widget_dict[\u0026#34;name\u0026#34;] if Widget.find_by_name(name): error = f\u0026#34;Widget name: {name} already exists, must be unique.\u0026#34; abort(HTTPStatus.CONFLICT, error, status=\u0026#34;fail\u0026#34;) widget = Widget(**widget_dict) owner = User.find_by_public_id(create_widget.public_id) widget.owner_id = owner.id db.session.add(widget) db.session.commit() response = jsonify(status=\u0026#34;success\u0026#34;, message=f\u0026#34;New widget added: {name}.\u0026#34;) response.status_code = HTTPStatus.CREATED response.headers[\u0026#34;Location\u0026#34;] = url_for(\u0026#34;api.widget\u0026#34;, name=name) return response   Let\u0026rsquo;s take a look at how the create_widget function performs the tasks listed above:\n  Line 13: Per the specification in Table 1, the ability to create a widget object is limited to users with the administrator role. To enforce this, we decorate the create_widget method with @admin_token_required. If you would like to review how this decorator is implemented, click here.\n  Line 14: After the request data has been parsed and validated, it is passed to the create_widget function as a dict object named widget_dict.\n  Lines 15-18: The function starts by checking if a widget already exists with the same name provided in the request. If this is true, the request is aborted. The first argument to the abort function is the HTTP status code to include in the response. In this case, the appropriate response code is 409 HTTPStatus.CONFLICT.\n  Line 19: ** is the dictionary unpacking operator, you can find more info on it and the related list unpacking operator (*) in PEP 448. It is a concise way to pass the name, info_url, and deadline values to the Widget constructor.\n  Lines 20-21: Thanks to the @admin_token_required decorator, the public_id of the user who sent the request is stored in create_widget.public_id (If you don't remember why this is is the case, review the last section where we broke down how the decorators are designed).\nWe retrieve the User object that corresponds to the public_id of the user that sent the request and assign it to owner. Then, owner.id is set as the value of widget.owner_id.\n  Why can't we just use public_id as the value of widget.owner_id, instead of going through the process of retrieving the User object and using the id attribute? Since widget.owner_id is defined as a foreign key referencing site_user.id (the primary key of the site_user table), we must store the value of the id attribute in order to correctly configure the relationship between the two tables.\n    Lines 22-23: The new widget is added to the database and the changes are committed..\n  Line 24: As we have seen previously, if we need to include a custom field in the header of a response, we must create the response object manually. One way to do this is with the flask.jsonify function.\n  Line 25: Since the request to create a new widget was successful, the correct HTTP status code for the response is 201 HTTPStatus.CREATED.\n  Line 27: After ensuring that the response is configured with all required header fields, it is sent to the client.\n     The call to the url_for function in Line 26 relies on the api.widget endpoint being implemented. We haven't implemented either of the two endpoints defined in Table 1 at this point, so this will result in an unhandled exception until both have have been fully implemented.\n  Next, we need to create the API endpoint for the create operation and incorporate it with the create_widget_reqparser and the create_widget function.\napi.widget_list Endpoint (POST Request) According to Table 1, the operation to create a widget is accessed by sending a POST request to the api.widget_list resource, located at /api/v1/widgets. To create this endpoint, create a new file endpoints.py in src/flask_api_tutorial/api/widgets and enter the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27  \u0026#34;\u0026#34;\u0026#34;API endpoint definitions for /widgets namespace.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_restx import Namespace, Resource from flask_api_tutorial.api.widgets.dto import create_widget_reqparser from flask_api_tutorial.api.widgets.business import create_widget widget_ns = Namespace(name=\u0026#34;widgets\u0026#34;, validate=True) @widget_ns.route(\u0026#34;\u0026#34;, endpoint=\u0026#34;widget_list\u0026#34;) @widget_ns.response(int(HTTPStatus.BAD_REQUEST), \u0026#34;Validation error.\u0026#34;) @widget_ns.response(int(HTTPStatus.UNAUTHORIZED), \u0026#34;Unauthorized.\u0026#34;) @widget_ns.response(int(HTTPStatus.INTERNAL_SERVER_ERROR), \u0026#34;Internal server error.\u0026#34;) class WidgetList(Resource): \u0026#34;\u0026#34;\u0026#34;Handles HTTP requests to URL: /widgets.\u0026#34;\u0026#34;\u0026#34; @widget_ns.doc(security=\u0026#34;Bearer\u0026#34;) @widget_ns.response(int(HTTPStatus.CREATED), \u0026#34;Added new widget.\u0026#34;) @widget_ns.response(int(HTTPStatus.FORBIDDEN), \u0026#34;Administrator token required.\u0026#34;) @widget_ns.response(int(HTTPStatus.CONFLICT), \u0026#34;Widget name already exists.\u0026#34;) @widget_ns.expect(create_widget_reqparser) def post(self): \u0026#34;\u0026#34;\u0026#34;Create a widget.\u0026#34;\u0026#34;\u0026#34; widget_dict = create_widget_reqparser.parse_args() return create_widget(widget_dict)   There\u0026rsquo;s nothing in the code above that we haven\u0026rsquo;t already encountered and explained while implementing the auth_ns API endpoints. Click here if you need a refresher on Namespace objects, Resource objects, @doc, @response, or @expect decorators, etc.\nThe important part is Lines 26-27 where the parse_args method of create_widget_reqparser is used to validate the request data, which is then passed to the create_widget function which we just defined.\n  You may have noticed that the @response decorator is sometimes applied to the WidgetList Resource and sometimes applied to the post method. Why would you do this instead of solely documenting one or the other? Flask-RESTx documentation decorators applied to a class cascade and apply to all methods within the class. In this case, HTTPStatus.BAD_REQUEST, HTTPStatus.UNAUTHORIZED, and HTTPStatus.INTERNAL_SERVER_ERROR are valid responses to any HTTP request made to this endpoint. Placing the @response decorators on the Resource prevents duplicating them on each method.\n  Add widget_ns Namespace to api In order to register the widget_ns namespace with the api object, open src/flask_api_tutorial/api/__init__.py and add the highlighted lines (Line 6 and Line 27):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28  \u0026#34;\u0026#34;\u0026#34;API blueprint configuration.\u0026#34;\u0026#34;\u0026#34; from flask import Blueprint from flask_restx import Api from flask_api_tutorial.api.auth.endpoints import auth_ns from flask_api_tutorial.api.widgets.endpoints import widget_ns  api_bp = Blueprint(\u0026#34;api\u0026#34;, __name__, url_prefix=\u0026#34;/api/v1\u0026#34;) authorizations = { \u0026#34;Bearer\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;apiKey\u0026#34;, \u0026#34;in\u0026#34;: \u0026#34;header\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Authorization\u0026#34; } } api = Api( api_bp, version=\u0026#34;1.0\u0026#34;, title=\u0026#34;Flask API with JWT-Based Authentication\u0026#34;, description=\u0026#34;Welcome to the Swagger UI documentation for the Widget API\u0026#34;, doc=\u0026#34;/ui\u0026#34;, authorizations=authorizations, ) api.add_namespace(auth_ns, path=\u0026#34;/auth\u0026#34;) api.add_namespace(widget_ns, path=\u0026#34;/widgets\u0026#34;)   We can verify that the endpoint was created by running flask routes:\nflask-api-tutorial $ flask routes Endpoint Methods Rule ------------------- ------- -------------------------- api.auth_login POST /api/v1/auth/login api.auth_logout POST /api/v1/auth/logout api.auth_register POST /api/v1/auth/register api.auth_user GET /api/v1/auth/user api.doc GET /api/v1/ui api.root GET /api/v1/ api.specs GET /api/v1/swagger.json api.widget_list POST /api/v1/widgets restplus_doc.static GET /swaggerui/\u0026lt;path:filename\u0026gt; static GET /static/\u0026lt;path:filename\u0026gt; As expected, a new endpoint has been created named api.widget_list that responds to requests sent to /api/v1/widgets. Currently, this endpoint only supports requests where the method type is POST.\nNormally, we would create unit tests to verify the endpoint does create widget objects correctly. However, it was noted while explaining the design of the create_widget function that currently an unhandled exception occurs when running this function since the business logic for creating a new Widget depends on the api.widget endpoint existing. We will create unit tests for both endpoints in the widget_ns namespace when both have been fully implemented.\nCheckpoint Even though we only implemented one of the five CRUD operations specified in Table 1, we actually satisfied several of the remaining requirements. However, since we have not created any test coverage and confirmed that the process of creating a widget object is working correctly, these will be marked as only half-complete at this point:\nUser Management/JWT Authentication\nNew users can register by providing an email address and password\nExisting users can obtain a JWT by providing their email address and password\nJWT contains the following claims: time the token was issued, time the token expires, a value that identifies the user, and a flag that indicates if the user has administrator access\nJWT is sent in access_token field of HTTP response after successful authentication with email/password\nJWTs must expire after 1 hour (in production)\nJWT is sent by client in Authorization field of request header\nRequests must be rejected if JWT has been modified\nRequests must be rejected if JWT is expired\nIf user logs out, their JWT is immediately invalid/expired\nIf JWT is expired, user must re-authenticate with email/password to obtain a new JWT\n API Resource: Widget List\nAll users can retrieve a list of all widgets\nAll users can retrieve individual widgets by name\nUsers with administrator access can add new widgets to the database\nUsers with administrator access can edit existing widgets\nUsers with administrator access can delete widgets from the database\nThe widget model contains attributes with URL, datetime, timedelta and bool data types, along with normal text fields.\nURL and datetime values must be validated before a new widget is added to the database (and when an existing widget is updated).\nThe widget model contains a \"name\" attribute which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nThe widget model contains a \"deadline\" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.\nWidget name must be validated before a new widget is added to the database (and when an existing widget is updated).\nIf input validation fails either when adding a new widget or editing an existing widget, the API response must include error messages indicating the name(s) of the fields that failed validation.\n  In the next section, we will finish implementing the Widget API. If you have any questions/feedback, please leave a comment!\n","href":"/series/flask-api-tutorial/part-5/","title":"How To: Create a Flask API with JWT-Based Authentication (Part 5)"},{"categories":"Flask ; Python ; Tutorial-Series","content":"Project Structure The chart below shows the folder structure for this section of the tutorial. In this post, we will work on all files marked as NEW CODE. Files that contain code from previous sections but will not be modified in this post are marked as NO CHANGES.\n. (project root folder) |- src | |- flask_api_tutorial | |- api | | |- auth | | | |- __init__.py | | | |- business.py | | | |- decorators.py | | | |- dto.py | | | |- endpoints.py | | | | | |- widgets | | | |- __init__.py | | | |- business.py | | | |- dto.py | | | |- endpoints.py | | | | | |- __init__.py | | | |- models | | |- __init__.py | | |- token_blacklist.py | | |- user.py | | |- widget.py | | | |- util | | |- __init__.py | | |- datetime_util.py | | |- result.py | | | |- __init__.py | |- config.py | |- tests | |- __init__.py | |- conftest.py | |- test_auth_login.py | |- test_auth_logout.py | |- test_auth_register.py | |- test_auth_user.py | |- test_config.py | |- test_create_widget.py | |- test_delete_widget.py | |- test_retrieve_widget.py | |- test_retrieve_widget_list.py | |- test_update_widget.py | |- test_user.py | |- util.py | |- .env |- .gitignore |- .pre-commit-config.yaml |- pyproject.toml |- pytest.ini |- README.md |- run.py |- setup.py |- tox.ini KEY: FOLDER NEW CODE NO CHANGES EMPTY FILE   Introduction In the previous section, we began implementing the Widget API. I\u0026rsquo;ve reproduced the table from the previous section which gives the specifications for all endpoints in the widget_ns namespace:\n  Table 1   Widget API endpoint specifications   Endpoint Name URI HTTP Method CRUD Operation Required Token     api.widget_list /api/v1/widgets POST Create a new widget Admin user   api.widget_list /api/v1/widgets GET Retrieve a list of widgets Regular user   api.widget /api/v1/widgets/\u0026lt;name\u0026gt; GET Retrieve a single widget Regular user   api.widget /api/v1/widgets/\u0026lt;name\u0026gt; PUT Update an existing widget Admin user   api.widget /api/v1/widgets/\u0026lt;name\u0026gt; DELETE Delete a single widget Admin user      At this point, we have only implemented the process to create a new widget object. Per the specification in Table 1, to do so we created a new endpoint named api.widget_list that responds to requests sent to /api/v1/widgets. Clients can create a new widget object by sending a POST request to this endpoint. Table 1 indicates that the api.widget_list endpoint must also support GET requests in order to allow clients to retrieve a list of widget objects.\nWhat is the best way to send a list of complex objects to a client in an HTTP response? Before we implement the process/endpoint for retrieving a list of widgets, let\u0026rsquo;s review the established best practices.\nPagination When a client sends a GET request to the api.widget_list endpoint, they are requesting every widget in the database. As the number of widgets increases, so does the size of the HTTP response. As the size of the response increases, the time required to send and receive the data increases, resulting in a sluggish API.\nObviously, returning every widget in the database at once would be foolish. Instead, REST APIs typically employ pagination to limit the number of items sent in the response to the first 10, 20, etc database items (the maximum number of items per page is controlled by the server).\nClients can navigate through the full set of database objects by including a page parameter and a per_page parameter with their request data. Combining the ability to specify the number of items that the server should send per page as well as the page number, the client can retrieve any item from the database and avoid a sluggish response.\nAn example of a request/response pair containing paginated data is given below:\n1 flask-api-tutorial $ http :5000/api/v1/widgets?page=1\u0026per_page=10 Authorization:\"Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1NjY4MjA3NDksImlhdCI6MTU2NjgxOTg0NCwic3ViIjoiMzUyMDg5N2EtZWQ0My00YWMwLWIzYWYtMmZjMTY3NzE5MTYwIiwiYWRtaW4iOmZhbHNlfQ.AkpscH6QoCrfHYeyJTyouanwyj4KH34f3YmzMnyKKdM\" 2 3 GET /api/v1/widgets?page=1\u0026per_page=10 HTTP/1.1 4 Accept: */* 5 Accept-Encoding: gzip, deflate 6 Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1NjY4MjA3NDksImlhdCI6MTU2NjgxOTg0NCwic3ViIjoiMzUyMDg5N2EtZWQ0My00YWMwLWIzYWYtMmZjMTY3NzE5MTYwIiwiYWRtaW4iOmZhbHNlfQ.AkpscH6QoCrfHYeyJTyouanwyj4KH34f3YmzMnyKKdM 7 Connection: keep-alive 8 Host: localhost:5000 9 User-Agent: HTTPie/2.0.0 10 11 12 HTTP/1.0 200 OK 13 Access-Control-Allow-Origin: * 14 Content-Length: 1407 15 Content-Type: application/json 16 Date: Mon, 26 Aug 2019 11:45:39 GMT 17 Link: \u0026lt;/api/v1/widgets?page=1\u0026per_page=10; rel=\"self\", 18 \u0026lt;/api/v1/widgets?page=1\u0026per_page=10; rel=\"first\", 19 \u0026lt;/api/v1/widgets?page=2\u0026per_page=10; rel=\"next\", 20 \u0026lt;/api/v1/widgets?page=5\u0026per_page=10; rel=\"last\" 21 Server: Werkzeug/0.16.1 Python/3.7.6 22 Total-Count: 23 23 24 { 25 \"links\":{  26 \"self\": \"/api/v1/widgets?page=1\u0026per_page=10\", 27 \"first\": \"/api/v1/widgets?page=1\u0026per_page=10\", 28 \"next\": \"/api/v1/widgets?page=2\u0026per_page=10\", 29 \"last\": \"/api/v1/widgets?page=5\u0026per_page=10\" 30 }, 31 \"has_prev\": false, 32 \"has_next\": true, 33 \"page\": 1 34 \"total_pages\": 5, 35 \"items_per_page\": 10, 36 \"total_items\": 23, 37 \"items\": [ 38 { 39 //... 40 \"name\": \"test\" 41 \"link\": \"/api/v1/widgets/test\",  42 }, 43 //... 44 ] 45 } In this example, the database contains a total of 23 widget objects. If the client sends a GET request to /api/v1/widgets?page=1\u0026amp;per_page=10, the server response will contain the first ten objects from the database in the order that they were created. (i.e., items #1-10) The client can step through the full set of widget objects by continually sending GET requests and incrementing the page attribute. With page=2 and per_page=10, the server response will contain the next ten objects from the database (i.e., items #11-20). With page=3 and per_page=10, only three items will be sent in the response (i.e., items #21-23).\nHATEOAS If you\u0026rsquo;ve spent time reading about REST APIs, you have probably encountered the term HATEOAS, which stands for Hypertext As The Engine Of Application State. This is part of Roy Fielding\u0026rsquo;s formal definition of REST, and the main takeaway is that clients shouldn\u0026rsquo;t need to construct URLs in order to interact with a REST API, since this ties the client to an implementation. This is bad because any change to the structure/syntax of an existing URI is a breaking change for the client.\nIdeally, a REST API should provide links in the header or body of the response that direct the client to the next logical set of actions/resources based on the client\u0026rsquo;s request. The navigational links in the header (Lines 18-21) and body of the response (Lines 25-29) above allow the client to browse through the collection of widgets.\nThe object named \u0026ldquo;items\u0026rdquo; contains the first ten (of 23 total) widget objects. Each widget contains a \u0026ldquo;link\u0026rdquo; attribute (Line 38) containing the URI for the widget. Thanks to the link attribute and the navigational links, the client can navigate through the API and perform CRUD actions on widget instances without manually entering a single endpoint URL.\npage and per_page are query string parameters. Query string parameters can be parsed using the same techniques that we used to parse email/password values from form data in the auth_ns namespace, and widget information in the previous section.\nIf the client sends a GET request to http://localhost:5000/api/v1/widgets (i.e., neither page or per_page are sent with the request), what happens? Typically, default values are assumed for these parameters. The default values are configured when we create the RequestParser arguments for the pagination parameters.\n  While Fielding defined HATEOAS and stipulated that it is a requirement for a REST API, he did not specify the format for doing so. Depending on the project, navigational links could be provided in either the response header or body (obviously I have provided both for this demonstration). The format of the Link header field is defined in RFC 8288, which is a Proposed Standard that is widely employed throughout the internet (i.e., it\u0026rsquo;s pretty safe to use it in your application, too).\n  Retrieve Widget List With the background info regarding pagination and HATEOAS in mind, we are ready to begin implementing the API endpoint that responds to GET requests sent to /api/v1/widgets. Per Table 1, this endpoint and request type allows clients to retrieve a list of widget objects. As before, we start by creating request parsers/API models to validate request data and serialize response data.\npagination_reqparser Request Parser When a client sends a request to retrieve a list of widgets, what data should we expect to be included with the request? The answer should be fairly obvious based on the information covered in the Introduction.\nThe request to retrieve a list of widget objects should include two values: the page number and number of items per page. Luckily, both of these values are integers, and there are several pre-built types in the flask_restx.inputs module that convert request data to integer values.\nOpen src/flask_api_tutorial/api/widgets/dto.py and update the import statements to include the flask_restx.inputs.positive class (Line 6):\n1 2 3 4 5 6 7 8 9 10  \u0026#34;\u0026#34;\u0026#34;Parsers and serializers for /widgets API endpoints.\u0026#34;\u0026#34;\u0026#34; import re from datetime import date, datetime, time, timezone from dateutil import parser from flask_restx.inputs import positive, URL from flask_restx.reqparse import RequestParser from flask_api_tutorial.util.datetime_util import make_tzaware, DATE_MONTH_NAME   Next, add the content below:\n70 71 72 73 74  pagination_reqparser = RequestParser(bundle_errors=True) pagination_reqparser.add_argument(\u0026#34;page\u0026#34;, type=positive, required=False, default=1) pagination_reqparser.add_argument( \u0026#34;per_page\u0026#34;, type=positive, required=False, choices=[5, 10, 25, 50, 100], default=10 )   By specifying type=positive, the value provided in the request data will be coerced to an integer. If the value represents a positive, non-zero integer, the request will succeed and the server will send the paginated list to the client. Otherwise, the server will reject the request with status code 400 HTTPStatus.BAD_REQUEST.\n  Why did I choose the flask_restx.inputs.positive class? For both the page and per_page parameters, zero and negative values are invalid. Checking the parsed values to ensure they are positive numbers would be simple, but since a class already exists that performs the same check, IMO, it is wasteful to re-implement the same logic.\n  This is the first time that we have specified a RequestParser argument as required=False. This allows the client to send a request without either parameter and the request will still succeed (e.g., GET /api/v1/widgets will return the same response as GET /api/v1/widgets?page=1\u0026amp;per_page=10).\n  What would happen if required=True and the client sends a request without either parameter? Rather than using a default value, the server would reject the request with status code HTTPStatus.BAD_REQUEST (400) and include an error message indicating that one or more required values were missing.\n  The range of valid values for the page parameter is any positive integer. However, the per_page parameter must have an upper bound since the point of employing pagination is to prevent the API from becoming sluggish due to sending/receiving a large amount of data.\nFlask-RESTx includes a pre-built type (flask_restx.inputs.int_range) that will restrict values to a range of integers. This would allow the client to request any number of items per page, but I think it makes more sense to restrict the page size to a small, fixed set of choices.\nThe list provided to the choices keyword defines the set of allowable values. This has an additional benefit — on the Swagger UI page, the input form for per_page will render a select element containing the list of choices.\nImplementing the request parser was trivial, but constructing a response containing a list of widget objects is significantly more complicated. Keep in mind, the response must be formatted as a paginated list, including navigational links and values for the current page number, number of items per page, total number of items in the collection, etc.\nIt is possible to create the paginated list manually from scratch and define API models to serialize the whole thing to JSON (it would also be tedious). Instead of needlessly wasting time, let\u0026rsquo;s take a look at a method that will do most of the work for us.\nFlask-SQLAlchemy paginate Method The Flask-SQLAlchemy extension provides a paginate method that produces Pagination objects. The paginate method is a member of the Query class, and I think the easiest way to understand how it works is with a demonstration in the interactive shell:\n(flask-api-tutorial) flask-api-tutorial $ flask shell Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 10.0.1 (clang-1001.0.46.4)] on darwin App: flask-api-tutorial [development] Instance: /Users/aaronluna/Projects/flask_api_tutorial/instance  len(Widget.query.all()) 6  pagination = Widget.query.paginate(page=1, per_page=5)  pagination \u0026lt;flask_sqlalchemy.Pagination object at 0x10b44bf90\u0026gt; I created six Widget instances in my test environment, which is verified by the first statement len(Widget.query.all()) returning a result of 6. Next, we create a Pagination object for the first page of Widget objects with five items per page by calling Widget.query.paginate(page=1, per_page=5). The last statement verifies that we did, in fact, create a Pagination object.\n  I recommend reading the SQLAlchemy documentation for the Query object, as well as the Flask-SQLAlchemy documentation for the Pagination object and the paginate method.\n   pagination.pages 2  pagination.per_page 5  pagination.total 6 These three attributes will remain the same for all pages with the same per_page value. pages is the total number of pages, per_page is (obviously) the number of items on a single page, and total is the total number of items in the collection.\n pagination.page 1  pagination.has_next True  pagination.has_prev False  len(pagination.items) 5  pagination.items [\u0026lt;Widget name=first_widget, info_url=https://www.widgethost.net\u0026gt;, \u0026lt;Widget name=next, info_url=https://www.thisorthat.feh\u0026gt;, \u0026lt;Widget name=another, info_url=https://www.wontbelong.now\u0026gt;, \u0026lt;Widget name=foo, info_url=https://www.foo.bar\u0026gt;, \u0026lt;Widget name=baz, info_url=https://www.baz.bar\u0026gt;] Next, we call pagination.page to verify that the page number matches the value specified in the paginate method. Since we specified per_page=5 and there are six total items we know that there are two total pages, which is confirmed by the value of pagination.has_next being True. Also, since this is the first page we know that there is no previous page, which is why pagination.has_prev is False.\nlen(pagination.items) verifies that this page contains five widgets. Finally, we inspect the widget list directly by executing pagination.items. As expected, a list containing five Widget instances is returned.\n pagination.items[0].name 'first_widget'  pagination.items[0].owner \u0026lt;User email=admin@test.com, public_id=475807a4-8497-4c5c-8d70-109b429bb4ef, admin=True\u0026gt;  pagination.items[0].owner.email 'admin@test.com' Let\u0026rsquo;s take a look at pagination.items[0], the first widget added to the database. First, the name attribute is checked, followed by owner. owner contains a User object that corresponds to the user that created this Widget. The create_widget function (which performs the process of creating a widget after the request has been fully validated) stores the id of the User that sent the request in the Widget.owner_id attribute.\nowner_id is defined as a SQLAlchemy Column to which a ForeignKey construct has been applied and this integer value is stored in the widget database table. Widget.owner is defined as a SQLAlchemy relationship between the Widget table and the User table, and is not stored in the database. Whenever a Widget object is retrieved from the database, Widget.owner is populated with a User object thanks to the foreign-key relationship and the magic of the SQLAlchemy ORM.\n pagination = Widget.query.paginate(page=2, per_page=5)  pagination.page 2  pagination.has_next False  pagination.has_prev True  len(pagination.items) 1  pagination.items [\u0026lt;Widget name=bim, info_url=http://www.baz.bar\u0026gt;] Finally, we retrieve the second (and final) page of Widget objects with five items per page by calling Widget.query.paginate(page=2, per_page=5). We then verify that this is, in fact, the second page by calling pagination.page. We know pagination.has_next should be False since this is the final page of Widgets, and pagination.has_prev should be True. len(pagination.items) is one since there are six total Widgets and items #1-5 were shown on page=1. Lastly, we verify that pagination.items contains a single Widget object.\nHopefully, this helps you understand the structure of the Pagination class and the behavior of the paginate method. Understanding both is crucial to implementing the api.widget_list endpoint. Next, we need to create an API model for the Pagination class which will be considerably more complex than the API model we created for the User class.\npagination_model API Model In order to send a paginated list of widgets as part of an HTTP response, we need to serialize it to JSON. I explained the purpose of API Models and how Flask-RESTx uses them to serialize database objects in Part 4. If you need a refresher, please review it.\nFirst, we need to update the import statements in src/flask_api_tutorial/api/widgets/dto.py to include the Flask-RESTx Model class, as well as a bunch of classes from the fields module . Add Line 6 and Line 7 and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12  \u0026#34;\u0026#34;\u0026#34;Parsers and serializers for /widgets API endpoints.\u0026#34;\u0026#34;\u0026#34; import re from datetime import date, datetime, time, timezone from dateutil import parser from flask_restx import Model from flask_restx.fields import Boolean, DateTime, Integer, List, Nested, String, Url from flask_restx.inputs import positive, URL from flask_restx.reqparse import RequestParser from flask_api_tutorial.util.datetime_util import make_tzaware, DATE_MONTH_NAME   Next, add the content below:\n78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112  widget_owner_model = Model(\u0026#34;Widget Owner\u0026#34;, {\u0026#34;email\u0026#34;: String, \u0026#34;public_id\u0026#34;: String}) widget_model = Model( \u0026#34;Widget\u0026#34;, { \u0026#34;name\u0026#34;: String, \u0026#34;info_url\u0026#34;: String, \u0026#34;created_at_iso8601\u0026#34;: DateTime(attribute=\u0026#34;created_at\u0026#34;), \u0026#34;created_at_rfc822\u0026#34;: DateTime(attribute=\u0026#34;created_at\u0026#34;, dt_format=\u0026#34;rfc822\u0026#34;), \u0026#34;deadline\u0026#34;: String(attribute=\u0026#34;deadline_str\u0026#34;), \u0026#34;deadline_passed\u0026#34;: Boolean, \u0026#34;time_remaining\u0026#34;: String(attribute=\u0026#34;time_remaining_str\u0026#34;), \u0026#34;owner\u0026#34;: Nested(widget_owner_model), \u0026#34;link\u0026#34;: Url(\u0026#34;api.widget\u0026#34;), }, ) pagination_links_model = Model( \u0026#34;Nav Links\u0026#34;, {\u0026#34;self\u0026#34;: String, \u0026#34;prev\u0026#34;: String, \u0026#34;next\u0026#34;: String, \u0026#34;first\u0026#34;: String, \u0026#34;last\u0026#34;: String}, ) pagination_model = Model( \u0026#34;Pagination\u0026#34;, { \u0026#34;links\u0026#34;: Nested(pagination_links_model, skip_none=True), \u0026#34;has_prev\u0026#34;: Boolean, \u0026#34;has_next\u0026#34;: Boolean, \u0026#34;page\u0026#34;: Integer, \u0026#34;total_pages\u0026#34;: Integer(attribute=\u0026#34;pages\u0026#34;), \u0026#34;items_per_page\u0026#34;: Integer(attribute=\u0026#34;per_page\u0026#34;), \u0026#34;total_items\u0026#34;: Integer(attribute=\u0026#34;total\u0026#34;), \u0026#34;items\u0026#34;: List(Nested(widget_model)), }, )   There is a lot to digest here. This is the first time that we are encountering API models that are composed of other API models. pagination_model contains a list of widget_model objects as well as a single pagination_links_model instance, Also, widget_model contains a single instance of widget_owner_model. Let\u0026rsquo;s take a look at how each API model is defined and how they interact with each other.\nwidget_owner_model and widget_model Let\u0026rsquo;s work our way from the inside-out. As demonstrated in the interactive shell, the structure of a Pagination object is:\npagination - items - widget - owner owner is a User object. Previously, we created the user_model API model to serialize a User object to JSON. This API model exposes every attribute of the User class, most of which are unnecessary in this context.\nRather than re-using user_model, we will create widget_owner_model which exposes only the email and public_id values of the User object:\nwidget_owner_model = Model( \u0026#34;Widget Owner\u0026#34;, { \u0026#34;email\u0026#34;: String, \u0026#34;public_id\u0026#34;: String, }, ) The widget_model has a bunch of features that we are seeing for the first time. Let\u0026rsquo;s take a look at it in more depth:\n80 81 82 83 84 85 86 87 88 89 90 91 92 93 94  widget_model = Model( \u0026#34;Widget\u0026#34;, { \u0026#34;name\u0026#34;: String, \u0026#34;info_url\u0026#34;: String, \u0026#34;created_at\u0026#34;: String(attribute=\u0026#34;created_at_str\u0026#34;), \u0026#34;created_at_iso8601\u0026#34;: DateTime(attribute=\u0026#34;created_at\u0026#34;), \u0026#34;created_at_rfc822\u0026#34;: DateTime(attribute=\u0026#34;created_at\u0026#34;, dt_format=\u0026#34;rfc822\u0026#34;), \u0026#34;deadline\u0026#34;: String(attribute=\u0026#34;deadline_str\u0026#34;), \u0026#34;deadline_passed\u0026#34;: Boolean, \u0026#34;time_remaining\u0026#34;: String(attribute=\u0026#34;time_remaining_str\u0026#34;), \u0026#34;owner\u0026#34;: Nested(widget_owner_model), \u0026#34;link\u0026#34;: Url(\u0026#34;api.widget\u0026#34;), }, )     Lines 86-87: This is the first time that we are using the flask_restx.fields.DateTime class, which formats a datetime value as a string. There are two supported formats: RFC 822 and ISO 8601. The format which is returned is determined by the dt_format parameter.\nBy default, ISO 8601 format is used. Since dt_format is not specified, created_at_iso8601 will use this format. On the other hand, created_at_rfc822 specifies dt_format=\"rfc822\" so the same date will be returned using RFC 822 format.\nWhat do these two formats look like? Here's an example:\n\"created_at_iso8601\": \"2019-09-20T04:47:50\", \"created_at_rfc822\": \"Fri, 20 Sep 2019 04:47:50 -0000\", The benefit of using a standard output format is that it can be easily parsed back to the original datetime value. If this is not a requirement and you (like me) find these formats difficult to quickly parse visually, there are other ways to format datetime values within an API model.\n  Line 88: deadline_str is a formatted string version of deadline, which is a datetime value. Since deadline is localized to UTC, deadline_str converts this value to the local time zone where the code is executed.\nHere's an example of the format used by deadline_str:\n\"deadline\": \"09/20/19 10:59:59 PM UTC-08:00\", I prefer this style of formatting to either ISO 8601 or RFC 822 format since it is localized to the user's timezone and is (IMO) more readable. However, if the datetime value will not be read by humans and/or will be provided to a function expecting either ISO 8601 or RFC 822 format, obviously use the built-in flask_restx.fields.Datetime class.\n  Line 89: We used the Boolean class already (in the user_model API model), so refer back to that section if you need to review how it works.\n  Line 90: time_remaining_str is a formatted string version of time_remaining, which is a timedelta value. Since Flask-RESTx does not include built-in types for serializing timedelta values, formatting time_remaining as a string is the only way to include it in the serialized JSON.\nHere's an example of the format used by time_remaining_str:\n\"time_remaining\": \"16 hours 41 minutes 42 seconds\",   Line 91: In order to serialize a Widget object and preserve the structure where owner is a User object nested within the parent Widget object, we use the Nested class in conjunction with the widget_owner_model.\nHere's what the widget_owner_model will look like within the parent Widget object:\n\"owner\": { \"email\": \"admin@test.com\", \"public_id\": \"475807a4-8497-4c5c-8d70-109b429bb4ef\", }   For more information and examples of serializing complex structures to JSON, please read the Complex Structures and Nested Field sections of the Flask-RESTx documentation.\n    Line 92: The Widget class doesn't contain an attribute named link, so what's going on here? I think the best explanation of the fields.Url class is given in the Flask-RESTx documentation:\nFlask-RESTx includes a special field, fields.Url, that synthesizes a uri for the resource that’s being requested. This is also a good example of how to add data to your response that’s not actually present on your data object. By including a link to the URI with each Widget, the client can perform CRUD actions without manually constructing or storing the URI (which is an example of HATEOAS). By default, the value returned for link will be a relative URI as shown below:\n\"link\": \"/api/v1/widgets/first_widget\", If the link should be an absolute URI (containing scheme, hostname, and port), include the keyword argument absolute=True (e.g., Url(\"api.widget\", absolute=True)). In my local test environment, this returns the URI below for the same Widget resource:\n\"link\": \"http://localhost:5000/api/v1/widgets/first_widget\",   The implementation of the Url field in widget_model relies on the api.widget endpoint, which we haven\u0026rsquo;t created at this point. This will result in an unhandled exception until the api.widget endpoint has been fully implemented.\n     I hope that all of the material we encountered for the first time in the widget_model and widget_owner_model was easy to understand. The remaining API models are much simpler, and contain only a few small bits of new information. If you\u0026rsquo;re comfortable with all of the information we just covered, let\u0026rsquo;s move on and finish creating the pagination_model.\npagination_links_model and pagination_model If you go back and look at the pagination example in the Introduction , there\u0026rsquo;s something important that is included in the example that is not part of the Flask-RESTx Pagination object. Here\u0026rsquo;s a hint: it has to do with HATEOAS. The answer is: navigational links:\npagination_links_model = Model( \u0026#34;Nav Links\u0026#34;, { \u0026#34;self\u0026#34;: String, \u0026#34;prev\u0026#34;: String, \u0026#34;next\u0026#34;: String, \u0026#34;first\u0026#34;: String, \u0026#34;last\u0026#34;: String, }, ) Since all of the fields in pagination_links_model are serialized using the String class, there\u0026rsquo;s nothing new to discuss in that regard.\nFinally, the widget_model and pagination_links_model are integrated into the pagination_model:\n101 102 103 104 105 106 107 108 109 110 111 112 113  pagination_model = Model( \u0026#34;Pagination\u0026#34;, { \u0026#34;links\u0026#34;: Nested(pagination_links_model, skip_none=True), \u0026#34;has_prev\u0026#34;: Boolean, \u0026#34;has_next\u0026#34;: Boolean, \u0026#34;page\u0026#34;: Integer, \u0026#34;total_pages\u0026#34;: Integer(attribute=\u0026#34;pages\u0026#34;), \u0026#34;items_per_page\u0026#34;: Integer(attribute=\u0026#34;per_page\u0026#34;), \u0026#34;total_items\u0026#34;: Integer(attribute=\u0026#34;total\u0026#34;), \u0026#34;items\u0026#34;: List(Nested(widget_model)), }, )   There are only a few things in pagination_model that we are seeing for the first time:\n  Line 104: In order to match the object structure shown in the Introduction, pagination_model must have a field named links containing navigational links that allow the client to access all Widget instances available in the database.\nNotice that we are using the Nested field type the same way we did in widget_model, the only difference is that now we are including the keyword argument skip_none=True. As explained in the Flask-RESTx documentation, by default, if any of the fields in pagination_links_model have value None, the JSON output will contain these fields with value null.\nThere are many situations where one or more of the navigational links will be None (e.g., for the first page of results prev will always be None). By specifying skip_none=True, these values will not be rendered in the JSON output, making it much cleaner and reducing the size of the response.\n  Lines 108-110: This isn't new information, I just want to point out that I have renamed these three fields to be more descriptive. IMHO, total_pages is a better name than pages, and the same thing goes for items_per_page/per_page, as well as total_items/total.\n  Line 111: We have already seen examples using the Nested type, which allows us to create complex API models which are composed of built-in types, custom types, and other API models. However, in order to serialize the most important part of the pagination object, we need a way to marshal a list of widget objects to JSON.\nThis is easily accomplished using the List type in conjunction with the Nested type. For more information on the List type, refer to the Flask-RESTx documentation for Response Marshalling.\n   pagination_model JSON Example We finally covered everything necessary to create the pagination_model API model. Here\u0026rsquo;s an example of the JSON that our API model will produce given a pagination object with page=2, per_page=5, and total=7:\n{ \u0026#34;links\u0026#34;: { \u0026#34;self\u0026#34;: \u0026#34;/api/v1/widgets?page=2\u0026amp;per_page=5\u0026#34;, \u0026#34;first\u0026#34;: \u0026#34;/api/v1/widgets?page=1\u0026amp;per_page=5\u0026#34;, \u0026#34;prev\u0026#34;: \u0026#34;/api/v1/widgets?page=1\u0026amp;per_page=5\u0026#34;, \u0026#34;last\u0026#34;: \u0026#34;/api/v1/widgets?page=2\u0026amp;per_page=5\u0026#34; }, \u0026#34;has_prev\u0026#34;: true, \u0026#34;has_next\u0026#34;: false, \u0026#34;page\u0026#34;: 2, \u0026#34;total_pages\u0026#34;: 2, \u0026#34;items_per_page\u0026#34;: 5, \u0026#34;total_items\u0026#34;: 7, \u0026#34;items\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;foo\u0026#34;, \u0026#34;info_url\u0026#34;: \u0026#34;https://www.foo.bar\u0026#34;, \u0026#34;created_at\u0026#34;: \u0026#34;11/18/19 06:07:20 AM UTC-08:00\u0026#34;, \u0026#34;created_at_iso8601\u0026#34;: \u0026#34;2019-11-18T14:07:20\u0026#34;, \u0026#34;created_at_rfc822\u0026#34;: \u0026#34;Mon, 18 Nov 2019 14:07:20 -0000\u0026#34;, \u0026#34;deadline\u0026#34;: \u0026#34;11/19/19 11:59:59 PM UTC-08:00\u0026#34;, \u0026#34;deadline_passed\u0026#34;: true, \u0026#34;time_remaining\u0026#34;: \u0026#34;No time remaining\u0026#34;, \u0026#34;owner\u0026#34;: { \u0026#34;email\u0026#34;: \u0026#34;admin@test.com\u0026#34;, \u0026#34;public_id\u0026#34;: \u0026#34;475807a4-8497-4c5c-8d70-109b429bb4ef\u0026#34; }, \u0026#34;link\u0026#34;: \u0026#34;/api/v1/widgets/foo\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;new-test-555\u0026#34;, \u0026#34;info_url\u0026#34;: \u0026#34;http://www.newtest.net\u0026#34;, \u0026#34;created_at\u0026#34;: \u0026#34;12/01/19 09:45:26 AM UTC-08:00\u0026#34;, \u0026#34;created_at_iso8601\u0026#34;: \u0026#34;2019-12-01T17:45:26\u0026#34;, \u0026#34;created_at_rfc822\u0026#34;: \u0026#34;Sun, 01 Dec 2019 17:45:26 -0000\u0026#34;, \u0026#34;deadline\u0026#34;: \u0026#34;12/02/19 11:59:59 PM UTC-08:00\u0026#34;, \u0026#34;deadline_passed\u0026#34;: false, \u0026#34;time_remaining\u0026#34;: \u0026#34;1 day 14 hours 12 minutes 46 seconds\u0026#34;, \u0026#34;owner\u0026#34;: { \u0026#34;email\u0026#34;: \u0026#34;admin@test.com\u0026#34;, \u0026#34;public_id\u0026#34;: \u0026#34;475807a4-8497-4c5c-8d70-109b429bb4ef\u0026#34; }, \u0026#34;link\u0026#34;: \u0026#34;/api/v1/widgets/new-test-555\u0026#34; } ] } retrieve_widget_list Method Next, we need to create a function that performs the following actions:\n Create a pagination object given the page and per_page values parsed from the request data. Serialize the pagination object to JSON using pagination_model and the flask_restx.marshal method. Construct dict of navigational links and add links to response header and body. Manually construct HTTP response using the flask.jsonify method and send response to client.  Before we begin, open src/flask_api_tutorial/api/widgets/business.py and make the following updates to the import statements:\n1 2 3 4 5 6 7 8 9 10 11 12  \u0026#34;\u0026#34;\u0026#34;Business logic for /widgets API endpoints.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask import jsonify, url_for from flask_restx import abort, marshal  from flask_api_tutorial import db from flask_api_tutorial.api.auth.decorators import token_required, admin_token_required from flask_api_tutorial.api.widgets.dto import pagination_model from flask_api_tutorial.models.user import User from flask_api_tutorial.models.widget import Widget     Line 4: Update to include the flask.url_for method.\n  Line 5: Update to include the flask_restx.marshal method.\n  Line 8: Update to include the token_required decorator.\n  Line 9: Update to include the pagination_model object.\n   Next, add the content below:\n31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67  @token_required def retrieve_widget_list(page, per_page): pagination = Widget.query.paginate(page, per_page, error_out=False) response_data = marshal(pagination, pagination_model) response_data[\u0026#34;links\u0026#34;] = _pagination_nav_links(pagination) response = jsonify(response_data) response.headers[\u0026#34;Link\u0026#34;] = _pagination_nav_header_links(pagination) response.headers[\u0026#34;Total-Count\u0026#34;] = pagination.total return response def _pagination_nav_links(pagination): nav_links = {} per_page = pagination.per_page this_page = pagination.page last_page = pagination.pages nav_links[\u0026#34;self\u0026#34;] = url_for(\u0026#34;api.widget_list\u0026#34;, page=this_page, per_page=per_page) nav_links[\u0026#34;first\u0026#34;] = url_for(\u0026#34;api.widget_list\u0026#34;, page=1, per_page=per_page) if pagination.has_prev: nav_links[\u0026#34;prev\u0026#34;] = url_for( \u0026#34;api.widget_list\u0026#34;, page=this_page - 1, per_page=per_page ) if pagination.has_next: nav_links[\u0026#34;next\u0026#34;] = url_for( \u0026#34;api.widget_list\u0026#34;, page=this_page + 1, per_page=per_page ) nav_links[\u0026#34;last\u0026#34;] = url_for(\u0026#34;api.widget_list\u0026#34;, page=last_page, per_page=per_page) return nav_links def _pagination_nav_header_links(pagination): url_dict = _pagination_nav_links(pagination) link_header = \u0026#34;\u0026#34; for rel, url in url_dict.items(): link_header += f\u0026#39;\u0026lt;{url}\u0026gt;; rel=\u0026#34;{rel}\u0026#34;, \u0026#39; return link_header.strip().strip(\u0026#34;,\u0026#34;)   This code implements the process of responding to a valid request for a list of widgets, please note the following:\n  Line 31: Per Table 1, the process of retrieving a list of widgets can only be performed by registered users (both regular and admin users). This is enforced by decorating the retrieve_widget_list function with @token_required.\n  Line 32: The page and per_page parameters are passed to retrieve_widget_list after the pagination_model has parsed the values provided by the client from the request data.\n  Line 33: As demonstrated in the Python interactive shell, pagination objects are created by calling Widget.query.paginate, with the page and per_page values provided by the client.\n  Line 34: This is the first time that we are seeing the flask_restx.marshal function. However, we already know how it works since we used the @marshall_with decorator in Part 4.\nThere is only a single, subtle difference between these two functions/decorators. Both operate on an object and filter the object's attributes/keys against the provided API model and validate the object's data against the set of fields configured in the API model.\nHowever, @marshal_with operates on the value returned from the function it decorates, while flask_restx.marshal operates on whatever is passed to the function as the first parameter. So why are we calling the marshal function directly? In Lines 37-38 custom header fields are added to the response before it is sent to the client, and there is no way to add these headers using @marshal_with.\n  Line 35: Remember, the output of the marshal function is a dict. Also remember that the flask_sqlalchemy.Pagination class does not contain navigational links. We will discuss the _pagination_nav_links function shortly, but what is important to know is that it returns a dict that matches the fields in pagination_links_model. This dict is then added to the pagination object with key-name links, which matches the field on pagination_model containing Nested(pagination_links_model, skip_none=True).\n  Line 36: After adding the navigational links to the pagination object, it is ready to send to the client. We discussed the flask.jsonify function in Part 3, please review it if you are drawing a blank trying to remember what it does.\nTL;DR, calling jsonify(response_data) converts response_data (which is a dict object) to JSON and returns a flask.Response object with the JSON object as the response body.\n  Line 37: Refer back to the pagination section, and note that the example includes navigational links in both the JSON response body AND the Link field in the response header. We will discuss the _pagination_nav_header_links function very soon, but what is important to know is that it returns a string containing all valid page navigation URLs in the format specified for the Link Header Field defined in RFC 8288.\n  Line 38: This isn't an official best practice, but it is very common to include other metadata about the paginated list in the response header. Here, we create a field named Total-Count that contains the total number of Widget objects in the database.\n  Line 39: After the response object is fully configured, we return it from the retrieve_widget_list function before sending it to the client.\n  Lines 42-58: The _pagination_nav_links function accepts a single parameter which is assumed to be a Pagination instance and returns a dict object named nav_links that matches the fields in pagination_links_model. By default, nav_links contains navigation URLs for self, first, and last pages (even if the total number of pages is one and all navigation URLs are the same). prev and next navigation URLs are not included by default. If pagination.has_prev is True, then the prev page URL is included (accordingly, next is included if pagination.has_next is True).\n  Lines 61-66: Finally, the _pagination_nav_header_links function also accepts a single parameter which is assumed to be a Pagination instance, but instead of a dict object a string is returned containing all valid page navigation URLs in the correct format for the Link header field.\nThis function calls _pagination_nav_links and uses the dict that is returned to generate the Link header field. This works because the keys of the dict object are the same as the Link field's rel parameter and the dict values are the page navigation URLs.\n   Now that the business logic has been implemented, we can add a method to the api.widget_list endpoint that will call retrieve_widget_list after parsing/validating the request data.\napi.widget_list Endpoint (GET Request) We created the api.widget_list endpoint in Part 5 and implemented the function that handles POST requests. According to Table 1, this endpoint also supports GET requests which allows clients to retrieve lists of widgets.\nOpen src/flask_api_tutorial/api/widgets/endpoints.py and make the following updates to the import statements:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15  \u0026#34;\u0026#34;\u0026#34;API endpoint definitions for /widgets namespace.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_restx import Namespace, Resource from flask_api_tutorial.api.widgets.dto import ( create_widget_reqparser, pagination_reqparser, widget_owner_model, widget_model, pagination_links_model, pagination_model, ) from flask_api_tutorial.api.widgets.business import create_widget, retrieve_widget_list      Line 8: Update to include the pagination_reqparser object.\n  Lines 9-12: Update to include the widget_owner_model, widget_model, pagination_links_model, and pagination_model API models.\n  Line 14: Update to include the retrieve_widget_list function.\n   Next, add the highlighted lines to endpoints.py and save the file:\n16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49  widget_ns = Namespace(name=\u0026#34;widgets\u0026#34;, validate=True) widget_ns.models[widget_owner_model.name] = widget_owner_model widget_ns.models[widget_model.name] = widget_model widget_ns.models[pagination_links_model.name] = pagination_links_model widget_ns.models[pagination_model.name] = pagination_model  @widget_ns.route(\u0026#34;\u0026#34;, endpoint=\u0026#34;widget_list\u0026#34;) @widget_ns.response(int(HTTPStatus.BAD_REQUEST), \u0026#34;Validation error.\u0026#34;) @widget_ns.response(int(HTTPStatus.UNAUTHORIZED), \u0026#34;Unauthorized.\u0026#34;) @widget_ns.response(int(HTTPStatus.INTERNAL_SERVER_ERROR), \u0026#34;Internal server error.\u0026#34;) class WidgetList(Resource): \u0026#34;\u0026#34;\u0026#34;Handles HTTP requests to URL: /widgets.\u0026#34;\u0026#34;\u0026#34; @widget_ns.doc(security=\u0026#34;Bearer\u0026#34;) @widget_ns.response(int(HTTPStatus.OK), \u0026#34;Retrieved widget list.\u0026#34;, pagination_model) @widget_ns.expect(pagination_reqparser) def get(self): \u0026#34;\u0026#34;\u0026#34;Retrieve a list of widgets.\u0026#34;\u0026#34;\u0026#34; request_data = pagination_reqparser.parse_args() page = request_data.get(\u0026#34;page\u0026#34;) per_page = request_data.get(\u0026#34;per_page\u0026#34;) return retrieve_widget_list(page, per_page)  @widget_ns.doc(security=\u0026#34;Bearer\u0026#34;) @widget_ns.response(int(HTTPStatus.CREATED), \u0026#34;Added new widget.\u0026#34;) @widget_ns.response(int(HTTPStatus.FORBIDDEN), \u0026#34;Administrator token required.\u0026#34;) @widget_ns.response(int(HTTPStatus.CONFLICT), \u0026#34;Widget name already exists.\u0026#34;) @widget_ns.expect(create_widget_reqparser) def post(self): \u0026#34;\u0026#34;\u0026#34;Create a widget.\u0026#34;\u0026#34;\u0026#34; widget_dict = create_widget_reqparser.parse_args() return create_widget(widget_dict)     Lines 17-20: We need to register all of the API models that we created in app.api.widgets.dto with the widget_ns namespace. This is an easy thing to forget and can be a difficult issue to debug. If the API models are not registered with the widget_ns namespace, the entire Swagger UI page will fail to render, displaying only a single cryptic error message: No API definition provided.\n  Line 31: The response decorator can be configured with an API model as an optional third argument. This has no effect on the resource's behavior, but the API model is displayed on the Swagger UI page with response code 200 as an example response body.\n  Line 32: The expect decorator was explained in depth in Part 3, please review the implementation of the function that handles POST requests for the api.auth_register endpoint if you need a refresher. Basically, applying the decorator @widget_ns.expect(create_widget_reqparser) to a function has two enormous effects: it specifies that create_widget_reqparser will be used to parse the client's request, AND it renders a form on the Swagger UI page with input text elements for widget.name, widget.info_url, and widget.deadline.\n  Lines 35-38: Everything here should be completely obvious to you since calling parse_args to get the client's request data and passing the data to our business logic is a process that we implement on (nearly) every API handler.\n   We can verify that the api.widget_list endpoint now supports both GET and POST requests by executing the flask routes command:\nflask-api-tutorial $ flask routes Endpoint Methods Rule ------------------- --------- -------------------------- api.auth_login POST /api/v1/auth/login api.auth_logout POST /api/v1/auth/logout api.auth_register POST /api/v1/auth/register api.auth_user GET /api/v1/auth/user api.doc GET /api/v1/ui api.root GET /api/v1/ api.specs GET /api/v1/swagger.json api.widget_list GET, POST /api/v1/widgets restplus_doc.static GET /swaggerui/\u0026lt;path:filename\u0026gt; static GET /static/\u0026lt;path:filename\u0026gt;   Please remember, currently an unhandled exception occurs if you attempt to execute either of the methods we have created for the api.widget_list endpoint since the business logic for both operations depends on the api.widget endpoint existing. We will create unit tests for all endpoints/CRUD operations in Table 1 when they have been fully implemented.\n  Retrieve Widget At this point, we have completed two of the five CRUD processes specified in Table 1. Thankfully, the remaining three should be much simpler and faster to implement since many of the elements can be reused (e.g. request parsers, API models).\nThe two processes we completed (create a widget, retrieve a list of widgets) are accessed via the /api/v1/widgets endpoint, which is also called the resource collection endpoint. However, the remaining three processes are accessed via the resource item endpoint, api/v1/widgets/\u0026lt;name\u0026gt;. The name parameter is provided by the client and is used to perform the requested action (retrieve, update, delete) on a specific widget instance.\nLet\u0026rsquo;s start with the process that is most similar to the last one we implemented, retrieving a single widget. We do not need to create any request parsers or API models since the name of the widget is provided in the URI requested by the client, and the widget_model can be re-used to serialize the requested object in the response.\nretrieve_widget Method The business logic for retrieving a single widget is very simple. First, the database is queried for widgets matching the name that was requested by the client. If a matching widget is found, it is serialized to JSON and sent to the client with status code 200 (HTTPStatus.OK). If no widget exists with the specified name, a 404 (HTTPStatus.NOT_FOUND) response is sent to the client.\nBecause this is such a common pattern in web applications, Flask-SQLAlchemy provides the first_or_404 method which does exactly what we need. It is modeled after the first function from SQLAlchemy, which returns either the first result of a query or None. first_or_404 raises a 404 error instead of returning None.\nOpen src/flask_api_tutorial/api/widgets/business.py and add the function below:\n42 43 44 45 46  @token_required def retrieve_widget(name): return Widget.query.filter_by(name=name.lower()).first_or_404( description=f\u0026#34;{name} not found in database.\u0026#34; )   This operation requires a valid access token, so the @token_required decorator is applied to the function (Line 42). The first_or_404 method (Line 44) accepts an optional description parameter which is used to include a message in the body of the HTTP response explaining why the request failed. Also, please note that the name value provided by the client is converted to lowercase before searching.\napi.widget Endpoint (GET Request) Next, we need to create the api.widget endpoint. Before we do so, open src/flask_api_tutorial/api/widgets/endpoints.py and update the import statements to include the retrieve_widget function (Line 17):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18  \u0026#34;\u0026#34;\u0026#34;API endpoint definitions for /widgets namespace.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_restx import Namespace, Resource from flask_api_tutorial.api.widgets.dto import ( create_widget_reqparser, pagination_reqparser, widget_owner_model, widget_model, pagination_links_model, pagination_model, ) from flask_api_tutorial.api.widgets.business import ( create_widget, retrieve_widget_list, retrieve_widget )   Next, add the content below:\n55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70  @widget_ns.route(\u0026#34;/\u0026lt;name\u0026gt;\u0026#34;, endpoint=\u0026#34;widget\u0026#34;) @widget_ns.param(\u0026#34;name\u0026#34;, \u0026#34;Widget name\u0026#34;) @widget_ns.response(int(HTTPStatus.BAD_REQUEST), \u0026#34;Validation error.\u0026#34;) @widget_ns.response(int(HTTPStatus.NOT_FOUND), \u0026#34;Widget not found.\u0026#34;) @widget_ns.response(int(HTTPStatus.UNAUTHORIZED), \u0026#34;Unauthorized.\u0026#34;) @widget_ns.response(int(HTTPStatus.INTERNAL_SERVER_ERROR), \u0026#34;Internal server error.\u0026#34;) class Widget(Resource): \u0026#34;\u0026#34;\u0026#34;Handles HTTP requests to URL: /widgets/{name}.\u0026#34;\u0026#34;\u0026#34; @widget_ns.doc(security=\u0026#34;Bearer\u0026#34;) @widget_ns.response(int(HTTPStatus.OK), \u0026#34;Retrieved widget.\u0026#34;, widget_model) @widget_ns.marshal_with(widget_model) def get(self, name): \u0026#34;\u0026#34;\u0026#34;Retrieve a widget.\u0026#34;\u0026#34;\u0026#34; return retrieve_widget(name)   The only thing that we are seeing for the first time is how to include a parameter in the endpoint path. Thankfully, Flask-RESTx uses the same process as Flask for URL route registration (via the @route decorator). From the Flask documentation:\n Variable parts in the route can be specified with angular brackets (/user/\u0026lt;username\u0026gt;). By default a variable part in the URL accepts any string without a slash however a different converter can be specified as well by using /\u0026lt;converter:name\u0026gt;.\nVariable parts are passed to the view function as keyword arguments.\nThe following converters are available:\n  string accepts any text without a slash (the default)   int accepts integers   float like int but for floating point values   path like the default but also accepts slashes   int matches one of the items provided   uuid accepts UUID strings    For the api.widget endpoint, the name parameter is documented in Lines 56-57 and is then passed to the get method as a parameter in Line 68.\nWe can verify that the api.widget endpoint has been registered and currently only responds to GET requests by executing the flask routes command:\nflask-api-tutorial $ flask routes Endpoint Methods Rule ------------------- --------- -------------------------- api.auth_login POST /api/v1/auth/login api.auth_logout POST /api/v1/auth/logout api.auth_register POST /api/v1/auth/register api.auth_user GET /api/v1/auth/user api.doc GET /api/v1/ui api.root GET /api/v1/ api.specs GET /api/v1/swagger.json api.widget GET /api/v1/widgets/\u0026lt;name\u0026gt; api.widget_list GET, POST /api/v1/widgets restplus_doc.static GET /swaggerui/\u0026lt;path:filename\u0026gt; static GET /static/\u0026lt;path:filename\u0026gt; Update Widget Working our way through Table 1, the next process to implement is updating a single widget. Clients with administrator access can perform this operation by sending a PUT request to the api.widget endpoint.\nI\u0026rsquo;d like you to try and imagine the business logic that the server should perform in this situation. Did you imagine something similar to: retrieve the widget whose name matches the URI requested by the client, parse the request data containing the updated data/values, and modify the attributes of the widget using the parsed values?\nThis is certainly a sensible, common sense answer. And the function that we implement will do the same thing in certain situations. However, RFC 7231 (HTTP/1.1 Semantics and Content) contains the formal specification for the PUT method. I\u0026rsquo;ve excerpted some of the most important parts of the spec below:\n4.3.4. PUT The PUT method requests that the state of the target resource be created or replaced with the state defined by the representation enclosed in the request message payload. ...\nIf the target resource does not have a current representation and the PUT successfully creates one, then the origin server MUST inform the user agent by sending a 201 (Created) response. If the target resource does have a current representation and that representation is successfully modified in accordance with the state of the enclosed representation, then the origin server MUST send either a 200 (OK) or a 204 (No Content) response to indicate successful completion of the request ....\nAn origin server SHOULD verify that the PUT representation is consistent with any constraints the server has for the target resource that cannot or will not be changed by the PUT. ....\nThe fundamental difference between the POST and PUT methods is highlighted by the different intent for the enclosed representation. The target resource in a POST request is intended to handle the enclosed representation according to the resource's own semantics, whereas the enclosed representation in a PUT request is defined as replacing the state of the target resource. Hence, the intent of PUT is idempotent and visible to intermediaries, even though the exact effect is only known by the origin server.\n I know, the language is highly technical and much more complex than the answer to the hypothetical question I posed. Fear not, implementing the PUT method per the specification doesn\u0026rsquo;t require much additional effort.\nupdate_widget_reqparser Request Parser The request data sent by the client for a PUT request is nearly identical to the data sent for a POST request, with one important difference. With a PUT request, the name parameter is provided in the URI instead of the body of the request. Because of this, we can\u0026rsquo;t just re-use the create_widget_reqparser as-is to parse a PUT request.\nIt turns out that the need to re-use portions of a request parser is such a common occurrence that Flask-RESTx provides methods to copy an existing parser, and then add/remove arguments. This is extremely useful since it obviates the need to re-write every argument and duplicate a bunch of code just to slightly tweak the behavior of a request parser.\nFor example, open src/flask_api_tutorial/api/widgets/dto.py, add the lines below, then save the file:\n72 73 74  update_widget_reqparser = create_widget_reqparser.copy() update_widget_reqparser.remove_argument(\u0026#34;name\u0026#34;)   Now we have exactly the request parser that we need for PUT requests received at the api.widgets endpoint. The update_widget_reqparser is created by simply copying the create_widget_reqparser and removing the name argument. We will import and use this when we are ready to implement the put method handler.\nupdate_widget Method Next, we need to create the business logic that implements the PUT method as specified in RFC 7231.\nFirst, open src/flask_api_tutorial/api/widgets/business.py and update the import statements to include the widget_name function from the app.api.widgets.dto module (Line 9):\n1 2 3 4 5 6 7 8 9 10 11 12  \u0026#34;\u0026#34;\u0026#34;Business logic for /widgets API endpoints.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask import jsonify, url_for from flask_restx import abort, marshal from flask_api_tutorial import db from flask_api_tutorial.api.auth.decorators import token_required, admin_token_required from flask_api_tutorial.api.widgets.dto import pagination_model, widget_name from flask_api_tutorial.models.user import User from flask_api_tutorial.models.widget import Widget   Then copy the update_widget function below and add it to business.py:\n49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65  @admin_token_required def update_widget(name, widget_dict): widget = Widget.find_by_name(name.lower()) if widget: for k, v in widget_dict.items(): setattr(widget, k, v) db.session.commit() message = f\u0026#34;\u0026#39;{name}\u0026#39; was successfully updated\u0026#34; response_dict = dict(status=\u0026#34;success\u0026#34;, message=message) return response_dict, HTTPStatus.OK try: valid_name = widget_name(name.lower()) except ValueError as e: abort(HTTPStatus.BAD_REQUEST, str(e), status=\u0026#34;fail\u0026#34;) widget_dict[\u0026#34;name\u0026#34;] = valid_name return create_widget(widget_dict)   Let\u0026rsquo;s make sure that the update_widget function correctly implements the PUT method:\n  Line 51: The first thing we do is check the database for a widget with the name provided by the client.   Remember, all widget names are converted to lowercase when written to the database. Because of this, we must be careful to convert the value provided by the client to lowercase before searching. You can review the validation logic for the widget name attribute in Part 5.\n    Lines 53-55: If the name provided by the client is found in the database, we save the retrieved Widget instance as widget. Next, we iterate over the items in widget_dict and overwrite the attributes of widget with the values provided by the client. Then, the updated widget is committed to the database.\n  Lines 56-57: Per the specification, if the name provided by the client already exists, and the widget was successfully updated using the values parsed from the request data, we can confirm that the request succeeded by sending either a 200 (HTTPStatus.OK) or 204 (HTTPStatus.NO_CONTENT) response.\n  Line 60: If we reach this point, it means that the database does not contain a widget with the name provided by the client. Before using this value to create a new widget, we must validate it with the widget_name function we created in the app.api.widgets.dto module. If it is valid, this function will return the value we passed in. If it is not valid, a ValueError will be thrown.\n  Line 62: If the name provided by the client is invalid we cannot create a new widget. The server rejects the request with a 400 (HTTPStatus.BAD_REQUEST) response containing an error message detailing why the value provided is not a valid widget name.\n  Line 63: If the name provided by the client was successfully validated by the widget_name function, we need to add it to the widget_dict object since the create_widget function expects to receive a dict object containing name, info_url, and deadline keys. widget_dict[\"name\"] = valid_name stores the validated name. At this point, widget_dict is in the format expected by the create_widget function.\n  Line 64: As specified in RFC 7231, if the name provided by the client does not already exist and this PUT request successfully creates one, we can confirm that the request succeeded by sending a 201 (HTTPStatus.CREATED) response.\n   I believe that the update_widget function satisfies the specification for the PUT method as specified in RFC 7231. If you disagree, please let me know in the comments, I would greatly appreciate it if my understanding of the spec is faulty in any way.\napi.widget Endpoint (PUT Request) Before we can bring everything together and expose the put method handler for the api.widget endpoint, open src/flask_api_tutorial/api/widgets/endpoints.py and update the import statements to include the update_widget_reqparser we created in app.api.widgets.dto (Line 8) and the update_widget function we created in app.api.widgets.business (Line 19):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20  \u0026#34;\u0026#34;\u0026#34;API endpoint definitions for /widgets namespace.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_restx import Namespace, Resource from flask_api_tutorial.api.widgets.dto import ( create_widget_reqparser, update_widget_reqparser,  pagination_reqparser, widget_owner_model, widget_model, pagination_links_model, pagination_model, ) from flask_api_tutorial.api.widgets.business import ( create_widget, retrieve_widget_list, retrieve_widget, update_widget, )   Next, add the highlighted lines to endpoints.py and save the file:\n57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82  @widget_ns.route(\u0026#34;/\u0026lt;name\u0026gt;\u0026#34;, endpoint=\u0026#34;widget\u0026#34;) @widget_ns.param(\u0026#34;name\u0026#34;, \u0026#34;Widget name\u0026#34;) @widget_ns.response(int(HTTPStatus.BAD_REQUEST), \u0026#34;Validation error.\u0026#34;) @widget_ns.response(int(HTTPStatus.NOT_FOUND), \u0026#34;Widget not found.\u0026#34;) @widget_ns.response(int(HTTPStatus.UNAUTHORIZED), \u0026#34;Unauthorized.\u0026#34;) @widget_ns.response(int(HTTPStatus.INTERNAL_SERVER_ERROR), \u0026#34;Internal server error.\u0026#34;) class Widget(Resource): \u0026#34;\u0026#34;\u0026#34;Handles HTTP requests to URL: /widgets/{name}.\u0026#34;\u0026#34;\u0026#34; @widget_ns.doc(security=\u0026#34;Bearer\u0026#34;) @widget_ns.response(int(HTTPStatus.OK), \u0026#34;Retrieved widget.\u0026#34;, widget_model) @widget_ns.marshal_with(widget_model) def get(self, name): \u0026#34;\u0026#34;\u0026#34;Retrieve a widget.\u0026#34;\u0026#34;\u0026#34; return retrieve_widget(name) @widget_ns.doc(security=\u0026#34;Bearer\u0026#34;) @widget_ns.response(int(HTTPStatus.OK), \u0026#34;Widget was updated.\u0026#34;, widget_model) @widget_ns.response(int(HTTPStatus.CREATED), \u0026#34;Added new widget.\u0026#34;) @widget_ns.response(int(HTTPStatus.FORBIDDEN), \u0026#34;Administrator token required.\u0026#34;) @widget_ns.expect(update_widget_reqparser) def put(self, name): \u0026#34;\u0026#34;\u0026#34;Update a widget.\u0026#34;\u0026#34;\u0026#34; widget_dict = update_widget_reqparser.parse_args() return update_widget(name, widget_dict)   We have previously encountered and explained everything in the put method, so you should be comfortable moving on without explaining the design/implementation. We can verify that the api.widget endpoint now supports both GET and PUT requests by executing the flask routes command:\nflask-api-tutorial $ flask routes Endpoint Methods Rule ------------------- --------- -------------------------- api.auth_login POST /api/v1/auth/login api.auth_logout POST /api/v1/auth/logout api.auth_register POST /api/v1/auth/register api.auth_user GET /api/v1/auth/user api.doc GET /api/v1/ui api.root GET /api/v1/ api.specs GET /api/v1/swagger.json api.widget GET, PUT /api/v1/widgets/\u0026lt;name\u0026gt; api.widget_list GET, POST /api/v1/widgets restplus_doc.static GET /swaggerui/\u0026lt;path:filename\u0026gt; static GET /static/\u0026lt;path:filename\u0026gt; Alright, four down, one to go! The only remaining CRUD process we have not yet implemented is deleting a single widget.\nDelete Widget Implementing the process to delete a single widget will be simple and very similar to implementing the process to retrieve a single widget. The only data that is sent by the client is the name of the widget, so we do not need to create a request parser for this process. Also, a successful response will not include any data in the response body, so we do not need to create any API models to serialize the response.\ndelete_widget Method Open src/flask_api_tutorial/api/widgets/business.py and add the content below:\n67 68 69 70 71 72 73 74 75  @admin_token_required def delete_widget(name): widget = Widget.query.filter_by(name=name.lower()).first_or_404( description=f\u0026#34;{name} not found in database.\u0026#34; ) db.session.delete(widget) db.session.commit() return \u0026#34;\u0026#34;, HTTPStatus.NO_CONTENT   The delete_widget function relies on the Flask-SQLAlchemy first_or_404 method which we used previously in the retrieve_widget function. If the database does not contain a widget with the name provided by the client, the request is rejected and a 404 (HTTPStatus.NOT_FOUND) response is sent. If a widget was successfully retrieved, however, it is deleted and the changes are committed to the database. Then, a 204 (HTTPStatus.NO_CONTENT) response is sent with an empty request body.\napi.widget Endpoint (DELETE Request) I told you this would be easy! Open src/flask_api_tutorial/api/widgets/endpoints.py and update the import statements to include the delete_widget function that we just created in app.api.widgets.business (Line 20):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21  \u0026#34;\u0026#34;\u0026#34;API endpoint definitions for /widgets namespace.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from flask_restx import Namespace, Resource from flask_api_tutorial.api.widgets.dto import ( create_widget_reqparser, update_widget_reqparser, pagination_reqparser, widget_owner_model, widget_model, pagination_links_model, pagination_model, ) from flask_api_tutorial.api.widgets.business import ( create_widget, retrieve_widget_list, retrieve_widget, update_widget, delete_widget, )   Next, add the highlighted lines to endpoints.py and save the file:\n58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90  @widget_ns.route(\u0026#34;/\u0026lt;name\u0026gt;\u0026#34;, endpoint=\u0026#34;widget\u0026#34;) @widget_ns.param(\u0026#34;name\u0026#34;, \u0026#34;Widget name\u0026#34;) @widget_ns.response(int(HTTPStatus.BAD_REQUEST), \u0026#34;Validation error.\u0026#34;) @widget_ns.response(int(HTTPStatus.NOT_FOUND), \u0026#34;Widget not found.\u0026#34;) @widget_ns.response(int(HTTPStatus.UNAUTHORIZED), \u0026#34;Unauthorized.\u0026#34;) @widget_ns.response(int(HTTPStatus.INTERNAL_SERVER_ERROR), \u0026#34;Internal server error.\u0026#34;) class Widget(Resource): \u0026#34;\u0026#34;\u0026#34;Handles HTTP requests to URL: /widgets/{name}.\u0026#34;\u0026#34;\u0026#34; @widget_ns.doc(security=\u0026#34;Bearer\u0026#34;) @widget_ns.response(int(HTTPStatus.OK), \u0026#34;Retrieved widget.\u0026#34;, widget_model) @widget_ns.marshal_with(widget_model) def get(self, name): \u0026#34;\u0026#34;\u0026#34;Retrieve a widget.\u0026#34;\u0026#34;\u0026#34; return retrieve_widget(name) @widget_ns.doc(security=\u0026#34;Bearer\u0026#34;) @widget_ns.response(int(HTTPStatus.OK), \u0026#34;Widget was updated.\u0026#34;, widget_model) @widget_ns.response(int(HTTPStatus.CREATED), \u0026#34;Added new widget.\u0026#34;) @widget_ns.response(int(HTTPStatus.FORBIDDEN), \u0026#34;Administrator token required.\u0026#34;) @widget_ns.expect(update_widget_reqparser) def put(self, name): \u0026#34;\u0026#34;\u0026#34;Update a widget.\u0026#34;\u0026#34;\u0026#34; widget_dict = update_widget_reqparser.parse_args() return update_widget(name, widget_dict) @widget_ns.doc(security=\u0026#34;Bearer\u0026#34;) @widget_ns.response(int(HTTPStatus.NO_CONTENT), \u0026#34;Widget was deleted.\u0026#34;) @widget_ns.response(int(HTTPStatus.FORBIDDEN), \u0026#34;Administrator token required.\u0026#34;) def delete(self, name): \u0026#34;\u0026#34;\u0026#34;Delete a widget.\u0026#34;\u0026#34;\u0026#34; return delete_widget(name)   We have previously encountered and explained everything in the delete method, so there\u0026rsquo;s nothing we need to elaborate on. We can verify that the api.widget endpoint now supports DELETE, GET and PUT requests by executing the flask routes command:\nflask-api-tutorial $ flask routes Endpoint Methods Rule ------------------- ---------------- -------------------------- api.auth_login POST /api/v1/auth/login api.auth_logout POST /api/v1/auth/logout api.auth_register POST /api/v1/auth/register api.auth_user GET /api/v1/auth/user api.doc GET /api/v1/ui api.root GET /api/v1/ api.specs GET /api/v1/swagger.json api.widget DELETE, GET, PUT /api/v1/widgets/\u0026lt;name\u0026gt; api.widget_list GET, POST /api/v1/widgets restplus_doc.static GET /swaggerui/\u0026lt;path:filename\u0026gt; static GET /static/\u0026lt;path:filename\u0026gt; We have finally implemented all of the API routes/CRUD processes specified in Table 1, but we actually have no idea if they are working correctly. At the absolute minimum, we need to create unit tests that verify the \u0026ldquo;happy path\u0026rdquo; behavior for each CRUD process. We also need unit tests that verify our request parsers are configured correctly, and requests containing invalid data are rejected.\nUnit Tests At this point, I would like you to attempt to create as many unit tests you can think of for the Widget API endpoints/CRUD operations we implemented in this section and the previous section (Part 5). For each, I will provide a few tests to get you started, and demonstrate how the @pytest.mark.parametrize decorator makes testing multiple values for a single parameter much simpler.\nutil.py First, open tests/util.py and update the import statements to include the datetime.date module (Line 2), and also add the string values highlighted below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17  \u0026#34;\u0026#34;\u0026#34;Shared functions and constants for unit tests.\u0026#34;\u0026#34;\u0026#34; from datetime import date  from flask import url_for EMAIL = \u0026#34;new_user@email.com\u0026#34; ADMIN_EMAIL = \u0026#34;admin_user@email.com\u0026#34; PASSWORD = \u0026#34;test1234\u0026#34; BAD_REQUEST = \u0026#34;Input payload validation failed\u0026#34; UNAUTHORIZED = \u0026#34;Unauthorized\u0026#34; FORBIDDEN = \u0026#34;You are not an administrator\u0026#34; WWW_AUTH_NO_TOKEN = \u0026#39;Bearer realm=\u0026#34;registered_users@mydomain.com\u0026#34;\u0026#39; DEFAULT_NAME = \u0026#34;some-widget\u0026#34; DEFAULT_URL = \u0026#34;https://www.fakesite.com\u0026#34; DEFAULT_DEADLINE = date.today().strftime(\u0026#34;%m/%d/%y\u0026#34;)    Next, add the create_widget function to util.py and save the file:\n47 48 49 50 51 52 53 54 55 56 57 58 59  def create_widget( test_client, access_token, widget_name=DEFAULT_NAME, info_url=DEFAULT_URL, deadline_str=DEFAULT_DEADLINE, ): return test_client.post( url_for(\u0026#34;api.widget_list\u0026#34;), headers={\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {access_token}\u0026#34;}, data=f\u0026#34;name={widget_name}\u0026amp;info_url={info_url}\u0026amp;deadline={deadline_str}\u0026#34;, content_type=\u0026#34;application/x-www-form-urlencoded\u0026#34;, )   This function uses the Flask test client to send a POST request to the api.widget_list endpoint, which is responsible for creating new widgets. The function requires two parameters: the test_client and an access_token. The remaining three parameters are optional since they have default values, and it should be obvious what these three values are used for. widget_name, info_url and deadline_str are the values that will be used for the name, info_url and deadline values of the new widget object.\nconftest.py We also need to update conftest.py with a new fixture. First, update the import statements to include the ADMIN_EMAIL value from tests.util:\n1 2 3 4 5 6 7 8  \u0026#34;\u0026#34;\u0026#34;Global pytest fixtures.\u0026#34;\u0026#34;\u0026#34; import pytest from flask_api_tutorial import create_app from flask_api_tutorial import db as database from flask_api_tutorial.models.user import User from tests.util import EMAIL, ADMIN_EMAIL, PASSWORD    Next, add the admin test fixture and save the file:\n37 38 39 40 41 42 43  @pytest.fixture def admin(db): admin = User(email=ADMIN_EMAIL, password=PASSWORD, admin=True) db.session.add(admin) db.session.commit() return admin   This fixture creates a new user with administrator privileges, which will be needed to create new widgets (it will also be needed to modify and delete widgets). We have previously used the user fixture in test cases where a User object was needed, and the admin fixture will be used in the same way.\nCreate Widget Create a new file named test_create_widget.py in tests, enter the content below and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20  \u0026#34;\u0026#34;\u0026#34;Unit tests for POST requests sent to api.widget_list API endpoint.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus import pytest from tests.util import ADMIN_EMAIL, login_user, create_widget @pytest.mark.parametrize(\u0026#34;widget_name\u0026#34;, [\u0026#34;abc123\u0026#34;, \u0026#34;widget-name\u0026#34;, \u0026#34;new_widget1\u0026#34;]) def test_create_widget_valid_name(client, db, admin, widget_name): response = login_user(client, email=ADMIN_EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = create_widget(client, access_token, widget_name=widget_name) assert response.status_code == HTTPStatus.CREATED assert \u0026#34;status\u0026#34; in response.json and response.json[\u0026#34;status\u0026#34;] == \u0026#34;success\u0026#34; success = f\u0026#34;New widget added: {widget_name}.\u0026#34; assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == success location = f\u0026#34;http://localhost/api/v1/widgets/{widget_name}\u0026#34; assert \u0026#34;Location\u0026#34; in response.headers and response.headers[\u0026#34;Location\u0026#34;] == location   This is the first time we are using the @pytest.mark.parameterize decorator, which is used to parameterize an argument to a test function. In the test_create_widget_valid_name function, we are parameterizing the widget_name argument, and the test will be executed three times; once for each value defined for widget_name. For example, this is the output from pytest if we were to execute just this single test function:\nflask-api-tutorial $ pytest tests/test_create_widget.py ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/venv/bin/python cachedir: .pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 5 items tests/test_create_widget.py::FLAKE8 PASSED [ 20%] tests/test_create_widget.py::BLACK PASSED [ 40%] tests/test_create_widget.py::test_create_widget_valid_name[abc123] PASSED [ 60%] tests/test_create_widget.py::test_create_widget_valid_name[widget-name] PASSED [ 80%] tests/test_create_widget.py::test_create_widget_valid_name[new_widget1] PASSED [100%] =================================================== warnings summary =================================================== tests/test_create_widget.py::test_create_widget_valid_name[abc123] /Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask_restx/model.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, MutableMapping tests/test_create_widget.py::test_create_widget_valid_name[abc123] /Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask_restx/api.py:28: DeprecationWarning: The import 'werkzeug.cached_property' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.utils import cached_property' instead. from werkzeug import cached_property tests/test_create_widget.py::test_create_widget_valid_name[abc123] /Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask_restx/swagger.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, Hashable -- Docs: https://docs.pytest.org/en/latest/warnings.html ============================================ 5 passed, 3 warnings in 0.80s ============================================= You can see the three tests executed by the parameterized test highlighted above. pytest helpfully includes the values used to parameterize the test in brackets after the name of the test. This function is designed to verify the response to a successful request to create a new widget, so the three values used for parameterization are all valid widget names, per the specs we followed. An obvious next step would be to create a test_create_widget_invalid_name test case, but I will leave that to you.\n  You can find more information on the @pytest.mark.parameterize decorator in the official pytest documents.\n  Creating valid/invalid values for the info_url argument should be straightforward, and you should definitely create parameterized test cases for both successful/rejected requests that isolate the info_url value. However, testing the deadline_str value is more complex. In the same file, test_create_widget.py, update the import statements to include the datetime.date and datetime.timedelta modules (Line 2), as well as the DEFAULT_NAME value from tests.util (Line 7):\n1 2 3 4 5 6 7  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.widget_list API endpoint.\u0026#34;\u0026#34;\u0026#34; from datetime import date, timedelta from http import HTTPStatus import pytest from tests.util import ADMIN_EMAIL, DEFAULT_NAME, login_user, create_widget   Why is the deadline attribute more difficult to test than widget_name or info_url? Remember, deadline is a string value that is parsed to a datetime value which must not be in the past. We could use hard-coded values that won\u0026rsquo;t be in the past 100 years from now, but that is a pretty hacky way to test our code.\nWhat approach can we use that would be better than hard-coded strings? Enter the content below and save the file:\n24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43  @pytest.mark.parametrize( \u0026#34;deadline_str\u0026#34;, [ date.today().strftime(\u0026#34;%m/%d/%Y\u0026#34;), date.today().strftime(\u0026#34;%Y-%m-%d\u0026#34;), (date.today() + timedelta(days=3)).strftime(\u0026#34;%b %d%Y\u0026#34;),  ], ) def test_create_widget_valid_deadline(client, db, admin, deadline_str): response = login_user(client, email=ADMIN_EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = create_widget(client, access_token, deadline_str=deadline_str) assert response.status_code == HTTPStatus.CREATED assert \u0026#34;status\u0026#34; in response.json and response.json[\u0026#34;status\u0026#34;] == \u0026#34;success\u0026#34; success = f\u0026#34;New widget added: {DEFAULT_NAME}.\u0026#34; assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == success location = f\u0026#34;http://localhost/api/v1/widgets/{DEFAULT_NAME}\u0026#34; assert \u0026#34;Location\u0026#34; in response.headers and response.headers[\u0026#34;Location\u0026#34;] == location   In the first two highlighted lines above (Lines 27-28), we call the datetime.strftime method on objects created with datetime.date.today. This generates a string value that always represents the current date, even if this test case is executed 10,000 years from now. datetime.strftime accepts a string value that can be configured to generate a string in any format, containing any combination of values such as month, year, hour, time zone, etc.\nThe third highlighted line (Line 29) utilizes a timedelta object to generate a date three days in the future (we obviously want to test dates other than the current date). Run pytest tests/test_create_widget.py to execute these test cases:\nflask-api-tutorial $ pytest tests/test_create_widget.py ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/venv/bin/python cachedir: .pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 8 items tests/test_create_widget.py::FLAKE8 PASSED [ 12%] tests/test_create_widget.py::BLACK PASSED [ 25%] tests/test_create_widget.py::test_create_widget_valid_name[abc123] PASSED [ 37%] tests/test_create_widget.py::test_create_widget_valid_name[widget-name] PASSED [ 50%] tests/test_create_widget.py::test_create_widget_valid_name[new_widget1] PASSED [ 62%] tests/test_create_widget.py::test_create_widget_valid_deadline[02/29/2020] PASSED [ 75%] tests/test_create_widget.py::test_create_widget_valid_deadline[2020-02-29] PASSED [ 87%] tests/test_create_widget.py::test_create_widget_valid_deadline[Mar 03 2020] PASSED [100%] =================================================== warnings summary =================================================== tests/test_create_widget.py::test_create_widget_valid_name[abc123] /Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask_restx/model.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, MutableMapping tests/test_create_widget.py::test_create_widget_valid_name[abc123] /Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask_restx/api.py:28: DeprecationWarning: The import 'werkzeug.cached_property' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.utils import cached_property' instead. from werkzeug import cached_property tests/test_create_widget.py::test_create_widget_valid_name[abc123] /Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask_restx/swagger.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, Hashable -- Docs: https://docs.pytest.org/en/latest/warnings.html ============================================ 8 passed, 3 warnings in 1.13s ============================================= We can easily see what values were used to test the deadline value in the results above. For the first two parameterized values, the current date was converted to string values 01/29/2020 and 2020-01-29, which succeeded in creating a new widget object in the database. Finally, Feb 01 2020 was used to verify that a date in the future is a valid value for deadline.\n  Did you notice that we used three different date formats for the deadline_str test parameter, and all three were successful? This is due to the validation performed by the future_date_from_string function, specifically by the dateutil.parser.parse function.\n  Before we create test cases for rejected requests, update the import statements to include the BAD_REQUEST value from tests.util (Line 7):\n1 2 3 4 5 6 7 8  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.widget_list API endpoint.\u0026#34;\u0026#34;\u0026#34; from datetime import date, timedelta from http import HTTPStatus import pytest from tests.util import ADMIN_EMAIL, BAD_REQUEST, DEFAULT_NAME, login_user, create_widget    Then, add the test_create_widget_invalid_deadline function and save the file:\n45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61  @pytest.mark.parametrize( \u0026#34;deadline_str\u0026#34;, [ \u0026#34;1/1/1970\u0026#34;, (date.today() - timedelta(days=3)).strftime(\u0026#34;%Y-%m-%d\u0026#34;), \u0026#34;a long time ago, in a galaxy far, far away\u0026#34;,  ], ) def test_create_widget_invalid_deadline(client, db, admin, deadline_str): response = login_user(client, email=ADMIN_EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = create_widget(client, access_token, deadline_str=deadline_str) assert response.status_code == HTTPStatus.BAD_REQUEST assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == BAD_REQUEST assert \u0026#34;errors\u0026#34; in response.json and \u0026#34;deadline\u0026#34; in response.json[\u0026#34;errors\u0026#34;]   Unlike the parameters in the \u0026ldquo;happy\u0026rdquo; path test case, we can use hardcoded strings to verify that invalid deadline values are rejected by the API. The first parameterized value is 1/1/1970 *(Line 48), which will be parsed as a valid datetime value by dateutil.parser, but must be rejected since it has already passed.\nNext, we construct a string value for a date three days in the past using a timedelta object (Line 49). This is the same technique we used in the \u0026ldquo;happy\u0026rdquo; path to construct dates in the future. The third parameter is a string that is not a formatted datetime value, and should obviously be rejected by datetuil.parser (Line 50).\nRun pytest tests/test_create_widget.py to execute these test cases:\nflask-api-tutorial $ pytest tests/test_create_widget.py ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/venv/bin/python cachedir: .pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 11 items tests/test_create_widget.py::FLAKE8 PASSED [ 9%] tests/test_create_widget.py::BLACK PASSED [ 18%] tests/test_create_widget.py::test_create_widget_valid_name[abc123] PASSED [ 27%] tests/test_create_widget.py::test_create_widget_valid_name[widget-name] PASSED [ 36%] tests/test_create_widget.py::test_create_widget_valid_name[new_widget1] PASSED [ 45%] tests/test_create_widget.py::test_create_widget_valid_deadline[02/29/2020] PASSED [ 54%] tests/test_create_widget.py::test_create_widget_valid_deadline[2020-02-29] PASSED [ 63%] tests/test_create_widget.py::test_create_widget_valid_deadline[Mar 03 2020] PASSED [ 72%] tests/test_create_widget.py::test_create_widget_invalid_deadline[1/1/1970] PASSED [ 81%] tests/test_create_widget.py::test_create_widget_invalid_deadline[2020-02-26] PASSED [ 90%] tests/test_create_widget.py::test_create_widget_invalid_deadline[a long time ago, in a galaxy far, far away] PASSED [100%] =================================================== warnings summary =================================================== tests/test_create_widget.py::test_create_widget_valid_name[abc123] /Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask_restx/model.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, MutableMapping tests/test_create_widget.py::test_create_widget_valid_name[abc123] /Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask_restx/api.py:28: DeprecationWarning: The import 'werkzeug.cached_property' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.utils import cached_property' instead. from werkzeug import cached_property tests/test_create_widget.py::test_create_widget_valid_name[abc123] /Users/aaronluna/Projects/flask-api-tutorial/venv/lib/python3.7/site-packages/flask_restx/swagger.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, Hashable -- Docs: https://docs.pytest.org/en/latest/warnings.html ============================================ 11 passed, 3 warnings in 1.18s ============================================   Don\u0026rsquo;t be confused by the PASSED result for these three test cases. This does not mean that the request succeeded and a new widget was created, since the assert statements verify that the status code of the response is 400 HTTPStatus.BAD_REQUEST (Line 58), and that the response includes an error message indicating that the value of the deadline attribute was invalid (Line 60).\n  Hopefully, these three parameterized test cases are illustrative and enable you to create the remaining necessary test cases for the widget_name and info_url attributes.\nThere are plenty of other requirements and expected behaviors that need test coverage other than the input validation performed on the client\u0026rsquo;s request data. I\u0026rsquo;ll show you two more test cases for the create widget operation, but you must attempt to create other test cases on your own for any remaining functionality.\nFirst, we should verify that it isn\u0026rsquo;t possible to create a widget if a widget with the same name has already been created. We can easily write a test case to check this scenario. Add the test case below (test_create_widget_already_exists) to test_create_widget.py and save the file:\n63 64 65 66 67 68 69 70 71 72  def test_create_widget_already_exists(client, db, admin): response = login_user(client, email=ADMIN_EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = create_widget(client, access_token) assert response.status_code == HTTPStatus.CREATED response = create_widget(client, access_token) assert response.status_code == HTTPStatus.CONFLICT name_conflict = f\u0026#34;Widget name: {DEFAULT_NAME} already exists, must be unique.\u0026#34; assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == name_conflict   The main thing that we need to verify is that the HTTP response code from the server is 409 (HTTPStatus.CONFLICT) (Line 70), indicating that the new widget was not created due to a conflict with an existing widget. This is also verified in the error message sent in the server\u0026rsquo;s response (Lines 72-73).\nSo far, in every test case for the create widget process we used the admin user account. However, we absolutely must verify that users without administrator access cannot create new widgets. Before we begin, we need to update the import statements to include the EMAIL (Line 8) and FORBIDDEN (Line 11) string values from tests.util:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15  \u0026#34;\u0026#34;\u0026#34;Unit tests for api.widget_list API endpoint.\u0026#34;\u0026#34;\u0026#34; from datetime import date, timedelta from http import HTTPStatus import pytest from tests.util import ( EMAIL,  ADMIN_EMAIL, BAD_REQUEST, FORBIDDEN,  DEFAULT_NAME, login_user, create_widget, )   Next, add the test case below (test_create_widget_no_admin_token) to test_create_widget.py and save the file:\n83 84 85 86 87 88 89 90  def test_create_widget_no_admin_token(client, db, user): response = login_user(client, email=EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = create_widget(client, access_token) assert response.status_code == HTTPStatus.FORBIDDEN assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == FORBIDDEN   As in the previous test case, the main thing we need to verify is that the HTTP status code of the response is 403 (HTTPStatus.FORBIDDEN) (Line 89), and that the error message indicates that request was rejected since the user does not have administrator privileges (Line 91).\nLet\u0026rsquo;s verify that all of our test cases are still passing by running tox:\n(flask-api-tutorial) flask-api-tutorial $ tox GLOB sdist-make: /Users/aaronluna/Projects/flask-api-tutorial/setup.py py37 create: /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37 py37 installdeps: black, flake8, pydocstyle, pytest, pytest-black, pytest-clarity, pytest-dotenv, pytest-flake8, pytest-flask py37 inst: /Users/aaronluna/Projects/flask-api-tutorial/.tox/.tmp/package/1/flask-api-tutorial-0.1.zip py37 installed: alembic==1.4.0,aniso8601==8.0.0,appdirs==1.4.3,attrs==19.3.0,bcrypt==3.1.7,black==19.10b0,certifi==2019.11.28,cffi==1.14.0,chardet==3.0.4,Click==7.0,entrypoints==0.3,flake8==3.7.9,Flask==1.1.1,flask-api-tutorial==0.1,Flask-Bcrypt==0.7.1,Flask-Cors==3.0.8,Flask-Migrate==2.5.2,flask-restx==0.1.1,Flask-SQLAlchemy==2.4.1,idna==2.9,importlib-metadata==1.5.0,itsdangerous==1.1.0,Jinja2==2.11.1,jsonschema==3.2.0,Mako==1.1.1,MarkupSafe==1.1.1,mccabe==0.6.1,more-itertools==8.2.0,packaging==20.1,pathspec==0.7.0,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pycparser==2.19,pydocstyle==5.0.2,pyflakes==2.1.1,PyJWT==1.7.1,pyparsing==2.4.6,pyrsistent==0.15.7,pytest==5.3.5,pytest-black==0.3.8,pytest-clarity==0.3.0a0,pytest-dotenv==0.4.0,pytest-flake8==1.0.4,pytest-flask==0.15.1,python-dateutil==2.8.1,python-dotenv==0.12.0,python-editor==1.0.4,pytz==2019.3,regex==2020.2.20,requests==2.23.0,six==1.14.0,snowballstemmer==2.0.0,SQLAlchemy==1.3.13,termcolor==1.1.0,toml==0.10.0,typed-ast==1.4.1,urllib3==1.25.8,wcwidth==0.1.8,Werkzeug==0.16.1,zipp==3.0.0 py37 run-test-pre: PYTHONHASHSEED='2677345098' py37 run-test: commands[0] | pytest ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/bin/python cachedir: .tox/py37/.pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 92 items run.py::FLAKE8 PASSED [ 1%] run.py::BLACK PASSED [ 2%] setup.py::FLAKE8 PASSED [ 3%] setup.py::BLACK PASSED [ 4%] src/flask_api_tutorial/__init__.py::FLAKE8 PASSED [ 5%] src/flask_api_tutorial/__init__.py::BLACK PASSED [ 6%] src/flask_api_tutorial/config.py::FLAKE8 PASSED [ 7%] src/flask_api_tutorial/config.py::BLACK PASSED [ 8%] src/flask_api_tutorial/api/__init__.py::FLAKE8 PASSED [ 9%] src/flask_api_tutorial/api/__init__.py::BLACK PASSED [ 10%] src/flask_api_tutorial/api/exceptions.py::FLAKE8 PASSED [ 11%] src/flask_api_tutorial/api/exceptions.py::BLACK PASSED [ 13%] src/flask_api_tutorial/api/auth/__init__.py::FLAKE8 PASSED [ 14%] src/flask_api_tutorial/api/auth/__init__.py::BLACK PASSED [ 15%] src/flask_api_tutorial/api/auth/business.py::FLAKE8 PASSED [ 16%] src/flask_api_tutorial/api/auth/business.py::BLACK PASSED [ 17%] src/flask_api_tutorial/api/auth/decorators.py::FLAKE8 PASSED [ 18%] src/flask_api_tutorial/api/auth/decorators.py::BLACK PASSED [ 19%] src/flask_api_tutorial/api/auth/dto.py::FLAKE8 PASSED [ 20%] src/flask_api_tutorial/api/auth/dto.py::BLACK PASSED [ 21%] src/flask_api_tutorial/api/auth/endpoints.py::FLAKE8 PASSED [ 22%] src/flask_api_tutorial/api/auth/endpoints.py::BLACK PASSED [ 23%] src/flask_api_tutorial/api/widgets/__init__.py::FLAKE8 PASSED [ 25%] src/flask_api_tutorial/api/widgets/__init__.py::BLACK PASSED [ 26%] src/flask_api_tutorial/api/widgets/business.py::FLAKE8 PASSED [ 27%] src/flask_api_tutorial/api/widgets/business.py::BLACK PASSED [ 28%] src/flask_api_tutorial/api/widgets/dto.py::FLAKE8 PASSED [ 29%] src/flask_api_tutorial/api/widgets/dto.py::BLACK PASSED [ 30%] src/flask_api_tutorial/api/widgets/endpoints.py::FLAKE8 PASSED [ 31%] src/flask_api_tutorial/api/widgets/endpoints.py::BLACK PASSED [ 32%] src/flask_api_tutorial/models/__init__.py::FLAKE8 PASSED [ 33%] src/flask_api_tutorial/models/__init__.py::BLACK PASSED [ 34%] src/flask_api_tutorial/models/token_blacklist.py::FLAKE8 PASSED [ 35%] src/flask_api_tutorial/models/token_blacklist.py::BLACK PASSED [ 36%] src/flask_api_tutorial/models/user.py::FLAKE8 PASSED [ 38%] src/flask_api_tutorial/models/user.py::BLACK PASSED [ 39%] src/flask_api_tutorial/models/widget.py::FLAKE8 PASSED [ 40%] src/flask_api_tutorial/models/widget.py::BLACK PASSED [ 41%] src/flask_api_tutorial/util/__init__.py::FLAKE8 PASSED [ 42%] src/flask_api_tutorial/util/__init__.py::BLACK PASSED [ 43%] src/flask_api_tutorial/util/datetime_util.py::FLAKE8 PASSED [ 44%] src/flask_api_tutorial/util/datetime_util.py::BLACK PASSED [ 45%] src/flask_api_tutorial/util/result.py::FLAKE8 PASSED [ 46%] src/flask_api_tutorial/util/result.py::BLACK PASSED [ 47%] tests/__init__.py::FLAKE8 PASSED [ 48%] tests/__init__.py::BLACK PASSED [ 50%] tests/conftest.py::FLAKE8 PASSED [ 51%] tests/conftest.py::BLACK PASSED [ 52%] tests/test_auth_login.py::FLAKE8 PASSED [ 53%] tests/test_auth_login.py::BLACK PASSED [ 54%] tests/test_auth_login.py::test_login PASSED [ 55%] tests/test_auth_login.py::test_login_email_does_not_exist PASSED [ 56%] tests/test_auth_logout.py::FLAKE8 PASSED [ 57%] tests/test_auth_logout.py::BLACK PASSED [ 58%] tests/test_auth_logout.py::test_logout PASSED [ 59%] tests/test_auth_logout.py::test_logout_token_blacklisted PASSED [ 60%] tests/test_auth_register.py::FLAKE8 PASSED [ 61%] tests/test_auth_register.py::BLACK PASSED [ 63%] tests/test_auth_register.py::test_auth_register PASSED [ 64%] tests/test_auth_register.py::test_auth_register_email_already_registered PASSED [ 65%] tests/test_auth_register.py::test_auth_register_invalid_email PASSED [ 66%] tests/test_auth_user.py::FLAKE8 PASSED [ 67%] tests/test_auth_user.py::BLACK PASSED [ 68%] tests/test_auth_user.py::test_auth_user PASSED [ 69%] tests/test_auth_user.py::test_auth_user_no_token PASSED [ 70%] tests/test_auth_user.py::test_auth_user_expired_token PASSED [ 71%] tests/test_config.py::FLAKE8 PASSED [ 72%] tests/test_config.py::BLACK PASSED [ 73%] tests/test_config.py::test_config_development PASSED [ 75%] tests/test_config.py::test_config_testing PASSED [ 76%] tests/test_config.py::test_config_production PASSED [ 77%] tests/test_create_widget.py::FLAKE8 PASSED [ 78%] tests/test_create_widget.py::BLACK PASSED [ 79%] tests/test_create_widget.py::test_create_widget_valid_name[abc123] PASSED [ 80%] tests/test_create_widget.py::test_create_widget_valid_name[widget-name] PASSED [ 81%] tests/test_create_widget.py::test_create_widget_valid_name[new_widget1] PASSED [ 82%] tests/test_create_widget.py::test_create_widget_valid_deadline[02/29/2020] PASSED [ 83%] tests/test_create_widget.py::test_create_widget_valid_deadline[2020-02-29] PASSED [ 84%] tests/test_create_widget.py::test_create_widget_valid_deadline[Mar 03 2020] PASSED [ 85%] tests/test_create_widget.py::test_create_widget_invalid_deadline[1/1/1970] PASSED [ 86%] tests/test_create_widget.py::test_create_widget_invalid_deadline[2020-02-26] PASSED [ 88%] tests/test_create_widget.py::test_create_widget_invalid_deadline[a long time ago, in a galaxy far, far away] PASSED [ 89%] tests/test_create_widget.py::test_create_widget_already_exists PASSED [ 90%] tests/test_create_widget.py::test_create_widget_no_admin_token PASSED [ 91%] tests/test_user.py::FLAKE8 PASSED [ 92%] tests/test_user.py::BLACK PASSED [ 93%] tests/test_user.py::test_encode_access_token PASSED [ 94%] tests/test_user.py::test_decode_access_token_success PASSED [ 95%] tests/test_user.py::test_decode_access_token_expired PASSED [ 96%] tests/test_user.py::test_decode_access_token_invalid PASSED [ 97%] tests/util.py::FLAKE8 PASSED [ 98%] tests/util.py::BLACK PASSED [100%] =================================================== warnings summary =================================================== src/flask_api_tutorial/api/exceptions.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/model.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, MutableMapping src/flask_api_tutorial/api/exceptions.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/api.py:28: DeprecationWarning: The import 'werkzeug.cached_property' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.utils import cached_property' instead. from werkzeug import cached_property src/flask_api_tutorial/api/exceptions.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/swagger.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, Hashable -- Docs: https://docs.pytest.org/en/latest/warnings.html =========================================== 92 passed, 3 warnings in 25.40s ============================================ _______________________________________________________ summary ________________________________________________________ py37: commands succeeded congratulations :) Please do not assume that these test cases are sufficient for the create widget operation, there are many requirements that have not been verified by the test cases I provided. You absolutely must attempt to identify these gaps and create all necessary test cases.\n  For the remaining CRUD operations I will be providing far fewer test cases since many of the test requirements must be repeated for each. Yes, testing is always (and I mean ALWAYS) repetitive. You might as well just accept it, keep your head down and write test cases until your body refuses to cooperate.\n  Retrieve Widget List In order to test the retrieve widget list operation, we need to create a function that uses the Flask test client to send a GET request to the api.widget_list endpoint. Remember, this endpoint expects the client\u0026rsquo;s request to include pagination query parameters (i.e., page and per_page), refer back to this section if you need to review how this was implemented.\nOpen tests/util.py, add the content below and save the file:\n62 63 64 65 66  def retrieve_widget_list(test_client, access_token, page=None, per_page=None): return test_client.get( url_for(\u0026#34;api.widget_list\u0026#34;, page=page, per_page=per_page), headers={\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {access_token}\u0026#34;}, )   Hopefully the retrieve_widget_list function above makes sense to you since it is very similar (and much simpler than) the create_widget function we used to test the create widget operation.\nI\u0026rsquo;m only going to provide a single test case for the retrieve widget list operation. Before you get all mopey, this test case is quite thorough. In fact, it is so thorough, I had to add comments explaining what is being tested and verified. Create a new file named test_retrieve_widget_list.py in the tests folder and add the content below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142  \u0026#34;\u0026#34;\u0026#34;Test cases for GET requests sent to the api.widget_list API endpoint.\u0026#34;\u0026#34;\u0026#34; from datetime import date, timedelta from http import HTTPStatus from tests.util import ADMIN_EMAIL, login_user, create_widget, retrieve_widget_list NAMES = [ \u0026#34;widget1\u0026#34;, \u0026#34;second_widget\u0026#34;, \u0026#34;widget-thrice\u0026#34;, \u0026#34;tetraWIDG\u0026#34;, \u0026#34;PENTA-widg-GON-et\u0026#34;, \u0026#34;hexa_widget\u0026#34;, \u0026#34;sep7\u0026#34;, ] URLS = [ \u0026#34;http://www.one.com\u0026#34;, \u0026#34;https://www.two.net\u0026#34;, \u0026#34;https://www.three.edu\u0026#34;, \u0026#34;http://www.four.dev\u0026#34;, \u0026#34;http://www.five.io\u0026#34;, \u0026#34;https://www.six.tech\u0026#34;, \u0026#34;https://www.seven.dot\u0026#34;, ] DEADLINES = [ date.today().strftime(\u0026#34;%m/%d/%y\u0026#34;), (date.today() + timedelta(days=3)).strftime(\u0026#34;%m/%d/%y\u0026#34;), (date.today() + timedelta(days=5)).strftime(\u0026#34;%m/%d/%y\u0026#34;), (date.today() + timedelta(days=10)).strftime(\u0026#34;%m/%d/%y\u0026#34;), (date.today() + timedelta(days=17)).strftime(\u0026#34;%m/%d/%y\u0026#34;), (date.today() + timedelta(days=23)).strftime(\u0026#34;%m/%d/%y\u0026#34;), (date.today() + timedelta(days=78)).strftime(\u0026#34;%m/%d/%y\u0026#34;), ] def test_retrieve_paginated_widget_list(client, db, admin): response = login_user(client, email=ADMIN_EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] # ADD SEVEN WIDGET INSTANCES TO DATABASE for i in range(0, len(NAMES)): response = create_widget( client, access_token, widget_name=NAMES[i], info_url=URLS[i], deadline_str=DEADLINES[i], ) assert response.status_code == HTTPStatus.CREATED # REQUEST PAGINATED LIST OF WIDGETS: 5 PER PAGE, PAGE #1 response = retrieve_widget_list(client, access_token, page=1, per_page=5) assert response.status_code == HTTPStatus.OK # VERIFY PAGINATION ATTRIBUTES FOR PAGE #1 assert \u0026#34;has_prev\u0026#34; in response.json and not response.json[\u0026#34;has_prev\u0026#34;] assert \u0026#34;has_next\u0026#34; in response.json and response.json[\u0026#34;has_next\u0026#34;] assert \u0026#34;page\u0026#34; in response.json and response.json[\u0026#34;page\u0026#34;] == 1 assert \u0026#34;total_pages\u0026#34; in response.json and response.json[\u0026#34;total_pages\u0026#34;] == 2 assert \u0026#34;items_per_page\u0026#34; in response.json and response.json[\u0026#34;items_per_page\u0026#34;] == 5 assert \u0026#34;total_items\u0026#34; in response.json and response.json[\u0026#34;total_items\u0026#34;] == 7 assert \u0026#34;items\u0026#34; in response.json and len(response.json[\u0026#34;items\u0026#34;]) == 5 # VERIFY ATTRIBUTES OF WIDGETS #1-5 for i in range(0, len(response.json[\u0026#34;items\u0026#34;])): item = response.json[\u0026#34;items\u0026#34;][i] assert \u0026#34;name\u0026#34; in item and item[\u0026#34;name\u0026#34;] == NAMES[i] assert \u0026#34;info_url\u0026#34; in item and item[\u0026#34;info_url\u0026#34;] == URLS[i] assert \u0026#34;deadline\u0026#34; in item and DEADLINES[i] in item[\u0026#34;deadline\u0026#34;] assert \u0026#34;owner\u0026#34; in item and \u0026#34;email\u0026#34; in item[\u0026#34;owner\u0026#34;] assert item[\u0026#34;owner\u0026#34;][\u0026#34;email\u0026#34;] == ADMIN_EMAIL # REQUEST PAGINATED LIST OF WIDGETS: 5 PER PAGE, PAGE #2 response = retrieve_widget_list(client, access_token, page=2, per_page=5) assert response.status_code == HTTPStatus.OK # VERIFY PAGINATION ATTRIBUTES FOR PAGE #2 assert \u0026#34;has_prev\u0026#34; in response.json and response.json[\u0026#34;has_prev\u0026#34;] assert \u0026#34;has_next\u0026#34; in response.json and not response.json[\u0026#34;has_next\u0026#34;] assert \u0026#34;page\u0026#34; in response.json and response.json[\u0026#34;page\u0026#34;] == 2 assert \u0026#34;total_pages\u0026#34; in response.json and response.json[\u0026#34;total_pages\u0026#34;] == 2 assert \u0026#34;items_per_page\u0026#34; in response.json and response.json[\u0026#34;items_per_page\u0026#34;] == 5 assert \u0026#34;total_items\u0026#34; in response.json and response.json[\u0026#34;total_items\u0026#34;] == 7 assert \u0026#34;items\u0026#34; in response.json and len(response.json[\u0026#34;items\u0026#34;]) == 2 # VERIFY ATTRIBUTES OF WIDGETS #6-7 for i in range(5, response.json[\u0026#34;total_items\u0026#34;]): item = response.json[\u0026#34;items\u0026#34;][i - 5] assert \u0026#34;name\u0026#34; in item and item[\u0026#34;name\u0026#34;] == NAMES[i] assert \u0026#34;info_url\u0026#34; in item and item[\u0026#34;info_url\u0026#34;] == URLS[i] assert \u0026#34;deadline\u0026#34; in item and DEADLINES[i] in item[\u0026#34;deadline\u0026#34;] assert \u0026#34;owner\u0026#34; in item and \u0026#34;email\u0026#34; in item[\u0026#34;owner\u0026#34;] assert item[\u0026#34;owner\u0026#34;][\u0026#34;email\u0026#34;] == ADMIN_EMAIL # REQUEST PAGINATED LIST OF WIDGETS: 10 PER PAGE, PAGE #1 response = retrieve_widget_list(client, access_token, page=1, per_page=10) assert response.status_code == HTTPStatus.OK # VERIFY PAGINATION ATTRIBUTES FOR PAGE #1 assert \u0026#34;has_prev\u0026#34; in response.json and not response.json[\u0026#34;has_prev\u0026#34;] assert \u0026#34;has_next\u0026#34; in response.json and not response.json[\u0026#34;has_next\u0026#34;] assert \u0026#34;page\u0026#34; in response.json and response.json[\u0026#34;page\u0026#34;] == 1 assert \u0026#34;total_pages\u0026#34; in response.json and response.json[\u0026#34;total_pages\u0026#34;] == 1 assert \u0026#34;items_per_page\u0026#34; in response.json and response.json[\u0026#34;items_per_page\u0026#34;] == 10 assert \u0026#34;total_items\u0026#34; in response.json and response.json[\u0026#34;total_items\u0026#34;] == 7 assert \u0026#34;items\u0026#34; in response.json and len(response.json[\u0026#34;items\u0026#34;]) == 7 # VERIFY ATTRIBUTES OF WIDGETS #1-7 for i in range(0, len(response.json[\u0026#34;items\u0026#34;])): item = response.json[\u0026#34;items\u0026#34;][i] assert \u0026#34;name\u0026#34; in item and item[\u0026#34;name\u0026#34;] == NAMES[i] assert \u0026#34;info_url\u0026#34; in item and item[\u0026#34;info_url\u0026#34;] == URLS[i] assert \u0026#34;deadline\u0026#34; in item and DEADLINES[i] in item[\u0026#34;deadline\u0026#34;] assert \u0026#34;owner\u0026#34; in item and \u0026#34;email\u0026#34; in item[\u0026#34;owner\u0026#34;] assert item[\u0026#34;owner\u0026#34;][\u0026#34;email\u0026#34;] == ADMIN_EMAIL # REQUEST PAGINATED LIST OF WIDGETS: DEFAULT PARAMETERS response = retrieve_widget_list(client, access_token) assert response.status_code == HTTPStatus.OK # VERIFY PAGINATION ATTRIBUTES FOR PAGE #1 assert \u0026#34;has_prev\u0026#34; in response.json and not response.json[\u0026#34;has_prev\u0026#34;] assert \u0026#34;has_next\u0026#34; in response.json and not response.json[\u0026#34;has_next\u0026#34;] assert \u0026#34;page\u0026#34; in response.json and response.json[\u0026#34;page\u0026#34;] == 1 assert \u0026#34;total_pages\u0026#34; in response.json and response.json[\u0026#34;total_pages\u0026#34;] == 1 assert \u0026#34;items_per_page\u0026#34; in response.json and response.json[\u0026#34;items_per_page\u0026#34;] == 10 assert \u0026#34;total_items\u0026#34; in response.json and response.json[\u0026#34;total_items\u0026#34;] == 7 assert \u0026#34;items\u0026#34; in response.json and len(response.json[\u0026#34;items\u0026#34;]) == 7 # VERIFY ATTRIBUTES OF WIDGETS #1-7 for i in range(0, len(response.json[\u0026#34;items\u0026#34;])): item = response.json[\u0026#34;items\u0026#34;][i] assert \u0026#34;name\u0026#34; in item and item[\u0026#34;name\u0026#34;] == NAMES[i] assert \u0026#34;info_url\u0026#34; in item and item[\u0026#34;info_url\u0026#34;] == URLS[i] assert \u0026#34;deadline\u0026#34; in item and DEADLINES[i] in item[\u0026#34;deadline\u0026#34;] assert \u0026#34;owner\u0026#34; in item and \u0026#34;email\u0026#34; in item[\u0026#34;owner\u0026#34;] assert item[\u0026#34;owner\u0026#34;][\u0026#34;email\u0026#34;] == ADMIN_EMAIL   Wow, that test case is a doozy! Let\u0026rsquo;s take a look at what exactly is being tested since there\u0026rsquo;s a lot going on here:\n  Lines 8-36: These three lists contain the data required to create seven widget instances. Each item in each list is unique, in order to create test data that is as realistic as possible. This also makes the verifications that we perform more effective since we prevent any \"false positive\" results from occurring.\n  Lines 45-53: The seven widget instances are added to the database. Why seven? When we created the pagination_reqparser, we defined that the only valid values for the per_page parameter are [5, 10, 25, 50, 100]. Creating seven widget instances and requesting a paginated list with per_page=5 allows us to verify that our pagination logic is working correctly since the expected result is that the seven widget instances will generate two pages: the first with a total of five widgets, and the second with a total of two widgets.\n  Lines 56-57: We use the retrieve_widget_list function that we created in tests.util to send a request for the first page of widgets with 5 items per page (page=1, per_page=5), and verify that the HTTP status code of the response is 200 HTTPStatus.OK.\n  Lines 60-65: Since the request was sucessful, we know that response.json contains the pagination object. If you need to remind yourself how this object is structured, click here.\nWe verify that the pagination object correctly represents the first page of two total pages of widgets containing five widgets per page. For example, has_prev should be False, has_next should be True, page should be equal to 1, etc.\n  Line 66: Next, we verify that items (which is the list of widgets on page 1) contains five elements.\n  Lines 69-75: After verifying the pagination attributes, we verify that the widgets retrieved from the database contain the exact values for widget_name, info_url and deadline_str that were taken from the NAMES, URLS and DEADLINES lists when we created the first five widget instances and added them to the database. This is done in a simple, compact way with a for loop.\n  Lines 78-79: After verifying all data on page 1, we send a request for the second page of widgets with 5 items per page (page=2, per_page=5), which should return a response with status code 200 (HTTPStatus.OK).\n  Lines 82-87: Next, we verify that the pagination object correctly represents the second page of two total pages of widgets containing five widgets per page. For example, has_prev should be True, has_next should be False, page should be equal to 2, etc.\n  Line 88: Since there are seven total widgets in the database and page 1 contained #1-5, we expect that items on page 2 will contain two elements (i.e., widgets #6-7)\n  Lines 91-97: In the same way that we did for page 1, we verify that the widgets retrieved from the database contain the exact values for widget_name, info_url and deadline_str that were taken from the NAMES, URLS and DEADLINES lists when we created the final two widget instances and added them to the database.\nAt this point, we have verified that our pagination scheme is working correctly when a client requests a list of widgets with five items per page. Let's see if everything is working as expected with a larger per_page value.\n  Lines 100-101: We send a request for the first page of widgets with 10 items per page (page=1, per_page=10), and verify that the HTTP status code of the response is 200 HTTPStatus.OK.\n  Lines 104-109: We verify that the pagination object correctly represents the first page of one total page of widgets containing ten widgets per page. For example, has_prev should be False, has_next should be False, page should be equal to 1, etc.\n  Lines 110: Next, we verify that items (which is the list of widgets on page 1) contains seven elements.\n  Lines 113-119: This should look familiar to you by now. We verify that the widgets retrieved from the database contain the exact values for widget_name, info_url and deadline_str that were taken from the NAMES, URLS and DEADLINES lists when we created all seven widget instances and added them to the database.\n  Lines 122-141: I'm going to summarize the rest of the test case since it is nearly identical to the page=1, per_page=10 results that we just explained. When requesting the list of widgets in Line 122, no values are specified for page and per_page. Since we specified that default values of page=1 and per_page=10 will be used when the request does not contain either value, we perform the same verifications which we just explained for Lines 103-119.\n   Are you still with me? I know that this section has been quite tedious but bear with me! The end is in sight, and when we get there you will have a fully-featured REST API with a satisfactory level of test-coverage. Just imagine how excited you will be.\nRetrieve Widget Hopefully you can figure out what we need to do at this point, since we did the same thing for the previous two sets of test cases: create a function that sends a request to the API endpoint responsible for handling the current operation, with the correct HTTP method type. Since we now are testing the retrieve widget operation, we need to use the Flask test client to send a GET request to the api.widget endpoint, specifying the name of the widget we wish to retrieve in the URL path.\nOpen tests/util.py and add the content below:\n69 70 71 72 73  def retrieve_widget(test_client, access_token, widget_name): return test_client.get( url_for(\u0026#34;api.widget\u0026#34;, name=widget_name), headers={\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {access_token}\u0026#34;}, )   This is very similar to the retrieve_widget_list function that we created in the same file so it should be easy to understand. Create a new file named test_retrieve_widget.py in the tests folder and add everything you see below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45  \u0026#34;\u0026#34;\u0026#34;Test cases for GET requests sent to the api.widget API endpoint.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from tests.util import ( ADMIN_EMAIL, EMAIL, DEFAULT_NAME, DEFAULT_URL, DEFAULT_DEADLINE, login_user, create_widget, retrieve_widget, ) def test_retrieve_widget_non_admin_user(client, db, admin, user): response = login_user(client, email=ADMIN_EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = create_widget(client, access_token) assert response.status_code == HTTPStatus.CREATED response = login_user(client, email=EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = retrieve_widget(client, access_token, widget_name=DEFAULT_NAME) assert response.status_code == HTTPStatus.OK assert \u0026#34;name\u0026#34; in response.json and response.json[\u0026#34;name\u0026#34;] == DEFAULT_NAME assert \u0026#34;info_url\u0026#34; in response.json and response.json[\u0026#34;info_url\u0026#34;] == DEFAULT_URL assert \u0026#34;deadline\u0026#34; in response.json and DEFAULT_DEADLINE in response.json[\u0026#34;deadline\u0026#34;] assert \u0026#34;owner\u0026#34; in response.json and \u0026#34;email\u0026#34; in response.json[\u0026#34;owner\u0026#34;] assert response.json[\u0026#34;owner\u0026#34;][\u0026#34;email\u0026#34;] == ADMIN_EMAIL def test_retrieve_widget_does_not_exist(client, db, user): response = login_user(client, email=EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = retrieve_widget(client, access_token, widget_name=DEFAULT_NAME) assert response.status_code == HTTPStatus.NOT_FOUND assert ( \u0026#34;message\u0026#34; in response.json and f\u0026#34;{DEFAULT_NAME} not found in database\u0026#34; in response.json[\u0026#34;message\u0026#34;] )   The first test case, test_retrieve_widget_non_admin_user, is a basic happy-path scenario with one small wrinkle. This operation requires a valid access token, but does not require administrator privileges. So, while the request to create a new widget is sent by the admin user (Line 20), the request to retrieve the same widget is sent by the regular non-admin user (Line 26). After that request succeeds, we verify the attributes of the retrieved widget match the values used to create it.\nThe second test case, test_retrieve_widget_does_not_exist, is a very simple unhappy-path scenario. The first four lines (Lines 37-40) are exactly the same as Lines 23-26 in the previous test case. Why is this noteworthy? Because in this case, we are attempting to retrieve a widget with name=\u0026quot;some-widget\u0026quot; before it has been created and added to the database. We expect to receive a response with status code 404 (HTTPStatus.NOT_FOUND), and an error message explaining that there is no widget with that name in the database (Lines 41-45).\nUpdate Widget You know the routine by now, open tests/utils.py, add the update_widget function and save the file:\n76 77 78 79 80 81 82  def update_widget(test_client, access_token, widget_name, info_url, deadline_str): return test_client.put( url_for(\u0026#34;api.widget\u0026#34;, name=widget_name), headers={\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {access_token}\u0026#34;}, data=f\u0026#34;info_url={info_url}\u0026amp;deadline={deadline_str}\u0026#34;, content_type=\u0026#34;application/x-www-form-urlencoded\u0026#34;, )   update_widget is (obviously) the function that we will use to send a PUT request to the api.widget endpoint. Next, create a new file named test_update_widget.py in tests, enter the content below and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41  \u0026#34;\u0026#34;\u0026#34;Test cases for GET requests sent to the api.widget API endpoint.\u0026#34;\u0026#34;\u0026#34; from datetime import date, timedelta from http import HTTPStatus from tests.util import ( ADMIN_EMAIL, DEFAULT_NAME, login_user, create_widget, retrieve_widget, update_widget, ) UPDATED_URL = \u0026#34;https://www.newurl.com\u0026#34; UPDATED_DEADLINE = (date.today() + timedelta(days=5)).strftime(\u0026#34;%m/%d/%y\u0026#34;) def test_update_widget(client, db, admin): response = login_user(client, email=ADMIN_EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = create_widget(client, access_token) assert response.status_code == HTTPStatus.CREATED response = update_widget( client, access_token, widget_name=DEFAULT_NAME, info_url=UPDATED_URL, deadline_str=UPDATED_DEADLINE, ) assert response.status_code == HTTPStatus.OK response = retrieve_widget(client, access_token, widget_name=DEFAULT_NAME) assert response.status_code == HTTPStatus.OK assert \u0026#34;name\u0026#34; in response.json and response.json[\u0026#34;name\u0026#34;] == DEFAULT_NAME assert \u0026#34;info_url\u0026#34; in response.json and response.json[\u0026#34;info_url\u0026#34;] == UPDATED_URL assert \u0026#34;deadline\u0026#34; in response.json and UPDATED_DEADLINE in response.json[\u0026#34;deadline\u0026#34;] assert \u0026#34;owner\u0026#34; in response.json and \u0026#34;email\u0026#34; in response.json[\u0026#34;owner\u0026#34;] assert response.json[\u0026#34;owner\u0026#34;][\u0026#34;email\u0026#34;] == ADMIN_EMAIL   This is the normal, happy-path scenario for updating an existing widget. After logging in with an admin user and creating a widget (Lines 19-23), we update the values for info_url and deadline_str (Lines 25-30). Next, we send a request to retrieve the same widget from the database (Line 33) so that we can verify that the info_url and deadline_str values were successfully updated (Lines 37-38).\n  Please keep in mind the level of detail and attention that was paid to the formal definition of a PUT request earlier in this section. The test coverage for the update widget operation should be designed to verify that our implementation adheres to the requirements from Section 4.3.4 of RFC 7321.\n  Delete Widget There is only a single, remaining CRUD operation in Table 1 that we need to create test coverage for: delete widget. Without further ado, open tests/utils.py, add the delete_widget function and save the file:\n85 86 87 88 89  def delete_widget(test_client, access_token, widget_name): return test_client.delete( url_for(\u0026#34;api.widget\u0026#34;, name=widget_name), headers={\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {access_token}\u0026#34;}, )   delete_widget is the function that we will use to send a DELETE request to the api.widget endpoint. Next, create a new file named test_delete_widget.py in tests, enter the content below and save the file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41  \u0026#34;\u0026#34;\u0026#34;Test cases for GET requests sent to the api.widget API endpoint.\u0026#34;\u0026#34;\u0026#34; from http import HTTPStatus from tests.util import ( ADMIN_EMAIL, EMAIL, DEFAULT_NAME, FORBIDDEN, login_user, create_widget, retrieve_widget, delete_widget, ) def test_delete_widget(client, db, admin): response = login_user(client, email=ADMIN_EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = create_widget(client, access_token) assert response.status_code == HTTPStatus.CREATED response = delete_widget(client, access_token, widget_name=DEFAULT_NAME) assert response.status_code == HTTPStatus.NO_CONTENT response = retrieve_widget(client, access_token, widget_name=DEFAULT_NAME) assert response.status_code == HTTPStatus.NOT_FOUND def test_delete_widget_no_admin_token(client, db, admin, user): response = login_user(client, email=ADMIN_EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = create_widget(client, access_token) assert response.status_code == HTTPStatus.CREATED response = login_user(client, email=EMAIL) assert \u0026#34;access_token\u0026#34; in response.json access_token = response.json[\u0026#34;access_token\u0026#34;] response = delete_widget(client, access_token, widget_name=DEFAULT_NAME) assert response.status_code == HTTPStatus.FORBIDDEN assert \u0026#34;message\u0026#34; in response.json and response.json[\u0026#34;message\u0026#34;] == FORBIDDEN   The first test case, test_delete_widget, is a basic happy-path scenario. The admin user creates a widget, then sends the request to delete the same widget (Lines 17-22). If the delete request was successful, we expect the response to have a status code of 204 (HTTPStatus.NO_CONTENT) (Line 23). To verify that the widget was deleted in another way, the admin user sends a request to retrieve a widget with name equal to the name of the widget which was deleted (Line 24). In this case, we expect the response to have a status code of 404 (HTTPStatus.NOT_FOUND) (Line 25).\nThe second test case, test_delete_widget_no_admin_token, is a very simple unhappy-path scenario. The first five lines (Lines 29-33) are exactly the same as Lines 17-21 in the previous test case. However, this test case differs because now we login as the non-admin user and send a request to delete the widget which was created previously (Lines 35-38). However, since this operation can only be performed by admin users, we expect the response to have a status code of 403 (HTTPStatus.FORBIDDEN), and for the response to contain a field named message explaining that the requested operation can only be performed by users with administrator privileges (Lines 39-4`).\nLet\u0026rsquo;s verify that all of our test cases are still passing by running tox:\n(flask-api-tutorial) flask-api-tutorial $ tox GLOB sdist-make: /Users/aaronluna/Projects/flask-api-tutorial/setup.py py37 create: /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37 py37 installdeps: black, flake8, pydocstyle, pytest, pytest-black, pytest-clarity, pytest-dotenv, pytest-flake8, pytest-flask py37 inst: /Users/aaronluna/Projects/flask-api-tutorial/.tox/.tmp/package/1/flask-api-tutorial-0.1.zip py37 installed: alembic==1.4.0,aniso8601==8.0.0,appdirs==1.4.3,attrs==19.3.0,bcrypt==3.1.7,black==19.10b0,certifi==2019.11.28,cffi==1.14.0,chardet==3.0.4,Click==7.0,entrypoints==0.3,flake8==3.7.9,Flask==1.1.1,flask-api-tutorial==0.1,Flask-Bcrypt==0.7.1,Flask-Cors==3.0.8,Flask-Migrate==2.5.2,flask-restx==0.1.1,Flask-SQLAlchemy==2.4.1,idna==2.9,importlib-metadata==1.5.0,itsdangerous==1.1.0,Jinja2==2.11.1,jsonschema==3.2.0,Mako==1.1.1,MarkupSafe==1.1.1,mccabe==0.6.1,more-itertools==8.2.0,packaging==20.1,pathspec==0.7.0,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pycparser==2.19,pydocstyle==5.0.2,pyflakes==2.1.1,PyJWT==1.7.1,pyparsing==2.4.6,pyrsistent==0.15.7,pytest==5.3.5,pytest-black==0.3.8,pytest-clarity==0.3.0a0,pytest-dotenv==0.4.0,pytest-flake8==1.0.4,pytest-flask==0.15.1,python-dateutil==2.8.1,python-dotenv==0.12.0,python-editor==1.0.4,pytz==2019.3,regex==2020.2.20,requests==2.23.0,six==1.14.0,snowballstemmer==2.0.0,SQLAlchemy==1.3.13,termcolor==1.1.0,toml==0.10.0,typed-ast==1.4.1,urllib3==1.25.8,wcwidth==0.1.8,Werkzeug==0.16.1,zipp==3.0.0 py37 run-test-pre: PYTHONHASHSEED='4200454201' py37 run-test: commands[0] | pytest ================================================= test session starts ================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/bin/python cachedir: .tox/py37/.pytest_cache rootdir: /Users/aaronluna/Projects/flask-api-tutorial, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, dotenv-0.4.0, flask-0.15.1, flake8-1.0.4 collected 106 items run.py::FLAKE8 PASSED [ 0%] run.py::BLACK PASSED [ 1%] setup.py::FLAKE8 PASSED [ 2%] setup.py::BLACK PASSED [ 3%] src/flask_api_tutorial/__init__.py::FLAKE8 PASSED [ 4%] src/flask_api_tutorial/__init__.py::BLACK PASSED [ 5%] src/flask_api_tutorial/config.py::FLAKE8 PASSED [ 6%] src/flask_api_tutorial/config.py::BLACK PASSED [ 7%] src/flask_api_tutorial/api/__init__.py::FLAKE8 PASSED [ 8%] src/flask_api_tutorial/api/__init__.py::BLACK PASSED [ 9%] src/flask_api_tutorial/api/exceptions.py::FLAKE8 PASSED [ 10%] src/flask_api_tutorial/api/exceptions.py::BLACK PASSED [ 11%] src/flask_api_tutorial/api/auth/__init__.py::FLAKE8 PASSED [ 12%] src/flask_api_tutorial/api/auth/__init__.py::BLACK PASSED [ 13%] src/flask_api_tutorial/api/auth/business.py::FLAKE8 PASSED [ 14%] src/flask_api_tutorial/api/auth/business.py::BLACK PASSED [ 15%] src/flask_api_tutorial/api/auth/decorators.py::FLAKE8 PASSED [ 16%] src/flask_api_tutorial/api/auth/decorators.py::BLACK PASSED [ 16%] src/flask_api_tutorial/api/auth/dto.py::FLAKE8 PASSED [ 17%] src/flask_api_tutorial/api/auth/dto.py::BLACK PASSED [ 18%] src/flask_api_tutorial/api/auth/endpoints.py::FLAKE8 PASSED [ 19%] src/flask_api_tutorial/api/auth/endpoints.py::BLACK PASSED [ 20%] src/flask_api_tutorial/api/widgets/__init__.py::FLAKE8 PASSED [ 21%] src/flask_api_tutorial/api/widgets/__init__.py::BLACK PASSED [ 22%] src/flask_api_tutorial/api/widgets/business.py::FLAKE8 PASSED [ 23%] src/flask_api_tutorial/api/widgets/business.py::BLACK PASSED [ 24%] src/flask_api_tutorial/api/widgets/dto.py::FLAKE8 PASSED [ 25%] src/flask_api_tutorial/api/widgets/dto.py::BLACK PASSED [ 26%] src/flask_api_tutorial/api/widgets/endpoints.py::FLAKE8 PASSED [ 27%] src/flask_api_tutorial/api/widgets/endpoints.py::BLACK PASSED [ 28%] src/flask_api_tutorial/models/__init__.py::FLAKE8 PASSED [ 29%] src/flask_api_tutorial/models/__init__.py::BLACK PASSED [ 30%] src/flask_api_tutorial/models/token_blacklist.py::FLAKE8 PASSED [ 31%] src/flask_api_tutorial/models/token_blacklist.py::BLACK PASSED [ 32%] src/flask_api_tutorial/models/user.py::FLAKE8 PASSED [ 33%] src/flask_api_tutorial/models/user.py::BLACK PASSED [ 33%] src/flask_api_tutorial/models/widget.py::FLAKE8 PASSED [ 34%] src/flask_api_tutorial/models/widget.py::BLACK PASSED [ 35%] src/flask_api_tutorial/util/__init__.py::FLAKE8 PASSED [ 36%] src/flask_api_tutorial/util/__init__.py::BLACK PASSED [ 37%] src/flask_api_tutorial/util/datetime_util.py::FLAKE8 PASSED [ 38%] src/flask_api_tutorial/util/datetime_util.py::BLACK PASSED [ 39%] src/flask_api_tutorial/util/result.py::FLAKE8 PASSED [ 40%] src/flask_api_tutorial/util/result.py::BLACK PASSED [ 41%] tests/__init__.py::FLAKE8 PASSED [ 42%] tests/__init__.py::BLACK PASSED [ 43%] tests/conftest.py::FLAKE8 PASSED [ 44%] tests/conftest.py::BLACK PASSED [ 45%] tests/test_auth_login.py::FLAKE8 PASSED [ 46%] tests/test_auth_login.py::BLACK PASSED [ 47%] tests/test_auth_login.py::test_login PASSED [ 48%] tests/test_auth_login.py::test_login_email_does_not_exist PASSED [ 49%] tests/test_auth_logout.py::FLAKE8 PASSED [ 50%] tests/test_auth_logout.py::BLACK PASSED [ 50%] tests/test_auth_logout.py::test_logout PASSED [ 51%] tests/test_auth_logout.py::test_logout_token_blacklisted PASSED [ 52%] tests/test_auth_register.py::FLAKE8 PASSED [ 53%] tests/test_auth_register.py::BLACK PASSED [ 54%] tests/test_auth_register.py::test_auth_register PASSED [ 55%] tests/test_auth_register.py::test_auth_register_email_already_registered PASSED [ 56%] tests/test_auth_register.py::test_auth_register_invalid_email PASSED [ 57%] tests/test_auth_user.py::FLAKE8 PASSED [ 58%] tests/test_auth_user.py::BLACK PASSED [ 59%] tests/test_auth_user.py::test_auth_user PASSED [ 60%] tests/test_auth_user.py::test_auth_user_no_token PASSED [ 61%] tests/test_auth_user.py::test_auth_user_expired_token PASSED [ 62%] tests/test_config.py::FLAKE8 PASSED [ 63%] tests/test_config.py::BLACK PASSED [ 64%] tests/test_config.py::test_config_development PASSED [ 65%] tests/test_config.py::test_config_testing PASSED [ 66%] tests/test_config.py::test_config_production PASSED [ 66%] tests/test_create_widget.py::FLAKE8 PASSED [ 67%] tests/test_create_widget.py::BLACK PASSED [ 68%] tests/test_create_widget.py::test_create_widget_valid_name[abc123] PASSED [ 69%] tests/test_create_widget.py::test_create_widget_valid_name[widget-name] PASSED [ 70%] tests/test_create_widget.py::test_create_widget_valid_name[new_widget1] PASSED [ 71%] tests/test_create_widget.py::test_create_widget_valid_deadline[02/29/2020] PASSED [ 72%] tests/test_create_widget.py::test_create_widget_valid_deadline[2020-02-29] PASSED [ 73%] tests/test_create_widget.py::test_create_widget_valid_deadline[Mar 03 2020] PASSED [ 74%] tests/test_create_widget.py::test_create_widget_invalid_deadline[1/1/1970] PASSED [ 75%] tests/test_create_widget.py::test_create_widget_invalid_deadline[2020-02-26] PASSED [ 76%] tests/test_create_widget.py::test_create_widget_invalid_deadline[a long time ago, in a galaxy far, far away] PASSED [ 77%] tests/test_create_widget.py::test_create_widget_already_exists PASSED [ 78%] tests/test_create_widget.py::test_create_widget_no_admin_token PASSED [ 79%] tests/test_delete_widget.py::FLAKE8 PASSED [ 80%] tests/test_delete_widget.py::BLACK PASSED [ 81%] tests/test_delete_widget.py::test_delete_widget PASSED [ 82%] tests/test_delete_widget.py::test_delete_widget_no_admin_token PASSED [ 83%] tests/test_retrieve_widget.py::FLAKE8 PASSED [ 83%] tests/test_retrieve_widget.py::BLACK PASSED [ 84%] tests/test_retrieve_widget.py::test_retrieve_widget_non_admin_user PASSED [ 85%] tests/test_retrieve_widget.py::test_retrieve_widget_does_not_exist PASSED [ 86%] tests/test_retrieve_widget_list.py::FLAKE8 PASSED [ 87%] tests/test_retrieve_widget_list.py::BLACK PASSED [ 88%] tests/test_retrieve_widget_list.py::test_retrieve_paginated_widget_list PASSED [ 89%] tests/test_update_widget.py::FLAKE8 PASSED [ 90%] tests/test_update_widget.py::BLACK PASSED [ 91%] tests/test_update_widget.py::test_update_widget PASSED [ 92%] tests/test_user.py::FLAKE8 PASSED [ 93%] tests/test_user.py::BLACK PASSED [ 94%] tests/test_user.py::test_encode_access_token PASSED [ 95%] tests/test_user.py::test_decode_access_token_success PASSED [ 96%] tests/test_user.py::test_decode_access_token_expired PASSED [ 97%] tests/test_user.py::test_decode_access_token_invalid PASSED [ 98%] tests/util.py::FLAKE8 PASSED [ 99%] tests/util.py::BLACK PASSED [100%] =================================================== warnings summary =================================================== src/flask_api_tutorial/api/exceptions.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/model.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, MutableMapping src/flask_api_tutorial/api/exceptions.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/api.py:28: DeprecationWarning: The import 'werkzeug.cached_property' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.utils import cached_property' instead. from werkzeug import cached_property src/flask_api_tutorial/api/exceptions.py::FLAKE8 /Users/aaronluna/Projects/flask-api-tutorial/.tox/py37/lib/python3.7/site-packages/flask_restx/swagger.py:12: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working from collections import OrderedDict, Hashable -- Docs: https://docs.pytest.org/en/latest/warnings.html =========================================== 106 passed, 3 warnings in 27.42s =========================================== _______________________________________________________ summary ________________________________________________________ py37: commands succeeded congratulations :) As you can see, thanks to pytest and various plugins, we have a faily robust test suite as well as automated enforcement of our code formatter (black) and linter (flake8) for all files inside the src and tests folders.\nSwagger UI It\u0026rsquo;s been a while since we looked at the Swagger UI page, and it has changed considerably due to the API endpoints and models we created in this section. Fire up the development server with the flask run command and navigate to http://localhost:5000/api/v1/ui. You should see the page shown below:\n Figure 1 - Swagger UI with all API routes\n  You should spend some time testing all of the endpoints. If you need a refresher on requesting and retrieiving an access token, and how to use the access token to send a request for a protected resource, refer to this step-by-step explanation in Part 4\nCheckpoint At long last, we have implemented all of the required features for this project. However, that does not mean that our application is finished and ready to be deployed/shared with the world. In fact, the current state of the project is lacking many vital features and tools to make it more maintainable and approachable for users in the open-source community. At this point we will focus on improving the quality of our project by focusing on test coverage, CICD, logging and security hardening.\nI am always interested in hearing feedback on the tutorial, I hope you have enjoyed what has been provided so far.\nUser Management/JWT Authentication\nNew users can register by providing an email address and password\nExisting users can obtain a JWT by providing their email address and password\nJWT contains the following claims: time the token was issued, time the token expires, a value that identifies the user, and a flag that indicates if the user has administrator access\nJWT is sent in access_token field of HTTP response after successful authentication with email/password\nJWTs must expire after 1 hour (in production)\nJWT is sent by client in Authorization field of request header\nRequests must be rejected if JWT has been modified\nRequests must be rejected if JWT is expired\nIf user logs out, their JWT is immediately invalid/expired\nIf JWT is expired, user must re-authenticate with email/password to obtain a new JWT\n API Resource: Widget List\nAll users can retrieve a list of all widgets\nAll users can retrieve individual widgets by name\nUsers with administrator access can add new widgets to the database\nUsers with administrator access can edit existing widgets\nUsers with administrator access can delete widgets from the database\nThe widget model contains attributes with URL, datetime, timedelta and bool data types, along with normal text fields.\nURL and datetime values must be validated before a new widget is added to the database (and when an existing widget is updated).\nThe widget model contains a \"name\" attribute which must be a string value containing only lowercase-letters, numbers and the \"-\" (hyphen character) or \"_\" (underscore character).\nThe widget model contains a \"deadline\" attribute which must be a datetime value where the date component is equal to or greater than the current date. The comparison does not consider the value of the time component when this comparison is performed.\nWidget name must be validated before a new widget is added to the database (and when an existing widget is updated).\nIf input validation fails either when adding a new widget or editing an existing widget, the API response must include error messages indicating the name(s) of the fields that failed validation.\n  ","href":"/series/flask-api-tutorial/part-6/","title":"How To: Create a Flask API with JWT-Based Authentication (Part 6)"}]