Setting Up A Pelican Blog with Cloudflare, Github Pages, and Travis CI

  • By Jamie Duncan
  • Sat 26 November 2016
  • Updated on Sat 26 November 2016

It’s all come full circle. Again.

Way back when we had to build static HTML sites. Then somebody invented Joomla or Wordpress or whatever thefirst web framework was, and we moved into the age of dynamic, database-driven websites.

But they had a problem. They’re hard to make microservice-y. It’s hard to make a CMS not a monolith. People are working on it, but it’s not how they were designed.

Now, coupled with some of the amazing stuff that javascript can do, we’re moving back to generating static sites and makin them feel dynamic.

Overview

The workflow I’ve adopted uses a handful of things

  • Pelican for configuration and content creation

  • Asciidoc for lightweight markdown

  • Github for source control

  • Github pages for serving the rendered content

  • Travis CI for tying it all together

  • Cloudflare for managing DNS and SSL termination

I certainly didn’t come up with this process on my own. But I did have to take the 4-5 howtos I found on the internet and consolidate and tweak them to work for me. So I’m documenting this process to help me the next time I need it. And to maybe help you too.

Pelican Static Site Generator

The whole key to this process is to be able to easily manage a static HTML site, updating and changing it when you need (or want) to. Luckily, Paul Frields turned me on to Pelican. It’s a site content generator that’s written in Python. It has a ton of easily applicable and modifiable themes, and tons of plugins to make your site powerful and integrated into lots of other services so it can do good work for you.

Getting started with Pelican

Pelican is available through PyPi. We use pip to install it.

$ sudo pip install pelican

Once we have it installed we need to get a Pelican site started. Luckily, it comes with a small applicaton called pelican-quickstart. It’s an interactive script that sets up a basic Pelican website in the directory you specify. For all of the parameters you give answers for, they can be edited later in pelicanconf.py in the root directory of your project

jduncan@dhcp-192-168-1-140 ~$ pelican-quickstart
Welcome to pelican-quickstart v3.6.3.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files
needed by Pelican.

> Where do you want to create your new web site? [.] /home/jduncan/Code/samplesite
> What will be the title of this web site? Example Site
> Who will be the author of this web site? Jamie Duncan
> What will be the default language of this web site? [en]
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n) Y
> What is your URL prefix? (see above example; no trailing slash) http://blog.example.com
> Do you want to enable article pagination? (Y/n) Y
> How many articles per page do you want? [10]
> What is your time zone? [Europe/Paris] America/New_York
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n) Y
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n)
> Do you want to upload your website using FTP? (y/N) N
> Do you want to upload your website using SSH? (y/N)
> Do you want to upload your website using Dropbox? (y/N)
> Do you want to upload your website using S3? (y/N)
> Do you want to upload your website using Rackspace Cloud Files? (y/N)
> Do you want to upload your website using GitHub Pages? (y/N) y
> Is this your personal page (username.github.io)? (y/N) y
Done. Your new project is available at /home/jduncan/Code/samplesite

Here’s what a basic layout looks like

jduncan@dhcp-192-168-1-140 ~$ ll /home/jduncan/Code/samplesite/
total 32
drwxr-xr-x. 2 jduncan wheel 4096 Nov 25 12:40 content
-rwxr-xr-x. 1 jduncan wheel 2201 Nov 25 12:40 develop_server.sh
-rw-r--r--. 1 jduncan wheel 2473 Nov 25 12:40 fabfile.py
-rw-r--r--. 1 jduncan wheel 4327 Nov 25 12:40 Makefile
drwxr-xr-x. 2 jduncan wheel 4096 Nov 25 12:40 output
-rw-r--r--. 1 jduncan wheel  870 Nov 25 12:40 pelicanconf.py
-rw-r--r--. 1 jduncan wheel  531 Nov 25 12:40 publishconf.py

The most important files in here are pelicanconf.py (your site’s configuration), Makefile (which we’ll use later to push our rendered content up to Github Pages) and the content directory, which is where we’ll put our actual stuff to render.

Pelican Basics

I’m not going to go into the workflows in Pelican because they are well documented. But a 10k foot overview is this

  1. Create your site content using

    1. reStructuredText

    2. Markdown

    3. Asciidoc (my personal preference).

  2. Use Pelican to render your markdown into HTML, sucking in all of your configurations, plugins and themes into a polished websites

  3. Upload that rendered content into your hosting platform of choice

  4. World Domination

My Pelican Configuration

My pelicanconf.py is a little too long to paste in whole into this blog post, so I’ll just link it. But a few of the highlights

Atom and RSS Feed Generation
FEED_DOMAIN = 'blog.jeduncan.com'
FEED_ALL_ATOM = 'feeds/all.atom.xml'
CATEGORY_FEED_ATOM = None
TRANSLATION_FEED_ATOM = None
AUTHOR_FEED_ATOM = None
AUTHOR_FEED_RSS = None
FEED_RSS = 'feeds/rss.xml'
Auto-creation of thumbnails for all images
# thumbnailer settings
IMAGE_PATH = 'images'
THUMBNAIL_DIR = 'images'
THUMBNAIL_KEEP_NAME = True
THUMBNAIL_KEEP_TREE = True
Pelican Plugins I am Currently Using
PLUGINS = ['asciidoc_reader',
            'sitemap',
            'gravatar',
            'filetime_from_git',
            'gallery',
            'thumbnailer',
            'disqus_static',]

The docs for these plugins explain what they do and how to use them very well. The most important one to me is asciidoc_reader

Asciidoc

I discovered asciidoc, specifically asciidoctor several projects ago. It lets me creat fast and powerful markdown for documentation and websites. If you have to write a lot of stuff and haven’t checked it out before, I highly recommend you give it a look.

Pelican Theme

There are dozens and dozens of themes available for Pelican. To keep it as simple as possible, I simply copy a theme’s contents from the github repo for them all into ~/pelican-theme for my project. That way I don’t have to change my config when I want to change my theme (which is often). They are jinja2 templates and CSS. So they are easy to understand and modify to make them your own (with proper attribution, of course).

Creating Content

Once you have your project configured, you need actual content. That is kept in, surprisingly enough, in the content directory in your project. You can include posts, pages, and static content within this directory. Just refer to the documentation on how to structure it and how to generate your nice new website’s code.

Summary

The ease of use and power built into Pelican is a great reason to give it a try for this type of website. When you couple that with asciidoc, you have a website that is 100% maintained in a single codebase and can be updated super-fast.

Web Hosting Setup

Paying for web hosting is so 2013. With Github Pages you can host static websites easily and for free. The way it works is pretty simple. Github Pages lets you serve static content for free, and even use your own custom domain to point to it.

Take this blog for example. It is available at https://jduncan-rva.github.io/blog as well as at https://blog.jeduncan.com. To do that I have to do a few things.

For my custom domain

Github Pages is designed to easily work with custom domains. Only a few steps are required to get Pelican to set up my rendered content properly.

  1. I need to create a CNAME record for blog.jeduncan.com that points to jduncan-rva.github.io. For my DNS, I’m using Cloudflare for a few reasons I’ll talk about later.

  2. After that I created a file named CNAME in the content/extras directory in my site code. It needs to contain one line that is the custom domain I want my site to resolve to. In my case, blog.jeduncan.com

CNAME file
$ cat content/extra/CNAME
blog.jeduncan.com

This file needs to be in the root directoy of my github pages hosted website. For this to happen properly, I add the following configuration in my pelicanconf.py.

custom domain config for Pelican
STATIC_PATHS = [
    'images',
    'extra/robots.txt',
    'extra/favicon.ico',
    'extra/CNAME'
]

EXTRA_PATH_METADATA = {
    'extra/robots.txt': {'path': 'robots.txt'},
    'extra/favicon.ico': {'path': 'favicon.ico'},
    'extra/CNAME': {'path': 'CNAME'}
}

This tells Pelican that the files in STATIC_PATHS should not be interpreted through the Pelican engine. The EXTRA_PATH_METADATA parameter tells Pelican that some of the static content shouldn’t be placed in their default location when they are rendered.

At this point, once my CNAME record is created, github takes the request and handles the rest to get the HTTP request to the proper site.

Rendering Content

At this point we have code that can become a website. But how do we get that content into something that looks nice in Chrome?

Content by hand

the pelican command has a content parameter that takes my config and content and renders it all out for me. By default it puts all of my content into the output directory. From within my site’s checked out code I simply run

samplesite$ pelican content
WARNING: No valid files found in content.
Done: Processed 0 articles, 0 drafts, 0 pages and 0 hidden pages in 0.06 seconds.

Since I’ve created no content for this sample site it didn’t generate any actual pages. But it did create my skeleton website.

samplesite$ ll output/
total 24
-rw-r--r--. 1 jduncan wheel 2090 Nov 25 16:12 archives.html
-rw-r--r--. 1 jduncan wheel 2111 Nov 25 16:12 authors.html
-rw-r--r--. 1 jduncan wheel 2007 Nov 25 16:12 categories.html
-rw-r--r--. 1 jduncan wheel 2058 Nov 25 16:12 index.html
-rw-r--r--. 1 jduncan wheel 2106 Nov 25 16:12 tags.html
drwxr-xr-x. 4 jduncan wheel 4096 Nov 25 16:12 theme

Putting Content out for Github Pages

By default, any repository can serve github pages content by creating and populating a gh-pages code branch. So that’s exactly what we’ll do. I actually found a tool out there called ghp-import that will take my output directory and add its contents to the gh-pages branch of the repository it’s in and upload it. It’s pretty awesome and easy to use.

Automating All the things

At this point, you could create content, configure your domain to point to the traffic, and then push it all to the proper place in your github repo to serve the content. But that’s way too manual, so let’s automate all the things to do that automatically for me.

Pelican makefile

When you created your Pelican directory, it created a makefile. One of the options inside this make file is to publish content to github. But the makefile won’t do exactly what we need it to do, so we tweak it a little bit.

Changes made to Pelican makefile
github: publish
        ghp-import -m "Generate Pelican site with Travis CI" $(OUTPUTDIR)
        git push -fq https://$(GH_USER):$(GH_TOKEN)@github.com/$(GH_USER)/$(GH_REPO).git $(GH_BRANCH)

OUTPUTDIR is set to ~/output earlier in the makefile. All of the GH_* variables are set in our CI system’s environment at build time.

Travis CI

We’re using Travis CI to build out site for us every time we push a commit to master. It will require a Github Access Token so it can automatically push our content for us.

I learned that you will need to add the repo scope to your API token for it to work properly. Other docs on the interwebs say different, so YMMV, but it’s what worked for me.

Once we enable our site’s repository in Travis, we then need to add a .travis.yml to our codebase. Here is mine for reference and explanation

travis.yaml
  language: python
  branches:
    only:
    - master
  install:
  - pip install pelican ghp-import python-dateutil gitpython Pillow disqus-python
  addons:
    apt:
      packages:
      - asciidoc
  script:
  - make publish github
  env:
    global:
    - GH_USER=jduncan-rva
    - GH_REPO=blog
    - GH_BRANCH=gh-pages
    - secure: somerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhashsomerandomhash

Here you can see that

  • I’m only building out the master branch

  • I’m installing several python packages, as well as asciidoc so all of my dependencies for plugins are met

  • I’m using my modified makefile to do what I need it to do

  • I’m setting my GH_* variables as environment variables to be read during the build process

For my API Access token, I’m actually able to use the travis ruby gem to encrypt it so people can’t make random commits to my directory. This will add the encrypted token to .travis.yml in your curent working directory.

SSL and Github Pages

One thing I did run into is that Github Pages doesn’t do a great job of handling SSL for custom domains. I was able to get around this by using Cloudflare and their flexible SSL option. Cloudflare acts as a CDN for my site and handles SSL termination for me. So the SSL certificate wonkiness of Github Pages is masked to the end user. It’s pretty awesome, free, and took 5 minutes to set up.

Summary

Static HTML pages are anything but static these days. With javascript I am able to have comments, galleries, and all sorts of crazy awesome stuff. And with the above method, I’m able to store my site’s code and serve its content from the same github repository. Toss in some easy to do automation with Travis and it’s a great solution for projects / small websites. I could even hook in OAuth2 style stuff if I need to down the road. But as it is, access is maintained by the repository itself.