11 Nov 2024

expandfile History

This note describes the history of expandfile, a simple Unix command line program for expanding templates.

History

I wrote expandfile in 2002 to replace a collection of ad hoc Perl programs that I had been using to maintain the multicians.org website since 1995. Since 2002, I've added features to expandfile occasionally.

I wanted to make it easy to change the site, to ensure website performance remained acceptable, and to prevent or catch mistakes.

Web pages are written in the HTML language and sent from a web server program on a server host to a web browser program running on a user device. I chose to create static HTML pages and have the web server serve them without any change at request time. Because multicians.org had mirror sites where I could not modify the web server configuration or execute code on the server, I could not dynamically generate web pages, even if I wanted to. Furthermore, at the time, people implementing web sites by generating pages dynamically from databases, like bulletin board systems, were encountering web server performance and security problems.

My ISP's web servers supported a "server side include" feature, which let users insert the contents of auxiliary files when serving a web page. When I tried to separate my pages into content and boilerplate, I found that I wanted to have the boilerplate be slightly different for each page, so I sought a method where the boilerplate content could be tailored depending on the page. The organization I chose was:

The idea of a simple macro expander dates back to Christopher Strachey's GPM macro expander, which I had experimented with in the mid 60s on CTSS at MIT Project MAC. Later, I used several computer languages that used pre-processors to add features to their languages, such as ALM, PL/I, and C: they provided features such as including other input files.

RUNOFF is an archetype text-transformation language system; I had used it in the 60s on CTSS. RUNOFF's input is either text to be copied to the output, or commands that change the state of execution and affect how later processing works. Later implementations of RUNOFF-like languages in Multics and elsewhere included the ideas of semantic markup, macro execution, and multi-pass processing.

I decided to write my own source text expansion tool that did not parse the underlying input language, similar to GPM: it would just transform text strings into other text strings, with a minimal way of defining macros. This made the program more general and freed it from dependency on the syntax of the underlying language; I didn't have to write or maintain an HTML parser, and changes in the HTML spec would rarely require the tool to change. As in GPM and some RUNOFFs, I could set and evaluate string variables, and expand macros that accepted string variables as arguments.

I looked at the Unix m4 tool, written by friends of mine from Multics days. It wasn't available for the computer I had then; Perl was available on Unix, Windows, and Mac. Perl was very efficient and supported SQL and regular expressions.

Wherever possible, I used what was already available: I didn't invent a new file storage system: I used used my computer's OS file system; I didn't invent a text file format or editor, using tools provided on my computer; I didn't invent a file caching system: I depended on the server computer's file system caches. I didn't try to replace HTML: I added a few abbreviation methods that expanded into HTML to avoid syntax errors.

I studied the common errors I made when maintaining multicians.org, and chose or wrote tools that would prevent them. I implemented ways to store facts in a database on my personal computer to avoid contradictions between pages. Having each fact in one place was good only if I made sure to regenerate all object pages that used that fact: this led me to use the Unix make command. Another common error was forgetting to update server files when I modified a file on my computer: this led me to use the Unix rsync command.

Using Expandfile to Generate Web Sites

I first used expandfile to translate "HTML with extensions" input (which I called HTMX) into HTML, mostly to include common boilerplate, such as page banners and CSS layout instructions used on all my web pages. I added features to allow source files to set variables that boilerplate files could use, like "page title" and "date updated" in the page header and footer.

Adding builtin functions that could transform variables' values came next, then the ability to capture the output of external shell commands, and then integration with SQL. I used these features to simplify my work flow for maintaining websites I created, and to eliminate special-purpose Perl programs in favor of logic in HTMX web page templates.

The big advance for me was introducing the *block builtin, and the pattern of writing HTMX files that

The wrapper pattern separates site boilerplate from page content, provides an independent source file for each HTML page, and makes it easy to regenerate a single page.

Using expandfile was valuable when I made global changes to every page on a site to change every page's appearance, to conform to changing HTML specifications, to improve support for mobile devices, or to use new browser features. Changes to wrapper files were incorporated into every object page with minimum editing.

I added the *shell builtin function to expandfile so that it could access operating system commands like date and additional helper programs I wrote in Perl, like fmtnum and filedaysold.

Connecting expandfile to my local MySQL database and supporting *sqlloop was the next big step. This provided consistent formatting for lists of people, publications, glossary entries, and website page indexes, and defined a lightweight way for any HTMX file to refer to data from these lists. I used this to have multiple pages refer to facts that were kept in one place in SQL data. These changes reduced the chance that an editing mistake would screw up a whole page, or that I'd forget to update all the pages that used a fact.

The third big step I took was using traditional Unix tools to automate site building and publishing. Using make (created years earlier for Unix by Multician Stu Feldman) to invoke expandfile only when an HTMX file was newer than its corresponding HTML files meant that I could make a one-line change to a file and then just type make install to recompile the minimum number of files and automatically rsync them to the deployment site.

Macros

There were times when I wanted to create repetitive formatting of a sequence of items, like setting up picture galleries. I added the *callv builtin to expandfile to be able to easily execute a series of steps, for example generating the HTML code to display a photo. Then I set up a library of these macros that I could *include into a source file when generating an HTML object.

The macros for inserting IMG tags into a page output a warning if the image file cannot be found; if the file is found, the macros invoke a small helper program using *shell to get the dimensions of the picture so they can be included in the IMG tag's WIDTH and LENGTH parameters. This speeds up the loading of the HTML page.

Another example of macro use is the main expandfile page, whose source you can view. That page has a section for each builtin function implemented by expandfile. The page defines a block called bif and begins each section with an invocation of the bif macro, like *callv,bif,prototype,result,text. Using the macro ensures that each section heading is correctly formatted. The macro also formats a <tr> row for the summary table of all builtin functions, which is added to a text variable that will be inserted into a TABLE of functions using *include. The HTMX source that generates the main page is here.

Security

Several design choices I made were aimed at reducing the chance of attacks on my sites.

Early web sites were attacked from outsiders who found bugs in host operating systems and web servers. I decided that I wanted to use only the simplest features of a web server to transmit the contents of read-only files. This design meant that each web page could be read-only, cached in the server memory, and served with quick response.

I chose not to require dynamic generation of pages, or to allow site visitors to cause any computation on the server. These choices limited some page designs: I couldn't provide a "hit counter" on every page. I ruled out "Chat" and "comment" features: if site visitors couldn't upload any data, there would be less chance that attack code could get executed, and I didn't have to try to detect spam and inappropriate content.

Other Applications of Expandfile

As expandfile developed, I found other uses for it, including reformatting database files and preparing input for other programs such as input to procmail, RSS feed declarations in XML, shell scripts, input to the dot graphical compiler, and XML sitemap files for the Google crawler.

For some features, I use expandfile to expand a template which generates HTMX files which are in turn expanded by expandfile; this lets me do "two pass" expansions so I can add up counters and then display the values in headings above the detailed information.

In 2004, I wrote a web statistics application, Super Webtrax, that uses expandfile to produce a daily web page with charts and tables generated from web server log data loaded into SQL.

For a document formatting application, I extracted data from data files in a proprietary format, translated it to SQL and loaded it into a local MySQL database, used expandfile's *sqlloop builtin to generate HTML, and a browser to generate visually formatted output, printed the browser output to PostScript, and used page impression tools to generate a booklet.

I have also built template files that use the *shell builtin to fire off curl commands that fetch XML data from Web APIs, and then parse the result with *xmlloop to generate HTML reports.

Language Issues

I originally wrote expandfile in Perl 5, and used it on Unix, macOS and Windows, through years of evolution of my program, the Perl language, and the features provided on different platforms. I showed expandfile to friends, but they were put off by the difficulty of installing and configuring the Perl implementation:

Later Improvements

Early versions of expandfile had some features that I later decided were mistakes. Fortunately, nobody but me was using the program, and I knew where all the HTMX source files were. So I backed up the program and sources, created and tested a new program version, modified every source file that had to change, recompiled everything, compared old output to new, and accounted for changes before switching over to the new version.

Some of the changes were bug fixes or added new features, such as *xmlloop and *format. A few changes were made when Perl syntax changed and the program had to be updated.

I made the following changes in early 2021:

Comparison to other approaches

PHP

PHP programs are parsed and interpreted at runtime on every view; their execution can output HTML, access databases, and so on. PHP constructs look like <?php echo '<p>Hello World</p>'; ?> You can set and refer to variables that have string values. PHP has over 1000 builtin functions, including SQL access, and is very popular. I didn't choose PHP, because I wanted a tool that could be deployed without modifying the web server configuration. (PHP also has a long history of security bugs, and I thought they might scare users away from a tool that required it.)

Markdown

Markdown converts a simple text language into HTML. Some Web bulletin board systems allow users to type in their postings in Markdown, and translate the input to HTML. Markdown is often used as a way to collect user comments and input, and allow formatting of the input, without requiring users to learn HTML syntax. There are several different dialects of Markdown. It's easy to learn, but it doesn't support the features I needed for a full-featured web page.

Hugo

Just learned a little about Hugo recently. Hugo is based on Markdown instead of Perl. It is implemented in Go and uses Go's conventions about modules and source organization. Similar to expandfile, Hugo input consists of HTML and extension constructs -- Hugo uses {{ ... }} instead of %[ ... ]% . It has variables, function expansion, and function definition. Looks slick. It supports multi-language translation, templates, themes, and static websites.

Copyright (c) 2018-2024, Tom Van Vleck