My development environment

This is a long post about the development environment I’m currently using.

The Basics

There are lots of technology available for developers that can make development much easier and many are free. I’ve been working on a birding website, using the Perl web framework Mojolicious. My development system is a Macbook Pro. It comes with an old version of Perl, so I used PerlBrew to manage different Perl versions. The editor I use is Atom.

Database Support

Most websites are connected to a database, and my birding website is no different. I’m currently using MySQL. MySQL has support for spatial types, which I hope to take advantage of in the website. The standard way for using a database in Perl is to use DBI and the corresponding DBD driver for the database that is being connected to. Many of these DBD drivers are linked to a C library implementation. However, a few are written that do not link to a C library. A native pure Perl DBD driver is nice because it doesn’t require the database software to be installed. I first experienced a native driver when writing some Go code that connected to MySQL.

Mojolicious has some modules that simplifies querying databases. Being a long time DBI user, I didn’t really see the need for such modules. However, the Mojolicious module Mojo::mysql has an interesting method named migrations. It is way to automatically update/downgrade a database when schema changes are made. It seems like most of the Mojolicious database drivers try to support migrations. Apparently Postgres has the best support for migrations since the DML statements can be rolled back. A great resource on the theory behind migrations can be found here.

The Mojo::mysql driver doesn’t have direct support for DBD::mysqlPP (the pure Perl driver for MySQL). But it is simple enough to get around that limitation by setting the dsn parameter of the new method to dbi:mysqlPP:<remaining parts>. However, I quickly learned that the DBD::mysqlPP driver was missing some functionality and it had a bug. The bug is exhibited itself when DBI’s bind_param method is used. The missing functionality is transaction support. I’ve have a fork of DBD::mysql for those interested in a fix. With those fixes, my Mojolicious app seems to functional, but I have since decided not to pursue the native driver approach.

Docker

For development, the MySQL database is run within a Docker container. The default Docker container for MySQL is very capable and can be used as is. Here’s was my initial startup script:

mysql_opts="-e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=migratus -e MYSQL_USER=migratus -e MYSQL_PASSWORD=migratus"
 docker run -p 3306:3306 --name migratus-db ${mysql_opts} -d mysql:8 --default-authentication-plugin=mysql_native_password

The DBD::mysqlPP driver does not support the newer authentication method, so the --default-authentication-plugin=mysql_native_password is needed.

Carton

With PerlBrew on OS X and MySQL within a Docker container, development works fairly well, but a better way of dealing with Perl application development is to use Carton. With Carton, a file named cpanfile is created that lists the application’s depencies, but only at the highest level. Issuing the command carton install pulls over the listed modules and their dependencies. This also creates a file cpanfile.snapshot which contains any additional modules the dependencies needed. With these two files, another developer can recreate the same environment. Carton installs all the modules in directory named local alongside the application. I’ve long preferred this method of building applications with Perl versus installing modules into the system Perl. My experience with Carton is new, so expect more posts regarding this tool in the future.

Carton and Docker finds a bug

Since the birding website is going to run on a Linux based system, it makes sense that Mojolicious framework is developed on Linux as well. Perl is suppose to be platform neutral, but I ran into an interesting problem when moving the code into a Docker container. I had the following code:

use Base Mojo::mysql;

This worked fine on OS X. However, it broke when moving to Linux. I got the following error:

Can't locate Base.pm in @INC

Base is actually base, but OS X uses a case-insensitive filesystem, so Perl was able to find the module.

Containerizing Perl

To move the Mojolicious app to Docker, I use the following Dockerfile that initializes a base container with a recent Perl and some additional packages:

FROM perl:5.30

RUN cpanm Carton \
    && mkdir -p /usr/src/app
RUN apt-get update
RUN apt install -y default-libmysqlclient-dev default-mysql-client default-mysql-client-core

EXPOSE 3000

WORKDIR /usr/src/app

With this Dockerfile, I created an image:

docker build -t macfisherman/migratus .

Create a Docker Network

To simplify container to container communication, Docker’s networking feature is used:

docker network create migratus

This creates a named network that containers can connect to. All containers that use the same network can connect directly to each other. Since the birding website is a website that uses a database backend, having these as two separate containers make sense. With the network in place, the development system can be brought up. The script brings up the database connected to the migratus network:

#!/bin/bash
 
mysql_opts="-e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=migratus -e MYSQL_USER=migratus -e MYSQL_PASSWORD=migratus"
docker run -P --network migratus --name migratus-db ${mysql_opts} -d mysql:8 --default-authentication-plugin=mysql_native_password

Initializing the application dependencies

This new Perl container is then started and Carton is used to install the dependencies. Docker’s virtual mount point is used to keep the source code on OS X so Atom can be used.

docker run -it --rm --name migratus --network migratus -p 3000:3000 -v "$PWD":/usr/src/app -w /usr/src/app macfisherman/migratus bash

This command causes a Bash shell to be running within the container. The following command is run in the container:

carton install

This creates a local directory. Because /usr/src/app is also mapped to the development directory in OS X it means the modules loaded into local will survive a container removal. The local directory should be exclude from any source code management system, but the files cpanfile and cpanfile.snapshot can and should be version controlled.

Running the application

With all the applications dependencies in place, it can be run. But first some Docker related checks. The database container should be pingable within the Perl Docker container. Since in this development environment it was named migratus-db, The following command should return positive results:

ping migratus-db

Although Carton provides an exec command that runs a Perl application with the correct module paths, the environment variable PERL5LIB can also be used. Since the application requires additional module paths, this is the route I chose. To run the app, the following is used:

PERL5LIB=$PWD/local/lib/perl5:$PWD/lib local/bin/morbo migratus

Summary

I can now do development on my birding site, keeping the application and its dependencies isolated from the OS X environment. The development website is reachable at http://localhost:3000. Code is locally accessible from OS X, and Perl related modules all all contained in a central spot. This post contained a lot of information, feel free to ask for any clarifications.

Testing legacy Perl code

At my day job, I maintain a legacy Perl code base. When I started, there were very few tests. Over the years I’ve been adding mostly integration tests, since the code base is essentially a collection of scripts, with lots of logic. There are very few modules.

I’m now tasked to improve a feature of one of the programs in this code base. The functionality that needs improvement is spread across two functions. These functions are several hundred lines. The code uses regular expressions to find certain features present in report. At times it fails, because the format of the report has changed through the years.

To test these two functions, which only exist in the script itself, I’m using two techniques. The first is to turn the script into a module that can also run as a script. The second is to override a CPAN module by creating a clone of it.

The technique of turning a script into a module is nicely covered here https://perlmaven.com/modulino-both-script-and-module. I only needed to change/add about a dozen lines. In short, any code that isn’t part of subroutine is wrapped into one. The subroutine can have any name, but in this case, I’ve called it main. Then the body of the script has this line:

main() if not caller();

The script itself had lots of globals, so those were left outside of the main function and declared with our. One final detail is to add a package line to the script. This keeps the script/module in its own namespace, providing isolation.

Before:

#!/usr/bin/perl
use DBI;
use Log::Log4perl;

use strict;
use warnings;

my $DBH;
my $LOGGER;
# other globals

Log::Log4perl::init(...);

$LOGGER=Log::Log4perl->get_logger(...);
$DBH=DBI->connect("DBI::mysql:...", ...);

process();
# more code

sub process {
...
}

After:

#!/usr/bin/perl
package process_reports;
use DBI;
use Log::Log4perl;

use strict;
use warnings;

our $DBH;
our $LOGGER;
# other globals

main() if not caller();

sub main {
  Log::Log4perl::init(...);

  $LOGGER=Log::Log4perl->get_logger(...);
  $DBH=DBI->connect("DBI::mysql:...", ...);
  process();
  # more code
}

sub process {
  ...
}

With these changes in place, it is now possible to load the script as a module in a test. Each of the subroutines in the script will be callable, allowing much better testing. My next post will cover the second technique of overriding system modules.