Posted on

Forking GoLang modules

Go uses path names in its import statements. When you fork a module and try to build it, the build will fail. For example, I want to add some functionality to go-echarts. I fork it at github, and then on my Macbook Pro, I clone the fork:

$ git clone git@github.com:macfisherman/go-echarts.git
Cloning into 'go-echarts'...
Warning: Permanently added the RSA host key for IP address '140.82.113.3' to the list of known hosts.
remote: Enumerating objects: 2181, done.
remote: Total 2181 (delta 0), reused 0 (delta 0), pack-reused 2181
Receiving objects: 100% (2181/2181), 1.75 MiB | 6.39 MiB/s, done.
Resolving deltas: 100% (1416/1416), done.

I then want to see if it builds.

jeffs-mbp:projects jeff$ cd go-echarts
jeffs-mbp:go-echarts jeff$ cd charts/
jeffs-mbp:charts jeff$ go build .
geo.go:9:2: cannot find package "github.com/go-echarts/go-echarts/datasets" in any of:
	/usr/local/go/src/github.com/go-echarts/go-echarts/datasets (from $GOROOT)
	/Users/jeff/go/src/github.com/go-echarts/go-echarts/datasets (from $GOPATH)
base.go:4:2: cannot find package "github.com/go-echarts/go-echarts/datatypes" in any of:
	/usr/local/go/src/github.com/go-echarts/go-echarts/datatypes (from $GOROOT)
	/Users/jeff/go/src/github.com/go-echarts/go-echarts/datatypes (from $GOPATH)
engine.go:13:2: cannot find package "github.com/go-echarts/go-echarts/templates" in any of:
	/usr/local/go/src/github.com/go-echarts/go-echarts/templates (from $GOROOT)
	/Users/jeff/go/src/github.com/go-echarts/go-echarts/templates (from $GOPATH)

In this case there is no go.mod.

jeffs-mbp:charts jeff$ cd ..
jeffs-mbp:go-echarts jeff$ ls go.mod
ls: go.mod: No such file or directory

By adding one, building will work:

jeffs-mbp:go-echarts jeff$ go mod init github.com/go-echarts/go-echarts
go: creating new go.mod: module github.com/go-echarts/go-echarts
jeffs-mbp:go-echarts jeff$ cd charts/
jeffs-mbp:charts jeff$ go build .
go: finding module for package github.com/gobuffalo/packr
go: found github.com/gobuffalo/packr in github.com/gobuffalo/packr v1.30.1

I can now use this local fork in a project that needs the modification the fork has by adding a line to go.mod (the project already has a go.mod) 

jeffs-mbp:charts$ cd ~/projects/covid-19
jeffs-mbp:covid-19 jeff$ go mod edit -replace="github.com/go-echarts/go-echarts => ../go-echarts"
jeffs-mbp:covid-19 jeff$ go list -m all | grep go-echarts
github.com/go-echarts/go-echarts v0.0.0-20190915064101-cbb3b43ade5d => ../go-echarts

The key is to use a recent Go. I’m using 1.14.2, but I believe any version since 1.11 will work too.

Posted on

Use UTC/epoch for Date/Time

Always use UTC for storing date/time information. Have the browser convert the epoch timestamp into the local time like so:

function localDate(timestamp) {
    let d = new Date();
    d.setTime(timestamp);
    return d.toLocaleDateString();
}

Docker MySQL instances are configured to UTC. MySQL uses seconds for epoch while Javascript uses milliseconds, so divide by 1000 when going from Javascript to MySQL and multiply when going the other direction. When storing timestamps, use the getTime method on the Date object using Javascript, and in MySQL, use the FROM_UNIXTIME function (after dividing by 1000). When retrieving timestamps, use the unix_timestamp function (and multiply by 1000). I’m surprised that Javascript and SQL don’t take epoch values by default.

Posted on

My development environment

This is a long post about the development environment I’m currently using.

The Basics

There are lots of technology available for developers that can make development much easier and many are free. I’ve been working on a birding website, using the Perl web framework Mojolicious. My development system is a Macbook Pro. It comes with an old version of Perl, so I used PerlBrew to manage different Perl versions. The editor I use is Atom.

Database Support

Most websites are connected to a database, and my birding website is no different. I’m currently using MySQL. MySQL has support for spatial types, which I hope to take advantage of in the website. The standard way for using a database in Perl is to use DBI and the corresponding DBD driver for the database that is being connected to. Many of these DBD drivers are linked to a C library implementation. However, a few are written that do not link to a C library. A native pure Perl DBD driver is nice because it doesn’t require the database software to be installed. I first experienced a native driver when writing some Go code that connected to MySQL.

Mojolicious has some modules that simplifies querying databases. Being a long time DBI user, I didn’t really see the need for such modules. However, the Mojolicious module Mojo::mysql has an interesting method named migrations. It is way to automatically update/downgrade a database when schema changes are made. It seems like most of the Mojolicious database drivers try to support migrations. Apparently Postgres has the best support for migrations since the DML statements can be rolled back. A great resource on the theory behind migrations can be found here.

The Mojo::mysql driver doesn’t have direct support for DBD::mysqlPP (the pure Perl driver for MySQL). But it is simple enough to get around that limitation by setting the dsn parameter of the new method to dbi:mysqlPP:<remaining parts>. However, I quickly learned that the DBD::mysqlPP driver was missing some functionality and it had a bug. The bug is exhibited itself when DBI’s bind_param method is used. The missing functionality is transaction support. I’ve have a fork of DBD::mysql for those interested in a fix. With those fixes, my Mojolicious app seems to functional, but I have since decided not to pursue the native driver approach.

Docker

For development, the MySQL database is run within a Docker container. The default Docker container for MySQL is very capable and can be used as is. Here’s was my initial startup script:

mysql_opts="-e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=migratus -e MYSQL_USER=migratus -e MYSQL_PASSWORD=migratus"
 docker run -p 3306:3306 --name migratus-db ${mysql_opts} -d mysql:8 --default-authentication-plugin=mysql_native_password

The DBD::mysqlPP driver does not support the newer authentication method, so the --default-authentication-plugin=mysql_native_password is needed.

Carton

With PerlBrew on OS X and MySQL within a Docker container, development works fairly well, but a better way of dealing with Perl application development is to use Carton. With Carton, a file named cpanfile is created that lists the application’s depencies, but only at the highest level. Issuing the command carton install pulls over the listed modules and their dependencies. This also creates a file cpanfile.snapshot which contains any additional modules the dependencies needed. With these two files, another developer can recreate the same environment. Carton installs all the modules in directory named local alongside the application. I’ve long preferred this method of building applications with Perl versus installing modules into the system Perl. My experience with Carton is new, so expect more posts regarding this tool in the future.

Carton and Docker finds a bug

Since the birding website is going to run on a Linux based system, it makes sense that Mojolicious framework is developed on Linux as well. Perl is suppose to be platform neutral, but I ran into an interesting problem when moving the code into a Docker container. I had the following code:

use Base Mojo::mysql;

This worked fine on OS X. However, it broke when moving to Linux. I got the following error:

Can't locate Base.pm in @INC

Base is actually base, but OS X uses a case-insensitive filesystem, so Perl was able to find the module.

Containerizing Perl

To move the Mojolicious app to Docker, I use the following Dockerfile that initializes a base container with a recent Perl and some additional packages:

FROM perl:5.30

RUN cpanm Carton \
    && mkdir -p /usr/src/app
RUN apt-get update
RUN apt install -y default-libmysqlclient-dev default-mysql-client default-mysql-client-core

EXPOSE 3000

WORKDIR /usr/src/app

With this Dockerfile, I created an image:

docker build -t macfisherman/migratus .

Create a Docker Network

To simplify container to container communication, Docker’s networking feature is used:

docker network create migratus

This creates a named network that containers can connect to. All containers that use the same network can connect directly to each other. Since the birding website is a website that uses a database backend, having these as two separate containers make sense. With the network in place, the development system can be brought up. The script brings up the database connected to the migratus network:

#!/bin/bash
 
mysql_opts="-e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=migratus -e MYSQL_USER=migratus -e MYSQL_PASSWORD=migratus"
docker run -P --network migratus --name migratus-db ${mysql_opts} -d mysql:8 --default-authentication-plugin=mysql_native_password

Initializing the application dependencies

This new Perl container is then started and Carton is used to install the dependencies. Docker’s virtual mount point is used to keep the source code on OS X so Atom can be used.

docker run -it --rm --name migratus --network migratus -p 3000:3000 -v "$PWD":/usr/src/app -w /usr/src/app macfisherman/migratus bash

This command causes a Bash shell to be running within the container. The following command is run in the container:

carton install

This creates a local directory. Because /usr/src/app is also mapped to the development directory in OS X it means the modules loaded into local will survive a container removal. The local directory should be exclude from any source code management system, but the files cpanfile and cpanfile.snapshot can and should be version controlled.

Running the application

With all the applications dependencies in place, it can be run. But first some Docker related checks. The database container should be pingable within the Perl Docker container. Since in this development environment it was named migratus-db, The following command should return positive results:

ping migratus-db

Although Carton provides an exec command that runs a Perl application with the correct module paths, the environment variable PERL5LIB can also be used. Since the application requires additional module paths, this is the route I chose. To run the app, the following is used:

PERL5LIB=$PWD/local/lib/perl5:$PWD/lib local/bin/morbo migratus

Summary

I can now do development on my birding site, keeping the application and its dependencies isolated from the OS X environment. The development website is reachable at http://localhost:3000. Code is locally accessible from OS X, and Perl related modules all all contained in a central spot. This post contained a lot of information, feel free to ask for any clarifications.

Posted on

Testing legacy Perl code – part 2

When Perl loads a module, it looks for the module in the subdirectories listed in the special Perl variable @INC. The first module it finds, it uses. Using this knowledge, a programmer can craft his own implementation of a module to substitute for the real module. In a test, the @INC can be manipulated by setting the environment variable PERL5LIB or by using the -I parameter when using prove.

jeffs-mbp:aws-npk jeff$ perl -V
Summary of my perl5 (revision 5 version 30 subversion 0) configuration:
...
  @INC:
    /Users/jeff/perl5/perlbrew/perls/perl-5.30.0-mojo/lib/site_perl/5.30.0/darwin-2level
    /Users/jeff/perl5/perlbrew/perls/perl-5.30.0-mojo/lib/site_perl/5.30.0
    /Users/jeff/perl5/perlbrew/perls/perl-5.30.0-mojo/lib/5.30.0/darwin-2level
    /Users/jeff/perl5/perlbrew/perls/perl-5.30.0-mojo/lib/5.30.0

jeffs-mbp:aws-npk jeff$ PERL5LIB=$PWD/lib perl -V
...
  @INC:
    /Users/jeff/projects/aws-npk/lib
    /Users/jeff/perl5/perlbrew/perls/perl-5.30.0-mojo/lib/site_perl/5.30.0/darwin-2level
    /Users/jeff/perl5/perlbrew/perls/perl-5.30.0-mojo/lib/site_perl/5.30.0
    /Users/jeff/perl5/perlbrew/perls/perl-5.30.0-mojo/lib/5.30.0/darwin-2level
    /Users/jeff/perl5/perlbrew/perls/perl-5.30.0-mojo/lib/5.30.0

By setting PERL5LIB to $PWD/lib, any modules found in the directory /Users/jeff/projects/aws-npk/lib will be used before any other modules.

Overriding a module that has lots of methods would be tedious and probably error prone. Perl has a solution for that, called AUTOLOAD. If a module has this method, then Perl will call it when a method does not exist in the module. Here is my implementation of DBD::mysql using this feature:

package DBD::mysql;

use Data::Dumper;
use strict;

sub new {
    my $class = shift;
    my %args = @_;

    return bless { LOG     => undef,
                   VERSION => "test DBD::mysql 1.0", 
                   %args}, $class;
}

sub display {
    my $self = shift;
    my $function = shift;
    my $content = shift;

    push(@{$self->{LOG}}, "$function: $content");

    if($self->{debug} == 1) {
        print("# $function:\n", "# $content\n");
    }
}

sub prepare {
    my $self = shift;
    my @args = @_;

    $self->display(Dumper(\@args));
    return $self;
}

sub fetchrow_arrayref {
    my $self = shift;
    my @args = @_;

    $self->display(Dumper(\@args));
    return [];
}

sub AUTOLOAD {
    my $self = shift;
    my @args = @_;

    our $AUTOLOAD;

    $self->display($AUTOLOAD, "@args");
}

1;

This module only defines two methods that are part of DBI::DBD, prepare and fetchrow_arrayref. Some code may require more to be defined, but in this case, the program I was testing only needed these two. prepare needed to return a reference to the object itself, because prepare normally returns a statement handle object, but Perl does not care as long as the object has the needed methods that are called upon it. fetchrow_arrayref returns an empty reference. All other methods are handled by the AUTOLOAD function. Most of the methods call the display function, which prints out the method and its arguments if the object was initialized with debug = 1.

Posted on

Testing legacy Perl code

At my day job, I maintain a legacy Perl code base. When I started, there were very few tests. Over the years I’ve been adding mostly integration tests, since the code base is essentially a collection of scripts, with lots of logic. There are very few modules.

I’m now tasked to improve a feature of one of the programs in this code base. The functionality that needs improvement is spread across two functions. These functions are several hundred lines. The code uses regular expressions to find certain features present in report. At times it fails, because the format of the report has changed through the years.

To test these two functions, which only exist in the script itself, I’m using two techniques. The first is to turn the script into a module that can also run as a script. The second is to override a CPAN module by creating a clone of it.

The technique of turning a script into a module is nicely covered here https://perlmaven.com/modulino-both-script-and-module. I only needed to change/add about a dozen lines. In short, any code that isn’t part of subroutine is wrapped into one. The subroutine can have any name, but in this case, I’ve called it main. Then the body of the script has this line:

main() if not caller();

The script itself had lots of globals, so those were left outside of the main function and declared with our. One final detail is to add a package line to the script. This keeps the script/module in its own namespace, providing isolation.

Before:

#!/usr/bin/perl
use DBI;
use Log::Log4perl;

use strict;
use warnings;

my $DBH;
my $LOGGER;
# other globals

Log::Log4perl::init(...);

$LOGGER=Log::Log4perl->get_logger(...);
$DBH=DBI->connect("DBI::mysql:...", ...);

process();
# more code

sub process {
...
}

After:

#!/usr/bin/perl
package process_reports;
use DBI;
use Log::Log4perl;

use strict;
use warnings;

our $DBH;
our $LOGGER;
# other globals

main() if not caller();

sub main {
  Log::Log4perl::init(...);

  $LOGGER=Log::Log4perl->get_logger(...);
  $DBH=DBI->connect("DBI::mysql:...", ...);
  process();
  # more code
}

sub process {
  ...
}

With these changes in place, it is now possible to load the script as a module in a test. Each of the subroutines in the script will be callable, allowing much better testing. My next post will cover the second technique of overriding system modules.