Friday, October 16, 2015

SQL Relay 0.64 Release Announcement

Version 0.64 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.

This release features support for Informix databases, support for remote and centralized configuration files, a better configure script for building from source on Windows, improvements to the drop-in replacement library for PostgreSQL, and a "softttl" parameter for dynamically spawned database connections.


Informix

SQL Relay now supports Informix databases. It can compile against fairly modern versions (probably 10.X and up) and access versions older than that. Informix may not be the most modern database, but it has a large user base and free developer versions have been available from IBM for a while now. It's high time SQL Relay supported it.

To configure SQL Relay to talk to an Informix database, use an sqlrelay.conf file like this:

<?xml version="1.0"?>
<!DOCTYPE instances SYSTEM "sqlrelay.dtd">
<instances>
 <instance id="informixtest" dbase="informix">
  <users>
   <user user="test" password="test"/>
  </users>
  <connections>
   <connection
        string="informixdir=/opt/informix;servername=ol_informix1210;
                db=testdb;user=testdbuser;password=testdbpass"/>
  </connections>
 </instance>
</instances>

(Actually, the connection tag should be all-on-one-line, I just had to split it up to it would fit on the page)

Then you can access it with sqlrsh like:

[dmuse@windows7 ~]$ sqlrsh -id informixtest
SQLRShell - Version 0.64
        Connected to: localhost:9000 as test

        type help; for help.

0> use sysmaster;
0> show tables;
tabname
=================
flags_text
logmessage
smi_build_status
sysactptnhdr
sysadtinfo
sysaqt
...
systxptab
sysuserthreads
sysvplst
sysvpprof
sysxatab
        Rows Returned   : 196
        Fields Returned : 196
        System time     : 8651

0>

Informix!

Remote and Centralized Configuration Files

The -config option that all of SQL Relay's command line programs take can now be a local file, directory, or URL.

For example:

sqlr-start -id oracleexample -config \
/home/myuser/mysqlrelay.conf

sqlr-start -id oracleexample -config \
file:///home/myuser/mysqlrelay.conf

sqlr-start -id oracleexample -config \
dir:///home/myuser/mysqlrelay.conf.d

sqlr-start -id oracleexample -config \
http://configserver.mydomain.com/sqlrconfig/sqlrelay.conf

sqlr-start -id oracleexample -config \
http://myuser:mypassword@configserver.mydomain.com/sqlrconfig/sqlrelay.conf

sqlr-start -id oracleexample -config \
http://[/usr/local/firstworks/etc/sqlruserpwd.txt]@configserver.mydomain.com/sqlrconfig/sqlrelay.conf

(of course, the commands can be all-on-one-line, I just split them up here for readability)

In the case of a dir:// URL, all files under the given directory will be parsed as config files.

In the last http:// URL, the file between the square brackets is a user-password file, which can contain a single line like:

myuser:mypassword

Using a file rather than embedding the user/password in the URL helps keep the user/password out of process listings and init scripts.

If Rudiments (the library upon which SQL Relay is built) was built with support for libcurl, the https, ftp, scp, and sftp urls can be used as well.

The -config option can now take a comma-separated list of URL's too. For example:

sqlr-start -id oracleexample -config \
/home/myuser/mysqlrelay-1.conf,\
/home/myuser/mysqlrelay-2.conf

sqlr-start -id oracleexample -config \
/home/myuser/mysqlrelay-1.conf,\
/home/myuser/mysqlrelay-2.conf,\
dir:///home/myuser/mysqrelay.conf.d

sqlr-start -id oracleexample \
-config /home/myuser/mysqlrelay-1.conf,\
/home/myuser/mysqlrelay-2.conf,\
dir:///home/myuser/mysqrelay-1.conf.d,\
dir:///home/myuser/mysqrelay-2.conf.d

sqlr-start -id oracleexample \
-config http://configserver1.mydomain.com/sqlrconfig/sqlrelay.conf,\
http://configserver2.mydomain.com/sqlrconfig/sqlrelay.conf,\
http://configserver3.mydomain.com/sqlrconfig/sqlrelay.conf

(again, I just split the commands over multiple lines for readability)

Any combination of local files, directories and remote URL's can be specified. Files will be processed in the order that they are specified.

SQL Relay now supports "link files" too. Instead of containing an XML configuration, a "link file" just contains the URL's of other configuration files. For example:

# oracle configuration
http://myuser:mypassword@configserver.mydomain.com/sqlrconfig/oracle.conf

# sap/sybase configuration
http://myuser:mypassword@configserver.mydomain.com/sqlrconfig/sap.conf

# db2 configuration
http://myuser:mypassword@configserver.mydomain.com/sqlrconfig/db2.conf

Any url that would be valid on the command line is valid in a link file. Link files may simply refer to other link files too.

Link files can be used to centralize configuration. For example, if you have several SQL Relay servers, rather than distributing configuration files across the servers, you could create an identical sqlrelay.conf file on each of them like:

http://myuser:mypassword@configserver.mydomain.com/sqlrconfig/sqlrelay.conf 

And then, on configserver.mydomain.com, host an sqlrelay.conf file like:

http://myuser:mypassword@configserver.mydomain.com/sqlrconfig/oracle.conf
http://myuser:mypassword@configserver.mydomain.com/sqlrconfig/sap.conf
http://myuser:mypassword@configserver.mydomain.com/sqlrconfig/db2.conf

The files oracle.conf, sap.conf, and db2.conf could then be hosted by and maintained on that server as well.

See the SQL Relay Configuration Guide for more information. In particular, the sections on Remote Configuration Files and Link Files.

A Better Configure Script for Windows

In the past, the Windows configure.vbs script didn't do much configuring. It basically detected what compiler and version of Windows you were using, and set all the right options for those, but didn't do much to detect different versions of databases or languages.

It does a lot more now. It's not as full featured as its linux/unix counterpart yet, but it's a lot better than it was in the past. It can now detect various versions of databases and languages, provided they're installed in reasonably standard locations. It also has options allowing you to specify locations and versions, in case you did install something in an odd location, and it provides feedback about what it could and couldn't find, like its linux/unix counterpart.

For example:

C:\Users\dmuse\src\sqlrelay>cscript /nologo configure.vbs

***** Platform ***************
Visual C++ Version: 16
Visual C++ Architecture: x64
Windows Version: 0x0601
******************************

***** Oracle *****************
Oracle includes... /I "C:\Program Files\Oracle\instantclient_12_1\sdk\include"
Oracle libs... /LIBPATH:"C:\Program Files\Oracle\instantclient_12_1\sdk\lib\msvc
" oci.lib
******************************

***** MySQL ******************
MySQL includes... /I "C:\Program Files\MySQL\MySQL Connector.C 6.1\include"
MySQL libs... /LIBPATH:"C:\Program Files\MySQL\MySQL Connector.C 6.1\lib" libmys
ql.lib
******************************

***** PostgreSQL *************
PostgreSQL includes... /I "C:\Program Files\PostgreSQL\9.4\include"
PostgreSQL libs... /LIBPATH:"C:\Program Files\PostgreSQL\9.4\lib" libpq.lib
******************************

***** SAP/Sybase *************
SAP/SYBASE includes... /I "C:\SAP\OCS-16_0\include"
SAP/SYBASE libs... /LIBPATH:"C:\SAP\OCS-16_0\lib" libsybct64.lib libsybblk64.lib
 libsybcs64.lib
******************************

***** ODBC *******************
ODBC includes...
ODBC libs... user32.lib gdi32.lib odbc32.lib odbccp32.lib
******************************

***** DB2 ********************
DB2 includes... /I "C:\Program Files\IBM\SQLLIB\include"
DB2 libs... /LIBPATH:"C:\Program Files\IBM\SQLLIB\lib" db2api.lib
******************************

***** Firebird ***************
Firebird includes... /I "C:\Program Files\Firebird\Firebird_2_5\include"
Firebird libs... /LIBPATH:"C:\Program Files\Firebird\Firebird_2_5\lib" fbclient_
ms.lib
******************************

***** Informix ***************
Informix includes... /I "C:\Program Files\IBM Informix Software Bundle\incl\cli"

Informix libs... /LIBPATH:"C:\Program Files\IBM Informix Software Bundle\lib" ic
lit09b.lib
******************************

***** C# *********************
******************************

***** Perl *******************
Perl prefix... C:\Perl64
Perl version... 520
******************************

***** Python *****************
Python prefix... C:\Python27
Python version... 27
******************************

***** Ruby *******************
Ruby prefix... C:\Ruby
Ruby version... 2.2.0
******************************

***** PHP ********************
PHP prefix... C:\PHP
******************************

***** Java *******************
Java prefix... C:\Program Files\Java\jdk1.8.0_25
******************************

***** TCL ********************
TCL prefix... C:\Tcl
******************************

***** node.js ****************
node.js prefix... C:\Program Files\nodejs
******************************

***** Summary ***********************************************
 Version      : 0.64

 APIs         : C/C++       yes           Perl       yes
                Python      yes           Ruby       yes
                PHP         yes           Java       yes
                PHP PDO     yes           ODBC       yes
                TCL         yes           C#         yes
                node.js     yes

 Databases    : Oracle8     yes           MySQL      yes
                PostgreSQL  yes           SAP/Sybase yes
                ODBC        yes           DB2        yes
                Firebird    yes           Informix   yes
*************************************************************

If you expected a Database or API that doesn't show up in the Summary
then the configure script probably couldn't find a package it needed to
build it.  You can manually specify package locations using command line
options.

Type:  cscript /nologo configure.vbs --help   for a list of options.
Drop-In Replacement Library for PostgreSQL

The drop-in replacement library for PostgreSQL has been improved somewhat. PQprepare/PQexecPrepared and bind variables are now supported.

"softttl"

There's also a new, "softttl" parameter for dynamically spawned connections.

From the Configuration Reference:

softttl - The total number of seconds that a dynamically spawned connection intends to live. When the connection notices that it has been alive for this number of seconds, it voluntarily shuts down, but it only checks after each client session. Thus, the connection will ignore this parameter until it has handled at least one client session, and it could live longer than this time if a client session takes a long time, or if it sits idle for a long time between client sessions. Setting this parameter to 0 disables it. Defaults to 0 (disabled).

It's just another way to prevent dynamically spawned connections from living too long. It can be used as an alternative to maxsessioncount.

Of course, this release features a few bug fixes too.

Full ChangeLog follows:

  • fixed bug in sqlrsh that caused it to filter out non-ascii characters
  • added informix connection module
  • added shutdown loop detection in sqlr-listener
  • fixed dbhostname/dbipaddress requests
  • added bind variable format validation to php pdo driver
  • added softttl parameter
  • added db/language detection to configure.vbs
  • added passwordfunction parameter to "sqlrelay" authentication module
  • added -disable-crash-handler command line option to server processes
  • postgresql connection module PQclear()'s the result of PQprepare() now, fixing a memory leak
  • updated postgresql connection module to handle invalid bind variables properly, and without crash
  • added various missing postgresql functions to the postgresql drop-in library, most notably PQprepare, PQexecPrepared, PQexecParams
  • simplified sqlrpaths/sqlrcmdline classes a bit
  • renamed sqlrconfigfile to sqlrconfig
  • implemented a modular configuration framework
  • config files can be loaded over protocols supported by libcurl now or http if rudiments compiled withougt libcurl support
  • config files can now be "link files" which just refer to other local config files, local config directories, or remote config files
  • updated sqlr-start and sqlr-scaler not to pass the default -config value if sqlr-start wasn't explicitly started with -config
  • fixed a bug that could cause sqlrsh to act funny if the last character prior to the command delimiter was a single or double quote
  • fixed a stacked result-set translation module bug
  • applied George Carrette's patch to fix potential PDO null-dereferences

Rudiments 0.54 Release Announcement

Version 0.54 of Rudiments, the C++ class library for developing systems and applications, is now available.

There are a few bug fixes in this release, and one or two new features.

URL Class

The most significant feature in this release is the addition of a url class, and incorporation of it into the XML classes.

The url class inherits from file and implements read-only access to arbitrary URL's. HTTP URL's, with basic-authentication are supported on all platforms. On Linux/Unix, if you have a modern enough version of libcurl, then Rudiments will use it and some URL's supported by libcurl will also be supported.

See the url class documentation for more info.

The url class has been integrated into the xmlsax class. So, xmlsax::parseFile() can now be passed a url, rather than just a local file. Since the xmldom class uses xmlsax, then xmldom::parseFile() can also take a url.

Other New Features

The leftTrim(), rightTrim() and bothTrim() methods of the charstring class trim all whitespace now, not just spaces.

There's a charstring::isNullOrEmpty() method now which returns true if the string is either NULL or "" and false otherwise.

The filedescriptor::read() methods which read until a specified terminator is found now have variants with a maxbytes option, which sets the maximum number of bytes that can be read before giving up if no terminator is found.

Full ChangeLog follows:

  • tweaked code in configure script to remove -Werror* to be non-greedy
  • updated semaphoreset/sharedmemory destructors to remove the semaphoreset/sharedmemory segment last
  • added url class that uses libcurl (read-only for now)
  • added support for urls to xmlsax (and thus xmldom and codetree)
  • added http support to the url class for platforms that lack an appropriate libcurl
  • charstring 1-argument trim methods trim all whitespace now, not just spaces
  • added charstring::isNullOrEmpty()
  • added read-until-terminator methods with maxbytes option
  • applied George Carrette's listener class patch to not close invalid-epoll filedescriptors

Friday, September 25, 2015

Retrocomputing with Solaris 9 Sparc

Introduction

Qemu 2.4.0 came out on August 11th and I've been so busy that I didn't even notice. This week though, I've been sick, all week. ...and my mind just hasn't been up to its customary level of problem solving. Feeble attempts to get work done gave way to browsing the web, and eventually to sleep. During one of those periods of consciousness though, I noticed that Qemu 2.4.0 had been released. And the ChangeLog included something that piqued my interest.

In the SPARC notes:


Fix SunOS 4.1.4 boot on sun4m with OpenBIOS

SunOS 4.1.4 is also known as Solaris 1.something-or-other. Last I checked, OpenBIOS couldn't boot anything except Solaris 7, 8 and 9, in single-user mode, and there were even some limitations to that. Older versions of Solaris were completely unsupported.

I knew that Artyom Tarasenko and friends had been poking at OpenBIOS and QEMU over the years, and bit-by-bit it could boot more and more things, but the last I'd heard, a year ago, they'd just barely gotten it to boot Solaris 9 in full single-user mode.

What's this I hear about SunOS 4.1.4? Has a bunch of progress been made since the last time I checked?

I would have to find out.

But I didn't. Not at first. Instead I slept and slept and tried to kick this cold. But later in the week, I improved somewhat, and rather than spending my conscious moments browsing the web, I nudged along some projects, including an attempt to get some version of Solaris for Sparc running.

I started with Solaris 9. I'm not sure why, really. I own copies of 1.1.1B, 2.1 and versions 7 through 11. No idea why I settled on 9. My subconscious was making most of my decisions at the time. I think it's because I'd also read that 9 sort-of-boots further and 9 was still on my mind.

At any rate, I gave 9 a try.

VM Configuration

I sort-of followed some of the directions at Artyom's Solaris/sparc under qemu how-to but with a few modifications. I had a few false starts, but it didn't take long, or much effort (thankfully) to work my way through them.

So here's what I ultimately did...

I created a 36G disk.


qemu-img create solaris9_sparc.img 36G

Why 36G in particular? Wait and see...

This is the QEMU command that I ended up using too.


qemu-system-sparc -drive file=solaris9_sparc.img,format=raw \
-cdrom solaris-9-sparc-installation.iso \
-nographic -net bridge -net nic,macaddr=aa:00:00:00:00:a0 \
-prom-env "auto-boot?=false"

Modern QEMU complains if you use a raw disk but don't tell it that you're using a raw disk, thus the -drive argument rather than just using -hda.

The -nographic option basically runs it in the terminal window rather than simulating the framebuffer, which is great because it runs much faster and the only supported framebuffer resolution is big and unwieldy.

I'm also using bridged networking. Here's a little info on that.

The weird -prom-env argument keeps it from trying to boot. There are various issues related to using the QEMU -boot option. If you tell it to boot to the CD, it does ok, but it universally attempts to boot to the wrong disk partition. It seems better to boot manually from the OpenBIOS prompt.

Also, the Solaris 9 distribution comes with a bunch of CD's, including an Installation disc, 2 Software discs, a Supplements disc, and Companion CD. I configured QEMU to boot to the installation disc here.

Pre-Installation

Running that QEMU command dropped me to an OpenBIOS prompt, where I booted to the CD, in single-user mode.


boot cdrom:d -vs

(-v means verbose, -s means single-user)

If you do this, and you look carefully, you'll notice something like this scroll up the screen:


(sd0):
Corrupt label; wrong magic number

As neozeed has pointed out before, Sun-provided disks were apparently pre-formatted, but if you used a non-Sun disk, you had to format it yourself, or the installer won't recognize it. My QEMU disk image was certainly a non-Sun-provided disk.

The single-user boot dropped me at a root prompt, and I ran the "format" command.


format

Which disk? Disk 0. What disk type? Other.

And then it prompted me for a dozen odd parameters. The important ones were:

  • data cylinders: 24622
  • heads: 27
  • data sectors/trac: 107
  • disk type name: qemu

I accepted defaults for the others.

BTW, the reason I used a 36G disk, in particular, is because those parameters are known to work with a 36G disk. Sun disks apparently have odd geometries, and the old geometry calculator web page is long gone.

After that, I labelled the disk, quit and rebooted.


format> label
Ready to label disk, continue? y

WARNING: /iommu@0,10000000/sbus@0,10001000/espdma@5,8400000/esp@5,8800000/sd@0,0 (sd0):
Corrupt label; wrong magic number
format> quit
# reboot

Ok. Disk formatted. Time to get down to business for real.

Installation

At the OpenBIOS prompt, I booted to the CD normally:


boot cdrom:d -v

This time, no "Corrupt label" errors.

The installer ran, I chose to install in English. Then it wanted a location to copy the installation software to. I allowed it to repartition the disk, accepted the default 512MB swap slice size, and allowed it to start the slice at beginning of the disk. The installer then copied itself to the disk and attempted to reboot.

But, since I'd disabled auto-boot, I had to tell it where to boot:


boot disk:b -v

At this point, the installer would sometimes just run, or other times crash. If it crashed, I could restart it by running disk0_install, and then it would run fine. I never figured out exactly why it was crashing sometimes. Later I read that maybe giving QEMU more memory (using -m 256) might have fixed the problem. I never tried that though. Your mileage may vary.

When the installer came up, it was intuitive. I supplied various networking parameters, time zone parameters, and a root password.

Then, after the weird (but familiar) Web Start message...


Solaris Web Start will assist you in installing software for Solaris.

(Whatever Web Start means)

...the installer asked a few more intuitive questions and asked for Software Disc 1.

If I'd been installing on real hardware, the CD tray would have been ejected and it would have been clear that I needed to change CD's. Everything being virtual, I didn't realize that I needed to do anything, and forgot that there are like 5 CD's with software on them, so it took me a while to realize that I needed to switch into the QEMU monitor and change CD's.


(qemu) change scsi0-cd2 solaris-9-sparc-software-disc1.iso

But after I figured that out, and hit return, and went through a few more prompts, the installer continued with a crude (but familiar) progress bar.

Woohoo!

Then it wanted Software Disc 2.


(qemu) change scsi0-cd2 solaris-9-sparc-software-disc2.iso

And there was more of the same, before it finally tried to reboot again.

Once again, I had to specify the disk and partition to boot to:


boot disk:a

Though this time to a different partition, and without the verbose flag.

The installer wasn't done yet though, it wanted the Supplements CD.


(qemu) change scsi0-cd2 solaris-9-sparc-software-supplement.iso

And it installed a some stuff that I recognized like Java3D and OpenGL, but then also stuff like PC Launcher, PCfileviewer, RSC, SunForum, SunHSI PCI, and SunVTS. What are these things?

After a few more prompts, it tried to reboot again, and again I told it to boot to disk:a...


boot disk:a

But this time, everything started up for real.

1-solaris-9-sparc-firstboot

I could log in as root, and networking worked on the first try!

Post-Install Tweaks

So, I now had a working Solaris 9 Sparc installation. I hadn't seen such a thing in more than 10 years.

But the system as it was, wasn't all that useful yet. I wanted languages (well, other than java) and compilers and web servers. Companion CD to the rescue...

I swapped the CD again:


(qemu) change scsi0-cd2 solaris-9-sparc-software-companion.iso

And, after a bit of trial and error, figuring out which device was the CD, mounted it and ran the installer.


mount -F hsfs /dev/dsk/c0t2d0s0 /cdrom
cd /cdrom
./installer

But it failed miserably because it's requires a graphical environment to run. Fortunately, I had one of those, just not on that machine.

Locally, I gave the VM permission to use my display:


xhost +192.168.123.191

And then ran the installer, piping it's display here:


DISPLAY=192.168.123.7:0 ./installer

Success!

2-solaris-9-sparc-companion

It took a while to start up, and there were a bunch of intuitive prompts, and it took a really long time to install, but eventually, it did, and I had most of the utilities I'm so accustomed to having.

I did need to tweak a few things though.

First, sudo had the wrong permissions.


chmod u+s /opt/sfw/bin/sudo

Second, even though I had a 36G disk, the root division is only 2G, and it was getting full, so I moved /opt/sfw to /export/home (which had a ton of space), and symlinked it back to /opt.


cd /opt
mv sfw /export/home
ln -s /export/home/sfw sfw

I also wanted to add some paths to everyone's PATH, but vi didn't work very well. The TERM environment variable is set to "sun" by default, but even setting it to xterm, vt100 or vt220 didn't help. I'm not sure why exactly, but I was eventually able to get it to work by running a dtterm (which supports the "sun" terminal type) and editing /etc/profile from there:


DISPLAY=192.168.123.7:0 /usr/dt/bin/dtterm

And then in that terminal:


vi /etc/profile

I added the following, near the top:


PATH=$PATH:/usr/ccs/bin:/usr/ucb:/usr/openwin/bin:/usr/openwin/demo:/usr/dt/bin:/usr/sfw/bin:/opt/sfw/bin
LD_LIBRARY_PATH=/usr/sfw/lib:/opt/sfw/lib
...
export LOGNAME PATH LD_LIBRARY_PATH

I also created a user for myself:


useradd -m -d /export/home/dmuse dmuse
passwd dmuse

And I gave myself some sudo privileges, though the sudoers file was in /usr/sfw/etc/sudoers rather than /etc/sudoers.

ssh appeared to be running, but I enabled X11 forwarding in /etc/ssh/sshd_config


X11Forwarding yes

...and restarted it.


/etc/init.d/sshd stop
/etc/init.d/sshd start

I was then able to ssh into the VM, and run X apps without having to fiddle with the DISPLAY variable.

All right!

The Web

Solaris 9 comes with some version of Apache. I copied the example config file.


cd /etc/apache
cp httpd.conf-example httpd.conf

And tweaked a few things (in httpd.conf):


#ServerName 127.0.0.1
...
AddHandler cgi-script .cgi
...
Options Indexes FollowSymLinks MultiViews ExecCGI
...
DirectoryIndex index.html index.cgi

And restarted apache.


/etc/init.d/apache restart

And wrote a quick and dirty CGI (/var/apache/htdocs/env.cgi)


#!/bin/sh
echo "Content-type: text/plain"
echo
echo
env

And it worked!

3-solaris-9-sparc-web

Solaris 9 also comes with Netscape Communicator 4.78 which tries to go to www.sun.com and crashes unless you manage to stop it in time. It appeared to support png's but that's about the most modern thing it could do.

Development

Ok, so all that had been fun, but could I get any of my software to run?

Turns out yes. I'd gotten it to work on Solaris 9 x86 long ago, and wrung out all the endian, alignment, and 32/64-bit issues over the years. The companion CD provided all the dev tools that I needed, including gcc 2.95.3. Pretty old, but good enough.

I was able to build and install Rudiments and SQL Relay and access Oracle 12 via an SQL Relay server on another machine.

4-solaris-9-sparc-sqlrelay

The companion CD came with MySQL too, Relay found it, and built a connection plugin for it, but it was such an old version that it was incompatible with modern MariaDB:


mysql_real_connect failed: Client does not support authentication protocol requested by server; consider upgrading MariaDB client

Yeah, pretty old:


/opt/sfw/mysql/bin/mysql Ver 11.15 Distrib 3.23.43, for sun-solaris2.9 (sparc)

I have various versions of Oracle for Sparc Solaris. One of them probably works on Solaris 9.

Fun for another day.

Conclusions

Having used Solaris 9 on x86 quite a bit, it was very familiar on Sparc. Backspace is delete and /home is /export/home, but I'm used to that. The most difficult thing to deal with was the inability to use vi on the console, and that wasn't too big of an issue. Other than that, it does slam the host cpu, so I can't just leave it running, but I think it does that on real hardware too, so it's not unexpected.

I've been collecting versions of Sparc Solaris for a few years now, in the hopes that someday QEMU/OpenBIOS might support it. Yeah, I know you can use the Sun ROMs but the dubious legality of those isn't my style. It's really great to find that it all works now. I'll have to try out some of the other versions soon too.

Monday, August 31, 2015

SQL Relay 0.63 Release Announcement

Version 0.63 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.

This release mainly focused on platform-compatibility, fixing failures to detect or build against various dependencies on modern versions of Fedora, Debian, Solaris, FreeBSD, and NetBSD.

A few bugs have been fixed as well, and there are a few changes.

Changes

The "oracle8" database type has been updated to just plain "oracle". So, in your sqlrelay.conf file, you should use dbase="oracle" rather than dbase="oracle8". "oracle8" is still supported, but the docs all say to use "oracle" now. Oracle 8i, 9i, 10g, 11g, and 12c have all come out since the driver was first written, and OCI hasn't undergone any massive rewrites in all of that time, so the name change definitely seems overdue.

Similarly, the "sybase" database type has been updated to "sap". SAP bought out Sybase a few years ago and ASE 16.0 is marketed as SAP ASE, not Sybase ASE. "sybase" still works, but the docs all say to use "sap" now.

There is also now an "id" parameter that can be set in the database connect string. If it's set, then if a client calls the identify() method or the sqlrcon_identify() function, it will return whatever value "id" is set to.

This is useful, for example, if an app requires that SQL Relay identify the database as "sybase" or "oracle8", as the name changes cause them to identify themselves as "sap" and "oracle" now, by default. It's also useful when connecting to an ODBC database. By default, SQL Relay identifies an ODBC database as "odbc", but it might be useful to the app to know what actual type of database is being accessed, rather than just "odbc". Similarly, FreeTDS connections can be id'ed as "sap", "sybase", "mssql", etc. rather than "freetds".

For example, with this configuration:

<instance id="mssqlexample" dbase="freetds" ... >
    ...
    <connections>
        <connection string="user=...;password=...;id=mssql;..." ...>
    </connections>
</instance>

In this code:

    ...
    sqlrconnection sqlrc("sqlrserver",9000,NULL,"...","...",0,1);
    ...
    identity=sqlrc.identify();
    ...

"identity" would be "mssql".

Bug Fixes

The default "maxitembuffersize" for Firebird is 32768 now, bumped up from 4096. Firebird has supported 32768-byte varchars for quite some time now. This should have been the default for all that time.

A few of the ADO.NET Data Provider classes implement the IDisposable pattern, but were missing destructors. The missing destructors have been added.

The setAuthTimeout() and setResponseTimeout() methods weren't properly exposed in the Ruby API. They are now.

And a few other things too... Full ChangeLog follows:

  • suppressed ruby configure warning
  • fixed glib detection if pkg-config isn't present
  • fixed mono detection on freebsd
  • fixed node.js yes/no misreport bug in configure script
  • added freetds tcl test
  • fixed slashes in tcl tests for windows
  • updated docs to mention firebird buffer size parameters
  • updated MAX_ITEM_BUFFER_SIZE to 32768 for firebird
  • fixed ruby detection on fedora 22
  • fixed missing export for setAuth/ResponseTimeout in ruby api
  • added manual include of inttypes.h in php api to work around issue with define/undef games, revealed on openbsd 5.7
  • tweaked ruby cflags script for debian 8
  • refactored mysql detection
  • tweaked unixodbc detection to also look in /usr/include/odbc
  • renamed "oracle8" connection plugin to "oracle"
  • renamed "sybase" connection plugin to "sap"
  • updated identity tests in api's/cmdline clients to use charstring::contains() rather than !charstring::compare()
  • implemented missing destructors for ADO.NET Data Provider

Rudiments 0.53 Release Announcement

Version 0.53 of Rudiments, the C++ class library for developing systems and applications, is now available.

This release features a bit of clean-up, and a few new features.

XML Improvements

The XML classes xmlsax, xmldom, and xmldomnode now support tag namespaces. The namespace is now an element of the xmldomnode and the various methods for navigating, adding, updating, and deleting nodes, all support namespaces now.

The xmldomnode class also features new appendXml/insertXml and appendXmlFile/insertXmlFile methods which allow a string or file containing XML to be parsed and inserted or appended directly into an existing tree.

codetreegrammar Class Exposed

The codetreegrammar class is now exposed. This allows the codetree class to use the same already-parsed grammar for successive parse or write passes without having to re-parse it.

DTD Class Removed

The outdated and long-unused dtd class has been removed.

Serial-port Classes Disabled By Default

The serialport, serialportprofile, modemserver and modemclient classes are omitted from the default build now, but can be enabled using the configure script's command line options.

The full ChangeLog follows:

  • exposed codetreegrammar class and updated codetree class so that successive parses/writes can use the same grammar without having to reload it
  • added insert/appendXml(File) methods to xmldomnode to parse and insert/append xml strings or files to a node
  • fixed FILE->fileno detection for solaris 11.2, which doesn't have it at all
  • llabs() is detected and preferred to abs()
  • added partial xml namespace support (namespaces for tags)
  • added partial support for extended ascii character set to character class
  • added setgroups() call before setuid() call to drop extraneous groups
  • updated rudiments-config man page, slightly
  • removed outdated and unused dtd class
  • updated file::resolveSymbolicLink() to use directory::maxPathLength(filename) as a starting point
  • updated directory::getCurrentDirectory() to use MAX_PATH as a starting point
  • serialport/serialportprofile, modemserver/modemclient classes are disabled in the default build now
  • updated intro docs

Friday, August 7, 2015

SQL Relay 0.62 release announcement

Version 0.62 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy is now available

Yes, that's right 0.62, not 0.61. 0.61 has been skipped for administrative reasons.

This release includes some significant new features.



node.js Client API

The most notable new feature is a client API for node.js. It's now possible to access any database supported by SQL Relay, from a node.js application. That includes Oracle, Microsoft SQL Server, Sybase/SAP, IBM DB2, MySQL, PostgreSQL, Firebird, SQLite, as well as any database accessible via ODBC.

See the Programming Guide for step-by-step instructions and examples, and the API Reference for details on the classes and methods.

Filters, Translations, and other Modules

SQL Relay has long had password encryption and logger modules, but new frameworks have been added for authentication, query and result set translation and query filtering. Some default modules have been included as well.

For more details on these frameworks, the included modules, and how to use them, see:

Firebird Updates

SQL Relay pre-allocates buffers to fetch result sets into, before sending the result set along to the client. It is now possible to tune the size of these buffers, and in-effect, memory usage, when using the Firebird database. The parameters involved are included in the database connect string, and are: maxitembuffersize, maxselectlistsize, and maxbindcount.

These parameters have long been supported with Oracle, Sybase/SAP and IBM DB2. Now they are supported with Firebird too.

Non-Lazy Connect

By default, SQL Relay does lazy connects. I.e. no connection is established to the SQL Relay server when the connection object is created. The connect doesn't occur until the first query (or other database operation) is run.

This is inconsistent with the behavior of most PDO and Perl DBI drivers though. Some apps rely on "new PDO" or "DBI->connect" failing if the database is inaccessible. Neither of those classically fail with SQL Relay though. The database can't be determined to be inaccessible until the first query is run. This behavior can cause problems when trying to insert SQL Relay between an existing app and the database that it used to connect to directly.

To address the issue, a new connect string variable has been added to the PHP PDO and Perl DBI drivers: lazyconnect.

If set to 1 (the default), then the default behavior is observed. If lazyconnect=0 is used, then a connection will be established immediately and "new PDO" and "DBI->connect" will fail if the database is inaccessible.

See the PHO PDO driver and Perl DBI driver documentation for more info.

Old PostgreSQL

Some apps rely on being able to send multiple, semicolon-delimited queries to PostgreSQL, in a single statement. However, changes to the PostgreSQL API in PostgreSQL 8, to support native bind variables, disabled this ability. SQL Relay's configure script detects PostgreSQL 8 and uses the modern API if it's available. This can cause problems for apps that needed the old behavior, when placing SQL Relay between them and the database using the drop-in replacement library for PostgreSQL. To remedy this problem, a --disable-postgresql8-api option has been added to the configure script. Using it disables the new API and makes it possible to send multiple, semicolon-delimited queries to PostgreSQL, in a single statement.

License Update

The documentation and examples given in the documentation are now explicitly covered by licenses. The documentation itself is GPL'ed and the examples are covered by the FSF Unlimited License:

The FSF Unlimited License

This software is free software; you have unlimited permission
to copy, distribute and modify it.

The SQLRelay Server, as defined in the file COPYING at the root of the source distribution, now carries an explicit exception that unquestionably allows it to be used with various non-GPL-compatible libraries. This was the intent all along, but modern interpretation of the GPL called it into question. There should be no question now. See a much longer discussion about this for more info.

Bug Fixes

The rest of the changes are just bug fixes.

Full ChangeLog follows:

  • added query filter plugin framework
  • added regex, string and pattern filters
  • added normalization translation
  • added reformatdatetime result set translation
  • fixed error in sqlrconfigfile class that could cause sqlr-start to crash if one instance defines addresses but the next doesn't
  • added query status flag to sqlrservercursor
  • tweaked odbc driver's SQLGetTypeInfo()
  • field name is passed into runResultSetTranslations() now
  • added support for maxitembuffersize, maxselectlistsize, and maxbindcount for firebird
  • added native api for node.js
  • added recognition of bigint, ubigint and uniqueidentifier fields in freetds/sybase
  • added nullsasnulls command to sqlrsh
  • added workaround for freetds empty (but non-null) text fields not getting properly converted to null-terminated empty strings
  • fetchatonce, maxitembuffersize and maxselectlistsize are set to defaulting values if invalid values are given for them
  • added lazyconnect option to PHP PDO and Perl DBI drivers (defaults to 1)
  • direct Transact SQL which returns a result set is now supported with Sybase/SAP/MSSQLServer/FreeTDS
  • added docs for authentication, query translation, query filter and result set translation modules
  • added -fPIC to sqlrserver-config --cflags, if supported
  • the "default" auth module is now called "userlist"
  • added database auth module that is analagous to authtier="database"
  • replaced passwordencryption attribute with passwordencryptionid
  • added --disable-postgresql8-api configure option
  • tweaked -Werror detection in configure script
  • refactored sqlr-bench build to work on windows
  • documentation, examples and tests are now covered by a license
  • reflecting modern interpretation of the GPL, The license for The SQLRelay Server (as designated in COPYING) license now carries an exception, explicitly allowing it to be linked with various non-GPL compatible libraries
  • license is installed now

Rudiments 0.52 is out

Rudiments version 0.52 is now available.

The ChangeLog is fairly short though each item involved more work, on average, than usual.

Over the past few years, I've done some trans-compiler work. The codetree class is one result of all that. The compiler class is another. So are the new xmldomnode methods. The compiler class is really just a framework for plugins. The plugins, codetree, and xmldomnode do all the real work.

The rpcentry class has been removed. I only originally included it for the sake of completeness, but have had no occasion to use it in 15 years. The rest are bug fixes and cleanup.

Full (though short) ChangeLog follows:

  • tweaked codetree's recursive break logic a bit
  • added compiler class
  • added case-insensitive methods to xmldomnode
  • added methods to xmldomnode that operate on descendents (as opposed to just operating on direct children)
  • added methods to xmldomnode for deleting, wrapping, unwrapping
  • fixed dynamiclib::getError() bug on windows
  • tweaked -Werror detection in configure script
  • removed rpcentry class
  • added file::setPermissions() methods

Wednesday, August 5, 2015

SQL Relay License Update

Background

SQL Relay is made up of a bunch of different parts: the server, the command line clients, API's for various languages, tests, documentation, etc. The server and command line clients have always been licensed under the GPL. The API's have been licensed under various licenses, as appropriate to each API - LGPL, Artistic License, Creative Commons, etc.

In the late 90's, when I first decided to use the GPL for the server, the open source community seemed to interpret the GPL as allowing dynamic linking of non-GPL-compatible libraries/plugins/modules/drivers into GPL-licensed programs. The Linux kernel was the poster boy for this interpretation, as hardware vendors provided (and still provide) closed-source drivers, under various non-GPL-compatible licenses. In the database arena, GPL-licensed programs like sqsh and tora illustrated this same interpretation.

The rationale seemed to be that the GPL-licensed program included instructions for loading and interfacing with a non-GPL library, but not the library itself. Headers generally "didn't count" so presumably neither did whatever bits of the library the linker needed to examine.

The GPL-licensed program and non-GPL library/plugin/module/driver were considered separate. They were only combined to create a derivative work at runtime. Copies of the pages of memory that composed that work could not be lawfully distributed, but distribution as separate components was fine.

That seemed to be conventional wisdom at the time.

I wasn't totally ambivalent to the politics associated with the GPL, but really, I needed to choose a license, the GPL was popular, and seemed to fit.

Since then, debates have raged. GNU has made their position abundantly clear: programs that are capable of sharing the same address space with a library/module/plugin/driver are a derivative work of the two, retroactively. Community opinion seems to be divided. The Linux development community is said to have a "gentlemen's agreement" with hardware manufacturers, but nothing in writing. There have been a few court cases and they've gone both ways.

These developments put projects like SQL Relay in an interesting position. Obviously it is the intention of the authors to allow linking with various non-GPL-compatible libraries, but under modern interpretation, the license doesn't explicitly allow that.

As of the next release, 0.61, it will.

The License Exception

For cases like this, GNU recommends a "license exception". The SQLRelay Server, as defined in the file COPYING, at the root of the source distribution, will carry the following exception:

Linking The SQLRelay Server statically or dynamically with other modules is
making a combined work based on The SQLRelay Server. Thus, the terms and
conditions of the GNU General Public License cover the whole combination.

In addition, as a special exception, the copyright holders of The SQLRelay
Server give you permission to combine The SQLRelay Server with free software
programs or libraries that are released under the GNU LGPL and with code
included in the standard release of the following libraries (or modified
versions of such code, with unchanged license):

* OpenSSL and its dependencies.
* Perl-Compatible Regular Expressions (PCRE) and its dependencies.
* IBM DB2 database client libraries and their dependencies.
* Firebird database client libraries and their dependencies.
* FreeTDS client libraries and their dependencies.
* MDB Tools client libraries and their dependencies.
* MySQL database client libraries and their dependencies.
* ODBC client libraries and their dependencies.
* Oracle database client libraries and their dependencies.
* PostgreSQL database client libraries and their dependencies.
* SQLite client libraries and their dependencies.
* Sybase/SAP client libraries and their dependencies.
* Any other programs or libraries as designated, in writing, now or in the
  future, by David Muse, his authorized agents, authorized heirs, or authorized
  agents of his authorized heirs.

You may copy and distribute such a system following the terms of the GNU GPL
for The SQLRelay Server and the licenses of the other code concerned{, provided
that you include the source code of that other code when and as the GNU GPL
requires distribution of source code}.

Note that people who make modified versions of The SQLRelay Server are not
obligated to grant this special exception for their modified versions; it is
their choice whether to do so. The GNU General Public License gives permission
to release a modified version without this exception; this exception also makes
it possible to release a modified version which carries forward this exception. 

Some of the named libraries are either LGPL already or otherwise GPL-compatible, but they are listed in case that changes in the future. If I missed something, the "Any other programs or libraries..." clause makes it possible to add whatever I missed too.

The big surprise: OpenSSL. Apparently it's licensed in a way that makes it non-GPL-compatible. Who knew? Apparently the authors of wget did. Did anyone else?

Modules

In addition to making it clear that those specific libraries are OK, there is another motivation to the license update.

SQL Relay has several module frameworks. You can do all kinds of things with modules - logging, password encryption, alternative authentication mechanisms, query filtering, query translation, result set translation, and more... And additional frameworks are planned.

Without the license update, module development is sketchy. For example:

Can I write a connection module for Informix? Good question.

OpenLDAP has an odd license, is it legal to develop an OpenLDAP authentication module? Who knows?

Can a software development firm write a module for a client that interfaces with some system that the client already uses via a non-GPL-compatible library? Maybe.

Can a software vendor write a module to interface with some proprietary system they developed? Also hard to say.

What about me, personally? Can I, personally, write proprietary plugins? As the primary copyright holder, I'd think so. But can I really?

The "Any other programs or libraries..." clause in the license update provides a path to these kinds of things. New libraries can be approved. If in doubt, just ask david.muse@firstworks.com and get a definitive answer, in writing.

As a guideline, the answer will almost certainly be yes for "client libraries" and their dependencies. By client libraries, I mean, there's some system running in its own address space and there's a library who's primary function is to provide access to it. That library is a "client library."

For other libraries, it would depend on the details.

Authority

By what authority do I change the license?

Mostly my own, partly that of the open source community, and partly that which was granted by other contributors.

I am the primary copyright holder for The SQLRelay Server. I'm not exactly sure what percentage of it I either wrote or own, but I'd bet that it's over 99%. The API's were largely contributed, but they're not affected by this change.

Over the years, patches from various contributors were applied to The SQLRelay Server, but the vast majority were "trivial bugfixes." While important, they were short and didn't demonstrate novel insight. I.e. the patch affected fewer than ten lines, the bug was easy to find, the bug is obvious now that it's been pointed out, and the fix couldn't have been implemented much differently. If the patch was short and met 2 out of 3 of the other criteria, then I deemed it trivial. Trivial bugfixes are likely not subject to copyright. This analysis and action seem consistent with those of the open source community in license update scenarios.

Some non-trivial patches were also integrated. But, I was able to get approval from the current copyright holders to make the license change. For good measure, I also got approval from all the trivial bugfixers that I could get a hold of too.

I did some contract jobs to integrate non-trivial patches against older versions into the then-current version of SQL Relay, but the language of the contract entitled me to the results.

There were also a few non-trivial patches that were once integrated into The SQLRelay Server, but have since been completely removed.

If anyone that I haven't received approval from can demonstrate that they contributed non-trivial code to The SQLRelay Server, a derivative of which still exists in the current version, and disagrees with the license change, please contact me at david.muse@firstworks.com and we'll work something out.

Ramifications

So, what does the license change mean to you?

Most likely nothing at all. The license is likely now just more consistent with what you assumed already. The GPL, in general, still applies. It is just definitively legal to use SQL Relay as it was intended.

The only practical difference is that if you want to write a module that integrates with some other system via a non-GPL-compatible client library, then it is very likely possible to do so.

"But this license change undermines the purpose of the GPL!"

Perhaps. SQL Relay was not written to crusade for the GPL. I don't intend to undermine the GPL though. In fact, rather than argue over how to interpret the GPL, I'm doing this in deference to GNU, who recommend exactly this action.

The GPL is great. Free software wouldn't be where it is without the GPL. With this update, it's still a good fit for this project.

Friday, July 31, 2015

Sourceforge is back

About a week and a half ago, SourceForge suffered a storage outage that took down just about everything they host.

SQL Relay, Rudiments, and a few other firstworks.com projects are hosted at SourceForge, and during the outage, the project websites, downloads, and CVS access were all down for periods of time. The SourceForge folks worked pretty tirelessly on getting everything restored, and as of this evening, everything appears to be back to normal.

CVS access to SQL Relay and Rudiments source was the last thing to be restored. Apparently they're hosted on non-Allura-backed CVS, the oldest platform that SourceForge has. I guess that makes sense, given that both projects began being hosted by SourceForge in 1999.

This has been the only outage that these projects have been affected by in all that time, so for now, I'm sticking with SourceForge. However, the source code repositories will likely be moved from CVS to Git sometime in the next few months. I'd been meaning to do that anyway. I guess this is a good reminder to get on it.

Saturday, June 20, 2015

SQL Relay 0.60 is out

This release fixes various obscure (though potentially crippling) bugs and shortcomings, and issues with specific compilers.

The most notable fix is, on Windows, that you can now successfully specify an installation path when using the installer, and it will actually work. In previous releases, you could specify a path, and everything would be installed there, but it was largely ignored by the applications, which looked for files in and tried to write files to the default path.

Full ChangeLog follows...

  • fixed true->false transposition in sqlrservercontroller::interceptQuery that could lead to a reLogIn loop
  • disabled -Werror for gcc < 2.7
  • applied George Carrette's patch to fix PDO connectstring options
  • migrated directory/file paths info into sqlrpaths class
  • removed undocumented and not-so-relevent-these-days MAX_CONNECTIONS/overridemaxconnections failsafe in sqlr-start
  • fixed java header detection for javac located in /usr/bin
  • updated java api to support non-null-tolerant implementations of environment::NewStringUTF()
  • the perl api builds correctly on OSR5 again
  • the postgresql sslmode is omitted entirely from the connect string now, if it's disabled, to prevent problems with older versions of postgresql that don't support the parameter at all
  • mono 2.8 or greater is required now
  • added a datedelimiters attribute to the instance tag in sqlrelay.conf to limit what date delimiters are used when translating dates
  • re-added mssqlserver detection, which had been accidentally removed
  • fixed statically-linked build
  • fixed mysql lob field bug that could cause a crash

Rudiments 0.51 is out

The 0.51 release of rudiments is largely a clean-up release. Regressions, inconsistencies, and issues with specific compilers have been addressed. Work on support for OpenVMS has begun as well.

Full ChangeLog follows...

  • added missing inline qualifiers to dynamicarray/staticarray method implementations
  • added test for clock_settime, datetime class uses it if other set-time functions aren't available, returns false and sets errno=ENOSYS if no set-time function is available
  • directory::fpathConf now falls back to MAX_PATH code if fpathconf exists but there's dirfd() or anything like it
  • file::createFifo returns false and sets errno=ENOSYS on platforms that don't support fifo's now
  • file::generateKey returns -1 and sets errno=ENOSYS on VMS now
  • file::createHardLink returns -1 and sets errno=ENOSYS on platforms don't support hard links now
  • fixed missing breaks in stdio::flush
  • userentry::getPassword/getRealName return NULL on VMS now
  • unix sockets are faked on VMS now
  • threadmutex is built and installed now whether or not threads are supported, but if threads are not supported, its methods are degenerate and return success
  • filesystem::getCurrentProperties() returns false and sets errno=ENOSYS on VMS now
  • implemented degenerate groupentry for VMS
  • fixed file::getBlockCount() crash that could occur if the filesystem's block size was reported as 0
  • tweaked copy constructors and = operators in staticarray/dynamicarray classes to work with older compilers
  • disabled -Werror for gcc < 2.7
  • filedescriptor::printf uses fprintf or fdopen/vfprintf when possible, in leiu of falling back to charstring::printf
  • charstring::printf tries the null device before falling back to a scratch file
  • the rate at which charstring::printf expands its buffer is now exponential up to 1024 bytes
  • added sys::getDirectorySeparator()
  • fixed trailing-slash-followed-by-quote issue when building windows command lines
  • added --enable-enosys-notify configure option for runtime notifcations when methods set ENOSYS
  • cleared errno before each EINTR loop
  • fixed sys::sync() segfault
  • fixed ssl tests
  • removed intervaltimer class
  • updated sys::getPhysicalPageCount/getAvailablePhysicalPageCount for Windows
  • added sys::getAllocationGranularity, updated code to use it instead of getPageSize, in most cases
  • fixed copy-constructor/=-operator bugs in dynamicarray class
  • added config_vs2013.h for VS 2013
  • fixed several permissions-related issues for WinNT 4
  • allowShortReads/useBlockingMode are set by default for stdinput now
  • added wrapper for atexit()
  • changed process::getRealUser/GroupId to getUser/GroupId to match setUser/Groupid
  • added process::setEffectiveUser/Group methods
  • fixed and documented auto-resume behavior in snooze methods
  • fixed detection of double-dashed command line parameters without values
  • removed redundant charstring::rightPad()
  • renamed charstring::padString() to charstring::pad()
  • added safePrint methods that take unsigned char arguments
  • fixed bytestring::findFirst() with unsigned char needle, when needle is 0
  • fixed bugs in linkedlist::moveBefore/moveAfter
  • fixed bugs in singlylinkedlist::removeAll()
  • removed arg parameter from thread::setFunction(), added thread::setArgument(), and added thread::create(arg)
  • changed thread::create() to thread::run()
  • wrote lots of example code and updated the programming docs
  • removed unused shmfile class
  • fixed several LocalFree's that should have been delete[]'s

Friday, May 8, 2015

Retrocomputing with UnixWare 7.0.1

This one's been a long time coming.

Introduction

Months and months ago I acquired a scattered collection of old SCO software from a seller on eBay. It included a random hodgepodge of all kinds of stuff; a dozen manuals, dozens of CD's and floppies, packs and packs of licenses... It took a few hours just to sort out what went with what. Among the haul: a complete copy of OpenServer 5.0.0, two complete copies of OpenServer 5.0.2, and the license pack for UnixWare 7.0.1, but no UnixWare media. Seriously?! What happened to the media?

At the time I was a bit frustrated, but not terribly because I had plenty to play with. In fact, I still haven't gone through most of it. And, so I just sat on the licenses and waited.

Months later, someone was selling a copy of OpenServer 5.0.5 and the photos showed media for UnixWare 7.0.1 included in the pile. This wasn't too surprising though. SCO used to give away promotional copies of UW 7.0.0 and 7.0.1 with OSR 5.0.5. In fact, that's how I was first exposed to it, way back. They didn't give away licenses though. You could install everything, but you only had about 60 days to play with it before the demo timed out.

But, I had an actual license! And a few days later I had media to go with it!

Score.

Still, it was a few weeks later before I had enough time on my hands to start trying to get it to run. That turned into a bit of an adventure too.

Here's how it went.

UnixWare

Actually, first... What the heck is UnixWare?

I'm a little fuzzy on the history actually.

Novell and somebody else (AT&T?) got together at some point in the early 90's to put together a version of SVR4 Unix that could interoperate with Novell NetWare, for some reason. I'm not sure what the motivation was, actually. But they succeeded, released a few versions, and eventually got the attention of SCO. At the time, SCO had OpenServer, which was an SVR3 implementation. UnixWare had feature-parity with OpenServer, aside from the configuration interfaces and back-compatibility with older SCO platforms. I guess they figured it would be easier to buy an SVR4 platform and port their goodies to it than rework OpenServer.

Whatever the motivation, SCO bought UnixWare, or licensed it, or something, and made several 2.x releases. In the late 90's Sun released Solaris version 2.7 as "Solaris 7". SCO was coming out with another version of UnixWare around then, and rather than name it 3.0, they named it "UnixWare 7" as well.

SCO made a few more releases and tried to get everyone to migrate from OpenServer to UnixWare, but there weren't a lot of takers. SCO's users had legacy apps galore that still needed to run, like apps written in FoxPro for 3.2v4.2, and apps written in Microsoft Basic for Xenix, and instances of Oracle7. They could recompile their modern C/C++ apps, but FoxPro, MS Basic, and Oracle7 didn't even run on UnixWare. Those apps would have to be completely rewritten in different languages or migrated away from SCO platforms. No thanks.

SCO tried a few stunts. Unixware 7.1.2 was released as "OpenUnix 8", for example, but people stuck with OpenServer. Eventually, SCO released OpenServer 6, which piled everything that everybody liked from OpenServer (including back-compatibility) onto a UnixWare base. It looked and felt like OpenServer, ran all the legacy apps, but it was UnixWare underneath.

Technologically, they'd finally achieved what both they and their users wanted, but it might have been too late. When OSR 6 was released, SCO's courtroom shenanigans had turned the world against them. I wonder how many takers OSR6 actually has.

But I digress. My adventure begins about half of that story ago, with version 7.0.1, right about when SCO got serious about trying to get everyone to migrate off of OpenServer.

Here's how it went. (For real, this time)

VM Configuration

I'd had good success with various OpenServer's in VMware, but UnixWare is a totally different breed. Turns out it'll run too, but only if you're careful about how you do it.

In particular, UnixWare can't make heads or tails out of VMware's IDE emulation. SCSI's the way to go. And, in particular, BusLogic SCSI. Don't even fool around with LSI, it's not going to work. The hard drive has to be SCSI, and so does the CD ROM.

But, UnixWare 7.0.1 predates bootable CD's by a year or two, so you've got to come up with some boot floppies. None came with my CD, but it turns out there are floppy images on the CD. Not in an intuitive place, mind you, but they're on there. It took a little bit of digging, but I found them under INFO/IMAGES. The files are named BOOT.IMA.1 and BOOT.IMA.2. There are other disk images there too, but they're not necessary for VMware installation.

Installation basically involved adding a floppy, aiming it at BOOT.IMA.1 and powering on the VM.

Right away I was greeted with the familiar UnixWare splash screen.

unixware 7.0.1 - 1. splash

And the installation was kind-of straightforward.

I answered a few intuitive questions, swapped floppies when prompted, answered a few more intuitive questions, eventually answered some less-intuitive ones...

Do I want to enter the DCU? Maybe. What's a DCU? Ahh, "Driver Configuration Utility." I'm not sure. As it turns out, letting it auto-configure is the way to go.

The installer detects the emulated PCnet ethernet card but the driver has a lot of trouble with it. The "IBM PCI Ethernet" driver works well though.

Configuring TCP/IP is intuitive, except that you can select the ethernet frame format. The default of ETHERNET_II works well though.

Since this is UnixWare, it also supports IPX, and as it turns out, IPX can't not be installed. That's OK though, the defaults seem to work.

Unfortunately, NIS also can't not be installed. I gave my installation an NIS Type of "client" and NIS domain of "firstworks" and as I have no NIS servers anywhere, left the optional NIS servers blank.

Just about everything else was totally intuitive, and before too long, the installation proceeded unattended.

unixware 7.0.1 - 2. install

When the installation was complete, I had to disconnect the cdrom and floppy before reboot. Failing to do so resulted in several very insistent messages.

On reboot, I got a funny message about swap being smaller than memory. I'd given my VM a monstrous 256MB of RAM. In the late 90's, 32mb would have been a lot.

I also got some apparently-unimportant SCSI errors and a bunch of messages about configuring NetWare and other things. Eventually I had to configure the mouse and then it asked me to insert disc #2 to continue installing the system. I only had 1 system disc though. The other disc was the development system. I tried that disc, but apparently that's not what it was looking for. It also didn't like it when I gave it the first disc again. Oh, well. Eventually I just hit F8 to defer that step. It then prompted me for disc #3 and I just deferred it again. I might have chuckled a little too.

Fortunately it didn't ask me for disc #4, but instead started doing boot-type things and eventually presented me with a CDE login.

unixware 7.0.1 - 3. first boot

All right!

Post-Install Tweaks

Logging in got me to a familiar CDE desktop.

unixware 7.0.1 - 4. cde desktop

But, not a lot worked. I could poke around the system, but it was an island unto itself. I couldn't ping, ftp or telnet to anywhere.

This took days to figure out.

It turned out that there were two big problems. First, though above I say "the PCnet driver sucks, use the IBM PCI Ethernet driver", I hadn't done that at this point. I was still using the PCnet driver, which sucks. Tremendous packet loss.

After switching to the IBM PCI Ethernet driver, things were a lot better, but I couldn't yet tell that they were.

It eventually turned out that NIS was wreaking widespread havoc. I never figured out exactly what was going wrong, but my guess is that the /etc/hosts file is an NIS-able file, and since there were no NIS servers anywhere, all DNS lookups had to wait for an NIS timeout before proceeding past the /etc/hosts file. The effect was that if I tried to ftp or telnet to somewhere, it would hang for quite a while before actually trying the address. Telnetting or ftp'ing in from other hosts would hang too, during the reverse-lookup. If I waited, each these operations would eventually succeed, but in the beginning, I wasn't patient enough to realize this.

Once I did determine that NIS was the culprit, disabling it was still fairly tricky. I had to edit /etc/init.d/nis, set the domain to an empty string early in the file:


domain=""

And then also comment out the setting of the variable "d" because the domainname program would somehow still return my domain, even though domain has been set to "".


#d=`domainname`

But! After that, and a reboot, everything started working really well.

I was able to download a bunch of useful software from the SCO Skunkware FTP site, but unfortunately none of it worked without first installing a libc patch. Apparently the shared version of the libc that ships with 7.0.1 is missing some symbols. Weird. The patch fixed it though.

So I installed that patch, and several bits of software, like bash, sudo, gzip, etc., all using the standard SVR4 package management tools, which are worth discussing a bit.

Package management on UnixWare, while SVR4-compliant, is (not-unexpectedly) archaic. There's no dependency-resolving "yum install xxx" or "apt-get install yyy". You just download a .pkg file, hope it doesn't have any not-too-obvious dependencies, and install it in a completely unintuitive manner.

For example, if you download a file named xxx.pkg, then you'd install it using:


pkgadd -d `pwd`/xxx.pkg

The -d meaning "the packages are contained on this device or in this directory".

What? I'm trying to install the package xxx.pkg, why do I declare it to be the directory containing packages?

Well, the pkg file is actually a cpio archive.

Aaaand?

In antiquity, packages weren't distributed individually. Dozens of expanded package trees were cpio'ed en-masse to a tape, and the tape was distributed. If you aimed pkgadd at the tape device, like pkgadd -d /dev/rmt0, or similar, it would tell you what packages are available and allow you to install one or more of them at a time. If you already knew which one(s) you wanted to install, you could supply the name(s) on the command line too.

As hard drives got bigger, someone got the idea to distribute packages as individual files, rather than collected up on tape. And, since on unix, everything is a file, pkgadd doesn't care whether the argument to -d is a file or a device, as long as it's a cpio archive.

Clever. Agreed. But completely unintuitive.

In fact, the man page for pkgin doesn't help much either. It goes on and on about how -d can refer to a directory, or a pipe, or a device (like a tape), and about the default locations, and other stuff, and unless one notices that -d can refer to a "directory, file, or named pipe", and it occurs to one that file might mean .pkg file, and that .pkg files might simulate tapes, then one might overlook that option altogether.

That's what I did, way back, when I first encountered UnixWare, and it drove me nuts. Even after I figured out the whole file=tape thing, I still couldn't get it to work because the fact that the argument to -d has to be the full pathname to the .pkg file is not mentioned ANYWHERE, AT ALL!

Somewhere I found an example though, and had to laugh, in hindsight, at how obvious it should have been.

Remote Access

It was fun using the CDE desktop and all, but not practical in the long run. Telnet worked, but it's difficult to automate.

OpenSSH packages were available, but they were apparently built for a later version of UnixWare. sshd kept complaining about libc missing the strlcmp function.

No problem though, I've built OpenSSH for a bunch of old Linux and Solaris platforms. Maybe I could get it to work on UnixWare too.

I had to install a few packages first though. OpenSSL for starters, and it required prngd and zlib. Also, I didn't have a lot of faith in the native development system, so I installed gcc-2.95.2pl1 too.

And, it turned out, building OpenSSH-3.4p1 from source was quick and easy.


./configure --prefix=/usr/local/openssh-3.4p1
make
sudo make install

Configuring it was familiar from OSR5 too.

sshd_config:


X11Forwarding yes
...
UsePrivilegeSeparation no

And I created an init script at /etc/init.d/sshd...


#!/bin/sh

case "$1" in
start)
/usr/local/openssh-3.4p1/sbin/sshd
;;
stop)
kill `ps -efa | grep sshd | grep -v grep | cut -c10-15`
;;
*)
echo $"Usage: $0 {start|stop}"
exit 1
esac

exit 0

...made it executable, and set it up to run at boot:


chmod 755 /etc/init.d/sshd
cd /etc/rc2.d
ln -s ../init.d/sshd S98sshd

The last step was to uninstall OpenSSL. Oddly, for both UnixWare and OpenServer, the OpenSSL packages only contain static libraries. This creates problems for my software, and since once OpenSSL is compiled it no longer needs the OpenSSL libraries, it's safe to remove the packages.

Reboot!

unixware 7.0.1 - 5. ssh

Remote access!

But, it wasn't perfect. ksh on UnixWare is all but unusable over SSH. Neither backspace nor delete work. Bash worked though, so it was easy enough to work around.

I could ssh out too.

Looking good.

File Transfer

As mentioned earlier, ftp worked pretty well. OpenSSH came with scp and sftp. I'd installed gzip earlier. UnixWare comes with tar and cpio, though neither can gunzip on the fly. Packages for GNU tar, as well as zip, unzip and bzip were all available from skunkware.

After installing a few more packages, I was well equipped to transfer files.

The Web

No system is complete without being able to get on the web, right?

Well...

UnixWare 7.0.1 came with Netscape Navigator 3.0.4. Cutting edge for 1999, but it struggled pretty badly with today's web.

I installed wget and lynx packages from skunkware though. Good old lynx.

On the server side, things were slightly better.

UnixWare comes with the Netscape FastTrack web server. To turn it on, you have to log into a CDE desktop, run SCOAdmin and select Netscape Server Admin. This launches Netscape and aims it at FastTrack's web-based configuration interface.

After logging in as admin and using the password for the root user, you get this snazzy UI.

unixware 7.0.1 - 6. netscape fasttrack web server

The toggle switch on the left is for the web server on port 80. Switching it to the "ON" position started up the web server, and it was immediately accessible from my local browser.

unixware 7.0.1 - 7. default web page

Oh yeah!

It took a bit of digging around in the interface, but I figured out how to enable CGI's and a quick "print out the environment variables" script worked.

Skunkware packages for Apache were also available, but I didn't try them. Maybe later.

Graphics

The CDE login and desktop was enabled by default.

It can be disabled using:


scologin stop

And re-enabled using:


scologin start

When disabled, running startx from the command line, surprisingly, didn't start a CDE session though.

unixware 7.0.1 - 8. non-cde desktop

Weird. I'd apparently never done that before because I didn't recognize the desktop from OpenServer, UnixWare, or anywhere else.

It turns out that it is possible to get a CDE session with startx though, if you first create a .xinitrc file containing:


dtsession
Development

There were Skunkware packages for GNU make and CVS. What about the compiler though?

Well, I'd intalled gcc earlier to build OpenSSH, but I had 2 CD's. One with the OS, and the other with the development system. And, I had actual licenses for it in the same pack as the OS licenses.

The OpenServer development system isn't super mature, at least as of OSR 5.0.5. There's no support for 64-bit integers, and the C++ compiler is just a c-front and both quirky and not terribly capable with templates, even simple ones. I wasn't sure about the UnixWare development system though. Back in the late 90's, I never got around to playing with it.

Now would be a good time, I figured.

The CD had a README on it, so I followed its instructions to run the Development System's installer.

unixware 7.0.1 - 9. dev system

Turns out, it installed the C and C++ compilers, a ton of libraries and headers, and Java version 1.1.3. Whooo! 1.1.3. That's a blast from the past.

The UnixWare C++ compiler was a lot more capable than it's OSR cousin. In fact, calling them cousins isn't even inaccurate. It didn't appear to be a c-front, and compiled my Rudiments library with only a few tweaks.

But, when I got around to running the tests, I discovered one of its deficiencies. Like earlier versions of gcc, it didn't handle nested template invocations correctly. Darnit.

There were issues with variadic macros too, but I could just comment those out. The template thing was a show-stopper.

That's OK, there's always gcc, right?

Not exactly.

I'd installed gcc-2.95.2p1 earlier, and it was able to compile all of my software quite easily, but when I tried to run just about anything, I'd get a segfault, almost immediately.

I tracked it down to a dynamic linker error of some sort. Basically, my charstring class had an overloaded compare() function with two different signatures, and attempts to call the one with 3 arguments would crash.

I'd actually seen this same error on OpenServer 6 years ago, and eventually fixed it by compiling gcc-2.95.3, with some SCO patches from source, using the native compiler.

No such luck though. Same error.

I tried other versions of gcc-2.95.x, both packages and builds from source. Nothing worked. At best I'd get the same error. At worst, the compiler would segfault. Eventually I gave up.

And that's where it sat, for months. Sometimes you've got to get away from a problem for a while though. This afternoon I took another look at it. I noticed that there were Skunkware packages for older versions of gcc, like 2.8.0 and the egcs compilers. I couldn't remember if I'd tried them before or not.

I must not have because the compiler from the egcs-1.1.1 package ran without segfaulting, and produced working code.

The rudiments tests mostly ran fine. SQL Relay built and the command line client ran.

unixware 7.0.1 - 10. sqlrsh

SQL Relay supports lots of languages and database backends, but unfortunately not many of them are well supported on UnixWare 7.0.1.

I couldn't get it working with Perl, Python, PHP, TCL, Ruby or Java. In some cases, even the oldest language package was clearly meant for a newer version of UnixWare. In other cases, there were odd header quirks that prevented them from compiling cleanly. I'm not sure what the issue is with Java yet. By rights, it ought to work but I get weird errors at runtime.

Things weren't any better on the database front. PostgreSQL was the only database available, and the only package that would install was for version 6.5.3 - older than the oldest version I've ever tried before. UnixODBC was available, but the package was for a newer version of UnixWare.

SQL Relay builds against that old PostgreSQL, but there's some shared-memory-related issue that keeps it from running.

Ehh, who knows. There's always some quirky thing with every platform.

That's where I left it.

Quirks and Endearing Features

So, what's weird about UnixWare 7.0.1?

Not all that much, actually, especially given it's age.

The most significant thing is an issue with the filesystem. It's very easy to run out of inodes, waaaaay before you run out of space:


bash-2.05a$ df -v /
Mount Dir Filesystem blocks used avail %used
/ /dev/root 16000740 1538304 14462436 10%
bash-2.05a$ df -i /
Mount Dir Filesystem iused ifree itotal %iused
/ /dev/root 55862 9674 65536 85%

Multiple times I had to tar and zip stuff up and delete directory trees to free up enough space to untar something else. It looks like they're using a 16-bit number for the number of inodes. I guess back in the day, when a 1.6G hard drive was gigantic, file systems would have been smaller, there would have been more of them, and that was enough. But I'd swear that one of the selling points of UnixWare was support for petabyte and exabyte sized volumes. Maybe the filesystem does support more inodes, but there's no option for it at install time, and 65536 is a small default.

Other quirks...

The ksh shell has trouble over ssh, neither backspace nor delete work.

The halt and reboot commands are in /usr/ucb, which isn't in the default PATH.

Adding directories to the PATH is remarkably difficult. /etc/profile exports PATH before setting it. Lord knows where it's getting the default PATH from. Also, CDE sessions don't appear to source /etc/profile at all. Again, it's unclear where their PATH is set.

socklen_t is defined but a lot of network functions use size_t instead.

The chmod command accepts a shorter argument list than commands like cp. I had to update several Makefiles to work around this.

If you make a change that requires a kernel relink, the relink is done during the reboot, not before. As such, there's no unix.old to boot to until after the boot fails. Actually, for that matter, it's not clear how to get to a bootloader prompt.

Ha!

Strange, old unix.

Gotta love it.

Friday, April 24, 2015

SQL Relay 0.59 is out

SQL Relay version 0.59 is now available!

Significant New Features

There are a few...

Improved Support for Windows

This is the main new feature in this release.

Support for Ruby and TCL have been added. Now, on Windows, you can develop in C, C++, any .NET-supported language, Java, Perl, Python, PHP, Ruby and TCL.

Support for many additional databases has been added. SQL Relay can now be run on Windows, against Oracle, Sybase/SAP, DB2, Firebird, Postgresql, Mysql and ODBC-supported databases. The query router is also supported now too.

Performance isn't as good on Windows as it is on Unix/Linux yet, but it works and it appears to be stable.

On the development side, the build process has been completely refactored to use nmake and now supports a wide variety of versions of Visual Studio. It has been tested with VS2005, 2008, 2010, 2012 and 2013. VS6 and VS5 mostly work, but not entirely, yet. The build takes infinitely less time and it's now possible to fix something and nmake and nmake install from within the directory of the modified file, like one would be accustomed to on Unix/Linux. The Windows build process isn't yet as capable as the Unix/Linux build process, but it is greatly improved. See Installation on Windows for more information.

ODBC Improvements

Arguably this falls under Improved Support for Windows, but technically some of these improvements apply to UnixODBC/iODBC as well.

ODBC has long worked on Windows, but configuring it required manual modifications to the registry. No longer! The ODBC driver manager dialog exists now. On Windows, it should now be as easy to configure an ODBC connection to SQL Relay as it is to any other database.

The driver supports getting and setting many more attributes, and SQLTables works now. This enables many, many more ODBC applications to work with SQL Relay. It's been tested against a long list of apps with good success. If you've had trouble with it in the past, try this version.

MySQL Improvements

The MySQL database connection configuration now supports the maxselectlistsize and maxitembuffersize parameters, similar to Oracle, Sybase, DB2 and ODBC. This enables tuning of memory usage at runtime. See Tuning SQL Relay - Memory Usage for details.

LOB support is much improved with MySQL now too. In the past, the maximum number of bytes that could be fetched from a LOB column was limited by a macro in the source code - the same macro that defined the maximum size of a non-lob column. LOB's are now fetched in chunks and the amount of data that can be fetched is unlimited.

Support for Mono

SQL Relay has long had a C# API and ADO.NET driver, but they were classically only supported on Windows. These API's now work on Unix and Linux platforms as well, using Mono.

Less Significant, But Still Interesting New Features

The number of bind variables per-query was originally limited by a macro in the source code. Now, on the client-side, the number of bind variables is unlimited. Well, it's only limited by memory. On the server side, there's a new maxbindvars parameter in the sqlrelay.conf file that can be used to limit the number of bind variables that the server will accept. maxbindvars defaults to 256, but if you have an application that needs to send 1000 bind variables, you can just set it to 1000.

Login warnings are now printed on the screen, but not interpreted as errors. For example, if the database password is going to expire soon, but hasn't yet expired, the warning is displayed, but no longer interpreted as an error.

The sqlrsh command line client now supports the "cache" and "opencache" commands to cache result sets to a file and fetch result sets from a file. Not sure how many people actually use that feature, but sqlrsh supports fiddling with it now.

The logger modules have an enabled parameter which can be set to "yes" or "no". It's no longer necessary to comment out or remove the logger configuration to disable logging, you can just set enabled="no". See Logger Modules for more information.

Firebird recently added support for global temporary tables. Their implementation is very similar to Oracle's. SQL Relay now manages global temporary tables identically for Oracle and Firebird. There is even a droptemptables parameter in the Firebird connection configuration, analogous to the same parameter for Oracle.

In fact, support for global temporary tables has been generally improved. In the past, tables created during the current SQL Relay session were tracked and truncated at the end of the session, but global temporary tables created during previous runs of SQL Relay or outside of SQL Relay altogether were ignored, leading to those global temporary tables not being truncated at the end of the session. The Oracle and Firebird connections now support a globaltemptables parameter that can be set to a list of tables that should be truncated at the end of each session, or to "%", which causes SQL Relay to query the database for the list of global temp tables to truncate.

The Ruby DBI driver has been retired. I guess that's not a new feature, but it's something that was done in this release. Ruby DBI seems to have fallen out of favor and the driver was becoming increasingly difficult to maintain.

Tweaks and Fixes

Lots of small things have been tweaked, fixed, or improved as well

  • Binding a NULL to an integer works with db2 now.
  • Added a semaphore to ensure that the listener doesn't hand off the client to the connection until the connection is ready, eliminating a race condition on the handoff socket that could occur if the connection timed out waiting for the listener just after the listener had decided to use that connection.
  • Oracle temp tables that need to be truncated at the end of the session are truncated with "truncate table xxx" now rather than "delete from xxx".
  • Oracle temp tables that need to be dropped at the end of the session are truncated first, rather than the connection re-logging in.
  • An ora-14452 error (basically indicating that a temp table can only be dropped after being truncated, or if the current session ends) does not automatically trigger a re-login any more.
  • Updated cachemanager to use directory::read() directly instead of directory::getChildName(index).
  • Made cache ttl a 64-bit number.
  • Updated odbc connection code to use new/delete and rudiments methods rather than malloc/free and native calls.
  • Fixed command line client crash when using -id "instance" with an instance that uses authtier="database".
  • Fixed bugs that could make reexecuted db2 selects fail and cause a database re-login loop.
  • Tweaked spec file to remove empty directories on uninstall.
  • Fixed typo that could sometimes cause a listener crash.
  • Postgresql and MDBTools return error code of 1 rather than 0 for all errors now.
  • Fixed bugs related to autocommit with db's that support transaction blocks.
  • Fixed various bugs in sqlrconfigfile that caused sqlr-start with no -id to crash or behave strangely sometimes.
  • Updated the slow query logger to show the date/time that the query was executed.
  • Consolidated c, c++ and server source/includes down a few levels.
  • Fixed a misspelling in sqlrelay.dtd.
  • Swapped order of init directory detection, looking for /etc/init.d ahead of /etc/rc.d/init.d to resolve conflict with dkms on SuSE Enterprise.
  • sqlr-start spawns a new window on Windows now.
  • updated mysql connection to allow mysql_init to allocate a mysql struct on platforms that support mysql_init, rather than using a static struct.
  • Fixed subtle noon/midnight-related bugs in date/time translation.
  • Updated mysql connection to get affected rows when not using the statement api.
  • Updated mysql connection not to use the statement API on windows, for now.
  • Disabled mysql_change_user, for now.
  • Fixed blob-input binds on firebird.

Give it a try, report any issues!