Monday, August 31, 2015

SQL Relay 0.63 Release Announcement

Version 0.63 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.

This release mainly focused on platform-compatibility, fixing failures to detect or build against various dependencies on modern versions of Fedora, Debian, Solaris, FreeBSD, and NetBSD.

A few bugs have been fixed as well, and there are a few changes.

Changes

The "oracle8" database type has been updated to just plain "oracle". So, in your sqlrelay.conf file, you should use dbase="oracle" rather than dbase="oracle8". "oracle8" is still supported, but the docs all say to use "oracle" now. Oracle 8i, 9i, 10g, 11g, and 12c have all come out since the driver was first written, and OCI hasn't undergone any massive rewrites in all of that time, so the name change definitely seems overdue.

Similarly, the "sybase" database type has been updated to "sap". SAP bought out Sybase a few years ago and ASE 16.0 is marketed as SAP ASE, not Sybase ASE. "sybase" still works, but the docs all say to use "sap" now.

There is also now an "id" parameter that can be set in the database connect string. If it's set, then if a client calls the identify() method or the sqlrcon_identify() function, it will return whatever value "id" is set to.

This is useful, for example, if an app requires that SQL Relay identify the database as "sybase" or "oracle8", as the name changes cause them to identify themselves as "sap" and "oracle" now, by default. It's also useful when connecting to an ODBC database. By default, SQL Relay identifies an ODBC database as "odbc", but it might be useful to the app to know what actual type of database is being accessed, rather than just "odbc". Similarly, FreeTDS connections can be id'ed as "sap", "sybase", "mssql", etc. rather than "freetds".

For example, with this configuration:

<instance id="mssqlexample" dbase="freetds" ... >
    ...
    <connections>
        <connection string="user=...;password=...;id=mssql;..." ...>
    </connections>
</instance>

In this code:

    ...
    sqlrconnection sqlrc("sqlrserver",9000,NULL,"...","...",0,1);
    ...
    identity=sqlrc.identify();
    ...

"identity" would be "mssql".

Bug Fixes

The default "maxitembuffersize" for Firebird is 32768 now, bumped up from 4096. Firebird has supported 32768-byte varchars for quite some time now. This should have been the default for all that time.

A few of the ADO.NET Data Provider classes implement the IDisposable pattern, but were missing destructors. The missing destructors have been added.

The setAuthTimeout() and setResponseTimeout() methods weren't properly exposed in the Ruby API. They are now.

And a few other things too... Full ChangeLog follows:

  • suppressed ruby configure warning
  • fixed glib detection if pkg-config isn't present
  • fixed mono detection on freebsd
  • fixed node.js yes/no misreport bug in configure script
  • added freetds tcl test
  • fixed slashes in tcl tests for windows
  • updated docs to mention firebird buffer size parameters
  • updated MAX_ITEM_BUFFER_SIZE to 32768 for firebird
  • fixed ruby detection on fedora 22
  • fixed missing export for setAuth/ResponseTimeout in ruby api
  • added manual include of inttypes.h in php api to work around issue with define/undef games, revealed on openbsd 5.7
  • tweaked ruby cflags script for debian 8
  • refactored mysql detection
  • tweaked unixodbc detection to also look in /usr/include/odbc
  • renamed "oracle8" connection plugin to "oracle"
  • renamed "sybase" connection plugin to "sap"
  • updated identity tests in api's/cmdline clients to use charstring::contains() rather than !charstring::compare()
  • implemented missing destructors for ADO.NET Data Provider

Rudiments 0.53 Release Announcement

Version 0.53 of Rudiments, the C++ class library for developing systems and applications, is now available.

This release features a bit of clean-up, and a few new features.

XML Improvements

The XML classes xmlsax, xmldom, and xmldomnode now support tag namespaces. The namespace is now an element of the xmldomnode and the various methods for navigating, adding, updating, and deleting nodes, all support namespaces now.

The xmldomnode class also features new appendXml/insertXml and appendXmlFile/insertXmlFile methods which allow a string or file containing XML to be parsed and inserted or appended directly into an existing tree.

codetreegrammar Class Exposed

The codetreegrammar class is now exposed. This allows the codetree class to use the same already-parsed grammar for successive parse or write passes without having to re-parse it.

DTD Class Removed

The outdated and long-unused dtd class has been removed.

Serial-port Classes Disabled By Default

The serialport, serialportprofile, modemserver and modemclient classes are omitted from the default build now, but can be enabled using the configure script's command line options.

The full ChangeLog follows:

  • exposed codetreegrammar class and updated codetree class so that successive parses/writes can use the same grammar without having to reload it
  • added insert/appendXml(File) methods to xmldomnode to parse and insert/append xml strings or files to a node
  • fixed FILE->fileno detection for solaris 11.2, which doesn't have it at all
  • llabs() is detected and preferred to abs()
  • added partial xml namespace support (namespaces for tags)
  • added partial support for extended ascii character set to character class
  • added setgroups() call before setuid() call to drop extraneous groups
  • updated rudiments-config man page, slightly
  • removed outdated and unused dtd class
  • updated file::resolveSymbolicLink() to use directory::maxPathLength(filename) as a starting point
  • updated directory::getCurrentDirectory() to use MAX_PATH as a starting point
  • serialport/serialportprofile, modemserver/modemclient classes are disabled in the default build now
  • updated intro docs

Friday, August 7, 2015

SQL Relay 0.62 release announcement

Version 0.62 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy is now available

Yes, that's right 0.62, not 0.61. 0.61 has been skipped for administrative reasons.

This release includes some significant new features.



node.js Client API

The most notable new feature is a client API for node.js. It's now possible to access any database supported by SQL Relay, from a node.js application. That includes Oracle, Microsoft SQL Server, Sybase/SAP, IBM DB2, MySQL, PostgreSQL, Firebird, SQLite, as well as any database accessible via ODBC.

See the Programming Guide for step-by-step instructions and examples, and the API Reference for details on the classes and methods.

Filters, Translations, and other Modules

SQL Relay has long had password encryption and logger modules, but new frameworks have been added for authentication, query and result set translation and query filtering. Some default modules have been included as well.

For more details on these frameworks, the included modules, and how to use them, see:

Firebird Updates

SQL Relay pre-allocates buffers to fetch result sets into, before sending the result set along to the client. It is now possible to tune the size of these buffers, and in-effect, memory usage, when using the Firebird database. The parameters involved are included in the database connect string, and are: maxitembuffersize, maxselectlistsize, and maxbindcount.

These parameters have long been supported with Oracle, Sybase/SAP and IBM DB2. Now they are supported with Firebird too.

Non-Lazy Connect

By default, SQL Relay does lazy connects. I.e. no connection is established to the SQL Relay server when the connection object is created. The connect doesn't occur until the first query (or other database operation) is run.

This is inconsistent with the behavior of most PDO and Perl DBI drivers though. Some apps rely on "new PDO" or "DBI->connect" failing if the database is inaccessible. Neither of those classically fail with SQL Relay though. The database can't be determined to be inaccessible until the first query is run. This behavior can cause problems when trying to insert SQL Relay between an existing app and the database that it used to connect to directly.

To address the issue, a new connect string variable has been added to the PHP PDO and Perl DBI drivers: lazyconnect.

If set to 1 (the default), then the default behavior is observed. If lazyconnect=0 is used, then a connection will be established immediately and "new PDO" and "DBI->connect" will fail if the database is inaccessible.

See the PHO PDO driver and Perl DBI driver documentation for more info.

Old PostgreSQL

Some apps rely on being able to send multiple, semicolon-delimited queries to PostgreSQL, in a single statement. However, changes to the PostgreSQL API in PostgreSQL 8, to support native bind variables, disabled this ability. SQL Relay's configure script detects PostgreSQL 8 and uses the modern API if it's available. This can cause problems for apps that needed the old behavior, when placing SQL Relay between them and the database using the drop-in replacement library for PostgreSQL. To remedy this problem, a --disable-postgresql8-api option has been added to the configure script. Using it disables the new API and makes it possible to send multiple, semicolon-delimited queries to PostgreSQL, in a single statement.

License Update

The documentation and examples given in the documentation are now explicitly covered by licenses. The documentation itself is GPL'ed and the examples are covered by the FSF Unlimited License:

The FSF Unlimited License

This software is free software; you have unlimited permission
to copy, distribute and modify it.

The SQLRelay Server, as defined in the file COPYING at the root of the source distribution, now carries an explicit exception that unquestionably allows it to be used with various non-GPL-compatible libraries. This was the intent all along, but modern interpretation of the GPL called it into question. There should be no question now. See a much longer discussion about this for more info.

Bug Fixes

The rest of the changes are just bug fixes.

Full ChangeLog follows:

  • added query filter plugin framework
  • added regex, string and pattern filters
  • added normalization translation
  • added reformatdatetime result set translation
  • fixed error in sqlrconfigfile class that could cause sqlr-start to crash if one instance defines addresses but the next doesn't
  • added query status flag to sqlrservercursor
  • tweaked odbc driver's SQLGetTypeInfo()
  • field name is passed into runResultSetTranslations() now
  • added support for maxitembuffersize, maxselectlistsize, and maxbindcount for firebird
  • added native api for node.js
  • added recognition of bigint, ubigint and uniqueidentifier fields in freetds/sybase
  • added nullsasnulls command to sqlrsh
  • added workaround for freetds empty (but non-null) text fields not getting properly converted to null-terminated empty strings
  • fetchatonce, maxitembuffersize and maxselectlistsize are set to defaulting values if invalid values are given for them
  • added lazyconnect option to PHP PDO and Perl DBI drivers (defaults to 1)
  • direct Transact SQL which returns a result set is now supported with Sybase/SAP/MSSQLServer/FreeTDS
  • added docs for authentication, query translation, query filter and result set translation modules
  • added -fPIC to sqlrserver-config --cflags, if supported
  • the "default" auth module is now called "userlist"
  • added database auth module that is analagous to authtier="database"
  • replaced passwordencryption attribute with passwordencryptionid
  • added --disable-postgresql8-api configure option
  • tweaked -Werror detection in configure script
  • refactored sqlr-bench build to work on windows
  • documentation, examples and tests are now covered by a license
  • reflecting modern interpretation of the GPL, The license for The SQLRelay Server (as designated in COPYING) license now carries an exception, explicitly allowing it to be linked with various non-GPL compatible libraries
  • license is installed now

Rudiments 0.52 is out

Rudiments version 0.52 is now available.

The ChangeLog is fairly short though each item involved more work, on average, than usual.

Over the past few years, I've done some trans-compiler work. The codetree class is one result of all that. The compiler class is another. So are the new xmldomnode methods. The compiler class is really just a framework for plugins. The plugins, codetree, and xmldomnode do all the real work.

The rpcentry class has been removed. I only originally included it for the sake of completeness, but have had no occasion to use it in 15 years. The rest are bug fixes and cleanup.

Full (though short) ChangeLog follows:

  • tweaked codetree's recursive break logic a bit
  • added compiler class
  • added case-insensitive methods to xmldomnode
  • added methods to xmldomnode that operate on descendents (as opposed to just operating on direct children)
  • added methods to xmldomnode for deleting, wrapping, unwrapping
  • fixed dynamiclib::getError() bug on windows
  • tweaked -Werror detection in configure script
  • removed rpcentry class
  • added file::setPermissions() methods

Wednesday, August 5, 2015

SQL Relay License Update

Background

SQL Relay is made up of a bunch of different parts: the server, the command line clients, API's for various languages, tests, documentation, etc. The server and command line clients have always been licensed under the GPL. The API's have been licensed under various licenses, as appropriate to each API - LGPL, Artistic License, Creative Commons, etc.

In the late 90's, when I first decided to use the GPL for the server, the open source community seemed to interpret the GPL as allowing dynamic linking of non-GPL-compatible libraries/plugins/modules/drivers into GPL-licensed programs. The Linux kernel was the poster boy for this interpretation, as hardware vendors provided (and still provide) closed-source drivers, under various non-GPL-compatible licenses. In the database arena, GPL-licensed programs like sqsh and tora illustrated this same interpretation.

The rationale seemed to be that the GPL-licensed program included instructions for loading and interfacing with a non-GPL library, but not the library itself. Headers generally "didn't count" so presumably neither did whatever bits of the library the linker needed to examine.

The GPL-licensed program and non-GPL library/plugin/module/driver were considered separate. They were only combined to create a derivative work at runtime. Copies of the pages of memory that composed that work could not be lawfully distributed, but distribution as separate components was fine.

That seemed to be conventional wisdom at the time.

I wasn't totally ambivalent to the politics associated with the GPL, but really, I needed to choose a license, the GPL was popular, and seemed to fit.

Since then, debates have raged. GNU has made their position abundantly clear: programs that are capable of sharing the same address space with a library/module/plugin/driver are a derivative work of the two, retroactively. Community opinion seems to be divided. The Linux development community is said to have a "gentlemen's agreement" with hardware manufacturers, but nothing in writing. There have been a few court cases and they've gone both ways.

These developments put projects like SQL Relay in an interesting position. Obviously it is the intention of the authors to allow linking with various non-GPL-compatible libraries, but under modern interpretation, the license doesn't explicitly allow that.

As of the next release, 0.61, it will.

The License Exception

For cases like this, GNU recommends a "license exception". The SQLRelay Server, as defined in the file COPYING, at the root of the source distribution, will carry the following exception:

Linking The SQLRelay Server statically or dynamically with other modules is
making a combined work based on The SQLRelay Server. Thus, the terms and
conditions of the GNU General Public License cover the whole combination.

In addition, as a special exception, the copyright holders of The SQLRelay
Server give you permission to combine The SQLRelay Server with free software
programs or libraries that are released under the GNU LGPL and with code
included in the standard release of the following libraries (or modified
versions of such code, with unchanged license):

* OpenSSL and its dependencies.
* Perl-Compatible Regular Expressions (PCRE) and its dependencies.
* IBM DB2 database client libraries and their dependencies.
* Firebird database client libraries and their dependencies.
* FreeTDS client libraries and their dependencies.
* MDB Tools client libraries and their dependencies.
* MySQL database client libraries and their dependencies.
* ODBC client libraries and their dependencies.
* Oracle database client libraries and their dependencies.
* PostgreSQL database client libraries and their dependencies.
* SQLite client libraries and their dependencies.
* Sybase/SAP client libraries and their dependencies.
* Any other programs or libraries as designated, in writing, now or in the
  future, by David Muse, his authorized agents, authorized heirs, or authorized
  agents of his authorized heirs.

You may copy and distribute such a system following the terms of the GNU GPL
for The SQLRelay Server and the licenses of the other code concerned{, provided
that you include the source code of that other code when and as the GNU GPL
requires distribution of source code}.

Note that people who make modified versions of The SQLRelay Server are not
obligated to grant this special exception for their modified versions; it is
their choice whether to do so. The GNU General Public License gives permission
to release a modified version without this exception; this exception also makes
it possible to release a modified version which carries forward this exception. 

Some of the named libraries are either LGPL already or otherwise GPL-compatible, but they are listed in case that changes in the future. If I missed something, the "Any other programs or libraries..." clause makes it possible to add whatever I missed too.

The big surprise: OpenSSL. Apparently it's licensed in a way that makes it non-GPL-compatible. Who knew? Apparently the authors of wget did. Did anyone else?

Modules

In addition to making it clear that those specific libraries are OK, there is another motivation to the license update.

SQL Relay has several module frameworks. You can do all kinds of things with modules - logging, password encryption, alternative authentication mechanisms, query filtering, query translation, result set translation, and more... And additional frameworks are planned.

Without the license update, module development is sketchy. For example:

Can I write a connection module for Informix? Good question.

OpenLDAP has an odd license, is it legal to develop an OpenLDAP authentication module? Who knows?

Can a software development firm write a module for a client that interfaces with some system that the client already uses via a non-GPL-compatible library? Maybe.

Can a software vendor write a module to interface with some proprietary system they developed? Also hard to say.

What about me, personally? Can I, personally, write proprietary plugins? As the primary copyright holder, I'd think so. But can I really?

The "Any other programs or libraries..." clause in the license update provides a path to these kinds of things. New libraries can be approved. If in doubt, just ask david.muse@firstworks.com and get a definitive answer, in writing.

As a guideline, the answer will almost certainly be yes for "client libraries" and their dependencies. By client libraries, I mean, there's some system running in its own address space and there's a library who's primary function is to provide access to it. That library is a "client library."

For other libraries, it would depend on the details.

Authority

By what authority do I change the license?

Mostly my own, partly that of the open source community, and partly that which was granted by other contributors.

I am the primary copyright holder for The SQLRelay Server. I'm not exactly sure what percentage of it I either wrote or own, but I'd bet that it's over 99%. The API's were largely contributed, but they're not affected by this change.

Over the years, patches from various contributors were applied to The SQLRelay Server, but the vast majority were "trivial bugfixes." While important, they were short and didn't demonstrate novel insight. I.e. the patch affected fewer than ten lines, the bug was easy to find, the bug is obvious now that it's been pointed out, and the fix couldn't have been implemented much differently. If the patch was short and met 2 out of 3 of the other criteria, then I deemed it trivial. Trivial bugfixes are likely not subject to copyright. This analysis and action seem consistent with those of the open source community in license update scenarios.

Some non-trivial patches were also integrated. But, I was able to get approval from the current copyright holders to make the license change. For good measure, I also got approval from all the trivial bugfixers that I could get a hold of too.

I did some contract jobs to integrate non-trivial patches against older versions into the then-current version of SQL Relay, but the language of the contract entitled me to the results.

There were also a few non-trivial patches that were once integrated into The SQLRelay Server, but have since been completely removed.

If anyone that I haven't received approval from can demonstrate that they contributed non-trivial code to The SQLRelay Server, a derivative of which still exists in the current version, and disagrees with the license change, please contact me at david.muse@firstworks.com and we'll work something out.

Ramifications

So, what does the license change mean to you?

Most likely nothing at all. The license is likely now just more consistent with what you assumed already. The GPL, in general, still applies. It is just definitively legal to use SQL Relay as it was intended.

The only practical difference is that if you want to write a module that integrates with some other system via a non-GPL-compatible client library, then it is very likely possible to do so.

"But this license change undermines the purpose of the GPL!"

Perhaps. SQL Relay was not written to crusade for the GPL. I don't intend to undermine the GPL though. In fact, rather than argue over how to interpret the GPL, I'm doing this in deference to GNU, who recommend exactly this action.

The GPL is great. Free software wouldn't be where it is without the GPL. With this update, it's still a good fit for this project.