Version 2.1.1 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
This patch release adds --sysconfdir to sqlrelay-config, fixes a bug which caused the port="..." attribute of the instance tag to be ignored, and loads the Oracle libclntsh library into its own linker namespace (if the platform supports it) to work around conflicts between its internal versions of other libraries (eg. openldap) and system libraries.
Version 2.1.0 of Rudiments, the C++ class library for developing systems and applications, is now available.
This release features some refactoring of the mvc classes and other things that are used when building MVC apps, some improvements to the logger classes, support for linker namespaces on platforms that support dlmopen(), and some fixes for older platforms.
Version 2.1.0 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
This release adds much improved debug logging. The debug logger module now supports options for logging to standard output and standard error. Most tags in the sqlrelay.conf file now support a debug="yes/no" attribute which can be used to enable or disable debug for that specific module or subsystem. Debug to standard output for a particular module or subsystem can be enabled by just adding a debug="yes" to the appropriate tag, without specifically configuring a debug logger, now. However, if a debug logger is configured, then debug will be sent to it, and logged to either standard output, standard error, or files, as specified by its configuration. The debug itself is now more sane and readable as well. See the SQL Relay Configuration Guide for more detail.
Version 2.0.2 of Rudiments, the C++ class library for developing systems and applications, is now available.
This patch release just moves things around internally such that syslog-related macros are exposed when including logger.h.
Version 2.0.1 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
The SQL Relay native APIs, database abstraction layers, and clients now supports attempts to connect to multiple hosts and host randomization, providing another option for front-end load balancing and failover. If the host/server is a comma-separated list of hosts, then the list of hosts will be randomized, and an attempt will be made to connect to each until the attempt succeeds. See the Front-End Load Balancing - Host Randomization section of the Configuration Guide for more details.
Version 2.0.1 of Rudiments, the C++ class library for developing systems and applications, is now available.
This release contains one main new feature. The inetsocketclient class now supports attempts to connect to multiple hosts and host randomization, providing a basic load balancing and failover feature to projects that use the class. If the host is a comma-separated list of hosts, then an attempt will be made to connect to each until the attempt succeeds, or there are no more hosts left to try. By default, the list of hosts is randomized, but this can be controlled by the randomizeHosts() method.
Version 2.0.0 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
Rudiments 2.0.0 was just released as well, and since many of its classes underwent significant overhaul, it was necessary to overhaul parts of SQL Relay to take advantage of Rudiments' changes and new features. As such, most of the changes in SQL Relay 2.0.0 are internal. However, there are still quite a few significant user-facing features worth mentioning in this release.
As usual, support for modern versions of languages, operating systems, and databases has been included and wrung out - Ubuntu 24.04, Oracle 18c and 19c, and PDO in PHP 8.1. Support for a few more legacy platforms has been included as well - older versions of OS X, PHP and FreeTDS. Support for mdbtools has finally been dropped entirely.
Historically, authentication has been one of the biggest points of confusion with SQL Relay. If you include "user=scott;password=tiger" in the database connect string, then why don't you use scott/tiger when connecting to SQL Relay? Or if you want that to work, why do you have to create an entirely separate <users> tag elsewhere in the config file that just has scott/tiger in it again?
Well... Reasons. But not very good ones.
Starting in 2.0.0, the user/password that you use to log in to the database, is the user/password that you use to log in to SQL Relay, unless you specifically choose another user/password, or another authentication method. This is accomplished by the connect-string auth module, which is loaded by default, unless some other auth module is specifically chosen.
Protocol Modules vs. Drop-in-Replacement LibrariesSQL Relay 1.9 provided drop-in-replacement libraries for MySQL and PostgreSQL - libraries that reimplemented libmysqlclient and libpq with functions that called the functions of the SQL Relay client API. When LD_PRELOAD'ED, they enabled some MySQL and PostgreSQL applications to use SQL Relay without modification. This wasn't an ideal solution though. It required installation and configuration of every client machine, it didn't work at all on Windows, many apps (most notably PHP) were statically linked against libmysqlclient, and other apps just used pure reimplementations of the database protocols in Java, C#, or some other language.
To remedy this, SQL Relay 1.9 also provided server-side protocol modules for MySQL and PostgreSQL. With one of these modules enabled, apps could just be aimed at SQL Relay rather than at MySQL or PostgreSQL and work without modification. This was a better solution - it required no client-side changes, worked with windows clients, worked with PHP clients, and worked with non-C/C++ clients.
In SQL Relay 2.0.0, the server-side protocol modules are mature and the drop-in-replacement libraries are no more. If you want to drop SQL Relay in between your app and database, server-side protocol modules are the way to go.
Fledgling Protocol ModulesThe MySQL and PostgreSQL protocol modules are quite mature, but work has been in progress, off and on, for years, on protocol modules for other databases as well. Specifically, there are fledgling protocol modules for Oracle, MSSQL Server, Firebird, and Teradata.
The Teradata protocol is actually almost entirely complete, lacking only password encryption, for which I cannot find a detailed enough description to implement. The MSSQL Server module can kind-of perform some non-MARS tasks. The Oracle protocol can talk to an 8.0 client and return result sets from tables containing only varchars. Old enough Firebird clients can talk to the Firebird module and do similar things.
They could use some attention. Hopefully open-sourcing them will draw some.
Error Translation ModulesSQL Relay 2.0.0 provides a new error translation framework. If you're using a protocol module to make your PostgreSQL app think that it's talking to PostgreSQL, but it's really talking to Oracle via SQL Relay with some query translations, then that app may not understand what to do when it receives a native Oracle database error. Error translation modules allow you to translate error codes and messages on the back-end, similarly to how query translation modules allow you to translate queries on the front-end.
Upsert Trigger ModuleThe new upsert trigger module transparently converts inserts to updates if the insert fails with an error indicating that the row already exists.
Split Multi-Insert Trigger ModuleThe replay module automatically recovers from databse deadlocks by logging queries, rolling back, and replaying the query log if a deadlock is detected. However, multi-inserts (inserts that insert multiple rows at a time) into tables with an autoincrementing keys were always problematic for the replay module, as it was only every possible to get the last_insert_id of the final insert. The split multi-insert trigger module provides a solution for this by splitting multi-inserts into a set of single-inserts, for which the last_insert_id of each is available. It works well with the upsert module too.
Fully Commented Server-Side MethodsSQL Relay is highly modular, and the various module frameworks are fairly mature, but for the longest time, there has been no real documentation explaining how to develop modules, or how to use the myriad of server-side methods available to them. The first steps have been taken in that direction in 2.0.0. 100% of the server-side API has been commented, sufficient for doxygen to understand. A proper module-building guide and reference is forthcoming.
The SQL Relay client API has always supported substitution variables, both as an alternative to bind variables, and in cases where you can't use bind variables. You can, for example: "select * from sometable where $(WHERECLAUSE)" and then replace $(WHERECLAUSE) with an actual where clause. In SQL Relay 2.0.0, you can embed substitution variables into the values of other substitution variables. For example, in 2.0.0, $(WHERECLAUSE) can be "$(TABLE1).$(COLUMN1) = $(TABLE2).$(COUMN2)".
SQLR CRUDThe sqlrcrud class implements the new Rudiments mvccrud interface, providing doCreate(), doRead(), doUpdate(), and doDelete() methods against a SQL Relay backend. RUD methods that take a criteria argument support jsonlogic where clauses, and doRead() supports a JSON-formatted sort-by argument. Results can be fetched using various SQLR Collections. When paired with the Rudiments MVC framework, it's possible to build MVC apps in C++, with an SQL Relay backend.
SQLR CollectionsSQLR Collections implement Rudiments collections. sqlrrowlist and sqlrrowdictionary provide access to a row of the result set as if it were a read-only linked list or dictionary. sqlrresultsettable provides access to the entire result set as a read-only, sequential, block-based table.
Many MVC DAOs create lists, dictionaries, or tables containing entire result sets, then pass those up, ultimately to the view. This involves buffering entire result sets, creating nodes (and the pointers between them), and all too-often copying values, not to mention writing the code to populate the collections. The SQLR Collections eschew all of that by only buffering a block of results at a time, using API methods to navigate the result set, and only presenting the data as node-based collection, rather than actually creating and managing nodes. And the code is already-written and reusable.
sqlrimport and sqlrexportThe sqlrimport and sqlrexport classes (and their children) have existed for a while, but their implementations have always been inconsistent, and not especially usable by developers. They have been overhauled in 2.0.0 with a robust class hierarchy, consistent behavior across child classes, and well-commented and doxygened methods. A proper guide and reference are coming.
Of course, there are many other improvements and bug fixes that were included in this release, but this announcement is already too long as it is. See the ChangeLog in the source distribution if you're interested in every single detail.
Version 2.0.0 of Rudiments, the C++ class library for developing systems and applications, is now available.
This release contains a tremendous number of changes and improvements...
Much like Java, nearly every class now inherits from a base object class. This eliminates the need for some awkward void pointers, simplifies a bunch of internals, and facilitates a wastebasket class. Speaking of which...
WastebasketEver need to create an arbitrary collection of objects, down in the depths of some body of code, and hang on to them until some higher level code is done with them? If that sounds abstract... In an MVC architecture, have you ever created a container full of objects way down in a DAO implementation and needed them to hang around until the view was done with them? In languages with garbage collection, they'll just hang around on their own until nobody needs them any more, but it's more complex than that in C++. The wastebasket class allows you to attach objects (children of the object class) to the wastebasket for bulk disposal later. You can create an arbitrary set of objects, throw them to the wastebasket, pass them back up to wherever, and continue to use them until you empty the wastebasket, at which point they'll all be destroyed.
MVC FrameworkSpeaking of DAO implementations and views... Rudiments 2.0.0 provides a basic MVC framework, which provides base classes for building out MVC applications. For example, there's an mvcsecurity class, an httpbasicsecurity implementation, an mvcproperites class that provides support for java-style properties files, mvccontroller, mvcview, mvcservice, and mvcdao classes, an mvcresult class for passing data up from mvcdao implementations, and an mvccrud class that defines an interface for classes that implement the CRUD paradigm. A resourcepool class is also provided for managing shared pools of resources.
Collection OverhaulThe collection classes (lists, trees, dictionaries, arrays, etc.) have undergone a major overhaul. Everything inherits from a base collection class now, which implements a bunch of functionality common to all containers. You can set a comparator to be used in sorting (eg. to sort in a non-sequential, "natural", or reverse order). Collections can be read-only, block/unit-based, or sequential-only.
Most notably though, a collection can now either "manage" its values or not. A collection that manages its values deletes them when the collection itself is deleted, and in the case of node-based collections like lists and trees, values are deleted when nodes are removed. In a dictionary, keys can also be managed. In rudiments 1.4, there were methods that you could call to delete or array-delete keys and values before deleting the collection, but this is a cleaner strategy.
There are also 2 new collections: table and scalar. The table container implements a table, with rows, columns, and column names. The scalar collection is a degenerate collection that only stores one value - useful when you have a single value but want to use a class that only operates on collections.
Character Set ImprovementsRudiments 1.4 had wcharacter/wcharstring/wstringbuffer classes. Rudiments 2.0 improves them and adds ucs2character/ucs2charstring/ucs2stringbuffer classes. Translation between character sets is improved as well. For example you can wcharstring::copy() or wcharstring::duplicate() an ascii or utf2 string, and vice-versa. The filedescriptor class provides methods for reading and writing wchar and ucs2 characters and strings. There's an iconvert class for translation of characters and strings between character sets. There's also a new locale class for managing locales and locale components.
Shutdown FlagWhen an app crashes or receives a signal telling it to shut down, the signal handler shoudn't really be responsible for actually shutting down the app. Signal handlers really shouldn't do very much, at all! Arguably, a signal handler ought to just set a flag, the rest of the app should check for that flag in various places, and shut the app down gracefully if the flag was set. I've implemented that paradigm independently a dozen times, and finally decided to move it up into rudiments.
The process class now provides a set of methods like setShutDownFlagOnCrashOrShutDown() which just set a flag if a signal is caught that would crash or shut down the process, and also provides a getShutDownFlag() method to query that flag. Rudiments methods that retry on EINTR check this flag, and bail if it was set.
Block-Aligned and mmap() BufferingThe filedescriptor class can now differentiate between streams and storage, and buffer differently depending on which the descriptor is. If setIsStream(true) is called (the default) then it buffers the way it always has. If setIsStream(false) is called (for storage), then it aligns its buffers with block boundaries, and will use mmap(), when possible, if setMmapBufferingEnabled(true) is called. The various setPositionRelativeTo*() methods also now work as expected when buffering of storage is enabled.
byte_tHistorically, rudiments used unsigned char to represent a byte. Now there's a proper byte_t type that represents a byte.
VerbsMethods tend to do things, and, as such ought to be verbs. Lots of methods have been renamed such that they "do" things now - eg. getIntegerLength() instead of integerLength(). Other methods that just describe a state or condition at least start with "is" - eg. isSupported() instead of supported().
Size, Length, and CountFor a long time, Rudiments (along with many other software projects) have confused size, length, and count. In Rudiments 2.0.0, size refers to the number of bytes, length to the number of characters, and count to the number of items. Eg. a ucs2 string might have a size of 10 but a length of 5. A linked list containing 5 ucs2 strings would have a count of 5. Method names and parameters have been overhauled, project-wide.
Bug Fixes and ImprovementsThe list of bug fixes and internal improvements could go on for days. Nearly evey class has been overhauled, updated, examined, or at least touched in some way. See the ChangeLog enclosed with the source distribution for the gory details.
I just updated the release files for version 1.9.3 of SQL Relay. There was a problem that caused the master branch rather than the 1.9.3 branch to get packaged as 1.9.3, which led to all kinds of build errors. That's been fixed. The files that are out there now should contain source for the actual 1.9.3 release. If you had trouble building, then that's almost certainly why. Try re-downloading the SQL Relay source. It should work this time.
Version 1.9.3 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
This release has 2 main features:
However, it does include some bug fixes as well:
If any of that sounds familiar to you, or if you need support for PHP 8.1 or Oracle 18+, give this release a try!
Version 1.4.2 of Rudiments, the C++ class library for developing systems and applications, is now available.
This release fixes bugs that caused timeouts to not timeout on systems that use epoll_wait() (eg. Linux) and kqueue() (eg. FreeBSD).
Version 1.9.1 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
This release has 2 main features:
Improved Ruby APISome time ago Ruby added a Global Interpreter Lock, similar to the one that Python has had forever. To improve performance with modern Ruby, I added code to release/reacquire the lock around SQL Relay C-API calls in case they perform I/O, similar to what the Python API does.
Password File Support in the Userlist Auth Modules and Connect StringsIf you don't want to store passwords directly in the sqlrelay.conf files any more, it's now possible to store them in external files (with potentially more restrictive permissions) and refer to them in the sqlrelay.conf file enclosed in square brackets.
The userlist, mysql_userlist, and postgresql_userlist auth modules now support this. It is also supported in the "password" parameter of the "string" attribute of the "connection" tag for all databases.
Bug FixesThe sqlrclient protocol module had some quirks related to how it handled cases where it was out of cursors, but was sent a really long query. These are resolved now. There was also a bug that could cause the result set not to be closed and the "current error" not to be cleared between reexecutions of the same query (eg. with updated bind variables) if the previous execution resulted in an error. That's also fixed. Both fairly obscure errors, but if they sound familiar then try this release.
Version 1.4.1 of Rudiments, the C++ class library for developing systems and applications, is now available.
Though this release just adds the sensitivedata class required by SQL Relay 1.9.2.
Well, I can't explain how, but apparently a few files inside of the previously released sqlrelay source code got gzipped. Specifically config.guess and config.sub. This led to configure not working, and general impossibility compiling things. I just repackaged everything again, double-checked it prior to upload, and double-checked the download. All appears to be well at the moment. If you had trouble building SQL Relay from source, please re-download and try again.
It looks like I made a mistake during the 1.9.1 release. The sqlrelay-1.9.1.tar.gz and sqlrelay-1.9.1.zip files containing the source for SQL Relay actually contained the source code for the trunk, rather than the 1.9.1 release. Similarly, the source code for both rudiments and sqlrelay that was included inside of the binary distributions was also the source code for the trunk, rather than the appropriate vesions. This has just been fixed. So, if you had trouble building the source, or if you managed to successfully build the source, but running "sqlrsh --version" returns "sqlrsh 2.0.0" rather than "sqlrsh 1.9.1", then please re-download/build/install the 1.9.1 release.
There was no issue with the binary release. The rpms, debs, and windows installers all included the correct versions. The mistake was, oddly, just in the source distribution.
Version 1.9.1 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
This release fixes 2 bugs: a potential crash in the PostgreSQL front-end module, and a fix/workaround for an isuse where if a NULL was bound to the first argument of MS SQL Server's isnull() function then it wouldn't actually be interpreted as a NULL.
If you've run into either of these issues, then this release is for you!
Version 1.9.0 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
The main features of this release are:
Support for TLS Encryption in the MySQL and PostgreSQL Protocol Modules
SQL Relay has long supported the MySQL and PostgreSQL protocols, on the front-end, allowing SQL Relay to be dropped-in, transparently between many MySQL or PostgreSQL applications and the database. However, SQL Relay's implementation of the MySQL and PostgreSQL protocols has lacked support for TLS encryption, until now. It is now possible to do end-to-end encryption (without a tunnel) from a MySQL or PostgreSQL application, through SQL Relay, and into the database.
Support for UTF-16 Characters in SQL Server nchar/nvarchar Fields
SQL Server 2012 introduced the ability to store UTF-16 characters in nchar/nvarchar fields in addition to the standard UCS-2 that they traditionally supported. However, SQL Relay has traditionally forced a UCS-2 encoding when using nchar/nvarchar fields, preventing SQL Server users from being able to take advantage of this feature. Starting with this release, when using ODBC to connect SQL Relay to a SQL Server database, you can now specify ncharencoding=UCS-2 or ncharencoding=UTF-16 in the connect string to instruct SQL Relay how to deal with nchar/nvarchar fields.
PHP 8 Support
SQL Relay should now compile against, and work seamlessly with PHP 8.
NodeJS 14+ Support
SQL Relay should now compile against, and work seamlessly with NodeJS 14+.
Of course, there are many other subtle new features, improvements, and bug fixes. Full ChangeLog follows:
Version 1.4.0 of Rudiments, the C++ class library for developing systems and applications, is now available.
Though this release does include some new classes, additions to existing classes, this release mainly features changes to improve consistency between classes, as well as some under-the-hood improvements.
ChangeLog follows:
Version 1.3.1 of Rudiments, the C++ class library for developing systems and applications, is now available.
This patch release fixes a bug in the rpm spec file and a missing library dependency in the link command for librudiments-apache.
Version 1.8.0 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
This release mainly lays the groundwork for some future features, including a generic import/export framework. Some notable progress was also made on the JDBC driver, though it's still not finished. The most significant (finshed) new feature is an aes128 password encryption module.
There are some bugfixes as well. A long-standing issue with postgresql that could cause results from multiple open cursors to get confused has been resolved. A subtle error that could cause counting of bind variables on the client-side to hang has been fixed. A long-standing, but apparently obscure, issue that could cause tables from other MySQL schemas to be included in a "show tables" command has been fixed. As well as various other issues.
Full ChangeLog follows: