Rudiments 0.50 is out

Rudiments 0.50 is now available!

The most significant feature in this release is a refactored, extent-based implementation of the variablebuffer and stringbuffer classes. The original implementation was crude, sufficient for storing fairly short strings, small in code size, but scaled poorly. The new implementation scales well while the code size still remains small.

There are also new dynamicarray and staticarray classes. Dynamicarray implements an array that expands as needed. It's also extent-based and scales well. Staticarry is just a thin wrapper around a C array. It only exists because you might want a dynamicarray who's members are static arrays, and there's no good way to do that in C++ without a wrapper. Or, at least none that I'm aware of.

This release also includes improved support for Windows and various other odd platforms, as well as various tweaks and fixes.

Full ChangeLog follows:

0.50 - added dictionary::detach and dictionary::getValue(key)
        added xmldom::getNullNode()
        improved xml printing for text nodes
        signalmanager::waitForSignal(NULL) works as expected now
        updated charstring::httpEscape not to escape legal symbols
        added flush() method to stdio classes
        added detection and support for vsnprintf_s
        fixed serviceentry, protocolentry and rpcentry on windows
        updated *entry classes for NULL and uninitialization safety
        implemented filesystem::initialize(fd) for Windows
        updated filesystem::initialize() with NULL and -1 to behave in
                predictable ways now, updated documentation for them too
        abstracted _get_osfhandle calls and added an invalid parameter callback
        updated filedescriptor to use bit fields for all of its flags
        implemented file::changeOwner() for windows
        removed xattrs stuff
        removed SSL-related ifdef's from headers and implemented degenerate
                SSL routines for when SSL isn't supported
        timezone conversion works on windows now
        refactored bytebuffer and stringbuffer to be extent-based
        fixed a subtle stringbuffer with initialcontents bug
        refactored stringbuffer to use bytebuffer::write/appendFormatted
                directly instead of charstring::parseNumber
        implemented dynamicarray and staticarray classes
        refactored build process to use nmake and be compatible with many
                different versions of MS Visual Studio
        updated vsnprintf_s calls for platforms where it's not defined
        added configure test for bool and true/false and definitions for
                platforms that don't define them
        updated timed semaphore waits - if seconds or nanoseconds are negative
                then the timed wait acts like an un-timed wait
        added support for getsockopt with size_t optlen parameter
        fixed strncasecmp on windows when str2 is longer than str1
        xmlsax uses memorymap on Windows now
        fixed file::lockFile() and file::unlockFile() on Windows
        fixed charstring::printf on Solaris 2.5.1
        fixed datetime::getSystemDateAndTime() for Windows without 64-bit
                integers
        fixed potential double-CloseHandle in thread class

SQL Relay 0.58.1 is out

It appears that 0.58 shipped with a bug that caused a few people to scratch their heads for quite a while.

The bug caused sqlr-start to crash, sometimes, under very specific circumstances.

The first is if an instance is defined with no port and no socket (ie. falling back to the default port of 9000). This would fairly reliably crash.

The second is very intermittent. Basically, if you call sqlr-start by itself, with no id, expecting that it will start all instances where enabled="yes" is configured, it will crash. Sometimes. Depending on a lot of random things. It turned out that a few variables were uninitialized, and random crashing a fairly classic behavior in that case.

Both issues are fixed in the 0.58.1 release. So, if you've had problems like those with 0.58, then give 0.58.1 a try.

It can be downloaded here.

ODBC Progress

SQL Relay has long supported ODBC to some extent, both on the server and client side. Ironically though, the client-side has always worked a lot better on Unix/Linux than Windows.

That's just a terrible state of affairs. Windows is where ODBC is most commonly used. Yeah, ADO.NET's the in-thing these days, but there's still a whole world of ODBC apps out there for Windows. You ought to be able to aim them at SQL Relay.

A week or so ago, I got the driver working well enough to be usable if you don't mind manually editing .reg files and loading them using regedit. A good step, but not good enough for the average user. The driver needed a real configuration interface.

Last night I made some good progress though.

I guess I could have drawn a better looking UI using modern tools, but I kind-of want to maximize compatibility, and it didn't seem right to pull in the .NET framework just to be able to configure an ODBC DSN.

It's been a looong time since I did anything with the WinAPI directly, and it was an interesting challenge... "How do I get the background colors to match on the labels again?" "What's the trick to getting tab-navigation to work if your edit controls are attached to a child window?"

But, I got it going, and that plain little window works pretty well.

Apps are working pretty well too.

Little-by-little, they're revealing discrepancies, but it's been a while since anything just didn't work. Things are looking good.

When 0.58 is out, you ought to be able to aim your Windows-based ODBC apps at Relay.

SQL Relay 0.58 is out

SQL Relay version 0.58 is now available!

Windows Support

The most significant feature of this release is preliminary support for the SQL Relay server on Windows.  The client API's and command line programs have long supported Windows, but in this release, it's possible to run the SQL Relay server on Windows as well.

This is a preliminary release,it currently only supports Oracle, and there are some quirks, but it's a good first step.

Running the SQL Relay server on Windows is much the same as on Linux or Unix, just run sqlr-start as described in the documentation.  The biggest quirk is that it doesn't run as a service yet.  You must currently run it from a console window and if you close the console window, it kills the processes.  Other quirks: sqlr-status doesn't work yet and the various loadable modules such as password encryption, authentication and logging don't work yet.  Also, only Oracle databases are currently supported.  You can't run SQL Relay on Windows, against Sybase, DB2, MySQL, PostgreSQL, SQLite, MS Access or MS SQL Server yet.

Building SQL Relay on Windows is tricky and currently requires the right versions of things to be in the right places (as described in the documentation) so if you want to try SQL Relay on Windows, I recommend the SQL Relay Binary Distribution for Windows, available on the downloads page for a small fee.

Also, if you are running a 64-bit OS then use the 64-bit version of SQL Relay.  The 32-bit version doesn't perform well on 64-bit systems.  The issues it has appear to affect other applications as well though, and don't appear to be specific to SQL Relay.


Configuration and Init Improvements

Another significant new feature is support for an sqlrelay.conf.d directory.  As of this release, in addition to the sqlrelay.conf file, all configuration files in the sqlrelay.conf.d directory are also processed.  This makes it possible to keep instance configurations separate.

The init script and sqlr-start program have also been updated.  Rather than running sqlr-start with each instance listed in /etc/sqlrelay or /etc/sysconfig/sqlrelay, the init script now just runs sqlr-start without specifying an instance.  If run without specifying an instance, sqlr-start now digs through the configuration files and starts all instances with enabled="yes" attributes.

Together, these updates make configuration much cleaner, but in particular make it much easier for third party applications to install a configuration and cause an instance to start at boot.


Threaded Listener

The sqlr-listener can run in either multi-process or multi-thread mode and now runs in multi-thread mode by default, which is much faster and lighter weight.


PHP Improvements

PHP ini files are now installed to enable the PHP and PHP PDO modules.  It's no longer necessary (on modern platforms, at least) to manually enable them.


PHP PDO Improvements

The PHP PDO driver features many improvements.

The connect-string now includes resultsetbuffersize, dontgetcolumninfo and nullsasnulls parameters.  The debug connect-string parameter can be set to a file name to send debug information to a file instead of to the browser.

Output bind variables can be bound to streams now.  It is, for example, possible to fetch a blob directly into a file, using an output bind variable.

Samat Yusupov contributed a set of driver-methods for ending, suspending and resuming sessions.  They have been incorporated and improved a little, as described in the documentation.


Perl DBI Improvements

The Perl DBI driver is much improved as well.

Support for the maxrows parameter to fetchall_arrayref has been added.  The begin_work method works now.  The get_info method works properly now.

RowCacheSize and RowsInCache attributes work correctly now.  The entire result set is now buffered by default (as is the case with other SQL Relay client API's) unless the RowCacheSize is set.

A custom ext_SQLR_Debug database handle attribute now allows debug to be turned on or off or sent to a file.

Type, length, precision and scale attributes can now be set for bind variables.  Output bind BLOBs and CLOBs  work now.

The ParamValues, ParamTypes and ParamArrays attributes work correctly now.

The driver works correctly with very old versions of Perl DBI now too.

Lots of fixes.


SAP ASE Support

SAP/Sybase ASE 16.0 is supported now.  The configure script detects it properly.  Version 16.0 revealed a few bugs that have been fixed too.

 
PostgreSQL Improvements

It is now possible to connect to PostgreSQL databases over SSL using the sslmode option.


JMeter  Support

The ODBC driver has been improved sufficiently to work with JMeter via the JDBC-ODBC bridge.  JMeter can now be used to benchmark all tiers of the application directly.  This is helpful in determining whether a performance problem lies in the database, SQL Relay, or the app.


Changes Under the Hood

Internally, quite a few things were restructured.  A lot of tightly coupled code has been decoupled and refactored into frameworks with plugins.  This ought to be invisible to the end user, but should enable much more modular development in the future.
 

Bug Fixes

A slew of bugs have been fixed.

Some are significant:

• Oracle JDK 7 and 8 are now detected correctly on Debian/Ubuntu systems.
• Blobs work when faking input binds now.
• Everything ought to compile cleanly with clang now.
• DB2 output bind BLOBs aren't truncated any more.

Others only occurred in obscure circumstances, but I guess they're still significant if they happened to you:

• sqlrsh doesn't lose the timezone when binding dates any more.
• The PostgreSQL drop-in replacement library had buggy implementations of PQreset, PQresetStart, PQresetPoll.  Those are fixed.
• The MySQL drop-in replacement library had implementations of mysql_row_seek, mysql_row_tell., mysql_stmt_row_seek and mysql_stmt_row_tell that didn't work on platforms with 32-bit void pointers.  Those are fixed.
• Subtle bugs related to keeping track of the total row count in the sybase and freetds connections have been fixed.
• Old versions of Perl (5.00x) are supported now.
• An obscure bug that caused DB2 output bind integers to be copied out incorrectly sometimes has been fixed.
• A very obscure bug that could cause Oracle connections to crash sometime later if an output bind cursor was used to fetch blob data has been fixed.

Rudiments 0.49 is out

Rudiments version 0.49 is now available.

Plenty of good stuff in this release: much improved support for Windows, some refactoring of the signal-handling classes, unification of the passwdentry and shadowentry classes into a single userentry class, improvements when compiling with clang, and a slew of subtle bug fixes.

Full changelog follows:

• combined passswdentry and shadowentry classes into userentry class
• updated file::sync() to use FlushFileBuffers directly
• implemented sys::sync(), reboot(), halt() and shutdown() for windows
• implemented various sys::getXXX() methods for windows
• added "detached" option to process::spawn()
• crash-related methods in process class catch SIGABRT, SIGFPE, SIGILL, SIGBUS, SIGIOT, SIGEMT and SIGSYS in addition to SIGSEGV now
• shutdown-related methods in process class catch SIGQUIT and SIGHUP in addition to SIGINT and SIGTERM now
• on windows, signal classes now support catching, sending/raising SIGABRT, SIGFPE, SIGILL, SIGINT, SIGSEGV and SIGTERM and sending/raising SIGKILL
• disabled threads outright on OSR5
• fixed threads detection for solaris 2.6
• codetree bails properly at end of string now
• fixed a break-related condition that could cause codetree to loop indefinitely
• added support for a beginning-of-line character to codetree
• added support for recursive breaks to codetree
• made charstring::compareIgnoringCase null-safe for platforms that don't have str(n)casecmp
• improved Windows support in signal classes
• -Wno-overloaded-virtual is now set when clang is used
• fixed some subtle kqueue-related bugs
• added sys/byteorder.h and htonll/ntohll for solaris 11
• fixed endian-detection macros for solaris 10 and older
• added support for SIGLARM and signalmanager::alarm() on windows
• improved windows implementation of semaphoreset
• added sys::signalsInterruptSystemCalls()
• added process::supportsFork()

Sudo on Minix 3.1.8

Minix 3.3.0 came out a while ago and I had all kinds of fun playing with it.  I'd long been running 3.2.1 but never gave 3.1.8 a try at all.  The fun I had with 3.3.0 got me motivated though and a few weeks back I gave it a shot.

It was fairly straightforward, for the most part.  A few things here and there reminded me of 2.0 but the overall user experience was similar to 3.2.1.  Getting various bits of software running required many of the same tweaks that 3.3.0 and 3.2.1 required, but the most difficult bit by far was sudo.

Sudo 1.7.something-or-other is available via pkgin but it dumps core whenever you do anything with it.

Sudo version 1.6.9p23 has been reliably portable for me though, so I gave it a try.  It took a while to work around various issues, but eventually I got it working.  If anyone out there is interested, here are the tricks:

The code requires a little hackery.

Add the following line after the includes in interfaces.c

#define IFF_LOOPBACK IFF_UP

Edit sudo.tab.c and comment out:

# include <search.h>

it should look like this when you're done:

/*# include <search.h>*/

Minix 3.1.8's implementation of getgroups always returns -1.  This causes a few problems for sudo.  To solve them, edit set_perms.c and make 3 modifications.

Make the declaration of runas_groups (around line 376) look like:

static GETGROUPS_T *runas_groups = NULL;

Update a section of the function runas_setgroups() as follows:

...
    if (runas_ngroups <= 0) {
        pw = runas_pw ? runas_pw : sudo_user.pw;
        if (initgroups(pw->pw_name, pw->pw_gid) < 0)
            log_error(USE_ERRNO|MSG_ONLY, "can't set runas group vector");

        if ((runas_ngroups = getgroups(0, NULL)) > 0) {
            runas_groups = emalloc2(runas_ngroups, sizeof(GETGROUPS_T));
            if (getgroups(runas_ngroups, runas_groups) < 0)
                log_error(USE_ERRNO|MSG_ONLY, "can't get runas group vector");
        }

    } else {
...

Update the function restore_groups() as follows:

static void
restore_groups()
{
    if (user_ngroups > 0) {
        if (setgroups(user_ngroups, user_groups) < 0)
            log_error(USE_ERRNO|MSG_ONLY, "can't reset user group vector");
    }
}

That third modification is actually a workaround for a bug (or feature) in setgroups().  It goes completely haywire if user_ngroups is -1.  I'll bet that this is the problem that the version of sudo available via pkgin has too.  There might be a libc update that fixes this issue but for now we'll just work around it.

Now build and install.  Some options needs to be enabled in the header files with the _POSIX_SOURCE macro, so we'll pass that in.  I also like to install it under it's own directory tree, so I use the --prefix option for that.

CPPFLAGS="-D_POSIX_SOURCE" ./configure --prefix=/usr/local/sudo-1.6.9p23
make
su
make install

The default permissions are wrong for Minix though, so they need to be set manually:

chmod 555 /usr/local/sudo-1.6.9p23/bin/sudo
chmod u+s /usr/local/sudo-1.6.9p23/bin/sudo

Now add a line for your user (dmuse in my case) to /etc/sudoers.

dmuse ALL=(ALL) NOPASSWD: ALL

Yes, the sudoers file is installed under /etc independent of the --prefix option used earlier.  The NOPASSWD option makes it so you don't have to enter a password.

And the last step is to add /usr/local/sudo-1.6.9p23/bin to your PATH in ~/.ashrc

export PATH=$HOME/bin:/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin:\
/usr/pkg/X11R6/bin:/usr/pkg/bin:/usr/pkg/sbin:/usr/llvm/bin:\
/usr/local/sudo-1.6.9p23/bin

And that's it.  Re-login and sudo should work as expected.

$ sudo ls -l /
total 44
drwxr-xr-x  2 root  operator   2880 Sep  4  2010 bin
drwxr-xr-x  3 root  operator    384 Sep  4  2010 boot
drwxr-xr-x  2 root  operator  29120 Sep 20 02:47 dev
drwxr-xr-x  3 root  operator   2304 Nov 20 22:53 etc
drwxrwxrwx  5 bin   bin         320 Sep 20 02:55 home
drwxrwxrwx  2 root  operator    128 Sep 20 02:45 mnt
drwx------  2 root  operator    768 Sep 20 04:09 root
drwxr-xr-x  2 root  operator    896 Sep  4  2010 sbin
drwxrwxrwt  2 root  operator    640 Nov 20 23:52 tmp
drwxrwxrwx 23 bin   bin        1536 Sep 20 03:27 usr
drwxr-xr-x  3 root  operator    256 Sep 20 02:45 var

Woohoo!  Sudo on Minix 3.1.8.

Windows Progress...

SQL Relay has classically run on Unix and Linux but not Windows. A few years ago I got the C/C++ API working on Windows and added a C# API and .NET data provider. I got a few of the command-line programs working too, but that was about all you could do on Windows for quite a while. A few months ago when I got the Perl, Python and PHP API's working on Windows along with the rest of the command line clients, but still, only the client side software worked.

Since then I've made quite a bit of progress getting the server working too. And tonight things really started to come together...



That's the SQL Relay server running on Windows, connected to a database running on Linux, accessed by sqlrsh, also running on Windows.

And it can be accessed remotely too.



That's the same SQL Relay server running on Windows, accessed by sqlrsh running in a separate Linux VM.

There are a number of things that don't work yet like sqlr-stop and various plugins, and there are several things that just haven't been tested yet, but the big things are coming along nicely. I expect to be able to announce preliminary support for the SQL Relay server software on Windows in the next release.