Friday, April 24, 2015

SQL Relay 0.59 is out

SQL Relay version 0.59 is now available!

Significant New Features

There are a few...

Improved Support for Windows

This is the main new feature in this release.

Support for Ruby and TCL have been added. Now, on Windows, you can develop in C, C++, any .NET-supported language, Java, Perl, Python, PHP, Ruby and TCL.

Support for many additional databases has been added. SQL Relay can now be run on Windows, against Oracle, Sybase/SAP, DB2, Firebird, Postgresql, Mysql and ODBC-supported databases. The query router is also supported now too.

Performance isn't as good on Windows as it is on Unix/Linux yet, but it works and it appears to be stable.

On the development side, the build process has been completely refactored to use nmake and now supports a wide variety of versions of Visual Studio. It has been tested with VS2005, 2008, 2010, 2012 and 2013. VS6 and VS5 mostly work, but not entirely, yet. The build takes infinitely less time and it's now possible to fix something and nmake and nmake install from within the directory of the modified file, like one would be accustomed to on Unix/Linux. The Windows build process isn't yet as capable as the Unix/Linux build process, but it is greatly improved. See Installation on Windows for more information.

ODBC Improvements

Arguably this falls under Improved Support for Windows, but technically some of these improvements apply to UnixODBC/iODBC as well.

ODBC has long worked on Windows, but configuring it required manual modifications to the registry. No longer! The ODBC driver manager dialog exists now. On Windows, it should now be as easy to configure an ODBC connection to SQL Relay as it is to any other database.

The driver supports getting and setting many more attributes, and SQLTables works now. This enables many, many more ODBC applications to work with SQL Relay. It's been tested against a long list of apps with good success. If you've had trouble with it in the past, try this version.

MySQL Improvements

The MySQL database connection configuration now supports the maxselectlistsize and maxitembuffersize parameters, similar to Oracle, Sybase, DB2 and ODBC. This enables tuning of memory usage at runtime. See Tuning SQL Relay - Memory Usage for details.

LOB support is much improved with MySQL now too. In the past, the maximum number of bytes that could be fetched from a LOB column was limited by a macro in the source code - the same macro that defined the maximum size of a non-lob column. LOB's are now fetched in chunks and the amount of data that can be fetched is unlimited.

Support for Mono

SQL Relay has long had a C# API and ADO.NET driver, but they were classically only supported on Windows. These API's now work on Unix and Linux platforms as well, using Mono.

Less Significant, But Still Interesting New Features

The number of bind variables per-query was originally limited by a macro in the source code. Now, on the client-side, the number of bind variables is unlimited. Well, it's only limited by memory. On the server side, there's a new maxbindvars parameter in the sqlrelay.conf file that can be used to limit the number of bind variables that the server will accept. maxbindvars defaults to 256, but if you have an application that needs to send 1000 bind variables, you can just set it to 1000.

Login warnings are now printed on the screen, but not interpreted as errors. For example, if the database password is going to expire soon, but hasn't yet expired, the warning is displayed, but no longer interpreted as an error.

The sqlrsh command line client now supports the "cache" and "opencache" commands to cache result sets to a file and fetch result sets from a file. Not sure how many people actually use that feature, but sqlrsh supports fiddling with it now.

The logger modules have an enabled parameter which can be set to "yes" or "no". It's no longer necessary to comment out or remove the logger configuration to disable logging, you can just set enabled="no". See Logger Modules for more information.

Firebird recently added support for global temporary tables. Their implementation is very similar to Oracle's. SQL Relay now manages global temporary tables identically for Oracle and Firebird. There is even a droptemptables parameter in the Firebird connection configuration, analogous to the same parameter for Oracle.

In fact, support for global temporary tables has been generally improved. In the past, tables created during the current SQL Relay session were tracked and truncated at the end of the session, but global temporary tables created during previous runs of SQL Relay or outside of SQL Relay altogether were ignored, leading to those global temporary tables not being truncated at the end of the session. The Oracle and Firebird connections now support a globaltemptables parameter that can be set to a list of tables that should be truncated at the end of each session, or to "%", which causes SQL Relay to query the database for the list of global temp tables to truncate.

The Ruby DBI driver has been retired. I guess that's not a new feature, but it's something that was done in this release. Ruby DBI seems to have fallen out of favor and the driver was becoming increasingly difficult to maintain.

Tweaks and Fixes

Lots of small things have been tweaked, fixed, or improved as well

  • Binding a NULL to an integer works with db2 now.
  • Added a semaphore to ensure that the listener doesn't hand off the client to the connection until the connection is ready, eliminating a race condition on the handoff socket that could occur if the connection timed out waiting for the listener just after the listener had decided to use that connection.
  • Oracle temp tables that need to be truncated at the end of the session are truncated with "truncate table xxx" now rather than "delete from xxx".
  • Oracle temp tables that need to be dropped at the end of the session are truncated first, rather than the connection re-logging in.
  • An ora-14452 error (basically indicating that a temp table can only be dropped after being truncated, or if the current session ends) does not automatically trigger a re-login any more.
  • Updated cachemanager to use directory::read() directly instead of directory::getChildName(index).
  • Made cache ttl a 64-bit number.
  • Updated odbc connection code to use new/delete and rudiments methods rather than malloc/free and native calls.
  • Fixed command line client crash when using -id "instance" with an instance that uses authtier="database".
  • Fixed bugs that could make reexecuted db2 selects fail and cause a database re-login loop.
  • Tweaked spec file to remove empty directories on uninstall.
  • Fixed typo that could sometimes cause a listener crash.
  • Postgresql and MDBTools return error code of 1 rather than 0 for all errors now.
  • Fixed bugs related to autocommit with db's that support transaction blocks.
  • Fixed various bugs in sqlrconfigfile that caused sqlr-start with no -id to crash or behave strangely sometimes.
  • Updated the slow query logger to show the date/time that the query was executed.
  • Consolidated c, c++ and server source/includes down a few levels.
  • Fixed a misspelling in sqlrelay.dtd.
  • Swapped order of init directory detection, looking for /etc/init.d ahead of /etc/rc.d/init.d to resolve conflict with dkms on SuSE Enterprise.
  • sqlr-start spawns a new window on Windows now.
  • updated mysql connection to allow mysql_init to allocate a mysql struct on platforms that support mysql_init, rather than using a static struct.
  • Fixed subtle noon/midnight-related bugs in date/time translation.
  • Updated mysql connection to get affected rows when not using the statement api.
  • Updated mysql connection not to use the statement API on windows, for now.
  • Disabled mysql_change_user, for now.
  • Fixed blob-input binds on firebird.

Give it a try, report any issues!

Rudiments 0.50 is out

Rudiments 0.50 is now available!

The most significant feature in this release is a refactored, extent-based implementation of the variablebuffer and stringbuffer classes. The original implementation was crude, sufficient for storing fairly short strings, small in code size, but scaled poorly. The new implementation scales well while the code size still remains small.

There are also new dynamicarray and staticarray classes. Dynamicarray implements an array that expands as needed. It's also extent-based and scales well. Staticarry is just a thin wrapper around a C array. It only exists because you might want a dynamicarray who's members are static arrays, and there's no good way to do that in C++ without a wrapper. Or, at least none that I'm aware of.

This release also includes improved support for Windows and various other odd platforms, as well as various tweaks and fixes.

Full ChangeLog follows:

0.50 - added dictionary::detach and dictionary::getValue(key)
        added xmldom::getNullNode()
        improved xml printing for text nodes
        signalmanager::waitForSignal(NULL) works as expected now
        updated charstring::httpEscape not to escape legal symbols
        added flush() method to stdio classes
        added detection and support for vsnprintf_s
        fixed serviceentry, protocolentry and rpcentry on windows
        updated *entry classes for NULL and uninitialization safety
        implemented filesystem::initialize(fd) for Windows
        updated filesystem::initialize() with NULL and -1 to behave in
                predictable ways now, updated documentation for them too
        abstracted _get_osfhandle calls and added an invalid parameter callback
        updated filedescriptor to use bit fields for all of its flags
        implemented file::changeOwner() for windows
        removed xattrs stuff
        removed SSL-related ifdef's from headers and implemented degenerate
                SSL routines for when SSL isn't supported
        timezone conversion works on windows now
        refactored bytebuffer and stringbuffer to be extent-based
        fixed a subtle stringbuffer with initialcontents bug
        refactored stringbuffer to use bytebuffer::write/appendFormatted
                directly instead of charstring::parseNumber
        implemented dynamicarray and staticarray classes
        refactored build process to use nmake and be compatible with many
                different versions of MS Visual Studio
        updated vsnprintf_s calls for platforms where it's not defined
        added configure test for bool and true/false and definitions for
                platforms that don't define them
        updated timed semaphore waits - if seconds or nanoseconds are negative
                then the timed wait acts like an un-timed wait
        added support for getsockopt with size_t optlen parameter
        fixed strncasecmp on windows when str2 is longer than str1
        xmlsax uses memorymap on Windows now
        fixed file::lockFile() and file::unlockFile() on Windows
        fixed charstring::printf on Solaris 2.5.1
        fixed datetime::getSystemDateAndTime() for Windows without 64-bit
                integers
        fixed potential double-CloseHandle in thread class

Wednesday, April 1, 2015

SQL Relay 0.58.1 is out

It appears that 0.58 shipped with a bug that caused a few people to scratch their heads for quite a while.

The bug caused sqlr-start to crash, sometimes, under very specific circumstances.

The first is if an instance is defined with no port and no socket (ie. falling back to the default port of 9000). This would fairly reliably crash.

The second is very intermittent. Basically, if you call sqlr-start by itself, with no id, expecting that it will start all instances where enabled="yes" is configured, it will crash. Sometimes. Depending on a lot of random things. It turned out that a few variables were uninitialized, and random crashing a fairly classic behavior in that case.

Both issues are fixed in the 0.58.1 release. So, if you've had problems like those with 0.58, then give 0.58.1 a try.

It can be downloaded here.

Thursday, March 12, 2015

ODBC Progress

SQL Relay has long supported ODBC to some extent, both on the server and client side. Ironically though, the client-side has always worked a lot better on Unix/Linux than Windows.

That's just a terrible state of affairs. Windows is where ODBC is most commonly used. Yeah, ADO.NET's the in-thing these days, but there's still a whole world of ODBC apps out there for Windows. You ought to be able to aim them at SQL Relay.

A week or so ago, I got the driver working well enough to be usable if you don't mind manually editing .reg files and loading them using regedit. A good step, but not good enough for the average user. The driver needed a real configuration interface.

Last night I made some good progress though.

I guess I could have drawn a better looking UI using modern tools, but I kind-of want to maximize compatibility, and it didn't seem right to pull in the .NET framework just to be able to configure an ODBC DSN.

It's been a looong time since I did anything with the WinAPI directly, and it was an interesting challenge... "How do I get the background colors to match on the labels again?" "What's the trick to getting tab-navigation to work if your edit controls are attached to a child window?"

But, I got it going, and that plain little window works pretty well.

Apps are working pretty well too.

Little-by-little, they're revealing discrepancies, but it's been a while since anything just didn't work. Things are looking good.

When 0.58 is out, you ought to be able to aim your Windows-based ODBC apps at Relay.

Friday, January 2, 2015

Weather Cache - 1.0

Announcing firstworks Weather Cache for Linux version 1.0 !

What is a Weather Cache?

The firstworks Weather Cache is a light, simple, caching proxy, targeted for use with Weather API's.



Weather API's provide web-services that allow you to display current conditions, forecasts or historical weather information on your website or mobile app.  For example:





But there are several issues with these innocuous-looking badges.

Downloading weather data from across the internet takes time. Fetching it on the server-side while building a page can slow page loads significantly.

You could update your site so that the data is fetched on the client-side after the rest of the page has loaded but that introduces Cross-Site Scripting (XSS) issues.  Yes, you can work around these issues with JSONP and CORS, but only if the Weather API supports them.

But the bigger issue, the one that crops up when your site starts getting popular, is request count limits.  When you exceed some number of requests during some period of time you get throttled, blocked, or have to pay more.

The firstworks Weather Cache sits between your application and the Weather API's web-service, forwarding requests from your application and receiving results from the web-service. Each result is cached in a local database and tagged with an expiration date before being returned to your application. If the cache receives a request for the same data prior to the expiration date, the result is fetched from the database, rather than from the web-service.
  • When results are fetched from the local database rather than from across the internet, page load times are improved dramatically.
  • Since duplicate requests are resolved locally, the number of requests to the Weather API are minimized.
  • Cross-Site Scripting issues are avoided because the domain of the cache is the same as the rest of your site.
There are other caching technologies out there, but they are complex and subject to a few Weather-API-specific drawbacks.

The firstworks Weather Cache is extremely simple.  It's distributed as a self-contained, statically-linked x86 Linux binary and should be compatible with any x86 or x86_64 Linux platform. Its only requirements are a web server configured to support CGI's and read/write access to the /var/tmp directory.  Simply install it and aim it at the Weather API of your choice:

Before:
http://api.wservice.com/current.asp?q=Atlanta,ga&mode=json&apikey=a1b2c3d4e5

After:
http://mydomain.com/weathercache.cgi/api.wservice.com/current.asp?q=Atlanta,ga&mode=json&apikey=a1b2c3d4e5
 
The Weather Cache is designed to be API-agnostic but it does a few semi-intelligent things to work around some common cache-defeating issues.
  • Parameters are normalized so subtly different requests for the same information will be recognized as the same request.
  • Mobile devices are "snapped" to a grid. Successive requests from mobile devices tend to send slightly different latitudes and longitudes, especially if the device is moving. When snapped to a grid, the location of the device only changes when it has has moved far enough to snap to a different grid point. Multiple mobile devices in close proximity snap to the same grid point.
 If your website or mobile app displays weather information, it could probably benefit from the firstworks Weather Cache.  Check it out.

Sunday, December 28, 2014

SQL Relay 0.58 is out

SQL Relay version 0.58 is now available!

Windows Support

The most significant feature of this release is preliminary support for the SQL Relay server on Windows.  The client API's and command line programs have long supported Windows, but in this release, it's possible to run the SQL Relay server on Windows as well.

This is a preliminary release,it currently only supports Oracle, and there are some quirks, but it's a good first step.

Running the SQL Relay server on Windows is much the same as on Linux or Unix, just run sqlr-start as described in the documentation.  The biggest quirk is that it doesn't run as a service yet.  You must currently run it from a console window and if you close the console window, it kills the processes.  Other quirks: sqlr-status doesn't work yet and the various loadable modules such as password encryption, authentication and logging don't work yet.  Also, only Oracle databases are currently supported.  You can't run SQL Relay on Windows, against Sybase, DB2, MySQL, PostgreSQL, SQLite, MS Access or MS SQL Server yet.

Building SQL Relay on Windows is tricky and currently requires the right versions of things to be in the right places (as described in the documentation) so if you want to try SQL Relay on Windows, I recommend the SQL Relay Binary Distribution for Windows, available on the downloads page for a small fee.

Also, if you are running a 64-bit OS then use the 64-bit version of SQL Relay.  The 32-bit version doesn't perform well on 64-bit systems.  The issues it has appear to affect other applications as well though, and don't appear to be specific to SQL Relay.


Configuration and Init Improvements

Another significant new feature is support for an sqlrelay.conf.d directory.  As of this release, in addition to the sqlrelay.conf file, all configuration files in the sqlrelay.conf.d directory are also processed.  This makes it possible to keep instance configurations separate.

The init script and sqlr-start program have also been updated.  Rather than running sqlr-start with each instance listed in /etc/sqlrelay or /etc/sysconfig/sqlrelay, the init script now just runs sqlr-start without specifying an instance.  If run without specifying an instance, sqlr-start now digs through the configuration files and starts all instances with enabled="yes" attributes.

Together, these updates make configuration much cleaner, but in particular make it much easier for third party applications to install a configuration and cause an instance to start at boot.


Threaded Listener

The sqlr-listener can run in either multi-process or multi-thread mode and now runs in multi-thread mode by default, which is much faster and lighter weight.


PHP Improvements

PHP ini files are now installed to enable the PHP and PHP PDO modules.  It's no longer necessary (on modern platforms, at least) to manually enable them.


PHP PDO Improvements

The PHP PDO driver features many improvements.

The connect-string now includes resultsetbuffersize, dontgetcolumninfo and nullsasnulls parameters.  The debug connect-string parameter can be set to a file name to send debug information to a file instead of to the browser.

Output bind variables can be bound to streams now.  It is, for example, possible to fetch a blob directly into a file, using an output bind variable.

Samat Yusupov contributed a set of driver-methods for ending, suspending and resuming sessions.  They have been incorporated and improved a little, as described in the documentation.


Perl DBI Improvements

The Perl DBI driver is much improved as well.

Support for the maxrows parameter to fetchall_arrayref has been added.  The begin_work method works now.  The get_info method works properly now.

RowCacheSize and RowsInCache attributes work correctly now.  The entire result set is now buffered by default (as is the case with other SQL Relay client API's) unless the RowCacheSize is set.

A custom ext_SQLR_Debug database handle attribute now allows debug to be turned on or off or sent to a file.

Type, length, precision and scale attributes can now be set for bind variables.  Output bind BLOBs and CLOBs  work now.

The ParamValues, ParamTypes and ParamArrays attributes work correctly now.

The driver works correctly with very old versions of Perl DBI now too.

Lots of fixes.


SAP ASE Support

SAP/Sybase ASE 16.0 is supported now.  The configure script detects it properly.  Version 16.0 revealed a few bugs that have been fixed too.

 
PostgreSQL Improvements

It is now possible to connect to PostgreSQL databases over SSL using the sslmode option.


JMeter  Support

The ODBC driver has been improved sufficiently to work with JMeter via the JDBC-ODBC bridge.  JMeter can now be used to benchmark all tiers of the application directly.  This is helpful in determining whether a performance problem lies in the database, SQL Relay, or the app.


Changes Under the Hood

Internally, quite a few things were restructured.  A lot of tightly coupled code has been decoupled and refactored into frameworks with plugins.  This ought to be invisible to the end user, but should enable much more modular development in the future.
 

Bug Fixes

A slew of bugs have been fixed.

Some are significant:
  • Oracle JDK 7 and 8 are now detected correctly on Debian/Ubuntu systems.
  • Blobs work when faking input binds now.
  • Everything ought to compile cleanly with clang now.
  • DB2 output bind BLOBs aren't truncated any more.
Others only occurred in obscure circumstances, but I guess they're still significant if they happened to you:
  • sqlrsh doesn't lose the timezone when binding dates any more.
  • The PostgreSQL drop-in replacement library had buggy implementations of PQreset, PQresetStart, PQresetPoll.  Those are fixed.
  • The MySQL drop-in replacement library had implementations of mysql_row_seek, mysql_row_tell., mysql_stmt_row_seek and mysql_stmt_row_tell that didn't work on platforms with 32-bit void pointers.  Those are fixed.
  • Subtle bugs related to keeping track of the total row count in the sybase and freetds connections have been fixed.
  • Old versions of Perl (5.00x) are supported now.
  • An obscure bug that caused DB2 output bind integers to be copied out incorrectly sometimes has been fixed.
  • A very obscure bug that could cause Oracle connections to crash sometime later if an output bind cursor was used to fetch blob data has been fixed.

Rudiments 0.49 is out

Rudiments version 0.49 is now available.

Plenty of good stuff in this release: much improved support for Windows, some refactoring of the signal-handling classes, unification of the passwdentry and shadowentry classes into a single userentry class, improvements when compiling with clang, and a slew of subtle bug fixes.

Full changelog follows:
  • combined passswdentry and shadowentry classes into userentry class
  • updated file::sync() to use FlushFileBuffers directly
  • implemented sys::sync(), reboot(), halt() and shutdown() for windows
  • implemented various sys::getXXX() methods for windows
  • added "detached" option to process::spawn()
  • crash-related methods in process class catch SIGABRT, SIGFPE, SIGILL, SIGBUS, SIGIOT, SIGEMT and SIGSYS in addition to SIGSEGV now
  • shutdown-related methods in process class catch SIGQUIT and SIGHUP in addition to SIGINT and SIGTERM now
  • on windows, signal classes now support catching, sending/raising SIGABRT, SIGFPE, SIGILL, SIGINT, SIGSEGV and SIGTERM and sending/raising SIGKILL
  • disabled threads outright on OSR5
  • fixed threads detection for solaris 2.6
  • codetree bails properly at end of string now
  • fixed a break-related condition that could cause codetree to loop indefinitely
  • added support for a beginning-of-line character to codetree
  • added support for recursive breaks to codetree
  • made charstring::compareIgnoringCase null-safe for platforms that don't have str(n)casecmp
  • improved Windows support in signal classes
  • -Wno-overloaded-virtual is now set when clang is used
  • fixed some subtle kqueue-related bugs
  • added sys/byteorder.h and htonll/ntohll for solaris 11
  • fixed endian-detection macros for solaris 10 and older
  • added support for SIGLARM and signalmanager::alarm() on windows
  • improved windows implementation of semaphoreset
  • added sys::signalsInterruptSystemCalls()
  • added process::supportsFork()