Saturday, June 20, 2015

SQL Relay 0.60 is out

This release fixes various obscure (though potentially crippling) bugs and shortcomings, and issues with specific compilers.

The most notable fix is, on Windows, that you can now successfully specify an installation path when using the installer, and it will actually work. In previous releases, you could specify a path, and everything would be installed there, but it was largely ignored by the applications, which looked for files in and tried to write files to the default path.

Full ChangeLog follows...

  • fixed true->false transposition in sqlrservercontroller::interceptQuery that could lead to a reLogIn loop
  • disabled -Werror for gcc < 2.7
  • applied George Carrette's patch to fix PDO connectstring options
  • migrated directory/file paths info into sqlrpaths class
  • removed undocumented and not-so-relevent-these-days MAX_CONNECTIONS/overridemaxconnections failsafe in sqlr-start
  • fixed java header detection for javac located in /usr/bin
  • updated java api to support non-null-tolerant implementations of environment::NewStringUTF()
  • the perl api builds correctly on OSR5 again
  • the postgresql sslmode is omitted entirely from the connect string now, if it's disabled, to prevent problems with older versions of postgresql that don't support the parameter at all
  • mono 2.8 or greater is required now
  • added a datedelimiters attribute to the instance tag in sqlrelay.conf to limit what date delimiters are used when translating dates
  • re-added mssqlserver detection, which had been accidentally removed
  • fixed statically-linked build
  • fixed mysql lob field bug that could cause a crash

Rudiments 0.51 is out

The 0.51 release of rudiments is largely a clean-up release. Regressions, inconsistencies, and issues with specific compilers have been addressed. Work on support for OpenVMS has begun as well.

Full ChangeLog follows...

  • added missing inline qualifiers to dynamicarray/staticarray method implementations
  • added test for clock_settime, datetime class uses it if other set-time functions aren't available, returns false and sets errno=ENOSYS if no set-time function is available
  • directory::fpathConf now falls back to MAX_PATH code if fpathconf exists but there's dirfd() or anything like it
  • file::createFifo returns false and sets errno=ENOSYS on platforms that don't support fifo's now
  • file::generateKey returns -1 and sets errno=ENOSYS on VMS now
  • file::createHardLink returns -1 and sets errno=ENOSYS on platforms don't support hard links now
  • fixed missing breaks in stdio::flush
  • userentry::getPassword/getRealName return NULL on VMS now
  • unix sockets are faked on VMS now
  • threadmutex is built and installed now whether or not threads are supported, but if threads are not supported, its methods are degenerate and return success
  • filesystem::getCurrentProperties() returns false and sets errno=ENOSYS on VMS now
  • implemented degenerate groupentry for VMS
  • fixed file::getBlockCount() crash that could occur if the filesystem's block size was reported as 0
  • tweaked copy constructors and = operators in staticarray/dynamicarray classes to work with older compilers
  • disabled -Werror for gcc < 2.7
  • filedescriptor::printf uses fprintf or fdopen/vfprintf when possible, in leiu of falling back to charstring::printf
  • charstring::printf tries the null device before falling back to a scratch file
  • the rate at which charstring::printf expands its buffer is now exponential up to 1024 bytes
  • added sys::getDirectorySeparator()
  • fixed trailing-slash-followed-by-quote issue when building windows command lines
  • added --enable-enosys-notify configure option for runtime notifcations when methods set ENOSYS
  • cleared errno before each EINTR loop
  • fixed sys::sync() segfault
  • fixed ssl tests
  • removed intervaltimer class
  • updated sys::getPhysicalPageCount/getAvailablePhysicalPageCount for Windows
  • added sys::getAllocationGranularity, updated code to use it instead of getPageSize, in most cases
  • fixed copy-constructor/=-operator bugs in dynamicarray class
  • added config_vs2013.h for VS 2013
  • fixed several permissions-related issues for WinNT 4
  • allowShortReads/useBlockingMode are set by default for stdinput now
  • added wrapper for atexit()
  • changed process::getRealUser/GroupId to getUser/GroupId to match setUser/Groupid
  • added process::setEffectiveUser/Group methods
  • fixed and documented auto-resume behavior in snooze methods
  • fixed detection of double-dashed command line parameters without values
  • removed redundant charstring::rightPad()
  • renamed charstring::padString() to charstring::pad()
  • added safePrint methods that take unsigned char arguments
  • fixed bytestring::findFirst() with unsigned char needle, when needle is 0
  • fixed bugs in linkedlist::moveBefore/moveAfter
  • fixed bugs in singlylinkedlist::removeAll()
  • removed arg parameter from thread::setFunction(), added thread::setArgument(), and added thread::create(arg)
  • changed thread::create() to thread::run()
  • wrote lots of example code and updated the programming docs
  • removed unused shmfile class
  • fixed several LocalFree's that should have been delete[]'s

Friday, April 24, 2015

SQL Relay 0.59 is out

SQL Relay version 0.59 is now available!

Significant New Features

There are a few...

Improved Support for Windows

This is the main new feature in this release.

Support for Ruby and TCL have been added. Now, on Windows, you can develop in C, C++, any .NET-supported language, Java, Perl, Python, PHP, Ruby and TCL.

Support for many additional databases has been added. SQL Relay can now be run on Windows, against Oracle, Sybase/SAP, DB2, Firebird, Postgresql, Mysql and ODBC-supported databases. The query router is also supported now too.

Performance isn't as good on Windows as it is on Unix/Linux yet, but it works and it appears to be stable.

On the development side, the build process has been completely refactored to use nmake and now supports a wide variety of versions of Visual Studio. It has been tested with VS2005, 2008, 2010, 2012 and 2013. VS6 and VS5 mostly work, but not entirely, yet. The build takes infinitely less time and it's now possible to fix something and nmake and nmake install from within the directory of the modified file, like one would be accustomed to on Unix/Linux. The Windows build process isn't yet as capable as the Unix/Linux build process, but it is greatly improved. See Installation on Windows for more information.

ODBC Improvements

Arguably this falls under Improved Support for Windows, but technically some of these improvements apply to UnixODBC/iODBC as well.

ODBC has long worked on Windows, but configuring it required manual modifications to the registry. No longer! The ODBC driver manager dialog exists now. On Windows, it should now be as easy to configure an ODBC connection to SQL Relay as it is to any other database.

The driver supports getting and setting many more attributes, and SQLTables works now. This enables many, many more ODBC applications to work with SQL Relay. It's been tested against a long list of apps with good success. If you've had trouble with it in the past, try this version.

MySQL Improvements

The MySQL database connection configuration now supports the maxselectlistsize and maxitembuffersize parameters, similar to Oracle, Sybase, DB2 and ODBC. This enables tuning of memory usage at runtime. See Tuning SQL Relay - Memory Usage for details.

LOB support is much improved with MySQL now too. In the past, the maximum number of bytes that could be fetched from a LOB column was limited by a macro in the source code - the same macro that defined the maximum size of a non-lob column. LOB's are now fetched in chunks and the amount of data that can be fetched is unlimited.

Support for Mono

SQL Relay has long had a C# API and ADO.NET driver, but they were classically only supported on Windows. These API's now work on Unix and Linux platforms as well, using Mono.

Less Significant, But Still Interesting New Features

The number of bind variables per-query was originally limited by a macro in the source code. Now, on the client-side, the number of bind variables is unlimited. Well, it's only limited by memory. On the server side, there's a new maxbindvars parameter in the sqlrelay.conf file that can be used to limit the number of bind variables that the server will accept. maxbindvars defaults to 256, but if you have an application that needs to send 1000 bind variables, you can just set it to 1000.

Login warnings are now printed on the screen, but not interpreted as errors. For example, if the database password is going to expire soon, but hasn't yet expired, the warning is displayed, but no longer interpreted as an error.

The sqlrsh command line client now supports the "cache" and "opencache" commands to cache result sets to a file and fetch result sets from a file. Not sure how many people actually use that feature, but sqlrsh supports fiddling with it now.

The logger modules have an enabled parameter which can be set to "yes" or "no". It's no longer necessary to comment out or remove the logger configuration to disable logging, you can just set enabled="no". See Logger Modules for more information.

Firebird recently added support for global temporary tables. Their implementation is very similar to Oracle's. SQL Relay now manages global temporary tables identically for Oracle and Firebird. There is even a droptemptables parameter in the Firebird connection configuration, analogous to the same parameter for Oracle.

In fact, support for global temporary tables has been generally improved. In the past, tables created during the current SQL Relay session were tracked and truncated at the end of the session, but global temporary tables created during previous runs of SQL Relay or outside of SQL Relay altogether were ignored, leading to those global temporary tables not being truncated at the end of the session. The Oracle and Firebird connections now support a globaltemptables parameter that can be set to a list of tables that should be truncated at the end of each session, or to "%", which causes SQL Relay to query the database for the list of global temp tables to truncate.

The Ruby DBI driver has been retired. I guess that's not a new feature, but it's something that was done in this release. Ruby DBI seems to have fallen out of favor and the driver was becoming increasingly difficult to maintain.

Tweaks and Fixes

Lots of small things have been tweaked, fixed, or improved as well

  • Binding a NULL to an integer works with db2 now.
  • Added a semaphore to ensure that the listener doesn't hand off the client to the connection until the connection is ready, eliminating a race condition on the handoff socket that could occur if the connection timed out waiting for the listener just after the listener had decided to use that connection.
  • Oracle temp tables that need to be truncated at the end of the session are truncated with "truncate table xxx" now rather than "delete from xxx".
  • Oracle temp tables that need to be dropped at the end of the session are truncated first, rather than the connection re-logging in.
  • An ora-14452 error (basically indicating that a temp table can only be dropped after being truncated, or if the current session ends) does not automatically trigger a re-login any more.
  • Updated cachemanager to use directory::read() directly instead of directory::getChildName(index).
  • Made cache ttl a 64-bit number.
  • Updated odbc connection code to use new/delete and rudiments methods rather than malloc/free and native calls.
  • Fixed command line client crash when using -id "instance" with an instance that uses authtier="database".
  • Fixed bugs that could make reexecuted db2 selects fail and cause a database re-login loop.
  • Tweaked spec file to remove empty directories on uninstall.
  • Fixed typo that could sometimes cause a listener crash.
  • Postgresql and MDBTools return error code of 1 rather than 0 for all errors now.
  • Fixed bugs related to autocommit with db's that support transaction blocks.
  • Fixed various bugs in sqlrconfigfile that caused sqlr-start with no -id to crash or behave strangely sometimes.
  • Updated the slow query logger to show the date/time that the query was executed.
  • Consolidated c, c++ and server source/includes down a few levels.
  • Fixed a misspelling in sqlrelay.dtd.
  • Swapped order of init directory detection, looking for /etc/init.d ahead of /etc/rc.d/init.d to resolve conflict with dkms on SuSE Enterprise.
  • sqlr-start spawns a new window on Windows now.
  • updated mysql connection to allow mysql_init to allocate a mysql struct on platforms that support mysql_init, rather than using a static struct.
  • Fixed subtle noon/midnight-related bugs in date/time translation.
  • Updated mysql connection to get affected rows when not using the statement api.
  • Updated mysql connection not to use the statement API on windows, for now.
  • Disabled mysql_change_user, for now.
  • Fixed blob-input binds on firebird.

Give it a try, report any issues!

Rudiments 0.50 is out

Rudiments 0.50 is now available!

The most significant feature in this release is a refactored, extent-based implementation of the variablebuffer and stringbuffer classes. The original implementation was crude, sufficient for storing fairly short strings, small in code size, but scaled poorly. The new implementation scales well while the code size still remains small.

There are also new dynamicarray and staticarray classes. Dynamicarray implements an array that expands as needed. It's also extent-based and scales well. Staticarry is just a thin wrapper around a C array. It only exists because you might want a dynamicarray who's members are static arrays, and there's no good way to do that in C++ without a wrapper. Or, at least none that I'm aware of.

This release also includes improved support for Windows and various other odd platforms, as well as various tweaks and fixes.

Full ChangeLog follows:

0.50 - added dictionary::detach and dictionary::getValue(key)
        added xmldom::getNullNode()
        improved xml printing for text nodes
        signalmanager::waitForSignal(NULL) works as expected now
        updated charstring::httpEscape not to escape legal symbols
        added flush() method to stdio classes
        added detection and support for vsnprintf_s
        fixed serviceentry, protocolentry and rpcentry on windows
        updated *entry classes for NULL and uninitialization safety
        implemented filesystem::initialize(fd) for Windows
        updated filesystem::initialize() with NULL and -1 to behave in
                predictable ways now, updated documentation for them too
        abstracted _get_osfhandle calls and added an invalid parameter callback
        updated filedescriptor to use bit fields for all of its flags
        implemented file::changeOwner() for windows
        removed xattrs stuff
        removed SSL-related ifdef's from headers and implemented degenerate
                SSL routines for when SSL isn't supported
        timezone conversion works on windows now
        refactored bytebuffer and stringbuffer to be extent-based
        fixed a subtle stringbuffer with initialcontents bug
        refactored stringbuffer to use bytebuffer::write/appendFormatted
                directly instead of charstring::parseNumber
        implemented dynamicarray and staticarray classes
        refactored build process to use nmake and be compatible with many
                different versions of MS Visual Studio
        updated vsnprintf_s calls for platforms where it's not defined
        added configure test for bool and true/false and definitions for
                platforms that don't define them
        updated timed semaphore waits - if seconds or nanoseconds are negative
                then the timed wait acts like an un-timed wait
        added support for getsockopt with size_t optlen parameter
        fixed strncasecmp on windows when str2 is longer than str1
        xmlsax uses memorymap on Windows now
        fixed file::lockFile() and file::unlockFile() on Windows
        fixed charstring::printf on Solaris 2.5.1
        fixed datetime::getSystemDateAndTime() for Windows without 64-bit
        fixed potential double-CloseHandle in thread class

Wednesday, April 1, 2015

SQL Relay 0.58.1 is out

It appears that 0.58 shipped with a bug that caused a few people to scratch their heads for quite a while.

The bug caused sqlr-start to crash, sometimes, under very specific circumstances.

The first is if an instance is defined with no port and no socket (ie. falling back to the default port of 9000). This would fairly reliably crash.

The second is very intermittent. Basically, if you call sqlr-start by itself, with no id, expecting that it will start all instances where enabled="yes" is configured, it will crash. Sometimes. Depending on a lot of random things. It turned out that a few variables were uninitialized, and random crashing a fairly classic behavior in that case.

Both issues are fixed in the 0.58.1 release. So, if you've had problems like those with 0.58, then give 0.58.1 a try.

It can be downloaded here.

Thursday, March 12, 2015

ODBC Progress

SQL Relay has long supported ODBC to some extent, both on the server and client side. Ironically though, the client-side has always worked a lot better on Unix/Linux than Windows.

That's just a terrible state of affairs. Windows is where ODBC is most commonly used. Yeah, ADO.NET's the in-thing these days, but there's still a whole world of ODBC apps out there for Windows. You ought to be able to aim them at SQL Relay.

A week or so ago, I got the driver working well enough to be usable if you don't mind manually editing .reg files and loading them using regedit. A good step, but not good enough for the average user. The driver needed a real configuration interface.

Last night I made some good progress though.

I guess I could have drawn a better looking UI using modern tools, but I kind-of want to maximize compatibility, and it didn't seem right to pull in the .NET framework just to be able to configure an ODBC DSN.

It's been a looong time since I did anything with the WinAPI directly, and it was an interesting challenge... "How do I get the background colors to match on the labels again?" "What's the trick to getting tab-navigation to work if your edit controls are attached to a child window?"

But, I got it going, and that plain little window works pretty well.

Apps are working pretty well too.

Little-by-little, they're revealing discrepancies, but it's been a while since anything just didn't work. Things are looking good.

When 0.58 is out, you ought to be able to aim your Windows-based ODBC apps at Relay.

Friday, January 2, 2015

Weather Cache - 1.0

Announcing firstworks Weather Cache for Linux version 1.0 !

What is a Weather Cache?

The firstworks Weather Cache is a light, simple, caching proxy, targeted for use with Weather API's.

Weather API's provide web-services that allow you to display current conditions, forecasts or historical weather information on your website or mobile app.  For example:

But there are several issues with these innocuous-looking badges.

Downloading weather data from across the internet takes time. Fetching it on the server-side while building a page can slow page loads significantly.

You could update your site so that the data is fetched on the client-side after the rest of the page has loaded but that introduces Cross-Site Scripting (XSS) issues.  Yes, you can work around these issues with JSONP and CORS, but only if the Weather API supports them.

But the bigger issue, the one that crops up when your site starts getting popular, is request count limits.  When you exceed some number of requests during some period of time you get throttled, blocked, or have to pay more.

The firstworks Weather Cache sits between your application and the Weather API's web-service, forwarding requests from your application and receiving results from the web-service. Each result is cached in a local database and tagged with an expiration date before being returned to your application. If the cache receives a request for the same data prior to the expiration date, the result is fetched from the database, rather than from the web-service.
  • When results are fetched from the local database rather than from across the internet, page load times are improved dramatically.
  • Since duplicate requests are resolved locally, the number of requests to the Weather API are minimized.
  • Cross-Site Scripting issues are avoided because the domain of the cache is the same as the rest of your site.
There are other caching technologies out there, but they are complex and subject to a few Weather-API-specific drawbacks.

The firstworks Weather Cache is extremely simple.  It's distributed as a self-contained, statically-linked x86 Linux binary and should be compatible with any x86 or x86_64 Linux platform. Its only requirements are a web server configured to support CGI's and read/write access to the /var/tmp directory.  Simply install it and aim it at the Weather API of your choice:


The Weather Cache is designed to be API-agnostic but it does a few semi-intelligent things to work around some common cache-defeating issues.
  • Parameters are normalized so subtly different requests for the same information will be recognized as the same request.
  • Mobile devices are "snapped" to a grid. Successive requests from mobile devices tend to send slightly different latitudes and longitudes, especially if the device is moving. When snapped to a grid, the location of the device only changes when it has has moved far enough to snap to a different grid point. Multiple mobile devices in close proximity snap to the same grid point.
 If your website or mobile app displays weather information, it could probably benefit from the firstworks Weather Cache.  Check it out.