Thursday, March 12, 2015

ODBC Progress

SQL Relay has long supported ODBC to some extent, both on the server and client side. Ironically though, the client-side has always worked a lot better on Unix/Linux than Windows.

That's just a terrible state of affairs. Windows is where ODBC is most commonly used. Yeah, ADO.NET's the in-thing these days, but there's still a whole world of ODBC apps out there for Windows. You ought to be able to aim them at SQL Relay.

A week or so ago, I got the driver working well enough to be usable if you don't mind manually editing .reg files and loading them using regedit. A good step, but not good enough for the average user. The driver needed a real configuration interface.

Last night I made some good progress though.

I guess I could have drawn a better looking UI using modern tools, but I kind-of want to maximize compatibility, and it didn't seem right to pull in the .NET framework just to be able to configure an ODBC DSN.

It's been a looong time since I did anything with the WinAPI directly, and it was an interesting challenge... "How do I get the background colors to match on the labels again?" "What's the trick to getting tab-navigation to work if your edit controls are attached to a child window?"

But, I got it going, and that plain little window works pretty well.

Apps are working pretty well too.

Little-by-little, they're revealing discrepancies, but it's been a while since anything just didn't work. Things are looking good.

When 0.58 is out, you ought to be able to aim your Windows-based ODBC apps at Relay.

Friday, January 2, 2015

Weather Cache - 1.0

Announcing firstworks Weather Cache for Linux version 1.0 !

What is a Weather Cache?

The firstworks Weather Cache is a light, simple, caching proxy, targeted for use with Weather API's.



Weather API's provide web-services that allow you to display current conditions, forecasts or historical weather information on your website or mobile app.  For example:





But there are several issues with these innocuous-looking badges.

Downloading weather data from across the internet takes time. Fetching it on the server-side while building a page can slow page loads significantly.

You could update your site so that the data is fetched on the client-side after the rest of the page has loaded but that introduces Cross-Site Scripting (XSS) issues.  Yes, you can work around these issues with JSONP and CORS, but only if the Weather API supports them.

But the bigger issue, the one that crops up when your site starts getting popular, is request count limits.  When you exceed some number of requests during some period of time you get throttled, blocked, or have to pay more.

The firstworks Weather Cache sits between your application and the Weather API's web-service, forwarding requests from your application and receiving results from the web-service. Each result is cached in a local database and tagged with an expiration date before being returned to your application. If the cache receives a request for the same data prior to the expiration date, the result is fetched from the database, rather than from the web-service.
  • When results are fetched from the local database rather than from across the internet, page load times are improved dramatically.
  • Since duplicate requests are resolved locally, the number of requests to the Weather API are minimized.
  • Cross-Site Scripting issues are avoided because the domain of the cache is the same as the rest of your site.
There are other caching technologies out there, but they are complex and subject to a few Weather-API-specific drawbacks.

The firstworks Weather Cache is extremely simple.  It's distributed as a self-contained, statically-linked x86 Linux binary and should be compatible with any x86 or x86_64 Linux platform. Its only requirements are a web server configured to support CGI's and read/write access to the /var/tmp directory.  Simply install it and aim it at the Weather API of your choice:

Before:
http://api.wservice.com/current.asp?q=Atlanta,ga&mode=json&apikey=a1b2c3d4e5

After:
http://mydomain.com/weathercache.cgi/api.wservice.com/current.asp?q=Atlanta,ga&mode=json&apikey=a1b2c3d4e5
 
The Weather Cache is designed to be API-agnostic but it does a few semi-intelligent things to work around some common cache-defeating issues.
  • Parameters are normalized so subtly different requests for the same information will be recognized as the same request.
  • Mobile devices are "snapped" to a grid. Successive requests from mobile devices tend to send slightly different latitudes and longitudes, especially if the device is moving. When snapped to a grid, the location of the device only changes when it has has moved far enough to snap to a different grid point. Multiple mobile devices in close proximity snap to the same grid point.
 If your website or mobile app displays weather information, it could probably benefit from the firstworks Weather Cache.  Check it out.

Sunday, December 28, 2014

SQL Relay 0.58 is out

SQL Relay version 0.58 is now available!

Windows Support

The most significant feature of this release is preliminary support for the SQL Relay server on Windows.  The client API's and command line programs have long supported Windows, but in this release, it's possible to run the SQL Relay server on Windows as well.

This is a preliminary release,it currently only supports Oracle, and there are some quirks, but it's a good first step.

Running the SQL Relay server on Windows is much the same as on Linux or Unix, just run sqlr-start as described in the documentation.  The biggest quirk is that it doesn't run as a service yet.  You must currently run it from a console window and if you close the console window, it kills the processes.  Other quirks: sqlr-status doesn't work yet and the various loadable modules such as password encryption, authentication and logging don't work yet.  Also, only Oracle databases are currently supported.  You can't run SQL Relay on Windows, against Sybase, DB2, MySQL, PostgreSQL, SQLite, MS Access or MS SQL Server yet.

Building SQL Relay on Windows is tricky and currently requires the right versions of things to be in the right places (as described in the documentation) so if you want to try SQL Relay on Windows, I recommend the SQL Relay Binary Distribution for Windows, available on the downloads page for a small fee.

Also, if you are running a 64-bit OS then use the 64-bit version of SQL Relay.  The 32-bit version doesn't perform well on 64-bit systems.  The issues it has appear to affect other applications as well though, and don't appear to be specific to SQL Relay.


Configuration and Init Improvements

Another significant new feature is support for an sqlrelay.conf.d directory.  As of this release, in addition to the sqlrelay.conf file, all configuration files in the sqlrelay.conf.d directory are also processed.  This makes it possible to keep instance configurations separate.

The init script and sqlr-start program have also been updated.  Rather than running sqlr-start with each instance listed in /etc/sqlrelay or /etc/sysconfig/sqlrelay, the init script now just runs sqlr-start without specifying an instance.  If run without specifying an instance, sqlr-start now digs through the configuration files and starts all instances with enabled="yes" attributes.

Together, these updates make configuration much cleaner, but in particular make it much easier for third party applications to install a configuration and cause an instance to start at boot.


Threaded Listener

The sqlr-listener can run in either multi-process or multi-thread mode and now runs in multi-thread mode by default, which is much faster and lighter weight.


PHP Improvements

PHP ini files are now installed to enable the PHP and PHP PDO modules.  It's no longer necessary (on modern platforms, at least) to manually enable them.


PHP PDO Improvements

The PHP PDO driver features many improvements.

The connect-string now includes resultsetbuffersize, dontgetcolumninfo and nullsasnulls parameters.  The debug connect-string parameter can be set to a file name to send debug information to a file instead of to the browser.

Output bind variables can be bound to streams now.  It is, for example, possible to fetch a blob directly into a file, using an output bind variable.

Samat Yusupov contributed a set of driver-methods for ending, suspending and resuming sessions.  They have been incorporated and improved a little, as described in the documentation.


Perl DBI Improvements

The Perl DBI driver is much improved as well.

Support for the maxrows parameter to fetchall_arrayref has been added.  The begin_work method works now.  The get_info method works properly now.

RowCacheSize and RowsInCache attributes work correctly now.  The entire result set is now buffered by default (as is the case with other SQL Relay client API's) unless the RowCacheSize is set.

A custom ext_SQLR_Debug database handle attribute now allows debug to be turned on or off or sent to a file.

Type, length, precision and scale attributes can now be set for bind variables.  Output bind BLOBs and CLOBs  work now.

The ParamValues, ParamTypes and ParamArrays attributes work correctly now.

The driver works correctly with very old versions of Perl DBI now too.

Lots of fixes.


SAP ASE Support

SAP/Sybase ASE 16.0 is supported now.  The configure script detects it properly.  Version 16.0 revealed a few bugs that have been fixed too.

 
PostgreSQL Improvements

It is now possible to connect to PostgreSQL databases over SSL using the sslmode option.


JMeter  Support

The ODBC driver has been improved sufficiently to work with JMeter via the JDBC-ODBC bridge.  JMeter can now be used to benchmark all tiers of the application directly.  This is helpful in determining whether a performance problem lies in the database, SQL Relay, or the app.


Changes Under the Hood

Internally, quite a few things were restructured.  A lot of tightly coupled code has been decoupled and refactored into frameworks with plugins.  This ought to be invisible to the end user, but should enable much more modular development in the future.
 

Bug Fixes

A slew of bugs have been fixed.

Some are significant:
  • Oracle JDK 7 and 8 are now detected correctly on Debian/Ubuntu systems.
  • Blobs work when faking input binds now.
  • Everything ought to compile cleanly with clang now.
  • DB2 output bind BLOBs aren't truncated any more.
Others only occurred in obscure circumstances, but I guess they're still significant if they happened to you:
  • sqlrsh doesn't lose the timezone when binding dates any more.
  • The PostgreSQL drop-in replacement library had buggy implementations of PQreset, PQresetStart, PQresetPoll.  Those are fixed.
  • The MySQL drop-in replacement library had implementations of mysql_row_seek, mysql_row_tell., mysql_stmt_row_seek and mysql_stmt_row_tell that didn't work on platforms with 32-bit void pointers.  Those are fixed.
  • Subtle bugs related to keeping track of the total row count in the sybase and freetds connections have been fixed.
  • Old versions of Perl (5.00x) are supported now.
  • An obscure bug that caused DB2 output bind integers to be copied out incorrectly sometimes has been fixed.
  • A very obscure bug that could cause Oracle connections to crash sometime later if an output bind cursor was used to fetch blob data has been fixed.

Rudiments 0.49 is out

Rudiments version 0.49 is now available.

Plenty of good stuff in this release: much improved support for Windows, some refactoring of the signal-handling classes, unification of the passwdentry and shadowentry classes into a single userentry class, improvements when compiling with clang, and a slew of subtle bug fixes.

Full changelog follows:
  • combined passswdentry and shadowentry classes into userentry class
  • updated file::sync() to use FlushFileBuffers directly
  • implemented sys::sync(), reboot(), halt() and shutdown() for windows
  • implemented various sys::getXXX() methods for windows
  • added "detached" option to process::spawn()
  • crash-related methods in process class catch SIGABRT, SIGFPE, SIGILL, SIGBUS, SIGIOT, SIGEMT and SIGSYS in addition to SIGSEGV now
  • shutdown-related methods in process class catch SIGQUIT and SIGHUP in addition to SIGINT and SIGTERM now
  • on windows, signal classes now support catching, sending/raising SIGABRT, SIGFPE, SIGILL, SIGINT, SIGSEGV and SIGTERM and sending/raising SIGKILL
  • disabled threads outright on OSR5
  • fixed threads detection for solaris 2.6
  • codetree bails properly at end of string now
  • fixed a break-related condition that could cause codetree to loop indefinitely
  • added support for a beginning-of-line character to codetree
  • added support for recursive breaks to codetree
  • made charstring::compareIgnoringCase null-safe for platforms that don't have str(n)casecmp
  • improved Windows support in signal classes
  • -Wno-overloaded-virtual is now set when clang is used
  • fixed some subtle kqueue-related bugs
  • added sys/byteorder.h and htonll/ntohll for solaris 11
  • fixed endian-detection macros for solaris 10 and older
  • added support for SIGLARM and signalmanager::alarm() on windows
  • improved windows implementation of semaphoreset
  • added sys::signalsInterruptSystemCalls()
  • added process::supportsFork()

Wednesday, November 12, 2014

Windows Progress...

SQL Relay has classically run on Unix and Linux but not Windows. A few years ago I got the C/C++ API working on Windows and added a C# API and .NET data provider. I got a few of the command-line programs working too, but that was about all you could do on Windows for quite a while. A few months ago when I got the Perl, Python and PHP API's working on Windows along with the rest of the command line clients, but still, only the client side software worked.

Since then I've made quite a bit of progress getting the server working too. And tonight things really started to come together...



That's the SQL Relay server running on Windows, connected to a database running on Linux, accessed by sqlrsh, also running on Windows.

And it can be accessed remotely too.



That's the same SQL Relay server running on Windows, accessed by sqlrsh running in a separate Linux VM.

There are a number of things that don't work yet like sqlr-stop and various plugins, and there are several things that just haven't been tested yet, but the big things are coming along nicely. I expect to be able to announce preliminary support for the SQL Relay server software on Windows in the next release.

Tuesday, October 28, 2014

oracle-jvm error...

I'm not sure how long ago this happened, but there are now lots of different JVM's available for Debian and Ubuntu systems: openjdk-6, openjdk-7, openjdk-8, oracle-jvm-7 and oracle-jvm-8. Possibly more...


SQL Relay 0.57 has no trouble detecting the openjdk's, but doesn't properly detect oracle-jvm's and the build fails. I ran into this error trying to build on a Raspberry PI running Raspbian, but a few people have reported the same error on Debian and Ubuntu sytems.


There are 3 workarounds:


Disable Java

If you're not using SQL Relay with java then you can add --disable-java to the configure command and SQL Relay won't even try to build against oracle-jvm.


Replace Oracle JVM with OpenJDK

This may be tricky. I'm not sure which packages have a hard dependency on oracle-jvm as opposed to just needing some version of Java. Your mileage may vary.


Patch the Configure Script

Edit the configure script in the SQL Relay source distribution and look for a line like:


   for i in `ls -d /usr/java/jdk* /usr/java/j2sdk* /usr/local/jdk* 2> /dev/null` /usr/java 
/usr/local/java `ls -d /usr/local/openjdk* /usr/pkg/java/openjdk* 2> /dev/null` `ls -d /usr/lib64/jvm/java 2> /dev/null` 
`ls -d /usr/lib64/jvm/java-1.8* 2> /dev/null` `ls -d /usr/lib64/jvm/java-1.7* 2> /dev/null` `ls -d /usr/lib64
/jvm/java-1.6* 2> /dev/null` `ls -d /usr/lib/jvm/java 2> /dev/null` `ls -d /usr/lib/jvm/java-1.8* 2> /dev/null` `ls -d 
/usr/lib/jvm/java-1.7* 2> /dev/null` `ls -d /usr/lib/jvm/java-1.6* 2> /dev/null` /System/Library/Frameworks
/JavaVM.framework/Versions/Current /usr /usr/local

and replace it with these lines:


   for i in `ls -d /usr/java/jdk* /usr/java/j2sdk* /usr/local/jdk* 2> /dev/null` \
    /usr/java \
    /usr/local/java \
    `ls -d /usr/local/openjdk* /usr/pkg/java/openjdk* 2> /dev/null` \
    `ls -d /usr/lib64/jvm/java 2> /dev/null` \
    `ls -d /usr/lib64/jvm/java-1.8* 2> /dev/null` \
    `ls -d /usr/lib64/jvm/java-1.7* 2> /dev/null` \
    `ls -d /usr/lib64/jvm/java-1.6* 2> /dev/null` \
    `ls -d /usr/lib64/jvm/jdk-7-* 2> /dev/null` \
    `ls -d /usr/lib64/jvm/jdk-8-* 2> /dev/null` \
    `ls -d /usr/lib/jvm/java 2> /dev/null` \
    `ls -d /usr/lib/jvm/java-1.8* 2> /dev/null` \
    `ls -d /usr/lib/jvm/java-1.7* 2> /dev/null` \
    `ls -d /usr/lib/jvm/java-1.6* 2> /dev/null` \
    `ls -d /usr/lib/jvm/jdk-7-* 2> /dev/null` \
    `ls -d /usr/lib/jvm/jdk-8-* 2> /dev/null` \
    /System/Library/Frameworks/JavaVM.framework/Versions/Current \
    /usr \
    /usr/local

One of those three solutions should work.


The source has been updated and it's fixed in CVS. The next release will include the fix too.

Saturday, October 11, 2014

Oops...

It appears that I forgot to run "make distclean" before building that last round of tar.gz and zip files for both Rudiments and SQL Relay. So, if you downloaded last week (the week of October 6th, 2014), or the week before, then you may have had trouble building. Configure probably ran, but make might not have made anything.


Who knows what platform the binaries that were in there ran on? Fedora 20 x64 maybe... Maybe not though.


Anyway...


I just updated the tar.gz and zip files with clean versions. I also downloaded and verified them this time. They ought to be correct.


Thanks Florin... If you hadn't pointed it out, I might never have noticed.