Wednesday, November 12, 2014
Since then I've made quite a bit of progress getting the server working too. And tonight things really started to come together...
That's the SQL Relay server running on Windows, connected to a database running on Linux, accessed by sqlrsh, also running on Windows.
And it can be accessed remotely too.
That's the same SQL Relay server running on Windows, accessed by sqlrsh running in a separate Linux VM.
There are a number of things that don't work yet like sqlr-stop and various plugins, and there are several things that just haven't been tested yet, but the big things are coming along nicely. I expect to be able to announce preliminary support for the SQL Relay server software on Windows in the next release.
Tuesday, October 28, 2014
I'm not sure how long ago this happened, but there are now lots of different JVM's available for Debian and Ubuntu systems: openjdk-6, openjdk-7, openjdk-8, oracle-jvm-7 and oracle-jvm-8. Possibly more...
SQL Relay 0.57 has no trouble detecting the openjdk's, but doesn't properly detect oracle-jvm's and the build fails. I ran into this error trying to build on a Raspberry PI running Raspbian, but a few people have reported the same error on Debian and Ubuntu sytems.
There are 3 workarounds:
If you're not using SQL Relay with java then you can add --disable-java to the configure command and SQL Relay won't even try to build against oracle-jvm.
Replace Oracle JVM with OpenJDK
This may be tricky. I'm not sure which packages have a hard dependency on oracle-jvm as opposed to just needing some version of Java. Your mileage may vary.
Patch the Configure Script
Edit the configure script in the SQL Relay source distribution and look for a line like:
for i in `ls -d /usr/java/jdk* /usr/java/j2sdk* /usr/local/jdk* 2> /dev/null` /usr/java /usr/local/java `ls -d /usr/local/openjdk* /usr/pkg/java/openjdk* 2> /dev/null` `ls -d /usr/lib64/jvm/java 2> /dev/null` `ls -d /usr/lib64/jvm/java-1.8* 2> /dev/null` `ls -d /usr/lib64/jvm/java-1.7* 2> /dev/null` `ls -d /usr/lib64 /jvm/java-1.6* 2> /dev/null` `ls -d /usr/lib/jvm/java 2> /dev/null` `ls -d /usr/lib/jvm/java-1.8* 2> /dev/null` `ls -d /usr/lib/jvm/java-1.7* 2> /dev/null` `ls -d /usr/lib/jvm/java-1.6* 2> /dev/null` /System/Library/Frameworks /JavaVM.framework/Versions/Current /usr /usr/local
and replace it with these lines:
for i in `ls -d /usr/java/jdk* /usr/java/j2sdk* /usr/local/jdk* 2> /dev/null` \ /usr/java \ /usr/local/java \ `ls -d /usr/local/openjdk* /usr/pkg/java/openjdk* 2> /dev/null` \ `ls -d /usr/lib64/jvm/java 2> /dev/null` \ `ls -d /usr/lib64/jvm/java-1.8* 2> /dev/null` \ `ls -d /usr/lib64/jvm/java-1.7* 2> /dev/null` \ `ls -d /usr/lib64/jvm/java-1.6* 2> /dev/null` \ `ls -d /usr/lib64/jvm/jdk-7-* 2> /dev/null` \ `ls -d /usr/lib64/jvm/jdk-8-* 2> /dev/null` \ `ls -d /usr/lib/jvm/java 2> /dev/null` \ `ls -d /usr/lib/jvm/java-1.8* 2> /dev/null` \ `ls -d /usr/lib/jvm/java-1.7* 2> /dev/null` \ `ls -d /usr/lib/jvm/java-1.6* 2> /dev/null` \ `ls -d /usr/lib/jvm/jdk-7-* 2> /dev/null` \ `ls -d /usr/lib/jvm/jdk-8-* 2> /dev/null` \ /System/Library/Frameworks/JavaVM.framework/Versions/Current \ /usr \ /usr/local
One of those three solutions should work.
The source has been updated and it's fixed in CVS. The next release will include the fix too.
Saturday, October 11, 2014
It appears that I forgot to run "make distclean" before building that last round of tar.gz and zip files for both Rudiments and SQL Relay. So, if you downloaded last week (the week of October 6th, 2014), or the week before, then you may have had trouble building. Configure probably ran, but make might not have made anything.
Who knows what platform the binaries that were in there ran on? Fedora 20 x64 maybe... Maybe not though.
I just updated the tar.gz and zip files with clean versions. I also downloaded and verified them this time. They ought to be correct.
Thanks Florin... If you hadn't pointed it out, I might never have noticed.
Saturday, October 4, 2014
SQL Relay 0.57 is now available!
Support for Windows is much improved in this release. The command line clients all work now - sqlrsh, sqlr-export, sqlr-import, etc. The C/C++, C# and ADO.NET API's have worked for a while, but are now joined by native API's for Perl, Python, PHP and Java as well as drivers for Perl DBI, Python DB and PHP PDO.
It is possible to build all of this on Windows using MS Visual Studio or MS Visual Studio Express but it is currently quite difficult and requires specific versions of things to be installed in the right places. PHP is especially difficult to work with. In fact, I'm honestly not sure how 3rd party developers are expected to develop drivers for Windows. I had to shoehorn the headers in manually from the source code.
However... The SQL Relay Binary Distribution for Windows is now available (for a few bucks) and provides installers for binary versions of the above-mentioned software.
So... in addition to just accessing SQL Relay from command line clients, SQL Relay applications, including web-based applications, can now be developed for Windows in C, C++, C# (and other .NET languages), Perl, Python, PHP and Java.
Give it a try, let me know if you run into trouble.
Oracle "describe table" Improvements
In sqlrsh if you run "describe tablename" it returns info about the columns of the table - name, type, size, etc. This is accomplished by a call to the getColumnList() method of the C++ API. All SQL Relay API's support this method, and it has been improved.
For oracle tables, if a column is a primary, unique or foreign key, that information is now given. In previous releases it was not. This is a fairly expensive operation, as it turns out, so there is a config parameter to disable looking up keys. See disablekeylookup=yes/no in the oracle connection string in the SQL Relay Configuration Reference for more info.
Also, if you run "describe" on a synonym to a table in another schema, it will now return information about that table. In previous releases, only tables could be described.
The Rudiments library that SQL Relay depends on has been updated to randomize the list of hosts returned by DNS server when using round-robin DNS. If you are using round robin DNS to distribute SQL Relay clients over a pool of SQL Relay servers, then the behavior may different than in previous releases. In particular, if one host is down, then rather than slamming the next host in the list with all of the downed host's traffic, load will be distributed over the remaining hosts. This feature was added to work around an issue with many recent-ish implementations of getaddrinfo().
Rudiments has also been updated to support kqueue, epoll, port_create, /dev/poll, poll and select in its listener class. There should be no concerns now about large numbers of clients queuing up on the listener, or with listener performance with regard to using select internally.
See the rudiments release announcement for more information on these updates.
Bug Fixes and Minor Improvements
This release features a few bug fixes and minor improvements too.
The Python API's getRowDictionary method had a bug where None's were being returned as 0's when getNullsAsNones was set. That's fixed now.
Sensible errors are now returned when the format of a numeric bind variable is incorrect when using mysql, firebird or oracle. An error was fixed that could cause mysql connection to loop up if alphanumeric bind variable names were used instead of numbers. A bind-variable translation bug was fixed that could cause variables not to be translated when multiple formats are used in the same query and one of them is the correct format too.
The --with-system-libtool configure option was broken but it's fixed now.
Minix 3.3.0 is now supported. At least the client side is. Ie. you can access Oracle from Minix 3.3.0 using sqlrsh and SQL Relay running on another system. The server-side stuff doens't work yet. The Minix kernel doesn't support semaphores by default and Minix 3.3.0 ships without kernel source. I'm not sure if it supports semaphores at all though, so maybe that's moot.
The drop-in replacement library for MySQL had bugs with its implementations of mysql_row_seek() and mysql_row_tell(). Those are fixed now.
The configure script properly detects Maria DB on Ubuntu 14.04.1 now.
Give it a try and report any bugs that you find!
Friday, October 3, 2014
Rudiments 0.48 is now available!
This release has a few fairly important features.
Random Number Generation
The randomnumber class has been updated to support CryptGenRandom (on Windows), arc4random, random_r, rand_r, lrand48_r, random, rand and lrand48. It will use whichever of those it can find, in that order. The class has static and non-static members now. When the non-static members are used, the class can be seeded once and numbers (or scaled numbers) can be generated over and over without explicitly re-seeding.
Inet Socket Clients
Several improvements have been made to the inetsocketclient class.
Internally, inetsocketclient uses either getaddrinfo() or some variant of gethostbyname(), depending on what's available.
First, a bug was fixed that caused timeouts not to work on systems that don't have getaddrinfo().
Second, it seems that the behavior of getaddrinfo() has changed over the years and a workaround has been introduced to deal with some issues that the new behavior causes...
When round-robin DNS is used, gethostbyname() and getaddrinfo() fetch the entire list of IP addresses that the host name resolves to. The inetsocketclient's connect() method gets this list and tries to connect to each of these IP's, in the order that they were returned by gethostbyname()/getaddrinfo(). When round-robin DNS is used, the order of the IP's in the list is rotated with each DNS request. This can be used to implement a simple load-balancing scheme.
"Recent" (I'm not sure how recent) implementations of getaddrinfo() have begun sorting the IP's, defeating round-robin DNS entirely. Some implementations have a /etc/gai.conf file that allow a sortv4=no directive, but many do not. Some implementations randomize the list of IP's. Some implementations return them in the order that the DNS server returned them in.
Rudiments has no way of knowing what the behavior of getaddrinfo() is, so now, by default, the list of IP addresses is randomized. For the sake of consistency, it's randomized whether it uses getaddrinfo() internally or some variant of gethostbyname(). Methods have been added to the class to enable or disable this behavior too.
Randomization provides a benefit over round-robin DNS as well. With round-robin DNS, when a host goes down, the next host in the list will receive all of the downed host's traffic. With randomization, traffic is distributed evenly over the hosts that are still up.
The listener class has been refactored entirely to match the paradigms established by kqueue, epoll, /dev/poll, etc. It now supports kqueue, epoll, port_create, /dev/poll, poll and select internally as well, and will use whichever is available, in that order.
In the previous release, support was added for kqueue, epoll and poll but the implementation still did some inefficient things like rebuilding the list of file descriptors to listen on before each iteration. The list is only rebuilt if a file descriptor is added or removed now.
Minix 3.3.0 came out since the last release and it's now supported.
General Bug Fixes and Improvements
Some general bug fixes and improvements were implemented as well. Cygwin builds ought to work correctly now. Setting file permissions ought to work natively on Windows now too.
Monday, July 14, 2014
SQL Relay version 0.56 is now available!
You might want to get this one. A subtle memory leak is fixed in this release. If you really can't upgrade and need a fix for a previous release, contact me and we can work something out.
Amazing new features:
Perl, PHP, Python and Java API's all build and work on Windows. You still have to build them from source though, and it's a pain. If you need Windows binaries, contact me and we can work something out.
Intelligent defaults have been added for a lot of the configuration parameters. The sqlrelay.conf file can be much more stripped down now. I also added a Configuration Guide that walks you through various configurations from minimal to complex. Enjoy.
The PHP PDO API now has a custom statement attribute PDO::SQLRELAY_ATTR_GET_NULLS_AS_EMPTY_STRINGS that can be set to true or false to cause it to return NULL values as NULLs or empty strings.
New parameters have been added for connecting to MySQL via SSL. See the Configuration Reference for more detail.
Amazing bug fixes:
The PHP PDO driver handles float values correctly now. Previously it was converting them to strings. It also handles NULL integers correctly now rather than converting them to 0's.
There was a bug in the sqlr-scaler that could cause it to use a noticeable amount of the CPU when sitting idle. That's fixed now.
Some installations of Oracle instant client might not have been properly detected before, but they are now.
There was a bug that caused problems with DB2 7.X. It's fixed now.
sessionhandler="thread" and listenertimeout!="0" ought to work now. This is still sketchy and experimental though.
A lot of code has been reorganized internally. Many things have been modularized and simplified. Frameworks have been built and some things have been re-implemented using them. This is ongoing. More of this will be done over the next few releases.
Full ChangeLog follows:
- removed VERSION from perl api bootstrap
- reorganized and simplified perl api code
- got perl api working on windows with ActivePerl
- perl api uninstall cleans up better now
- got python api working on windows with ActivePython
- consolidated php include tweaks
- got java api working on windows
- updated pdo driver to convert float fields to strings rather than ints
- updated pthread test to match rudiments pthread test
- moved everything done by children of sqlwriter interface into individual translations
- added plugin-based authentication framework
- reorganized code tree a bit
- reworked default values to enable more minimal configurations
- updated configuration docs and added configuration guide
- fixed cast issue with firebird 1.5
- added result set translation framework and re-implemented date translation using it
- fixed millisecond->nanosecond confusion in sqlr-scaler
- fixed subtle oracle instantclient version detection bug
- added null input bind support to sqlrsh
- fixed php pdo driver to return null rather than 0 for null integer fields and bind null rather than 0 for null integer input binds
- added attribute to return nulls as nulls or empty strings to php pdo
- added parameters for connecting to mysql via ssl
- added a few tweaks to support DB2 < 8.0
- fixed bad return value type in python getConnectionPort method
- fixed sessionhandler="thread" with listenertimeout!="0"
For immediate release!
Rudiments 0.47 is now available.
This update to rudiments fixes a few obscure things including a memory leak in the datetime class that could cause SQL Relay to chew up memory (very slowly) and a potential null-dereference in the xmldomnode class. Not sure how I ever missed those bugs before, given how often I use those classes, but at least they're fixed now.
Another significant update in this release is internal support for poll, epoll and kqueue. In the past, the listener class used select exclusively, but select has some issues. Most notably, attempting to listen on "large numbers" (more than 1024 on most systems) of file descriptors could cause undefined problems (references beyond the end of a 1024-member array). Now, if the platform supports epoll (like most modern linux systems) or kqueue (like most modern BSD systems) then those will be used in favor of select. If the platform supports neither epoll nor kqueue, but supports regular poll, then poll will also be used in favor of select. /dev/poll is not yet supported on solaris yet, but it's on the TODO list. Also, the poll/epoll/kqueue implementations aren't optimized yet. They still do inefficient things like rebuild the list of file descriptors every time, which has to be done with select, but they are a bit faster than select, and safe for large numbers of file descriptors.
Two classes have been renamed to be more consistent - variablebuffer is now bytebuffer and rawbuffer is now bytestring. Hey, it's pre-1.0 software. Things get renamed.
The linkedlist class has been refactored and a singlylinkedlist class has been added. Both now have sort methods now too: selectionSort() and heapSort(). Writing them took me back to my data structures class in college. selectionSort() is slow but uses no additional storage. heapSort() is much faster but uses N additional storage. Choose!
I also fixed an obscure bug on windows involving file::open() with O_CREAT but not O_EXCL.
Solaris 2.5.1 is also supported, just in case anyone still uses that!
Full ChangeLog follows:
- fixed possible null-dereference in xmldomnode::safeAppend
- added charstring::inSetIgnoringCase
- xmldomnode::setAttributeValue does nothing when the null node is referenced now
- fixed a codetree bug that could cause indentation to attempt to go negative when using an unsigned number
- added a configure test to see if -Wno-format is needed
- configure tests for mlockall/munlockall attempt link now
- added sys/types.h before sys/un.h in sys/un.h test for older freebsd
- implemented stubs for pure virtual methods of client and server classes so instances of them could be created to attach already-open file descriptors to
- unified usage of select/poll and prefer poll
- pushed all select/poll-related code into listener class
- removed problematic and unused useListener-related methods in filedescriptor class
- epoll is used in place of select/poll for systems that support it
- kqueue is used in place of select/poll for systems that support it
- consolidated getpagesize() calls to use sys::getPageSize()
- added detection and support for __vsnprintf for platforms that have that instead of vsnprintf
- added cancel and raiseSignal methods to the thread class
- fixed datetime memory leak
- added xmldomnode::clone method to clone a node
- renamed variablebuffer to bytebuffer
- renamed rawbuffer to bytestring
- refactored linkedlist
- added singlylinkedlist class
- added sort, detach, move and insert methods to linkedlist classes
- fixed file::open using O_CREAT without O_EXCL on windows