Sunday, November 10, 2013

Consulting and Development Services

Just a reminder...

Consulting, Support and Integration Services for SQL Relay or other firstworks technologies are available and General Consulting and Software Development Services are also available.
  • I have expertise in:
    • C/C++ on Linux/Unix
    • Embedded Linux systems
    • Database-Driven Applications
    • System and platform migration
    • Systems and applications for small businesses
  • And experience with:
    • Java, PHP, Python, C#, Visual Basic and other languages.
    • Windows and Mac OS X
So how can I help you?
  • Need help converting your existing apps to use SQL Relay?
  • Need help developing new apps that use SQL Relay?
  • Need a feature that SQL Relay doesn't currently provide?
  • Need SQL Relay to run on a new platform?
I'm your guy.
  • Need a custom app for your small business?
  • Need a legacy app migrated to a modern platform?
  • Need to squeeze some life out of an old app or older Unix platform?
  • Need to integrate some odd set of dissimilar applications?
  • Need help with an Embedded Linux app?
I've done plenty of that, all of that, for clients all over the world.  I might be your guy.

Send me an email or give me a ring.  For contact info, click here.

Wednesday, November 6, 2013

.1 Updates

Hmm, it appears that I was a little too hasty in that last release of Rudiments and SQL Relay.

It turns out there were some documentation errors and they didn't build successfully on every platform in my build farm.  Most significantly though, there was an error in one of the Makefiles that cased a "make clean" to abort partway through the build.  This could cause problems for automated build systems.

So, .1 updates of Rudiments and SQL Relay are now out - 0.44.1 and 0.53.1 respectively.

Enjoy!

Wednesday, October 30, 2013

SQL Relay - 0.53 is out

SQL Relay 0.53 is now available!

This release contains several significant new features.

The most significant is the inclusion of a PHP PDO driver.  PDO is the most popular abstraction layer for PHP and SQL Relay has been without a driver for it for too long.  Give it a try:
http://sqlrelay.sourceforge.net/sqlrelay/programming/phppdo.html
It is new though, there could be bugs.  Report them to david.muse@firstworks.com if you find any.

Also significant is support for Multiarch systems.  Debian and Ubuntu systems have supported Multiarch for a while now and packages weren't successfully detected on those systems.  They are now, even on non x86/x64 platforms.

The init scripts have been refactored as well.  After installed, SQL Relay should integrate properly with the init processes on most Linux and Unix systems and OS X.

The SQLite Statement API is now supported, both versions of it.  Platforms that can take advantage of it should perform a little better and use less memory.


A few bugs have been fixed too, including a long-standing issue where IPC and socket-related files were left lying around, owned by whoever last ran SQL Relay.  This caused problems if a different user tried to run it.  They are removed now and things work as expected.


The PHP Pear DB and Zope drivers have been removed too.  Packages aren't even available for them on most systems.  Zope is still well-used and I might resurrect the driver some day.  PHP Pear DB seems to have fallen so out of fashion that it's hard to even test it any more.

The sqlr-start program no longer attempts to start the sqlr-cachemanager.  Honestly, it's quirky that it ever did.  There are separate init scripts for them now as well.

What else..

Fixes for a few unlikely-to-be-encountered memory leaks and a few more obscure features.

Complete ChangeLog follows:
  • added support for sqlite statement api and native binds
  • fixed some leaks related to using sys::getHostName()
  • added multiarch detection
  • added PHP PDO driver
  • fixed a bind variable translation bug where output binds followed by := would not be detected
  • dropped zope support (for now)
  • dropped PHP Pear DB support
  • refactored init script - one script should work on all platforms now
  • updated init script installation - should work on virtually all unixes
  • added OS X launchd configuration
  • updated the sqlr-listener to clean up files related to ipc, sockets and marking whether the db is up or down on exit
  • sqlr-start no longer starts the cache manager
  • added a second init script for the cache manager
  • plugins are statically linked into libsqlrserver if the platform doesn't support shared libraries (or if --disable-shared is specified at configure time)
  • the perl API should build with old versions of perl (5.00X) on older platforms (redhat 4.2, 5.2, 6.2, etc.) now
  • updated postgresql bind docs
  • added dateyyyyddmm parameter
  • added yyyyddmm parameter to translatedates translation
  • added SQLR_MYSQL_DATE_YYYYDDMM envrionment variable to mysql drop-in library
  • added SQLR_ODBC_DATE_YYYYDDMM envrionment variable to odbc driver

Rudiments - 0.44 is out

For immediate release!

Rudiments 0.44

This release has lots of small fixes, optimizations and compatibility improvements.  There are no major changes but lots of refactoring and... fiddling around.

The most significant change is support for multiarch platforms like Ubuntu and Debian.  OpenSSL and PCRE should be detected properly on those platforms now.

Here's the complete ChangeLog for this release.
  • fixed inet_aton test to attempt link, not just compile
  • fixed vsnprintf test to work on arm linux
  • filedescriptor::printf now uses vdprintf, if available, if writes are not being buffered, and vasprintf, if available, if writes are being buffered
  • fixed a memory leak in filedescriptor::printf
  • added multiarch detection
  • applied Simon Martin's getenv-related patch to reset errno and allow getenv to return NULL - fixed a situation where an infinte loop could occur if getenv returned NULL and the most recent error from another system call was EINTR
  • added missing print() for const char *'s in linkedlistutils
  • tweaks for OSR505
  • a few xmlsax optimization fixes
  • renamed *Data methods to *Value in linkedlist and dictionary classes
  • removed print methods and unlikely-to-be-used static methods from *entry classes
  • refactored the static convenience methods of the *entry classes
  • refactored xattr code a little to make it smaller
  • removed static methods from filesystem class to make it smaller
  • removed some static methods from file class to make it smaller
  • removed the clientserverfactory class
  • slight refactoring of linkedlist and dictionary classes
  • various process class fixes for Windows

Wednesday, October 16, 2013

The Horizon...

Looking ahead...

I like to look ahead.  It's so exciting!  Sometimes at least.  Hopefully this look ahead will be exciting to you.

Coming soon to SQL Relay...
  • PHP PDO Support
  • SQLite Statement API Support
  • Multiarch Support
The next release will include all three.  Exciting?  Maybe.  I guess that depends on your interests.

I've been meaning to implement these things for long time and just now getting around to doing it.

PDO

The PDO support will probably have the greatest impact.  PDO is the most popular PHP database abstraction layer these days and SQL Relay has long lacked support for it.  It's funny, for as popular as it is, it is a little quirky.  There's no obvious way to bind a floating point number, for example.  It looks like you just can't do it.  It doesn't appear to differentiate between Clobs and Blobs either.  They're both just LOB's.  That's actually giving me problems right now.  And it doesn't appear to support out-variables.  It supports in-binds and in/out-binds but not just plain out-binds.  SQL Relay supports in-binds and out-binds but not in/out-binds, so I guess I can't be too particular about another API's lack of support for something, but it's odd.  I'm currently mapping SQL Relay's out-binds to PDO's in/out's.  I hope that works out.

SQLite Statement API

This is an under-the hood change that won't likely have any noticeable effect, other than maybe subtle performance or memory-usage.  SQLite has long had a statement-oriented API and supported bind-variables natively, but SQL Relay has always used the older API and faked bind variables by rewriting the query.

No longer!

The SQLite Statement API is now supported.  Actually both versions of the statement API are supported.  There appears to have been an initial effort and then a follow-up where _v2 was appended to some of the functions and error handling was improved.  Both statement API's are now supported, as is the old sqlite3 API and the even older sqlite API.  So, it shouldn't matter what version of SQLite you're using, they're all supported.

From the outside, there are only two noticeable differences.

The statement API exposes column types, so column types will now be visible and not just "UNKNOWN".

Also, the statement API provides methods for fetching one row at a time, while the older API's either required you to register a callback to handle a row or returned the entire result set at once.  I opted to use the return-the-entire-result-set method and as a result, knew how many rows there were in the result set.  When using the statement API though, the total number of rows in the result set is unknown until they've all been fetched.  So, going forward, if you use setResultSetBufferSize() on the client-side, rowCount() will be unknown.

It's possible that performance will be improved using the statement API but I haven't run any tests to see.  It's likely that memory usage will be better on the server-side, as the sqlr-connection daemon doesn't buffer the entire result set any more.  Depending on the size of the result set, this could improve performance too.  Again, I haven't run any tests to see.

Multiarch Support

Debian and Ubuntu have recently embraced Multiarch.  It's a great idea but it creates headaches for configure scripts.

What is Multiarch?

Some architectures support multiple binary formats.  For example, an x86_64 CPU can run code compiled for x86_64 or x86.  An ARM chip with hardware floating point support can run ARM binaries that depend on hardware floating point support, or soft-float binaries.  Many ARM chips can run thumb code as well.  A MIPS64 chip can run 32 or 64-bit MIPS binaries.  Similar for SPARC64.

In the past, support for multiple architectures on the same system has been shoehorned in by creating /lib and /lib64 directories and putting 32-bit libs in /lib and 64-bit libs in /lib64.  BTW, this also created its share of headaches for my configure scripts, but happened so long ago that I'd forgotten entirely about it.

Multiarch aims to support more than just 32 vs. 64 bit variants.  On a Multiarch system, under /lib and /usr/lib, there are directories like i386-linux-gnu, amd64-linux-gnu, mipsel-linux-gnu and arm-linux-gnueabihf with architecture-specific libraries in them.  An amd64 machine might have both amd64-linux-gnu and i386-linux-gnu directories, enabling it to run 32 and 64-bit binaries.

This presented a challenge.  The configure script has to look in the appropriate subdirectory but the arch command doesn't help.  For example, arch might return i586 or armv7l rather than i386 or arm.  And how should we know to look in arm-linux-gnueabihf?  Where does that gnueabihf come from?  This plagued me until I discovered that on newer multiarch systems,  gcc -print-multiarch will return the multiarch tuple.  Woohoo!

So, now the configure scripts look in the appropriate directory.  If you use gcc -m32 on an x86_64 system, it should look in the i386-linux-gnu subdirectory.  If you use gcc -m64 (or leave out the -m option), it'll look in the x86_64-linux-gnu or amd64-linux-gnu subdirectory, as appropriate.

What does this mean for the user?  Basically on modern Debian and Ubuntu releases, The configure script will "just work" and correctly detect everything installed on the system rather than requiring painful manual intervention.

That's half of the challenge.  The other half is installing my libraries in the right places.  For now, I'm just still installing them in the lib directory.  Currently, the --libdir option to the configure script can be used to install them in the appropriate multiarch subdirectory.  I'll have to see what the debian package build scripts are doing.  They might just be using --libdir and I may not have to do anything.

So, that's what's on the horizon.

There will, of course, be a few bug fixes, and it's possible that the ODBC driver might have some improvements too, but I wouldn't hold my breath.

Wednesday, September 25, 2013

libfirstworks 0.2 is out

Quick on the heels of the most recent Rudiments and SQL Relay releases comes the 0.2 release of libfirstworks.  Basically since I added support for 64-bit builds on Windows to Rudiments and SQL Relay, I also had to update the build script in libfirstworks.  That's the only change.

SQL Relay - 0.52 is out

Yes.  SQL Relay 0.52 is now available for general consumption.

Usually when I make a release of SQL Relay there is at least one significant new features.  This release though, is mainly just for compatibility with all of the changes I recently made to Rudiments.  A side effect of those changes is platform compatibility, so arguably that's a new feature, but most of the platforms that it now runs on are old or obscure in one way or another.  There were some recent fixes for ARM platforms which are actually growing in popularity these days, but the fixes for SCO OSR6, NetBSD Vax or Linux libc4 aren't likely to get too many peoples' attention.

It is kind of neat though, to see screenshots of an Oracle 12c database being accessed from platforms that Oracle has probably never remotely considered supporting.

http://sqlrelay.sourceforge.net/sqlrelay/features/proxying.html

There were a few non-platform-compatibility-related updates to this release.  sqlr-export supports CSV output now.  I had a need for that so I added code for it.  sqlrsh now reports the correct version.  It had been stuck at 0.22 probably since release 0.22.  sqlrsh also supports multiple queries on the command line.  You can now do:

sqlrsh -id myinstance -command "select * from table1; select * from table2"

I'd long thought you could but apparently hadn't written it that way.

The rest of the changes are either bug fixes, platform-compatibility updates or changes under the hood.  If the version you're using is satisfying then there's probably no pressing need to upgrade.

Full ChangeLog:

  • sqlrsh has long reported the wrong version, fixed that
  • added csv support to sqlr-export and a -format xml|csv option
  • replaced snprintf calls with charstring::printf calls
  • added a test for sys/vnode.h to work around an issue with perl on SCO OSR6
  • added workarounds for old versions of sqlite without sqlite3_malloc and sqlite3_free with char * argument
  • fixed error with sqlserver where "describe " would return columns in alphabetical order rather than the order they are in the table
  • the code freetds uses to get column names from MS SQL Server works for temp tables now
  • added informix->mssqlserver translation for select into queries
  • added translation to convert ||'s to concat() calls or +'s
  • added a long-missing "delete clientsock" to sqlrlistener that could cause a crash after enough time
  • various documentation tweaks
  • added handling for "select into" queries to temptableslocalize/sybaseize translations
  • fixed various crashes that could occur when a query filter was used
  • updated the query routing and filtering doc
  • sqlrsh supports multiple commands on a single line or on the command-line now
  • exit/quit in a sqlrsh script aborts the script now
  • improved date/time parsing for translations and result-set conversions
  • fixed lots of cases where const char *'s were being deleted
  • added 64-bit build support for windows
  • combined sqlrclient files for faster compiling
  • converted \t to actual tab in regular expressions for posix-compatibility

Rudiments - 0.43 is out

As recently promised, Rudiments 0.43 is out.

This release of rudiments features some significant changes under the hood and a few visible ones.  Most of the changes center around fixing compatibility issues with older platforms and compilers.  Some are in the spirit of general code cleanup and decluttering.  Some were to make the code simpler and smaller.

The most earth shattering changes were the consolidation of the daemonprocess class into the process class, the renaming of the mutex class to threadmutex and the renaming of *clientsocket and *serversocket to *socketclient and *socketserver.  The daemonprocess class has long been quirky and needed an overhaul.  The mutex rename was to avoid namespace collisions and the socket class renames were to comply with a general client/server class naming scheme I've been meaning to do for years.

Some method names were changed to avoid namespace collisions as well.  There are surprisingly many platforms that use macros (#defines) to alias one function to another and it doesn't matter if you use C++ namespaces or not, on those platforms, you'll get collisions.  Terrible!

Speaking of namespaces, I'd long had goofy macros to use a rudiments namespace if the compiler supported them and not use them if the compiler didn't.  After fixing all the other name-related collisions, I eventually realized that the rudiments namespace wasn't necessary at all and removed it altogether.  This also helped with older platform compatibility.

Rudiments aims for compatibility.  Apps built on it might not be so compatible but Rudiments itself ought to be.  At least that's one of my goals for it.

Other semi-earth-shattering changes include the addition of printf-like methods in the charstring, filedescriptor and variablebuffer classes and inclusion of the proper header files high in the chain to make sure NULL is defined (correctly even, on older platforms).  Another goal that I've had for a long time is to be able to write programs that don't include any system header files.  Everything I need should be wrapped in Rudiments and platform-agnostic.  Ideally.  Well, I'm a lot closer.  I don't have to include stddef.h for NULL or stdio.h for any of the printf functions.  Yeah, I know it's C++ and by rights I ought to be using cout but that's a-whole-nother story.  At any rate, most of my programs, including most of SQL Relay, don't need to include any system headers.  Not all the way there yet, but getting much closer.

The rest of the changes are obscure, platform-specific fixes or workarounds.  You ought to see the hack I put in for platforms that don't provide a vsnprintf.  Terrible.  Definitely going to rewrite that someday.

Full ChangeLog:

  • added a workaround for platforms that don't support blocking/non-blocking modes for sockets
  • added *printf wrappers to charstring, filedescriptor and stringbuffer classes
  • added test/include for sys/signal.h for platforms that need it
  • added support for uadmin with int vs. char 3rd argument
  • added snooze and retry if fork fails with EAGAIN
  • moved safePrint and printBits methods into the filedescriptor class
  • added a test and handling for fsetxattr with non-const third parameter
  • "hid openssl" by abstracting SSL types and moving includes into .cpp files
  • fixed erroneous end-of-buffer address when creating a variablebufferis with initial contents
  • added option to disable string cache in xmldom class
  • added "data" member to xmldomnode class for attaching app-specific data
  • improved codetree speed
  • fixed a "char" on ARM issue
  • fixed a longstanding issue where variablebuffers could get extended too much sometimes
  • renamed system class to sys to avoid collisions
  • added -Werror to configure-time tests for older systems where g++ doesn't return an error for an implicitly defined function
  • removed the rudiments namespace for compatibility with older compilers
  • replaced a bunch of unsigned long and long with [u]int(32|64)_t
  • removed the timezonefile class
  • renamed mutex class to threadmutex to avoid collisions
  • wrapped setsid() function
  • refactored and simplified dictionary and linkedlist classes
  • if thread detection fails it just disables threads now rather than erroring out
  • thread support is displayed in configure summary
  • refactored xmldomnnode::print() methods, added one to write to a filedescriptor
  • renamed memorypool methods malloc, calloc and free to allocate, allocateAndClear and deallocate to avoid collisions on platforms that define malloc and calloc using macros
  • fixed lots of cases where const char *'s were being deleted
  • renamed *serversocket to *socketserver
  • renamed *clientsocket to *socketclient
  • added 64-bit build support for Windows
  • refactored daemonprocess class and moved its methods into the process class

Tuesday, September 24, 2013

Coming Soon...

The next release of SQL Relay is just about wrapped up.  There is testing and documentation left to do but not much of either really.

So what's going to be in this release?

No major visible changes but quite a few under the hood.  The Rudiments library upon which SQL Relay is based has had a major overhaul and SQL Relay has benefited somewhat from this in the platform compatibility department.  I've long had a goal of not having to include any native header files or call any native functions.  Everything should be wrapped and platform compatibility should be purely a function of porting Rudiments.  I'm really close to that now.  I think SQL Relay calls ceil() and popen() and one or two other native functions but that's about it.  Probably one or two more releases and they'll be out of there too.

Speaking of platform compatibility, the next release will support a 64-bit build on Windows.  It will also run on a long list of non-x86 NetBSD, OpenBSD and Debian GNU/Linux platforms, a couple of old Redhat platforms and SCO OpenServer 6.0.

Why?

I'm a bit of a retrocomputing nerd.  It's fun to run new systems on old (emulated) hardware and old systems on modern (emulated) hardware.  It's also fun to try to get my software running on those platforms.

There are a few legitimate benefits.  The first is exposure to lots of odd compilers.  I recently found all kinds of cases where I was deleting consts and returning values from functions that were supposed to return void just because an older compiler complained about those things.  There were a few cases where a comparison was always true because the variable on the left side was unsigned.  The cleaner the code is, the more likely the code is to compile on a platform that I don't have access to.

There are also platforms that are gaining popularity and it's good to support a version of them.  ARM chips are all over the place these days.  I don't have a Raspberry PI but I can run Debian-armhf in qemu.

I'm going to start chronicling  my adventures in emulation.  Someone might find them useful.

There are a few practical changes in this upcoming release.  sqlr-export will support CSV output, in case you need that.  I did.  sqlrsh's -command option will support running multiple queries.  It should have in the past, but apparently it didn't.  It will now.  A lot of the docs have been updated with little pictures and better examples.  Some of that should already be on the web actually.

Coming soon!

Monday, August 5, 2013

libfirstworks - 0.1 is out

Announcing...

libfirstworks

libfirstworks eases development and deployment of SQL Relay client apps on Windows by combining Rudiments and the various SQL Relay client libraries into a single DLL. At first, you still have to actually download and build Rudiments and SQL Relay and then build the libfirstworks DLL, but once you have, you can deploy the libfirstworks DLL rather than having to deploy all of the individual Rudiments and SQL Relay DLL's. You can also link your apps directly against it.

There are some caveats and details, so go to the website and read the documentation.

SQL Relay - 0.51 is out

Not a whole lot of new stuff in this release, mainly updates to support newer versions of operating systems, databases and compilers. Applied some patches. Some of the database install docs were updated. An obscure date conversion bug was fixed and documented.

Changelog:

  • fixed minor perl DBI driver documentation issue
  • added support for oracle 12c
  • added support for db2 10.1 and 10.5
  • updated docs for oracle 12c, db2 10.1 and 10.5 and sybase 15.7
  • modernized database installation docs a little
  • fixed db2/sybase detection to take the system architecture into account when searching for libs
  • applied various build-related patches from Ville Silventoinen
  • fixed rpm->rpmbuild in installation docs
  • added ora-00020 to list of errors that will cause the oracle connection to attempt to re-login, per Ville Silventoinen
  • fixed some MS SQL Server/dateddmm parameter issues and updated docs to explain the remaining issues

Rudiments - 0.42 is out

Not much new at all in this release, just some fixes for some compiler issues and a documentation fix. Really minor stuff.

Changelog:

  • made a "char" array in the charstring class explicitly "signed char" for arm and similar platforms where "char" is actually "unsigned char"
  • fixed variadic macros in codetree class for MSVC
  • fixed rpm->rpmbuild in installation docs

Friday, February 22, 2013

Rudiments - 0.41 is out

Rudiments version 0.41 is now available.

This release adds support for Syllable OS and includes a few other minor bug fixes and improvements.

SQL Relay - 0.50 is out

SQL Relay version 0.50 is now available.

This release mainly fixes a few issues relevant to users of Microsoft SQL Server.

The following client C++ API methods, equivalent methods and functions in other API's and equivalent commands in sqlrsh now work properly with SQL Server via FreeTDS:

  • getDbVersion()
  • getDatabaseList()
  • getTableList()
  • getColumnList()

The FreeTDS code has been updated as well. FreeTDS (and the TDS protocol in general) doesn't support fetching multiple result sets simultaneously through the same connection. Any attempt to do so results in the error: "Attempt to initiate a new Adaptive Server operation with results pending." SQL Relay has always attempted to work around this by caching the entire result set on the client by default but there was a subtle bug that could cause the error to still occur under certain circumstances. That bug has been fixed and everything now works as expected. I added an entry to the FAQ about it as well.

Parameters have been added to sqlrelay.conf for reformatting dates in the result set as well. Many databases provide limited or no control over the formatting of dates and times in the result set. This is problematic when migrating from one database to another if the databases have different date/time formats and the application expects a particular format. To help remedy this the datetimeformat, dateformat, timeformat and dateddmm parameters have been added to allow dates in the result set to be reformatted by SQL Relay before they are sent to the client.

Various other bug fixes and updates have also been made:

  • fixed the sqlrserver-config script to return data for libsqlrserver rather than libsqlrconnection
  • added a test for SQLROWSETSIZE for odbc that doesn't define it
  • fixed a NULL dereference in custom_nw logger
  • fixed an uninitialized variable (sqltr) in sqlrcontroller class
  • fixed a memory leak in the freetds version-checking code
  • added new search paths and default prefix for Syllable OS
  • added support for null-terminated result bind lists to the mysql drop-in library so apps can optionally bind fewer columns than come back in the result set
  • added configure tests for php-config-"version"

Monday, January 28, 2013

SQL Relay - 0.49.1 is out

This release fixes two bugs:

  • sqlrserver-config returns the proper library now
  • there was a NULL-dereference in the custom_nw logger module

If you don't use either of those things then there's no need to upgrade.