Version 1.0.2 of Rudiments, the C++ class library for developing systems and applications, is now available.
There was actually an unannounced 1.0.1 as well. 1.0.1 mainly contained fixes for OpenSSL 1.1.x. 1.0.2 just adds another SSL tweak that's required by Fedora for packaging. No pressing need to upgrade unless you're having trouble building on a platform that uses OpenSSL 1.1.x.
OpenSSL 1.1.0 introduced several API-breaking changes, and with distributions starting to use it, I found myself scrambling to get my software working. I found various docs stating that 1.1.0 changed things, but nothing that clearly stated: "if you have this now, then change it to this..." So, that's what I'll do here, and hopefully it will help you if you find your app in the same situation.
Deprecated "method" FunctionsLots of "method" functions like TLSv1_server_method() and SSLv23_client_method() and TLSv1_2_method() are deprecated in OpenSSL 1.1.0, and if you compile your software with -Werror, then that presents a problem.
All you can call now is one of:
If your code allowed the user to select a SSL/TLS mode then it can't do it by calling a mode-specific function any more. It'll have to select the mode some other way that I'm not aware of. The new methods negotiate the most modern mode supported by the client and server. It's not immediately clear how to override that, or if you can. If someone knows a way, please leave a comment below.
It's easy enough to replace function calls, but if you want to support older versions, then you have to add tests for the old and new functions to your configure script and add all kinds of #ifdefs, and so forth. Actually, since various methods like these have been added over the years, you may already have such tests and #ifdefs (I did) and it won't be so bad.
Deprecated ASN1_STRING_dataunsigned char *ASN1_STRING_data(...);
has been replaced with
const unsigned char *ASN1_STRING_get0_data(...);
This is a straightforward replacement, but note the const'ness of the return value of the new function. If you're using C++ then you may have to change the type of the variable it's being assigned to (or un-const it).
Again, this is a brand new function, so, configure tests and #ifdefs.
Opaque EVP_PKEYThe EVP_PKEY struct (public key struct) is opaque now. The header files declare that an EVP_PKEY struct exists, but the definition is hidden up in the library. The error you get (with g++ at least) is fairly obtuse though, complaining about a forward declaration and an invalid use of an incomplete type. The error just means that the struct is opaque.
Fortunately there are a bunch of functions that return the various components of the struct.
I had to replace code to get the key type like:
EVP_PKEY *pubkey = X509_get_pubkey(c);
...
int t = pubkey->type;
with code like:
EVP_PKEY *pubkey = X509_get_pubkey(c);
...
int t = EVP_PKEY_base_id(pubkey);
Note that there IS an EVP_PKEY_type() function, but it does NOT return the "type" member of the struct. Ha! Not super intuitive, but the change is still straightforward.
Similarly, I had to replace code to get the key itself like:
EVP_PKEY *pubkey = X509_get_pubkey(c);
...
const unsigned char *key = pubkey->pkey.ptr;
with code like:
EVP_PKEY *pubkey = X509_get_pubkey(c);
...
const unsigned char *key = EVP_PKEY_get0(pubkey);
I'm not sure when those functions were introduced, but they appear to be available in OpenSSL 1.0.x but not in OpenSSL 0.9.x, so again, if you want to support really old versions, then you'll have to add configure tests and #ifdefs.
Opaque X509The X509 struct is also opaque now. Like the EVP_PKEY struct, you'll have to use functions to access its members.
The one case I needed was slightly trickier though.
In the past, I was getting the signature algorithm name like:
X509 *c = SSL_get_peer_certificate(ssl);
...
char sigalg[256];
OBJ_obj2txt(pvt->_sigalg,256,c->sig_alg->algorithm,0);
But I had to change that code to:
X509 *c = SSL_get_peer_certificate(ssl);
...
char sigalg[256];
OBJ_obj2txt(pvt->_sigalg,256,OBJ_nid2obj(X509_get_signature_nid(c)),0);
Which is a little contrived. In the old code, c->sig_alg->algorithm is an "obj", and I could pass it directly into OBJ_obj2txt. In the new code, I have to get the "nid", convert it to an object, and pass that in. Ideally I'd just get the object to begin with, but there doesn't appear to be an obvious way to do that.
As a side note, the function name X509_get_signature_nid() was a little confusing at first, because it appears to return the "nid" of the signature _algorithm_ as opposed to the signature itself. But I guess the signature itself wouldn't have a "nid", so it makes sense, but it wasn't immediately intuitive to me.
Again, X509_get_signature_nid() was introduced sometime in the 1.0.x series, so you'll have to add a test for it and #ifdefs if you want to support older versions.
ConclusionThe opacity of the structs ought to help OpenSSL be more maintainable in the future. They ought to be able to change the code at will now, without breaking the API or ABI, and that's always good. I did the same kind of thing with my Rudiments library way back, and recently to SQL Relay for that exact reason. Hopefully it will work out well for both of us.
I'm not sure about the TLS_*_method() thing though. It seems like there really ought to be a way to choose a specific SSL/TLS mode. Perhaps there is though, and I just didn't see it. Somebody let me know if they figure that one out.
There are a lot of reasons to use SQL Relay, but one of the most popular is just to improve throughput, usually measured in queries-per-second.
Logging in to most databases takes a noticable amount of time...
So much time, in fact, that transient apps (such as web-based applications) which start up, connect, query, disconnect, and shut down to deliver each bit of content, often take longer to log in than to run the queries. As a result, overall throughput (queries-per-second) is generally low unless many queries are run per connection.
SQL Relay maintains a pool of already-logged-in database connections, and logging in to a connection pool is substantially faster than logging into the database directly.
As a result, overall throughput (queries-per-second) should be maximized, and shouldn't depend nearly as much on the number of queries that are run per connection.
In theory.
Anecdotally this is the case and informal testing has always confirmed it. But, I wanted hard data and pretty graphs to prove it. Also, it had been a long time since I'd done any formal performance testing. For all I knew, I'd done something recently to make everything run horribly.
Of course there are popular tools for benchmarking apps and databases. JBench, for example. But they all use intermediate layers - JDBC, ODBC, web pages... I wanted raw comparisons: the database's native API vs. SQL Relay's native API. I also wanted it to be easy to run from the command-line, and automatable.
Months ago, I'd begun writing a program called "sqlr-bench" which aimed to run a battery of tests against the database directly, run the same tests through SQL Relay, collect stats, and produce a graph. It had to support every database that SQL Relay supported. It needed a dozen or more tunable paramters... It was a fairly ambitious project. I started and stopped work on it several times but I eventually got it working well enough.
It creates a table, populates it, and runs a series of selects. First it runs 1 select per connection, then 2 per connection, then 3, etc. It times all of this and calculates queries-per-second as a function of queries-per-connection. The whole process is done first against SQL Relay, and then directly against the database. The data is then correlated and gnuplot is used to produce a graph.
Various parameters are configurable, but the defaults are:
I ran the tests against each database, and against instances of SQL Relay running on the local machine (over a unix socket).
Theoretically I figured I should see...
Performance against the database directly was exactly as I expected, and it generally seemed to level off at about 20 queries-per-connection.
SQL Relay actually does have a small ramp-up period (2 or 3 queries per-connection is slower than 4 or more), which I didn't expect, but the number of queries-per-second were otherwise uncorrelated with the number of queries-per-connection, which I did.
Overall though, SQL Relay did not immediately perform as well as I expected. In fact, for most databases, there was a "break-even" between 5 and 10 queries per-connection.
While apps that use SQL Relay would still likely perform better than if run directly against the database (how many web pages run more than 5 queries that return 256 rows each), I really expected it to perform better.
Thus began days of profiling and analyzation. It appeared that, indeed, over the years new features had steadily introduced inefficiencies.
In most cases, things just needed to be reused instead of reallocated. Everything from database cursors to byte arrays. I replaced a few regular expressions with simple string compares, and made a few case-insensitive string compares case-sensitive. There were also some alignment and padding issues that prevented optimized strcpy/memcpy from being used. My socket buffer sizes were sub-optimal...
Lots of little changes.
In the end, I was pleased with the results. Most of them, anyway.
Informix:
Oh yeah, that looks great.
MySQL:
Still looking good.
Oracle:
Not bad, but why the falloff?
DB2:
It's good where it counts, but why does DB2 eventually outperform SQL Relay?
SAP/Sybase:
Again, it's good where it counts, but throughput is generally low. And again, why the falloff?
PostgreSQL:
My goodness, PostgreSQL is fast!
I checked the code over and over to make sure my test program was legitimately running queries and not just erroring out or something. It was. Why is it so much faster than everything else? Strace showed that the socket is in non-blocking mode and that they do a poll() then read(), basically blocking manually rather than doing a blocking read. My preliminary tests with a similar strategy didn't show much improvement with that over a blocking read though. Non-blocking poll()/read() is a good way to multiplex reads through a single thread, but if you're only reading from one socket at a time, I wouldn't expect it to help at all.
Very strange.
SQLite:
Well, I guess I shouldn't really be disappointed that SQL Relay is slower than SQLite :)
FreeTDS:
82 queries-per-second??? Does SQL Server limit throughput?
The falloffs are strange. It's odd that it's so inconsistent. There doesn't appear to be a memory leak. Heap fragmentation?
Performance against DB2 and SAP is odd. It shouldn't break even so quickly.
Performance against PostgreSQL is the most strange. Whatever they're doing, SQL Relay needs to be doing it too.
And what's the deal with MS SQL Server?
These things deserve investigation.
I ran tests with SQL Relay on a server between the client and database too. The results were similar, but there was more of a ramp-up. I should publish those results too.
To perform these tests, I configured 2 Fedora Linux VMware VM's. One for the databases and one to run SQL Relay. Both VM's had 1 CPU. The database VM had 2Gb of RAM. The SQL Relay VM had 512MB.
The DB VM ran modern versions of all databases, in their default configurations, without any performance tweaks.
I built Rudiments and SQL Relay without debug and with -O3 optimizations like: CPPFLAGS="-O3" ./configure
The SQL Relay configurations were all pretty standard too. For example, the Oracle configuration was something like:
<instance id="oracletest" port="9000" socket="/tmp/test.socket" dbase="oracle"> <users> <user user="test" password="test"/> </users> <connections> <connection string="user=...;password=...;oracle_sid=..."/> </connections> </instance>
For the PostgreSQL tests I disabled typemangling.
The sqlr-bench program can be found in the test/bench directory of the SQL Relay distribution. It must be run from that directory. I ran it with default parameters. There's a benchall.sh script there too, which starts and stops the appropriate SQL Relay instances and runs sqlr-bench. I actually used that script rather than running the tests individually.
Version 1.0.0 of SQL Relay, the powerful connection pool, load balancer, query router, and database proxy, is now available.
At great length: 1.0.0!
The client APIs has been stable for a long time. The server APIs have been stable for long enough. Recent efforts during the past few releases updated the internal structure such that significant internal changes can be made without affecting the API or ABI. Yes, it's definitely time for a 1.0.0 release.
Of course, no project is ever feature-complete. There are still plenty of tasks in the backlog, but it should be possible to implement most, or all, of them without breaking the API and ABI.
New FeaturesThe most noticeable new feature in this release is that all DB abstraction layer drivers support the same set of connect-string options, including TLS and Kerberos options, as well as columnnamecase, resultsetbuffersize, dontgetcolumninfo, nullsasnulls, and lazyconnect options. The options are named subtly differetly for each driver, following the conventions for the DB abstraction layer, but all options are present.
ChangesThis release mainly features a bunch of internal changes though. Most improve performance or stability in one way or another.
The most outwardly noticeable change is that column names are case-sensitive now when getting a field by name. The docs never promised case insensitivity, and the performance improvement is notable. Also, there are options for upper and lower-casing column names, if you need them to be converted one way or the other.
Another semi-noticeable changes is the removal of calls to mysql_stmt_store_result/mysql_stmt_num_rows in the mysql database connection module. Actually the removal of them isn't the noticeable part. Rather, that since they have been removed, the row count is no longer available immediately, if a result-set-buffer-size is being used. This is consistent with almost all other databases though, and by default, no result-set-buffer-size is used.
Another, nearly unnoticeable change, is that by default now, connections start with 1 cursor, but will scale up, on-demand, to 5 cursors. Of course, maxcursors can be set higher or lower than 5, but it defaults to 5.
Most other changes were fixes for subtle bugs or tweaks to improve performance.
Full ChangeLog follows:
Version 1.0.0 of Rudiments, the C++ class library for developing systems and applications, is now available.
Yes, that's right, 1.0.0. It's been a very long time coming, but Rudiments has finally become stable enough to call it version 1. The last few releases have been aggressively targeting this. Now it's time.
So, is Rudiments complete? Hardly. But, the core API has been stable enough, for long enough. Also, the general design has proven to make it easy enough to modify the internals quite a bit without changing the API or ABI. For the most part, anyway.
So what's new in this release? Lots of little fixes. Several features that I've been putting off adding. Several bugs that I've been putting off fixing.
Probably the most significant change is the addition of an avltree class and use of it in various places.
Other big changes include refactoring of the thread class such that the thread and process classes both follow the same spawn/detach/exit/wait paradigm, and refactoring of the filesystem class so that it and the file class both follow the same open/getCurrentProperties/close paradigm.
An md5 class was also added. It and the sha1 class are now children of a hash class.
Full ChangeLog follows: