The ANTs Data Server was a SQL database that I worked on from 2001 to 2007. At the time it was arguably one of the most technologically advanced databases in the world. The brainchild of Cliff Hersch, ADS used lock-free multiversion concurrency control, an internal architecture based on cooperative multitasking, and compiler that compiled user queries to machine code. It used a set of threads (typically one per core) that each concentrated on doing small tasks and only swapped tasks at points where there was very little context to record so there was never a full-scale context switch, and the more parallel work you gave it, the better it performed. The storage management was designed so that most allocations and deallocations were just a few instructions and each task thread had its own allocation regions to minimize the critical sections associated with memory management.
In future posts I’ll get into details of how all of this worked. I’d like to thank Achim Bode (whose tenure at ANTs was pretty much coextensive) with mine for reviewing these documents and pointing out quite a few errors in my memory of the technology.
About the Server
We had benchmarks for ANTs Data Server that showed us running up to 80 times faster than other commercial databases of the day. Of course, those were for special work loads that ADS did especially well on, but for typical high-volume transactional workloads, we were typically 10 times faster.
Performance numbers like that may make you think “in-memory database”, but ADS was a full ACID database that stored all data on disk and had the usual persistence guarantees of disk-based databases (although for performance it did rely on having enough RAM to keep most of the working set in memory). ADS had read-committed and serializable transaction isolation levels. It had fully functional logging, checkpointing, on-line backup, and transactionally safe read-only hot standby servers with automatic failover from the clients.
ADS supported pretty much all of SQL 92 with additions from later SQL standards as well as many compatibility features for Oracle, Microsoft, Sybase, and Informix SQL as well as a stored-procedure language inspired by PL/SQL. It had B+tree indexes, a cost-based planner and a facility to gather statistics for the planner. ADS used ODBC as its native driver and could cache query prepares on the client to unload work from the server.
Some History
Unfortunately, ANTs Data Server never sold well in the commercial space to the great disappointment of the team who created it --one of the most talented and personable groups of people I have ever worked with.
Some time in the year 2000, Eugene Malik created a project in Microsoft Visual Studio to begin the implementation of Cliff’s revolutionary idea for Asynchronous Non-blocking Threads. I wouldn’t join the team until about a year later. In those days, all of the cool kids wrote enterprise software on Linux or Unix, but the choice to develop in Windows was a critical factor in our rapid progress. There were no development platforms on Unix or Linux that could compare with Microsoft Visual Studio for convenience or ease of use.
In fact, even today, a decade later, I’ve never found a development platform on Linux that is as good as the Visual Studio of 2001, although Delphi was also a tremendous development environment in those days and I never got a chance to try Kylix, the version of Delphi that ran on Linux. I haven’t used Visual Studio since then, but given Microsoft’s propensity over the last decade to favor novelty over consistency and usability, I expect that modern iterations of Visual Studio are probably not as good.
Leading the team at that point were Cliff Hersch and Jeff Spirn. Jeff was the only one on the initial team with real experience in database internals. It was hard for a small company to hire experienced database engineers from real companies like Oracle, IBM and Microsoft, and Cliff and Jeff had the philosophy that experience was less important because they were using a novel implementation. Consequently, their hiring strategy was just to look for smart people that they thought they would enjoy working with. Jeff used to say that when interviewing candidates he tried to answer these three questions: “Can they do it? Will they do it? Can I stand them while they are doing it?” Frankly, that’s a risky policy, but it worked out very well at ANTs.
What neither Cliff nor Jeff had was experience in managing a commercial software product. For that the company hired Girish Mundata who came on board a few months after I did, sometime in 2002, I believe. Girish had enormous energy and enthusiasm and managed to single-handedly change the culture from that of a research project to that of a project to produce a commercial product. Personally, I preferred the research-project environment that Cliff and Jeff had fostered, but at some point you do have to put out a product if you want investors to keep giving you money. It’s a cruel and heartless world.
We went through the usual pains of a startup and spent a lot of time in the endless rounds of the sales-driven development cycle:
1. we tell the sales team that we have X
2. the sales team goes out to sell X
3. the sales team comes back and says that they can’t sell X but they could sell Y if we had it.
4. the engineering team creates Y
5. Set X=Y and go to 1.
If you ever find yourself in that loop in a startup, then I advise you to run, not walk, to the nearest exit. That goes whether you are an employee or an investor.
I believe our first release was in 2003 and sales were practically non-existent. After five years of the sales-driven development cycle, our investors had had enough (they shouldn’t have put up with it that long) and the CEO responded by getting out of the server business and selling off the intellectual property in ADS to two different companies. In 2008, ANTs sold a source code license to Sybase to use ADS in their database products, but inside sources tell me it was never used. Later that year, ANTs sold ADS to Four J’s, a company that writes database applications based on Informix. Four J’s sold a descendent of ADS as “Genero db” and a lot of the original team went to Four J’s to continue working on the product.
The total revenue that ANTs saw from those sales was about 4 or 4.5 million --I don’t recall exactly. I estimated at the time that we sold ADS for about 1/10 of the amount that we had invested in creating it. Our team, which was never very large to begin with, was split into three groups: some went to Sybase, the some went to Four J’s, and some stayed on at ANTs to work on another database-related product (several of the Sybase and remaining ANTs team eventually ended up at Four J’s).
Those were some unpleasant times. Some of the team didn’t want to leave but ANTs was required by the terms of the sale to send to each of Sybase and Four J’s a team with the talent and knowledge to independently continue work on ADS. I was a manager at the time and had to help divide up the teams which was not fun since so many of the team were friends. And naturally there were hard feelings from some of those we chose to send off because they felt like they were being coerced. Upper management didn’t handle it well, and Sybase, at least, made no effort to help people feel better about it. Sybase later sued ANTs because so many of the people we sent there quickly left. I have no idea what their legal argument was for ANTs being responsible that Sybase couldn’t motivate their own employees to stay.
I stayed on at ANTs and became the Director of Development but soon left because I really didn’t enjoy management, wasn’t excited about the new product, and had an opportunity to work elsewhere on a shared-nothing distributed big-data database. For political reasons I couldn’t tell anyone where I was going, so I unfortunately couldn’t explain to my friends why I didn’t want to join them at Four J’s when I left ANTs. I think I made up some lame excuse about wanting to take a long vacation before deciding.
Genero DB, the Four J’s rebranding of ADS, is still apparently available, although almost none of the original crew is still working there and my understanding is that they stopped all development work around 2010 or so. Most of the people who developed ADS are now at Oracle, Sybase/SAP, Google, or other big companies and about half are still working on databases.
In a surprise move (to me, anyway), ANTs has bought ADS back from Four J’s. I don’t know what their plans are with the product.
Revealing the Secrets
Now, for the first time ever (not counting lots of casual conversations over the years) I shall reveal the secrets of the ANTs Data Server: how the non-locking architecture worked, how the compiler worked, how storage management worked, etc. I’ll publish in a series of posts about specific features or components.
continued...
No comments:
Post a Comment