I’ve been working with technology for a very long time. Long enough that I have seen the words “data base” mean exactly that: a base body of data. The term implied no specific format, no particular layout, or even an engine to process it. Just raw data used for a given storage of information.
Just Databases
Early on in my mainframe days I worked at the Kenedy Space Center for NASA here in Florida. My first foray into databases on a mainframe was IBM’s Information Management System (IMS). It was a hierarchical database and transaction manager that is still in use, even though it wa developed in the 1960s to track parts for NASA’s Apollo program. Yes, primitive now, but at the time it eventually became a staple for over 95% of the Fortune 1000. It’s still in use, processing billions of transactions every day, even now.
I also used a data base software application called SQL/DS on the mainframe, which became DB2. I still have a book around my office somewhere with the “System R” logo on it where it was developed at IBM.
In all of those cases, data was just data. We moved from COBOL flat-files to an engine over relational tables, but it was just data. Applications (green-screen terminals, thank you very much) put data in, and took data out. The function of the “database” was to do that as quickly, efficiently, and securly as possible.
Surely You Can do Better than That
And then I disovered and managed and used Oracle. Oracle had *Forms*. It had a reporting system , all built in. And then they added Java! And all kinds of new data types outside text, dates and numbers.
It was this kind of thing, pioneered by companies like Oracle, that allowed organizations to think about centering their data estate (that became a thing) around a database platform.
SQL Server – the Little Engine That Could
In the late 1980’s, in addition to the mainframes NASA used, we also had IBM’s OS/2. I joined the team that focused on this product, and one of the interesting applications it had was a command-line based database called “Microsoft SQL Server” – version 1.0. It was even client-server, meaning the database engine could run on one computer and another computer (the client) could access it over the (Token Ring) network. I was fascinated with the “Transact-SQL” dialiect of SQL it used.
IBM and Microsoft broke up over OS/2, and I started my journey with Windows operating sytems, including “Windows NT Advanced Server”. By now I was using SQL Server version 6+. While Microsoft added Stored Procedures and other engine features, they were just that, engine features. For a database engine.
Except one: The opening of the system with the Replication feature. The ability to send some or all of the data in one database to another using a dedicated system. Data could now be used differently by entirely different application needs. And companies started thinking about the database engine a little differently.
SQL Server – Let’s Be a Platform
Then came SQL Server version 7 – a completely rewrite of the platform but with (mostly) backwards compatibility. It included not only a highly capable database engine with a wide suite of programability features, but Microsoft started adding things right in the box that drastically extended the ability of the system to host business-critical data estate functions. One of the most impressive was “OLAP Services”, which as the name suggests, was a full Business Intelligence (then a pretty new term) server. To fill, process and orchestrate that system, they added “Data Transformation Services” or DTS, a feature that was pushed by us administrators far past its intended purpose of Extract, Transform and Load (ETL).
SQL Server – The Kitchen Sink
Fast forward to SQL Server 2005, and we got all the “S’s”: Analysis Services for BI, Reporting Services for, well, reporting, and Integration Services which is actually hard to pin down. Integration Services does so much that I’ve seen pure Oracle shops buy SQL Server just to use Integration Services for everything from data transforms and movement to a full job system.
Then I Joined Microsoft on the SQL Server team. What an eye-opener for me. This product was massive and had grown FAR beyond a database. You scaled out the data functions to multiple servers, with complicated disk arrangements and multiple network cards.
And from there, more and more features showed up in heavy and rapid fashion. We added every data type you can think of: you could store whole Microsoft Office documents, music, pictures, and search them all. Full text. In-memory. We added not only geographic datatypes, but geo-spatial datatypes. I had one client using that feature to map out their entire warehouse, down to the shelf dimensions so they could find exactly which boxes would fit where for the most efficient storage. In a database!
I moved from the SQL Server engineering team to work as a technical pro at Microsoft in the field, I then joined the team that built Azure, and then I joined the team that was implementing Machine Learning at Microsoft. And of course then that went into SQL Server. You could (and still can) run your own self-contained AI constructs and solutions and applications and never leave SQL Server.
I was also part of the ill-fated “SQL Server Big Data Clusters” – a system ahead of its time that could scale from on-premsises to the cloud, with so many features running in Linux an Kubernetes no less.
Companies large and small, around the world, built their business logic, data processing and analysis, reporting, and now even AI into the SQL Server product.
But it got a bit, well, overwhelming. Having everything, and I mean everything, in the box can make for complex upgrades, complicated troubleshooting, and make it hard to keep up with what the system can really do.
Adding to this, some of the features are more mature and stable than others. Some very much so.
SQL Server – Just a Database
Things, as they always do, have changed. Developers are now using scale-out, cloud-first, micro-services, per-server data processing, which feeds much larger more spread-out systems. Data storage is now separated away from (as in my COBOL days of old) from the processing engines that used it. Vendors – even Microsoft – tended to focus on larger distributed sytems.
The expensive licenses of SQL Server doesn’t hold the same value if you’re just using it as a database. Postgres is “free” (not really, but that’s the mindshare) and if your data is small you can use SQLite (love me some SQLite) and the stunningly amazing DuckDB. DuckDB, for me, holds the same excitement level as the SQL Server version 7 days. The little engine that could.
SQL Server – So Wither Forth?
In some clients I work with, they feel they are being left behind if they don’t move to the newest cloud analytics platforms currently battling for their data budget.
In some cases, after we do a deep audit of their needs, SQL Server still fills the bill quite well. Yes, it needs to be designed and implemented properly, but if you do, there’s a lot going for it. Yes, pricing is pretty high, but compared to cloud costs, it can actually be significantly cheaper.
So – will SQL Server be your “Platform in a Box” or cast aside for cheaper, smaller, faster “Just a Database” products? I’ll fall back here on the most famous saying in the Database Administrator (DBA) world: “It Depends”. Sorry, but there it is. I think for many, many shops, if you take the time to plan, implement and manage it well, SQL Server cannot be beat. It really can’t.
For some workloads, however, distributed storage and compute is a great idea. If you haven’t investigated with some of the major systems, I highly recommend you do. Oh, and while you’re at it, play around with DuckDB. It will amaze you with what it can do, directly over storage like Delta/Parquet. And while you’re at it, take a look at some of the newer systems that bypass the “Reporting” layer entirely. I’ve seen some lately that are an absolute sea-change.
Everything old is new again.