Does Automation Kill the DBA “Store”?
In my opinion, automation will enhance DBAs. It will allow DBAs to provide more value to the business. Good DBAs will be fine. If you are a DBA with ten years of DBA experience who just repeats year one ten times, this might be a different story for a different blog post. Therefore, automation enhances the Good DBAs careers just like the sync button enhances turntablists but the sync button doesn’t replace good turntablists’ careers.
Follow along to see if you agree or disagree. I look forward to seeing your feedback in the comments section on this topic.
Some of my good friends know I have a passion for DJing on the ones and twos. A subset of those friends know I took a job in high school just so I could buy two Technics 1200s and records. In my free time, I started taking an online DJ Masterclass and it quickly jumped out to me that the sync button for turntablists, club DJs (see just like DBAs there are several different types of DJs) isn’t much different from automation for good DBAs. The sync button is a way to automate the tedious task of making sure two different songs are played at the same speed so they could be mixed together. In the past few years, I started to hear buzz that automation will replace DBAs so I wanted to write a blog post providing more details into a recent tweet of mine.
Saying the DBA's job is going away due to automation is just like saying the sync button will replace all DJs. #NotGoingToHappen
— John Sterrett (@johnsterrett) July 29, 2017
What is this sync Button You Speak Of? How does it relate to DBAing?
Going back to my high school days for a second. One of the core fundamentals of being a turntablist is using beat matching to keep the dance floor going as you transition from one song to another song. It took me months to learn this fundamental skill. I had to learn how to measure BPM (beats per minute) to see if the songs could even mix together. I had to learn beats (think of pages) and bars (think of extents) and learn how to mix the songs on the same beat on right bars. I had to learn how to adjust the speed of each of the two records to make the BPMs match on my own. When mixing, I had to make sure the two records stayed on beat during the mix. Syncing records isn’t the hard part of DJing, it’s just a tedious part.
Today, most DJ software packages have a sync button so this process can be automated. Now we can hit the sync button and the BPMs will match almost all of the time. What would you do if you didn’t learn the basics and relied on the sync button and the BPM for your song is wrong, or worse, your venue doesn’t have the sync button? I think this would be similar to having corruption occur and not knowing the basics behind DBCC CHECKDB, Backups and Restores. You won’t be keeping people on the dance floor for long, or have a successful DBA Career.
If you don’t know the basics, you will rely on the software and have no other options if and when the software doesn’t automatically work for you.
I Love Automation, I have loved it for years!
I love automation because it allows me to automate tedious tasks and focus on ones I love and which provide value. For now, I will talk about automation in general and will focus a bit later on some new features with SQL Server that help automate things.
I have been using automation as a tool in my toolbox for almost my whole DBA career. I have given talks on Getting Started with PowerShell for DBAs; my first PASS Member Session was on utilizing Policy-Based Management to automate basic health checks. You can even use 3rd party tools or custom scripts like this, this, this, this, or this to help with some automation. Just make sure they work for you and your environment. I have seen a nice automated SQL Server installer built a very long time ago that used JavaScript before JavaScript was cool. I bet it actually started with JScript (Insert painful JScript vs JavaScript memories from my web development days).
The language used doesn’t matter, what matters is what the process does.
At the same time, I have actually seen a software service provider application that in my opinion didn’t monitor transactional log backups correctly, yikes. I mention this not to scare you, but to remind you that you need to know the basics to know if an automation process works for your needs. If you don’t know what the automated process is supposed to do how do you know it is doing it correctly? Using the wrong automation tool, or using the tools incorrectly, could be very hurtful to your career. This looks a lot like the sync button for DJing to me. Automation tools/scripts also evolve and change so this isn’t a ‘set it and forget it’ thing either, in my humble opinion.
Good, experienced DBAs have been leveraging automation for years.
It has been a tool in their toolbag for quite a while, not a new thing. DBAs also know how to leverage it as a great tool instead of seeing it as a new tool that replaces their jobs. BIML is a great automation tool for Business Intelligence (BI) Developers. Does BIML automate good BI Developers out of their jobs? Nope. Does it help them automate so they can focus more time toward high-value tasks and get more things done? Yep, it sure does.
If automation is new to you as a DBA, or maybe you are starting out as a DBA, I suggest checking out some well-proven open source tools in a non-production environment. If you try to test out automation, several solutions include options to script out but not execute the tasks. This can be very helpful for learning (just like I used the script feature in SSMS to learn a lot about T-SQL behind the GUI). If you are looking for a place to start, I highly recommend dbatools.io . It’s amazing to look at all the things one could do with this single tool.
Can Self-Tuning Databases Completely Automate DBA Tuning?
Performance tuning is one of my favorite parts of being a DBA; therefore, there is no way I would skip talking about some of the great enhancements that allow SQL Server to tune your own environment in Azure SQL Database and the box product. It’s purely amazing to see some of the new cutting-edge tuning features that, in my opinion, are game changers that will help Microsoft further separate themselves from all other database management systems on the market.
Don’t hold your breath while you try to read about some of these great performance improvements provided just since SQL Server 2014. There is cardinality estimator changes, auto tuning for adding and removing indexes, adaptive plan changes, query store, resumable index rebuilds, heck good old Database Tuning Advisor gets better and wait for it…. Automate Tuning. Don’t take my word for it that these features will be amazing. Go check out Lance Tidwell’s about some of these new features in his PASS Summit talk in November. Actually, take a second to read his abstract. Notice the following, “We will also look at some of the pitfalls and potential issues that can arise from these new features.” I find it interesting that it looks like Automated Tuning and other features might not safely automate all your tuning needs.
I don’t foresee Self-Tuning Databases being able to self-tune everything.
A friend of mine Joey D’Antoni wrote a great blog post recently on Self-Tuning Databases. Let’s go by this statement in his blog post, “I’m sure this would take years and many versions to come into effect.” Let’s fast forward and assume this is fully in effect including even multiple features outside of execution plan changes. I will throw in an example, Artificial Intelligence that could find several common known T-SQL Anti-Patterns and rework the code for all of them so the optimizer friendly version executes.
Let’s take a look at why I do not foresee automated tuning fully replacing Performance Tuning DBAs.
The way people implement business processes with technology will prevent automated tuning from fixing everything in your environment. It could be hardware, database design, could be bad coding, bad anti-patterns in code, misuse of ORMs, ORMs struggling to your data model, etc. My point is, I don’t see performance tuning getting completely automated to the point where performance tuning skills will be obsolete. I am not alone, another good friend of mine, Grant Fritchey writes, “Finally, despite the fact of adaptive plans and automated regressions, you’re still writing code that NO amount of automation can fix.”
Will the Cloud Automate DBAs Out of their Jobs?
The cloud will not automate DBAs out of their jobs. First, let me start with the fact that I love the cloud. I write about Azure, and I give presentations on Azure SQL Database. Azure SQL Database has grown a lot over the years and almost has all the same features as the box product. Microsoft has also added great additional features to differentiate themselves from their competitors like Query Performance Insight, Automated Tuning to add and remove indexes, Geo-Replication and more to help make Azure SQL Database my favorite platform as a service offering. Soon, Azure Managed Instances of SQL Server will even make admin tasks easier for DBAs as they will be able to apply their skills directly with fewer changes like the box product.
Even though it is a fantastic time to leverage cloud services not all companies are going to put all of their critical data there. Many are still using SQL Server 2005. Some still work with mainframes. Using the same logic, shouldn’t those DBAs have been automated away years ago?
migrations to multiple cloud providers and even helped clients spin up their own cloud environments that they offer as a service, while also providing them Remote DBA Services as well.
I know what you’re thinking. Why do I need DBA help if I am in the cloud?
If someone is using a managed service for their databases in a platform as a service model (PaaS) like Amazon RDS or Azure SQL Databases, why would they need a DBA to manage the databases, isn’t the managed service managing them for you? Well, it’s simple, your needs might not be completely aligned with the service level agreements that are offered. Take a second and read Amazon RDS SLA and then Azure SQL Database SLA. Keep in mind, I am not telling you to not leverage these services; I just want you to know what is covered and what gaps DBAs can still fill. If you also take a look at Amazon RDS Best Practices you see several things that can keep DBAs busy.
Did you notice any mentions of your data in the SLA’s?
Is anything mentioned in writing about what will happen when stack dumps occur or data corruption happens? Did you see anything written on how you get notified if corruption happens and what actions will be taken on your behalf that might cause data loss, or if you should make business decisions and take action?
Thinking this could not happen to you? I ask because while performing our health check on a non-Microsoft cloud PaaS provider for a new client, we detected stack dumps… and this was the first time our client was notified, yikes!
The DBA Profession Is Not Going Away
To wrap this up, I don’t see the DBA profession going away. The DBA role has pivoted and evolved constantly over the years. I don’t see that changing. It’s one of the many reasons why I am passionate about helping clients solve their business problems with SQL Server. While some think machines are close to automating away your job as a DBA, I would hate it if new DBAs or experienced DBAs see written content like this and assume they need to change their career. Not only do I respectfully disagree, I do not see this happening and I am not alone.
When you want to focus on growing your skills, I see several benefits towards investing time both inside and outside of the role of a DBA.
I think automation will enhance good DBAs and reduce time spent on repeatable trivial tasks allowing DBAs to move faster and provide more value. I don’t see good DBAs being replaced by a handful of PowerShell scripts. To me, this is the DBA version of the sync button that allows turntablists to spend less time on preparing a mix and more time to providing a better experience to the dance floor.
Seriously, don’t just take my words for it. NPR predicts only a 3% chance of automation taking over DBAs’ jobs. Even better, take a look at US Bureau of Labor Statistics you will see DBAs are not losing their jobs. When the tedious parts of both database administration and DJing have been automated away, it leaves people like me more time doing what we love. Embrace automation and find ways to add value so the business loves you.
Don’t make DBA stand for Don’t Bother Asking or you might get replaced by another DBA or a managed service provider.
Very well said, all of it. I’ve preached about this for quite some time and I have to fight it in some companies when I go in. CIOs telling me that they don’t know if they even need any DBAs cause they’re taking everything to the cloud.
Thank you for the complement and for sharing your thoughts, Sean. You are defiantly not alone.
John, Thanks for this great article. Recently I have come across this attitude that DBA’s will be phased out because of automation. My view is aliglined with yours in that automation augments a DBA’s skills and reduces tedious task from consuming a DBA’s time. Personally, if I have to do a repetitive task more than 3 times I am looking for a way to automate it by the 4th. I remember when MSSQL ver7 was realeased; the same attitude was floating that a DBA role be phased out. It’s absurd to think about it now, even with all the game changes SQL7 introduced, that a DBA would not be needed. I think it’s just absurd now to think that way.
Michael,
Thank you for sharing your thoughts. I love how you mention SQL 7.0 its amazing to see all the cool things Microsoft has provided since then to help DBAs get more done and show more value. Its a fun time for DBAs. We have so many opportunities to connect, share and learn.
Regards,
John
Excellent post John. Like you I have been a fan of automation for many years as a DBA. Dating back to 2011 when I read John Sansom’s article https://www.johnsansom.com/the-best-database-administrators-automate-everything/ It is a battle that is fought every day in terms of professionals thinking their jobs will dissipate. I firmly believe that, while our roles may change over time, that it is okay for it to happen. It’s a must actually. What most people have difficult is dealing with said change. Any where you go and in any shop change is difficult to have; just like some companies going to the cloud with their data. I think you hit the nail on the head in terms of knowing the groundwork and what it takes to get there; automation is key and will continue to be key to run as efficiently as one possibly can. Great article ~ +1000 points for DJ skills.
Chris,
Thank you for sharing your thoughts. You bring up a great point about people dealing with change.
Regards,
John
And yet in my DBA automation Precons, only 5{ba73dddca7982ae34a1fd88deacd14d81bde8c1566bcc84b8da0bfb5e7162b7b} of DBAs claim to do any scripted installs. But that’s just an anectdote. Maybe that’s why they chose that Precon
Those people are going to be surprised when a new hire or consultant is added to the team and shows the director that half of what the Next, Next, Next “I can only use GUI” DBA has been wasting his time on. I’d provide links to the experts who advocate that DBAs should never learn command line, but I don’t want to give them more traffic.
I think your position has put this question into a Yes/No situation. Like all things in IT, in real life we don’t have the the luxury of boiling down answers to “always yes” or “always no.” You’ve used worlds like “completely” “forever” or implied them. Those of us who are talking about automation aren’t thinking of it as an all or nothing question.
The future of the DBA isn’t a question about whether automated tuning is good or bad. The future of the DBA isn’t a question of whether automated backup and restore testing should never or always be done. It’s about having an architect’s mindset of applying cost, benefits and risks to decide when and how to best support the business that employs us.
The fact is that we should embrace, not be fearful of, leveraging compute to do the tasks that humans are bad at and leverage humans to do the tasks that we are good at.
Over the last 35 years of working as a data professional, I’ve seen those sets change a great deal. There are tasks DBAs did 10 years ago that engines do now. Someone can still turn off engine assisted tuning and continue to do those tasks by hand, with a GUI, in a fire-fighting mode because they don’t trust computers. But eventually someone who understands how they work, knows the CBR of using them and knows how to use them effectively will be more valuable to the business. The new guy will be more valuable. And will also be able to spend more time converting the data processes from reactive to proactive.
It’s not a Y/N question. It’s a “how does a DBA with 20+ years of a career ahead of her prepare for the job of the future?” I’m guessing in even 5 years many of the day to day tasks we do now won’t be daily tasks done by clicking through a GUI. No one be has said that there will be no DBAs, unless you are just reading post titles.
Finally, and most importantly, automation isn’t just using scripts. It involves many more complex tasks, ones that tend to be declarative, not just scripted. It involves integrating services, not just managing SW and HW. It involves leveraging engine features that have been added to the platform. It involves finding the right level of unattended, yet monitored responses to issues. It involves self-healing. It will involve self-scaling. It will involve ML and AI. It will be different that it is now. More tasks will be able to be handed over to compute services. Even storage management. The whole “think automation” isn’t just about PowerShell.
I suggest you attend one of Conor’s talks on how DB management is changing. It’s enlightening to see how Microsoft thinks the future of the DBA will change. And how we can use data we have right now to leverage compute to cover DBA tasks better.
But it’s still not a Y/N question. We as a profession *should be* discussing where the line between services and humans should be today. Boiling it down to “Yes” or “No” is a losing battle.
(I guess I should have just blogged this. Maybe I will later)
Karen,
Thank you for adding your thoughts over here. I think there are multiple great conversations to be had based on this topic and your feedback.
I like your analogy of the “I can only use GUI” DBA. Maybe that falls inline with my reference of the ten year experienced DBA who repeats year one ten times? Maybe not. My point is, good DBAs who connect, learn, grow and share will be fine. Let me know if you disagree with that.
My blog post was motivated by reading multiple things from multiple people that imply the end is near for DBAs. This is not just how it is received by me but multiple people in the PASS community. I hear this was an interesting topic discussed at least one SQL Saturday last weekend.
I agree that IT Professionals including DBAs should embrace leveraging compute and many other things as well. That’s part of the learn and grow I mentioned above. This is a long blog post so I couldn’t touch on everything I would have wanted or go in-depth as much as I would have liked.
I could have done a better job mentioning that I actually expect more people to be involved with tuning then less due to the new features. Tedious task that prevented people from wanting to focus on performance tuning will be automated. Even without enabling automate tuning (not saying this should be done) people can see recommended changes and learn from them in their environment when they have the time. That is a great thing!
I think self-healing processes are great as well. I believe overall it will greatly improve systems. I do believe there will be some cases where ML and AI can make mistakes as they are continuously learning and making decisions. Maybe this will never happen? I am not an ML or AI expert but humor me on this next question. What if the business impact of a mistake made requires manual intervention and you don’t know the basics when you could override?
We can agree to disagree but I think some good IT questions can have “always yes” or “always no” answers to them. Some statements in IT can also always be true or false. I agree with Grant that you can write code that NO automation can fix. I will also give another example. Should an IT team be able to recover mission critical data? I believe the answer should always be yes.
Who are you referring to when you mention “Those of us who are talking about automation”?
Regards,
John
As I was reading this I had an interesting view of the evolution of database professionals. Think of the formation of a universe. In the beginning a “DBA” might manage the instance (or equivalent), do performance tuning as needed, write reports, even develop the app using tools like Foxpro, Clarion, Access etc. This is the center. As time has gone on the job has “spread”. You soon had specialties like BI, DB Dev, and Admins forming. Then even a bit later things spread even more. Now you have a number of BI specialties (ETL, Reporting, “Big Data”), Dev specialties (tuners, coders, architects), etc. I think the people who say “The DBA is dying” see just that center and miss all of the exciting activity spreading out from there.
I don’t actually see the DBA going away, I see it specializing and growing. Some tasks will be phased out. No one should be taking regular manual backups these days, you use some form of automation, be it maintenance plan or a tool like Minion Backup or Ola’s scripts. But the computer can’t know how to schedule your backups. Full backups daily? Weekly? Transactional backups every 30 minutes? Every 5? What if you have replication and it fails? I haven’t seen anything coming along to automate that (although it may eventually, who knows).
It’s funny, these “The DBA job is dying” conversations seem to come along every 5 years or so. In fact in one of my very first conferences ~5 years ago one of the presenters said “The DBA job will be dead in a few years” “They aren’t needed anymore”. Since then I’ve seen the job of the data professional grow larger and larger, never dying, only spreading out.
Kenneth,
I love your thoughts on the evolution of database professionals. Yes, this isn’t the first time or will be the last time we see peoples opinions state that The DBA job is dying.
Regards,
John
I agree with what you said John. A good DBA embraces automation, after all, that’s how we get time to do the fun stuff. I encourage automating mundane repetitive tasks, you only have so much time in the day, and I’d rather spend it on more interesting things. I also agree that DBA’s cannot go away, there will always be a case where you need eyes on something and the understanding of how everything works is critical. Great post!
Thank you for sharing your thoughts, Monica.
This was a great post John. I love the references back to DJ’ing and the “sync” button. I agree with that our role is definitely going to change, I do not think it is going to go away. In a discussion I was having with a friend /co-worker just the other day I said “Even if tuning is taken away, that is only part of our job. I treat it like this: This is my database, there are many like it, but this one is mine.”
There are many roles in a DBA career that constantly change or “evolve”, but it is our job to adapt and keep on doing our jobs. Sure, I love automation when it comes time for a hardening sprint and I need to spin up 10 SQL Servers for QA/DEV to test their release of code and all I have to do is run an ARM template, get coffee, come back and give out the RDP connections. Who wouldn’t!? But just because I can do that with Azure, doesn’t mean I can’t install without ARM. I believe you hit the nail on the head where you must learn your craft(in your case “Beat Matching” ), before you should automate it. If you don’t understand how it works without that process, how do you know the automated process is working correctly?
I don’t feel as if it is a yes/no question either though. Do I feel some things can be better off automated? Absolutely! That would just give me more time to “learn my craft” on other pieces or focus on a different pain point.
Great post, +10000000
Jim,
Glad you enjoyed the references to DJ’ing. This is not the first time or last time roles are going to change, in my opinion. To me its one of my favorite thing about being an IT Professional and a DBA.
I also love how you treat your databases.
Regards,
John
I agree the DBA job is not going anywhere, it will get more complex and we will need to know more. So in some ways it will stay the same as it always has, ever changing. I think that is why many people like being in this industry, it is never static, so we don’t get bored. The key to automation it is needed to prevent mistakes. DBAs like to say they are lazy so they automate, but really to do the same thing over and over is time consuming and boring. Thanks John for keeping this conversation going. So many of my clients need to have this conversation in their organizations and blogs like this help get the word out to embrace this.
Melody,
Thank you for sharing your thoughts. I hope it is helpful reference material.
Regards,
John
Very nice read my friend! A lot of areas covered and a lot of questions answered inline with facts and examples. This topic has been a real concern and fear for some DBAs I know over the last few years. It’s even trickled into the minds of traditional BI developers with all of the new technology services and software for leveraging big data, self service BI and real time reporting. Understanding and facilitating architecture and having the ability to spend more time learning and focusing on other DBA and development skills is priceless when your able to utilize automation.
Hi Jeremy,
Thank you for providing your feedback. I love how you touch on how automation trickled into the minds of traditional BI developers as well. I hope I get to catch up with you in person at PASS Summit this year.
Regards,
John