To be clear, this content isn’t just available to conference attendees; this is freely available to anyone who’s interested in it. So take a few minutes to learn about what’s new in the Hadoop community and what the tech giants are doing with Hadoop.
Shameless Plug: if you’re wondering what the frilly heck to do with all your data, check out my session on Data Warehousing in Hadoop 🙂
How can we take advantage of the veritable treasure trove of data stored in Hadoop to augment our traditional data warehouses? In this session, Michelle will share her experience with migrating GoDaddy’s data warehouse to Hadoop. She’ll explore how GoDaddy has adapted traditional data warehousing methodologies to work with Hadoop and will share example ETL patterns used by her team. Topics will also include how the integration of structured and unstructured data has exposed new insights, the resulting business impact, and tips for making your own Hadoop migration project more successful.
I’ll be honest: I didn’t know what to expect from Hadoop Summit. I’ve been surprised at the lack of overlap between the PASS community that I know so well and dearly love, and this new community of open-source aficionados, data hackers, and Ph.D. data scientists. Would this new community be interested in data warehousing, a topic traditionally — and fallaciously, in my opinion 🙂 — associated with all things BI and relational? Combine this with the fact that I’ve never even attended Hadoop Summit before, and well… this was easily the most nervous I’ve been before a presentation since my first major presentation in 2009. However, all my fears were for naught… the session was packed — clearly, folks are interested in this topic! And judging from the quantity of conversations I had with people afterwards — many of whom are from companies you’d readily recognize, too — this is a topic that is only going to grow.
For those who were unable to attend but are interested in this topic, I have good news! The session recording should also be available online within the next couple of weeks. I’ll post the link once it becomes available. 🙂
Lastly, I typically find the conversations I have with session attendees after presentations to be my favorite part of conferences, and this was no exception. Thank you to everyone who attended and reached out to me afterwards! I met some great people, and I regret not doing a better job of exchanging contact information amidst the chaos of the event. If we connected at Hadoop Summit, let’s connect on LinkedIn too. 🙂
I’ll save you the suspense of a long post and answer the second question first: No, it’s not.
SQL Server is Still Relevant Here’s why. SQL Server does what it does *extremely* well. I would not hesitate to suggest SQL Server in numerous scenarios, such as the database backend for an OLTP application, a data store for small-to-medium sized data marts or data warehouses, or an OLAP solution for building and serving cubes. Honestly, with little exception, it remains my go-to solution over MySQL and Oracle.
Now that we’ve cleared that up, let’s go back to the first question. If SQL Server is still a valid and effective solution, why did I switch my focus to Hadoop?
Excellent question, dear reader! I’m glad you asked. 🙂
Before I get to the reason behind my personal decision, let’s discuss arguably the biggest challenge we face in the data industry.
Yes, Data Really Is Exploding We’re in the midst of a so-called Data Explosion. You’ve probably heard about this… it’s one of the few technical topics that has actually made it intomainstreammedia. But I still think it’s important to understand just how quickly it’s growing.
Every year, EMC sponsors a study called The Digital Universe, which “is the only study to quantify and forecast the amount of data produced annually.” I’ve reviewed each of their studies and taken the liberty of preparing the following graphic* based on past performance and future predictions. Also worth noting is that, EMC historically tends to be conservative in their data growth estimates.
* Feel free to borrow this graphic with credit to: Michelle Ufford & EMC’s The Digital Universe
Take a moment and just really absorb this graphic. They say a picture is worth a thousand words. My hope is that this picture explains why the concept of Big Data is so important to all data professionals. DBAs, ETL developers, data warehouse engineers, BI analysts, and more are affected by the fact that data is growing at an alarming rate, and the majority of that data growth is coming in the form of unstructured and semi-structured data.
Throughout my career, I have been focused on using data to do really cool things for the business. I have built systems to personalize marketing offers, predict customer behaviors, and improve the customer experience in our applications. There is no doubt in my mind that Hadoop is absolutely critical to the ability of an enterprise to perform these types of activities.
The Bottom Line SQL Server isn’t going away. Arguably, the most valuable raw data in an enterprise will still be managed in a SQL Server database, such as inventory, customer information, and order data.
So again: why did I make the decision to focus on Hadoop over the past year?
I once had the pleasure to work for a serial entrepreneur. One day over lunch, he gave me a piece of advice that resonated with me and would come to influence my whole career: “Michelle, to be successful in whatever you do, you need to find the point where your heart and the money intersect.”
My heart is in data, the money is in the ability to effectively consume data, and Hadoop is where they intersect.