In our modern world of data driven decision making, the pressures on data teams (Executive to Developer) are increasing more than ever. Businesses are looking for rapid turnaround insights into their data, that is structured, easy to understand and reliable. Unfortunately, rapid turnaround is usually achieved at the cost of high reliability and accuracy. This means that business is basically looking for what seems to be impossible…
To meet these seemingly impossible demands: Data and Analytics teams have had to evolve and develop new methodologies, enter DataOps. DataOps is a framework that combines the fundamentals of Agile, DevOps and Lean Manufacturing to better serve business’ needs for rapid, reliable, and differentiating insights.
According to the research firm Gartner, half of all chief data officers in large organizations will not be deemed a success in their role. Per Forrester Research, 60% of the data and analytics decision-makers surveyed said they are not very confident in their analytical insights. Why do Data and Analytics teams lack confidence in the analytics that they deliver and, why would business constantly doubt the veracity of data being presented?
DataOps starts out by identifying a few of the reasons why modern Data and Analytics teams fail to provide business with reliable insights and a rapid development cycle, from moving goalposts to siloed data, the list goes on. This article will focus on two reasons that we believe are the most pervasive, silent killers of an effective Data and Analytics Process:
Heroism and Hope.
Heroism is when Data-analytics teams work long hours to compensate for the gap between performance and expectations. When a deliverable is met, the data-analytics team is considered heroes. However, yesterday’s heroes are quickly forgotten when there is a new deliverable to meet. Also, this strategy is difficult to sustain over a long period of time, and it, ultimately, just resets expectations at a higher level without providing additional resources. The heroism approach is also difficult to scale up as an organization grows.
Hope is when a deadline must be met, it is tempting to just quickly produce a solution with minimal testing, push it out to the users and hope it does not break. This approach has inherent risks. Eventually, a deliverable will contain data errors, upsetting the users and harming the hard-won credibility of the data-analytics team.
With many years’ experience in the Data and Analytics space, we have seen (and ourselves been guilty of) both these practices. Perhaps, it is a result of the personality types that are drawn to Data and Analytics, or it might be to cope with the pressures of the modern Data and Analytics landscape (in our experience it’s a combination of both), but Data and Analytics teams are all guilty of Heroism and Hope. The type of developers / managers in Data and Analytics teams want to be the ones that are enabling business. They’re excited by the opportunity to deeply contribute to the growth of an organisation with clear insights into what the data is telling us, but they unknowingly adopt practices that paradoxically only adds to business’ frustration and confusion.
By falling back to Heroism and Hope, Data and Analytics teams are perceived by business as bottlenecks, providing inaccurate reporting and insight long after the accurate answer was required. Business then simply falls back to self-developed spreadsheets built off manual extracts from the source system and as inaccurate and inefficient as that may, at least they feel in control.
The above paints a grim picture of the state of Data and Analytics, but this would be an incorrect conclusion. By implementing a few small processes and systems Data and Analytics teams will be able to meet the demands of business. And by taking responsibility for Heroism and hope, they can more effectively provide business with deeper, accurate insights.
As a start it is necessary for Data and Analytics teams to agree, that the pace of business is in no way about to slow down. And with the on-demand economy driving the way that people think, the need for speed will only increase. It is therefore necessary for Data and Analytics teams to bring in systems and processes that mediate the conflict between reliability and speed. There are many great tools on the market that provide data Catalogues, Data lineage, cross platform testing and end to end documentation. These tools ease the barriers to a good DataOps process and enable teams to achieve DataOps goals, but they are in no way Magic. Data and Analytics teams also need to be trained out of the bad habits of Heroism and Hope. They need to be guided into following DataOps processes by understanding how it would benefit them.
One of the founders of DataKitchen stated that the idea for DataOps was born out of his frustration with always being reactive to inaccuracies in the data. Always feeling like you are one step behind the users of the reports that you built. All Data and Analytics teams share this experience and would greatly benefit from small adjustments to processes and mindsets.
