Just as hitching a cart to the front of a horse is going to get you into trouble (and get you nowhere fast), it’s important to focus on data at scale before you jump to analytics at scale for that data. 

In other words, if the data’s not ready, you’re going to be stuck from the beginning. It’s important to research ahead of developing something. You need to ask yourself a lot of questions as you’re investigating and planning for large-scale analytics. Several come to mind right away: 

  1. What systems do you use? 
  2. What data platforms do you use? 
  3. How often are you going to have analytics workloads running against them? 
  4. Will your analytics be more interactive or more batch? 
  5. How fast do you need the response to be? 
  6. Is there going to be real-time support? 
  7. Are there regulatory and legal requirements in place you need to review and plan for?

For all these reasons it’s important not to jump in and rush a lot of these things because you don’t want to eliminate your chances of getting large-scale analytics right the first time. Data is going to be complex in most organizations. You’re going to have a mixture of things that are more simple, with structured data, but you might have semi-structured and unstructured data as well. 

That’s why planning the right databases and the right data platforms is going to be important so that you can get that data as you need it. Security is always going to be a driving factor as well, as is thinking about each step of the design of that ecosystem, that platform. Think about data access, roles, and requirements for your various team members. In other words, take the time to do things right. 

GenAI: the efficiency catalyst 

GenAI is of course important in analytics as well as scaling it. But you should start scaling analytics with the simple things that don’t necessarily need a human hand on them, like ETL processes, data quality checks, and metadata management. These things can be automated at scale to enable your analyst and developers to focus on things that bring more impact. Gen AI can serve us when it comes to analytics, uncovering simple trends and low-level predictions, making recommendations,  and identifying certain compliance and security vulnerabilities and violations. But it’s important to remember that we’re just getting started with GenAI. I’d recommend to anyone in a leadership role to keep an eye out, or check quarterly, as to what’s new and what’s generating buzz.

We’re already seeing greater efficiency thanks to using GenAI in analytics. Like a lot of organizations, there are plenty of things that we do that are simple, repeated routines that can be automated. Automating frees you up to focus on delivering things faster and getting faster time to insight. It also lowers the barrier for general users to advance their use of analytics. For example, having another set of eyes, whether that’s a person or through generative AI, can help you with improving accuracy, identifying issues and making sure that the data you’re actually providing to others is sound. This co-pilot model is being integrated into many tools. As is the case with generating code, GenAI can get you 80 percent of the way there. The other 20 percent is still going to require that a human with experience, knowledge, and ethical frameworks who knows the process and what they’re trying to develop. ChatGPT still assumes that any code it generates will work, no testing needed.  

5 best practices to speed up analytics at scale 

Based on my own experience, here are a couple of things to keep in mind as you build your analytics capability.

  1. Review and invest in cloud. Most of the best software, utilities, and innovation are happening in the cloud. Cloud gives you opportunities to scale more fluidly more quickly, and it is the way of the industry now. As you start your journey to scale up, start with cloud.
     
  2. Investigate and plan for your data architecture. Your data infrastructure and architecture should present you with many opportunities to modernize.
     
  3. Use the right programs or data platforms for your use cases. Also make sure you’re ready to go with that automation where possible. 
     
  4. Don’ t forget the simple stuff. ETL and metadata management focus on using generative AI in ways that automate the simple so you can focus where it makes more sense.
     
  5. Build with security in mind from the start. Every type of organization has seen a lot of major leaks lately. Security is just something that can’t be a second thought at any point.
     
  6. Foster a data-driven culture. When I say foster culture, I mean advocating for your users at all levels. That’s a challenge with a lot of organizations when it comes to data leadership. They aren’t focusing on the ground floor, the people that are actually using these products, and what the impact to them is going to be and how will they use them. 
     
  7. Invest in your people. Offer lots of opportunities for continuous learning, for work practice sessions, for workshops, any sort of training opportunities to make sure that people have the opportunity to invest in the company strategy when it comes to data practices.

Yes, achieving analytics at scale is great, but it’s even better when you take the time to do it right.