AI Governance for Enterprises: A DataArt Case Study

Yeah thank you Dimitri, absolutely. So as you mentioned, DataArt is a user of the technology and a supplier. Among the many things that I'm doing, I'm leading our AI lab, and we think about DataArt as our own client. So we face very similar challenges as our enterprise clients. In this presentation I will go through some key points about governance, the challenges, the vision, and what we have done internally that helps us A) transform ourselves and B) work better with our customers.

Custom software development, everything around it, and services. And industries that actually — you see it's not just one industry, it's across the board. Even industries that are not in software development but they're using technology, they're also changing.

The perception of the complexity has changed, as I mentioned. What it means is that before you had to have a team of data scientists and everything in AI and machine learning, you had to invest heavily, there was very high risk and there was a chance that it will never work because of data issues, because of the quality of the model, and because of the limiting functionality of these models. Yes we talk about conventional AI use cases — the same clusterization, the same regression and statistical analysis — all of that had to be consumed by other systems, it was not as user friendly. And although the value was there, when ChatGPT was released and became available to the wider audience, everyone now sees the value of it by interacting with it. You can ask anything, it can generate you a response, and you can see how easily and quickly you can use it in different fields — from writing source code to creating articles to analyzing large volumes of documents. And you can use it not only in the chat perspective but you can create — think about many different aspects of how you can use GPT and generative AI. It just changes, for example, NLP. It used to be a very important field, a very important pillar in AI and machine learning, and nowadays with GPT and all of it on the market we talk about generative AI, not about NLP. The NLP problem pretty much has been solved. So this is what has changed.

It's a very good question. The enterprise, it's an organization, and there might be different definitions. I'm coming from my pragmatic perspective: it's a complex organization that is exposed to regulations in different countries, it has revenue, it has a large number of employees, it has a somewhat complex structure internally from the legal perspective, from the organizational development perspective. It's a very complex living organism with lots of context dependencies and challenges that are specific to large-scale organizations. So if we're talking examples, Disney would be an enterprise, Kmart would be an enterprise. DataArt, using the same definition — this is very pragmatic — DataArt fits the enterprise definition because we are a large organization and we have presence in many different countries around the world, and we have internal departments and we have our clients, and we have thousands of engineers that work every day on writing software. So we are an enterprise from this perspective.

So when we talk about AI governance and what makes companies that adopted AI successful — what creates value to the company rather than be a destructor and waste of resources — here are the keys to success that we learned and discovered when working with our clients and advisors. One key thing here is there should be a roadmap that connects AI initiatives to business value. It's no longer pure R&D, it has to see a use, it should generate value.

I think you need to think about what you're doing in your AI program and how it will create value. Let me elaborate. There's something I wanted to argue with, because from my point of view, if we're talking about business, the first and most important task of any business owner, any business decision maker is not actually thinking about how can I plug a particular initiative into my business. It's basically thinking how can I make my business more profitable, how can I make it more efficient, how can I make it bigger. So from my point of view all of the technologies that are currently being used on the market, including generative AI, in the first place most importantly should be viewed from the point of view of how can they actually bring business value. Can we have a look at AI in particular from how it actually brings value to the business?

So think about, for example, in our own processes — the success of our own projects that we deliver to customers depends on their productivity, how quickly they can do certain things. And yes, one of the things that we're doing — we are exploring AI assistance productivity tools and we try to measure how they impact the productivity of our developers so they can work faster, that they can be more creative, that they don't make obvious mistakes. They have this assistance tool that guides them. It's not a silver bullet, some people don't like it, some people like it, but there is evidence from practical experience that in certain areas it is a significant improvement when you work with a productivity tool. It helps you move legacy systems to the new stack, it helps you create — like split the monolith into microservices, for example — you're not doing this alone, you're not writing code on your own, you have a tool that allows you to do it quickly. We used to have similar tools before, now the quality of them has changed and they just work faster. They're driven by generative AI and they can understand the context. So this is just one of the examples. Basically, if developers no longer have to spend significant amounts of time on mundane tasks which could be very efficiently and very accurately performed by an AI assistant, they actually can concentrate on something which cannot be done by AI and not be distracted by something else. So basically from the point of view of DataArt as a business, what we're doing is we're using AI to make ourselves more efficient. And at the same time, think not only about developers — we have multiple departments here. The way how we work with our engineers, HR, recruitment, how we approach the market, how we create marketing materials — we use AI to save time, increase productivity and allow people to be creative and spend their time on the most important part and spend less time on less important. And by using generative AI we can now connect siloed data. We have maybe a system that has all the quantitative information about our developers with something that is in form of feedback or documents, different kinds of materials and data here and there. We can connect all of it together to gain new insights about individuals, about projects, about accounts.

Well, yes and no. Really what I've observed in the last six months is that generative AI instead of replacing people actually creates new opportunities. We didn't talk about prompt engineering as a specialty 12 months ago, now it is a thing. We never talked about RAG — retrieval augmented generation — use cases before, now we talk about it. Again, some of the jobs, yes, they will be covered by AI, but there will be new roles that emerge just because we have generative AI, like with any other technology. And I also wanted to make one comment. I mentioned a couple of the scenarios and things targeted at optimizing our own efficiency and operations. We work with clients, so if we do the same thing but faster, that's just half of the equation. We also want to do new things for our customers and we want to be there where they want us to be. Because many customers are trying to approach their own transformation, their own journey with AI and generative AI, it is in our interest to be prepared, to know the best practices, to know solutions, to have offerings to help them get to the point where they want to be.

Yeah, understand the technology, understand use cases, tools and challenges, and new issues that emerge just because this new technology is here. How to handle it differently — for example, in cybersecurity, generative AI and advancement in AI in general is both a threat and a tool. An opportunity to be more efficient and find new threats quickly, and at the same time attackers can use the same AI to create new types of attacks. So again, technology is an opportunity, it's a risk, and as with any other tool in technology we talk about the same questions — how to govern it, how to make it efficient, scalable and so on.

This is the vision that defines how we started this transformation earlier this year. AI is a mainstream and a new norm, meaning that every solution, every architecture, every product, every project of DataArt that we design and work on considers AI as a first-class citizen. Some projects don't require AI right now, some solutions there is no room for this at the moment, but when we look at the pace of change we understand that it should not be a surprise to us that the same technology, the same product that we were developing two years ago now requires transformation and there will be AI-driven new features, new changes. This is where we want to be as a company — that every engineer in the company, every person to a certain degree, knows AI capabilities that are relevant to them. If we talk about a developer, an architect, a penetration testing engineer, or even a designer or business analyst, every person should understand the impact of AI, its capability, and how they can use it when they work with their clients.

Yes, it is very accessible now.

Having a vision is just the first step. We try to unpack it, to convert it into a roadmap and tangible steps in our own journey, in our own transformation. Here you can see an example of what we use. There are multiple stages — crawl, walk, and run. You cannot boil the ocean, you need to approach every challenge in a certain way. And we do have certain pillars: it's outreach, it's technical skills, technology domain expertise and subject matter expertise. Each and every pillar here has its own objectives, goals, milestones. We talk about developing both technical skills — as I mentioned, every engineer and every person in the company understands certain components and capabilities of AI — but at the same time we are going deep into AI itself by doing prototypes, by doing target...

Of this program — so we talk about our own labs, our own verticals, our own practices, and it's our alliances, our partnerships with third parties, with hyperscalers and with AI leaders. I'm not using too many details here. When for example we investigate and we invest into creation of prototypes for industry-specific use cases, it goes in one of the cells in subject matter expertise development. It's just one of the things that we are investing in. So we talk about engineers, architects and subject matter experts that work with clients.

So we can probably go to the next slide at this point because here we exactly talk about different segments. We started by looking at our own engineers and architects and people who work with clients, but as a company, as you mentioned, there are different groups that are not even technical — they don't write code.

But to be fair, we have a very strong engineering culture. And you yourself did writing code in the past, so you can understand. Having a vision is just the first step, having the roadmap is the second. What now? We started actually discovering different use cases and opportunities within the company where we can use AI, what value it can deliver, what are the use cases that make sense. We started talking to different departments within the company — sales, marketing, operations, support, our own resources and recruitment and HR. We discovered more than 50 use cases where it was a clear fit — that yes, generative AI can do something good here. We talk about creating a knowledge base for sales so business development professionals can find resources quickly, get answers quickly — did we do this or that in the past? And now you can use chat to get the answer. What about case studies and references or partnership work? Again, there is one thing that can help you with this, and by means of working with a generative AI-assisted solution you can find answers to that. Analysis of our own engineers and accounts — so we've collected more than 50 different use cases across different departments.

Oh, that's a very good question. Probably, say, an interview like we have right now. When we do analysis of account management work, of account portfolios, and when we try to understand the bigger picture, we can think about a number of interviews between different people. But at the same time we want to get some key data points, signals from these interviews. Sometimes it's a one-hour-long conversation, and to post-process that, to understand, to get some insight from it, you need to summarize it. You need to convert voice into text, text into summary, you need to extract data points, and you need to get something that you can then analyze across the board. Sometimes it can go very deep into our policies. And with the chatbot that I will be describing in a moment — it's a knowledge base, it's L1 support to our employees. To what degree it's technical? Well, you work with a chatbot. It is integrated in our corporate ecosystem and at the same time you can ask non-technical questions about when is the next holiday, if we use DataArt as an example of an enterprise.

Yeah we did this to prioritize them. So among the 50 use cases we cannot start 50 at once — there should be a roadmap and priorities. We analyzed: what if we do it and it works, what is the reward? We save time or we get new perspective, we get new data, new insights from this. What is the value of this particular use case and how it fits the bigger picture? Because our business is not in the research itself — we are not creating a new model just as a mathematical abstraction or a very nice experiment. What will it make? What kind of value does it create for the company? How can we use it when we work with our clients? For example, our own AI platform we now use as a reference, as a blueprint when we talk to our customers. Our accelerators — we enhance them with generative AI and now we can do projects faster and deliver capabilities that are not on the market.

So you see, this is the journey — the vision, the roadmap, the definition of use cases and where we can use AI in the company. And now, all right, how do we start doing it? Yes, many things start with a POC and sometimes you do a first step. But when we talk about an enterprise it has to be secure, it has to be regulated, it has to be transparent, it has to have the right access control and user management control. It needs to be scalable and accessible across the board because there are so many different use cases and you cannot repeat the same thing over and over. You need to have a platform. And the platform here is a solution — yes, to enable generative AI and ML capabilities, but it's not only in the tech field and landscape. We also talk about compliance, legal, how we manage risk, how we approach conversations about AI, how we prioritize things, and how we structure our teams within the lab and within the company so we can develop these use cases faster. So this platform is a concept that has both technical components and organizational non-technical components. It's a solution that has architecture for both.

So, saying several years ago it would be purely technical. Nowadays I would start with the people and with the org structure. In our case at DataArt we do have an AI committee, we talk about AI at our board. We have ambassadors in all of our departments that understand it. We have a cross-functional team that is concerned with and learned about development of this platform and what are the use cases and risks. And we started conversations with compliance and infosec early on because we all understand the impact of this. So I would say start with people and with org structure first and then proceed carefully with the technology implementation. Because again, remember the perception has changed — it is not as complex now. You don't need to develop your own foundational model anymore. In some cases you just need to tweak parts here and there and it works. But because it is so easy, some people stop thinking about the same data privacy, security and compliance. Ask these questions early and then proceed with the technical implementation.

There are several risks. One is yes, it's about costs — you need to control it because as with any cloud-based or modern technology, it can be very powerful and very expensive. You need to have a clear vision on the burn-down rates, the model, how you project costs associated with this solution. The other thing, outside of data privacy and security and compliance as you mentioned, there is one thing that is very specific to generative AI, which is how you tackle hallucinations and ethical concerns when people work with a chatbot.

It's a real thing. In the generative AI field it's a concept that is very well known — it's when the model will produce you sentences with very high confidence. The model can tell you that 2 plus 2 is 5, or I'm simplifying things, but it can mix in things that don't exist and be very confident about that. Something like 'Do Androids Dream of Electric Sheep?' And the ethical concerns — it is much more difficult because every person has their own core values, and as a company, as an enterprise, there are certain corporate values and beliefs and what we think is the right way, our own position on questions and topics. It's about policy, it's about geopolitical questions and issues, it's about racial questions. When people start working with generative AI, the first thing many people do is they start exploring: what do you tell me about this or that? And we've seen early days of our prototype how the model can — the responses can be not in line with our values. Because the model was trained years ago, because we didn't explain our own position on this or that. And that's why it's another major risk and you need to think about it. Because when you're dealing with an enterprise ecosystem you have thousands of people in different countries and everyone — there is a third rail always, even friends can discuss something and find that they have difference of opinion on this or that. The engineers who are working on the use case and the platform need to spend time in prompt engineering, need to spend time in architecting, getting the feedback, controlling the responses so they are in line with how you want them.

So you start elaborating the request-response — the prompt and the response generation. By means of prompt engineering you define the context, you define the thinking process and decision-making process so the model will act as a certain person, will have certain guardrails when it generates responses. And yeah, prompt engineering is quite an extensive field now. This is just one thing. You start by exploring how you actually work with the model, giving the request from the user. Then you can implement a feedback loop and run the response through another model, through another prompt-engineered chain, so you can make an assessment of whether the factual information there is correct. You can start challenging the response — whether it has any offensive statements or whether it tackles the question in the right way from ethical and hallucination standpoint. And you can even make another loop that is less on the generative AI side and more on the practical side — extract key data points from the response and literally run it against something that you can quantify, you can trust, you can trace. And if it works then it goes to the customer; if it doesn't work it is being flagged and compliance and engineers will review what is happening here. So it's a combination of both how you approach the question, how you start working with the model, and how you post-process the response itself.

To a certain degree, yes. But training has thousands of definitions, and in this particular case training of a new model is a very expensive and complex thing. I don't think that we personally, as a company in this field, we don't want to compete with those who create these foundational models. But we need to know what we need to do on the platform level to control the quality of responses, and we need to train something that consumes, that works, that touches the model. That training — yes, we do it.

Using that platform, what we've done — and I can be very brief on these case studies. We developed a new feature in one of our accelerators which we found is very useful. If you remember the problem of OCR — you have thousands of PDFs and you want to get data from these PDFs. There are solutions on the market, yes, some of them require manual mapping, some of them require a person to actually select segments and explain that from this segment extract this value, from this table extract this structure. We developed our own accelerator that is multimodal and you can throw in documents in different formats and it will get the data from these documents without any manual mapping at all. It can assign labels, it can create subsets and classes of data that were extracted, and it works with different languages. Then we realized, all right, we have this accelerator, now what? You have the raw data of the document — can we make another step forward and actually instead of getting the raw data, classify the document, or extract a summary, or derive insights from the same PDF? Sometimes these documents can be a financial proxy statement, something like 200 pages long, and you need to just get three key insights. So we created the pipelines that feed the data from the OCR, from the document accelerator into the model, and now the full chain between having a source document, extracting the data from it, classifying and labeling, then feeding it to a model and getting insights back — it's just drag and drop a file into your browser, boom, you have your insights on the right side. We've developed that and we realized it can be quite useful because every second enterprise has something similar as a challenge in their roadmap and people are trying to tackle this problem. We developed this with generative AI and our platform.

Yeah, you're right. Another comment I can make — not about the case study but about the thing that you explained. Sometimes we think about AI as a solution to find the right something. However, what we find in practice is that AI and generative AI solutions allow you to fail faster. So if it doesn't work, if say with the same legal example, if there is something in the contract that is a red flag to us — with AI you can find it faster. If all of the AI capabilities that we have will tell you that this is a good contract, perhaps a person will still read it, but that person will know that there are no red flags and you don't need to waste time. Because if there were any red flags you would have known about them.

There is another one. I can briefly touch base on this. This is the chatbot that I mentioned at the beginning. What we've done, we've collected all the resources that we have — our knowledge base, our policies. Sometimes when you work in a company like DataArt, like we have offices in 20, 30 countries around the world — I have a question like when is the next holiday in that particular country? Because I don't know and I need to know about this when I'm planning a new release in my project. Or what about how can I work from my own laptop, or what do I do about my vacation, how do I plan it, how do I request my vacation? Instead of searching and scanning through articles, I can ask this question and the chatbot will give me an answer and will give me a source link to the document where it is explained. We can handle 60-70% of L1 requests. Of course we have room to actually do the magic with ethical concerns and hallucinations and accuracy of responses, so the model does not mislead a person. But at the same time, we see the value that I can get answers quickly and I don't need to bother my colleagues or support team because I already have the answer.

On the next — here I just explained the two case studies. This is pretty much done and we are improving it. There is a lot that is in progress right now. I mentioned the productivity tools rollout — it is happening. We started experiments with our accounts and clients. We're going through an extensive education program with our own engineers, and we're talking about hundreds and hundreds of people going through training and getting access to productivity tools now. The training of all the labs and upskilling of the company — this is happening, this is actively developing. This cross-pollination is happening. We organize panel discussions, we organize technical sessions and meetings. Again, we are developing use cases for sales and HR and account management here and there. And outside of DataArt, because you see we have our internal and we have our external — external is our clients, this is the core, we're a customer-centric company. So we've developed more than 40 different projects already, prototypes and production projects for our clients. And our experience — how we work with AI internally, we learn a lot of things, what to do, what not to do, and we come to our clients with this perspective in mind.

And what are the risks, what to do, what not to do. Because clients when they come to DataArt they expect us to have this knowledge and expertise and history — that we know 25 different issues with this particular solution because we've done a number of them. And yes, we've done it, we know that your customers think about five things, there are 20 that customers don't think about, and we help our customers to learn about them, to educate people, to make decisions. This is I think how generative AI, this whole journey, helps us create more value.

So when it just emerged, yes of course, from the curiosity perspective I was asking various different questions. Sometimes just to understand the limitations of this capability, sometimes just to get some insights. Because when you start working and you have just a blank screen, it's difficult — because you've done this in the past so many times, you cannot copy-paste, so some inspiration.

Oh no.

Yeah, that is a very good comment. I should perhaps start doing this too.

Thank you.