Building accurate AI without clean data

Hey folks, I'm Arushrut. I lead the applied AI team here at PromptQL. And today I want to talk about busting this data readiness and semantic lab myth. Every one of us was trying to build an AI is either waiting to perfect their data, perfect their metadata, perfect their tables, their columns, their descriptions to start building AI on top of that. Or they are trying to create solutions like semantic layers and knowledge graphs and then perfecting those as well to get AI added. But that does not work because A, you will never have it ready and B, even if you have it ready today, tomorrow it's going to break and it's going to get outdated instantly. How do you make AI reliable with the messy realities of business? That is by making AI work with data, just how your best analysts and best engineers work with data. They also work with data. They deliver some great results, but they don't have perfect data or documentation to work with, but yet they are so productive. Why should we not let the AI work on it in a similar way? This is what the data really looks like in enterprise. Tell me if I'm wrong that many of you folks working in enterprises or even smaller companies, you have seen a database for the table like CUST underscore MSTR underscore table underscore B 2 underscore final underscore capital final. That's how we've been naming tables. for so long and inside that table, let's say there are 2 columns. One says customer ID new, one is a customer ID old. Which one is the right customer ID column? Some of them have null values. Some of them still might be used now that they've been deprecated. You have columns which are named poorly that it's up to the human or the AI's interpretation to understand what that column really means. What kind of values does it have? If there's a Boolean column called isActive, does it have one or a zero? Does it have a Y or an N or a true or a null? These kinds of things represent the reality of enterprise data. While you might have similar data or data that you use to join across multiple systems, but now that is also stored in very different ways. Let's say one database that you have which has a customer ID, which is a number, 1, 2, 3, 4, but another data source that the customer ID has actually like a prefix, like a CUST underscore one, 2, 3, 4. Now, how would you create this kind of an implicit relationship and then do joins across these 2 sources without have been able to define 3 clear foreign key relationships. That is what the reality of an enterprise data is. This is something that plagues every single business out there. How do we solve this? Do we start a month-long or 2yearlong exercise to start cleaning up this stuff? One of our customers, they were like, okay, 5, 6 years ago, we'll standardize everything. We'll send a memo internally in the company that everyone, this is the way to name tables and columns. This is how you keep your data. And then there's to be a complete standard across the company. I mean, it took 18 months. It was 40 percent complete and that thing was abandoned. People stopped following those rules because no one else was doing it. Then there was this master data management team that said that they're going to fix this. They're still working on it. Now, 2 years ago with AI coming in, they were like, let's add a semantic layer, which understands my tables and columns, my data, and stuff like that. That's great. They tried to implement that, but every single time a new piece of data comes, the business evolves, definitions evolve, the semantic layer just breaks. Then you spend more time maintaining the semantic layer than actually getting value from it. Now, still today we are waiting for perfect data, but still waiting. We're still not there. What's the solution? Semantic layers. Semantic layers is where you give LLMs context about your business. Yes, tables and column descriptions is great, but then also you want to give context on business terms, business definitions. Let's say customer acquisition cost is equal to marketing spend divided by new customers. Now, that's something which you have put in your semantic layer. But that's not enough. When you say marketing spend, what does it mean? What kind of marketing spend? There are so many different places we can see marketing spend. When you say new customers, what does new even mean? Is it their first purchase? Were they reactivated? Are the trial customers included into that? Things like those. then over what time period are we looking at this cost? If I want to do some analysis, strategy planning, financial planning, do I look at my historical customer acquisition cost last one year, last one month? What is it that my business cares about? all of these things which you all know working in your business for so many years, it comes intuitively to you that what should I be optimizing for? What does the term exactly mean? What kind of analysis do I want to do on top of my data? But for an AI, that's not implicit. We have to explicitly tell the AI everything. and how much of this context will we keep adding to the AI? We'll keep discovering new stuff that my AI does not have context to, and we have to keep adding it, adding it, adding it, and maintaining semantic layers just seems like an impossible task. Someone asked me that, but what about knowledge graphs? Knowledge graphs are what map entity relationships, and then I can give my AI complete context about my data domain. Sure. Let's take a very simple knowledge graph where you have sales information, sales data, where there is a deal, deal has a stage, stage has a date, or a deal also has an owner. Now, when my business user asks a question like, show me my deals at risk, very simple question, but hidden with so much business context. When I say at risk, What does that at-risk even mean? My knowledge graph knows that deals have a stage and stage have a close date. But the definition of at-risk is something that that business user knows. If I were an intern in this team and my leader asked me, show me deals at-risk, I would ask them back, when you say risk, what do you mean? How do we define risk in our business? He probably does not do that and start making some assumption to start trying to give you some answer. But that answer may or may not be correct. How do I let my AI know that that's what risk means? And how do I capture that kind of a thing in a knowledge graph? So probably that's also not the solution. The problem really is that the AI in your business today, the latest and greatest LMs that you have today, they speak a certain language which is not your business's language. Because when your team says GM, it might mean gross margin or general manager or general motors. I don't know. When the sales team says conversion, that's different from when a marketing team says conversion versus the finance team says conversion. When you say what happened in the last quarter, it really depends on your definition of the quarter, right? Because typically, I will assume quarter starts in January, ends in March. But maybe for my business, the quarter starts in February and ends in April. That's what happens at PromptQL as well. For what times are we talking about? What is the definition of an active customer? All of these things are these tacit pieces of knowledge that you know working in your business, but your AI does not speak the same language. Hence, working with AI feels so clunky and prone to breaking. As I said, this tribal knowledge that is in every single analyst or engineers head in your business today, which makes them so productive, which allows them to understand ambiguity, which allows them to operate on messy data, all of that is missing with AI today and we have not been able to crack that code. so that we can start making AI reliable on these business questions asked by business users or customers. How do we go about building this? The traditional approach is that you have a human who maintains the semantic layer and knowledge graph, which you keep on updating every single time something new happens in the business and the data. It keeps getting broken and then you still keep updating it. AI tries to use it, it does not work. With PromptQL, what we have tried to do is build something called an agentic semantic layer. What that means is with every interaction that you have with the AI, with every single time you had to give AI feedback, you had to course correct AI, AI should be learning. It should learn from those conversations. It should learn from the patterns it sees in the data. If it realizes that this column has null values, I should remember that so that the next time I'm doing that analysis, I make sure I impute those values or I handle those values gracefully. These pieces of information that your intern who was super smart, was really thrifty, really worked hard to understand the domain, today is an expert. This interaction design with data, with AI, with business users is the same thing that AI should have. So that they also become, like start as good as your smartest intern and then become as good as your best analyst in the company. With PromptQL, that's what we have built. The basic architecture of PromptQL, which some of you have seen some of these calls before know that the number one key innovation that we did with PromptQL is not letting the LLM answer your question So, not letting your AI answer the question, only using the AI for what it is good at and making it behave like a human, right? A human, when I ask them a question, they don't just directly give me an answer, right? Let me try to figure out how to answer this question. Let me come up with a few steps. I need to probably define your question as this more concrete definition of business terms and calculations that I need to do. Then I need to go into these 7 data systems, pull out the data, write some code, and then run an analysis, also writing some code. Then finally, the output of that code is my answer, which I'm going to give it to you. Human did not generate the answer in their head. They just use their head to do this kind of planning. That's exactly what we use AI to do at PromptQL as well. Let's only use the LLM for what it is good at, which is planning. A PromptQL LLM will come up with the PromptQL plan. This plan is something which is very understandable and auditable and even editable by the user, but also it's executable by a deterministic system because this plan looks like a domain-specific language which can be executed in a programmatic runtime. The AI only generates that plan and we execute that plan on a deterministic runtime. That execution talks to the distributed query engine under the hood which pulls the right data, does the right analysis, and then finally extracts the right answer from the data and presents it back to the user without involving the element generating the answer so that we don't risk any scope of hallucination. So that's great, right? Like if I show you PromptQL, how it works, like something very simple. Who are my key customers? Let's say it's connected to a SaaS company's internal database. So if I ask a question like this, first of all, when I say key customers, right? What does key even mean? So let's see how Fromm can handle this question. It says, okay, I'm going to join the user's projects invoice item table to calculate a bunch of these things and filter important users based on this criteria. See, it has some business knowledge about me that how I try to define key customers, right? It knows that important customers are those with more than 3 projects and more than twenty thousand average revenue per project. So this is great. So I have like 8 of these key customers, but maybe I add a more strict filter because maybe the business definition has evolved. That's not what key customer means anymore. I'll edit this query plan and be like, no, this is actually 4 projects and this has become thirty thousand dollars average revenue And yeah, like I'll be like new definition. Now my AI needs to learn from that, right? Because A, prompter is completely editable, right? It's completely transparent and explainable in how it approaches the problem. I have this ability to course correct it. Once I've done that, now this goes back into the learning loop for the AI. Frankel learns from that and realizes, okay, your definitions have changed, I should adapt to that. Next time I ask the same question, I'm going to get the same analysis because it has learned that my definition of an important customer has changed. So what allows us to do that is the fact that we have this learning mechanism which starts turning the PromptQL LLM, the PromptQL model or the PromptQL plan into your business's model and the business's plan. Let's say you are in a company called Acme. Every time PromptQL learns with interactions with you or the business user, it starts improving the model under the hood with the semantic context, with fine-tuning, with whatever else is required to capture that information. Next time if I ask that question, I do not miss it. Now, the foundational model has become an PromptQL model. The PromptQL model will generate PromptQL plans spoken in the PromptQL language. Now, I can do many things that I really wanted to do. Now, if I ask a question like this, which enterprise deals are at risk, this Q and Y? Now, this question has so much business context in it, which we don't even realize. first, AI needs to understand that this Q means current quarter. That is the lingo that we use in our company. And when I say quarter, our ACME's fiscal quarter is offset by a month. So the current quarter is August to October. Or if it's Q 2, then it's It ends in July. Then define at risk. At risk means that a deal which is at a three plus stage with a close date in quarter three and there hasn't been any activity in over 14 days or it has been pushed one time or the champion has departed from the company or the budget is not confirmed. That is what defines risk. Now this definition of risk, the AI must have learned at some point when they were working with a business user. Then it needs to understand that, okay, opportunity data is coming from Salesforce. When I say enterprise segment, enterprise means that their ARR is more than a hundred thousand dollars. Then the activity data is coming from outreach. I need to look at the call recordings in Gong. I need to look at the email engagement metrics. And then finally, create this risk score. And the way we define the risk metrics is like, 40 percent of activities and say, 30 percent relationship strength, 30 percent budget status. and now do all of this math to finally surface the enterprise at All of this, the AI needed to learn at some point. Now, with an approach like this is very much possible so that I don't have to manually keep managing the metadata or the semantic layer to keep my AI up to date and informed about what my business really means. And now you can ask these questions in any kind of business, right? You can ask questions in the financial services business, like show me borrowers who might need proactive outreach before they become delinquent. So much tacit knowledge inside this question. Show me which units have concerning readmission patterns that need intervention, like a healthcare question. Which production lines are underperforming and what's the root cause? Manufacturing question. Where should we prioritize safety stock investments to avoid stock house without over-investing? This kind of planning I want to do with my inventory management and supply chain. Let's just look at how PromptQL is able to learn these kinds of things. Some of you might have seen this demo, but for those who are new, I want to just reinforce the fact that how the PromptQL agentic semantic layer learns. I'm here in instance where I've connected from QL to a database which has horribly named tables and columns. Something like, if I ask a question, who are the employees that work in departments with more than UST budget? Now, when I ask this question, because the data is so bad, there's no way any AI or even a human for that system can answer that question. But with PromptQL, what happens is it says that first I need to analyze the schema to understand how departments and budgets are even represented because I see that your tables are named as more plugins or which means nothing to me. first, let me try to see what data you have. Then based on top of that, I will try to understand how to approach the problem and how to answer your first, it did this to understand how to answer that question. I can also sample data from each table and confirm if your assumptions Work correct, something like that. In fact, it already did. Looking at the data, I noticed the schema appears to store employee first names as Flim, employee last names as Flam. Even if your data is so horrible, a smart AI will look at the data to even understand what's happening or rely on you to course correct itself. Now I can see the actual data. That's what it really means. Let me modify the query so that I can correctly answer your question. Previously, it was wrong. my approach. Now it's saying, and this is the new answer, but I'm also like, no, this doesn't seem right. Millions of dollars in some departments, that's a lot of budget actually because the data is in cents, not dollars. That's why you see millions in numbers. Again, poor data, no context for the AI. How does it answer the question? It's like, oh, cool. I'll just divide by a hundred and give you the right answer. Makes sense. Again, it just modifies its query plan, executes the query plan, and then gives me an answer. Actually, the answer is just 2 of these customers are in departments with budget more than 10000. Now, all of this has happened. Now, how is Prankly learning through all of these things? Let me show you the technology that powers the learning. It's a feature called Autograph, which keeps running in the background to keep analyzing these conversations that we are having, keep analyzing the data that's under the hood, and keep improving the semantic layer. One of the examples is to update the metadata of the semantic layer. When I say metadata, I mean like the column descriptions, the table descriptions. I can simply be like last one thread. So you improve the metadata using insights from the last one thread. I'm just manually triggering autograph, but autograph just keeps running on its own in the background to agentically keep improving the semantic layer. Now it says, okay, I'm gonna get this current schema definitions that we have so that I make sure that I understand what the current state is. Then I'm going to get the thread data, all the conversations we have had. And then based on that, I need to make informed decisions on how to make improvements to that metadata. So like, okay, based on the condition threads I've analyzed, I can now generate improvements. I need to create improvements for each table, the purpose of the tables, the meaning of the key columns, important data types, like the monetary values are incensed, all of these things that we just spoke about. see, it created this kind of great description. And it said, in the organization, the department budget quality is stored in cents. For example, the value of this represents twelve thousand dollars. See how great it is able to create this kind of description. Now it says, would you like me to apply these changes to the metadata? I'm like, yeah, apply these changes. When I say yes, it's going to create a new build of the semantic layer. You see all of these builds created by Autograph. The semantic layer itself is version controlled and it creates immutable builds. Either you can manually create them or you can let the Autograph create it. Every build is immutable and you have the ability to basically roll back to a previous build, run your evals on a specific build so that you know any changes happen or still do not break the system. Here it says that, I'm about to push these changes, does that look okay to you? That's what allows us to build such powerful AI which keeps learning from your business context. Going back, now with this, today, then PromptQL becomes this veteran analyst, or even a veteran engineer. You can use PromptQL not just to do your data analysis, but you can ask it to do some kind of data automation, create business logic on the fly, and then one thing, deploy that business logic. Every single time a new customer comes in, make sure you trigger these 7 workflows, and then send me a confirmation email that the workflows have been completed. All of these things can just happen with simple natural language prompts. So now, on day one, when your AI didn't know what enterprise means, how do customer ideas match, does not know what quarters mean. But by day 30, it understood so many business terms, it understands, it has created these map-to-map values relationships across systems by discovering patterns in them, understand different calculation metrics, and then now is able to give you a hundred percent accuracy on your complex answer questions. So remove the accuracy, get the accuracy bottleneck, and stop waiting for your perfect data. Just deploy AI which keeps improving with you, with the data, with the user. These are some quotes from some of our customers that Director of Data had a Fortune Five hundred food chain. They were like, we tried building things ourselves, we tried building semantic layers ourselves, our AI architecture ourselves, we evaluated hundreds of vendors, and then finally chose PromptQL. for this reliable AI. Another CEO for high-growth fintech companies like PromptQL was able to demonstrate a hundred percent accuracy on the hardest question in their eval set, which they had planned to solve next year. And today just out of the box, PromptQL delivered a hundred percent So that is PromptQL. Stop waiting for perfect data. Start getting perfect answers today.