Name: Everlaw AI Assistant Virtual Event
Uploaded: 2024-08-12T18:51:39.726Z
Duration: 1 h 4 s
Description: Everlaw AI Assistant Virtual Event

Transcript for "Everlaw AI Assistant Virtual Event": We think about Everlaw as a truth finding machine. Everlaw AI helps us do that better by helping you find the critical piece of evidence to tell the best story you can. I think the most surprising part is how fast progress is made. What these elements are capable of doing is quite extraordinary. Generative AI has definitely jumped on the scene very quickly. It has different strengths and capabilities that are exciting. I think for us, it's that reading and writing which is so core to legal work. When you use generative AI, it is both far more powerful and expressive, but also it's not always gonna give you right answer. It's not actually particularly close to, you know, search algorithm. It's actually a lot closer to a person. While these models have improved a lot over the last few years, they're still not at a place where they're perfect. We work many hours behind the scenes to ensure that these models are producing well written output that clients can actually use and iterate from. People are seeing that Gen AI can elevate the profession if used responsibly. We knew we needed a framework in place. At the core of Everlaw's approach to building LLM tools are our Gen AI principles, control, confidence, and transparency. We have a great group of engineers and designers and product leads working on this. And having that sort of expertise is so important to getting the best outcomes with the product. And then it's not just about having a great team, it's about putting in the work to make this a reality. So the first thing we do is we test out how the L. M. Responds to different kinds of input. There is, for instance, something called temperature, which signifies the variety of responses that you're going to get. A field like legal, we want more deterministic answers. We try to build towards the strengths of generative AI. We're really thinking about what are the applications for it, the best ways to implement it. We don't just release something when it's half baked. We're building this for the long term. I think these tools will have a big impact for our customers immediately. I'm excited to see it take the next step and mature to, generally available part of the platform. Hello, and thank you so much for being here today. Welcome to our Everlaw AI assistant virtual launch event. I'm so excited to have you all here. It is possible that I have met some of you before. If that's the case, great to see you again. If not, I'm Rachel Gonzales, and I lead customer advocacy here at Everlaw. In my time here at the company, I have had the opportunity to speak with so many of our customers and really witness the growth and advancement that they have made in their careers as technology has changed over time. So that could range from IT leaders who are moving their organizations to the cloud, to ediscovery managers who are building and running their very first predictive coding models to address efficiencies and massive document reviews to even partners at some of the largest law firms in the world who are transforming litigation at their organizations. So I'm so happy to be here. As we discuss the next step in the transformation of the legal practice, We are very happy to announce the general availability of our Everlaw AI assistant suite of tools. This is an AI powered suite of tools that can help speed up and enhance document review and narrative building. Okay. So let's talk about how we're gonna spend our time together today. First, we will hear from Everlaw founder and CEO, AJ Shankar, on Everlaw's philosophy to building generative AI tools and why you can act with confidence. He'll then give us a demo to talk about how this will impact your workflows. And finally, we will hear from some customers to talk through the impacts of this on their day to day work. Okay. Let's talk through what you're seeing on your screen. On the right hand side, you'll see a chat functionality. Section where we'll have relevant information related to today's content, section where we'll have relevant information related to today's content. And finally, there will be a q and a tab. You are welcome to send questions in of us today. If we don't get to them live, we will try our best to follow-up with the answers afterwards. But I will address the number one question you probably have on your mind, which is, will this event be recorded? Yes. It will. Using the same link that you used to access today's event, you can go back afterwards and rewatch the recording. So let's get started. I am thrilled to have AJ here with us today. AJ, thank you so much for joining us. Hey, Rachel. AJ, you've been thinking deeply about the application of technology to the legal profession for over a decade, but I know that things started earlier than that for you. So you studied applied mathematics and computer science at both Harvard and UC Berkeley. So I have to ask you, what gets you excited about the application of generative AI to legal technology? Thanks, Rachel. Great question. Yeah. So my career has been around, providing access to technology to practitioners in law. And one thing that you see pretty quickly, there is no silver bullet for solving these problems. You need a whole toolbox of solutions. And what's really compelling about generative AI is that it stands out as a really, powerful new addition to this toolbox, which has the potential, to solve a wide variety of problems. And, ultimately, we think this is gonna, potentially help you do your job better, get better outcomes, and also provide greater access to the law. Okay. Well, thanks. I'll let you take it from here. Thanks, Rachel. Alright. So let's dig into the impact Gen AI is gonna have on litigation, and we'll talk also about how Everlaw is, integrating Gen AI technology into the platform. So the first thing that we feel pretty strongly about is GenAI is gonna change litigation. It's gonna have a big impact. But it's not gonna replace practitioners. It's gonna augment how they work. And it's not gonna be one single killer feature where you go to this one place to get all of your answers. It's gonna be integrated into every step of the process all the way through the entire life cycle of a litigation. Fundamentally, we do think it's gonna help you find better needles in your haystack and find them faster. It's gonna help you tell more compelling stories at the end of the day. But it also has to be used responsibly. Like any powerful tool, there's opportunities and there's concerns and risks. So I want to talk about how we're addressing those things. Right? So Everlaw as a company has a set of generative AI principles, and, these are our commitments we make to our customers when they use our Gen AI tools. I wanna start with the 2 basic principles that I almost consider table stakes now. The first one is privacy and security. Right? So we know that your data is extremely confidential. Right? You have a responsibility to your clients to keep it, secure. And as a company, Everlaw has been, you know, very forward thinking about security in general, whether it's our certification, SOC 2 and SOC 3 and ISO 27,000 and 1, 2717, Cyber Essentials Plus in the UK, FedRAMP for our federal customers, you name it. And so our commitments there are the same commitments we have here. Right? We're going to validate and vet the vendors we have for these tools like we would any other vendor and hold them at the same standards and hold ourselves to the same auditing and testing practices that we have. But there's 2 key additional, guarantees that we make to our clients here, because of the specific concerns around how g n a I can be used. So number 1, first guarantee is your data will never be used to train a model. And number 2, any provider we use will have a zero data retention agreement with them, which means that when we send your data to them, with the request, the data will be deleted as soon as the request is returned. It's not gonna be kept around for auditing, purposes. It's gone immediately. And that really reduces the exposure you have for some kind of exfiltration risk. So that's on the privacy and security side, and those guarantees give you some comfort that, your data is gonna remain secure. The second core principle is control. Right? And the idea here is that you're in charge. You get to decide when you're gonna use Gen AI, on what cases, and with what users. Your users will be able to always know when they're interacting with Gen AI through visual cues. You're also gonna know what models we're using. We're happy to share with you our points of view on, foundation models and how they're applied in EverLock. So those are the 2 kind of table stakes foundational principles. The third principle is the most interesting one, in my opinion. And it's around how we ensure that your team and your users have confidence in using the Arcara Gen AI tools. And so building confidence in a tool like this is actually challenging. Right? We've all heard about some of the embarrassing gaps people have experienced in using these tools in the wrong way and some of the pitfalls that are possible. So we build confidence through a series of 4 integrated steps. Right? So the first thing that we do is that all of our, product experiences using Gen AI, are designed to play to the strengths of the underlying LLMs while staying away from some of the weaknesses. And I'll get really concrete about what I mean there. The second thing is all of our experience are deeply embedded in the existing workflows your users are already at. All they should have to do effectively is click a button to get a reliable answer. The third thing is we wanna work with you and your teams to change their mental model about how they interact with these tools because they are fundamentally different from other technology tools you might have available. And the 4th thing is we also want to ensure that any results you get from a Gen AI tool are immediately verifiable in the platform. So you can actually check their answers to gain confidence. Right? And so I'm gonna get into each of these 4 steps on the path towards building confidence with your team. Right. The first one is understanding enough about these large language models and how they work so that we can develop, experiences that play to the strengths and stay away from the weaknesses. So if you look at how a large language model works, you might have heard that their next word prediction machines. Right? That's their job, is to predict the next word over and over again to answer your question. So you might ask, what color is the sky? And the LLM is going to predict the sky is blue. Right? And by the way, I think it's it's commonplace, you know, sometimes to trivialize what this actually means. It's just predicting the next word. You know, it's a sarcastic parrot or what have you. It's actually incredibly sophisticated what it takes to predict the next word and have a coherent conversation. Right? If you think about it, you know, there's 50,000 words in the English language, but there's an infinite number of ways to combine those words. Predicting the next word in the right order is what separates, you know, Einstein from from from gibberish. Right? So being able to predict the next word well, it turns out, requires some surprisingly deep competencies that these large language models have to learn in training. And it's actually very illuminating to spend some time understanding what these competencies are so that we can see how they should be applied to the law. Right? So there's a lot of competencies, but the big four that I like focusing on are, first of all, fluency. Right? Fluency in natural language. We live in a world of text, especially in the law. And so these systems must understand, you know, natural language. They have to understand syntax and grammar and the meanings of words. Right? The second big competency is creativity. Right? The ability to connect the dots. Right? Maybe it's generating poetry. Maybe it's coming up with a novel conclusion. But it's an essential part of a free flowing conversation. The 3rd competency is knowledge, embedded knowledge. Right? You when you have a discourse with somebody, it turns out a healthy the capital of France is blank, it better say Paris. Right? That's a fact that has to learn in order to predict the next word, and that's a competency these systems have. They know an enormous number of facts. And the 4th and maybe the most surprising competency is logical reasoning. Right? Logical reasoning, you know, maybe in a trivial sense is solving logic problems as well as humans do, which some of these more advanced systems can do. But it actually means something a little deeper. Right? It means that a sentence should be coherent from beginning to end. Right? And a paragraph should consist of sentences that flow well together. And an entire conversation should make sense. Right? And so the ability to reason is essential to have this kind of discourse, and these systems do events this kind of reasoning. So if we think about, these particular competencies, some strike out as things that we really like. Like, we love fluency and we love reasoning when applied to the law. Right? Again, it's a lot of text we want to understand. The more reasoning ability we can bring to bear, the better in terms of putting together good arguments. Creativity, is an interesting competency where it actually varies how much we wanna have creativity involved. Right? So if you're asking, a Gen AI tool to to write a summary of a document, you don't want any creativity there. Right? But if you're asking it to take a bunch of documents and build a compelling narrative from the next and dots, you actually might want a little bit of creativity as long as it's citing its source material. So you can think about creativity as being on a slider and us being able to control the slider, which which we do. Now the last competency here is knowledge, and I think that's the most interesting one. Right? Knowledge is this understanding of facts and ability to regurgitate facts. Right? And I mentioned these systems have learned a lot, right, in their training. And it turns out that this kind of knowledge is actually how many people actually interact with large language models. It's one of the primary use cases. It's it's factual understanding. And so, for instance, you know, if you're going to Chicago, you might ask, a Gen AI tool to give you a history of the nineties bulls. You might ask it to tell you the top ten things to do in Chicago. Right? And it'll give you good answers because it has a lot of training data on these questions. And the questions aren't particularly precise. And, by the way, the stakes are pretty low. Right? So as long as it gets the major detail major gist of these things right, you're not gonna be too upset if it leaves out, you know, like, the number 9 best thing to do in Chicago. But this is relevant because the way these systems work, when you're asking them for factual information, they don't have a database of facts to consult. That's not how they work. Any factual information they have is actually embedded statistically in the model itself, which is basically, you know, a giant spreadsheet of a trillion numbers. And so these systems don't really know right from wrong, and all they're trying to do is predict the next word, and, hopefully, the next word is factually correct. If there's a lot of training data, the questions are pretty high level, it's most likely gonna be the case. But what if you're in an environment where the questions are very precise, where there may not be a lot of training data, or a slightly incorrect word might actually be very different in terms of meaning. And, also, an environment where the stakes are really high, where a mistake is not just, hey. I can't do the nice 9th best thing in Chicago, but I might get the facts of the case wrong. Right? I might get the case law wrong. That's the environment we work in in the law. Right? It's not a low stakes environment with general questions. It's a high stakes environment with very precise questions. And so for that reason, we don't think we should use large language models for their knowledge of the law. The reality is with such precision, when you're talking about case law or precedent or regulatory framers, it's just possible statistically that they're gonna get answers wrong. And with high stakes, those wrong answers lead to the hallucinations that we've seen so publicly get people into trouble. And so all of our experiences are designed to not actually use these systems for the knowledge of the law. We think there's a better resource for knowledge of the law, and that's your team. Your team has deep domain experts in the law. It's very complementary skill to some of the fluency and reasoning stuff that the all large language models can do themselves, but we're not gonna rely on them for what they know of the law. We're gonna make sure that your team is there. So we stay away from knowledge when it comes to how we design the system. So that's number 1. We think about the competencies here. We design all of our experiences to play to these strengths, fluency fluency and and reasoning with the right dash of creativity, and we stay away from relying on them for any knowledge of the law. Now the second big thing that we do is we embed each of these experiences deep into existing workflows in Everlaw. You don't have to go to a separate website or anything. You just are in your review window or you're in story builder, and you can click a button to in to use Gen AI. This allows us to do some critical things. Right? It allows us to pick very precise use cases that are well defined, that allow us to give you that button to click rather than a very open ended experience where you might have to prompt engineer, you know, teach your users the right way to interact with this or that model. That shouldn't be a thing they have to do. The second thing that embedding into workflows allows us to do is provide all the context that the large language model might need to give you an answer. Right? So instead of going to a website and uploading or exporting documents and uploading them, a very tedious process, we have that context right there. And that context yields dramatically better answers. Right? And we talked about this context is not embedded facts about the law. It's your evidence in your case, the four corners of the documents that you have. And that reduces the security impact and the hassle of interacting with these systems. So embedding into existing workflows, allows us to have precise use cases and incorporate all the context we need that ultimately yields reliable and efficient outputs. So the third big thing is changing the mental model of the user when interacting with the system. So what do I mean what do I mean here? You know, when I show an Everlaw AI demonstration, people ask me, you know, how do you know it's right? And the answer is we actually don't know it's right. Sometimes, even with all these safeguards, it it's wrong. And that's very different from interacting with, you know, a search engine or something that's deterministic and always gives you kind of the same answer. But that doesn't disqualify using these kinds of tools. The analogy I like to give here is if you think about imagine there's a cancer detecting AI. Right? If the threshold for releasing this cancer detecting AI was it had to be a 100% correct, it might never be released. It might not have any impact at all. But if the threshold was instead, can it detect cancer better than humans can, when it hit that threshold, you could release it, and it would be saving lives. Right? And so, luckily, in our industry, our threshold is not, you know, saving lives every day, but it's something more like, can this tool help you do your job better? And it turns out that that's a threshold that you can meet. And I think about it as a smart intern threshold. Right? These tools are smart interns. They work really hard. They're very fast, but they're still interns. Right? Imagine an intern on the 1st day of the summer. Alright? They might get some stuff right, and they might get some stuff wrong. And so you want to build trust with them over time. So that mental model shift shift of recognizing that, hey. As an intern, this thing is really flexible and can be used in a lot of ways, a lot more than I could use my other tools in, but I'm still gonna have to either check its worth work or build trust in that system. That's an important model shift, the mental model shift that you have to have. And we love working with our clients to, provide them with ways to educate their teams about that model shift. Right? And so once you have that model shift, you think about it like an intern. It's beginning of the summer. You wanna check all of its work. And as the summer progresses, let's say, you know, the metaphorical summer, you're gonna find some task where it's just so reliably right. You don't need to check its work anymore. You're gonna find some cases where it's helpful but not always right, and you wanna keep checking its work. And you might find some cases where you don't wanna work with the intern at all. All of those are totally fine. But having this model really helps you build that kind of practice and application. And so to facilitate this, we have the 4th component of building confidence, which is ensuring that you can always verify these results right there in the moment. And so all of our functionality and all of our features work in such a way that when the generative AI generates a result, it'll cite source material that you can check right away. Right? Either, it'll cite, verbatim text in the document that you have right there that you can ensure that that text supports its point of view, or it'll cite base numbers of a set of documents that you can, verify or supporting what it's saying. Right? So that ability to verify allows you to work with this intern in the right way to build your confidence over time in the use cases you're comfortable with. And we think that's a really important part of this process. Alright. So that's enough theory. Let's actually see how we've applied these points of view to developing, features in our law itself. Alright. Let's jump right in with a demonstration of Everlaw's gen AI capabilities. So we're gonna start with, the Enron database, which you might have heard of. Actually, I'm just kidding. This is a different database, which I'm real very relieved to talk about. This database has documents, involving a leading management consulting firm and its relationship and communications with a major opioids manufacturer. And the case involves, the some of the strategies the consulting firm suggested that the manufacturer used that might have led to an increase in the opioid usage despite the known risks. So it's an interesting matter. And I'm in Everlaw right now on this case, And I noticed here that I've got a message, in my inbox here in the Everlaw messaging system, and I can jump into this. And I can see right here that my colleague, Mondi, has shared with me a message asking me to look at this giant dump of documents to see if any of them are responsive to a particular RFP. Right? So request for production. Now this is a very stressful and common task that people sometimes face, and especially hard given the size of the document dumps and, the time frames involved. So let's see how our Everlaw AI coding suggestions can help address this problem. Okay. So the first thing I'm gonna do is actually take a look at this RFP document itself. And here's the document. You know, it's a mock RFP, but it's similar to what you might actually see in the wild. And what it does is it describes, several kinds of documents to be that we're looking for for this particular request. Right? There's 6 kinds of documents we wanna pay attention to. And what we can do in Everlaw is set up, a coding sheet, which with one code per document type, that we map over, to identify which documents are responsive to the request. Right? So if I go back to Everlaw, that's easy enough to set up. I can go to my project settings, and I'll go to my codes. And you can see here I've set up these 6 codes that reflect the 6 kinds of documents we're looking for in the RFP. Now, normally, this would be what you'd set up, and then you'd go hand this off to a human review team, and they would go evaluate all the documents in the dump with respect to these 6 different codes. But here, we can actually bring gen AI to the table as well. And so what we can do is go to our AI assistant configuration and effectively start telling the AI about the case. And you'll see here there's a case level description that's gonna give details about the the matter at hand. And then if you go down to your coding suggestions, you can see that there's information you provide about the category here, these RFP topics, and then information you also provide about each of the 6 codes that we've created. And you can notice that you're providing information in a human readable format much like you would tell, a human reviewer about, what to look for in the documents. And I'll say some best practices here are, not to assume the LLM knows anything. Right? Give it all the context you think it might need. Fill in any of the gaps. Don't assume that it has any outside context. But if you do that and if you're robust, you'll find that it often can do a really good job of evaluating your documents for responsiveness with respect to the codes that you're providing here. So let's take a look at that next. Alright. So the way we apply these coding suggestions is let's find a batch of documents to run them on. So we'll go back to the home screen, and I'm gonna open up this sample binder to run the coding suggestions on. And, of course, this is a small sample to to show for a demo, but you can actually run these coding suggestions on 100 or 1,000 or tens of thousands of documents if you choose. So I go to batch here, and I'm gonna generate coding suggestions for this set. I'm gonna add the coding suggestions column to my results table view, and I'm gonna go generate. And what this is gonna do is feed each of these documents to, the large language model, provide all the context that we just showed about the coding rubric, and then have it evaluate each document with respect to responsiveness for each of those, codes. So this task is started. It's running behind the scenes. It's right in my workflow here in my results table, and it's gonna farm these requests out. And you'll see, slowly that documents are, these coding suggestions are filtering in here. Right? So we'll let it run for a little bit as it's actually going through these documents, to make these coding suggestions determinations. Now we do this in parallel, and we have a lot of ways to speed this up. And you're gonna find, that it's actually quite responsive, over time. So it's it's running on these documents. It's filling in coding suggestions. And what I can do now, because it's, again, deeply integrated into EverLaw, is I can, interact with these coding suggestions. So the first thing I might do is is filter down by coding suggestions. I'm gonna say suggestions, yes, any of, and I'll pick, you know, financial analysis and and and marketing and sales strategy, and we'll filter down to, these particular suggestions. So it's gonna show me the documents that have these suggestions. And I can see them right here in the results table, and the next thing I can do is is dig into them. K. So now that we've done some filtering, let's actually take a peek at an individual document. So I'll pick this document here. And this is opening up the document in Everlaw's standard review window. And so you can see again how these experiences are just deeply integrated into the platform itself. So I'm gonna take a look at this particular, document developing a path forward for market access. I'll open up in my context window some of the review assistant coding suggestions that have been made. So we talked about financial analysis and reports. There's a couple other ones that suggest that we apply and then marketing and sales strategies. So there's a number of other codes that it's suggesting that we don't apply, but I'm gonna focus on, you know, the one one of the ones that filter down to marketing and sales strategies. And you'll notice that it's gonna give you, an explanation for its reasoning. Right? Something that you can can give you some confidence in its in in its justification for, applying this code. This document is heavily focused on the development and assessment of marketing and sales strategies for OxyContin and XMSA. So there's a little justification here, and it's also gonna point you to that relevant area of the documents. If I click on this here, it's gonna take me to, you know, this particular slide where, you know, it's got some text here. Given our central objective to ensure appropriate patient patient access, there are several potential options we could consider a bunch of sales and marketing strategy. Right? So this is how you can see the system both providing actionable, insight, but also a justification intern is bringing where that intern is bringing insights to you, but ultimately, you're the person developing confidence and trust and then validating the outputs of what it's giving. So when you have an interface like this, because it's integrated, there's a number of things you can do. Right? So one is you can, you know, go back and view the configuration of what information we provided to the AI around the the case and the codes, and you might wanna iterate on this. If you don't like the results you're seeing, we suggest a healthy amount of iterations so you're comfortable and confident in the results. Right? You can, give overall thumbs up, thumbs down feedback for our team so we can learn about your thoughts on whether the system's performing well. And, of course, if you agree with, one of the determinations here, you can just click apply and apply that code directly, right, to the document. So you can see how integrating, this Gen AI workflow can can improve your experience. Right? It can reduce the stress around these tight time constraint, deadlines. You can improve the quality and depth of your understanding and help you test against human protocols and reviewers. And really get you a jump start on, finding the critical evidence in your case. In our experiments we've seen that, the system can perform quite well given the right context and information. We see precision at 0.6 or higher. We see recall at 0.8 or higher. These are competitive to or even surpassing human review performance. So we think there's a lot of potential here. But we also know this is a new technology, and there's gonna be significant variance based on the nature of the case, the context you provide, the size of the evaluation set. And so we encourage you to experiment with it, and we wanna be your partner in figuring out the right way to use this kind of tool to augment and improve the review process. Alright. Now we're done with that. Let's get back to the home screen and see if there's any more work for us to do here. We go to our messages. We'll see oh, looks like Mondi has sent me another message. We need to investigate ASAP. So there's a couple particular questions she has for me. What's going on with Tarjanique, this attached search? And then what's the cluster of people around Tamar, and XJang that what what what were they talking about? So this is another common and difficult stressful request here. We've got some very specific topics. And this is less about doing high level codes, but more about doing a deep dive into the contents of these documents. If I open up the search here, you'll see look at look at these pages. There's a lot of pages, for these documents. This is this can take many hours to sort through and figure out what's really relevant here. So let's let's bring AI to the task again. So what I can do here is, batch generate descriptions and summaries for these documents. I'll also go ahead and batch generate topics for these documents here. And you'll see actually the system going through these documents and generating these summaries right in line for me to take a look. Now what's exciting about this is, I can use it in a lot of ways. Right? These summaries are visible to my whole team. We all can benefit. I can use them to, just give me a high level overview or show me where I should spend more time with the deep dive. And so you can see here, these are all filtering in here. These descriptions and topics is really relevant information for me, for all 40 of these documents. So let me open up this 112 page document here and take a deeper look. So I'll open up this document. It's a lot to look through. Again, I can use our AI description and summary to get an overview of the document, what it's about. I can show summaries by section, section by section here as much as I want. And I can look at the extracted topics here. Right? There's many topics in this big document. Maybe I wanna focus on prescription patterns here. And this is an interesting one. It's gonna give me some of the bullet points around this topic, what it's finding. It's gonna show me some of the entities that are relevant to this topic, and then the relevant area here. Right? So it's gonna take me to a number of key parts of this 112 page document that might be relevant to this particular topic. Right? And I can page through them here. You can get a sense of, what it's talking about. Let me zoom out so you can take a look here. Differences in prescription patterns by prescriber and so on. So really rich information that I can look at and understand how, this document supports this particular topic. So this looks like a really interesting document to me. And so I can apply any codes I want. I can apply, you know, a hot rating if I want. And then I can also take this, topic and add it to a note and say, you know, interesting stuff. And so my colleagues can, again, benefit from, all the learnings I'm taking from the Gen AI in this way. So it's a really nice way to dig deep into documents, these descriptions, the topics, the sentiment analysis, the any of the extraction, all coming together to really help, you get the most out of, any particular document you wanna look Okay. So we took a deep dive into these documents, and we have a good understanding of of how they're critical to the case. Now a theme here is that, you know, each of these experiences is embedded deeply into the Everlaw review experience and that the LLM is really constrained to the four corners of the documentary evidence that you have, and that ensures that it's gonna be as reliable as possible. Now let's get to the second task that, Mondi had asked me about, which is to analyze the communications of these two critical people here. And so let me open up my data dump here, and I can look at these documents and and visualize them. Right? So I'm gonna look at my communication. I'm gonna find the 2 people that I really care about, these folks, and apply this filter. And then we can go jump back to, the results table. And I'm what you notice here is I've integrated into our communication visualizer, our data visualization system. And now what I've gotten is a subset of documents that's relevant to the 2 people we've asked for. And, of course, we've got our AI generated description of topics right here. So I could look at these and try to make sense of this, but what I actually wanna do is tell them a deeper, more compelling story with these 460 documents. So what I'm gonna do instead is add them to story builder. And story builder is our tool for creating narratives. Right? So if our review platform focuses on taking your haystack and finding needles, story builder's about taking the needles you found and constructing a narrative. And so, what I've done here is I've taken these 460 documents and added them to a story builder draft. So I'm gonna use my AI assistant within the draft, and compose using all the evidence I've added to this draft, these 460 documents, a memo, and I'm gonna provide a prompt too. You don't have to watch me type it in, but it's basically asking questions that Mondi wanted me to figure out about these particular folks. What are the themes here? And I'm gonna start generating. So what happens here is that we feed, either summaries or the entire text of these documents, depending on how big the document is, all to the LLM, and it's gonna synthesize all this information with respect to the particular task I've asked it to do, again, staying within the four corners of these documents, and then generate some output. Now it takes a little while. It's different every time. It's doing a lot of work. Again, it's an intern. It's really fast. It's not instantaneous. But you can see all the different ways you could, conceivably use such a tool. Right? So, I might have it focused on a particular person or a particular, particular aspect of the case that's relevant to me. I can use it in any way I want here and generate all these drafts to, start creating some valuable work product. Right? So here's a memo here. It's generating it for me. I can see as it's doing it in real time, a whole bunch of stuff around what these communications are about. I'm gonna let it run here, and I can insert this right into, the document. Let me get out of the assistant here. And here I've got, you know, wonderful view. And you'll notice that it's actually cited documents. Right? These are documents, evidence in the platform that I can view right in Everlaw right here, as part of this experience, to support its point of view. And you can see as I navigate around this, there's a whole bunch of very, very interesting things here that I found. Now it's not just used for these kinds of, big memos. You can use it in a lot of different ways. Right? I might actually just select, I don't know, a whole bunch of text here and say, you know, here's a whole section. Let me use the assistant to rewrite this section and summarize this text into a concise, concise set of bullet points and have it do that. Right? And so you can imagine this again as this really helpful intern that's waiting for you, asking for your input, giving you a first draft that you can make your own, Being a really good partner here, you can see now it's done this output. I can read these bullet points. I can include them, etcetera. So an incredibly powerful set of tools to really help you get the most out of those needles that you found in your haystack. Okay. So we've seen the ways that Everlaw AI can be useful in story building. Let's now move a little further along in the case and imagine that we're getting ready to depose some critical folks. In this case, you know, Mondi has asked me to help her prep for a deposition here. And, what I can do again is use the Everlaw AI assistant in a whole number of ways to facilitate that preparation. So Monty has actually already gotten in here and has done a number of interesting, Gen AI tasks where she's asked, Everlaw AI to generate a table of certain scenarios and particular opioids involved and cite the relevant sources. Right? So it's a it's a table format. You can see here, she's also, had another really neat task where she's asked it to come up with a list of questions that she could conceivably ask these executives, based on specific evidence that we found. Right? So this is a great way to prep for a deposition again, with this outline format as well as, providing concrete citations, to specific documents. But now let's imagine we've actually done, the deposition. We've uploaded a transcript here. And now she's asking me for a summary. We know from our users, this is often a very tedious task. Let's go ahead and generate this task here. I wanna run an exhibit and witness summary based on the transcript that we've uploaded. So the AI will get to work here generating the summary, so that I can inject it into the draft here. And again, it's an intern. It's a starting point. It's a first draft. But often, it can provide you so much value, by generating that first draft so that, you can get to work, make it your own, and then release it. You can see here it's generating a summary both based on exhibit and based on people. As you can see here, I can insert this right into the document. Great. I've done that. Now I've got this great summary. Maybe I also wanna run a a custom task here myself. I might wanna go to my assistant and, do an analysis, run a custom task. And here, I've got a custom task. I've asked I've asked it to do based on some specific needs in the case. I can do that. And, again, it's going to go do follow my instruction, do the best it can, get that initial draft out for me to review and edit and make my own. Right. So you can see this the the myriad ways in which you can, use a technology like this to augment your work, especially in this challenging task of story building, and narration. So I can insert this here. Maybe I'm really happy with these two things together. I can go, share this whole draft with Mondy here, and, hopefully, she will give me the raise I've been looking for. Alrighty. Summary is ready. So in this way, you can see, so many ways in which we can incorporate j n a I into that critical story building phase of a litigation even after core discovery is done. Okay. Let's recap what we've accomplished today. We've used Everlaw AI in a number of critical ways. We've used its coding suggestions to automatically categorize documents according to a coding sheet that we prepared with context that we provided, based on natural language prompts. And that's enabled me to speed up my early review of the case to identify which documents are gonna be responsive. We also use review tools like summaries and topics to quickly and effectively do a deep dive into particularly complex and thorny documents to get a better understanding of their relevance. We also leveraged the writing assistant to start the synthesis and analysis of evidence to develop a really powerful narrative. And finally, we used a number of tools to help prep for a deposition both before the deposition in terms of what questions to ask and other critical information. And then also after the depot, in terms of synthesizing and summarizing the information, all along the way, citing evidence, and providing verifiable outputs every step of the way so that I can have confidence in using this information, and making it my own. Now, key thing here is it's ready to use now. Right? So if you wanna learn more about how our customers have been using Everlaw AI and how can you can use it today, let's jump back over to Rachel, who can share some customer stories. Great. Thank you so much, AJ. You've explained to us the theory behind the technology. We've seen what it looks like on the screen, and now we're getting to my favorite part, which is hearing from our actual Everlaw customers about how this is impacting their day to day life. So today, we are joined virtually by Julie Brown, who's the director of practice technology at Vorys. Vorys is a law firm that has over 300 attorneys located in 9 different offices. Julie, thank you so much for being here today. Rachel, thank you for having me. Okay. So, Julie, you are a thought leader in this space and a true expert in practice technology. So how are you thinking about the potential of generative AI tools like Everlong AI assistant? So Voorhees, we, I would like to say, took a very cautious approach with generative AI when it first came out. In fact, I think our first edict was don't use it. And so we understood it and learned more about it. Obviously, we deal with very confidential information from our clients. So we took a a slow approach, but I think we're very open to exploring and seeing what opportunities it presents. As long as we know it's the data is secure and nothing's going to happen, to cause issues with our clients' data. Okay. Thanks for sharing, Julie. So you've been an active member of our beta program over the last year. Where have you seen the most success? I think the most success has been having the generative AI assistant help with, creating content, basically, reviewing documents and then providing insights and information about those documents. Oftentimes, the generative AI can come up with perspectives or ideas that we may not consider, just reviewing documents individually. So you have a really developed sense of how Voorhees is thinking about generative AI. How would you say that that aligns with Everlaw's approach? So Voorhees' approach on generative AI is to, first understand it, understand the capabilities, and then explore and determine how well it can help either from an efficiency perspective or improving our work product. Overlaw's approach to generative AI has been so refreshing. I think they've also taken a very cautious approach to ensure that it's accurate and it's giving the best results we can expect. And I appreciate the conservative approach to developing the tool. Julie, thank you so much for being here today. We really appreciate it. Well, thank you so much for having me. It was a true pleasure. Alright. Well, I'm excited for our next guest. Steve Delaney is the director of Lit Support at Benish. Now, I have known Steve for a long time because he's a long time Everlaw user, So we've talked many times in the past. Now Benish is a large and growing law firm, but they have a lean ediscovery team. So if anyone understands the need for people, processes, and technology, it's Steve. So Steve, thank you so much for being here with us today. Thanks, Rachel. It's great to be here. So, Steve, can you tell us a bit about how you're using Everlaw AI assistant at Benish? For the coding suggestions, we are focusing initially on low value cases where there's a economic pressure to keep the cost of review to a minimum. And figured that was the best way to test that feature. And so far, it's been going very well. And another use case that has come up quite a bit actually is when we are in crunch time and it is they're in the middle of depositions and the other side serves an 11th hour production. And you need to get very quick insight into what's in that production, and you need to be able to, you know, report on, you know, up the food chain to to the partners, etcetera, and say, this is what's in here. And note the prior to having AI, you would assign attorneys and review this stuff as fast as possible. The capability of Everlaw AI to take this corpus of this production and very quickly generate descriptions, look at what topics are involved in it. And then the attorneys can then spin through those descriptions, looking at a preview of the doc and just have the amount of time they have to spend. And then at the end of it, they have a nice report that they can give on to the senior partners without having to do an exhaustive immediate review of that production. So I I found that to be a a huge benefit. That makes sense. Can I know you're really also thinking about how to scale these tools? So can you tell me a little bit about how you're doing that? So when you're using coding suggestions or really any of these AI tools, you don't wanna be really applying the process across all your documents in every reiterative step, because you want to be able to refine those results. You want to get the best results you can. And the most efficient way to get to that result is to start with a very small batch, of a random sample where it applies right away, and then you can have attorneys look at the results right away. And they can determine, Yeah, this is really working, or maybe 8 of the 10 codes that I apply this to are are great, but maybe we need to tweak this one prompt or that prompt. And then they do, and then we reapply it to the same sample to see if it fixes the problem. And if if it does, then we go to a larger sample. And we keep doing this, keep revising, keep improving. It goes pretty quickly, especially if you can get engagement from the attorney on this, and you can be working kind of in real time on it. And then eventually, you get to a point where you trust it enough, you think you've got a really good set of prompts and you apply it across all the documents. And now you've only had to apply it across all the documents once as opposed to if you were a little less disciplined about it. You might have applied it across all the documents 2, 3, 4 times. So this way, it saves you time and and resources. And then I have to ask, Steve, what would you say to those who are interested in Gen AI, but haven't yet taken the lead to start trying it out? If you're interested in it and haven't had a chance to try it out, I suggest that you get a test database going with just some documents that you have. Nothing. You know, there's there's no pressure. There's no client effect on this. And just look at how it works. And I think you're gonna find that it's it's a lot easier than, than you might imagine. Don't hesitate to try it. I think where you need to be cautious is in how you use it. Then you make sure you're using it in a way where you're doing your due diligence and you're you're making sure that you're vetting the results and and doing a good process. But I think that you would use a good process. You would want to be using a good process no matter what tools you use. I think AI is something that's really gonna give you an advantage in discovery over anyone who's not using it. Thank you, Steve. So great to see you. I hope to see you again soon. Oh, Rachel, it was my pleasure, and, good luck with the rest of the launch. So we have here today with us Jason Thomas. Now Jason is the chief information officer at Cole Scott Cassaine, and CSK is Florida's largest full service litigation firm. And Jason is someone who's been thinking about the future of generative AI and law for a while now. So, Jason, thank you for being here with us today. Hey, Rachel. It's really good to see you, and it's really great to be here. Thanks for having me. Jason, I know that CSK really thinks about efficiency deeply, and that's a big measure of success for you. Can you tell me a little bit about the philosophy behind this and how technology fits into it? A lot of the stuff that that that attorneys have to do is, you know, I would I would consider kind of a hassle or mundane or repetitive task. And I think, we can use AI to automate those tasks. I mean, if they could they reduce the amount of time that they're spending doing all these administrative tasks, they can do, spend more time doing stuff that they love and, you know, the practice of law and being able to do that type of work. And what about the benefits that you expect to see? We decided to, do the into Everlaw AI, beta because, honestly, AI was picking up popularity, and we are we were already on the platform. We figured it'd be easy just to to dive right in and start, testing the features. It's really fast. It's it's very easy to get somebody trained and and to learn how to use the tool, and and we use it in a couple of our practice areas with very large, very complex type litigation. And that's that's where we really, have seen seen the benefits of using, the tool. And what advice would you give to other practitioners like yourself considering the use of generative AI in their workflows? The advice I'd give to other legal practitioners is, you know, every everybody seems to have a tool these days, and everyone seems to have the best tool. I would definitely spend a lot of time, in in your your in in your research and evaluations. These companies, they're starting to realize that there's demand for more than just one specific thing that that we wanna do. So rather than have 10 pieces of software that that are all AI and do all do different things, let's let's I I would look and see if there's a platform or you can find something that'll just like a one one stop shop for everything that you need to go. Because the last thing that any lawyer wants is another, 3 platforms to to log in to to use to be able to get their job done. Amazing. Well, thank you so much, Jason, for joining us today. Hey, Rachel. Good talking to you again, and looking forward to seeing you at Summit. And last but not least, we have Greg McCullough here with us today. Now Greg is an independent litigation consultant, and he is brought into cases involving complex fire litigation. So Greg has actually been providing Everlaw with feedback about our beta tools over the last year, and he's been really giving us a lot of crucial information on how these tools can influence the work that he's doing day to day. And the work that he's doing is very personal to people. Greg has been most recently working on cases involving the Maui wildfires that happened just about a year ago. So, Greg, thank you so much for being here with us today. It is so great to see you. Thank you. It's good to see you too and very honored to be a part of this event. Okay. So, Greg, I know that you've been working in the fire litigation space for quite a while and that this is really personal to you, and you have deep empathy for those who have been impacted. Can you tell us a little bit about how you got started in this area? Absolutely. Back in 2007, my parents' home was actually destroyed by a wildfire, that was caused by a local utility. And through the process of helping them to recover from that, I got to understand how challenging it is for people who are trying to recover from that. And I helped them with their litigation, ultimately started working with the lawyers who assisted them with their part and have now started helping on a much wider basis, to help people get back whatever they've lost. Of course, you can't get back the real loss, but having funds to rebuild your life does make a difference, and I've seen that firsthand. So I work literally day and night to try to help those people to get back what they've lost, and move on with their lives, just like I did for my parents. So, yeah, I take it very personally. Thank you so much for sharing that, Greg. I would love to hear a little bit about your work more recently with the Maui wildfires as well. Absolutely. Back in August of 2023, a series of wildfires hit the island of Maui and were extremely devastating. Many people died, entire communities were wiped out, and the process of trying to help the people of Maui to recover from those has been one that I've been honored to be a part of. And the way that I've participated is by helping to manage a database of documents that is in Everlong, helping to find relevant data for the documents that are coming in, preparing for depositions, and leveraging tools like the AI tool to have Everlaw help us figure out what the cream is versus the stuff that we need to throw off. And, it's been very impactful for us. Greg, how would you say this compares to how this type of work was done in the past? So if you compare pre AI to my current workflow, Before, we would have to go through the documents and either leverage searches where we were looking for specific key terms in a document set or leverage things like clustering to try to draw relevant comparative documents or down and dirty look at every document, which tended to be the way you ended up going. So you start with you start at the top and you just start barreling through the documents to try to find the relevant pieces of information learning along the way. So that was in the pre AI world. Post AI world, now the first thing I do when I get one of these last minute sets is I throw it into AI. I tell AI to do as much as it can across that corpus of work. Give me those descriptions. I throw that description into the Everlaw review window so that I can see it across the documents, and I start using that to help me identify which documents might be important or which pieces of information are being pulled out as far as relevant facts. Because just looking at that summary, I can see facts that are inside the documents without ever opening the document. It's sort of like leveraging metadata, where you're learning a little bit about the document without opening the document. And if you've ever done large scale document review, the more you can learn about your documents without having to read them, the better off you are for getting through them faster and finding the most important information. And how do you see Everlaw impacting your practice in the long run? We have more data than we've ever had before to review tighter timelines than I've ever seen before where we have cases that are going from the actual cause of action to trial within a much shorter timeline. And so we have deadlines that you cannot believe. And so even though there are people who are working literally 7 days a week, maybe certain people, 18 or 20 hours in those days, we still can't get it all done. And so leveraging AI assistant will help us to get the job done in the timelines that we have because the timelines aren't gonna change, and the data is only going to increase. So the only thing we can do is to have the best tools and leverage the most effectively. Effectively. And now I would ask you, what kind of advice would you give and what advice have you already given to those that are still on the fence about using generative AI in their practice? When anyone asks me, what do I think about AI and what should I do? The advice that I give every single one of them is number 1, don't be afraid. AI is here regardless of whether you wanna believe it or not. AI is a part of our lives and ways that we don't even understand or imagine. So be willing to embrace it and leverage it for what it is. You know, going back to what AJ said, it's like having a little extra assistant that is working on your behalf. Don't be afraid to click the AI button and see what it says. It's not gonna harm you. It's only going to give you information that you can then use to help make informed decisions and move your case forward. Greg, thank you so much for being here with us today and for sharing how this technology is transforming your work. Thank you very much for having me a part of this event. I've enjoyed it, and I look forward to more AI tools coming down the pipeline in the future. Well, hopefully, you've all seen the impact of Everlaw AI assistant, and you now understand our unique approach to development and can act with confidence. Now is the time to act. So if you're new to Everlaw, join us at everlaw.com, or contact your account team and see how you can get started today. Thank you so much. Goodbye, everyone.