Google AI Hackathon: LLM based Evaluator for RAG

•Als PPTX, PDF herunterladen•

0 gefällt mir•34 views

Slides accompanying project submission video for Google AI Hackathon. Describes a LCEL and DSPy based evaluation framework inspired by the RAGAS project. Accompanying video URL: https://youtu.be/yOIU65chc98

Technologie

LLM based Evaluator for RAG
Mayank Bhaskar
Dave Campbell
Sujit Pal

question context
answer
ground truth
Retriever
LLM

question context
answer
ground truth
Retriever
LLM
Faithfulness
Answer
Relevance
Context
Precision
Context
Utilization
Context
Relevance
Answer
Correctness
Answer
Similarity
Context
Recall

Faithfulness
Answer
Relevance
Context
Precision
Context
Utilization
Context
Relevance
Answer
Correctness
Answer
Similarity
Context
Recall
JSON
Snapshot
LLM

https://github.com/sujitpal/llm-rag-eval
🙏

Weitere ähnliche Inhalte

Mehr von Sujit Pal

Python has a great ecosystem of tools for natural language processing (NLP) pipelines, but challenges arise when data sizes and computational complexity grows. Best case, a pipeline is left to run overnight or even over several days. Worst case, certain analyses or computations are just not possible. Dask is a Python-native parallel processing tool that enables Python users to easily scale their code across a cluster of machines. This talk presents an example of an NLP entity extraction pipeline using SciSpacy with Dask for parallelization, which was built and executed on Saturn Cloud. Saturn Cloud is an end-to-end data science and machine learning platform that provides an easy interface for Python environments and Dask clusters, removing many barriers to accessing parallel computing. This pipeline extracts named entities from the CORD-19 dataset, using trained models from the SciSpaCy project, and makes them available for downstream tasks in the form of structured Parquet files. We will provide an introduction to Dask and Saturn Cloud, then walk through the NLP code.

Accelerating NLP with Dask on Saturn Cloud: A case study with CORD-19

Sujit Pal

This deck discusses two papers by Dr Leslie Smith. The first paper discusses empirical findings around learning rate (LR) and other regularization parameters for neural networks, and leads to the idea of Cyclic Learning Rates (CLR). The second paper discusses CLR in depth, as well as how to estimate its parameters. The slides also covers LR Finder, a tool first introduced in the Fast.AI library to find optimal parameters for CLR, including how to run it and interpret its outputs.

Leslie Smith's Papers discussion for DL Journal Club

Sujit Pal

For the longest time, term-based vector representations based on whole-document statistics, such as TF-IDF, have been the staple of efficient and effective information retrieval. The popularity of Deep Learning over the past decade has resulted in the development of many interesting embedding schemes. Like term-based vector representations, these embeddings depend on structure implicit in language and user behavior. Unlike them, they leverage the distributional hypothesis, which states that the meaning of a word is determined by the context in which it appears. These embeddings have been found to better encode the semantics of the word, compared to term-based representations. Despite this, it has only recently become practical to use embeddings in Information Retrieval at scale. In this presentation, we will describe how we applied two new embedding schemes to Scopus, Elsevier’s broad coverage database of scientific, technical, and medical literature. Both schemes are based on the distributional hypothesis but come from very different backgrounds. The first embedding is a graph embedding called node2vec, that encodes papers using citation relationships between them as specified by their authors. The second embedding leverages Transformers, a recent innovation in the area of Deep Learning, that are essentially language models trained on large bodies of text. These two embeddings exploit the signal implicit in these data sources and produce semantically rich user and content-based vector representations respectively. We will evaluate these embedding schemes and describe how we used the Vespa search engine to search these embeddings for similar documents within the Scopus dataset. Finally, we will describe how RELX staff can access these embeddings for their own data science needs, independent of the search application.

Using Graph and Transformer Embeddings for Vector Based Retrieval

Sujit Pal

The Transformer architecture is responsible for many state of the art results in Natural Language Processing. A central feature behind its superior performance over Recurrent Neural Networks is its multi-headed self-attention mechanism. However, the superior performance comes at a cost, an O(n2) time and memory complexity, where n is the size of the input sequence. Because of this, it is computationally infeasible to feed large documents to the standard transformer. To overcome this limitation, a number of approaches have been proposed, which involve modifying the self-attention mechanism in interesting ways. In this presentation, I will describe the transformer architecture, and specifically the self-attention mechanism, and then describe some of the approaches proposed to address the O(n2) complexity. Some of these approaches have also been implemented in the HuggingFace transformers library, and I will demonstrate some code for doing document level operations using one of these approaches.

Transformer Mods for Document Length Inputs

Sujit Pal

In the last couple of years, we have seen enormous breakthroughs in automated Open Domain Restricted Context Question Answering, also known as Reading Comprehension, where the task is to find an answer to a question from a single document or paragraph. A potentially more useful task is to find an answer for a question from a corpus representing an entire body of knowledge, also known as Open Domain Open Context Question Answering. To do this, we adapted the BERTSerini architecture (Yang, et al., 2019), using it to answer questions about clinical content from our corpus of 5000+ medical textbooks. The BERTSerini pipeline consists of two components -- a BERT model fine-tuned for Question Answering, and an Anserini (Yang, Fang, and Lin, 2017) IR pipeline for Passage Retrieval. Anserini, in turn, consists of pluggable components for different kinds of query expansion and result reranking. Given a question, Anserini retrieves candidate passages, which the BERT model uses to retrieve the answer from. The best answer is determined using a combination of passage retrieval and answer scores. Evaluating this system using a locally developed dataset of medical passages, questions, and answers, we adapted the BERT Question Answering component to our content using a combination of fine-tuning with third party SQuAD data, and pre-training the model using our medical content. However, when we replaced the canned passages with passages retrieved using the Anserini pipeline, performance dropped significantly, indicating that the relevance of the retrieved passages was a limiting factor. The presentation will describe the actions taken to improve the relevance of passages returned by the Anserini pipeline.

Question Answering as Search - the Anserini Pipeline and Other Stories

Sujit Pal

Named Entity Recognition (NER) is foundational for many downstream NLP tasks such as Information Retrieval, Relation Extraction, Question Answering, and Knowledge Base Construction. While many high-quality pre-trained NER models exist, they usually cover a small subset of popular entities such as people, organizations, and locations. But what if we need to recognize domain specific entities such as proteins, chemical names, diseases, etc? The Open Source Named Entity Recognition for Data Scientists (NERDS) toolkit, from the Elsevier Data Science team, was built to address this need. NERDS aims to speed up development and evaluation of NER models by providing a set of NER algorithms that are callable through the familiar scikit-learn style API. The uniform interface allows reuse of code for data ingestion and evaluation, resulting in cleaner and more maintainable NER pipelines. In addition, customizing NERDS by adding new and more advanced NER models is also very easy, just a matter of implementing a standard NER Model class. Our presentation will describe the main features of NERDS, then walk through a demonstration of developing and evaluating NER models that recognize biomedical entities. We will then describe a Neural Network based NER algorithm (a Bi-LSTM seq2seq model written in Pytorch) that we will then integrate into the NERDS NER pipeline. We believe NERDS addresses a real need for building domain specific NER models quickly and efficiently. NER is an active field of research, and the hope is that this presentation will spark interest and contributions of new NER algorithms and Data Adapters from the community that can in turn help to move the field forward.

Building Named Entity Recognition Models Efficiently using NERDS

Sujit Pal

Natural Language embodies the human ability to make “infinite use of finite means” (Humboldt, 1836; Chomsky, 1965). A relatively small number of words can be combined using a grammar in myriad different ways to convey all kinds of information. Languages model inter-relationships between their words, just like graphs model inter-relationships between their vertices. It is not surprising then, that graphs are a natural tool to study Natural Language and glean useful information from it, automatically, and at scale. This presentation will focus on NLP techniques to convert raw text to graphs, and present Graph Theory based solutions to some common NLP problems. Solutions presented will use Apache Spark or Neo4j depending on problem size and scale. Examples of Graph Theory solutions presented include PageRank for Document Summarization, Link Prediction from raw text for Knowledge Graph enhancement, Label Propagation for entity classification, and Random Walk techniques to find similar documents.

Graph Techniques for Natural Language Processing

Sujit Pal

Learning to Rank Presentation (v2) at LexisNexis Search Guild

Sujit Pal

Search summit-2018-ltr-presentation

Sujit Pal

Search summit-2018-content-engineering-slides

Sujit Pal

SoDA v2 - Named Entity Recognition from streaming text

Sujit Pal

Slides for talk at Haystack Conference 2018. Covers evolution of an Image Similarity Search Proof of Concept built to identify similar medical images. Discusses various image vectorizing techniques that were considered in order to convert images into searchable entities, an evaluation strategy to rank these techniques, as well as various indexing strategies to allow searching for similar images at scale.

Evolving a Medical Image Similarity Search

Sujit Pal

Embed, Encode, Attend, Predict – applying the 4 step NLP recipe for text clas...

Sujit Pal

Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...

Sujit Pal

Deep Learning Models for Question Answering

Sujit Pal

Artificial Intelligence, Machine Learning and Deep Learning

Sujit Pal

Measuring Search Engine Quality using Spark and Python

Sujit Pal

Dictionary based Annotation at Scale with Spark, SolrTextTagger and OpenNLP

Sujit Pal

Mehr von Sujit Pal (18)

Accelerating NLP with Dask on Saturn Cloud: A case study with CORD-19

Leslie Smith's Papers discussion for DL Journal Club

Using Graph and Transformer Embeddings for Vector Based Retrieval

Transformer Mods for Document Length Inputs

Question Answering as Search - the Anserini Pipeline and Other Stories

Building Named Entity Recognition Models Efficiently using NERDS

Graph Techniques for Natural Language Processing

Learning to Rank Presentation (v2) at LexisNexis Search Guild

Search summit-2018-ltr-presentation

Search summit-2018-content-engineering-slides

SoDA v2 - Named Entity Recognition from streaming text

Evolving a Medical Image Similarity Search

Embed, Encode, Attend, Predict – applying the 4 step NLP recipe for text clas...

Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...

Deep Learning Models for Question Answering

Artificial Intelligence, Machine Learning and Deep Learning

Measuring Search Engine Quality using Spark and Python

Dictionary based Annotation at Scale with Spark, SolrTextTagger and OpenNLP

Kürzlich hochgeladen

In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method. We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy. Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations. Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today. Join us to discover how Zero-ETL can elevate your organization's data strategy.

The Zero-ETL Approach: Enhancing Data Agility and Insight

Safe Software

Welcome to this session on UiPath Manufacturing and AI. In this session, we will discuss the importance UiPath plays with technology in the manufacturing and we will do an overview of AI and Document Understanding. Please join us to hear from UiPath and Community experts on why you might consider leveraging UiPath in manufacturing. Topics covered Community team overview The importance of UiPath technology for manufacturing UiPath AI and Document Understanding overview What is Document Understanding? DU Process Studio Template Components of DU Framework Q&A Speakers: Sebastian Seutter, Senior Industry Practice Director, UiPath, Inc. Priya Darshini, UiPath MVP RPA Solutions Architect Dzmitry Belanovski, Account Executive, UiPath, Inc.

UiPath manufacturing technology benefits and AI overview

DianaGray10

WebRTC and SIP not just audio and video @ OpenSIPS 2024

Lorenzo Miniero

In today's digital world, trust is key to customer relationships, but keeping it is a huge challenge. Customers are well-informed and empowered, quick to change brands if their trust is broken, even if it costs them more. This puts a lot of pressure on organizations to handle trust and safety issues with great care and transparency. The challenge, however, is real. Fragmented solutions have left privacy, legal, and security teams in a perpetual cycle of catch-up, struggling to update privacy notices, manage customer data rights, and answer lengthy security questionnaires—all while trying to prove ROI to the business. It's a thankless job, filled with repetition, tedious tasks, and constant interdepartmental coordination. Combine this with fast regulatory changes and the quick evolution of AI, and it becomes overwhelming. Join this webinar to learn more about TrustArc's new innovative solution Trust Center, the only unified, no-code online hub for trust and safety information built for privacy, security, compliance, and legal teams. Trust Center streamlines your path to compliance, shortens the pre-sales cycle, and reduces both legal and regulatory risks, saving time, effort, and cost. This webinar will review: - Why companies are building unified Trust Centers for a robust privacy program. - How unified Trust Centers streamline sales cycles, ensure regulatory compliance, and reduce operational bottlenecks. - How compliance, legal, security, GRC, and privacy teams benefit from a unified Trust Center in terms of needs, pains, and outcomes. - How TrustArc Trust Center saves time and work while reducing legal, reputational, and compliance risk by effectively managing policies, notices, terms, and disclosures, and providing real-time updates on subprocessors.

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

TrustArc

In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.

Platformless Horizons for Digital Adaptability

WSO2

DBX First Quarter 2024 Investor Presentation

Dropbox

Decarbonising Commercial Real Estate: The Role of Operational Performance

IES VE

Overview of Hyperledger Foundation

Hyperleger Tokyo Meetup

Intro to Passkeys and the State of Passwordless.pptx

FIDO Alliance

Working together SRE & Platform Engineering

Marcus Vechiato

Introduction to use of FHIR Documents in ABDM

Kumar Satyam

How to Check CNIC Information Online with Pakdata cf

danishmna97

Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots

Leah Henrickson

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

FIDO Alliance

JohnPollard-hybrid-app-RailsConf2024.pptx

JohnPollard37

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

In this session, we will discuss the journey of API governance from its initial, ungoverned state to the development of sophisticated models that tackle contemporary challenges. We'll explore how APIs have become essential in the intersection of business and technology, adapting to advancements and evolving needs. We'll focus on how organizations have moved from launching to monetizing APIs, using models like pay-per-use and subscriptions, and finding the right balance between technical implementation and business strategy. We'll also highlight the impact of governance on monetization strategies, especially how data security, compliance, and service quality influence pricing. Real-world examples will demonstrate the effective integration of governance with monetization, including AI's role in dynamic pricing. Looking ahead, we'll share insights into future trends in API governance and monetization, emphasizing the importance of adaptability, continuous learning, and innovation.

API Governance and Monetization - The evolution of API governance

WSO2

Kürzlich hochgeladen (20)

The Zero-ETL Approach: Enhancing Data Agility and Insight

UiPath manufacturing technology benefits and AI overview

WebRTC and SIP not just audio and video @ OpenSIPS 2024

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

Platformless Horizons for Digital Adaptability

DBX First Quarter 2024 Investor Presentation

Decarbonising Commercial Real Estate: The Role of Operational Performance

Overview of Hyperledger Foundation

Intro to Passkeys and the State of Passwordless.pptx

Working together SRE & Platform Engineering

Introduction to use of FHIR Documents in ABDM

How to Check CNIC Information Online with Pakdata cf

Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

JohnPollard-hybrid-app-RailsConf2024.pptx

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

API Governance and Monetization - The evolution of API governance