Ion Moșnoi on LinkedIn: I helped a founder who was creating a search engine for insurance… (2024)

I fixed a generative AI app for a client, and it was a challenging task. The application was designed to respond to WhatsApp support messages, but whenever we tried to address one issue, it would inadvertently break the solution for another problem. This highlighted the importance of proper evaluation in generative AI systems before deploying them to production environments.The solution involved creating a more robust evaluation function that utilized multiple datasets tailored to different use cases and question types. By running these datasets against the modified model, we could assess its accuracy across various categories and ensure it met the required thresholds before pushing it live.Ultimately, the experience reinforced the crucial need for comprehensive evaluation datasets in generative AI deployments. This approach enables developers to identify and address potential issues proactively, ensuring a smoother and more reliable user experience.

8

I recently had the opportunity to assist a client in resolving a persistent issue with their generative AI chatbot, which had been developed by a different agency over several months. Despite having a system in place to generate new questions and answers based on user interactions, the chatbot's performance was not improving over time due to stagnant data quality.Upon investigating the root cause, it became evident that the data fed into the chatbot was not being processed effectively. The data chunking process lacked crucial context, resulting in incomplete and disjointed answer generation.To address this challenge, I implemented an automated entity recognition system that extracted relevant information from the client's documents, presentations, and files. This included extracting document titles and section headings, which were then incorporated into each data chunk. By providing this additional context, the chatbot could better understand the subject matter and leverage the full scope of the available information.One particularly notable improvement was in handling scenarios where a single slide or section lacked sufficient information to answer a query. Previously, the chatbot would struggle in such cases, but with the enhanced data processing capabilities, it could now draw upon the broader context from the entire document or presentation.Within two weeks, I successfully integrated the entity recognition system with the client's chatbot and deployed it into production. The result was a significant improvement in the chatbot's response quality, with more accurate, comprehensive, and contextually relevant answers. By addressing the underlying data quality issue, the chatbot could now leverage the client's extensive knowledge base more effectively, providing a better experience for end-users.

16

3 Comments

Should I open source the project for sorting and counting products, reviews, and other documents based on dates, categories, and more?Last week, I made a post about how I solved a problem for a client by enabling a chatbot to answer questions like "How many products do we have?", "How many open job listings are there?", "Show me the latest negative reviews" or "Show me the products from this category." It got hundreds of likes and tens of thousands of views, so I wanted to help others by making it available. However, I want to ensure there is a demand for it. If you would like to use or contribute to this project, please reply with a "+" or "yes" in the comments. If there is sufficient interest, I will proceed with open-sourcing it.

20

15 Comments

The Challenge of Big Data Integration for GenAI ApplicationsA fellow AI developer reached out for help after running into major issues with a generative AI project he had contracted for $40,000. During the proof-of-concept phase with a small dataset, everything worked brilliantly. But when it came time to integrate the full 50GB dataset of PDFs, suddenly 90% of the model's outputs were wrong.What went wrong? With large, real-world datasets, you often encounter many records with similar content but different important context and metadata. In this case, there were reports about car accidents that shared terminology like "vehicle collision", "injuries reported", etc. But the key details—dates, locations, parties involved—were different for each incident.To the generative model trained only on the small sample data, all these inputs looked basically the same. It couldn't effectively disambiguate and reason over the nuanced differences across tens of thousands of similar but distinct cases in the full dataset.Over the course of two weeks, I implemented a range of techniques to properly condition the model on the unique contexts and overcome the similarity challenges:Fine-tuned a custom sentence transformer embedding model on the domain data to obtain rich semantic representations.Trained a re-ranker model using 25 key examples to dramatically reduce the generative model's confusion between highly similar cases.Added an initial document classification stage to quickly filter by document type, avoiding needless comparisons across different categories.Identified and injected key entities (locations, names, etc.) from each document into the model prompt to reinforce the specific context....And numerous other data processing, filtering, and model tweaks to equip the generative model with stronger reasoning capabilities.The key takeaway is that simply throwing lots of data at a foundation model is rarely sufficient for high-accuracy outputs, especially with highly specialized domains and datasets. Careful data conditioning, retrieval augmentation, and incorporating relevant task-specific models are essential.For generative AI to truly become a transformative technology, we can't just rely on simplistic, out-of-the-box approaches. Addressing the nuanced challenges of real enterprise data will require rolling up our sleeves and applying considerable machine learning expertise—building custom architectures, creating high-quality training datasets, and intelligently orchestrating multiple models.Those of us working in this space must embrace (and charge accordingly for) the systems-level thinking and iterative development required to extract robust performance from large, complex datasets using generative AI. There are no easy shortcuts, but the rewards of finally making this technology an enterprise reality are immense.

53

8 Comments

The Real Problem with AI Automation in Business: A Failure to Get Hands-OnFor well over a decade, businesses have struggled to successfully adopt AI and machine learning to automate key processes and tasks. As far back as 2010, the frustration was palpable - executives would check out the latest AI solutions, try to apply them in simplistic ways, and then dismiss the technology as "not ready yet" when it failed to deliver.But the real problem isn't that AI wasn't ready. The problem is that businesses fail to understand that AI is just one piece of the automation puzzle. It's an extremely powerful tool, but a tool nonetheless. Successful automation requires integrating AI models with your existing data pipelines, backend systems, workforce processes, and more.Too many organizations still treat AI as a magic "make it happen" button rather than a component that needs to be properly embedded into their operational fabric. They expect to throw data over the wall to a machine learning model and have it spit out complete, integrated solutions.Let's take a common business use case like document processing. Deploying an AI system to merely extract data from invoices, contracts, or other documents is just the first step. That extracted data then needs to be:Verified against accounting rules and policiesMatched to product/vendor information in internal databasesClassified into appropriate accounting categories and cost centersApproved through human review workflowsIntegrated with accounting/ERP systems to update recordsThe AI model is a relatively small piece of that end-to-end process. But business domain expertise, data integration, human oversight, and orchestrating workflows are crucial elements that can't be neglected.The disconnect happens because business leaders look at AI through the lens of what they read in the news or what they see from tech vendors. They treat it as a plug-and-play product rather than a toolset to enhance and augment their operations. Nobody writes about the hard work of process re-engineering, data readiness, managing human-AI workforce workflows, and tying it all together.Until you get hands-on and start actually building AI products for your specific departmental needs, you'll never truly grasp all the components required to make it work effectively. There's no amount of conferences, workshops, or blog posts that can substitute for that practical experience.So if you want to move beyond the cycle of disillusionment with AI automation, stop treating it as an out-of-the-box solution delivered by vendors. Build small, functional AI products for your processes. Integrate them. Iterate based on learnings. Only then will you understand what's required to successfully augment your workforce and workflows using AI as the enabler, not the entire solution.The technology is ready. But you have to be ready to thoughtfully weave it into your operations. Start doing instead of waiting

12

5 use cases where fine-tuning a sentence transformer makes sense for generative AI applications:Intent Classification: Fine-tuning a sentence transformer on a dataset of labeled user intents can help classify the intent behind user queries or prompts. This is useful for routing queries to the appropriate generative model or response module in a conversational AI system.Query Clustering: By fine-tuning a sentence transformer on a corpus of user queries, you can obtain semantic representations that cluster similar queries together. This can aid in query understanding, retrieval of relevant information, and generating appropriate responses.Dialogue Context Encoding: Sentence transformers can be fine-tuned to encode the context of a conversation into dense vector representations. These context vectors can then be used by generative language models to produce more coherent and relevant responses grounded in the dialogue history.Semantic Search: Fine-tuning a sentence transformer on a domain-specific corpus can provide semantic vector representations that enable efficient similarity search and retrieval of relevant information or knowledge to assist in generating high-quality responses.Data Augmentation: By fine-tuning a sentence transformer on a small seed dataset, you can leverage its capability to generate semantically similar sentence embeddings. These embeddings can be used to find related examples in a larger unlabeled corpus, effectively augmenting the original dataset for better performance when fine-tuning generative models.

46

4 Comments

How I Fixed Counting and Sorting for a GenAI Chatbot in Just 5 DaysA client came to me frustrated after paying tens of thousands of dollars to an AI agency for a chatbot that couldn't handle even basic counting and sorting tasks. Things like "How many products do we have?", "How many open job listings are there?", and "Show me the latest negative reviews" were stumping their expensive chatbot.In just 5 days of work, I was able to get their generative AI assistant answering those types of queries accurately and reliably. And I charged a lot for that work - but it was worth it to the client to finally have a chatbot that could handle such fundamental requirements.How did I accomplish this so quickly when their previous AI developers couldn't? By going back to fundamental machine learning principles rather than treating the language model as a self-contained black box.The key techniques I used were:Named Entity Recognition (NER) to extract relevant pieces of information like product names, review text snippets, and job listing details from the client's dataset.Classification models to identify the distinct data categories like products, reviews, job postings, etc. present in the information.Combining the outputs of NER and classification to enable counting, filtering, sorting and other operations on the categorized data pieces.While the famous foundation models like GPT-3 can engage in very impressive open-ended dialogue, creating polished AI assistants for answering specific business queries requires incorporating more targeted data extraction and processing capabilities.For too many companies, there is a tendency to treat large language models as end-to-end solutions without understanding their underlying limitations. To create genuinely useful and robust generative AI applications, you need to pair the language model with carefully engineered data pipelines and supplementary machine learning models.If you simply try to force-feed all of your proprietary business data into a foundation model without any additional processing, you'll inevitably run into issues around core requirements that humans take for granted - like being able to count and sort items accurately.Each company should start building AI systems trained on their own data and personalized for their own needs. Relying solely on publicly-available foundation models and expecting a "one model fits all" solution is a recipe for disappointment. With thoughtful integration of the right complementary techniques, you can create AI assistants that truly understand your business.

129

13 Comments

As someone who specializes in solving the limitations of existing generative AI solutions or building new ones from scratch, my ideal client has a few key characteristics:They have thoroughly tested and understand the current limitations of off-the-shelf AI models and tools for their specific use case. They don't come to me with just a vague idea - they've done their homework on what's out there already and where the gaps are that need to be filled.They have a very clear and well-defined use case they need solved. It's not just a speculative "maybe we could use AI for X" idea. They know precisely what they need an AI model to do and what kind of inputs, outputs, and constraints it needs to work within.They have access to relevant data that can be used to train or fine-tune models. Having a large, high-quality dataset specifically for the target use case is critical for developing an effective AI solution.They have a prioritized list of questions/prompts and expected outputs. Even better - this list is organized by intent categories, difficulty levels, and importance to end-users. This focuses the AI development efforts on what really matters.They understand that AI development is fundamentally an iterative process of experimentation. We'll prototype, test, get feedback, refine, and repeat until we arrive at a production-ready model that meets their needs sufficiently.The ideal client is a collaborative partner who is deeply invested in the success of the project. They share domain expertise, ask excellent questions, provide constructive feedback, and have reasonable expectations around what can be achieved and how long it may take to get there.If this sounds like you and your requirements for generative AI, I'd love to discuss how I could help overcome your current limitations and take your AI use case to the next level. Reaching out is the first step!

4

Chatbot "Hallucinations" - or Just Bad Retrieval?When working with company chatbots and question-answering systems, a common complaint from users is that the AI seems to be "hallucinating" or making up information not contained in the underlying data. But more often than not, the root cause is not hallucination at all - it's simply a failure of the retrieval system to accurately find the right supporting information across documents.Take the example of a client I worked with recently who attempted to use an off-the-shelf SaaS chatbot offering to create an internal knowledge base system. They fed the tool their collection of procedures, product documentation, marketing materials, and more. However, the chatbot performed poorly because it could not effectively retrieve relevant information across this set of documents with many repeated sections like conclusions, financial details, etc.The client assumed the chatbot was hallucinating, making up responses wholecloth. But the real issue was simply the naive chunk retrieval technique breaking down in the face of redundant data across sources. The model wasn't hallucinating - it was just retrieving the wrong information from the wrong places.The solution was to invest effort into properly curating, classifying, and structuring the source data based on entities and content types. With some targeted preprocessing and organization using tools like named entity recognition and document classification, I was able to greatly improve the retrieval ability of a modern language model fine-tuned on this customized data set.The key lesson is that for most real-world enterprise use cases, simply throwing all your data at a generic chatbot offering will likely lead to subpar results. These cutting-edge AI models are incredibly powerful, but their performance still depends heavily on careful data preparation and modeling of the specific retrieval task.Companies should avoid thinking there will be a universal, out-of-the-box chatbot that can handle every internal knowledge need with zero configuration. At least for the first generation of these tools, the most successful deployments will focus on targeted use cases - customer support, report generation, HR knowledge bases, finance QA, and so on. By scoping the data and queries upfront, you can sidestep issues of "hallucination" in favor of accurate, well-grounded retrieval.So before calling out your chatbot for making things up, see if the root problem is actually one of data structure and retrieval modeling. The models may be incredibly capable, but they still need carefully engineered inputs and query setups to shine. Investing in that data preprocessing work upfront will be key to unlocking their full potential without fears of hallucinations.

8