bookmarks (hide)67
- Kedro versioned datasets can be mixed with incremental and partitioned datasets [ �� Unsure what kedro is? Check out this post. This was a question presented to
- Plan and execute agents promise faster, cheaper, and more performant task execution over previous agent designs. Learn how to build 3 types of planning agents in LangGraph in this post.
- As data engineers, you might have heard the terms functional data pipeline, factory pattern, singleton pattern, etc. One can quickly look up the implementation, but it can be tricky to understand what they are precisely and when to (& when not to) use them. Blindly following a pattern can help in some cases, but not knowing the caveats of a design will lead to hard-to-maintain and brittle code! While writing clean and easy-to-read code takes years of experience, you can accelerate that by understanding the nuances and reasoning behind each pattern. Imagine being able to design an implementation that provides the best extensibility and maintainability! Your colleagues (& future self) will be extremely grateful, your feature delivery speed will increase, and your boss will highly value your opinion. In this post, we will go over the specific code design patterns used for data pipelines, when and why to use them, and when not to use them, and we will also go over a few python specific techniques to help you write better pipelines. By the end of this post, you will be able to identify patterns in your data pipelines and apply the appropriate code design patterns. You will also be able to take advantage of pythonic features to write bug-free, maintainable code that is a joy to work on!
- Jeff from Sync Computing and Ian from SELECT sit down for an hour to discuss Snowflake and Databricks.
- If you haven't heard about Dagster Dagster is an open source data orchestrator for machine... Tagged with dataengineering, etl, dagster, kubernetes.
- Since 2021 we’ve been releasing the annual State of Data Engineering Report, a compilation of all the relevant categories that have a direct impact on data engineering infrastructure. In 2024, we see 3 primary trends that influence the categories which will be covered in this report.
- In today’s rapidly evolving tech landscape, the integration of advanced language models with robust data management systems is opening new horizons for data processing and analytics. One of the most…
- The recent release of this open-source project, LlamaFS, addresses the challenges associated with traditional file management systems, particularly in the context of overstuffed download folders, inefficient file organization, and the limitations of knowledge-based organization. These issues arise due to the manual nature of file sorting, which often leads to inconsistent structures and difficulty finding specific files. The disorganization in the file system hampers productivity and makes it challenging to locate important files quickly.
- I have been working with LangChain applications for quite a while now and as you might know there is always something new to learn in the GenAI universe. So a couple of weeks ago I was going through…
- TLDR — Extractive question answering is an important task for providing a good user experience in many applications. The popular Retriever-Reader framework for QA using BERT can be difficult to scale…
- Explore and run machine learning code with Kaggle Notebooks | Using data from multiple data sources
- As data scientists, we spend a lot of our time doing exploratory data analysis (EDA), cleaning data and making sure the data we use to generate insights is of good quality. Have you ever found…
- This talk explores the integration of Knowledge Graphs (KGs) and Large Language Models (LLM) to harness their combined power for improved natural language understanding. By leveraging KGs' structured knowledge and language models' text comprehension abilities, we can leverage the domain-specific–and potentially sensitive–data together with the general knowledge of LLMs. We also examine how language models can enhance KGs through knowledge extraction and refinement. The integration of these technologies presents opportunities in various domains, from question-answering to chatbots, fostering more intelligent and context-aware applications.
- I recently created a demo for some prospective clients of mine, demonstrating how to use Large Language Models (LLMs) together with graph databases like Neo4J. The two have a lot of interesting interactions, namely that you can now create knowledge graphs easier than ever before, by having AI find the graph entities and relationships from your unstructured data, rather than having to do all that manually. On top of that, graph databases also have some advantages for Retrieval Augmented Generation (RAG) applications compared to vector search, which is currently the prevailing approach to RAG.
- One of the key enablers of the ChatGPT magic can be traced back to 2017 under the obscure name of reinforcement learning with human feedback(RLHF). Large language models(LLMs) have become one of the most interesting environments for applying modern reinforcement learning(RL) techniques. While LLMs are great at deriving knowledge from vast amounts of text, RL can help to translate that knowledge into actions. That has been the secret behind RLHF.
- In this article, we will explore how we can use Llama2 for Topic Modeling without the need to pass every single document to the model. Instead, we will leverage BERTopic, a modular topic modeling technique that can use any LLM for fine-tuning topic representations.
- Large language models (LLMs) have proven to be valuable tools, but they often lack reliability. Many instances have surfaced where LLM-generated responses included false information. Specifically…
- Language Model Query Language
- Learn Prompting is the largest and most comprehensive course in prompt engineering available on the internet, with over 60 content modules, translated into 9 languages, and a thriving community.
publications (hide)177
- 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), page 1-6. IEEE, (July 2023)
- (2024)
- Technische Universität München, (2023)
- (2023)
- (2024)
- (2024)
- Forty-first International Conference on Machine Learning, (2024)
- The Twelfth International Conference on Learning Representations, (2024)
- (2024)
- Advances in Neural Information Processing Systems, 36, page 51991--52008. Curran Associates, Inc., (2023)
- (2023)
- (2023)
- Proceedings of the first Workshop on Information Extraction from Scientific Publications, page 67--77. Online, Association for Computational Linguistics, (November 2022)
- (2023)
- (November 2023)
- (2024)
- CoRR, (2019)
- (2018)
- CoRR, (2019)
- (2023)
